Attention level information generation method, device and equipment and computer readable medium

Abstract

The embodiment of the invention discloses an attention level information generation method and device, equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: acquiring a near-eye image sequence and an electroencephalogram signal; generating a fixation point coordinate sequence and a blink image sequence according to the near-eye image sequence; generating a noise time interval set corresponding to the electroencephalogram signal based on the fixation point coordinate sequence and the blink image sequence; denoising the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal; and generating an attention level information sequence based on the de-noised electroencephalogram signal, the fixation point information, the blink information sequence and a pre-trained attention level classification model. According to the embodiment, various eye movement information can be fully utilized to assist in removing the noise in the electroencephalogram signal, and the denoising quality is improved, so that the accuracy of the generated attention level information is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and computer-readable medium for generating attention level information. Background Art

[0002] Human attention is a complex cognitive function that plays a vital role in the cognitive process of an individual. The measurement and analysis of attention are of great significance for a variety of application fields, including but not limited to education, medical treatment, and human-computer interaction. At present, attention analysis mainly relies on the measurement of electroencephalogram (EEG) signals. EEG is a non-invasive neuroimaging technology that can record electrical signals generated by brain activity. By analyzing these signals, the attention state of an individual can be inferred. In recent years, the research on EEG signal denoising technology has been developing rapidly, especially in deep learning, graph signal processing, and data enhancement technology. Among them, denoising technology usually identifies noise by analyzing EEG signals and then separates it through relevant algorithms.

[0003] However, when using the above method to perform attention analysis, the following technical problems often occur: EEG signals are easily affected by noise caused by eye movements, blinking, and muscle activities, which can significantly affect the signal quality. The accuracy of identifying noise simply by analyzing EEG signals is low, resulting in low accuracy in identifying attention level information.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a method, apparatus, electronic device, and computer-readable medium for generating attention level information to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for generating attention level information, the method comprising: acquiring a near-eye image sequence and an electroencephalogram signal; generating a gaze point coordinate sequence and a blink image sequence based on the near-eye image sequence; generating a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence; denoising the electroencephalogram signal based on the noise time interval set to obtain a denoised electroencephalogram signal; generating an attention level information sequence based on the denoised electroencephalogram signal, the gaze point coordinate sequence, the blink image sequence and a pre-trained attention level classification model.

[0008] In a second aspect, some embodiments of the present disclosure provide an attention level information generating device, the device comprising: an acquisition unit, configured to acquire a near-eye image sequence and an electroencephalogram signal; a first generation unit, configured to generate a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence; a second generation unit, configured to generate a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence; a denoising unit, configured to denoise the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal; and a third generation unit, configured to generate an attention level information sequence based on the denoised electroencephalogram signal, the gaze point coordinate sequence, the blink image sequence and a pre-trained attention level classification model.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through a method for generating attention level information of the present disclosure, a variety of eye movement information can be fully utilized to assist in removing eye movement noise in the electroencephalogram signal. Compared with removing noise based on the analysis of a single electroencephalogram signal, analyzing multimodal information can more effectively distinguish the signal generated by brain activity and external noise, and improve the accuracy of noise recognition. Thus, the denoising quality and the accuracy of the generated attention level information can be improved. Specifically, the reason for the low denoising quality is that the electroencephalogram signal is easily affected by the noise caused by eye movement, blinking, and muscle activity, which will significantly affect the signal quality, and the accuracy of identifying noise by analyzing the electroencephalogram signal graph alone is low, resulting in low accuracy in identifying the attention level information. Based on this, the attention level information generation method of some embodiments of the present disclosure first obtains a near-eye image sequence and an electroencephalogram signal. Thus, multiple frames of near-eye images and unprocessed electroencephalogram signals can be obtained. Then, according to the above-mentioned near-eye image sequence, a gaze point coordinate sequence and a blink image sequence are generated. Thus, the multimodal information contained in the near-eye image sequence can be extracted to obtain the gaze point coordinate sequence and the blink image sequence required for denoising. Next, based on the above-mentioned fixation point coordinate sequence and the above-mentioned blink image sequence, a noise time interval set corresponding to the above-mentioned electroencephalogram signal is generated. Thus, the fixation point coordinate sequence and the blink image sequence can be integrated to determine the noise time interval set of the electroencephalogram signal. Then, according to the above-mentioned noise time interval set, the above-mentioned electroencephalogram signal is denoised to obtain a denoised electroencephalogram signal. Thus, the noise in the electroencephalogram signal can be removed according to the determined noise time interval. Finally, based on the above-mentioned denoised electroencephalogram signal, the above-mentioned fixation point coordinate sequence, the above-mentioned blink image sequence and the pre-trained attention level classification model, an attention level information sequence is generated. Thus, the attention level classification model can be used to combine the denoised electroencephalogram signal, the above-mentioned fixation point information and the above-mentioned blink information sequence trimodal information to identify the user's attention level information sequence. Also because denoising is performed by fusing electroencephalogram signals and multiple eye movement information, compared with removing noise based on analyzing a single electroencephalogram signal, analyzing multimodal information can more effectively distinguish signals generated by brain activity and external noise, and improve the accuracy of noise recognition. Thus, the denoising quality and the accuracy of the generated attention level information can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1is a flow chart of some embodiments of the method for generating attention level information according to the present disclosure;

[0014] Figure 2 is a schematic diagram of visualization of attention level information according to some embodiments of the attention level information generation method disclosed herein;

[0015] Figure 3 is a schematic structural diagram of some embodiments of the attention level information generating device according to the present disclosure;

[0016] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 The process 100 of some embodiments of the method for generating attention level information according to the present disclosure is shown. The method for generating attention level information comprises the following steps:

[0024] Step 101, acquiring a near-eye image sequence and an electroencephalogram signal.

[0025] In some embodiments, the execution subject (such as a computing device) of the attention level information generation method can obtain a near-eye image sequence and an electroencephalogram signal from an integrated device through a wired connection or a wireless connection. Wherein, the near-eye image sequence is a continuous multi-frame near-eye image. The near-eye image can be a single-eye image taken or a binocular image taken. Wherein, the above-mentioned integrated device includes a near-eye camera device and an electroencephalogram signal acquisition device, and the electroencephalogram signal acquisition device can be a standard EEG (Electro Encephalo Gram, electroencephalogram) machine or a portable EEG device. Wherein, the wearing method of the above-mentioned integrated device can be but is not limited to any one of the following: head-mounted, glasses-type or earring-type. In practice, the above-mentioned execution subject can obtain a near-eye image sequence and an electroencephalogram signal corresponding to a preset time period to synchronously acquire the electroencephalogram signal and the corresponding near-eye image sequence. For example, the integrated device can achieve data synchronization by sending a common marking signal to the near-eye camera device and the electroencephalogram signal acquisition device. The above-mentioned marking signal includes a TTL (Transistor-Transistor Logic, transistor-transistor logic) signal. It should be noted that the wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. For example, the execution subject may be a head-mounted display device.

[0026] Step 102: Generate a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence.

[0027] In some embodiments, the execution subject may generate a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence. The gaze point coordinates in the gaze point coordinate sequence may be three-dimensional coordinates. The blink image sequence may be each near-eye image representing blinking.

[0028] In some optional implementations of some embodiments, the execution subject may generate a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence through the following steps:

[0029] In the first step, each near-eye image in the near-eye image sequence is subjected to blink recognition processing to determine each near-eye image representing blinking. In practice, the execution subject may perform blink recognition processing on each near-eye image in the near-eye image sequence by using the eye aspect ratio (EAR) to determine each near-eye image representing blinking.

[0030] The second step is to determine the determined near-eye images representing blinking as a blinking image sequence. In practice, the above execution subject can arrange the determined near-eye images representing blinking as a blinking image sequence in ascending order of acquisition time.

[0031] The third step is to perform gaze point recognition processing on each near-eye image in the above-mentioned near-eye image sequence to obtain a gaze point coordinate sequence. The gaze point coordinates in the above-mentioned gaze point coordinate sequence correspond to the near-eye images in the above-mentioned near-eye image sequence. The gaze point coordinates may be coordinates in a three-dimensional coordinate system. In practice, the gaze point may be identified by eye tracking technology. The above-mentioned eye tracking technology is a technology that works by tracking the position and movement of the eyes, using a near-infrared or infrared light source and a camera to record reflections, and then calculating the position and gaze direction of the eyes.

[0032] Step 103: Generate a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence.

[0033] In some embodiments, the execution subject may generate a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence, wherein the noise time interval is a time interval corresponding to the electroencephalogram signal segment containing noise.

[0034] In some optional implementations of some embodiments, the execution subject may generate a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence through the following steps:

[0035] The first step is to generate a set of scanning time intervals according to the above-mentioned gaze point coordinate sequence and the above-mentioned near-eye image sequence, wherein the above-mentioned scanning time interval is the time interval corresponding to adjacent gaze points whose moving distance between adjacent gaze points is greater than a preset distance.

[0036] The second step is to generate a set of blink time intervals according to the blink image sequence, wherein the blink time intervals may be time intervals corresponding to near-eye images representing blinks.

[0037] The third step is to generate a noise time interval set corresponding to the electroencephalogram signal according to the above-mentioned saccade time interval set and the above-mentioned blink time interval set. In practice, the above-mentioned execution subject can determine the union of the above-mentioned saccade time interval set and the above-mentioned blink time interval set as the noise time interval set corresponding to the above-mentioned electroencephalogram signal.

[0038] In some optional implementations of some embodiments, the execution subject may generate a set of scanning time intervals according to the gaze point coordinate sequence and the near-eye image sequence through the following steps:

[0039] In the first step, for each two adjacent gaze point coordinates in the above gaze point coordinate sequence, perform the following steps:

[0040] The first sub-step is to determine the distance between the two gaze point coordinates as the gaze point distance. The distance between the two gaze point coordinates may be a straight-line distance between the two gaze point coordinates.

[0041] The second sub-step is to determine whether the gaze point distance is greater than a preset distance. The preset distance may be a preset distance. The specific setting of the preset distance is not limited.

[0042] A third sub-step, in response to determining that the gaze point distance is greater than the preset distance, determines the gaze point distance as the scanning distance.

[0043] The fourth sub-step is to generate a gaze point moving speed based on the above-mentioned scanning distance and the image time difference corresponding to the above-mentioned near-eye image sequence. The image time difference may be the time interval between two adjacent near-eye images collected. In practice, the above-mentioned execution subject may determine the ratio of the above-mentioned scanning distance and the above-mentioned image time difference as the gaze point moving speed.

[0044] In the second step, the generated movement speeds of each gaze point are determined as a set of scanning speeds.

[0045] In the third step, for each scanning speed in the above scanning speed set, perform the following steps:

[0046] In the first sub-step, the two gaze point coordinates corresponding to the above-mentioned scanning speed are determined as a scanning point coordinate sequence.

[0047] In the second sub-step, the two near-eye images corresponding to the above-mentioned scanning point coordinate sequence are determined as a scanning image sequence.

[0048] In a third sub-step, the image acquisition time corresponding to the first scanning image in the scanning image sequence is determined as the first time point.

[0049] The fourth sub-step is to determine the image acquisition time corresponding to the second scanning image in the scanning image sequence as the second time point.

[0050] The fifth sub-step is to combine the first time point and the second time point into a scanning time interval.

[0051] The fourth step is to determine the combined scanning time intervals as a scanning time interval set.

[0052] In some optional implementations of some embodiments, the execution subject may generate a set of blink time intervals according to the blink image sequence through the following steps:

[0053] The first step is to determine the blink type corresponding to each blink image in the blink image sequence. The blink type can represent closed eyes or open eyes. In practice, the execution entity can input the blink image into a pre-trained blink type recognition model to obtain the blink type corresponding to the blink image. The blink type recognition model can be a neural network model that takes the blink image as input data and the blink type corresponding to the blink image as output data. For example, the blink type recognition model can be a convolutional neural network model.

[0054] In the second step, the first two blink images in the blink image sequence are determined as an initial blink image group.

[0055] The third step is to perform the following loop steps based on the initial blink image group:

[0056] In the first sub-step, an initial blink image corresponding to the blink type representing closed eyes in the initial blink image group is determined as an eye-closed image.

[0057] In the second sub-step, the initial blink image of the initial blink image group, whose blink type represents open eyes, is determined as the open eyes image.

[0058] In a third sub-step, the image acquisition time corresponding to the eye-closed image is determined as a third time point.

[0059] The fourth sub-step is to determine the image acquisition time corresponding to the above-mentioned eye-opening image as the fourth time point.

[0060] The fifth sub-step is to combine the third time point and the fourth time point into a blink time interval.

[0061] The sixth sub-step, in response to determining that the initial blink image group is not the last two blink images in the above-mentioned blink image sequence, determines the two blink images arranged after the initial blink image group in the above-mentioned blink image sequence as the initial blink image group to update the initial blink image group, and based on the updated initial blink image group, executes the above-mentioned loop step again.

[0062] In the fourth step, the combined blink time intervals are determined as a blink time interval set.

[0063] Step 104: De-noise the EEG signal according to the noise time interval set to obtain a de-noised EEG signal.

[0064] In some embodiments, the execution subject may denoise the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal.

[0065] In some optional implementations of some embodiments, the execution subject may denoise the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal through the following steps:

[0066] The first step is to divide the above-mentioned EEG signal according to the above-mentioned noise time interval set to obtain a noise EEG signal segment set and a non-noise EEG signal segment set. In practice, the above-mentioned execution subject can determine the various signal segments in the above-mentioned EEG signal corresponding to the above-mentioned noise time interval set as the noise EEG signal segment set. Secondly, the above-mentioned execution subject can determine the remaining signal segments in the above-mentioned EEG signal as the non-noise EEG signal segment set.

[0067] The second step is to denoise each noisy EEG signal segment in the above-mentioned noisy EEG signal segment set to obtain a denoised EEG signal segment. In practice, the above-mentioned execution subject can denoise the above-mentioned noisy EEG signal segment by a wavelet transform algorithm to obtain a denoised EEG signal segment. Among them, the wavelet transform algorithm can denoise by threshold processing, setting the wavelet coefficients less than the threshold to zero, thereby removing noise. Among them, the selection of the threshold is usually not fixed, but is determined according to the characteristics of the signal and the noise level.

[0068] The third step is to splice the obtained denoised EEG signal segments and the above-mentioned non-noise EEG signal segment set in chronological order to obtain a denoised EEG signal. In practice, the above-mentioned execution subject can use the waverec function to splice the above-mentioned denoised EEG signal segments and the above-mentioned non-noise EEG signal segment set in chronological ascending order to obtain a denoised EEG signal.

[0069] In some optional implementations of some embodiments, the execution subject may denoise the noisy EEG signal segment to obtain the denoised EEG signal segment by the following steps:

[0070] In the first step, a preset component is determined as an initial first component, wherein the preset component may be a null value.

[0071] The second step is to construct a first component set and a second component set. The first component set may be a set for storing the first component. The second component set may be a set for storing the first component. The first component may be an approximate component, which refers to the high scale of the signal, i.e., low-frequency information. The second component may be a detail component, which refers to the low scale of the signal, i.e., high-frequency information. In practice, the execution subject may determine the empty set as the first component set and the second component set to construct the first component set and the second component set.

[0072] In the third step, based on the initial first component, the first component set, and the second component set, the following decomposition steps are performed on the noisy EEG signal segment:

[0073] In the first sub-step, the noisy EEG signal fragments are decomposed to obtain a first component and a second component. The frequency corresponding to the first component is less than the frequency corresponding to the second component. In practice, a denoising algorithm can be used to decompose the noisy EEG signal fragments. The denoising algorithm can be a discrete wavelet transform algorithm to decompose the noisy EEG signal fragments.

[0074] The second sub-step is to add the first component to the first component set to update the first component set.

[0075] The third sub-step is to add the second component to the second component set to update the second component set.

[0076] The fourth sub-step is to generate a signal characteristic value based on the first component and the initial first component. In practice, the above-mentioned execution subject can obtain the corresponding first component skewness value and the initial first component skewness value through the first component and the initial first component. Then, the absolute value of the first component skewness value and the initial first component skewness value can be taken. Then, the skewness value of the first component after taking the absolute value and the skewness value of the initial first component are subtracted, and the absolute value of the difference result is taken to obtain the signal characteristic value. Among them, the above-mentioned first component skewness value and the above-mentioned initial first component skewness value can be obtained by inputting the first component and the initial first component into the skewness calculation formula respectively.

[0077] The fifth sub-step is to generate a denoised EEG signal segment according to each second component in the updated second component set in response to determining that the signal characteristic value is greater than a preset threshold. The preset threshold may be pre-set. In practice, the above-mentioned execution subject may add each second component in the second component set to obtain a denoised EEG signal segment.

[0078] In the fourth step, in response to determining that the signal characteristic value is less than or equal to the above-mentioned preset threshold, the first component is determined as the initial first component to update the initial first component, and the second component is determined as the noise EEG signal segment to update the noise EEG signal segment. Based on the updated initial first component, the updated noise EEG signal segment, the updated first component set, and the updated second component set, the above-mentioned decomposition step is performed again.

[0079] The above-mentioned first step to the fourth step and their related contents, as an inventive point of the embodiment of the present disclosure, solve the technical problem that "the existing denoising technology has low accuracy in separating the features and noise contained in the EEG signal". The factors that lead to low accuracy in separating the features and noise contained in the EEG signal are often as follows: Most denoising technologies ignore the characteristics of noise and EEG signals at different scales, resulting in low denoising accuracy. If the above factors are solved, the denoising accuracy can be improved. In order to achieve this effect, first, the noisy EEG signal is decomposed for the first time using a denoising algorithm to obtain the first approximate component and the detail component. Thereafter, the approximate component is continuously decomposed until the signal characteristics are greater than a preset threshold, and the approximate component finally obtained can be noise. Finally, the noise is removed by the threshold processing method of the denoising algorithm. Therefore, the embodiment of the present disclosure uses a denoising algorithm to perform local analysis on the noisy EEG signal in both the time domain and the frequency domain, and decomposes the noisy EEG signal multiple times, which enables it to better detect the characteristics of noise and signal at different scales to determine more accurate noise areas and non-noise areas, and finally remove the noise. Thereby, the denoising accuracy is improved.

[0080] Step 105, generating an attention level information sequence based on the denoised EEG signal, the gaze point coordinate sequence, the blink image sequence and the pre-trained attention level classification model.

[0081] In some embodiments, the execution subject may generate an attention level information sequence based on the denoised EEG signal, the gaze point coordinate sequence, the blink image sequence and a pre-trained attention level classification model. The attention level classification model may be a neural network model that processes input information using a multimodal deep learning method to perform attention level classification. θ , where θ represents the model parameters. Cross entropy loss function The formula is as follows:

[0082]

[0083] The true label y can be pre-labeled. In the above cross entropy loss function Represents the model’s prediction function for the k-th class’s attention level, is the predicted probability that the i-th sample is the k-th attention level. The predicted probability of all samples can be expressed as Above represents the cross entropy loss, y i，k It indicates the sign function (output is 0 or 1), indicating whether the i-th sample belongs to the k-th class. If yes, the output is 1, otherwise it is 0. B indicates the batch size, K is the total number of classes, represents the input features of the denoised EEG signal corresponding to the i-th sample, Represents the input features of the gaze coordinate sequence corresponding to the i-th sample, represents the input features of the blink image sequence corresponding to the i-th sample. The above log represents the natural logarithm, whose base is the mathematical constant e (approximately equal to 2.71828). The cross entropy loss is calculated by calculating the predicted probability of all samples in all categories of a given batch. It is determined by taking the average value of the negative log-likelihood of the true label y. The above-mentioned attention classification level model can use Adam to optimize the model parameters to minimize the above-mentioned cross entropy loss function, thereby obtaining the best output result. The above-mentioned attention level information sequence is an information sequence in which the attention level level is marked in chronological order. Among them, the attention level information may include the attention level level. The attention level level can characterize the user's attention level to at least one object. The attention level information in the above-mentioned attention level information sequence corresponds to the gaze point coordinates in the above-mentioned gaze point coordinate sequence.

[0084] In some optional implementations of some embodiments, the above execution subject may generate the attention level information sequence through the following steps:

[0085] In the first step, the denoised EEG signal is input into the first two-dimensional convolutional network included in the attention level classification model to obtain the first feature map corresponding to the denoised EEG signal. Among them, the attention level classification model may include a first two-dimensional convolutional network, a second two-dimensional convolutional network, a three-dimensional convolutional network, a fully connected layer and an output layer. The first two-dimensional convolutional network and the second two-dimensional convolutional network may include respective two-dimensional convolutional layers. The three-dimensional convolutional network may include respective three-dimensional convolutional layers. Among them, the structures of the first two-dimensional convolutional network and the second two-dimensional convolutional network are exactly the same, and both may include a two-dimensional convolutional layer for initial feature extraction, a deep two-dimensional convolutional layer for further feature extraction, and an independent two-dimensional convolutional layer for capturing frequency information. The three-dimensional convolutional network is a convolutional network including multiple three-dimensional convolutional layers. The first two-dimensional convolutional network and the second two-dimensional convolutional network may be in parallel relationship, and both are connected to the three-dimensional convolutional network, the three-dimensional convolutional network is connected to the fully connected layer, the fully connected layer is connected to the output layer, and the output layer finally outputs the result. In practice, the above-mentioned execution entity can decompose the denoised EEG signal into a band signal set, and then input the band signal set into the above-mentioned first two-dimensional convolutional network to obtain the first feature map corresponding to the above-mentioned denoised EEG signal. Among them, the above-mentioned band signal set can be a plurality of different bands obtained by decomposing the denoised EEG signal through a filter. The above-mentioned band signal set may include but is not limited to: delta wave (0.5-4Hz), theta wave (4-8Hz) and alpha wave (8-13Hz). The above-mentioned filter may be a Butterworth filter or a Chebyshev filter.

[0086] The second step is to input the above-mentioned fixation point coordinate sequence into the second two-dimensional convolutional network included in the above-mentioned attention level classification model to obtain a second feature map corresponding to the above-mentioned fixation point coordinate sequence. In practice, the above-mentioned execution subject can input the above-mentioned fixation point coordinate sequence into the above-mentioned second two-dimensional convolutional network, and extract feature information through the above-mentioned second two-dimensional convolutional network to obtain a second feature map corresponding to the above-mentioned fixation point coordinate sequence.

[0087] In the third step, the first feature map and the second feature map are input into the three-dimensional convolutional network to obtain first fused feature information, wherein the first fused feature information is bimodal fused feature information obtained by fusing the electroencephalogram signal and the gaze point coordinate sequence using multiple three-dimensional convolutional layers.

[0088] The fourth step is to fuse the above-mentioned blink image sequence and the above-mentioned first fusion feature information into the second fusion feature information. Among them, the second fusion feature information is a tri-modal fusion information that fuses the blink image sequence and the first fusion feature information. In practice, the above-mentioned execution entity can determine each blink type corresponding to the above-mentioned blink image sequence as a blink type sequence. Then, the above-mentioned blink type sequence can be converted into a numerical sequence. Among them, the numerical value corresponding to the blink type that characterizes blinking can be 1. The numerical value corresponding to the blink type that characterizes not blinking can be 0. Finally, the above-mentioned numerical sequence and the above-mentioned first fusion feature information can be spliced ​​into the second fusion feature information.

[0089] In the fifth step, the second fused feature information is input into the fully connected layer to obtain the target feature information. In practice, the execution subject may classify the target feature information by attention level through the fully connected layer.

[0090] In step 6, the target feature information is input into the output layer to obtain the attention level information. In practice, the execution subject may determine the attention level grade obtained by classification as the attention level information.

[0091] The above-mentioned first step to the sixth step and their related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem that "the existing deep learning model for attention level classification lacks the fusion of feature information other than the EEG signal features, resulting in low accuracy of attention classification". The factors that lead to the low accuracy of attention classification are often as follows: the existing deep learning model for attention level classification can often only obtain the attention classification result by processing the EEG signal after denoising at a single place. There is a lack of integration and fusion of other eye movement information, and these eye movement information will also affect the deep learning model's assessment of the attention level level, resulting in low accuracy of the attention level classification result. If the above factors are solved, the accuracy of the attention level classification result can be improved. In order to achieve this effect, first, an attention level classification model is constructed. Among them, the attention level classification model has two two-dimensional neural networks with exactly the same structure, and the two two-dimensional neural networks can be in parallel and connected to a three-dimensional convolutional network. The three-dimensional convolutional network can be connected to the fully connected layer, the output layer, and can be a series structure. The two-dimensional neural network structure can be used to fuse the denoised EEG signal and the gaze point coordinate sequence to obtain a first feature map. Afterwards, the first feature map is fused with the numerical sequence obtained from the blink image sequence to obtain a second feature map that integrates the trimodal information. Finally, the second feature map is input into the fully connected layer to classify the attention level, and the attention level information is output through the output layer. Therefore, the embodiment of the present disclosure combines the denoised EEG signal and a variety of eye movement information, and increases the basis for judging the attention level grade through the attention level classification model that integrates the trimodal information. Thereby, the accuracy of the attention level classification result is improved.

[0092] Optionally, the above subject may further perform the following steps:

[0093] The first step is to visualize the various attention level information in the attention level information sequence according to the various attention level grades included in the attention level information sequence. The attention level grades can be divided into three grades of low, medium and high according to the duration of fixating a certain point and distinguished by different colors. The duration of fixating a certain point is a self-set threshold. In practice, the execution entity can display the various attention level grades in the form of a timeline with corresponding colors to visualize the various attention level information in the attention level information sequence. As an example, the visualization result of the visualization of the various attention level information in the attention level information sequence can be referred to. Figure 2 (a) in . Figure 2 In (a), the darker the color, the higher the corresponding attention level.

[0094] In the second step, based on the above-mentioned gaze point coordinate sequence, the eye movement trajectory is superimposed and displayed in the target observation space. The above-mentioned target observation space can be the target scene that the user is gazing at when acquiring the near-eye image sequence. The target scene can be the three-dimensional scene that the user is currently watching. The above-mentioned eye movement trajectory is the gaze point movement trajectory corresponding to the gaze point coordinate sequence. In practice, the above-mentioned execution entity can connect the various gaze point coordinates in the above-mentioned gaze point coordinate sequence in the above-mentioned target observation space in sequence as an eye movement trajectory, and superimpose the eye movement trajectory in the target observation space as an example. The visualization result of visualizing the various attention level information in the above-mentioned attention level information sequence can be referred to. Figure 2 (b) in .

[0095] The third step is to visualize the attention level information sequence in the target observation space according to the above-mentioned gaze point coordinate sequence and the above-mentioned attention level information sequence. In practice, when the above-mentioned execution subject accumulates eye movement trajectories for a period of time, the above-mentioned execution subject will record the user's gaze point position and the corresponding attention level at every moment according to the above-mentioned gaze point coordinate sequence and the above-mentioned attention level information sequence, and obtain a series of gaze point coordinate points with different attention levels. These gaze points will be weighted and superimposed according to the attention level levels obtained by the above-mentioned attention level component model. These points are then smoothed with a Gaussian kernel to obtain a heat map that can be directly viewed by the user. Among them, the area with a higher attention level and a more concentrated gaze point coordinate point has a brighter color on the heat map. As an example, the above-mentioned heat map that can be directly viewed by the user can refer to Figure 2 (c) in .

[0096] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through a method for generating attention level information of the present disclosure, a variety of eye movement information can be fully utilized to assist in removing eye movement noise in the electroencephalogram signal. Compared with removing noise based on the analysis of a single electroencephalogram signal, analyzing multimodal information can more effectively distinguish the signal generated by brain activity and external noise, and improve the accuracy of noise recognition. Thus, the denoising quality and the accuracy of the generated attention level information can be improved. Specifically, the reason for the low denoising quality is that the electroencephalogram signal is easily affected by the noise caused by eye movement, blinking, and muscle activity, which will significantly affect the signal quality, and the accuracy of identifying noise by analyzing the electroencephalogram signal graph alone is low, resulting in low accuracy in identifying the attention level information. Based on this, the attention level information generation method of some embodiments of the present disclosure first obtains a near-eye image sequence and an electroencephalogram signal. Thus, multiple frames of near-eye images and unprocessed electroencephalogram signals can be obtained. Then, according to the above-mentioned near-eye image sequence, a gaze point coordinate sequence and a blink image sequence are generated. Thus, the multimodal information contained in the near-eye image sequence can be extracted to obtain the gaze point coordinate sequence and the blink image sequence required for denoising. Next, based on the above-mentioned fixation point coordinate sequence and the above-mentioned blink image sequence, a noise time interval set corresponding to the above-mentioned electroencephalogram signal is generated. Thus, the fixation point coordinate sequence and the blink image sequence can be integrated to determine the noise time interval set of the electroencephalogram signal. Then, according to the above-mentioned noise time interval set, the above-mentioned electroencephalogram signal is denoised to obtain a denoised electroencephalogram signal. Thus, the noise in the electroencephalogram signal can be removed according to the determined noise time interval. Finally, based on the above-mentioned denoised electroencephalogram signal, the above-mentioned fixation point coordinate sequence, the above-mentioned blink image sequence and the pre-trained attention level classification model, an attention level information sequence is generated. Thus, the attention level classification model can be used to combine the denoised electroencephalogram signal, the above-mentioned fixation point information and the above-mentioned blink information sequence trimodal information to identify the user's attention level information sequence. Also because denoising is performed by fusing electroencephalogram signals and multiple eye movement information, compared with removing noise based on analyzing a single electroencephalogram signal, analyzing multimodal information can more effectively distinguish signals generated by brain activity and external noise, and improve the accuracy of noise recognition. Thus, the denoising quality and the accuracy of the generated attention level information can be improved.

[0097] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an attention level information generating device, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0098] like Figure 3As shown, the attention level generating device 300 of some embodiments includes: an acquisition unit 301, a first generation unit 302, a second generation unit 303, a denoising unit 304 and a third generation unit 305. The acquisition unit 301 is configured to acquire a near-eye image sequence and an electroencephalogram signal; the first generation unit 302 is configured to generate a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence; the second generation unit 303 is configured to generate a noise time interval set corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence; the denoising unit 304 is configured to denoise the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal; the third generation unit 305 is configured to generate an attention level information sequence based on the denoised electroencephalogram signal, the gaze point coordinate sequence, the blink image sequence and the pre-trained attention level classification model.

[0099] It is understood that the units described in the device 300 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units contained therein, and will not be described in detail here.

[0100] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 4 As shown, the electronic device 400 may include a processing device 401 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0102] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0103] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0104] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0105] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0106] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a near-eye image sequence and an electroencephalogram signal; generates a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence; generates a noise time interval set corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence; denoises the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal; generates an attention level information sequence based on the denoised electroencephalogram signal, the gaze point coordinate sequence, the blink image sequence and a pre-trained attention level classification model.

[0107] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a first generation unit, a second generation unit, a denoising unit, and a third generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring a target near-eye image sequence and an electroencephalogram signal".

[0110] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating attention level information, comprising: Acquire near-eye image sequences and EEG signals; generating a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence; Based on the gaze point coordinate sequence and the blink image sequence, generating a set of noise time intervals corresponding to the electroencephalogram signal; Denoising the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal; An attention level information sequence is generated based on the denoised electroencephalogram signal, the gaze point coordinate sequence, the blink image sequence and a pre-trained attention level classification model.

2. The method according to claim 1, wherein: The step of generating a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence comprises: Performing blink recognition processing on each near-eye image in the near-eye image sequence to determine each near-eye image representing a blink; Determining the determined near-eye images representing blinking as a blinking image sequence; Performing gaze point recognition processing on each near-eye image in the near-eye image sequence to obtain a gaze point coordinate sequence, wherein the gaze point coordinates in the gaze point coordinate sequence correspond to the near-eye images in the near-eye image sequence.

3. The method according to claim 1, wherein: The step of generating a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence includes: generating a set of scanning time intervals according to the gaze point coordinate sequence and the near-eye image sequence; generating a set of blink time intervals according to the blink image sequence; A noise time interval set corresponding to the electroencephalogram signal is generated according to the scan time interval set and the blink time interval set.

4. The method according to claim 3, wherein: The step of generating a set of scanning time intervals according to the gaze point coordinate sequence and the near-eye image sequence includes: For each two adjacent gaze point coordinates in the gaze point coordinate sequence, the following steps are performed: Determine the distance between the two fixation point coordinates as the fixation point distance; Determining whether the gaze point distance is greater than a preset distance; In response to determining that the gaze point distance is greater than the preset distance, determining the gaze point distance as a scanning distance; generating a gaze point movement speed based on the scanning distance and the image time difference corresponding to the near-eye image sequence; Determine the generated movement speeds of each fixation point as a scanning speed set; For each scanning speed in the set of scanning speeds, the following steps are performed: Determining the two gaze point coordinates corresponding to the scanning speed as a scanning point coordinate sequence; Determine two near-eye images corresponding to the scanning point coordinate sequence as a scanning image sequence; Determine the image acquisition time corresponding to the first scanning image in the scanning image sequence as the first time point; Determining the image acquisition time corresponding to the second scanning image in the scanning image sequence as the second time point; The first time point and the second time point are combined into a glance time interval; and the combined glance time intervals are determined as a glance time interval set.

5. The method according to claim 3, wherein: The step of generating a set of blink time intervals according to the blink image sequence comprises: Determine the blink type corresponding to each blink image in the blink image sequence; Determining the first two blink images in the blink image sequence as an initial blink image group; Based on the initial set of blink images, perform the following loop steps: Determine an initial blink image in the initial blink image group whose corresponding blink type represents closed eyes as an eye-closed image; Determine an initial blink image in the initial blink image group whose corresponding blink type represents open eyes as an open eye image; Determining the image acquisition time corresponding to the closed-eye image as the third time point; Determining the image acquisition time corresponding to the eye-opening image as the fourth time point; combining the third time point and the fourth time point into a blink time interval; In response to determining that the initial blink image group is not the last two blink images in the blink image sequence, determining two blink images in the blink image sequence that are arranged after the initial blink image group as the initial blink image group to update the initial blink image group, and executing the loop step again based on the updated initial blink image group; The combined blink time intervals are determined as a blink time interval set.

6. The method according to claim 1, wherein: The step of denoising the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal includes: Dividing the electroencephalogram signal according to the noise time interval set to obtain a set of noisy electroencephalogram signal segments and a set of non-noise electroencephalogram signal segments; For each noisy EEG signal segment in the noisy EEG signal segment set, denoising the noisy EEG signal segment to obtain a denoised EEG signal segment; The obtained denoised electroencephalogram signal segments and the set of non-noise electroencephalogram signal segments are spliced ​​in chronological order to obtain a denoised electroencephalogram signal.

7. The method according to claim 1, wherein: The method further comprises: According to each attention level grade included in the attention level information sequence, visualizing each attention level information in the attention level information sequence; According to the gaze point coordinate sequence, superimposing and displaying the eye movement trajectory in a target observation space, wherein the target observation space corresponds to the scene viewed by the user when acquiring the near-eye image sequence; According to the gaze point coordinate sequence and the attention level information sequence, the attention level information sequence is visualized in the target observation space.

8. An attention level information generating device, comprising: an acquisition unit configured to acquire a near-eye image sequence and an electroencephalogram signal; A first generating unit is configured to generate a gaze point coordinate sequence and a blink image sequence according to the near-eye image sequence; A second generating unit is configured to generate a set of noise time intervals corresponding to the electroencephalogram signal based on the gaze point coordinate sequence and the blink image sequence; a denoising unit configured to denoise the electroencephalogram signal according to the noise time interval set to obtain a denoised electroencephalogram signal; The third generating unit is configured to generate an attention level information sequence based on the denoised electroencephalogram signal, the gaze point coordinate sequence, the blink image sequence and a pre-trained attention level classification model.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images