User state recognition method and device, computer device, and storage medium

Abstract

The application relates to a user state recognition method and device, computer equipment and a storage medium, and is applied to the field of artificial intelligence. The method comprises the following steps: detecting each facial image in a facial video of a to-be-recognized user to obtain a plurality of eye images; the facial video is collected by a camera arranged on an interactive terminal; each facial image and the eye image corresponding to each facial image are input into a pre-trained neural network model to obtain a gaze point position corresponding to each eye image; an eye movement track of the to-be-recognized user in the operation process of the interactive terminal is generated according to each gaze point position; and the user state of the to-be-recognized user is determined based on the similarity between the eye movement track and a plurality of preset reference eye movement tracks. The facial image of the user is collected by the ordinary camera arranged on the interactive terminal, and the user state is determined, so that the user can be alarmed when the user state is abnormal, and the self-service of the user of the self-service interactive terminal is supervised.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a user state recognition method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Currently, bank branches widely use self-service terminals to provide self-service to users. However, self-service through these terminals differs from manual processing. Manual processing allows for transactions to be completed under the supervision of bank staff, and any abnormal user behavior can be manually identified. Self-service through terminals lacks supervision, making it difficult to identify some suspicious users. For example, users may be coerced into transferring funds using their bank cards, or they may be using other people's bank cards for illegal transfers. Therefore, monitoring self-service through these terminals is a pressing issue that needs to be addressed. Summary of the Invention

[0003] Therefore, it is necessary to address the aforementioned technical problem of lack of supervision in self-service conducted through self-service interactive terminals by providing a user status identification method, device, computer equipment, computer-readable storage medium, and computer program product.

[0004] Firstly, this application provides a user status recognition method, applied to an interactive terminal associated with financial transactions. The method includes:

[0005] Each facial image in the facial video of the user to be identified is detected to obtain multiple eye images of the user; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal;

[0006] Each facial image and its corresponding eye image are input into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image.

[0007] Based on the position of each gaze point, the eye movement trajectory of the user to be identified during the operation of the interactive terminal is generated;

[0008] The user state of the user to be identified is determined based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group.

[0009] In one embodiment, the baseline eye-tracking trajectory for each user state is determined in the following manner:

[0010] For each user state, obtain the sample gaze point location sequence set and the initial center sequence for that user state;

[0011] Repeat the following steps until the loop termination condition is met:

[0012] From the fixation point positions contained in each fixation point position sequence in the sample fixation point position sequence set, determine the associated fixation point positions of each fixation point position in the current center sequence; based on each fixation point position in the current center sequence and the associated fixation point positions of each fixation point position, determine a new center sequence; obtain the difference between a first distance corresponding to the new center sequence and a second distance corresponding to the current center sequence; the first distance represents the average distance between the new center sequence and each sample fixation point position sequence, and the second distance represents the average distance between the current center sequence and each sample fixation point position sequence;

[0013] If the difference is greater than the threshold, then the new center sequence is taken as the current center sequence;

[0014] The loop termination condition is: the difference is less than or equal to the threshold.

[0015] After the loop termination condition is met, the eye-tracking trajectory formed by the obtained center sequence is used as the reference eye-tracking trajectory for the user state.

[0016] In one embodiment, determining the associated fixation position of each fixation position in the current center sequence from the fixation positions contained in each sample fixation position sequence of the sample fixation position sequence set includes:

[0017] Construct a distance matrix between each sample gaze point position sequence and the current center sequence; each element in the distance matrix represents the distance between a set of sample gaze point positions corresponding to the position of that element;

[0018] Based on the distances in the distance matrix, the associated gaze point positions of each gaze point position in the current center sequence are determined.

[0019] In one embodiment, determining a new center sequence based on the fixation points in the current center sequence and the associated fixation points of those fixation points includes:

[0020] For each fixation point position in the current center sequence, the mean of the fixation point position and the associated fixation point positions is obtained, and used as the new fixation point position corresponding to the fixation point position.

[0021] The new center sequence is obtained based on the new fixation point position corresponding to each fixation point position.

[0022] In one embodiment, detecting individual facial images in the facial video of the user to be identified to obtain multiple eye images of the user to be identified includes:

[0023] Key point detection is performed on each of the facial images to obtain the eye images in each facial image;

[0024] The size of each eye image is adjusted to obtain an adjusted eye image of the same size;

[0025] Each adjusted eye image is then subjected to equalization processing, and the equalized adjusted eye images are used as multiple eye images of the user to be identified.

[0026] In one embodiment, before determining the user state of the user to be identified based on the similarity between the eye movement trajectory and a plurality of preset benchmark eye movement trajectories, the method further includes:

[0027] A distance matrix is ​​constructed based on the positions of each fixation point in the eye movement trajectory and the positions of each fixation point in any reference eye movement trajectory; each element in the distance matrix represents the distance between a set of fixation point positions corresponding to the position of that element.

[0028] Determine the normalized path in the distance matrix and obtain the cumulative distance corresponding to the normalized path; the normalized path is the shortest path from the starting element to the ending element in the distance matrix.

[0029] Based on the cumulative distance, the similarity between the eye movement trajectory and any reference eye movement trajectory is obtained.

[0030] In one embodiment, determining the user state of the user to be identified based on the similarity between the eye movement trajectory and a plurality of preset benchmark eye movement trajectories includes:

[0031] Based on the similarity, a target reference eye movement trajectory is determined from the plurality of reference eye movement trajectories; the similarity of the target reference eye movement trajectory is greater than the similarity of the other reference eye movement trajectories.

[0032] The user state corresponding to the target baseline eye movement trajectory is determined as the user state of the user to be identified.

[0033] In one embodiment, the eye images corresponding to each gaze point location carry a timestamp, the timestamp indicating the time when the facial image corresponding to the eye image was captured;

[0034] The step of generating the eye movement trajectory of the user to be identified during the operation of the interactive terminal based on the position of each gaze point includes:

[0035] By connecting the various gaze points in order of the timestamps from morning to night, the eye movement trajectory of the user to be identified during the operation of the interactive terminal can be obtained.

[0036] Secondly, this application also provides a user status recognition device. Located in an interactive terminal associated with financial transactions, the device includes:

[0037] The image detection module is used to detect each facial image in the facial video of the user to be identified, and obtain multiple eye images of the user to be identified; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal;

[0038] The position determination module is used to input the face images and the corresponding eye images into a pre-trained neural network model to obtain the gaze point positions corresponding to each eye image.

[0039] The trajectory generation module is used to generate the eye movement trajectory of the user to be identified during the operation of the interactive terminal based on the position of each gaze point;

[0040] The state determination module is used to determine the user state of the user to be identified based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group.

[0041] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0042] Each facial image in the facial video of the user to be identified is detected to obtain multiple eye images of the user; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal;

[0043] Each facial image and its corresponding eye image are input into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image.

[0044] Based on the position of each gaze point, the eye movement trajectory of the user to be identified during the operation of the interactive terminal is generated;

[0045] The user state of the user to be identified is determined based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group.

[0046] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0047] Each facial image in the facial video of the user to be identified is detected to obtain multiple eye images of the user; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal;

[0048] Each facial image and its corresponding eye image are input into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image.

[0049] Based on the position of each gaze point, the eye movement trajectory of the user to be identified during the operation of the interactive terminal is generated;

[0050] The user state of the user to be identified is determined based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group.

[0051] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0052] Each facial image in the facial video of the user to be identified is detected to obtain multiple eye images of the user; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal;

[0053] Each facial image and its corresponding eye image are input into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image.

[0054] Based on the position of each gaze point, the eye movement trajectory of the user to be identified during the operation of the interactive terminal is generated;

[0055] The user state of the user to be identified is determined based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group.

[0056] The aforementioned user status recognition method, device, computer equipment, storage medium, and computer program product detects various facial images of the user to be identified, obtains corresponding eye images, processes each facial image and corresponding eye image using a neural network model to obtain the gaze point position, generates an eye movement trajectory based on the gaze point position, and finally determines the user status based on the similarity between the eye movement trajectory and a reference eye movement trajectory. This method uses a regular camera on the interactive terminal to capture the user's facial images, analyzes the eye images within the facial images to identify the user's eye movement trajectory, and thus determines the user status. This allows for alerts when the user's status is abnormal, thereby enabling supervision of the user's self-service at the self-service interactive terminal. Attached Figure Description

[0057] Figure 1 This is an application environment diagram of the user state recognition method in one embodiment;

[0058] Figure 2 This is a flowchart illustrating a user state recognition method in one embodiment;

[0059] Figure 3 This is a flowchart illustrating the steps for determining the baseline eye movement trajectory in one embodiment;

[0060] Figure 4 This is a schematic diagram of a sequence of multiple gaze points in one embodiment;

[0061] Figure 5 This is a schematic diagram of facial key points in one embodiment;

[0062] Figure 6 This is a schematic diagram of the resolving path of the DTW algorithm in one embodiment;

[0063] Figure 7 This is a flowchart illustrating the user state recognition method in another embodiment;

[0064] Figure 8 This is a structural block diagram of a user state recognition device in one embodiment;

[0065] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0068] The user status recognition method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown is illustrated. The interactive terminal 102 is an interactive terminal associated with financial services; for example, it could be an interactive terminal in a bank branch. The interactive terminal 102 is equipped with a camera, which can be a regular camera, used to capture facial video of the user. The interactive terminal 102 also includes a processing system for detecting each facial image in the facial video of the user to be identified captured by the camera, obtaining multiple eye images of the user to be identified. Then, each facial image and its corresponding eye image are input into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image. Further, based on the gaze point positions, the eye movement trajectory of the user to be identified during operation of the interactive terminal is generated. Finally, based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories, the user state of the user to be identified is determined. The intelligent interactive terminal and method based on gaze estimation provided in this application can acquire the user's eye movement data through the ordinary camera built into the interactive terminal, analyze the user state, and thus issue alarms when the user state is abnormal, thereby realizing the supervision of user self-service at the self-service interactive terminal.

[0069] In one embodiment, such as Figure 2 As shown, a user state recognition method is provided, which can be applied to... Figure 1 Taking the interactive terminal 102 as an example, the following steps are included:

[0070] Step S210: Detect each facial image in the facial video of the user to be identified to obtain multiple eye images of the user to be identified; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal.

[0071] Among them, the user to be identified is the user who is operating the interactive terminal.

[0072] The camera can be a regular camera, and can be set up as follows: Figure 1 The interactive terminal 102 shown is located above the display screen, at a position where the user's facial video can be captured.

[0073] In practice, facial videos of the user to be identified can be captured using a regular camera on the interactive terminal. Each facial image in the video has a timestamp indicating the time when the corresponding facial image was captured / acquired. After obtaining each facial image, calibration and scaling operations can be performed on each facial image, and key points around the eyes can be located. Based on the located key points, eye images are cropped from each facial image, where these eye images have the same timestamp as the corresponding facial image before cropping. After obtaining the eye images, further preprocessing operations can be performed on each eye image to improve its quality. The eye images are sorted according to their timestamps to obtain an eye image sequence. Based on this sequence, the changes in pupil position in each frame of the facial video can be determined. These changes in position and the corresponding timestamps can be used to simulate eye movement.

[0074] Step S220: Input each face image and the corresponding eye image of each face image into the pre-trained neural network model to obtain the gaze point position corresponding to each eye image.

[0075] Among them, the neural network model can be a convolutional neural network (CNN).

[0076] In practice, end-to-end eye tracking is learned through a neural network model, without the need for any manually designed features. Each facial image and its corresponding eye image are used as input variables to the neural network model. Through the model's inference and computation, the gaze point position of each eye image can be obtained. It is crucial that the input to the neural network model ensures a one-to-one correspondence between each facial image and eye image, thus enabling the capture of the user's eye movement trajectory during interaction with the interactive terminal.

[0077] Step S230: Generate the eye movement trajectory of the user to be identified during the operation of the interactive terminal based on the position of each gaze point.

[0078] In the specific implementation, the eye movement trajectory is a time-series data composed of multiple fixation point positions, which is located as P = {p1, p2, ..., p...} m}, p i =(x i ,y i ) represents time t iThe fixation point position, t1 <t2<,...,t n When a user observes something, the position of their gaze is constantly changing and fluctuates with changes in the brain's thinking. These changes in the position of the gaze (eye movement trajectory) constitute the user's eye movement behavior when observing the whole object, and therefore the user's state can be determined based on the eye movement trajectory.

[0079] Step S240: Based on the similarity between the eye-tracking trajectory and multiple preset benchmark eye-tracking trajectories, determine the user status of the user to be identified; each benchmark eye-tracking trajectory has a corresponding user status, and the benchmark eye-tracking trajectory is obtained by calculating the common features of the eye-tracking behavior patterns of the corresponding user group.

[0080] Eye movement patterns, typically representing a series of alternating fixations and saccades, provide information about a participant's search behavior and can directly or indirectly reflect a person's psychological activities and behavioral awareness. Therefore, a user's state can be identified by classifying their eye movement patterns.

[0081] The user's state can be divided into two types: normal and abnormal. These state patterns are also the basic actions performed by the eyes during the process. Therefore, the user's business processing is modeled as a sequence of these patterns. The similarity between the eye movement trajectory of the user to be identified and the baseline eye movement trajectory of these two states is calculated. The user to be identified is considered to be in the state with the highest similarity to the baseline eye movement trajectory of the user transition.

[0082] In practice, the types of user states can be determined in advance. Then, for each user state, sample data of the corresponding user group can be collected. The sample data may include the sample gaze position sequence of multiple sample users in the user state. Based on the sample gaze position sequence, the common features of this user group can be extracted, that is, the center sequence corresponding to the sample gaze position sequence. The eye movement trajectory formed by the center sequence is used as the reference eye movement trajectory of the user state.

[0083] After determining the eye movement trajectory of the user to be identified, the eye movement trajectory can be compared with the baseline eye movement trajectory corresponding to each user state to determine the similarity. The user state to which the baseline eye movement trajectory with the highest similarity belongs is taken as the user state of the user to be identified.

[0084] Furthermore, after determining the user status of the user to be identified, a pre-set strategy can be executed. Specifically, when the user status is normal, no action is required; when the user status is abnormal, an alarm message can be generated and sent to the alarm terminal, so that the personnel at the alarm terminal can handle it, for example, by notifying the user to be identified to go to the manual window for processing.

[0085] In the aforementioned user status recognition method, each facial image of the user to be identified is detected to obtain the corresponding eye images. A neural network model is then used to process each facial image and its corresponding eye image to obtain the gaze point position. An eye movement trajectory is generated based on the gaze point position, and finally, the user status is determined based on the similarity between the eye movement trajectory and a reference eye movement trajectory. This method uses a regular camera on the interactive terminal to capture facial images of the user. By analyzing the eye images within the facial images, the user's eye movement trajectory is identified, thereby determining the user status. This allows for alerts when the user's status is abnormal, thus enabling supervision of the self-service functions of the self-service interactive terminal.

[0086] In one exemplary embodiment, such as Figure 3 As shown, the baseline eye-tracking trajectory for each user state is determined through the following steps:

[0087] Step S310: For each user state, obtain the sample gaze point location sequence set and the initial center sequence under the user state;

[0088] Repeat the following steps until the loop termination condition is met:

[0089] Step S320: Determine the associated fixation locations of each fixation location in the current center sequence from the fixation locations contained in each sample fixation location sequence in the sample fixation location sequence set.

[0090] Step S330: Determine a new center sequence based on the positions of each fixation point in the current center sequence and the associated fixation point positions of each fixation point.

[0091] Step S340: Obtain the difference between the first distance corresponding to the new center sequence and the second distance corresponding to the current center sequence; the first distance represents the average distance between the new center sequence and each sample fixation point position sequence, and the second distance represents the average distance between the current center sequence and each sample fixation point position sequence.

[0092] Step S350: If the difference is greater than the threshold, then the new center sequence is used as the current center sequence;

[0093] The loop termination condition is: the difference is less than or equal to the threshold.

[0094] Step S360: After the loop termination condition is met, the eye-tracking trajectory formed by the obtained center sequence is used as the reference eye-tracking trajectory for the user state.

[0095] The initial center sequence is any sequence in the set of sample gaze point position sequences.

[0096] The gaze position sequence is a time series; therefore, the first distance and the second distance are DTW distances obtained based on the Dynamic Time Warping (DTW) algorithm.

[0097] In practice, the common characteristics of eye movement behavior of user groups in each user state, namely the baseline eye movement trajectory, or representative eye movement trajectory, can be determined by combining the centroid-averaged dynamic time-of-day (DTW) dynamic time-of-day (DBA) algorithm with the K-means clustering algorithm. The specific process is as follows:

[0098] After obtaining a set of sample fixation point position sequences for a specific user state, arbitrarily select one sequence as the initial center sequence. Calculate the DTW distance between this initial center sequence and all other sequences in the sample fixation point position sequence set. Based on this DTW distance, determine the associated fixation point position corresponding to each fixation point position in the initial center sequence. Group each fixation point position and its corresponding associated fixation point position into a group, calculate the average value of each group, and obtain the new fixation point position corresponding to each fixation point position. Combine these new fixation point positions to form a new center sequence. Calculate the average DTW distance between the new center sequence and all other sequences in the sample fixation point position sequence set as the first distance, and calculate the average DTW distance between the initial center sequence and all other sequences in the sample fixation point position sequence set as the second distance. Obtain the difference between the first distance and the second distance. If the difference is greater than the threshold, it indicates that the center sequence has converged. Then, take the new center sequence as the current center sequence and repeat the loop until the difference between the first distance corresponding to the new center sequence and the second distance corresponding to the previous center sequence is less than or equal to the threshold. Then, end the loop and output the final center sequence. Use the eye movement trajectory formed by the center sequence as the reference eye movement trajectory for the user state.

[0099] In this embodiment, a baseline eye-tracking trajectory for customer group eye-tracking behavior is calculated using the centroid-average dynamic time warping (DBA) algorithm, and the dynamic time warping (DTW) algorithm is used to measure the similarity between different eye-tracking trajectories. The k-means algorithm is used to cluster the eye-tracking trajectories. The baseline eye-tracking trajectory determined in this way can better represent the common features of the eye-tracking behavior patterns of the user group in each state, thereby improving the reliability of the user state of the user to be identified based on the baseline eye-tracking trajectory.

[0100] To facilitate understanding of the concept of determining the baseline eye-tracking trajectory in this application by those skilled in the art, the method is further described below. DBA is an iterative algorithm that adjusts the average sequence in the target set, where each iteration follows an expectation-maximization scheme and involves two stages:

[0101] a) Randomly select a time series as the initial center series, calculate the DTW distance between the initial center series and each individual series in the sample fixation location sequence set, thereby finding the association between data points (i.e. fixation locations) on the initial center series and data points on other series.

[0102] b) Group each data point in the initial center sequence with its associated data points, calculate the average, and update the initial center sequence based on the calculation results. Repeat this process until convergence or the maximum number of iterations is reached. This yields the baseline eye-tracking trajectory in the group eye-tracking trajectory. The detailed DBA algorithm flow is as follows:

[0103] Input data: Target sequence set S = {S1,...,S...} N}, Initialize the central sequence

[0104] Output data: The center sequence after iterative calculation

[0105] S1: Start a loop.

[0106] S2: Alignment using the DTW algorithm and S j , where j = 1, 2, ..., N.

[0107] S3: S j Align each element in j = 1, 2, ..., N k = 1, 2, ..., l, obtaining the same as The set of elements associated with each element in C k ,

[0108]

[0109] S4: Calculate the central sequence The average value of each element and its associated elements is used to obtain a new center sequence. Understandably, since the number of elements associated with each sequence in the target set is different from the number of elements in the central sequence, the new central sequence... This is equivalent to calculating the weighted sum of the sequences in the target set.

[0110]

[0111] S5: When convergence is achieved, i.e. When the loop terminates, the loop exits and the latest center sequence A is output. Here, ave[] is the average function.

[0112] Corresponding to the first stage of the DBA iterative algorithm described above, in an exemplary embodiment, step S320, which involves determining the associated fixation positions of each fixation position in the current center sequence from the fixation positions contained in each sample fixation position sequence in the sample fixation position sequence set, includes: constructing a distance matrix between each sample fixation position sequence and the current center sequence; each element in the distance matrix represents the distance between a set of sample fixation positions corresponding to the position of that element; and determining the associated fixation positions of each fixation position in the current center sequence based on the distances in the distance matrix.

[0113] In the distance matrix, each element represents the Euclidean distance between a set of sample gaze points corresponding to the position of that element.

[0114] In the specific implementation, for each sample fixation point position sequence in the sample fixation point position sequence set, the Euclidean distance between each fixation point position in the sample fixation point position sequence and each fixation point position in the current center sequence can be calculated. Then, a two-dimensional distance matrix is ​​constructed based on the various Euclidean distances. For each fixation point position 'a' in each sample fixation point position sequence, the fixation point position 'b' in the current center sequence that is closer to the fixation point position 'a' is determined, and fixation point position 'a' is taken as the associated fixation point position 'b'.

[0115] For example, see reference. Figure 4 This is a schematic diagram illustrating multiple fixation point position sequences in one embodiment. It should be noted that the actual fixation point positions are curves; for a more intuitive representation, this example uses straight lines. Assume that sequence B in the diagram is the current center sequence, and sequences A and C are sequences from the sample fixation point position sequence set. Now, we need to determine the associated fixation point positions of each fixation point position in sequence B from sequences A and C. For example... Figure 4 As shown, for b1 in the current central sequence B, the associated fixation points of b1 in sequence A can be determined as a1 and a2 based on the Euclidean distance between each fixation point position in sequence A and C and b1, and the associated fixation point positions of b1 in sequence C can be determined as c1 and c2. Similarly, the associated fixation point positions of each fixation point position in sequence B can be determined using the same method.

[0116] In this embodiment, the associated fixation point positions of each fixation point position in the current center sequence are determined to improve the accuracy of the new center sequence determined subsequently based on the associated fixation point positions.

[0117] Corresponding to the second stage of the DBA iterative algorithm described above, in an exemplary embodiment, step S330, which determines a new center sequence based on each fixation point position in the current center sequence and the associated fixation point positions of each fixation point position, includes: for each fixation point position in the current center sequence, obtaining the average of the fixation point position and the associated fixation point positions of the fixation point position, as the new fixation point position corresponding to the fixation point position; and obtaining a new center sequence based on the new fixation point position corresponding to each fixation point position.

[0118] In practical implementation, it is still based on Figure 4 For example, for fixation point position b1 in the current center sequence B, its associated fixation point positions are a1, a2, c1, and c2. The mean of a1, a2, c1, c2 and b1 can be calculated, and the result can be used as the new fixation point position corresponding to fixation point position b1. By analogy, the new fixation point position corresponding to each fixation point position in the current center sequence can be obtained. The new center sequence can be obtained by connecting each new fixation point position according to the original order of each fixation point position in the current center sequence.

[0119] In this embodiment, a new center sequence is determined by the mean of each fixation point position in the current center sequence and its associated fixation point position. The center sequence obtained by this method can better characterize the common features of the user group in each state, thereby improving the representativeness of the determined baseline eye movement trajectory.

[0120] In an exemplary embodiment, the detection of each facial image in the facial video of the user to be identified in step S210 above to obtain multiple eye images of the user to be identified can be specifically achieved through the following steps:

[0121] Step S211: Perform key point detection on each facial image to obtain the eye image in each facial image;

[0122] Step S212: Adjust the size of each eye image to obtain adjusted eye images of the same size;

[0123] Step S213: Perform equalization processing on each adjusted eye image, and use the equalized adjusted eye images as multiple eye images of the user to be identified.

[0124] Understandably, since facial images captured by cameras vary in terms of individual differences, distance, lighting, and head posture, it is necessary to detect the eyes in facial images in order to stably track eye movements frame by frame in these facial images.

[0125] In practical implementation, the OpenCV algorithm library (a computer vision algorithm library that can be used to develop real-time image processing) can be used for facial landmark detection. (Refer to...) Figure 5 This is a schematic diagram of facial key points in one embodiment. The process of obtaining the eye image of any facial image includes: firstly, performing key point detection on the facial image to obtain, as shown below. Figure 5 The facial landmark diagram shown in (a) focuses on changes in the eyeballs. Therefore, it is necessary to determine, from the facial landmarks, features such as... Figure 5 The key points of the eye region shown in (b) and (c) are used to determine the bounding box of the eye region. This bounding box can be a quadrilateral bounding box, such as a rectangle or square. Then, the facial image is cropped according to this bounding box to obtain the eye image in the facial image, as shown below. Figure 5 As shown in (d) in the figure. This acquisition process is performed for each face image, resulting in a series of eye images that vary in size, lighting, and rotation.

[0126] Furthermore, to make subsequent calculations more convenient and accurate, the individual eye images can be adjusted by scaling them to the same size. Histograms can then be used to equalize the adjusted eye images, mitigating the impact of varying lighting conditions. It should be noted that there is no strict order for adjusting the size and equalizing the individual eye images. In practice, equalization can be performed first, followed by size adjustments.

[0127] In this embodiment, eye images are first extracted from each facial image through key point detection. Then, each eye image is scaled to a uniform size to ensure the accuracy of the subsequently determined gaze point position. At the same time, equalization processing is performed on each eye image to alleviate the influence of different lighting on the eye image and further improve the accuracy of the subsequently determined gaze point position.

[0128] In an exemplary embodiment, before step S240 determines the user state of the user to be identified based on the similarity between the eye-tracking trajectory and a plurality of preset benchmark eye-tracking trajectories, the method further includes:

[0129] Step S231: Construct a distance matrix based on the positions of each fixation point in the eye track and the positions of each fixation point in any reference eye track; each element in the distance matrix represents the distance between a set of fixation point positions corresponding to the position of that element.

[0130] Step S232: Determine the resetting path in the distance matrix and obtain the cumulative distance corresponding to the resetting path; the resetting path is the shortest path from the starting element to the ending element in the distance matrix.

[0131] Step S233: Based on the cumulative distance, obtain the similarity between the eye movement trajectory and any reference eye movement trajectory.

[0132] In practice, before determining the user status of the user to be identified, it is necessary to calculate the similarity between the eye movement trajectory of the user to be identified and multiple preset benchmark eye movement trajectories. Since eye movement trajectory is a time series data, the Dynamic Time Warping (DTW) algorithm should be used instead of the traditional Euclidean distance to calculate the similarity between eye movement trajectories.

[0133] The detailed process of the DTW algorithm is as follows: Consider two sequences X = (x1, x2, ..., xn) of length m and n respectively. i ,...,x m ) and Y = (y1, y2, ..., y j ,...,y n Let d ij Let x be the point i and y j Construct a matrix based on the Euclidean distance between them:

[0134]

[0135] The DTW algorithm needs to find an optimal path from (1,1) to (m,n). To match sequences, see Figure 6 To ensure that this path is globally optimal, three conditions must be met simultaneously:

[0136] ① Boundedness. The starting point of the path must be (1,1) and the ending point must be (m,n).

[0137] ② Continuity. The rule is that the next point in the path must be adjacent to the current point; a match cannot be made by skipping a point. That is, the next point after the current point (i,j) must be one of (i+1,j), (i,j+1), or (i+1,j+1).

[0138] ③ Monotonicity. The prescribed path maintains a monotonic temporal order without decreasing.

[0139] When the above three conditions are met, DTW can be defined as

[0140]

[0141] Wherein d(w k )=d(x i ,y j Let represent the i and j corresponding to position k in the path. The cumulative distance of the path can be calculated recursively.

[0142]

[0143] The cumulative distance between the eye movement trajectory of the user to be identified and each baseline eye movement trajectory can be calculated using the above formula. This cumulative distance can then be input into a preset conversion formula to obtain the similarity score. The principle behind determining this conversion formula is as follows:

[0144] DTW (Time-Tracking Wave) can measure the similarity between two time series to the greatest extent. Normalizing the similarity distance calculated by DTW yields a similarity score between the two time series, i.e., a similarity score between eye-tracking trajectories. Normalization calculates the difference between two time series as a value in the range [0,1], where 0 represents a similarity of 0 and 1 represents complete overlap. In this embodiment, based on the characteristics of the DTW distance, a Gaussian kernel function is selected as the transformation function:

[0145]

[0146] Here, σ describes the sensitivity of similarity to DTW distance. When the DTW value remains constant, a larger σ indicates higher similarity, and a smaller σ indicates lower similarity. Therefore, in the case of time series clustering, the value of σ should be chosen appropriately. When the value of σ is such that... When the time (where N is the number of sequences in the time series set) is greater, better clustering results can be achieved.

[0147] In this embodiment, considering that eye movement trajectory is a time series data, the DTW distance metric using the dynamic time-warping algorithm can ensure the accuracy of the determined similarity.

[0148] In an exemplary embodiment, step S240 determines the user state of the user to be identified based on the similarity between the eye-tracking trajectory and multiple preset benchmark eye-tracking trajectories. This can be achieved through the following steps:

[0149] Step S241: Based on similarity, determine the target reference eye movement trajectory from multiple reference eye movement trajectories; the similarity of the target reference eye movement trajectory is greater than the similarity of other reference eye movement trajectories.

[0150] Step S242: Determine the user state corresponding to the target baseline eye movement trajectory as the user state of the user to be identified.

[0151] In practice, based on similarity, the target baseline eye track with the highest similarity can be determined from multiple baseline eye track tracks, and the user state corresponding to the target baseline eye track can be determined as the user state of the user to be identified.

[0152] In this embodiment, the user state corresponding to the benchmark eye-tracking trajectory with the highest similarity to the eye-tracking trajectory of the user to be identified is determined as the user state of the user to be identified, so as to issue an alarm when the user state is abnormal, thereby realizing the supervision of the user self-service of the self-service interactive terminal.

[0153] In one exemplary embodiment, the eye image corresponding to each gaze point position carries a timestamp, which indicates the time when the facial image corresponding to the eye image was acquired;

[0154] In step S230 above, the eye movement trajectory of the user to be identified during the operation of the interactive terminal is generated according to the position of each gaze point. Specifically, the eye movement trajectory of the user to be identified during the operation of the interactive terminal can be obtained by connecting the positions of each gaze point in the order of timestamps from early to late.

[0155] In the specific implementation, since the timestamp carried by the eye image represents the time when the corresponding facial image was captured, the order of the timestamp from morning to night is the order of changes in the eyes of the user to be identified. Therefore, the eye movement trajectory of the user to be identified during the operation of the interactive terminal can be obtained by connecting the positions of each gaze point according to the order of the timestamp from morning to night.

[0156] In this embodiment, the positions of each gaze point are connected in order of timestamp from morning to night to obtain the eye movement trajectory of the user to be identified during the operation of the interactive terminal. This allows the user's state to be determined by comparing the eye movement trajectory with the baseline eye movement trajectory under different user states.

[0157] In one embodiment, such as Figure 7 The diagram shown is a flowchart illustrating a user state recognition method according to another embodiment of this application. This embodiment includes the following steps:

[0158] Step S710: Key point detection is performed on each facial image in the facial video of the user to be identified to obtain the eye image in each facial image; the facial video is acquired by a camera set on the interactive terminal;

[0159] Step S720: Adjust the size of each eye image to obtain adjusted eye images of the same size;

[0160] Step S730: Equalize each adjusted eye image and use the equalized adjusted eye images as multiple eye images of the user to be identified.

[0161] Step S740: Input each face image and the corresponding eye image of each face image into the pre-trained neural network model to obtain the gaze point position corresponding to each eye image;

[0162] Step S750: Connect the various gaze points in order from early to late according to the timestamps carried by the eye images corresponding to each gaze point position to obtain the eye movement trajectory of the user to be identified during the operation of the interactive terminal.

[0163] Step S760: Obtain multiple reference eye movement trajectories, and construct distance matrices between each reference eye movement trajectory and the eye movement trajectory of the user to be identified;

[0164] Step S770: Determine the normalized path in the distance matrix and obtain the cumulative distance corresponding to the normalized path; based on the cumulative distance, obtain the similarity between the eye movement trajectory and each benchmark eye movement trajectory;

[0165] Step S780: Based on similarity, determine the target reference eye movement trajectory from multiple reference eye movement trajectories; the similarity of the target reference eye movement trajectory is greater than the similarity of other reference eye movement trajectories.

[0166] Step S790: Determine the user state corresponding to the target baseline eye movement trajectory as the user state of the user to be identified.

[0167] This application uses a regular webcam to capture facial images of customers operating in front of a self-service terminal. These facial images are then processed to establish a sequence of changes in gaze position. A time-series algorithm (DBA algorithm) combined with k-means clustering is then used to classify different eye-tracking sequences, enabling alerts to be triggered when user status is abnormal, thereby achieving supervision of user self-service at the self-service terminal. This method has the following advantages over existing technologies:

[0168] (1) The hardware cost is low, and no additional equipment is required. Ordinary network cameras on self-service terminals can be used. Network cameras are the cheapest and most common image capture devices, and almost every type of terminal is equipped with them.

[0169] (2) The intelligent interactive terminal system based on gaze estimation can automatically detect abnormal behavior of users during the service process. By recognizing the user's eye movements, it can understand the user's intentions and behaviors during the use of the self-service terminal. When abnormal user status is detected, it can issue an alarm in a timely manner.

[0170] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0171] Based on the same inventive concept, this application also provides a user state identification device for implementing the user state identification method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more user state identification device embodiments provided below can be found in the limitations of the user state identification method described above, and will not be repeated here.

[0172] In one embodiment, such as Figure 8 As shown, a user status recognition device is provided, installed in an interactive terminal associated with financial business, including: an image detection module 810, a location determination module 820, a trajectory generation module 830, and a status determination module 840, wherein:

[0173] The image detection module 810 is used to detect each facial image in the facial video of the user to be identified, and obtain multiple eye images of the user to be identified; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal;

[0174] The position determination module 820 is used to input each face image and the corresponding eye image of each face image into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image.

[0175] The trajectory generation module 830 is used to generate the eye movement trajectory of the user to be identified during the operation of the interactive terminal based on the position of each gaze point.

[0176] The state determination module 840 is used to determine the user state of the user to be identified based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group.

[0177] In one embodiment, the above-described apparatus further includes a reference trajectory determination module, configured to acquire a set of sample gaze point position sequences and an initial center sequence for each user state; and to iteratively execute the following steps until a loop termination condition is met: determining the associated gaze point positions of each gaze point position in the current center sequence from the gaze point positions contained in each sample gaze point position sequence in the set of sample gaze point position sequences; determining a new center sequence based on each gaze point position and its associated gaze point positions in the current center sequence; acquiring the difference between a first distance corresponding to the new center sequence and a second distance corresponding to the current center sequence; the first distance represents the average distance between the new center sequence and each sample gaze point position sequence, and the second distance represents the average distance between the current center sequence and each sample gaze point position sequence; if the difference is greater than a threshold, the new center sequence is used as the current center sequence; wherein, the loop termination condition is: the difference is less than or equal to the threshold; after the loop termination condition is met, the eye movement trajectory formed by the obtained center sequences is used as the reference eye movement trajectory for the user state.

[0178] In one embodiment, the baseline trajectory determination module is further configured to construct a distance matrix between each sample gaze point position sequence and the current center sequence; each element in the distance matrix represents the distance between a set of sample gaze point positions corresponding to the position of that element; and based on each distance in the distance matrix, determine the associated gaze point positions of each gaze point position in the current center sequence.

[0179] In one embodiment, the baseline trajectory determination module is further configured to, for each fixation point position in the current center sequence, obtain the mean of the fixation point position and the associated fixation point positions of the fixation point position, as the new fixation point position corresponding to the fixation point position; and obtain a new center sequence based on the new fixation point position corresponding to each fixation point position.

[0180] In one embodiment, the image detection module 810 is further configured to perform key point detection on each facial image to obtain eye images in each facial image; adjust the size of each eye image to obtain adjusted eye images of the same size; perform equalization processing on each adjusted eye image, and use the equalized adjusted eye images as multiple eye images of the user to be identified.

[0181] In one embodiment, the state determination module 840 is further configured to construct a distance matrix based on the positions of each gaze point in the eye track and the positions of each gaze point in any reference eye track; each element in the distance matrix represents the distance between a set of gaze point positions corresponding to the position of the element; determine the normalization path in the distance matrix and obtain the cumulative distance corresponding to the normalization path; the normalization path is the shortest path from the starting element to the ending element in the distance matrix; and obtain the similarity between the eye track and any reference eye track based on the cumulative distance.

[0182] In one embodiment, the state determination module 840 is further configured to determine a target reference eye track from multiple reference eye tracks based on similarity; the similarity corresponding to the target reference eye track is greater than the similarity corresponding to other reference eye tracks; and the user state corresponding to the target reference eye track is determined as the user state of the user to be identified.

[0183] In one embodiment, the eye images corresponding to each gaze point position carry a timestamp, which indicates the time when the facial image corresponding to the eye image was acquired; the trajectory generation module 830 is also used to connect each gaze point position in order from early to late according to the timestamps to obtain the eye movement trajectory of the user to be identified during the operation of the interactive terminal.

[0184] Each module in the aforementioned user status identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0185] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a user status identification method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0186] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0189] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A user status recognition method, characterized in that, The method, applied to an interactive terminal associated with financial transactions, includes: Each facial image in the facial video of the user to be identified is detected to obtain multiple eye images of the user; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal; Each facial image and its corresponding eye image are input into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image. Based on the position of each gaze point, the eye movement trajectory of the user to be identified during the operation of the interactive terminal is generated; The user state of the user to be identified is determined based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group. The baseline eye-tracking trajectory for each user state is determined as follows: For each user state, obtain the sample gaze point location sequence set and initial center sequence for that user state; repeat the following steps until the loop termination condition is met: A distance matrix is ​​constructed between each sample fixation point position sequence and the current center sequence. Each element in the distance matrix represents the distance between a set of sample fixation point positions corresponding to that element. Based on the distances in the distance matrix, the associated fixation point positions of each fixation point position in the current center sequence are determined. Based on the fixation point positions in the current center sequence and the associated fixation point positions, a new center sequence is determined. The difference between a first distance corresponding to the new center sequence and a second distance corresponding to the current center sequence is obtained. The first distance represents the average distance between the new center sequence and each sample fixation point position sequence, and the second distance represents the average distance between the current center sequence and each sample fixation point position sequence. If the difference is greater than a threshold, the new center sequence is taken as the current center sequence. The loop termination condition is: the difference is less than or equal to the threshold. After the loop termination condition is met, the eye-tracking trajectory formed by the obtained center sequence is used as the reference eye-tracking trajectory for the user state.

2. The method according to claim 1, characterized in that, The step of determining a new center sequence based on the fixation points in the current center sequence and the associated fixation points of those fixation points includes: For each fixation point position in the current center sequence, the mean of the fixation point position and the associated fixation point positions is obtained, and used as the new fixation point position corresponding to the fixation point position. The new center sequence is obtained based on the new fixation point position corresponding to each fixation point position.

3. The method according to claim 1, characterized in that, The process involves detecting individual facial images in the facial video of the user to be identified, resulting in multiple eye images of the user, including: Key point detection is performed on each of the facial images to obtain the eye images in each facial image; The size of each eye image is adjusted to obtain an adjusted eye image of the same size; Each adjusted eye image is then subjected to equalization processing, and the equalized adjusted eye images are used as multiple eye images of the user to be identified.

4. The method according to claim 1, characterized in that, Before determining the user state of the user to be identified based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories, the method further includes: A distance matrix is ​​constructed based on the positions of each fixation point in the eye movement trajectory and the positions of each fixation point in any reference eye movement trajectory; each element in the distance matrix represents the distance between a set of fixation point positions corresponding to the position of that element. Determine the normalized path in the distance matrix and obtain the cumulative distance corresponding to the normalized path; the normalized path is the shortest path from the starting element to the ending element in the distance matrix. Based on the cumulative distance, the similarity between the eye movement trajectory and any reference eye movement trajectory is obtained.

5. The method according to claim 1, characterized in that, The step of determining the user state of the user to be identified based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories includes: Based on the similarity, a target reference eye movement trajectory is determined from the plurality of reference eye movement trajectories; the similarity corresponding to the target reference eye movement trajectory is greater than the similarity corresponding to other reference eye movement trajectories; The user state corresponding to the target baseline eye movement trajectory is determined as the user state of the user to be identified.

6. The method according to claim 1, characterized in that, The eye images corresponding to each gaze point position carry a timestamp, which indicates the time when the facial image corresponding to the eye image was captured. The step of generating the eye movement trajectory of the user to be identified during the operation of the interactive terminal based on the position of each gaze point includes: By connecting the various gaze points in order of the timestamps from morning to night, the eye movement trajectory of the user to be identified during the operation of the interactive terminal can be obtained.

7. A user status identification device, characterized in that, The device, located in an interactive terminal associated with financial business, includes: The image detection module is used to detect each facial image in the facial video of the user to be identified, and obtain multiple eye images of the user to be identified; wherein, each facial image corresponds to one eye image; the facial video is acquired by a camera set on the interactive terminal; The position determination module is used to input the face images and the corresponding eye images into a pre-trained neural network model to obtain the gaze point position corresponding to each eye image; The trajectory generation module is used to generate the eye movement trajectory of the user to be identified during the operation of the interactive terminal based on the position of each gaze point; The state determination module is used to determine the user state of the user to be identified based on the similarity between the eye movement trajectory and multiple preset benchmark eye movement trajectories; each benchmark eye movement trajectory has a corresponding user state, and the benchmark eye movement trajectory is obtained by calculating the common features of the eye movement behavior patterns of the corresponding user group; The baseline trajectory determination module is used to acquire a set of sample gaze point position sequences and an initial center sequence for each user state; and to repeatedly execute the following steps until the loop termination condition is met: A distance matrix is ​​constructed between each sample fixation point position sequence and the current center sequence. Each element in the distance matrix represents the distance between a set of sample fixation point positions corresponding to the element's position. Based on the distances in the distance matrix, the associated fixation point positions of each fixation point position in the current center sequence are determined. Based on the fixation point positions in the current center sequence and the associated fixation point positions, a new center sequence is determined. The difference between the first distance corresponding to the new center sequence and the second distance corresponding to the current center sequence is obtained. The first distance represents the average distance between the new center sequence and each sample fixation point position sequence, and the second distance represents the average distance between the current center sequence and each sample fixation point position sequence. If the difference is greater than a threshold, the new center sequence is used as the current center sequence. The loop termination condition is that the difference is less than or equal to the threshold. After the loop termination condition is met, the eye movement trajectory formed by the obtained center sequences is used as the baseline eye movement trajectory of the user state.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the user state recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the user state recognition method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the user state recognition method according to any one of claims 1 to 6.

Images