Individual state determination method and related device

Abstract

The invention discloses an individual state determination method and a related device, and the method comprises the steps: carrying out the feature extraction of a to-be-classified video frame, and obtaining a biological feature set of the to-be-classified video frame; the biological feature vector of each channel dimension in the biological feature set of the to-be-classified video frame is used for describing to-be-classified individuals in the to-be-classified video frame from different angles; performing attention processing on the biological feature vector of each channel dimension to update a biological feature set of the to-be-classified video frame; and based on the updated biological feature set of the to-be-classified video frame, classifying the state of the to-be-classified individual in the to-be-classified video frame to obtain a first classification result. According to the scheme, the accuracy of the first classification result can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an individual state determination method, an individual state determination device, an electronic device, and a computer-readable storage medium. Background Art

[0002] The individual state determination method is used to classify an individual's state based on their video frames to determine the category to which their state belongs. This method can be applied to fields such as education, health management, and behavioral analysis. For example, in education, the learner's concentration level can be determined based on their video frames.

[0003] However, the state classification results obtained by existing individual state determination methods are not accurate enough. Summary of the Invention

[0004] The present application provides an individual status determination method, an individual status determination device, an electronic device, and a computer-readable storage medium, which can improve the accuracy of status classification results.

[0005] The present application provides a method for determining an individual state, comprising: extracting features from a video frame to be classified to obtain a biometric feature set of the video frame to be classified; using a biometric feature vector of each channel dimension in the biometric feature set of the video frame to be classified to describe the individual to be classified in the video frame to be classified from different angles; performing attention processing on the biometric feature vector of each channel dimension to update the biometric feature set of the video frame to be classified; and classifying the state of the individual to be classified in the video frame to be classified based on the updated biometric feature set of the video frame to be classified to obtain a first classification result.

[0006] The present application provides an individual status determination device, comprising: a feature extraction module, an attention processing module, and a classification module. The feature extraction module is used to extract features from a video frame to be classified to obtain a set of biometric features of the video frame to be classified; the biometric feature vectors of each channel dimension in the biometric feature set of the video frame to be classified are used to describe the individual to be classified in the video frame to be classified from different angles; the attention processing module is used to perform attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified; and the classification module is used to classify the state of the individual to be classified in the video frame to be classified based on the updated biometric feature set of the video frame to be classified to obtain a first classification result.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-mentioned individual status determination method.

[0008] The present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned individual state determination method when the program instructions are executed by a processor.

[0009] In the above scheme, the biometric feature set of the video frame to be classified includes biometric feature vectors in multiple channel dimensions. Attention processing is performed on the biometric feature vectors in each channel dimension to update the biometric feature set of the video frame to be classified. The state of the individual to be classified is classified based on the updated biometric feature set of the video frame to be classified, thereby obtaining a first classification result. On the one hand, compared to biometric feature vectors in a single channel dimension, biometric feature vectors in multiple channel dimensions can describe the individual to be classified from different perspectives, expressing the individual's state more comprehensively. Biometric feature vectors based on multiple channel dimensions can more accurately classify the individual's state, thereby improving the accuracy of the first classification result. On the other hand, based on considering biometric feature vectors in multiple channel dimensions, performing attention processing on biometric feature vectors in multiple channel dimensions can increase the attention paid to biometric feature vectors in channel dimensions that are useful for classification and reduce the attention paid to biometric feature vectors in channel dimensions that are redundant or irrelevant to classification. This effectively integrates biometric feature vectors from different channel dimensions, further improving the accuracy of the first classification result.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0012] Figure 1 This is a flow chart of an embodiment of a method for determining an individual status provided by the present application;

[0013] Figure 2 is a flowchart of another embodiment of the method for determining individual status provided by the present application;

[0014] Figure 3 is a flowchart of another embodiment of the method for determining individual status provided by the present application;

[0015] Figure 4 This is a flow chart of an embodiment of a training method for an individual state determination model of the present application;

[0016] Figure 5 This is a flow chart of another embodiment of the training method of the individual state determination model of the present application;

[0017] Figure 61 is a flow chart of another embodiment of the training method of the individual state determination model of the present application;

[0018] Figure 7 is a flowchart of a specific example of the method for determining individual status provided in this application;

[0019] Figure 8 This is a schematic diagram of the channel attention processing flow of SENet in this application;

[0020] Figure 9 This is a schematic diagram of the channel attention processing and spatial attention processing flow of CBAM in this application;

[0021] Figure 10 This is a flowchart of a specific example of a training method for an individual state determination model of the present application;

[0022] Figure 11 This is a flow chart of an embodiment of the individual status determination device of the present application;

[0023] Figure 12 This is a structural diagram of an embodiment of an electronic device of the present application;

[0024] Figure 13 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0026] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0027] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] Figure 1 This is a flow chart of an embodiment of the method for determining individual status provided by this application. Figure 1 As shown, in this embodiment, the individual status determination method may include the following steps:

[0029] S110: Extract features from the video frame to be classified to obtain a set of biometric features of the video frame to be classified.

[0030] The biometric feature vectors of each channel dimension in the biometric feature set of the video frame to be classified are used to describe the individual to be classified in the video frame to be classified from different angles.

[0031] The execution subject of the embodiment of the individual status determination method of the present application is an individual status determination device, and the individual status determination device can be any electronic device with the ability to determine an individual status.

[0032] The individual state determination method of the present application is intended to classify the state of an individual to be classified in a video frame to be classified. The video frame to be classified can be a separate video frame or a video frame in a video to be classified.

[0033] The individuals to be classified can be living things such as people (such as students, teachers, white-collar workers, blue-collar workers), animals (such as cats, dogs, pigs), or still objects such as trees, flowers and plants.

[0034] The state of the individual to be classified may include, but is not limited to, exercise state, learning state, behavioral state, emotional state, and growth state. Candidate categories for exercise state may include intense, normal, and soothing. Learning state may be further divided into reading state and concentration state. Candidate categories for reading state may include reading and unreading. Candidate categories for concentration state may include very concentrated, concentrated, not concentrated, and extremely not concentrated. Candidate categories for behavioral state may include very standard, standard, and non-standard. Candidate categories for emotional state may include happy, normal, and sad. Candidate categories for growth state may include germination, growth, maturity, and dormancy.

[0035] The video frame to be classified contains a part or the whole of the individual to be classified. For example, the video frame to be classified contains the shoulders, neck, and head of the individual to be classified. For another example, the video frame to be classified contains only the head of the individual to be classified.

[0036] The channel dimension may correspond to a part or a combination of parts of the individual to be classified, depending on the state of the individual to be classified.

[0037] In some embodiments, when the state of the individual to be classified is a reading state, each channel dimension includes an eye channel dimension and a mouth channel dimension. The eye channel dimension corresponds to the eyes, and the mouth channel dimension corresponds to the mouth.

[0038] In some embodiments, the state of the individual to be classified is an emotional state, and each channel dimension includes an eye channel dimension, an eyebrow channel dimension, a nose channel dimension, a mouth channel dimension, and a cheek channel dimension. The eye channel dimension corresponds to the eyes, the eyebrow channel dimension corresponds to the eyebrows, the nose channel dimension corresponds to the nose, the mouth channel dimension corresponds to the mouth, and the cheek channel dimension corresponds to the cheeks.

[0039] In some embodiments, when the state of the individual to be classified is a focused state, the channel dimensions include a head / face channel dimension, a posture channel dimension, and an eye channel dimension. The head / face channel dimension corresponds to the head / face, the posture channel dimension corresponds to the head and shoulders, and the eye channel dimension corresponds to the eyes. The biometric feature vector of the head channel dimension can be used, but not limited to, for face recognition; the biometric feature vector of the posture channel dimension can be used, but not limited to, for posture analysis; and the biometric feature vector of the eye channel dimension can be used, but not limited to, for eye tracking.

[0040] S120: Perform attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified.

[0041] It can be understood that attention processing of the biometric feature vectors of each channel dimension can enhance the biometric feature vectors of the channel dimension that is useful for classification, increase the attention to the biometric feature vectors of the channel dimension that is useful for classification, reduce the attention to the biometric feature vectors of the channel dimension that is redundant or irrelevant for classification, and realize the effective fusion of the biometric feature vectors of each channel dimension.

[0042] S130: Based on the updated biometric feature set of the video frame to be classified, classify the status of the individual to be classified in the video frame to be classified to obtain a first classification result.

[0043] In some embodiments, the first classification result may include a predicted category label of the video frame to be classified. The predicted category label of the video frame to be classified represents the predicted category of the state of the individual to be classified.

[0044] In some embodiments, the first classification result may further include a confidence level of the predicted category label of the video frame to be classified. The confidence level of the predicted category label of the video frame to be classified is used to represent the credibility of the predicted category label of the video frame to be classified.

[0045] It is understandable that if a single-channel-dimensional biometric feature vector is used to classify the state of the individual to be classified in the video frame to be classified, due to the limited ability of the single-channel-dimensional biometric feature vector to express the state of the individual to be classified, it cannot fully express the state of the individual to be classified, especially when facing complex individual state determination scenarios or individual state determination tasks, the classification accuracy and robustness are low.

[0046] In the above scheme, the biometric feature set of the video frame to be classified includes biometric feature vectors in multiple channel dimensions. Attention processing is performed on the biometric feature vectors in each channel dimension to update the biometric feature set of the video frame to be classified. The state of the individual to be classified is classified based on the updated biometric feature set of the video frame to be classified, thereby obtaining a first classification result. On the one hand, compared to biometric feature vectors in a single channel dimension, biometric feature vectors in multiple channel dimensions can describe the individual to be classified from different perspectives, expressing the individual's state more comprehensively. Biometric feature vectors based on multiple channel dimensions can more accurately classify the individual's state, thereby improving the accuracy of the first classification result. On the other hand, based on considering biometric feature vectors in multiple channel dimensions, performing attention processing on biometric feature vectors in multiple channel dimensions can increase the attention paid to biometric feature vectors in channel dimensions that are useful for classification and reduce the attention paid to biometric feature vectors in channel dimensions that are redundant or irrelevant to classification. This effectively integrates biometric feature vectors from different channel dimensions, further improving the accuracy of the first classification result.

[0047] Furthermore, the biometric feature set (H×W×C) of the video frame to be classified has a channel dimension (C, along the spatial dimension) and a spatial dimension (H*W, along the channel dimension, where H is height and W is width). In terms of the channel dimension, the biometric feature set of the video frame to be classified includes multiple biometric feature vectors of the channel dimension. In terms of the spatial dimension, the biometric feature set of the video frame to be classified includes multiple biometric feature vectors of the spatial dimension.

[0048] In some embodiments, in S120, attention processing may be performed only on the channel dimension (hereinafter referred to as channel attention processing in this application).

[0049] In some embodiments, in S120, attention processing can also be performed on the spatial dimension (hereinafter referred to as spatial attention processing in this application) to increase the attention to the biometric feature vectors of the spatial dimensions that are useful for classification, and reduce the attention to the biometric feature vectors of the spatial dimensions that are redundant or irrelevant for classification, so as to achieve effective management of information focus.

[0050] The number of times channel attention processing / spatial attention processing is performed on the biometric feature set of the video frame to be classified can be one or more. The order of performing channel attention processing and spatial attention processing is not limited. Channel attention processing can be performed first, and then spatial attention processing. Alternatively, spatial attention processing can be performed first, and then channel attention processing. Alternatively, different times of spatial attention processing and different times of channel attention processing can be performed alternately.

[0051] Channel attention processing can be implemented using any module capable of channel attention processing, such as SENet (Squeeze-and-Excitation), ECANet, CBAM (Convolutional Block Attention Module), or SKNet. Spatial attention processing can be implemented using any module capable of spatial attention processing, such as CBAM or STN. SENet requires less computational overhead.

[0052] An embodiment further extending S120 is described below:

[0053] Figure 2 This is a flow chart of another embodiment of the method for determining individual status provided by this application. In this embodiment, channel attention processing is performed on the biometric feature set of the video frame to be classified. Figure 2 As shown, S120 may include the following steps:

[0054] S210: Perform attention processing on the biological feature vectors of each channel dimension according to the first channel attention processing method to obtain the first channel attention coefficient of each channel dimension.

[0055] The first channel attention processing method can be the channel attention processing method of any module with channel attention processing capability. Taking SENet implementation as an example, S210 includes the following S211-S212 (not shown).

[0056] S211: Compress the biometric feature vectors of each channel dimension respectively according to a first compression method to obtain a first feature compression value of each channel dimension. The first compression method includes a global pooling operation.

[0057] The global pooling operation can be a global average pooling operation or a global maximum pooling operation. The so-called global average pooling operation refers to taking the average eigenvalue of the biometric feature vector in the channel dimension. The so-called global maximum pooling operation refers to taking the maximum eigenvalue of the biometric feature vector in the channel dimension. By compressing the biometric feature vector (H×W×C) of each channel dimension according to the first compression method, the first feature compression value (C×1×1) of the C channel dimensions can be obtained.

[0058] S212: Perform a full connection operation on the first feature compression value of each channel dimension to obtain the first channel attention coefficient of each channel dimension.

[0059] The fully connected operation can be implemented through the fully connected layer (MLP).

[0060] S220: Apply the first channel attention coefficient of each channel dimension to the biological feature vector of the corresponding channel dimension respectively.

[0061] The first channel attention coefficient acts on the biological feature vector of the channel dimension by multiplying or adding the first channel attention coefficient to the biological feature vector of the channel dimension.

[0062] Figure 3 This is a flow chart of another embodiment of the method for determining individual status provided by this application. This embodiment is a further extension of S120. In this embodiment, based on S210-S220, channel attention processing and spatial attention processing are further performed on the biometric feature set of the video frame to be classified. Figure 3 As shown, after S220, S120 may further include the following steps:

[0063] S310: Perform attention processing on the biological feature vectors of each channel dimension according to the second channel attention processing method to obtain the second channel attention coefficient of each channel dimension.

[0064] The second channel attention processing mode can be a channel attention processing mode of any module with channel attention processing capability. The second channel attention processing mode can be the same as or different from the first channel attention processing mode.

[0065] In some embodiments, one of the first channel attention processing method and the second channel attention processing method is implemented by SENet, and the other is implemented by CBAM.

[0066] When the second channel attention processing method is implemented by CBAM, S310 may include the following S311-S312 (not shown).

[0067] S311: Compress the biological feature vectors of each channel dimension according to the second compression method to obtain a second feature compression value of each channel dimension.

[0068] The second compression method includes a global pooling operation.

[0069] In some embodiments, the global pooling operation is a global maximum pooling operation, and the second feature compression value is a global maximum pooling result. The so-called global maximum pooling operation refers to taking the maximum eigenvalue of the biological feature vector in the channel dimension.

[0070] In some embodiments, the global pooling operation is a global average pooling operation, and the second feature compression value is a global average pooling result. The so-called global average pooling operation refers to taking the average feature value of the biological feature vector in the channel dimension.

[0071] In some embodiments, the global pooling operation includes a global maximum pooling operation and a global average pooling operation. The second feature compression value includes a global maximum pooling result and a global average pooling result.

[0072] S312: Perform a full connection operation on the second feature compression value of each channel dimension to obtain the second channel attention coefficient of each channel dimension.

[0073] In some embodiments, the second feature compression value is a global maximum pooling result, and a full connection operation can be performed on the global maximum pooling result of each channel dimension to obtain the second channel attention coefficient of each channel dimension.

[0074] In some embodiments, the second feature compression value is a global average pooling result, and a full connection operation can be performed on the global average pooling result of each channel dimension to obtain the second channel attention coefficient of each channel dimension.

[0075] In some embodiments, the second feature compression value includes the global maximum pooling result and the global average pooling result. The global maximum pooling result of each channel dimension and the global average pooling result of each channel dimension can be fully connected to obtain the global maximum channel attention coefficient and the global average channel attention coefficient of each channel dimension. The global maximum channel attention coefficient of each channel dimension is added to the global average channel attention coefficient of the corresponding channel dimension to obtain the second channel attention coefficient of each channel dimension.

[0076] S320: Perform spatial attention processing on the biometric feature vectors of each spatial dimension in the biometric feature set of the video frame to be classified to obtain a spatial attention coefficient of each spatial dimension.

[0077] Spatial attention processing can be implemented by any module with spatial attention processing capabilities, such as CBAM, STN, etc. Taking CBAM as an example, S320 includes the following S321-S322 (not shown).

[0078] S321: Compress the biometric feature vectors of each spatial dimension respectively according to the third compression method to obtain a third feature compression value of each spatial dimension.

[0079] The third compression method includes at least one of a maximum pooling operation and an average pooling operation.

[0080] In some embodiments, the third compression method includes a maximum pooling operation, and the third feature compression value is a maximum pooling result. The so-called maximum pooling operation refers to taking the maximum eigenvalue of the biological feature vector in the spatial dimension.

[0081] In some embodiments, the third compression method includes an average pooling operation, and the third feature compression value is the average pooling result. The so-called average pooling operation refers to taking the average feature value of the biological feature vector in the spatial dimension.

[0082] In some embodiments, the third compression method includes a maximum pooling operation and an average pooling operation, and the third feature compression value includes a maximum pooling result and an average pooling result.

[0083] S322: Based on the third feature compression value of each spatial dimension, obtain the spatial attention coefficient of each spatial dimension.

[0084] In some embodiments, the third feature compression value of each spatial dimension is the maximum pooling result or the average pooling result, and the third feature compression value of each spatial dimension can be directly used as the spatial attention coefficient of each spatial dimension. Alternatively, the third feature compression value of each spatial dimension can be normalized to obtain the spatial attention coefficient of each spatial dimension.

[0085] In some embodiments, the third feature compression value of each spatial dimension includes the maximum pooling result and the average pooling result. The maximum pooling result and the average pooling result of each spatial dimension can be spliced ​​to obtain the spliced ​​pooling result of each spatial dimension; the spliced ​​pooling result of each spatial dimension is convolved and normalized to obtain the spatial attention coefficient of each spatial dimension.

[0086] S330: Apply the second channel attention coefficient of each channel dimension to the biological feature vector of the corresponding channel dimension.

[0087] The second channel attention coefficient acts on the biological feature vector of the channel dimension by multiplying or adding the second channel attention coefficient to the biological feature vector of the channel dimension.

[0088] S340: Applying the spatial attention coefficient of each spatial dimension to the biological feature vector of the corresponding spatial dimension.

[0089] The spatial attention coefficient may act on the biological feature vector of the spatial dimension by multiplying or adding the spatial attention coefficient to the biological feature vector of the spatial dimension.

[0090] In the above solution, the execution order of S310 - S340 is not limited to the order shown in the figure.

[0091] In some embodiments, S310 and S330 can be executed after S320 and S340. That is, S320 and S340 can first perform spatial attention processing on the biometric feature vectors of each spatial dimension to update the biometric feature set of the video frame to be classified; S310 and S330 can perform channel attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified again. In some embodiments, S310 and S330 can be executed before S320 and S340. That is, S310 and S330 can perform channel attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified; S320 and S340 can again perform spatial attention processing on the biometric feature vectors of each spatial dimension to update the biometric feature set of the video frame to be classified again.

[0092] In some embodiments, S310 and S320 may be executed first, and then S330 and S340 may be executed.

[0093] In some embodiments, the above-mentioned individual state determination method is implemented based on an individual state determination model. Before applying the individual state determination model to the individual state determination method, the individual state determination model needs to be trained so that the individual state determination model's individual state determination capability reaches the expected level. Alternatively, after applying the individual state determination model to the individual state determination task, the individual state determination model can be periodically trained to further improve the individual state determination capability of the individual state determination model.

[0094] The following describes the training of the individual state determination model. Generally speaking, the training of the individual state determination model described below is divided into the first phase of training, the second phase of training, and the third phase of training. The first phase of training, the second phase of training, and the third phase of training can all be performed independently.

[0095] The first and second phases of training are conducted before the individual state determination model is applied to the individual state determination method, while the third phase of training is conducted after the individual state determination model is applied to the individual state determination task. The third phase of training can be considered a supplement to the first or second phase of training.

[0096] When actually training the individual state determination model, the first, second, and third phases of training can be performed selectively, or all of them can be performed simultaneously. In the case of performing all of them simultaneously, the first phase of training can be performed first, and then the second phase of training can be performed based on the first phase of training, and then the third phase of training can be performed based on the second phase of training.

[0097] The following is an example of training an individual state determination model:

[0098] Figure 4 This is a flow chart of an embodiment of the training method of the individual state determination model of the present application. This embodiment is the first stage of training. Figure 4 As shown, the training method of the individual state determination model includes the following steps:

[0099] S410: Using the individual state determination model, feature extraction is performed on each first training video frame in the first training set to obtain a biological feature set of each first training video frame.

[0100] The training embodiment of the present application is performed by a training device, which can be any electronic device capable of determining individual status. The training device and the aforementioned individual status determination device can be the same or different.

[0101] The first training video frames have true category labels, and the biometric feature vectors of each channel dimension in the biometric feature set of each first training video frame are used to describe the training individuals in the corresponding first training video frames from different angles.

[0102] The true category label of the first training video frame represents the true category to which the state of the training individual in the first training video frame belongs.

[0103] The processing of S410 in this embodiment is similar to that of S110 and will not be described in detail here.

[0104] S420: Using the individual state determination model, perform attention processing on the biometric feature vectors of each channel dimension in the biometric feature set of each first training video frame to update the biometric feature set of each first training video frame.

[0105] The processing of S420 in this embodiment is similar to that of S120 and will not be described in detail here.

[0106] S430: Classifying the status of the training individuals in the corresponding first training video frames based on the updated biometric feature sets of the first training video frames to obtain a second classification result.

[0107] The second classification result may include a predicted category label of the first training video frame. The predicted category label of the first training video frame represents the predicted category to which the state of the training individual in the first training video frame belongs.

[0108] The second classification result may further include a confidence level of the predicted category label of the first training video frame. The confidence level of the predicted category label of the first training video frame is used to represent the credibility of the predicted category label.

[0109] S440: Based on each second classification result and the corresponding true category label, perform first-stage training on the individual state determination model.

[0110] In some embodiments, a loss function may be constructed based on the difference between the predicted category label of each first training video frame and the corresponding predicted category label; and the parameters of the individual state determination model may be adjusted based on the loss function.

[0111] In some embodiments, a target first training video frame with a confidence greater than a confidence threshold can be selected; a loss function is constructed based on the difference between the predicted category label and the true category label of the target first training video frame; and the parameters of the individual state determination model are adjusted based on the loss function.

[0112] Figure 5 This is a flow chart of another embodiment of the training method of the individual state determination model of the present application. This embodiment is the second stage training. Figure 5 As shown, after S440, the following steps may also be included:

[0113] S450: Classify each second training video frame in the second training set using the individual state determination model to obtain a third classification result.

[0114] The second training video frame may have a true category label or may not have a true category label. The true category label of the second training video frame represents the true category to which the state of the training individual in the second training video frame belongs.

[0115] The third classification result may include a predicted category label of the second training video frame. The predicted category label of the second training video frame represents the predicted category to which the state of the training individual in the second training video frame belongs.

[0116] The third classification result may further include a confidence level of the predicted category label of the second training video frame. The confidence level of the predicted category label of the second training video frame is used to represent the credibility of the predicted category label of the second training video frame.

[0117] The process of obtaining the third classification result is similar to the process of obtaining the second classification result, and will not be repeated here.

[0118] S460: Based on each third classification result, determine a representative training video frame from the second training set.

[0119] In some embodiments, a second training video frame whose corresponding predicted class label has a confidence score less than a confidence threshold can be identified from the second training set and used as a representative training video frame. It is understood that a confidence score less than the confidence threshold indicates that the second training video frame is a difficult sample for the individual state determination model to classify. Using this second training video frame as a representative training video frame for the second stage of training enables targeted optimization of the individual state determination model.

[0120] In some embodiments, the second training video frame has a true category label. In this case, S460 may include the following S461-S462 (not shown).

[0121] S461: Determine a second training video frame that meets a first preset condition from the second training set.

[0122] The first pre-set condition is that the difference between the predicted class label of the second training video frame and the corresponding true class label is greater than a difference threshold. It can be understood that a difference greater than the difference threshold indicates that the second training video frame is an incorrectly classified sample for the individual state determination model. Using this as a representative training video frame for the second stage of training enables targeted optimization of the individual state determination model.

[0123] S462: Determine a representative training video frame from the second training video frames that meet the first preset condition.

[0124] In some embodiments, in S462 , all second training video frames that meet the first preset condition may be directly used as representative training video frames.

[0125] In some embodiments, in S462 , a second training video frame that meets a first preset condition may be sent to a user; first identification information input by the user may be received; and the second training video frame indicated by the first identification information may be used as a representative training video frame.

[0126] The training device has a human-computer interaction interface, which can send the second training video frames that meet the first preset condition to the user in the form of a first list to be fed back through the human-computer interaction interface for the user to select.

[0127] The first list to be fed back may include a frame index, a true category label, the name of the training video to which it belongs, and the frame sequence number within the training video. In some embodiments, the first identification information may include the frame index of the second training video frame selected by the user. In some embodiments, the first identification information may also include the importance of the second training video frame, the difficulty level of classification, and the scarcity of the predicted category label distribution of each second training video frame within the training video frames.

[0128] In this case, the user may select at least some representative second training video frames from the first list to be fed back, and feed back the first identification information of the selected second training video frames to the training device.

[0129] Representative second training video frames can be those from difficult-to-classify categories or borderline categories. Borderline categories are those where the corresponding true class labels are close to each other; the closer the distance, the harder it is to distinguish. For example, for the four focus states of very focused, focused, unfocused, and extremely unfocused, very focused and focused are borderline categories, while unfocused and extremely unfocused are borderline categories.

[0130] S470: Performing second-stage training on the individual state determination model using representative training video frames.

[0131] In some embodiments, during the second stage training, the contribution of each representative training video frame is set to be equal.

[0132] In some embodiments, the contribution of each representative training video frame during the second stage of training is related to at least one of the corresponding importance, difficulty level, and scarcity of predicted class label distribution. For example, the higher the importance, the higher the difficulty level, and the higher the scarcity of predicted class label distribution, the greater the contribution.

[0133] Other detailed descriptions of the process of performing the second stage training using the representative training video frames are similar to the process of performing the first stage training using the first training set and are not repeated here.

[0134] The above scheme can, based on the third classification results of each second training video frame in the second training set, manually and / or automatically select at least some of the second training video frames as representative training video frames, and use the representative training video frames to perform the second stage training of the individual state determination model. Thus, during the training process, the training process can be targeted and optimized through automatic or manual intervention, further improving learning ability and classification accuracy while taking into account efficiency, ensuring that the individual state determination model has greater adaptability and accuracy when processing difficult-to-classify samples / misclassified samples, enabling the individual state determination model to adaptively respond to complex individual state determination scenarios, and ultimately achieving individual state determination in diverse individual state determination scenarios.

[0135] In addition, through the above-mentioned first-stage training and second-stage training methods, limited training resources with real category labels can be fully utilized.

[0136] Figure 6 This is a flow chart of another embodiment of the training method of the individual state determination model of the present application. This embodiment is the third stage of training. Figure 6 As shown, after S130, the following steps may also be included:

[0137] S510: Collect the video frames to be classified, and use the video frames to be classified to determine a third training set.

[0138] In some embodiments, all video frames to be classified may be directly determined as the third training set.

[0139] In some embodiments, S510 may include the following S511 - S512 (not shown).

[0140] S511: Determine a video frame to be classified that meets a second preset condition.

[0141] The second preset condition includes that the confidence of the predicted category label of the video frame to be classified is less than the confidence threshold, or the distribution of the predicted category labels of the video frames to be classified included in the same video to be classified is abnormal.

[0142] It is understood that a confidence score less than the confidence threshold indicates uncertainty in the individual state determination model's prediction of the video frame to be classified. This uncertainty may stem from the accuracy of the individual state determination model or the difficulty of classifying the video frame to be classified. This video frame to be classified is a difficult sample for the individual state determination model, and using it in the third stage of training can provide targeted optimization for the individual state determination model.

[0143] Understandably, an abnormal distribution of predicted class labels across frames within the same video to be classified indicates significant discrepancies and inconsistencies in the individual state determination results for different frames. These frames are difficult for the individual state determination model to classify, and using them in the third stage of training allows for targeted optimization of the individual state determination model.

[0144] S512: Determine a third training set based on the video frames to be classified that meet the second preset condition.

[0145] In some embodiments, in S512 , all the to-be-classified video frames that meet the second preset condition may be directly determined as the third training set.

[0146] In some embodiments, in S512, the video frames to be classified that meet the second preset condition can be fed back to the user; second identification information fed back by the user is received, and the second identification information is obtained based on the third training video frame selected by the user from the video frames to be classified that meet the second preset condition; the third training video frame pointed to by the frame index and the corresponding true category label are combined to obtain a third training set.

[0147] The training device has a human-computer interaction interface, which can send the second training video frames that meet the second preset condition to the user in the form of a second list to be fed back through the human-computer interaction interface for the user to select.

[0148] The second list to be fed back may include a frame index, the name of the video to be classified, and the frame sequence number within the video to be classified. In some embodiments, the second identification information includes the frame index and true category label of the third training video frame selected by the user. In some embodiments, the second identification information may also include the importance of the third training video frame, the difficulty level of classification, and the scarcity of the predicted category label distribution for each third training video frame within the training video frames.

[0149] In this case, the user can select at least some representative video frames to be classified from the second feedback list as the third training video frames, assign true category labels to the selected third training video frames, and feed back information such as the frame index and true category label of the selected third training video frames as the second identification information to the training device, so that the training device obtains the third training set based on the second identification information.

[0150] S520: Performing a third stage of training on the individual state determination model based on the third training set.

[0151] In some embodiments, during the third stage training, the contribution of each third training video frame is set to be equal.

[0152] In some embodiments, the contribution of each third training video frame during the third stage of training is related to at least one of the corresponding importance, difficulty level, and predicted category label distribution scarcity. For example, the higher the importance, the higher the difficulty level, and the higher the predicted category label distribution scarcity, the greater the contribution.

[0153] Other detailed descriptions related to the process of conducting the third stage training are similar to those of conducting the first and second stage training and will not be repeated here.

[0154] The above scheme extends the training of the individual state determination model to the application stage, and expands at least part of the video frames to be classified in the application stage into a third training set through manual and / or automatic selection, and uses the third training set to perform the third stage training of the individual state determination model, so as to perform targeted optimization of the individual state determination model in the application stage of the individual state determination model, while taking into account efficiency and further improving the learning ability and classification accuracy, and improving the individual state determination model to have stronger adaptability and accuracy when processing difficult-to-classify samples / incorrectly classified samples, so that the individual state determination model can adaptively respond to complex individual state determination scenarios, and ultimately realize individual state determination in diverse individual state determination scenarios.

[0155] Furthermore, in at least one of the first stage training, the second stage training, and the third stage training, a BN layer (normalization layer), Dropout, L2 regularization, and an early stopping mechanism can be added to prevent overfitting, and the cross entropy loss function and the Adam optimizer can be used to optimize the weights of the individual state determination model classification module.

[0156] To facilitate understanding of the present application, the following describes, in the form of a specific example, how the individual state determination method provided by the present application is used to perform the task of determining the student's concentration state.

[0157] Figure 7 This is a flow chart of a specific example of the method for determining individual status provided by this application. Figure 7 As shown, the individual state determination method includes biometric feature set extraction, feature enhancement based on attention mechanism, and classification stage.

[0158] 1. Biometric Feature Set Extraction Stage

[0159] 1. Obtain a video of student A to be classified. The video to be classified includes several consecutive video frames to be classified. Execute the subsequent steps for each video frame to be classified.

[0160] 2. Perform feature extraction on the video frame to be classified to obtain a set of biometric features of the video frame to be classified.

[0161] (1) Extract the head features of the video frame to be classified and obtain the head feature vector of student A in the video frame to be classified.

[0162] Specifically, the head of the video frame to be classified can be detected based on the target detection algorithm such as the Haar feature cascade classifier to obtain the head area in the video frame to be classified; the size of the head area can be cropped and scaled to make it uniform to a fixed size (for example, 224×224 pixels); the head area can be grayscale processed; and the features of the head area can be extracted using a feature extraction network such as ResNet50 as the head feature.

[0163] (2) Perform posture feature extraction on the video frames to be classified to obtain the posture feature vector of student A in the video frames to be classified. The posture feature vectors in multiple video frames to be classified can represent the movement changes of student A.

[0164] Specifically, the overall key point detection can be performed on the video frame to be classified to obtain the overall key point set. For example, multiple CNN convolutional layers in the posture recognition model MediaPipe Pose can be used for key point detection. The key point set includes the three-dimensional coordinates of several key points. For example, the overall key point set includes 11 key points, including 9 head key points (4 eye key points, 2 ear key points, 2 mouth key points and 1 nose key point, 2 shoulder key points. The three-dimensional coordinates of the key points can be expressed as (x, y, z), where x, y, and z represent the horizontal position, vertical position, and depth information of the key points, respectively.

[0165] (3) Eye feature extraction is performed on the video frames to be classified to obtain the eye feature vector of student A in the video frames to be classified. The eye feature vectors in multiple video frames to be classified can represent the eye movement behavior of student A.

[0166] Specifically, eye keypoint detection can be performed on the video frames to be classified to obtain an eye keypoint set. For example, the multi-layer residual network in the MediaPipeFacemesh face reconstruction model can be used for eye keypoint detection. The eye keypoint set includes several eye keypoints. For example, the eye keypoint set includes 16 eye keypoints, including the inner and outer corners of the eyes, the upper and lower center points of the eyes, and the intersection of the iris and sclera.

[0167] (4) Student A’s head features, posture features, and eye features constitute the biometric feature set X(H×W×C) of the video frame to be classified. Here, H×W represents the spatial dimensions, H and W represent the height and width, respectively, and C represents the channel dimension. C = 3, representing the head channel dimension, posture channel dimension, and eye channel dimension, respectively.

[0168] 2. Feature Enhancement Stage Based on Attention Mechanism

[0169] 1. Use SENet to perform channel attention processing on the biometric feature set X(H×W×C) of the classified video frame.

[0170] Figure 8 This is a schematic diagram of the channel attention processing flow of SENet in this application. Figure 8 As shown in Figure 1, SENet includes three stages: Squeeze (compression), Excitation (excitation) and Scale (weighting).

[0171] Squeeze stage: perform global average pooling operation F on the biological feature vectors of each channel dimension of X(H×W×C) Squeeze , and obtain the first feature compression value (1×1×C) of each channel dimension.

[0172] Excitation stage: Perform a full connection operation F on the first feature compression value of each channel dimension Excitation , and obtain the first channel attention coefficient (1×1×C) of each channel dimension.

[0173] Scale stage: multiply the first channel attention coefficient of each channel dimension by the biological feature vector of the corresponding channel dimension F Scale , and obtain the biometric feature set X'(H×W×C).

[0174] Among them, before the Squeeze stage, the feature transformation F of X can also be performed transform , to extract more advanced semantic features.

[0175] 2. Use the CBAM (Convolutional Block Attention Module) module to perform channel attention processing and spatial attention processing on the biological feature set X'(H×W×C) of the video frame to be classified.

[0176] Figure 9 This is a flow chart of the channel attention processing and spatial attention processing of CBAM in this application. Figure 9 As shown in Figure 2, the channel attention processing and spatial attention processing processes of CBAM include:

[0177] (1) Channel attention processing

[0178] The maximum pooling operation and the average pooling operation are performed on the biological feature vector of each channel dimension of X'(H×W×C) respectively to obtain the maximum pooling compression value (1×1×C) of each channel dimension and the average pooling compression value (1×1×C) of each channel dimension, that is, the second feature compression value.

[0179] Use the shared fully connected layer (Shared MLP) to perform full connection operations on the maximum pooling compression value of each channel dimension and the average pooling compression value of each channel dimension to obtain the maximum pooling attention coefficient of each channel dimension and the average pooling attention coefficient of each channel dimension.

[0180] The maximum pooling attention coefficient of each channel dimension is added to the average pooling attention coefficient of the corresponding channel dimension to obtain the second channel attention coefficient (1×1×C) of each channel dimension.

[0181] Multiply the second channel attention coefficient of each channel dimension by the biometric feature vector of the corresponding channel dimension to obtain the biometric feature set X”(H×W×C).

[0182] (2) Spatial attention processing

[0183] Perform maximum pooling and average pooling operations on the biological feature vectors of each spatial dimension of X' respectively to obtain the maximum pooling feature compression value (H × W × 1) of each spatial dimension and the average pooling feature compression value (H × W × 1) of each spatial dimension, that is, the third feature compression value.

[0184] The maximum pooled feature compression value of each spatial dimension is spliced ​​with the maximum pooled feature compression value of the corresponding spatial dimension to obtain the feature compression value splicing result of each spatial dimension (H×W×2).

[0185] Based on the concatenation results of the feature compression values ​​of each spatial dimension, the spatial attention coefficient (H×W×1) of each spatial dimension is obtained.

[0186] Multiply the spatial attention coefficient of each spatial dimension by the biometric vector of the corresponding spatial dimension to obtain the biometric set X''(H×W×C).

[0187] 3. Classification stage

[0188] For each video frame to be classified, the state of student A in the video frame to be classified is classified based on X' to obtain a first classification result. The first classification result includes the predicted category label of the video frame to be classified and its confidence.

[0189] For ease of understanding, the training method of the individual state determination model of the present application is described below in the form of a specific example.

[0190] Figure 10 This is a flowchart of a specific example of the training method of the individual state determination model of this application. Figure 10 As shown, the training method of the individual state determination model includes:

[0191] 1. Obtain a training dataset consisting of several training video frames, each with a true class label. The true class label represents the true class of the student's focus state in the training video frame. There are four true class labels: very focused, focused, not focused, and extremely not focused.

[0192] 2. The total training set is divided into a first training set and a second training set, and the training video frames in the first training set are used as the first training video frames, and the training video frames in the second training set are used as the second training video frames.

[0193] 3. Perform the first phase of training on the individual state determination model on the first training set. The first phase of training can be divided into the extraction of biometric feature sets, feature enhancement based on the attention mechanism, classification phase, and loss calculation phase.

[0194] In the biometric feature set extraction stage, the individual state determination model is used to extract features from each first training video frame to obtain a biometric feature set for each first training video frame.

[0195] In the feature enhancement stage based on the attention mechanism, the individual state determination model is used to perform attention processing on the biometric feature set of each first training video frame.

[0196] In the classification stage, based on the biometric feature set of each first training video frame, the state of the training individual in the biometric feature set of each first training video frame is classified to obtain a second classification result. The second classification result includes a predicted category label of the first training video frame.

[0197] In the loss calculation stage, a loss function is calculated based on the difference between the predicted category label of each first training video frame and the corresponding true category label; and the parameters of the individual state determination model are adjusted based on the loss function.

[0198] After iterating the above process for 30 rounds, the first stage of training ends and the second stage of training begins.

[0199] 4. Select representative training video frames from the second training set to perform the second stage training on the individual state determination model.

[0200] (1) Obtain a third classification result for each second training video frame using the individual state model. The third classification result includes the predicted category label and its confidence level for the second training video frame.

[0201] (2) Determine from the second training set a second training video frame whose predicted class label differs from the true class label by more than a difference threshold, and form a first list to be fed back to the user. The first list to be fed back may include a frame index (sample_index1), a true class label (true_label1), the name of the training video to which it belongs (video_name1), and a frame number (frame_count1) in the training video to which it belongs. An example is shown in Table 1.

[0202] Table 1 The first list to be fed back

[0203]

[0204] (3) Receive the frame index of the second training video frame input by the user.

[0205] (4) The second training video frame pointed to by the frame index is used as the representative training video frame. If the frame index is a valid index, the second training video frame pointed to by the frame index can be used as the representative training video frame. If the frame index is an invalid index, a prompt message is sent to the user to prompt the user to re-enter a valid index. When a specific identifier (e.g., "o") input by the user is received, it indicates that the user has completed the feedback.

[0206] (5) The second stage of training of the individual state determination model is performed using representative training video frames.

[0207] After the second stage of training, the individual state determination model can be deployed in the application scenario to carry out the individual state determination task.

[0208] 5. After being deployed in the application scenario, periodically collect the video frames to be classified in the application scenario, determine the third training set from the video frames to be classified, and use the third training video frames to perform the third stage training on the individual state determination model.

[0209] (1) Determine whether the confidence level of the predicted category label is less than the confidence threshold, and form a second list to be fed back and send it to the user. The second list to be fed back may include a frame index (sample_index2), the name of the video to be classified (video_name2), and the frame number in the video to be classified (frame_count2). See Table 2 for an example.

[0210] Table 2 The second list of feedback

[0211]

[0212] (2) Receive the frame index and the corresponding true category label input by the user to obtain the third training set.

[0213] (3) Use the third training set to conduct the third stage training of the individual state determination model.

[0214] Figure 11 This is a flow chart of an embodiment of the individual status determination device of the present application. Figure 11 As shown, the individual state determination device 60 includes a feature extraction module 61, an attention processing module 62 and a classification module 63.

[0215] The feature extraction module 61 is used to extract features from the video frame to be classified to obtain a set of biometric features of the video frame to be classified; the biometric feature vectors of each channel dimension in the set of biometric features of the video frame to be classified are used to describe the individual to be classified in the video frame to be classified from different angles;

[0216] an attention processing module 62 for performing attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified;

[0217] The classification module 63 is configured to classify the status of the individual to be classified in the video frame to be classified based on the updated biometric feature set of the video frame to be classified, and obtain a first classification result.

[0218] For other detailed descriptions of the individual status determination device 60 , please refer to the previous embodiments, which will not be repeated here.

[0219] Figure 12 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application. Figure 12 As shown, electronic device 70 includes memory 71 and processor 72. Processor 72 is configured to execute program instructions stored in memory 71 to implement the steps of any of the above method embodiments. In a specific implementation scenario, electronic device 70 may include, but is not limited to, a microcomputer and a server. In addition, electronic device 70 may also include a carrier device such as a laptop computer and a tablet computer, which is not limited here.

[0220] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0221] See also Figure 13 , Figure 13 The computer-readable storage medium 80 stores program instructions 81, which, when executed by a processor, implement the steps of any of the above method embodiments.

[0222] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0223] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0224] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0225] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0226] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A method for determining an individual state, characterized in that: include: Performing feature extraction on the video frame to be classified to obtain a biometric feature set of the video frame to be classified; a biometric feature vector of each channel dimension in the biometric feature set of the video frame to be classified is used to describe the individual to be classified in the video frame to be classified from different angles; Performing attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified; Based on the updated biometric feature set of the video frame to be classified, the status of the individual to be classified in the video frame to be classified is classified to obtain a first classification result.

2. The method according to claim 1, characterized in that The performing attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified includes: Performing attention processing on the biological feature vectors of each channel dimension according to the first channel attention processing method to obtain the first channel attention coefficient of each channel dimension; The first channel attention coefficients of the channel dimensions are respectively applied to the biological feature vectors of the corresponding channel dimensions.

3. The method according to claim 2, characterized in that The method of performing attention processing on the biological feature vectors of each channel dimension according to the first channel attention processing method to obtain the first channel attention coefficient of each channel dimension includes: Compressing the biometric feature vectors of each channel dimension respectively according to a first compression method to obtain a first feature compression value of each channel dimension, wherein the first compression method includes a global pooling operation; A full connection operation is performed on the first feature compression value of each channel dimension to obtain the first channel attention coefficient of each channel dimension.

4. The method according to claim 2, characterized in that After the first channel attention coefficients of the channel dimensions are applied to the biological feature vectors of the corresponding channel dimensions respectively, the method further includes: Performing attention processing on the biological feature vectors of each channel dimension according to the second channel attention processing method to obtain a second channel attention coefficient of each channel dimension; and Performing spatial attention processing on the biometric feature vectors of each spatial dimension in the biometric feature set of the video frame to be classified to obtain a spatial attention coefficient of each spatial dimension; Applying the second channel attention coefficient of each channel dimension to the biological feature vector of the corresponding channel dimension; and The spatial attention coefficient of each spatial dimension is applied to the biological feature vector of the corresponding spatial dimension.

5. The method according to claim 4, characterized in that The method of performing attention processing on the biological feature vectors of each channel dimension according to the second channel attention processing method to obtain the second channel attention coefficient of each channel dimension includes: Compressing the biometric feature vectors of each channel dimension respectively according to a second compression method to obtain second feature compression values ​​of each channel dimension, wherein the second compression method includes a global pooling operation; Performing a full connection operation on the second feature compression value of each channel dimension to obtain a second channel attention coefficient of each channel dimension; And / or, performing spatial attention processing on the biometric feature vectors of each spatial dimension in the biometric feature set to obtain a spatial attention coefficient of each spatial dimension includes: compressing the biometric feature vectors of each spatial dimension respectively according to a third compression method to obtain a third feature compression value of each spatial dimension, wherein the third compression method includes at least one of a maximum pooling operation and an average pooling operation; Based on the third feature compression value of each spatial dimension, a spatial attention coefficient of each spatial dimension is obtained.

6. The method according to claim 1, characterized in that The channel dimensions include head channel dimension, posture channel dimension, and eye channel dimension; and / or, The individual state determination method is implemented based on an individual state determination model, and the training step of the individual state determination model includes: Using the individual state determination model, feature extraction is performed on each first training video frame in the first training set to obtain a biometric feature set for each first training video frame, wherein the first training video frame has a true category label, and a biometric feature vector of each channel dimension in the biometric feature set for each first training video frame is used to describe the training individual corresponding to the first training video frame from different perspectives; Using the individual state determination model, attention processing is performed on the biometric feature vectors of each channel dimension in the biometric feature set of each first training video frame to update the biometric feature set of each first training video frame; classifying the status of the training individuals in the first training video frames based on the updated biometric feature sets of the first training video frames to obtain second classification results; Based on each of the second classification results and the corresponding true category labels, the individual state determination model is trained in the first stage.

7. The method according to claim 6, characterized in that After performing the first stage training on the individual status determination model based on each of the first classification results and the corresponding true category labels, the method further includes: classifying each second training video frame in the second training set using the individual state determination model to obtain a third classification result; determining a representative training video frame from the second training set based on each of the third classification results; The representative training video frames are used to perform a second stage of training on the individual state determination model.

8. The method according to claim 7, characterized in that The second training video frame has the true category label, the third classification result includes the predicted category label of the second training video frame, and determining a representative training video frame from the second training set based on each of the third classification results includes: Determining a second training video frame from the second training set that meets a first preset condition; the first preset condition includes that a difference between a predicted category label of the second training video frame and the corresponding true category label is greater than a difference threshold; The representative training video frame is determined from the second training video frames that meet the first preset condition.

9. The method according to claim 8, characterized in that The determining the representative training video frame from the second training video frames that meet the first preset condition includes: Sending the second training video frame that meets the first preset condition to the user; receiving first identification information input by the user; The second training video frame indicated by the first identification information is used as the representative training video frame.

10. The method according to claim 1, characterized in that The individual state determination method is implemented based on an individual state determination model. After classifying the state of the individual to be classified in the video frame to be classified based on the updated biometric feature set of the video frame to be classified and obtaining a first classification result, the method further includes: Collecting the video frames to be classified, and using the video frames to be classified to determine a third training set; The individual state determination model is trained in a third stage based on the third training set.

11. The method according to claim 10, characterized in that The first classification result includes a predicted category label of the video frame to be classified and a confidence level of the predicted category label of the video frame to be classified. The determining of the third training set using each of the video frames to be classified includes: Determining a video frame to be classified that meets a second preset condition; the second preset condition includes that the confidence of the predicted category label of the video frame to be classified is less than a confidence threshold, or that the distribution of the predicted category labels of the video frames to be classified included in the same video to be classified is abnormal; The third training set is determined based on the video frames to be classified that meet the second preset condition.

12. The method according to claim 11, characterized in that The determining the third training set based on the video frames to be classified that meet the second preset condition includes: Sending the video frames to be classified that meet the second preset condition to the user; receiving second identification information input by the user, where the second identification information is obtained based on a third training video frame selected by the user from the video frames to be classified that meet the second preset condition, and the second identification information includes a frame index and a true category label of the third training video frame selected by the user; The third training video frame pointed to by the frame index and the corresponding true category label are combined to obtain the third training set.

13. An individual status determination device, characterized in that: include: A feature extraction module is used to extract features from the video frame to be classified to obtain a set of biometric features of the video frame to be classified; the biometric feature vectors of each channel dimension in the set of biometric features of the video frame to be classified are used to describe the individual to be classified in the video frame to be classified from different angles; an attention processing module, configured to perform attention processing on the biometric feature vectors of each channel dimension to update the biometric feature set of the video frame to be classified; The classification module is configured to classify the status of the individual to be classified in the video frame to be classified based on the updated biometric feature set of the video frame to be classified, and obtain a first classification result.

14. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images