Concentration Evaluation Method, Device, Computer Equipment and Storage Medium

Abstract

The present application relates to a method, apparatus, computer device, and storage medium for assessing concentration. The method comprises: extracting global image features from a target image frame; extracting image context features from the target image frame; identifying the expression and posture of a target object in the target image frame, as well as an object in the target image frame, based on fusion features between the global image features and the image context features; determining the interaction between the target object and the object based on the fusion features; and determining the concentration of the target object based on the expression, posture, type of the object, and the interaction. This method can improve the accuracy of concentration assessment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a concentration assessment method, apparatus, computer equipment, and storage medium. Background Art

[0002] In recent years, the continuous development of artificial intelligence (AI) technology has led to its widespread application in various fields, such as attention assessment in classroom teaching. In classroom teaching, if students' attention levels are low, they can easily miss important learning points. Therefore, using AI technology to identify the attention levels of students captured from facial images can effectively intervene with low-attention students and optimize the quality of classroom teaching. However, determining attention levels through facial recognition can affect the accuracy of attention assessments. Summary of the Invention

[0003] Based on this, it is necessary to provide a concentration assessment method, device, computer equipment and storage medium that can improve the accuracy of concentration assessment in response to the above technical problems.

[0004] A concentration assessment method, comprising:

[0005] extracting global image features from the target image frame;

[0006] extracting image context features from the target image frame;

[0007] Recognizing the expression and posture of the target object in the target image frame, and the object in the target image frame based on the fusion features between the global image features and the image context features;

[0008] determining an interaction behavior between the target object and the item according to the fusion feature;

[0009] The concentration of the target object is determined based on the expression, the posture, the type of the object, and the interactive behavior.

[0010] A concentration assessment device, comprising:

[0011] A global feature extraction module, used to extract global image features from the target image frame;

[0012] A context feature extraction module, configured to extract image context features from the target image frame;

[0013] a recognition module, configured to recognize an expression and a posture of a target object in the target image frame, and an object in the target image frame based on a fusion feature of the global image feature and the image context feature;

[0014] an interaction determination module, configured to determine an interaction behavior between the target object and the item based on the fusion feature;

[0015] A concentration determination module is used to determine the concentration of the target object based on the expression, the posture, the type of the object and the interactive behavior.

[0016] In one embodiment, the apparatus further comprises:

[0017] A video decoding module is used to decode the video stream to obtain an image frame sequence;

[0018] An image frame extraction module, configured to extract image frames from the image frame sequence to obtain a target image frame;

[0019] The image processing module is used to perform image processing on the target image frame to obtain a processed target image frame.

[0020] In one embodiment, the image processing module is further configured to:

[0021] Performing inversion processing on the target image frame to obtain an inverted image frame;

[0022] Obtaining a pixel value of each pixel in the inverted image frame;

[0023] determining a transformation parameter corresponding to each pixel point in the inverted image frame based on the pixel value;

[0024] According to the pixel value of each pixel point in the inverted image frame and the corresponding transformation parameters, the target image frame is subjected to nonlinear transformation to obtain a processed target image frame.

[0025] In one embodiment, the global feature extraction module is further configured to:

[0026] performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame;

[0027] Inputting the normalized denoised image frame into a concentration assessment model;

[0028] The global image features in the normalized denoised image frame are extracted through the feature extraction network in the focus evaluation model.

[0029] In one embodiment, the context feature extraction module is further configured to:

[0030] performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame;

[0031] Performing block processing on the normalized denoised image frame to obtain image sequence blocks;

[0032] Inputting the image sequence blocks into a concentration assessment model;

[0033] Extracting contextual features of each image block in the image sequence block through the multi-attention encoding network in the focus evaluation model;

[0034] The multi-layer perceptron in the concentration evaluation model is used to perform feature fusion on the context features of each image block to obtain fused image context features.

[0035] In one embodiment, the concentration determination module is further configured to:

[0036] determining an expression score of the target object based on the expression;

[0037] determining a posture score of the target object based on the posture;

[0038] Determine an item score of the target object based on the type of the item; determine an interaction score of the target object based on the interaction behavior;

[0039] The concentration of the target object is determined according to the expression score, the posture score, the object score, and the interaction score.

[0040] In one embodiment, the concentration determination module is further configured to:

[0041] Get the weights corresponding to the expression dimension, posture dimension, object dimension, and interaction dimension respectively;

[0042] Based on the weights of the different dimensions, performing a weighted summation on the expression score, the posture score, the item score, and the interaction score;

[0043] The concentration of the target object is determined according to a weighted score obtained by weighted summation.

[0044] In one embodiment, the target object is a person in a target group; the apparatus further comprises a chart generating module for:

[0045] Obtaining the concentration of each character object at a historical moment;

[0046] Determining the average concentration of the target group at different moments based on the currently determined concentration of each character object and the concentration of the historical moments;

[0047] A concentration analysis graph of the target group is generated according to the average concentration at each moment.

[0048] In one embodiment, the chart generation module is further configured to:

[0049] Obtaining the concentration of each character object at a historical moment;

[0050] Based on the currently determined concentration of each character object and the concentration at the historical moment, a concentration analysis graph of each character object is generated.

[0051] In one embodiment, the identification module is further configured to:

[0052] Performing feature fusion on the global image feature and the image context feature to obtain a fused feature;

[0053] extracting target object features and item features from the fused features;

[0054] recognizing the expression and posture of the target object in the target image frame based on the target object features;

[0055] An object in the target image frame is identified based on the object features.

[0056] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0057] extracting global image features from the target image frame;

[0058] extracting image context features from the target image frame;

[0059] Recognizing the expression and posture of the target object in the target image frame, and the object in the target image frame based on the fusion features between the global image features and the image context features;

[0060] determining an interaction behavior between the target object and the item according to the fusion feature;

[0061] The concentration of the target object is determined based on the expression, the posture, the type of the object, and the interactive behavior.

[0062] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0063] extracting global image features from the target image frame;

[0064] extracting image context features from the target image frame;

[0065] Recognizing the expression and posture of the target object in the target image frame, and the object in the target image frame based on the fusion features between the global image features and the image context features;

[0066] determining an interaction behavior between the target object and the item according to the fusion feature;

[0067] The concentration of the target object is determined based on the expression, the posture, the type of the object, and the interactive behavior.

[0068] A computer program, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the following steps:

[0069] extracting global image features from the target image frame;

[0070] extracting image context features from the target image frame;

[0071] Recognizing the expression and posture of the target object in the target image frame, and the object in the target image frame based on the fusion features between the global image features and the image context features;

[0072] determining an interaction behavior between the target object and the item according to the fusion feature;

[0073] The concentration of the target object is determined based on the expression, the posture, the type of the object, and the interactive behavior.

[0074] The above-mentioned concentration assessment method, apparatus, computer equipment and storage medium extract global image features and image context features from the target image frame, and based on the fusion features between the global image features and the image context features, identify the expression and posture of the target object in the target image frame, as well as the objects in the target image frame, and determine the interaction behavior between the target object and the objects according to the fusion features. On the one hand, by fusing the features, the accuracy of the recognition results is improved. On the other hand, by comprehensively determining the concentration of the target object based on multiple dimensions such as expression, posture, type of object and interaction behavior, the accuracy of the concentration assessment is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A diagram showing an application environment of a concentration assessment method in one embodiment;

[0076] Figure 2 1 is a flow chart of a concentration evaluation method according to an embodiment;

[0077] Figure 3 Schematic diagram of the residual unit structure in one embodiment;

[0078] Figure 4 A schematic diagram of image block processing in one embodiment;

[0079] Figure 5 Schematic diagram of the structure of the Transformer Encoder module in one embodiment;

[0080] Figure 6 Schematic diagram of a flow chart of a target image frame acquisition step in one embodiment;

[0081] Figure 7 A schematic diagram of a concentration analysis chart of a target group in one embodiment;

[0082] Figure 8 A schematic diagram of an individual concentration analysis chart of a target group in one embodiment;

[0083] Figure 9 is a flowchart of a concentration assessment method according to another embodiment;

[0084] Figure 10 is a flowchart of a concentration assessment method according to another embodiment;

[0085] Figure 11 is a structural block diagram of a concentration evaluation device in one embodiment;

[0086] Figure 12 is a structural block diagram of a concentration evaluation device in one embodiment;

[0087] Figure 13 is a diagram of the internal structure of a computer device in one embodiment;

[0088] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0090] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0091] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0092] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0093] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common unmanned driving, autonomous driving, drones, robots, smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0094] The solution provided in the embodiments of this application involves artificial intelligence machine learning and other technologies, which are specifically explained through the following embodiments: The concentration evaluation method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The terminal 102 can capture the target image frame through a built-in or external image acquisition device, and then save the captured target image locally in the terminal 102 or upload it to the server 104, so that the terminal 102 or the server 104 executes the concentration evaluation method provided in this application. Taking the application of this method to the terminal 102 as an example, the terminal 102 extracts global image features from the target image frame; extracts image context features in the target image frame; based on the fusion features between the global image features and the image context features, identifies the expression and posture of the target object in the target image frame, as well as the objects in the target image frame; determines the interaction behavior between the target object and the object based on the fusion features; determines the concentration of the target object based on the expression, posture, type of object and interaction behavior.

[0095] Among them, the terminal 102 is but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud storage, network services, cloud communications, big data and artificial intelligence platforms. It can also be a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP). The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0096] In one embodiment, Figure 2 As shown in the figure, a method for evaluating concentration is provided. Figure 1 The computer device (including terminal 102 and server 104) in FIG. 1 is used as an example to illustrate the method, which includes the following steps:

[0097] S202: extract global image features from the target image frame.

[0098] The target image frame is an image frame containing the target object extracted from the image frame sequence contained in the video stream. For example, if the video stream is a video stream obtained by filming students during a classroom teaching process, the image frame containing the student can be extracted from the video stream as the target image frame; if the video stream is a video stream obtained by filming a teacher during a classroom teaching process, the image frame containing the teacher can be extracted from the video stream as the target image frame.

[0099] Global image features are features that represent the entire image. Global features, as opposed to local image features, describe overall characteristics of an image or object, such as color and shape. Global features include color, texture, and shape features, such as intensity histograms. Because they are low-level, pixel-level visual features, they exhibit good invariance, are computationally simple, and offer intuitive representation.

[0100] Specifically, after acquiring the target image frame, the computer device inputs the acquired target image frame into a pre-trained concentration evaluation model, and extracts the global image features of the target image frame through the feature extraction network of the concentration evaluation model.

[0101] Among them, the concentration assessment model is a pre-trained machine learning model for evaluating concentration, including a feature extraction network and a prediction network. The feature extraction network is used to extract global image features from the input image frame, and the prediction network is used to identify target objects, items and interactive behaviors in the image frame based on the input image features.

[0102] In one embodiment, the feature extraction network is constructed by Resnet Blockc residual units, which may specifically include multiple residual units. Image features with rich semantic information at different levels are extracted from the image frame through the multiple residual units, thereby obtaining the global image features of the target image frame.

[0103] Figure 3 A schematic diagram of the structure of a Resnet Block residual unit in one embodiment is shown, where X is the input of the residual unit, F(X) represents the residual mapping, and the output is: Y = F(X) + X. The Resnet Block residual unit establishes a direct association channel between the input and output by introducing identity mapping, so that the parameter layer focuses on learning the residual between the input and output.

[0104] S204: extracting image context features from the target image frame.

[0105] The target image frame is an image frame containing the target object extracted from the image frame sequence contained in the video stream. For example, if the video stream is a video stream obtained by filming students during a classroom teaching process, the image frame containing the student can be extracted from the video stream as the target image frame; if the video stream is a video stream obtained by filming a teacher during a classroom teaching process, the image frame containing the teacher can be extracted from the video stream as the target image frame.

[0106] Image context features refer to some or all information in the target image frame that can affect the scene and objects in the image, and can represent the relationship between the target object in the target image frame and other objects or the environment.

[0107] Specifically, after acquiring the target image frame, the computer device inputs the acquired target image frame into a pre-trained concentration evaluation model, and extracts the contextual features of the target image frame through the multi-attention encoding feature extraction network of the concentration evaluation model.

[0108] Among them, the concentration evaluation model is a pre-trained machine learning model for evaluating concentration, and also includes a multi-attention encoding feature extraction network, which is used to extract image context features from the input image frames.

[0109] In one embodiment, the multi-attention encoding feature extraction network includes a multi-attention encoding network and a multi-layer perceptron (MLP). The multi-attention encoding network is constructed by a Transformer Encoder module, which is composed of layers of multi-head self-attention (MSA) and MLP (multi-layer perceptron) blocks. By adopting a multi-head attention mechanism instead of the original single-head attention mechanism, the Transformer Encoder module can better fuse features and thus extract richer image context features.

[0110] The Transformer is a deep neural network primarily based on the self-attention mechanism, originally applied in the field of natural language processing. Inspired by the Transformer's powerful representational capabilities, researchers have proposed extending the Transformer to computer vision tasks. Compared to other network types (such as convolutional and recurrent networks), Transformer-based models have demonstrated superior performance on various vision benchmarks.

[0111] In one embodiment, after acquiring the target image frame, the computer device divides the target image frame into blocks to obtain image sequence blocks, and inputs the obtained image sequence blocks into a concentration evaluation model, extracts the context features of each image block through the multi-attention encoding feature extraction network in the concentration evaluation model, and performs feature fusion on the extracted context features of each image block to obtain fused image context features.

[0112] Specifically, after obtaining the target image frame, the computer device can also input the target image frame into a pre-trained concentration evaluation model, and perform linear projection mapping on the target image through the linear projection layer of the concentration evaluation model to obtain an image block sequence of a preset size.

[0113] In one embodiment, for a target image frame of size H×W×C, a linear projection mapping may be performed on it using the following formula to obtain an image block sequence of a preset size:

[0114]

[0115] Where H is the height of the target image frame, W is the width of the target image frame, C is the number of channels of the target image frame, P is the height and width of the obtained image block, and N is the number of obtained blocks. N will affect the length of the image sequence block and can be calculated by the following formula:

[0116]

[0117] Figure 4 A schematic diagram of image block processing in an embodiment is shown. In the figure, for the original target image frame, linear projection mapping is performed on it using formula (1) to obtain 9 16*16 image blocks, which are composed of image sequence blocks in order from left to right and from top to bottom.

[0118] In one embodiment, after obtaining an image sequence block, the computer device inputs the obtained image sequence block into a concentration assessment model, extracts contextual features of each image block in the image sequence block through a multi-attention encoding network in the concentration assessment model, and performs feature fusion on the contextual features of each image block through a multi-layer perceptron in the concentration assessment model to obtain fused image context features. The multi-attention encoding network is constructed by a Transformer Encoder module, and the Transformer Encoder module is composed of layers of Multi-head self-attention (MSA) and MLP (Multi-layer Perceptron) blocks.

[0119] Figure 5A structural schematic diagram of a Transformer Encoder module in one embodiment is shown. In the figure, the Transformer Encoder module is composed of layers of Multi-head self-attention (MSA) and MLP (Multi-layer Perceptron) blocks. Layer norm (LN) is applied before each block, and residual connection is applied after each block. The MLP includes two fully connected layers with GELU (Gaussian Error Linear Unit) nonlinearity. The image sequence blocks (Embedded Patches) for block processing of the target image frame are input into the Transformer Encoder module. The context features of each image block in the image sequence block are extracted through the Multi-head self-attention block and MLP block in the Transformer Encoder module. The context features of each image block in the obtained image sequence block are then input into the multi-layer perceptron in the concentration evaluation model. The context features of each image block are fused by the multi-layer perceptron in the concentration evaluation model to obtain fused image context features.

[0120] S206 , based on the fusion features between the global image features and the image context features, recognize the expression and posture of the target object in the target image frame, as well as the object in the target image frame.

[0121] Expressions include three emotion types: positive, neutral, and negative. It's understandable that happiness and surprise are positive emotions, while anger, disgust, fear, sadness, contempt, and doubt are negative. Postures include the target's head and body postures. Head posture refers to the pitch, yaw, and roll angles of the head, while body postures include raising a hand, standing, sitting upright, and lying on a table. Objects can include both normal and abnormal items. Normal items include study-related items like stationery and books, while abnormal items include non-study-related items like mobile phones and toys.

[0122] Specifically, after obtaining the global image features and image context features of the target image frame, the computer device fuses the global image features and image context features to obtain the fusion features of the target image frame, and identifies the expression and posture of the target object in the target image frame, as well as the objects in the target image frame based on the fusion features of the target image frame.

[0123] In one embodiment, after obtaining the fusion features of the target image frame, the computer device inputs the obtained fusion features into the prediction network of the concentration assessment model, and uses the prediction network to respectively identify the expression and posture of the target object in the target image frame, as well as the objects in the target image frame.

[0124] The prediction network of the focus assessment model can include multiple subnetworks, specifically an expression recognition subnetwork, a posture recognition subnetwork, and an object recognition subnetwork. The expression recognition subnetwork can identify the expression of the target object in the target image frame, the posture recognition subnetwork can identify the posture of the target object in the target image frame, and the object recognition subnetwork can identify the object contained in the target image frame. Each subnetwork is composed of fully connected layers.

[0125] S208: Determine the interaction behavior between the target object and the item based on the fusion features.

[0126] Among them, the interactive behavior can be the behavior of the target object operating or viewing the object, such as when the object is a book, the target object reading the book, when the object is a pen, the target object holding the pen to write, when the object is a mobile phone, the target object holding the mobile phone or viewing the mobile phone to play with the mobile phone.

[0127] Specifically, the computer device can directly identify the interaction behavior between the target object and the item based on the fusion features.

[0128] In one embodiment, after obtaining fused features of the target image frame, the computer device inputs the obtained fused features into the prediction network of the concentration assessment model. The gesture recognition subnetwork included in the prediction network directly identifies the interaction state between the target object and the object, and determines the interaction behavior between the target object and the object based on the identified interaction state category. The interaction state includes reading, writing, playing with a mobile phone, etc.

[0129] In one embodiment, after identifying the expression and posture of the target object in the target image frame and the object in the target image frame, the computer device can also determine the interaction behavior between the target object and the object based on at least one of the expression and posture of the target object.

[0130] In one embodiment, the computer device determines whether the expression of the target object matches the object. If so, it determines that there is an interaction between the target object and the object, and determines the interaction based on the target object.

[0131] For example, if the object is a book, if the target object's expression is neutral, it is determined that the target object has an interaction with the book, and the interaction is determined to be reading behavior; if the target object's expression is negative, it is determined that the target object does not have an interaction with the book; if the object is a mobile phone, if the target object's expression is positive, it is determined that the target object has an interaction with the mobile phone, and the interaction is determined to be playing with the mobile phone; if the target object's expression is negative, it is determined that the target object does not have an interaction with the mobile phone.

[0132] In one embodiment, the computer device determines whether the posture of the target object matches the object. If so, it determines that there is an interaction between the target object and the object, and determines the interaction based on the target object.

[0133] Specifically, the computer device can determine the position of the object and determine whether the posture of the target object matches that of the object. If the posture of the target object matches the position of the object, it is determined that the posture of the target object matches the object. If the posture of the target object does not match the position of the object, it is determined that the posture of the target object does not match the object.

[0134] For example, if the object is a book, if the target object's head posture matches the position of the book, it is determined that the target object's posture matches the object, it is determined that there is an interaction between the target object and the book, and the interaction behavior is determined to be a reading behavior. If the target object's head posture does not match the position of the book, it is determined that the target object's posture does not match the object, and it is determined that there is no interaction between the target object and the book. If the object is a mobile phone, if the target object's head posture matches the position of the mobile phone, it is determined that the target object's posture matches the book, it is determined that there is an interaction between the target object and the mobile phone, and the interaction behavior is determined to be a reading behavior. If the target object's head posture does not match the position of the mobile phone, it is determined that the target object's posture does not match the book, and it is determined that there is no interaction between the target object and the mobile phone.

[0135] In one embodiment, the computer device determines whether the expression of the target object matches the object, and whether the posture of the target object matches the object. If the expression and posture of the target object both match the object, it is determined that there is an interaction between the target object and the object, and the interaction is determined based on the target object; if the expression or posture of the target object does not match the object, it is determined that there is no interaction between the target object and the object.

[0136] For example, if the object is a book, if the target object's expression is neutral and the target object's head posture matches the position of the object, then it is determined that the target object has an interactive behavior with the book, and the interactive behavior is determined to be a reading behavior; if the target object's expression is negative and the target object's head posture matches the position of the object, then it is determined that there is no interactive behavior with the target object; if the target object's expression is positive and the target object's head posture does not match the position of the object, then it is determined that there is no interactive behavior with the target object; if the target object's expression is negative and the target object's head posture does not match the position of the object, then it is determined that there is no interactive behavior with the target object.

[0137] S210, determining the concentration of the target object based on the expression, posture, type of object and interactive behavior.

[0138] Among them, concentration is used to measure the degree of concentration of the target object during classroom teaching. The target object can be students or teachers.

[0139] Specifically, after identifying the target object's expression and posture, objects, and the target object's interaction behavior with objects, the computer device determines the target object's expression score and posture score, object score, and interaction score based on the target object's expression and posture, the object type, and the interaction behavior, and determines the target object's concentration based on the target object's expression score and posture score, object score, and interaction score.

[0140] In one embodiment, S210 includes the following steps: determining the expression score of the target object based on the expression; determining the posture score of the target object based on the posture; determining the item score of the target object based on the type of the item; determining the interaction score of the target object based on the interaction behavior; and determining the concentration of the target object based on the expression score, posture score, item score, and interaction score.

[0141] Specifically, the computer device determines the expression type of the target object's expression, and determines the score corresponding to the expression type as the expression score of the target object; determines the posture score of the target object based on the head posture and body posture of the target object; determines the type of object, and determines the score corresponding to the determined item type as the item score, and determines the interaction score based on the interaction behavior, and then determines the concentration of the target object based on the expression score, posture score, item score and interaction score.

[0142] For example, if the target object's expression is neutral, the score corresponding to the neutral emotion is obtained as the expression score of the target object. If the head posture is a pitch angle of 30° and a yaw angle of 35°, and the body posture is sitting upright, the head posture score corresponding to the pitch angle of 30° and the yaw angle of 35°, as well as the body posture score corresponding to sitting upright are obtained. If the object is a book, then the book is a normal object type, and the score corresponding to the normal object type is determined as the object score. If the interactive behavior is reading, then the interaction score corresponding to the reading behavior is obtained, and the concentration of the target object is determined based on the expression score, head posture score, body posture score, object score and interaction score.

[0143] In one embodiment, after obtaining the concentration of the target object, the computer device determines whether the concentration of the target object is lower than a preset concentration threshold. If so, a prompt message is generated for prompting.

[0144] For example, if the target object is a student, when the student's concentration is lower than the concentration threshold, a concentration prompt message can be generated and sent to the terminal corresponding to the teacher or the terminal corresponding to the student, so as to prompt the teacher to intervene in the student, or prompt the student to concentrate on listening; if the target object is a teacher, when the teacher's concentration is lower than the concentration threshold, a concentration prompt message can be generated and sent to the terminal corresponding to the teacher or the terminal corresponding to the teaching quality supervisor, so as to prompt the teacher to adjust the teaching status, or prompt the teaching quality supervisor to pay attention to the teacher's classroom teaching situation.

[0145] The above-mentioned concentration assessment method extracts global image features and image context features from the target image frame, and based on the fusion features between the global image features and the image context features, identifies the expression and posture of the target object in the target image frame, as well as the objects in the target image frame, and determines the interaction behavior between the target object and the objects according to the fusion features. On the one hand, by fusing the features, the accuracy of the recognition results is improved. On the other hand, by comprehensively determining the concentration of the target object based on multiple dimensions such as expression, posture, type of object and interaction behavior, the accuracy of the concentration assessment is improved.

[0146] In one embodiment, before extracting global image features from the target image frame, the computer device Figure 6 As shown, the above-mentioned concentration evaluation method further includes the following steps:

[0147] S602: Decode the video stream to obtain an image frame sequence.

[0148] In one embodiment, a computer device captures video streams in a classroom teaching environment in real time through a built-in or external image acquisition device. When the scene is offline classroom teaching, at least one image acquisition device can be used to capture images of the entire classroom teaching environment to obtain a video stream; when the scene is online classroom teaching, the image acquisition device of the terminal used by the student or teacher can be used to capture images of the individual classroom teaching environment of the corresponding user to obtain each video stream.

[0149] In one embodiment, when a computer device receives a video stream, it directly sends the received video stream to a decoder, which decodes the video stream to obtain image frames corresponding to each frame of the video stream, and composes an image frame sequence based on the image frames corresponding to each frame of the video stream.

[0150] In one embodiment, after obtaining the image frames corresponding to each video stream frame, the computer device directly extracts the image frames corresponding to the key frame video stream from the image frames corresponding to each video stream frame, and assembles an image frame sequence based on the image frames corresponding to each key frame video stream. The key frame video stream may be a video frame whose picture data is completely retained during encoding, that is, a video frame containing a complete picture.

[0151] In one embodiment, after receiving the video stream, the computer device extracts the key frame video stream from the received video stream, decodes the key frame video stream, obtains the image frames corresponding to each key frame video stream, and composes an image frame sequence based on the image frames corresponding to each key frame video stream.

[0152] S604: Extract image frames from the image frame sequence to obtain target image frames.

[0153] Among them, the target image frame contains at least one target object. For example, for the target image frame corresponding to the offline classroom teaching scene, each target image frame can contain multiple target objects. For the target image frame corresponding to the online classroom teaching scene, each target image frame can contain one target object.

[0154] Specifically, after decoding the video stream to obtain an image frame sequence, the computer device performs face detection on each image frame in the image frame sequence, and extracts the image frames containing faces from the image frames to obtain target image frames.

[0155] S606: Perform image processing on the target image frame to obtain a processed target image frame.

[0156] The image processing includes at least one of a cropping process and an image correction process, and the image correction process may be a geometric correction process or a grayscale correction process.

[0157] In one embodiment, if the target image frame contains multiple target objects, the computer device performs face detection on the target image frame, and crops the target image frame according to the face detection result to obtain image frames corresponding to each target object, and determines the image frames corresponding to each target object as the processed target image frames.

[0158] In one embodiment, after obtaining the target image frame, the computer device can perform gamma correction processing on at least a portion of the target image frame. By performing gamma correction processing on at least a portion of the target image frame, the grayscale of the image of the target image frame can be adjusted to reduce the impact of uneven light on the recognition effect.

[0159] In the above embodiment, the computer device decodes the video stream to obtain an image frame sequence, extracts image frames from the image frame sequence to obtain target image frames, and performs image processing on the target image frames, thereby obtaining processed target image frames suitable for target recognition, thereby improving the recognition accuracy and further improving the accuracy of concentration assessment.

[0160] In one embodiment, after obtaining a target image frame, a computer device performs gamma correction processing on the obtained target image frame, and the processing process includes the following steps: performing inversion processing on the target image frame to obtain an inverted image frame; obtaining the pixel value of each pixel point in the inverted image frame; determining the transformation parameters corresponding to each pixel point in the inverted image frame based on the pixel value; performing nonlinear transformation on the target image frame according to the pixel value of each pixel point in the inverted image frame and the corresponding transformation parameters to obtain a processed target image frame.

[0161] Inverting the target image frame, which essentially flips the R, G, and B values, is done by inverting the colors. If the color quantization level is 256, the R, G, and B values of the new image are 255 minus the R, G, and B values of the original image. This applies to all images, including true color, paletted color (also known as pseudo-color), and grayscale images.

[0162] Specifically, after obtaining the pixel value of each pixel in the inverted image frame, the computer device may substitute the pixel value of each pixel in the inverted image frame into the following formula to obtain the transformation parameter corresponding to each pixel:

[0163]

[0164] Among them, r[i,j,N(i,j)] is the transformation parameter corresponding to the Nth pixel point corresponding to the coordinates (i,j), and mask(i,j) is the pixel value of the pixel point corresponding to the Nth pixel point corresponding to the coordinates (i,j) in the inverted image frame.

[0165] After obtaining the transformation parameters corresponding to each pixel point, the computer device can substitute the transformation parameters and the pixel value of each pixel point in the target image frame into the following formula to obtain the pixel value of each pixel point after transformation:

[0166]

[0167] Among them, O(i,j) is the transformed pixel value corresponding to the Nth pixel point corresponding to the coordinates (i, j), r[i,j,N(i,j)] is the transformation parameter corresponding to the Nth pixel point corresponding to the coordinates (i, j), and I(i,j) is the pixel value corresponding to the Nth pixel point corresponding to the coordinates (i, j) in the target image frame.

[0168] In the above embodiment, the computer device obtains an inverted image frame by performing inversion processing on the target image frame, obtains the pixel value of each pixel in the inverted image frame, and determines the transformation parameters corresponding to each pixel in the inverted image frame based on the pixel value, thereby performing a nonlinear transformation on the target image frame according to the pixel value of each pixel in the inverted image frame and the corresponding transformation parameters, thereby obtaining a target image frame after gamma correction processing, reducing the impact of uneven light in the target image frame on the recognition effect, thereby improving the recognition accuracy, and further improving the accuracy of concentration assessment.

[0169] In one embodiment, S202 specifically includes the following steps: performing normalization and denoising on the processed target image frame in sequence to obtain a normalized denoised image frame; inputting the normalized denoised image frame into a concentration evaluation model; and extracting global image features in the normalized denoised image frame through a feature extraction network in the concentration evaluation model.

[0170] Normalization involves converting the original image to a unique standard form through a series of transformations. This involves using the image's invariant moments to find a set of parameters that eliminate the effects of other transformation functions on the image. The resulting standard form is invariant to affine transformations such as translation, rotation, and scaling. In neural networks, image normalization can also accelerate the convergence of trained networks.

[0171] De-noising the target image frame involves removing noise from it. Noise can be understood as "factors that hinder our sensory organs from understanding the source information they receive." For example, if the brightness distribution of a black-and-white image is f(x, y), then the brightness distribution R(x, y) that interferes with its reception is called image noise. Specifically, image denoising algorithms such as spatial-domain median filtering, wavelet-domain wavelet threshold denoising, PDE-based image denoising, or total variation (TV) image denoising can be used to denoise the target image frame.

[0172] Specifically, after obtaining the normalized denoised image frame, the computer device can directly input the normalized denoised image frame into the concentration evaluation model, and extract the global image features in the normalized denoised image frame through the feature extraction network in the concentration evaluation model.

[0173] Among them, the concentration assessment model is a pre-trained machine learning model for evaluating concentration, including a feature extraction network and a prediction network. The feature extraction network is used to extract global image features from the input image frame, and the prediction network is used to identify target objects, items and interactive behaviors in the image frame based on the input image features.

[0174] In one embodiment, the feature extraction network is constructed by a Resnet Blockc residual unit, which may specifically include multiple residual units. Image features with rich semantic information at different levels are extracted from the image frame through the multiple residual units, thereby obtaining the global image features of the normalized denoised image frame.

[0175] In the above embodiment, the computer device obtains a normalized denoised image frame by sequentially normalizing and denoising the processed target image frame, and inputs the normalized denoised image frame into the concentration evaluation model. The global image features in the normalized denoised image frame are extracted through the feature extraction network in the concentration evaluation model. On the one hand, the impact of noise in the target image frame on feature extraction is reduced, and on the other hand, the efficiency of the model in extracting features from the image frame is improved, thereby improving the timeliness of recognition, and further improving the accuracy and efficiency of concentration evaluation.

[0176] In one embodiment, S204 specifically includes the following steps: performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame; performing block processing on the normalized denoised image frame to obtain an image sequence block; inputting the image sequence block into a concentration evaluation model; extracting the context features of each image block in the image sequence block through the multi-attention encoding network in the concentration evaluation model; performing feature fusion on the context features of each image block through the multi-layer perceptron in the concentration evaluation model to obtain a fused image context feature.

[0177] Normalization involves converting the original image to a unique standard form through a series of transformations. This involves using the image's invariant moments to find a set of parameters that eliminate the effects of other transformation functions on the image. The resulting standard form is invariant to affine transformations such as translation, rotation, and scaling. In neural networks, image normalization can also accelerate the convergence of trained networks.

[0178] De-noising the target image frame involves removing noise from it. Noise can be understood as "factors that hinder our sensory organs from understanding the source information they receive." For example, if the brightness distribution of a black-and-white image is f(x, y), then the brightness distribution R(x, y) that interferes with its reception is called image noise. Specifically, image denoising algorithms such as spatial-domain median filtering, wavelet-domain wavelet threshold denoising, PDE-based image denoising, or total variation (TV) image denoising can be used to denoise the target image frame.

[0179] Specifically, after obtaining the normalized denoised image frame, the computer device can divide the normalized denoised image frame into blocks to obtain image sequence blocks, and input the image sequence blocks into the concentration evaluation model, and extract the image context features corresponding to the image sequence blocks through the multi-attention encoding feature extraction network in the concentration evaluation model.

[0180] The multi-attention encoding feature extraction network consists of a multi-attention encoding network and a multi-layer perceptron (MLP). The multi-attention encoding network is constructed using a Transformer Encoder module, which consists of layers of multi-head self-attention (MSA) and MLP (multi-layer perceptron) blocks. By using a multi-head attention mechanism instead of the original single-head attention mechanism, the Transformer Encoder module can better fuse features and thus extract richer image context features.

[0181] In the above embodiment, the computer device sequentially normalizes and denoises the processed target image frame to obtain a normalized denoised image frame, blocks the normalized denoised image frame to obtain an image sequence block, inputs the image sequence block into a concentration evaluation model, and extracts the fused image context features through the multi-attention encoding feature extraction network in the concentration evaluation model. On the one hand, it reduces the impact of noise in the target image frame on feature extraction, and on the other hand, it improves the efficiency of the model in extracting features from the image frame, thereby improving the timeliness of recognition, and further improving the accuracy and efficiency of concentration evaluation.

[0182] In one embodiment, the process of a computer device determining the concentration of a target object based on the expression score, posture score, item score and interaction score includes the following steps: obtaining the weights corresponding to the expression dimension, posture dimension, item dimension and interaction dimension respectively; performing a weighted summation of the expression score, posture score, item score and interaction score based on the weights of different dimensions; and determining the concentration of the target object based on the weighted score obtained by the weighted summation.

[0183] Among them, the weights corresponding to the expression dimension, posture dimension, object dimension and interaction dimension can be the same or different, and the weight of each dimension represents the degree of influence of the dimension on concentration.

[0184] Specifically, after obtaining the weights and scores corresponding to each dimension, the computer device can input the obtained weights and scores into the following formula to calculate the concentration of the target object:

[0185] P=αA+βB+γC+εD (5)

[0186] Among them, P is the concentration of the target object, A is the expression score, B is the posture score, C is the item score, and D is the interaction score. α is the weight corresponding to the expression dimension, β is the weight corresponding to the posture dimension, γ is the weight corresponding to the item dimension, and ε is the weight corresponding to the interaction dimension, and α+β+γ+ε=1.

[0187] In the above embodiment, the computer device obtains the weights corresponding to the expression dimension, posture dimension, object dimension and interaction dimension respectively, and performs weighted summation based on the weights and scores of different dimensions to obtain the concentration of the target object. The evaluation of concentration comprehensively considers the influence of each dimension on concentration, thereby further improving the accuracy of the concentration evaluation.

[0188] In one embodiment, the target object is a character object in the target group, and the above-mentioned concentration evaluation method also includes the following steps: obtaining the concentration of each character object at a historical moment; determining the average concentration of the target group at different moments based on the currently determined concentration of each character object and the concentration at the historical moment; and generating a concentration analysis chart of the target group based on the average concentration at each moment.

[0189] For example, if the target group is a class that contains 50 students, then the 50 students are the person objects in the target group. The concentration corresponding to each student at a historical moment and the concentration corresponding to each student at the current moment can be obtained. The average concentration of the 50 students at each moment can be calculated to obtain the average concentration corresponding to each moment. Then, a concentration analysis chart of the class can be generated with time as the horizontal axis and the average concentration as the vertical axis. Figure 7 A schematic diagram of a focus analysis chart for a target group in one embodiment.

[0190] In the above embodiment, the computer device obtains the concentration of each character object at a historical moment, and determines the average concentration of the target group at different moments based on the currently determined concentration of each character object and the concentration at the historical moment, so that a concentration analysis chart of the target group can be generated according to the average concentration at each moment. The generated group concentration analysis chart can be used to perform an overall evaluation of the target group's concentration, so as to adjust subsequent teaching strategies based on the overall concentration evaluation results.

[0191] In one embodiment, the target object is a person object in the target group, and the above-mentioned concentration evaluation method also includes the following steps: obtaining the concentration of each person object at the current moment, determining the current average concentration of the target group at the current moment, and if the current average concentration is less than the concentration threshold, generating a prompt message and sending the prompt message to the administrator terminal corresponding to the target group.

[0192] The administrator terminal corresponding to the target group may be the terminal of the teaching teacher or the terminal of the teaching quality supervision teacher.

[0193] For example, if the average concentration threshold is 0.5 and the average concentration of students at the current moment is 0.4, a classroom warning prompt message will be generated and sent to the teacher's terminal so that the teacher can adjust the teaching content or teaching strategy in time according to the classroom warning information.

[0194] In the above embodiment, the computer device determines the current average concentration of the target group at the current moment by obtaining the concentration of each character object at the current moment. If the current average concentration is less than the concentration threshold, a prompt message is generated and sent to the administrator terminal corresponding to the target group, so that the administrator can adjust the teaching content or teaching strategy in real time based on the received prompt information, thereby improving the efficiency of teaching feedback.

[0195] In one embodiment, the target object is a character object in the target group, and the above-mentioned concentration evaluation method also includes the following steps: obtaining the concentration of each character object at a historical moment; generating a concentration analysis chart for each character object based on the currently determined concentration of each character object and the concentration at the historical moment.

[0196] For example, if the target group is a class that contains 50 students, then the 50 students are the person objects in the target group. The concentration corresponding to each student at a historical moment and the concentration corresponding to each student at the current moment can be obtained. For any student, a concentration analysis chart of the student can be generated with time as the horizontal axis and the concentration of the student at each moment as the vertical axis. Figure 8A schematic diagram of a concentration analysis chart for an individual student in one embodiment is shown. Furthermore, the concentration levels of the instructor at historical moments and the current moment can be obtained, and a concentration analysis chart corresponding to the instructor can be generated with time as the horizontal axis and the instructor's concentration levels at each moment as the vertical axis.

[0197] In the above embodiment, the computer device obtains the concentration of each character object at a historical moment, and generates a concentration analysis graph for each character object based on the currently determined concentration of each character object and the concentration at the historical moment. The individual concentration analysis graph generated can be used to evaluate the individual concentration individually, so as to provide learning guidance to the individual based on the individual concentration evaluation result.

[0198] In one embodiment, the target object is a character object in the target group, and the above-mentioned concentration assessment method also includes the following steps: based on the concentration analysis diagram of each character object, determining the effective learning time for each character object's concentration to reach the concentration threshold; when the effective learning time reaches the effective time threshold, generating a first prompt message, and sending the first prompt message to the terminal of the corresponding character object; when the effective time does not reach the effective time threshold, generating a second prompt message, and sending the second prompt message to the terminal of the corresponding character object.

[0199] For example, a class lasts 45 minutes, the concentration threshold is 0.3, and the effective time threshold is 36 minutes. For any student, when the student's concentration threshold reaches 0.3 and the cumulative time reaches 36 minutes, a praise message for good class performance is generated and sent to the student's terminal; when the student's concentration threshold reaches 0.3 but the cumulative time does not reach 36 minutes, a study suggestion message requiring after-class review is generated and sent to the student's terminal.

[0200] In the above embodiment, the computer device determines the effective learning time for each character object's concentration to reach the concentration threshold based on the concentration analysis diagram of each character object, so that the individual's concentration can be evaluated separately, so as to provide learning guidance to the individual based on the individual's concentration evaluation results.

[0201] In one embodiment, S206 includes the following steps: performing feature fusion on global image features and image context features to obtain fused features; extracting target object features and item features from the fused features; identifying the expression and posture of the target object in the target image frame based on the target object features; and identifying the item in the target image frame based on the item features.

[0202] The feature fusion of the global image features and the image context features can be performed by concatenating the obtained global image features and the image context features, and determining the resulting features as the fused features. For example, if the size of the obtained global image features is 1024*1 and the size of the obtained image context features is 1024*1, the global image features and the image context features can be directly concatenated to obtain a fused feature size of 2048*1.

[0203] Specifically, after the computer device obtains the fusion features, it inputs the obtained fusion into the feature classification network, extracts the target object features and item features from the fusion features through the feature classification network, and inputs the extracted target object features into the expression recognition subnetwork and the posture recognition subnetwork respectively, and inputs the item features into the item recognition subnetwork, and uses the expression recognition subnetwork to recognize the expression of the target object in the target image frame based on the target object features, uses the posture recognition subnetwork to recognize the posture of the target object in the target image frame, and uses the item recognition subnetwork to recognize the item in the target image frame.

[0204] In the above embodiment, the computer device obtains fused features by performing feature fusion on global image features and image context features, and extracts target object features and item features from the fused features, so as to identify the expression and posture of the target object in the target image frame based on the target object features, and identify the item in the target image frame based on the item features, thereby improving the accuracy of the recognition results.

[0205] In one embodiment, Figure 9 As shown in the figure, a method for evaluating concentration is also provided. Figure 1 The computer device (including terminal 102 and server 104) in FIG. 1 is used as an example to illustrate the method, which includes the following steps:

[0206] S902: Decode the video stream to obtain an image frame sequence.

[0207] S904: Extract image frames from the image frame sequence to obtain target image frames.

[0208] S906: Perform inversion processing on the target image frame to obtain an inverted image frame.

[0209] S908: Obtain the pixel value of each pixel in the inverted image frame.

[0210] S910 , determining a transformation parameter corresponding to each pixel point in the inverted image frame based on the pixel value.

[0211] S912 , performing nonlinear transformation on the target image frame according to the pixel value of each pixel point in the inverted image frame and the corresponding transformation parameters to obtain a processed target image frame.

[0212] S914 , performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame.

[0213] S916: Input the normalized denoised image frame into the concentration evaluation model.

[0214] S918, extracting global image features from the normalized denoised image frame through the feature extraction network in the focus evaluation model.

[0215] S920 , performing block processing on the normalized denoised image frame to obtain image sequence blocks.

[0216] S922: Input the image sequence blocks into the concentration evaluation model.

[0217] S924, extracting contextual features of each image block in the image sequence block through the multi-attention encoding network in the concentration evaluation model.

[0218] S926 , performing feature fusion on the context features of each image block through the multi-layer perceptron in the focus evaluation model to obtain fused image context features.

[0219] S928 , based on the fusion features between the global image features and the image context features, recognize the expression and posture of the target object in the target image frame, as well as the object in the target image frame.

[0220] S930: Determine the interaction behavior between the target object and the item based on the fusion feature.

[0221] S932, determining an expression score, a posture score, an item score, and an interaction score based on the expression, posture, and item type and the interaction behavior.

[0222] S934, obtaining weights corresponding to the expression dimension, posture dimension, object dimension, and interaction dimension respectively.

[0223] S936: Based on the weights of different dimensions, perform weighted summation of the expression score, posture score, item score, and interaction score.

[0224] S938: Determine the concentration of the target object according to the weighted score obtained by weighted summation.

[0225] This application also provides an application scenario, which applies the above-mentioned concentration evaluation method. Specifically, the application of the concentration evaluation method in this application scenario is as follows:

[0226] refer to Figure 10The flow chart of the concentration evaluation method shown in the figure shows that after the computer device decodes the video stream to obtain the target image frame, it preprocesses the target image frame to obtain the preprocessed target image, wherein the image preprocessing can be at least one of normalization, denoising and gamma correction processing, and then on the one hand, the preprocessed target image frame is divided into blocks to obtain image sequence blocks S, and the obtained image sequence blocks S are input into the Transformer Encoder module, and the context features T of each image block are extracted through the Transformer Encoder module, and then the extracted context features T of each image block are input into the MLP module, and the context features of each image block are fused through the MLP module to obtain fused image context features F1; on the other hand, the preprocessed target image frame is input into the feature extraction network composed of the residual units of the Resnet Blockc, and the image sequence blocks S are input into the Transformer Encoder module. The feature extraction network composed of Blockc residual units extracts the global image feature F2 from the preprocessed target image frame, and then fuses the image context feature F1 and the global image feature F2 to obtain the fused feature F, and inputs the fused feature into the prediction network. The prediction network identifies the target object's expression, head posture, identity posture, object and interaction behavior, and determines the target object's expression score, head posture score, body posture score, object score and interaction behavior score based on the target object's expression, head posture, identity posture, object and interaction behavior. Then, the expression score, head posture score, body posture score, object score and interaction behavior score are weightedly summed to obtain the target object's concentration.

[0227] It should be understood that although Figure 2 、 6 The steps in the flowcharts of , 9 and 10 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 、 6 At least part of the steps in 9 and 10 may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0228] In one embodiment, Figure 11As shown, a concentration evaluation device is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: a global feature extraction module 1102, a context feature extraction module 1104, a recognition module 1106, an interaction determination module 1108, and a concentration determination module 1110, wherein:

[0229] The global feature extraction module 1102 is used to extract global image features from the target image frame.

[0230] The context feature extraction module 1104 is configured to extract image context features from the target image frame.

[0231] The recognition module 1106 is configured to recognize the expression and posture of the target object in the target image frame, as well as the object in the target image frame, based on the fusion features between the global image features and the image context features.

[0232] The interaction determination module 1108 is used to determine the interaction behavior between the target object and the item based on the fusion features.

[0233] The concentration determination module 1110 is used to determine the concentration of the target object based on the expression, posture, type of object and interactive behavior.

[0234] In the above embodiment, the concentration evaluation device extracts global image features and image context features from the target image frame, and identifies the expression and posture of the target object in the target image frame, as well as the objects in the target image frame based on the fusion features between the global image features and the image context features, and determines the interaction behavior between the target object and the objects according to the fusion features. On the one hand, the accuracy of the recognition result is improved by fusing the features. On the other hand, the concentration of the target object is comprehensively determined based on multiple dimensions such as expression, posture, type of object and interaction behavior, thereby improving the accuracy of the concentration evaluation.

[0235] In one embodiment, Figure 12 As shown, the device also includes: a video decoding module 1112, an image frame extraction module 1114 and an image processing module 1116, wherein: the video decoding module 1112 is used to decode the video stream to obtain an image frame sequence; the image frame extraction module 1114 is used to extract image frames from the image frame sequence to obtain target image frames; the image processing module 1116 is used to perform image processing on the target image frames to obtain processed target image frames.

[0236] In the above embodiment, the video stream is decoded to obtain an image frame sequence, image frames are extracted from the image frame sequence to obtain target image frames, and image processing is performed on the target image frames, so that processed target image frames suitable for target recognition can be obtained, thereby improving the recognition accuracy and further improving the accuracy of concentration assessment.

[0237] In one embodiment, the image processing module 1116 is further used to: perform inversion processing on the target image frame to obtain an inverted image frame; obtain the pixel value of each pixel in the inverted image frame; determine the transformation parameters corresponding to each pixel in the inverted image frame based on the pixel value; and perform nonlinear transformation on the target image frame according to the pixel value of each pixel in the inverted image frame and the corresponding transformation parameters to obtain a processed target image frame.

[0238] In the above embodiment, the target image frame is inverted to obtain an inverted image frame, and the pixel value of each pixel in the inverted image frame is obtained. The transformation parameters corresponding to each pixel in the inverted image frame are determined based on the pixel value, and then the target image frame is nonlinearly transformed according to the pixel value of each pixel in the inverted image frame and the corresponding transformation parameters, so that the target image frame after gamma correction processing can be obtained, reducing the influence of uneven light in the target image frame on the recognition effect, thereby improving the recognition accuracy, and further improving the accuracy of concentration assessment.

[0239] In one embodiment, the global feature extraction module 1102 is further used to: perform normalization and denoising on the processed target image frame in sequence to obtain a normalized denoised image frame; input the normalized denoised image frame into the concentration evaluation model; and extract global image features from the normalized denoised image frame through the feature extraction network in the concentration evaluation model.

[0240] In the above embodiment, the processed target image frame is normalized and denoised in sequence to obtain a normalized denoised image frame, which is input into the concentration evaluation model. The global image features in the normalized denoised image frame are extracted through the feature extraction network in the concentration evaluation model. On the one hand, the impact of noise in the target image frame on feature extraction is reduced, and on the other hand, the efficiency of the model in extracting features from the image frame is improved, thereby improving the timeliness of recognition, and further improving the accuracy and efficiency of concentration evaluation.

[0241] In one embodiment, the context feature extraction module 1104 is further used to: perform normalization and denoising on the processed target image frames in sequence to obtain normalized denoised image frames; perform block processing on the normalized denoised image frames to obtain image sequence blocks; input the image sequence blocks into the concentration evaluation model; extract the context features of each image block in the image sequence blocks through the multi-attention encoding network in the concentration evaluation model; perform feature fusion on the context features of each image block through the multi-layer perceptron in the concentration evaluation model to obtain fused image context features.

[0242] In the above embodiment, the processed target image frame is normalized and denoised in sequence to obtain a normalized denoised image frame, the normalized denoised image frame is divided into blocks to obtain image sequence blocks, and the image sequence blocks are input into the concentration evaluation model. The fused image context features are extracted through the multi-attention encoding feature extraction network in the concentration evaluation model. On the one hand, the impact of noise in the target image frame on feature extraction is reduced, and on the other hand, the efficiency of the model in extracting features from the image frame is improved, thereby improving the timeliness of recognition, and then improving the accuracy and efficiency of concentration evaluation.

[0243] In one embodiment, the concentration determination module 1110 is also used to: determine the expression score of the target object based on the expression; determine the posture score of the target object based on the posture; determine the item score of the target object based on the type of the item; determine the interaction score of the target object based on the interaction behavior; and determine the concentration of the target object based on the expression score, posture score, item score and interaction score.

[0244] In one embodiment, the concentration determination module 1110 is further used to: obtain the weights corresponding to the expression dimension, posture dimension, object dimension and interaction dimension respectively; based on the weights of different dimensions, perform weighted summation of the expression score, posture score, object score and interaction score; and determine the concentration of the target object based on the weighted score obtained by the weighted summation.

[0245] In the above embodiment, the concentration of the target object is obtained by obtaining the weights corresponding to the expression dimension, posture dimension, object dimension and interaction dimension respectively, and performing weighted summation based on the weights and scores of different dimensions. The evaluation of concentration comprehensively considers the degree of influence of each dimension on concentration, thereby further improving the accuracy of the concentration evaluation.

[0246] In one embodiment, the target object is a person object in the target group; Figure 12As shown, the device also includes a chart generation module 1118, which is used to: obtain the concentration of each character object at a historical moment; determine the average concentration of the target group at different moments based on the currently determined concentration of each character object and the concentration at the historical moment; and generate a concentration analysis chart of the target group based on the average concentration at each moment.

[0247] In the above embodiment, by obtaining the concentration of each character object at a historical moment, and based on the currently determined concentration of each character object and the concentration at the historical moment, the average concentration of the target group at different moments is determined, so that a concentration analysis chart of the target group can be generated according to the average concentration at each moment. The generated group concentration analysis chart can be used to perform an overall evaluation of the target group's concentration, so as to adjust subsequent teaching strategies based on the overall concentration evaluation results.

[0248] In one embodiment, the chart generating module 1118 is further configured to: obtain the concentration of each character object at a historical moment; and generate a concentration analysis chart for each character object based on the currently determined concentration of each character object and the concentration at the historical moment.

[0249] In the above embodiment, by obtaining the concentration of each character object at a historical moment, a concentration analysis graph of each character object is generated based on the currently determined concentration of each character object and the concentration at the historical moment. The individual concentration analysis graph generated can be used to evaluate the individual concentration separately, so as to provide learning guidance to the individual based on the individual concentration evaluation result.

[0250] In one embodiment, the recognition module 1106 is further used to: perform feature fusion on global image features and image context features to obtain fused features; extract target object features and item features from the fused features; recognize the expression and posture of the target object in the target image frame based on the target object features; and recognize the item in the target image frame based on the item features.

[0251] In the above embodiment, by determining the effective learning time for each character object's concentration to reach the concentration threshold based on the concentration analysis diagram of each character object, the individual's concentration can be evaluated separately, so that the individual can be given learning guidance based on the individual's concentration evaluation results.

[0252] The specific definition of the concentration assessment device can be found in the definition of the concentration assessment method above and will not be repeated here. The various modules in the above-mentioned concentration assessment device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0253] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 13 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a concentration assessment method is implemented.

[0254] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 14 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a concentration assessment method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0255] Those skilled in the art will understand that Figure 13 The structure shown in or 14 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0256] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0257] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0258] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0259] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0260] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0261] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A concentration assessment method, characterized in that: The method comprises: extracting global image features from the target image frame; The target image frame is divided into blocks to obtain image sequence blocks; the image sequence blocks are input into a concentration evaluation model; context features of each image block in the image sequence blocks are extracted through a multi-attention encoding network in the concentration evaluation model; and the context features of each image block are fused through a multi-layer perceptron in the concentration evaluation model to obtain fused image context features. Recognizing the expression and posture of the target object in the target image frame, and the object in the target image frame based on the fusion features between the global image features and the image context features; determining an interaction behavior between the target object and the item according to the fusion feature; The concentration of the target object is determined based on the expression, the posture, the type of the object, and the interactive behavior.

2. The method according to claim 1, characterized in that Before extracting the global image features from the target image frame, the method further includes: Decode the video stream to obtain an image frame sequence; Extracting image frames from the image frame sequence to obtain target image frames; Performing image processing on the target image frame to obtain a processed target image frame.

3. The method according to claim 2, characterized in that The performing image processing on the target image frame to obtain a processed target image frame includes: Performing inversion processing on the target image frame to obtain an inverted image frame; Obtaining a pixel value of each pixel in the inverted image frame; determining a transformation parameter corresponding to each pixel point in the inverted image frame based on the pixel value; According to the pixel value of each pixel point in the inverted image frame and the corresponding transformation parameters, the target image frame is subjected to nonlinear transformation to obtain a processed target image frame.

4. The method according to claim 3, characterized in that The extracting global image features from the target image frame includes: performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame; Inputting the normalized denoised image frame into a concentration assessment model; The global image features in the normalized denoised image frame are extracted through the feature extraction network in the focus evaluation model.

5. The method according to claim 3, characterized in that The block processing of the target image frame to obtain image sequence blocks includes: performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame; The normalized denoised image frame is divided into blocks to obtain image sequence blocks.

6. The method according to claim 1, characterized in that The determining the concentration of the target object based on the expression, the posture, the type of the object, and the interactive behavior includes: determining an expression score of the target object based on the expression; determining a posture score of the target object based on the posture; determining an item score for the target object based on the type of the item; determining an interaction score of the target object based on the interaction behavior; The concentration of the target object is determined according to the expression score, the posture score, the object score, and the interaction score.

7. The method according to claim 6, characterized in that The determining the concentration of the target object according to the expression score, the posture score, the object score, and the interaction score includes: Get the weights corresponding to the expression dimension, posture dimension, object dimension, and interaction dimension respectively; Based on the weights of the different dimensions, performing a weighted summation on the expression score, the posture score, the item score, and the interaction score; The concentration of the target object is determined according to a weighted score obtained by weighted summation.

8. The method according to any one of claims 1 to 7, characterized in that The target object is a person object in the target group; the method further includes: Obtaining the concentration of each character object at a historical moment; Determining the average concentration of the target group at different moments based on the currently determined concentration of each character object and the concentration of the historical moments; A concentration analysis graph of the target group is generated according to the average concentration at each moment.

9. The method according to any one of claims 1 to 7, characterized in that The target object is a person object in the target group; the method further includes: Obtaining the concentration of each character object at a historical moment; Based on the currently determined concentration of each character object and the concentration at the historical moment, a concentration analysis graph of each character object is generated.

10. The method according to any one of claims 1 to 7, characterized in that The identifying, based on the fusion features between the global image features and the image context features, the expression and posture of the target object in the target image frame, and the object in the target image frame comprises: Performing feature fusion on the global image feature and the image context feature to obtain a fused feature; extracting target object features and item features from the fused features; recognizing the expression and posture of the target object in the target image frame based on the target object features; An object in the target image frame is identified based on the object features.

11. A concentration assessment device, characterized in that: The device comprises: A global feature extraction module, used to extract global image features from the target image frame; a context feature extraction module configured to divide the target image frame into blocks to obtain image sequence blocks; input the image sequence blocks into a focus assessment model; extract context features of each image block in the image sequence blocks through a multi-attention encoding network in the focus assessment model; and perform feature fusion on the context features of each image block through a multi-layer perceptron in the focus assessment model to obtain fused image context features; a recognition module, configured to recognize an expression and a posture of a target object in the target image frame, and an object in the target image frame based on a fusion feature of the global image feature and the image context feature; an interaction determination module, configured to determine an interaction behavior between the target object and the item based on the fusion feature; A concentration determination module is used to determine the concentration of the target object based on the expression, the posture, the type of the object and the interactive behavior.

12. The device according to claim 11, characterized in that The device further comprises: A video decoding module is used to decode the video stream to obtain an image frame sequence; An image frame extraction module, configured to extract image frames from the image frame sequence to obtain a target image frame; The image processing module is used to perform image processing on the target image frame to obtain a processed target image frame.

13. The device according to claim 12, characterized in that The image processing module is further used for: Performing inversion processing on the target image frame to obtain an inverted image frame; Obtaining a pixel value of each pixel in the inverted image frame; determining a transformation parameter corresponding to each pixel point in the inverted image frame based on the pixel value; According to the pixel value of each pixel point in the inverted image frame and the corresponding transformation parameters, the target image frame is subjected to nonlinear transformation to obtain a processed target image frame.

14. The device according to claim 13, characterized in that The global feature extraction module is further used for: performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame; Inputting the normalized denoised image frame into a concentration assessment model; The global image features in the normalized denoised image frame are extracted through the feature extraction network in the focus evaluation model.

15. The device according to claim 13, characterized in that The context feature extraction module is further used for: performing normalization and denoising processing on the processed target image frame in sequence to obtain a normalized denoised image frame; The normalized denoised image frame is divided into blocks to obtain image sequence blocks.

16. The device according to claim 11, characterized in that The concentration determination module is further configured to: determining an expression score of the target object based on the expression; determining a posture score of the target object based on the posture; determining an item score for the target object based on the type of the item; determining an interaction score of the target object based on the interaction behavior; The concentration of the target object is determined according to the expression score, the posture score, the object score, and the interaction score.

17. The device according to claim 16, characterized in that The concentration determination module is further configured to: Get the weights corresponding to the expression dimension, posture dimension, object dimension, and interaction dimension respectively; Based on the weights of the different dimensions, performing a weighted summation on the expression score, the posture score, the item score, and the interaction score; The concentration of the target object is determined according to a weighted score obtained by weighted summation.

18. The device according to any one of claims 11 to 17, characterized in that The target object is a person object in the target group; the device also includes a chart generating module for: Obtaining the concentration of each character object at a historical moment; Determining the average concentration of the target group at different moments based on the currently determined concentration of each character object and the concentration of the historical moments; A concentration analysis graph of the target group is generated according to the average concentration at each moment.

19. The device according to any one of claims 11 to 17, characterized in that The target object is a person object in the target group; the device also includes a chart generating module for: Obtaining the concentration of each character object at a historical moment; Based on the currently determined concentration of each character object and the concentration at the historical moment, a concentration analysis graph of each character object is generated.

20. The device according to any one of claims 11 to 17, characterized in that The identification module is further configured to: Performing feature fusion on the global image feature and the image context feature to obtain a fused feature; extracting target object features and item features from the fused features; recognizing the expression and posture of the target object in the target image frame based on the target object features; An object in the target image frame is identified based on the object features.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a controller, the steps of the method according to any one of claims 1 to 10 are implemented.

Images