A method, system, device and medium for generating style profiles through behavioral data learning

Abstract

The present invention discloses a method, system, device and medium for generating a learning style portrait based on behavioral data, including: acquiring sample data, constructing a convolutional neural network model, processing first behavioral data, and obtaining cognitive emotional state recognition data; constructing a Transformer model that fuses spatiotemporal features, processing second behavioral data, and obtaining cognitive concentration recognition data; constructing a semantic recognition model, processing third behavioral data, and obtaining interactive activity recognition data; quantifying cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate a behavioral data sequence, constructing a learning style coding framework, generating a learning style portrait based on the behavioral data sequence, and generating a learning style by tracking the informal learning behavior data of deaf-mute people in a venue visiting situation, thereby grasping the learning status of the deaf-mute learners and improving the informal learning interaction experience of the deaf-mute people.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for generating a behavioral data learning style profile. Background Art

[0002] At present, there are two mainstream technical solutions for generating learning style profiles for deaf-mute people in informal learning venues:

[0003] The first approach draws on traditional qualitative educational observation. Drawing on educational theory and industry experience, observers provide qualitative descriptions of deaf learners' learning behaviors and offer suggestions for improvement. However, this approach suffers from a lack of personalized instruction. Qualitative observation, from the perspective of deaf learners, emphasizes a holistic description of learning activities and cannot accurately capture the learning status of each individual learner. Consequently, this approach is highly subjective and struggles to capture local variations between events.

[0004] The second type is quantitative observation based on empirical research. This approach first creates an observation coding framework, records learning behaviors using manual labeling, and uses statistical data to quantitatively analyze learning behaviors. These behavioral coding systems primarily target teaching behaviors and interactive information in learning activities, and offer valuable insights for analyzing informal learning videos of deaf individuals. However, these observational elements still need to be modified and adjusted based on the interactive characteristics of deaf individuals. Furthermore, because deaf individuals have limited hearing and speech, communication and interaction primarily rely on sign language and body language. From a machine understanding perspective, the lack of sound, a crucial information intake channel, presents a significant challenge for machines in understanding informal learning behaviors. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing technology cannot accurately grasp the learning status of deaf-mute learners. The purpose is to provide a method, system, equipment and medium for generating learning style portraits based on behavioral data, utilizing the informal learning behavior of deaf-mute people in the learning context of venues, and generating learning styles through learning cognitive emotional state recognition data, exhibition head posture, learning interactive behavior and other explicit information, thereby obtaining the exhibition interactive activity of deaf-mute learners, thereby grasping the learning status of deaf-mute learners, and thus improving the informal learning interactive experience of deaf-mute people.

[0006] The present invention is achieved through the following technical solutions:

[0007] A first aspect of the present invention provides a method for generating a style profile through behavioral data learning, comprising the following specific steps:

[0008] S1. Acquire sample data, where the sample data includes: first behavior data, second behavior data, and third behavior data;

[0009] S2. Build a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data; build a Transformer model that integrates spatiotemporal features to process the second behavior data to obtain cognitive concentration recognition data;

[0010] Build a semantic recognition model, process the third-party behavior data, and obtain interactive activity recognition data;

[0011] S3. Quantify cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate a behavioral data sequence;

[0012] S4. Construct a learning style coding framework and generate a learning style profile based on the behavioral data sequence.

[0013] The present invention tracks the informal learning behavior of deaf-mute people in the context of visiting venues, generates learning styles by learning cognitive emotional state recognition data, head posture when visiting exhibitions, learning interactive behavior and other explicit information, obtains the interactive activity of deaf-mute learners in exhibitions, and thus grasps the learning status of deaf-mute learners, thereby improving the informal learning interactive experience of deaf-mute learners.

[0014] Furthermore, the sample data is behavioral data of deaf-mute people in a learning situation at a venue. The first behavioral data includes facial expression data of the learner, the second behavioral data includes head posture data of the learner when viewing an exhibition, and the third behavioral data includes sign language movement data of the learner.

[0015] Furthermore, the constructing of a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data specifically includes:

[0016] Obtain the first behavior data, build a convolutional neural network model to extract features from the first behavior data, and obtain feature labels;

[0017] The feature labels are screened and the feature labels of different sizes are concatenated to obtain a feature vector of fixed length. The softmax function is used to determine the probability of the category to which the feature label belongs, and the feature label classification result is obtained.

[0018] The back-propagation algorithm is used to update the error between the feature label classification result and the actual expression category, and multiple rounds of iterative training of the convolutional neural network model parameters are performed;

[0019] The feature labels are input into the trained convolutional neural network model, and the feature labels obtained using convolution kernels of different sizes are fused. The classification result with the highest probability is output as the final expression recognition result, and the cognitive emotional state recognition data is determined based on the final expression recognition result.

[0020] Furthermore, the Transformer model for fusion of spatiotemporal features is constructed to process the second behavior data to obtain cognitive focus recognition data, specifically including:

[0021] The second behavior data is obtained, and a Transformer model with spatiotemporal feature fusion is constructed to extract the angular deflection features of the learner's head. The head posture is estimated based on the angular deflection features of the learner's head through a softmax regressor. The cognitive concentration recognition data is obtained based on the estimated horizontal and vertical deflection angles of the head posture.

[0022] Furthermore, estimating the head posture based on the angle deflection feature of the learner's head specifically includes:

[0023] Obtain the learner's head posture data while viewing the exhibition, uniformly slice the video frame by frame in the spatial dimension, convert the two-dimensional image into a one-dimensional linear sequence, and regress and predict the learner's head deflection angle through horizontal and vertical angles;

[0024] The sequence is flattened by the linear compression module to obtain the embedded vector of the image data as the input of the Transformer module;

[0025] The sequence contains image position information, and the spatial transformer treats each key point of the frame as a slice and obtains high-dimensional features through spatial position embedding;

[0026] By extracting high-dimensional features from space and encoding temporal information as a slice in the time dimension, the temporal correlation between multiple frames of images is obtained;

[0027] The head orientation description operator is obtained based on the temporal correlation between multiple frames of images, and the head orientation description operator is passed into the classification inference module to estimate the horizontal and vertical deflection angles of the head to obtain cognitive concentration recognition data.

[0028] Furthermore, the step of constructing a semantic recognition model and processing the third behavior data to obtain interactive activity recognition data specifically includes:

[0029] Obtaining the third behavior data, building a three-channel convolutional neural network model, and constructing feature maps from three channels: spatial, local time domain, and global time domain: the spatial channel CNN performs deep learning on the action image, the local time domain channel CNN performs deep learning on the optical flow field, and the global time domain channel CNN performs deep learning on the action difference image;

[0030] Concatenate feature maps of different sizes into feature vectors of fixed length and classify the concatenated features;

[0031] The three-channel video features are input into the feature extraction module to obtain the deep feature map which is input into the long short-term memory unit to extract the contextual time domain information of the body semantics;

[0032] The outputs of different feature channels are fused according to the contextual time domain information of body semantics to vote and obtain the recognition results of the learner's body semantics.

[0033] Furthermore, historical data of cognitive emotional state recognition data, visit cognitive concentration recognition data, and interactive activity are obtained, and weight factors of cognitive emotional state recognition data, visit cognitive concentration recognition data, and interactive activity are determined;

[0034] Construct a learning style coding framework, combine weight factors to perform weighted calculations on emotional engagement, cognitive concentration recognition data, and interactive activity recognition data, and generate a venue learning style portrait of deaf and mute learners.

[0035] A second aspect of the present invention provides a system for generating a style profile by learning from behavioral data, comprising:

[0036] Sample data collection module: used to collect first behavior data, second behavior data and third behavior data;

[0037] The first processing module is used to build a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data;

[0038] The second processing module is used to build a Transformer model for spatiotemporal feature fusion, process the second behavior data, and obtain cognitive focus recognition data;

[0039] The third processing module is used to build a semantic recognition model, process the third behavior data, and obtain interactive activity recognition data;

[0040] Data processing module: used to quantify cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate behavioral data sequences;

[0041] Learning style generation module: used to construct a learning style coding framework and generate a learning style portrait based on the behavioral data sequence.

[0042] A third aspect of the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for generating a style portrait through learning from behavioral data is implemented.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating a style profile through behavioral data learning.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] 1. This invention tracks the informal learning behavior of deaf-mute learners in the context of venue visits. By learning explicit information such as cognitive emotional state recognition data, head posture during exhibitions, and learning interaction behavior, it generates learning styles and obtains the interactive activity of deaf-mute learners in exhibitions, thereby understanding their learning status and improving their informal learning interaction experience.

[0046] 2. This invention collects facial expressions of deaf-mute learners during learning in a venue, designs a resizable convolutional neural network, performs fine-grained detection of facial features in deaf-mute videos, and recognizes their cognitive emotions. It also collects head postures of deaf-mute learners while viewing exhibits, designs a Transformer model that integrates spatiotemporal features, estimates the deaf-mute learners' head deflection angles, and calculates their cognitive focus state using a threshold judgment method. It also collects learners' interactive body movements in the venue context, designs a semantic recognition model to understand their body language, and derives the degree of interactive activity between the deaf-mute learners and the learners. It also quantifies the historical behaviors of deaf-mute learners and inputs them into a learning style coding framework. It generates learning style profiles based on the behavioral data sequences and displays them visually. This model can be applied to venue teaching for people with disabilities.

[0047] 3. Learning support services based on this learning style deduction method focus more on describing the behavior of deaf and mute students in venue learning scenarios, so that learning analysis can be based on objective data, thereby helping venue managers to make improvements in the exhibition room environment, exhibit placement and design based on learners' feedback, and promote the improvement of informal learning effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings:

[0049] Figure 1 This is a flow chart of a method for deducing learning styles in a venue context in an embodiment of the present invention;

[0050] Figure 2 A flow chart for generating learner cognitive emotional state recognition data in an embodiment of the present invention;

[0051] Figure 3A flow chart for generating learner cognitive concentration recognition data in an embodiment of the present invention;

[0052] Figure 4 A flow chart for generating learner interaction activity recognition data in an embodiment of the present invention;

[0053] Figure 5 A flow chart for generating a learner learning style profile in an embodiment of the present invention;

[0054] Figure 6 This is a diagram showing the effect of learning style deduction in an embodiment of the present invention;

[0055] Figure 7 This is a diagram showing a comparative implementation example of differentiated learning style portraits of different learners in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0057] Example 1

[0058] like Figure 1 As shown, the first aspect of this embodiment provides a method for generating a style profile by learning behavior data, including the following specific steps:

[0059] S1. Acquire sample data, where the sample data includes: first behavior data, second behavior data, and third behavior data;

[0060] S2. Build a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data; build a Transformer model that integrates spatiotemporal features to process the second behavior data to obtain cognitive concentration recognition data;

[0061] Build a semantic recognition model, process the third-party behavior data, and obtain interactive activity recognition data;

[0062] S3. Quantify cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate a behavioral data sequence;

[0063] S4. Construct a learning style coding framework and generate a learning style profile based on the behavioral data sequence.

[0064] By tracking the informal learning behavior of deaf-mute people in the context of visiting venues, generating learning styles through learning cognitive emotional state recognition data, head posture when visiting exhibitions, learning interactive behavior and other explicit information, the interactive activity of deaf-mute learners in exhibitions is obtained, thereby understanding the learning status of deaf-mute learners and improving their informal learning interactive experience.

[0065] In some possible embodiments, the sample data is behavioral data of deaf-mute people in a venue learning situation. The first behavioral data includes the learner's facial expression data, the second behavioral data includes the learner's head posture data when viewing the exhibition, and the third behavioral data includes the learner's sign language movement data.

[0066] like Figure 2 As shown, a convolutional neural network model is constructed to process the first behavior data to obtain cognitive emotional state recognition data, specifically including:

[0067] The facial expressions of deaf-mute learners during their visit to the venue were collected, and a variable-size convolutional neural network was designed to perform fine-grained detection of facial features in the deaf-mute videos, thereby achieving classification and recognition of the deaf-mute's cognitive emotions in six dimensions: neutrality, distraction, happiness, frustration, boredom, and doubt.

[0068] A convolutional neural network of variable size, including an input layer, at least 4 stacked convolutional layers, a pooling layer, a spatial pyramid pooling layer, and a softmax classification layer:

[0069] The input layer inputs the image of the venue learning situation into the convolutional layer for feature extraction, and obtains feature information for facial expression recognition;

[0070] The pooling layer discards some features and inputs them into the Spatial Pyramid Pooling (SPP) layer. After the feature maps of different sizes are spliced ​​through the spatial pyramid pooling layer, a feature vector of fixed length can be obtained, thereby ensuring the smoothness of the classifier before the features are input into the softmax layer.

[0071] Calculate the error between the features output by the softmax layer and the actual expression category. Based on the calculated error, backpropagate and calculate the gradient value of the parameter optimization of each layer. Then adjust the parameters of each layer in a loop until the error between the model output and the actual expression reaches a minimum and the iteration stops.

[0072] Variable size convolutional neural network in this embodiment Figure 1 There are two different sizes of convolution operations (Conv) to sample features from the image, the sizes are 1×1 and 3×3 respectively. The rate represents the number of convolution kernels. The convolution operation is implemented as follows:

[0073]

[0074] In the above formula, Represents the input image sample in the current state. The superscript x and y represent the length and width of the sample, respectively. Two sizes of convolution kernels are used for differential feature extraction. The subscript (l)n represents the nth feature map in the lth layer. represents the weight connecting the convolution kernel to the nth feature map, p and q represent the spatial size of the convolution kernel. represents the bias term of the jth feature map in the lth layer, and f represents the output after the convolution operation. The pooling layer follows immediately to discard irrelevant features. The pooling operation is expressed as:

[0075]

[0076] In the above formula, is the two-dimensional tensor output by the convolution operation, and m represents the size of the pooling operation in the spatial dimension, in which only the maximum value is retained.

[0077] In order to combat the low face resolution caused by different distances, a spatial pyramid pooling layer is added before the fully connected layer according to the different resolutions of the input sample data.

[0078] The pyramid pooling layer consists of three max pooling layers with grid sizes of 4×4, 2×2, and 1×1. For any input feature map of any size, the three max pooling layers divide the image into 16, 4, and 1 blocks, respectively, and then perform max pooling on each block. The first pooling layer extracts a 16-dimensional feature vector, the second pooling layer extracts a 4-dimensional feature vector, and the third pooling layer extracts a 1-dimensional feature vector. Finally, the feature vectors extracted by the three pooling layers are fused to obtain a 16+4+1=21-dimensional feature vector. Feature maps of different sizes are concatenated through the spatial pyramid pooling layer to obtain a fixed-length feature vector, ensuring that the feature dimensions of the input to the fully connected classifier are consistent.

[0079] The fully connected layer uses the softmax function to calculate the probability of the category to which the feature map belongs to obtain the classification result. The calculation logic of the softmax function is:

[0080]

[0081] In the above formula, x i represents the output of the i-th neuron in the output layer; k represents the number of output parameters, that is, the type of expression classification. In this way, k scalars x1,...,x k Convert to a probability distribution: p1,...,p kDuring the training phase, the model performs supervised learning based on the labels of the facial expression database, updating the model's gradient and weight information. During the testing phase, the model similarly fuses the features obtained by convolution kernels of different sizes and outputs the p-value with the highest probability as the final expression recognition result.

[0082] like Figure 3 As shown in the figure, a Transformer model with spatiotemporal feature fusion is constructed to process the second behavior data to obtain cognitive focus recognition data, specifically including:

[0083] First, the video is sliced ​​uniformly frame by frame in the spatial dimension. For the original input image x, the image is divided into N sub-regions of size N×N using a grid, where N = HW / P 2 , and the total length of the output sequence is determined by the value of N. Then, the image is adjusted to a one-dimensional linear sequence according to the sequence of sub-images, that is, Each block in the sequence contains the image position information, along with the one-dimensional vector of feature embedding, which is input into the model with the self-attention mechanism. The calculation logic is:

[0084]

[0085] In the above formula, Q, K, and V are obtained by linear transformation of the input vector x. Q represents the information to be queried, K represents the queried vector, V represents the value obtained by the query, d is the dimension of the query vector Q and the value vector K, and B represents the position offset matrix. Similarly, after high-dimensional features are extracted from space, each frame of the picture is treated as a slice in the time dimension, and the temporal correlation between multiple frames is obtained after temporal information encoding. Finally, the obtained head orientation description operator is passed to the classification inference module to estimate the horizontal and vertical deflection angles of the head, and the deflection angle of the head posture is returned in the pitch angle (pitch) and yaw angle (yaw) dimensions. Here, the judgment logic of the deflection angle and the cognitive attention of the audience is as follows:

[0086]

[0087] In the above formula, R pitch Indicates the pitch angle of the head in the horizontal direction, R roll Indicates the yaw angle in the vertical direction. If R pitch and R roll If the values ​​of are between the set thresholds -20° and 20°, the current moment is judged to be focused; otherwise, if R pitch and R roll If one of the values ​​of does not meet the threshold condition, it means that the learner is not focused at the current moment.

[0088] like Figure 4As shown in the figure, a video of a deaf-mute person expressing their body language is fed into a semantic recognition model. Feature maps are first constructed from three channels: spatial, local temporal, and global temporal. The three-channel video features are then fed into a feature extraction module consisting of four convolutional layers and two pooling layers. After convolution and pooling, the resulting deep feature map is fed into a long short-term memory (LSTM) unit to further extract contextual temporal information about the body language.

[0089] The LSTM unit controls the linear information to be transmitted in a loop by introducing a gate structure, and transmits the nonlinear information to the external state h of the hidden layer through the output. t :

[0090]

[0091] h t =o t ⊙tanh(c t )

[0092] In the above formula, ⊙ represents the Hadamard product of vector elements; c t-1 Represents the output of the memory unit at the previous moment. Represents the candidate state output by the nonlinear activation function, and its calculation logic is:

[0093]

[0094] In the above formula, x t Indicates the input information in the current state, h t-1 Represents the output of the previous moment, W and U c Represents the weight of the input information at the current moment and the previous moment, b c Represents the bias term; at time t, c t As a state unit, it contains all the historical information contained in the information transmitted to the current moment, and the residual ratio is controlled by the gate structure:

[0095] f t =σ(W i x t +U i h t-1 +b i )

[0096] i t =σ(W f x t +U f h t-1 +b f )

[0097] o t =σ(W o xt +U o h t-1 +b o )

[0098] In the above formula, f t represents the forget gate, i t represents the input gate, o t represents the output gate, σ(·) represents the nonlinear activation function, which can normalize the output values ​​of the three gates to the range of [0,1]. t Based on the current input x t and the output h at the previous moment t-1 Determine the information that the neuron needs to retain at the current moment; input gate i t According to x t 、h t-1 And the forget gate weight U f Controls added to candidate state The proportion of information in to generate a new state c t ; Output gate o t According to the updated c t State, determines the output value h at that moment t .

[0099] A spatial pyramid pooling layer is added between the softmax layer and the LSTM model to concatenate feature maps of different sizes into a fixed-length feature vector. The softmax function is then used to classify the current features. Finally, the outputs of different feature channels are fused and voted to obtain the recognition results of the deaf-mute learner's body language.

[0100] like Figure 5 As shown, the historical data of cognitive emotional state recognition data, visit cognitive concentration recognition data and interactive activity are obtained, and the weight factors of the cognitive emotional state recognition data, visit cognitive concentration recognition data and interactive activity are determined;

[0101] A learning style coding framework was constructed, and weighted calculations were performed on the emotional engagement, cognitive focus, and interactive activity identification data using weight factors to generate a venue learning style profile for deaf and mute learners. Specifically, the profiles include:

[0102] Based on the characteristic and statistical information of deaf-mute learners' learning cognitive emotions, head posture during the exhibition, and learning interaction behaviors in the machine-generated venue videos, models for learners' emotional engagement, learning focus, and interactive activity were constructed. Each piece of information was derived from the sub-models S1 through S3 and assigned weighting factors α, β, and η. Based on the established models for emotional engagement, learning focus, and behavioral engagement, the frequency of venue behaviors was statistically analyzed based on the learning behaviors identified in these three dimensions, following the learning style coding framework, to generate a visual assessment of individual venue learning styles.

[0103] like Figure 6 As shown, the system first uses the learner's initial state of entering a specific exhibit as the starting point. By identifying and recording both focused behaviors (active interaction, physical responses, interactive operations, and recording) and distracted behaviors (looking down, looking around, and quickly leaving), the system then compiles statistics on individual learner behaviors. The behavior understanding module detects and assesses learner movements. All detection results are uploaded to the exhibition center server in real time via the sensor storage interface. After the visit, the learning style is assessed based on four dimensions: independence, dependence, kinesthetic sense, and silence.

[0104] A coding system for venue learning style profiling was developed, segmenting learner behavior into four areas: visitor information expression, visitor activity, the influence of the instructor on learning, and the information flow of the learning scenario. Each area was further divided into four sub-levels, creating a behavioral coordinate system encompassing 16 dimensions. By statistically analyzing historical behavioral data, scores for each dimension were calculated. These values ​​were recorded and interconnected in the behavioral coordinate system, forming an irregularly zigzag closed-loop "venue learning effect cloud."

[0105] like Figure 7 As shown, the learning effects of different visitors can be reflected from the normalized area, external perimeter, center of gravity, range, and coefficient of dispersion. For example, compared with learner 1, the overall shape of learner 0's behavior cloud is fuller, more uniform, and has a smaller range, indicating that his ability has developed well and he has achieved relatively ideal learning results. Moreover, by further comparing the center of gravity of the behavior cloud, it can be found that the two have obvious differences in learning styles. Learner 1 tends to observe the exhibits one by one according to the pre-set rhythm of the venue, while learner 0 is more willing to explore and interact independently. In the later stage, we can combine the learners' learning style preferences and the differentiated characteristics of the learning groups to push more suitable exhibit explanation content and extended materials in a targeted manner; through high-quality learning support services, we can achieve "one thousand faces for one thousand people" and teach students in accordance with their aptitude.

[0106] The method described in this embodiment combines the learning process of deaf-mute people in venues with relevant measurement methods based on artificial intelligence. Based on the reference to traditional educational observation and analysis research, it uses data-driven machine intelligence to establish an association between the multi-faceted and heterogeneous explicit behaviors (learning cognitive emotions, exhibition viewing head posture, learning interactive behaviors) in the learning context of deaf-mute people in venues and potential implicit states (emotional involvement, learning concentration, behavioral participation); and generates learning styles based on this information, which has obvious advantages over traditional learning analysis methods.

[0107] A second aspect of this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a method for generating a style portrait through learning from behavioral data is implemented.

[0108] A third aspect of this embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, a method for generating a style profile through behavioral data learning is implemented.

[0109] Example 2

[0110] This embodiment provides a behavioral data learning style profile generation system, including:

[0111] Sample data collection module: used to collect first behavior data, second behavior data and third behavior data;

[0112] The first processing module is used to build a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data. The cognitive emotional state recognition data is segmented into six expressions: neutral, distracted, happy, frustrated, bored, and confused by the expression recognition framework;

[0113] The second processing module is used to build a Transformer model for spatiotemporal feature fusion, process the second behavior data, and obtain cognitive focus recognition data. The cognitive focus recognition data is calculated by the head posture estimation model to calculate the learner's head deflection angles in the horizontal and vertical angles. Combined with the threshold judgment method, it further returns the learner's concentration duration;

[0114] The third processing module is used to build a semantic recognition model, process the third behavior data, and obtain interactive activity recognition data. The interactive activity recognition data is obtained by counting the interaction frequency of deaf and mute learners in the venue context based on the body semantic understanding framework of deaf and mute learners;

[0115] Data processing module: used to quantify cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate behavioral data sequences;

[0116] Learning style generation module: used to construct a learning style coding framework, generate learning style portraits based on behavioral data sequences, automatically sample and count various types of information from 16 dimensions in four major areas: visitor interaction information, visitor activity behavior, lecturer's learning influence, and information flow of learning scenarios, form a data sequence of learning style portraits based on behavioral frequency, and use radar charts to display learning style portraits.

[0117] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a style profile by learning behavioral data, characterized in that: The specific steps include: S1. Obtain sample data, the sample data including: first behavior data, second behavior data, and third behavior data; wherein the sample data is behavior data of a deaf-mute person in a learning environment at a venue, the first behavior data including the learner's facial expression data, the second behavior data including the learner's head posture data while viewing an exhibition, and the third behavior data including the learner's sign language movement data; S2. Construct a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data; A Transformer model that integrates spatiotemporal features is constructed to process the second behavioral data to obtain cognitive focus recognition data. Processing the second behavioral data includes estimating the learner's head deflection angle and calculating the learner's cognitive focus state using a threshold determination method. Build a semantic recognition model, process the third-party behavior data, and obtain interactive activity recognition data; S3. Quantify cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate a behavioral data sequence; S4. Construct a learning style coding framework and generate a learning style profile based on the behavioral data sequence.

2. A method for generating a style profile based on behavioral data learning according to claim 1, characterized in that: The step of constructing a convolutional neural network model and processing the first behavior data to obtain cognitive emotional state recognition data specifically includes: Obtain the first behavior data, build a convolutional neural network model to extract features from the first behavior data, and obtain feature labels; The feature labels are screened and the feature labels of different sizes are concatenated to obtain a feature vector of fixed length. The softmax function is used to determine the probability of the category to which the feature label belongs, and the feature label classification result is obtained. The back-propagation algorithm is used to update the error between the feature label classification result and the actual expression category, and multiple rounds of iterative training of the convolutional neural network model parameters are performed; The feature labels are input into the trained convolutional neural network model, and the feature labels obtained using convolution kernels of different sizes are fused. The classification result with the highest probability is output as the final expression recognition result, and the cognitive emotional state recognition data is determined based on the final expression recognition result.

3. The method for generating a style profile by learning behavioral data according to claim 1, characterized in that: The Transformer model for fusion of spatiotemporal features is constructed to process the second behavior data to obtain cognitive concentration recognition data, specifically including: The second behavior data is obtained, and a Transformer model with spatiotemporal feature fusion is constructed to extract the angular deflection features of the learner's head. The head posture is estimated based on the angular deflection features of the learner's head through a softmax regressor. The cognitive concentration recognition data is obtained based on the estimated horizontal and vertical deflection angles of the head posture.

4. A method for generating a style profile by learning behavioral data according to claim 3, characterized in that: The step of estimating the head posture based on the angle deflection feature of the learner's head specifically includes: Obtain the learner's head posture data while viewing the exhibition, uniformly slice the video frame by frame in the spatial dimension, convert the two-dimensional image into a one-dimensional linear sequence, and regress and predict the learner's head deflection angle through horizontal and vertical angles; The sequence is flattened by the linear compression module to obtain the embedded vector of the image data as the input of the Transformer module; The sequence contains image position information, and the spatial transformer treats each key point of the frame as a slice and obtains high-dimensional features through spatial position embedding; By extracting high-dimensional features from space and encoding temporal information as a slice in the time dimension, the temporal correlation between multiple frames of images is obtained; The head orientation description operator is obtained based on the temporal correlation between multiple frames of images, and the head orientation description operator is passed into the classification inference module to estimate the horizontal and vertical deflection angles of the head to obtain cognitive concentration recognition data.

5. The method for generating a style profile by learning behavioral data according to claim 1, characterized in that: The step of constructing a semantic recognition model and processing the third behavior data to obtain interactive activity recognition data specifically includes: Obtaining the third behavior data, building a three-channel convolutional neural network model, and constructing feature maps from three channels: spatial, local time domain, and global time domain: the spatial channel CNN performs deep learning on the action image, the local time domain channel CNN performs deep learning on the optical flow field, and the global time domain channel CNN performs deep learning on the action difference image; Concatenate feature maps of different sizes into feature vectors of fixed length and classify the concatenated features; The three-channel video features are input into the feature extraction module to obtain the deep feature map which is input into the long short-term memory unit to extract the contextual time domain information of the body semantics; The outputs of different feature channels are fused according to the contextual time domain information of body semantics to vote and obtain the recognition results of the learner's body semantics.

6. A method for generating a style profile by learning behavioral data according to claim 1, characterized in that: Obtaining historical data on cognitive emotional state recognition data, visit cognitive concentration recognition data, and interactive activity, and determining weight factors for the cognitive emotional state recognition data, visit cognitive concentration recognition data, and interactive activity; Construct a learning style coding framework, combine weight factors to perform weighted calculations on emotional engagement, cognitive concentration recognition data, and interactive activity recognition data, and generate a venue learning style portrait of deaf and mute learners.

7. A behavioral data learning style profile generation system, characterized by: include: Sample data collection module: used to collect first behavior data, second behavior data and third behavior data; wherein the first behavior data includes learner's facial expression data, the second behavior data includes learner's head posture data when viewing the exhibition, and the third behavior data includes learner's sign language movement data; The first processing module is used to build a convolutional neural network model to process the first behavior data to obtain cognitive emotional state recognition data; The second processing module is used to construct a Transformer model for spatiotemporal feature fusion, process the second behavior data, and obtain cognitive focus recognition data. The processing of the second behavior data includes estimating the learner's head deflection angle and calculating the learner's cognitive focus state using a threshold determination method. The third processing module is used to build a semantic recognition model, process the third behavior data, and obtain interactive activity recognition data; Data processing module: used to quantify cognitive emotional state recognition data, cognitive concentration recognition data, and interactive activity recognition data to generate behavioral data sequences; Learning style generation module: used to construct a learning style coding framework and generate a learning style portrait based on the behavioral data sequence.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the method for generating a behavioral data learning style portrait as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for generating a behavior data learning style portrait is implemented as described in any one of claims 1 to 6.

Images