A hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning

Abstract

This application belongs to the field of image sentiment analysis technology, specifically disclosing a hand-drawn artwork sentiment recognition method based on multi-feature fusion and ensemble learning. The method includes: preprocessing a hand-drawn image to obtain a preprocessed hand-drawn image; image preprocessing is used to remove noise and enhance the image; based on the preprocessed hand-drawn image, performing multi-dimensional feature extraction to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features; based on the color temperature features, brushstroke dynamics features, and compositional spatial distribution features, performing cross-attention fusion encoding to obtain a sentiment feature vector; cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and fuse the extracted attention vectors; based on the sentiment feature vectors, ensemble learning is used to predict the sentiment classification of the hand-drawn image. This application enables efficient and accurate recognition of the sentiment classification of hand-drawn artworks.

Description

Technical Field

[0001] This application belongs to the field of image sentiment analysis technology, and more specifically, relates to a hand-drawn sentiment recognition method based on multi-feature fusion and ensemble learning. Background Technology

[0002] Hand-drawn artworks can serve as a non-verbal medium for expressing human emotions. Traditional analytical methods relying on expert subjective judgment suffer from inefficiency and low consistency, making them unsuitable for processing large volumes of data. Furthermore, existing automated technologies often focus on single feature dimensions, resulting in insufficient generalization ability and impacting prediction accuracy. Therefore, how to efficiently and accurately identify the emotional classification of hand-drawn artworks is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this application is to achieve efficient and accurate identification of the emotional classification of hand-drawn works.

[0004] To achieve the above objectives, firstly, this application provides a hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning, the method comprising: Based on the hand-drawn image (image of a hand-drawn artwork), image preprocessing is performed to obtain the preprocessed hand-drawn image. Image preprocessing is used to remove noise from the image and enhance the image. Based on the preprocessed hand-drawn image, multi-dimensional feature extraction is performed to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features. Based on color temperature features, brushstroke dynamics features, and compositional spatial distribution features, cross-attention fusion encoding is performed to obtain emotional feature vectors. Cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and to fuse and encode the extracted attention vectors. Each attention path is used to process two of the color temperature features, brushstroke dynamics features, and compositional spatial distribution features through a cross-attention mechanism. Based on sentiment feature vectors, ensemble learning is used to predict the sentiment classification (e.g., positive or negative emotions) of hand-drawn images.

[0005] Understandably, image preprocessing removes noise and enhances the image, providing a high-quality data foundation for subsequent feature extraction. Next, multi-dimensional features such as color temperature, brushstroke dynamics, and compositional spatial distribution are extracted from the preprocessed image. These features deconstruct emotional expression from three levels: visual perception, creative behavior, and spatial structure. Subsequently, cross-attention fusion encoding uses multiple attention paths to perform pairwise cross-processing on these three types of features, fusing the feature vectors extracted from multiple attention paths into a high-dimensional emotional feature vector. This achieves deep complementarity between features and enhancement of emotional semantics—by fusing multiple feature emotional cues, the model's generalization ability is improved, ensuring prediction accuracy. Finally, ensemble learning, based on this emotional feature vector and combined with the prediction results of multiple base models, outputs the final emotional classification through strategies such as voting or weighted averaging. From image preprocessing to multi-dimensional feature extraction, to cross-attention fusion encoding, and finally to emotional classification, the entire process is automated, ensuring high efficiency.

[0006] Therefore, by integrating multi-feature emotional cues with full-process automation, it is possible to achieve both high efficiency and accuracy in the emotional recognition of complex hand-drawn works.

[0007] In one possible implementation, the above-mentioned preprocessed hand-drawn image undergoes multi-dimensional feature extraction to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features, including: Based on the preprocessed hand-drawn image, the color temperature characteristics are obtained by statistically analyzing the difference between the red and green channels and the saturation value of the pixels using a weighted average method. Based on a preset grid (e.g., an 8×8 grid), the preprocessed hand-drawn image is divided into multiple cells (e.g., the preprocessed hand-drawn image is divided into 64 blocks using an 8×8 grid, with each block being a cell). The brushstroke force fluctuation and curvature are statistically analyzed for each cell to obtain brushstroke dynamics features. The preprocessed hand-drawn image is divided into multiple regions according to a specified region scale (the region scale is used to indicate the size of the region), and the effective pixel ratio of each region is counted. Based on the effective pixel ratio of each region under multiple region scales, the compositional spatial distribution characteristics are obtained.

[0008] In one possible implementation, the preprocessed hand-drawn image is used to statistically analyze the difference between the red and green channels and the saturation value of each pixel using a weighted average method to obtain color temperature features, including obtaining the color temperature features using the following formula: ; ; ; ; in, Indicates the overall temperature value. Indicates the proportion of warm colors in the area. Indicates the percentage of cool colors in the area. Indicates average saturation; Indicates the first The red channel value of each pixel Indicates the first The green channel value of each pixel Indicates the first The saturation value of each pixel in the HSV color space Indicates the first Visual attention weight per pixel; Color temperature characteristics include , , and .

[0009] In one possible implementation, Based on the saliency graph The saliency map is obtained by taking the values ​​of each pixel and using the Itti-Koch model. In the linear superposition stage of synthesizing saliency maps using the Itti-Koch model, the weight corresponding to brightness contrast (e.g., 0.5) is greater than the weight corresponding to color contrast (e.g., 0.3), and the weight corresponding to color contrast is greater than the weight corresponding to orientation contrast (e.g., 0.2).

[0010] In one possible implementation, the above-mentioned statistical analysis of the pen pressure fluctuation and curvature of each cell includes calculating the pen pressure fluctuation value and curvature frequency corresponding to each cell using the following formulas: ; ; in, This indicates the fluctuation value of pen strokes. This represents the number of stroke sampling points in the cell. For the cell's first The stroke width of each stroke sampling point Indicates the curvature frequency. This specifies the number of pen stroke sampling points in the cell whose curvature value is greater than a preset curvature (e.g., 0.8 rad / mm). The pen stroke dynamics features include the pen force fluctuation value and curvature frequency corresponding to each cell.

[0011] In one possible implementation, multiple attention paths include a color-brushstroke path and a brushstroke-composition path. These paths, based on color temperature features, brushstroke dynamics features, and compositional spatial distribution features, undergo cross-attention fusion encoding to obtain an emotional feature vector, including: Input color temperature features and brush stroke dynamics features into the color-brush path, and obtain the attention vector output by the color-brush path; Input the brushstroke dynamics features and composition space distribution features into the brushstroke-composition path, and obtain the attention vector output by the brushstroke-composition path; The attention vectors corresponding to the color-brushstroke path and the attention vectors corresponding to the brushstroke-composition path are concatenated to obtain the concatenated features. Based on the concatenated features, a multi-layer Transformer encoder is used to encode and obtain the sentiment feature vector. Among them, the color-brushstroke path is used to take the color temperature feature as the query vector and the brushstroke dynamics feature as the key and value vectors. By calculating the attention, the attention vector corresponding to this path is extracted. The brushstroke-composition path uses brushstroke dynamics features as a query vector and composition space distribution features as key and value vectors. By calculating attention, the attention vector corresponding to this path is extracted.

[0012] In one possible implementation, the above-mentioned prediction of the sentiment classification of hand-drawn images based on sentiment feature vectors through ensemble learning includes: Based on sentiment feature vectors, the probability of sentiment classification is predicted through multiple base models; Based on the sentiment classification probabilities predicted by each base model, an ensemble decision is made through a meta-learner to obtain the sentiment classification of the hand-drawn image. Several base models include: convolutional neural networks and Transformer.

[0013] In one possible implementation, the above image preprocessing includes: median filtering and contrast-limited adaptive histogram equalization (CLAHE).

[0014] Secondly, this application provides a hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning, comprising: The image preprocessing module is used to perform image preprocessing based on the hand-drawn image, and obtain the preprocessed hand-drawn image. Image preprocessing is used to remove noise from the image and enhance the image. The multi-dimensional feature extraction module is used to extract multi-dimensional features based on the preprocessed hand-drawn image, and obtain color temperature features, brushstroke dynamics features and compositional spatial distribution features. The cross-attention fusion encoding module is used to perform cross-attention fusion encoding based on color temperature features, brush stroke dynamics features, and compositional spatial distribution features to obtain emotional feature vectors. The cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and to fuse and encode the extracted attention vectors. Each attention path is used to process two of the color temperature features, brush stroke dynamics features, and compositional spatial distribution features through the cross-attention mechanism. The sentiment classification module is used to predict the sentiment classification of hand-drawn images based on sentiment feature vectors through ensemble learning.

[0015] Thirdly, this application provides an electronic device, including: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors invoke the computer instructions to cause the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0016] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0017] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: This application improves data quality by preprocessing images, and then analyzes color, brushstrokes, and composition. Figure 3 Features are extracted from multiple dimensions, and then these features are deeply fused and complemented using a cross-attention mechanism. Finally, the final sentiment classification result is output through an ensemble learning model. This process ensures the model's generalization ability and prediction accuracy in complex scenarios by integrating multi-dimensional sentiment cues, and ensures high efficiency through full automation. Attached Figure Description

[0018] Figure 1 This is one of the flowcharts illustrating the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning provided in the embodiments of this application; Figure 2 This is the second flowchart of the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning provided in the embodiments of this application; Figure 3 This is a diagram of the multi-dimensional feature fusion encoder architecture provided in the embodiments of this application; Figure 4 This is a schematic diagram of the improved VGG16 architecture provided in the embodiments of this application; Figure 5This is a schematic diagram of the improved Vision Transformer (ViT) architecture provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning provided in the embodiments of this application; Figure 7 This is a schematic diagram of the system layered architecture provided in the embodiments of this application; Figure 8 This is one of the schematic diagrams of a hand-drawn emotion recognition feature visualization example provided in the embodiments of this application; Figure 9 This is the second schematic diagram of a hand-drawn emotion recognition feature visualization example provided in the embodiments of this application; Figure 10 This is the third illustration of a hand-drawn emotion recognition feature visualization example provided in the embodiments of this application; Figure 11 This is the fourth illustration of a hand-drawn emotion recognition feature visualization example provided in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0022] First, the technical terms involved in the embodiments of this application will be introduced.

[0023] (1) Scanning noise; Scan noise mainly refers to visual imperfections that are not inherent to the painting itself, introduced during the digitization process of hand-drawn works (such as scanning or photographing).

[0024] (2) Itti-Koch model; The Itti-Koch model is a visual saliency calculation model. Its core idea is to mimic the human visual system and quickly identify the most "prominent" regions in complex scenes. It can be understood as an algorithm that highlights key areas in an image. The workflow of the Itti-Koch model mainly consists of the following three steps: step a, step b, and step c.

[0025] Step a, Feature extraction—Extract three basic visual features from the input image: color contrast features, brightness contrast features, and orientation contrast features.

[0026] Color contrast feature (using Euclidean distance in RGB space): Find areas with large color differences from the surrounding colors.

[0027] Brightness contrast characteristics (based on Lab space L channel): Find areas with high contrast to the surrounding light and dark areas.

[0028] Orientation contrast features (Gabor filter multi-directional response): Find edge or texture regions with a specific orientation.

[0029] Step b, multi-scale analysis and feature map generation—a Gaussian pyramid (i.e., different resolution versions of the same image) is constructed for each feature, and the contrast is calculated at different scales to generate multiple feature maps showing the prominence of the feature.

[0030] Step c, synthesizing the saliency map—normalizing and linearly superimposing the feature maps of color contrast, brightness contrast, and orientation contrast to synthesize a single saliency map. This map is like a heat map; the brighter the area, the more salient it is, and the more attention it attracts.

[0031] In the prior art, during the linear superposition process in step c, the weights corresponding to color contrast, brightness contrast, and orientation contrast remain equal and approximately equal to 0.33.

[0032] (3) Stacking integration; Stacking ensemble is a machine learning technique that typically consists of two layers: the first layer comprises multiple base models, and the second layer is a meta-learner. The base models output multiple predictions, and the meta-learner synthesizes these predictions from the first layer to form the final prediction. This is analogous to multiple experts providing multiple opinions, and the meta-learner combining these opinions to make a final decision.

[0033] The embodiments of this application are described below with reference to the accompanying drawings.

[0034] Figure 1 This is one of the flowcharts illustrating the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning provided in this application embodiment, such as... Figure 1 As shown, the method includes the following steps S101 to S104.

[0035] Step S101: Based on the hand-drawn image, perform image preprocessing to obtain the preprocessed hand-drawn image. Image preprocessing is used to remove noise from the image and enhance the image. Step S102: Based on the preprocessed hand-drawn image, perform multi-dimensional feature extraction to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features; Step S103: Based on color temperature features, brushstroke dynamics features, and compositional spatial distribution features, cross-attention fusion encoding is performed to obtain emotional feature vectors; cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and to fuse and encode the extracted attention vectors; each attention path is used to process two of the color temperature features, brushstroke dynamics features, and compositional spatial distribution features through a cross-attention mechanism; Step S104: Based on the sentiment feature vector, predict the sentiment classification of the hand-drawn image through ensemble learning.

[0036] Understandably, image preprocessing removes noise and enhances the image, providing a high-quality data foundation for subsequent feature extraction. Next, multi-dimensional features such as color temperature, brushstroke dynamics, and compositional spatial distribution are extracted from the preprocessed image. These features deconstruct emotional expression from three levels: visual perception, creative behavior, and spatial structure. Subsequently, cross-attention fusion encoding uses multiple attention paths to perform pairwise cross-processing on these three types of features, fusing the feature vectors extracted from multiple attention paths into a high-dimensional emotional feature vector. This achieves deep complementarity between features and enhancement of emotional semantics—by fusing multiple feature emotional cues, the model's generalization ability is improved, ensuring prediction accuracy. Finally, ensemble learning, based on this emotional feature vector and combined with the prediction results of multiple base models, outputs the final emotional classification through strategies such as voting or weighted averaging. From image preprocessing to multi-dimensional feature extraction, to cross-attention fusion encoding, and finally to emotional classification, the entire process is automated, ensuring high efficiency.

[0037] Therefore, by integrating multi-feature emotional cues with full-process automation, it is possible to achieve both high efficiency and accuracy in the emotional recognition of complex hand-drawn works.

[0038] The following examples illustrate the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning provided in this application.

[0039] Figure 2 This is the second flowchart illustrating the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning provided in this application embodiment, as shown below. Figure 2 As shown, the main processing steps include: (1) image preprocessing; (2) multi-dimensional feature extraction; (3) cross-attention fusion coding; and (4) ensemble classification.

[0040] Specifically, the input hand-drawn image is first preprocessed, standardized to a resolution of 512×512 pixels, and subjected to 3×3 median filtering to remove scanning noise. Local contrast is then enhanced using the CLAHE algorithm. Next, color, brushstroke, and composition are constructed. Figure 3 3D feature system, color temperature features are expressed by formula Calculation, where To improve the weights of the visual attention model, the brushstroke dynamics features include brush pressure fluctuations and curvature frequency, which are quantified by the rate of change of pressure and the proportion of high curvature points, respectively. The composition space distribution features are calculated based on the pixel distribution complexity of image blocks. Then, a cross-attention fusion encoder is used to encode the multimodal features into a unified sentiment vector. This encoder includes a dual-path attention mechanism and a Transformer encoding layer. Finally, a stacking ensemble strategy is used to fuse the outputs of CNN and Transformer base models, and the sentiment classification results are output through a meta-learner (in this application, the meta-learner is a logistic regression model).

[0041] The following is an example of the "image preprocessing" step.

[0042] The input hand-drawn image (obtained by scanning a hand-drawn artwork) is standardized to a resolution of 512×512 pixels. A 3×3 median filter is used to remove scanning noise. Local contrast is enhanced by Contrast Limited Adaptive Histogram Equalization (CLAHE). The clip limit parameter of the CLAHE algorithm is set to 2.0 to preserve brushstroke details while improving feature discriminability.

[0043] The following is an example of the "multi-dimensional feature extraction" step.

[0044] Constructing color, brushstrokes, and structure Figure 3 The dimensional feature system specifically includes the following: (1) color temperature features, (2) brushstroke dynamics features and (3) compositional spatial distribution features.

[0045] (1) Color temperature characteristics: A comprehensive color emotion feature vector is constructed by calculating the weighted average of the differences between the red and green channels, which quantifies color emotion from three aspects: hue trend, warm and cool distribution, and saturation. Specifically, it is characterized by the following four indicators: overall warm and cool value, warm color area ratio, cool color area ratio, and average saturation.

[0046] Overall temperature rating: ; The overall warm / cool value reflects the overall color tone bias of the image, and is calculated by visually weighted average of the differences between the red and green channels.

[0047] Warm color area proportion: ; The proportion of warm-toned areas measures the visual importance of warm-toned areas (i.e., pixels with red channel values ​​greater than green channel values) in an image.

[0048] Percentage of cool colors: ; The proportion of cool-toned areas measures the visual importance of cool-toned areas (i.e., pixels with red channel values ​​less than green channel values) in an image.

[0049] Average saturation: ; Average saturation reflects the overall color vibrancy of an image, and is also weighted by visual attention.

[0050] in pixels The red and green channel values ​​(0-255). For pixels Saturation values ​​(0-1) in the HSV color space. For the first Visual attention weights for each pixel (0-1). The saliency map is obtained through the Itti-Koch model, based on the first... The value of each pixel is determined. In this application, the Itti-Koch model is improved: in the linear superposition stage of synthesizing saliency maps, the weight corresponding to color contrast is configured as 0.3, the weight corresponding to brightness contrast is configured as 0.5, and the weight corresponding to orientation contrast is configured as 0.2. This represents the total number of pixels in the image.

[0051] Vector of color temperature characteristics Represented as: .

[0052] A vector representing color temperature comprehensively reflects the emotional tendency of a color. Typically, it represents an overall warm or cool value. And the proportion of warm colors When it is higher, it corresponds to positive and warm emotions; conversely, when And the proportion of cool colors A higher level corresponds to negative, calm emotions. Weighted saturation It further depicts the intensity of the emotion.

[0053] It should be noted that in the existing Itti-Koch model, during the linear superposition stage of saliency map synthesis, the weights corresponding to color contrast, brightness contrast, and orientation contrast are kept equal and approximately equal to 0.33. However, in this application, the weights corresponding to brightness contrast are configured to be greater than those corresponding to color contrast, which in turn are greater than those corresponding to orientation contrast. Experiments on emotion recognition of hand-drawn artworks show that, compared to existing weight configurations, the weight configuration in this application can more accurately guide the model to focus on the truly emotional parts of the artwork, thereby improving the final classification accuracy.

[0054] Optionally, the weight of the stroke edge region in the saliency map can be enhanced by 1.2 times through the detection results of the Canny operator.

[0055] (2) Brushstroke dynamics characteristics: A 64-dimensional dynamic feature vector was generated through 8×8 spatial grid analysis, which includes brush force fluctuation values. and curvature frequency The spatial distribution information, where the formula for calculating the pen force fluctuation value is: ; The formula for calculating curvature frequency is: ; in, This represents the number of pen stroke sampling points, with a sampling interval of 0.5 mm. For the first The stroke width of each sampling point (in pixels); The number of high curvature points with a curvature value > 0.8 rad / mm; the stroke feature vector is formed into a 64-dimensional vector by spatially encoding the stroke dynamics features of each grid in the 8×8 grid.

[0056] Experiments show that: A higher value indicates more drastic stress changes, corresponding to heightened emotions; high-frequency transitions ( It represents tension.

[0057] (3) Spatial distribution characteristics of the composition: A 60-dimensional spatial distribution vector is generated through multi-scale block pixel distribution analysis to quantify the layout complexity of the elements in the picture. This feature is based on the core idea of ​​"information entropy" in information theory—the uniformity and uncertainty of pixel distribution can reflect the chaos and tension of emotions, but in its implementation, richer original distribution data is used to preserve complete spatial information.

[0058] The specific generation process includes the following steps a to d.

[0059] Step a: Effective pixel segmentation; To accurately analyze the spatial distribution of the main subject in a painting, it is first necessary to separate the hand-drawn foreground from the background and identify the effective pixels (i.e., the non-background pixels in the painting): Convert the preprocessed RGB image to a grayscale image.

[0060] Set the background grayscale threshold T=240 (value range 0-255). Since the background of hand-drawn works is usually white or near-white paper, this threshold can effectively distinguish the background from the brushstrokes.

[0061] Create a binary mask: Iterate through each pixel in the grayscale image. If its grayscale value is greater than or equal to 240, it is determined to be a background pixel; otherwise, it is determined to be a valid pixel.

[0062] Step b: Multi-scale image segmentation; The image is divided using three different grid scales, and the effective pixel percentage within each grid is calculated based on the effective pixel mask obtained in step one: 4×4 uniform partitioning: The image is divided into 16 uniform regions (16D, 128×128 pixels each), and the effective pixel percentage of each region is calculated. 4×5 rectangular partitioning: The image is divided into 20 rectangular regions (20D) in a 4x5 grid, and the effective pixel percentage of each region is calculated. 6×4 rectangular partitioning: The image is divided into 24 rectangular regions (24D) in a 6x4 grid, and the effective pixel percentage of each region is calculated. It should be noted that in this application, 16 dimensions are abbreviated as 16D, 20 dimensions as 20D, and so on, 24 dimensions as 24D, 60 dimensions as 60D, and so on.

[0063] Step c: Calculate the pixel spatial distribution; The formula for calculating the effective pixel percentage of each block is: .

[0064] Step d: Vector of spatial distribution characteristics of the composition; The vector of spatial distribution features is obtained by concatenating the features of the three types of blocks mentioned above. Specifically, it is obtained by combining the proportion values ​​of all 60 effective pixels calculated by the three types of blocks. By sequentially piecing together vectors, a spatial distribution characteristic of the composition is formed. : ; The total dimensions are 60, and each dimension corresponds to the pixel proportion of a block. .

[0065] The distribution pattern of vectors representing the spatial distribution characteristics of a composition is closely related to the complexity of the composition. If the values ​​in the vectors are relatively even (i.e., uniformly distributed), their theoretical entropy value is high, indicating a dispersed composition, often corresponding to anxiety and tension. If some values ​​in the vectors are significantly larger than others (i.e., concentrated distribution), their theoretical entropy value is low, indicating an ordered composition, often corresponding to calm and stable emotions.

[0066] The following is an example of the "cross-attention fusion encoding" step.

[0067] Figure 3 This is a diagram of the multi-dimensional feature fusion encoder architecture provided in the embodiments of this application, such as... Figure 3 As shown, a dual-path cross-attention mechanism is used to achieve multi-feature fusion.

[0068] Path 1 – Color-Brushstroke Path: Color features (4-dimensional, abbreviated as 4D) are transformed by a linear transformation matrix. Mapped to a 64D Query vector, the brushstroke features (64D) are directly used as the Key / Value vector, and the attention calculation is as follows: ; in, For the dimension of the key vector, These represent the query vector, key vector, and value vector, respectively.

[0069] Path 2 – Brushstroke-Composition Path: Brushstroke features (64 dimensions, abbreviated as 64D) are used as the Query vector, and composition features (60D) are transformed through a linear transformation matrix. The mapping is to a 64D Key / Value vector, and the attention calculation method is the same as above; The 64D features output from the two paths (path 1 and path 2 above) are concatenated into a 128D vector (which can be called concatenated features), and then a 512D sentiment feature vector is generated by a 6-layer Transformer encoder (8-head self-attention, feedforward dimension 2048).

[0070] Optionally, the linear transformation matrix , The Xavier initialization method is used.

[0071] Optionally, the self-attention layer of the Transformer encoder employs a scaled dot product attention mechanism. The feedforward network contains two fully connected layers (2048→512), with GELU activation function. Each layer is equipped with layer normalization and residual connections to prevent gradient vanishing during training. For example, a 6-layer Transformer encoder (8-head self-attention, feedforward dimension 2048) encodes 128D concatenated features into a 512D global sentiment feature vector.

[0072] The following is an example of the "integrated classification" step.

[0073] A stacking ensemble strategy is employed to fuse the outputs of CNN and Transformer base models. The meta-learner is a logistic regression model, specifically defined by the following classification probability formula: ; in, The probability of sentiment classification predicted by CNN. The probability of sentiment classification predicted by the Transformer. The probability of sentiment classification predicted by the meta-learner. It is the sigmoid activation function. , As base model weights, This is the bias term; model training uses 5-fold cross-validation, and the optimizer is Adam (learning rate). Weight decay On a dataset containing 1200 hand-drawn artworks, the emotion classification accuracy reached 90.5%, with an F1 score of 0.892.

[0074] It should be noted that CNN excels at capturing local textures and brushstroke details, while Transformer excels at understanding global composition and color distribution.

[0075] Optionally, the CNN base model adopts an improved VGG16 architecture (removing the last 3 fully connected layers and adding a global average pooling layer and 2 fully connected layers), and the Transformer base model adopts a ViT-Base architecture (12-layer encoder, 12-head attention). The output probabilities of the base models are used to generate meta-features on the validation set through 5-fold cross-validation, and the regularization parameters of the logistic regression meta-learner are used. (L2 regularization), determined through grid search optimization.

[0076] Figure 4 This is a schematic diagram of the improved VGG16 architecture provided in the embodiments of this application, as shown below. Figure 4As shown, the existing VGG16 architecture is improved by removing the last three fully connected layers and adding a global average pooling layer and two fully connected layers.

[0077] for Figure 4 The global average pooling layer in the middle has the following function: to pool the output of the last convolutional layer. The feature map is globally averaged for each channel, and the output is... This results in a vector for each channel, replacing the fully connected layer in the original VGG16.

[0078] It is understandable that by using a global average pooling layer, the following beneficial effects are achieved: (1) significantly reducing parameters; (2) preventing overfitting. (1) Significantly reducing parameters: The original VGG16 flattens the 7x7x512 feature map into a 25088-dimensional vector after the convolutional layer, and then connects two 4096 fully connected layers, resulting in a huge number of parameters (approximately 120 million). Global average pooling only outputs 512 dimensions, with 0 parameters. (2) Preventing overfitting: Since no additional parameters are introduced, the risk of overfitting is reduced. Preserving spatial information: Global average pooling averages each feature map, which can preserve the feature response of each channel, making it more suitable for small-scale datasets.

[0079] for Figure 4 The first fully connected layer in the model serves to pass the 512-dimensional vector output from global average pooling through a 512-unit fully connected layer and apply the ReLU activation function. This layer is used for feature compression and integration. It is understood that employing this fully connected layer provides the following benefits: Significantly reduced parameter count: The original VGG16 has two 4096-node fully connected layers, while this application sets the number of nodes in the fully connected layer to 512, greatly reducing the number of parameters. Dropout (0.3) is added: 30% of neurons are randomly dropped during training to prevent overfitting.

[0080] for Figure 4 The second fully connected layer maps the 512-dimensional features to 2 dimensions, corresponding to positive and negative emotion categories. Understandably, employing this fully connected layer provides the following benefits: output dimension adaptation; the original VGG16 output is 1000 dimensions (corresponding to ImageNet 1000 classes), while this application changes it to a 2-dimensional output, directly corresponding to the binary classification task of hand-drawn emotions. Dropout (0.3) is added: also to prevent overfitting.

[0081] for Figure 4 The Softmax layer in the middle has the function of converting the output of the second fully connected layer into a probability distribution, representing the probability of belonging to positive and negative emotions.

[0082] Figure 5This is a schematic diagram of the improved Vision Transformer (ViT) architecture provided in the embodiments of this application, as shown below. Figure 5 As shown, the difference from the standard ViT is that the input of the model in this application is not the original image pixels, but a 512-dimensional feature vector extracted by the front-end module.

[0083] The following is an exemplary description of the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning provided in this application, with specific experimental examples. The experimental example includes the following steps: (1) preparing the experimental dataset, (2) setting up the experimental environment, (3) data preprocessing, (4) model training, and (5) analyzing the experimental results.

[0084] The following is an exemplary description of the "preparing the experimental dataset" step involved in this experimental example.

[0085] This application validates its effectiveness on a dataset containing 1200 hand-drawn artworks, including 800 from the publicly available ArtEmis dataset (52% positive, 48% negative) and 400 from a self-built dataset (containing 120 paintings by patients with depression). The dataset is divided into a training set (840 artworks), a validation set (180 artworks), and a test set (180 artworks) at 70% / 15% / 15% of the total. The data is independently labeled by three experts, with a Kappa coefficient of 0.82 (P<0.01), ensuring labeling consistency. The data labeling uses a three-class classification system of sentiment polarity (positive / negative / neutral), and the labeling process includes: independent expert labeling, consistency verification, and data augmentation.

[0086] Independent expert annotation: Each expert scores the work based on an emotional vocabulary list (containing 20 basic emotional words such as "joy" and "sadness"), giving it an emotional score (1-5 points). Consistency verification: The Fleiss-Kappa test was used to check the consistency of the annotations. When Kappa < 0.7, a collective discussion and revision were carried out. Data augmentation: The sample size of the self-built dataset was increased by means of rotation (±15°), scaling (0.8-1.2 times), and slight color jitter (brightness ±5%), with an expansion ratio of 1:1.5.

[0087] The following is an exemplary description of the "Experimental Environment Setup" step involved in this experimental example.

[0088] The experimental hardware environment consisted of an Intel i7-10700K CPU (8 cores, 16 threads), an NVIDIA RTX 3090 GPU (32GB VRAM), and 32GB of DDR4 RAM. The software environment included Python 3.8, TensorFlow 2.5, OpenCV 4.5, and Scikit-learn 0.24.2. The model training hyperparameters were set as follows: Optimizer: AdamW (β1=0.9, β2=0.999, weight decay). Learning rate: Initially 5e-5, decaying to 5e-5 using a cosine annealing strategy. Batch size: 32 (GPU parallel); Training epochs: 100 epochs, early stopping policy (patience=10, termination if the F1 score on the validation set does not improve for 10 consecutive epochs); Data loading: Multi-threaded preprocessing (image decoding, resize, normalize) is implemented using the tf.data.Dataset API, and the prefetch buffer size is twice the training batch size.

[0089] The following is an exemplary description of the "data preprocessing" step involved in this experimental example.

[0090] As a prerequisite for the technical solution, the preprocessing steps ensure the consistency and quality of the input data. Specifically, this includes: standardizing the hand-drawn images to a uniform size of 512×512 pixels; using a median filtering algorithm to remove image noise, preserving edge information while suppressing salt-and-pepper noise; and enhancing local contrast through contrast-limited adaptive histogram equalization (CLAHE) to improve the robustness of subsequent feature extraction.

[0091] The following is an illustrative example of the "model training" step involved in this experimental example.

[0092] This application is based on the emotional expressive characteristics of hand-drawn images, focusing on color, brushstrokes, and composition. Figure 3 Construct a feature system from multiple dimensions.

[0093] The first step in model training, after processing the data, is to extract multi-dimensional features and quantify the emotional characteristics. The core feature set includes three categories: color temperature features, which construct a 4-dimensional feature vector from three aspects: hue trend, warm / cool distribution, and saturation. = This feature dynamically weights emotionally salient areas through an improved Itti-Koch visual attention mechanism, thus more accurately reflecting the overall emotional tendency of the image. Brushstroke dynamics: By analyzing changes in brushstroke width and curvature frequency, dynamic emotional fluctuations during the drawing process are quantified, forming a 64-dimensional feature vector. Among these, the brushstroke force fluctuation value... The degree of pressure change is reflected by the average absolute value of the difference in stroke width between adjacent sampling points. Curvature frequency The turning frequency of brushstrokes is characterized by calculating the percentage of high-curvature points in the total sampling length. Spatial distribution characteristics of the composition: A 60-dimensional spatial distribution vector is constructed by calculating the effective pixel percentage of each region through multi-scale image segmentation (4×4, 4×5, 6×4). This feature is based on the theory of information entropy, which preserves complete spatial layout information through original distribution data to reflect the compositional complexity and emotional tension of the image.

[0094] The second step in model training involves fusing the features obtained in this application: cross-attention dual-path encoding. First, a cross-attention fusion layer is introduced. As a core component of multimodal feature fusion, this layer achieves deep association between color, brushstroke, and composition features through a dual-path attention mechanism, providing structured input for subsequent Transformer encoding. This layer design follows a three-order processing logic of "dimensional unification - modal association - feature encoding," ensuring that multi-source features form a collaborative representation in the semantic space.

[0095] A cross-attention fusion encoder is used as the core transformation module to encode 128-dimensional multimodal features (4-dimensional color temperature + 64-dimensional brushstroke dynamics + 60-dimensional composition entropy) into a 512-dimensional sentiment feature vector. This encoder innovatively designs a dual-path attention mechanism, incorporating two modeling types: (A) Color-Brushstroke Path: A self-attention layer captures the sentiment correlation between color temperature and brushstroke force fluctuations, with the weight matrix initialized based on pre-trained sentiment word vectors; (B) Brushstroke-Composition Path: A cross-attention mechanism is used to model the spatial distribution relationship between brushstroke curvature frequency and composition features, with attention scores normalized using a softmax function.

[0096] The following details the design principle of this dual-path attention mechanism. The cross-attention fusion layer includes two parallel processing paths: color-brushstroke and brushstroke-composition. Through linear transformation, it achieves dimensional unification and modal interaction of heterogeneous features.

[0097] (A) Color-Stroke Attention Path. Addressing the modal differences between color features (4D) and stroke features (64D), this path employs a query-key / value architecture to establish a connection. Color features are first processed through a linear transformation matrix. Mapping to a 64D semantic space generates a query vector; since the brushstroke features are already in the 64D space, they directly participate in attention calculation as key and value vectors. This design elevates the low-dimensional color features (4D) to the same dimensional space as the brushstroke features, laying the foundation for cross-modal attention weight calculation.

[0098] (B) Brushstroke-Composition Attention Path. The original dimension of the composition features is 60D, which needs to be mapped to a 64D space through linear transformation to match the dimensionality requirements of the attention mechanism. In this path, the brushstroke features serve as the query vector, and the dimensionally expanded composition features (64D) serve as the key and value vectors. The attention mechanism captures the semantic relationship between brushstroke dynamics and compositional structure. The parallel processing of the two paths enables the establishment of a relationship between color sentiment, brushstroke motion features, and compositional spatial layout in a unified dimensional space.

[0099] For the calculation of attention weights for both paths, the scaling dot product attention mechanism is used to calculate the modal association weights. The specific process is as follows: the transpose of the query vector and the key vector is multiplied by a matrix, and the result is divided by the square root of the dimension of the key vector. To mitigate the vanishing gradient problem, the gradient is normalized using the softmax function and then multiplied by the value vector to finally output the fused features. Through this calculation, the model can automatically learn the contribution weights of different dimensional features to emotional expression, strengthening the representational strength of key features.

[0100] The third step in model training involves feature concatenation and Transformer encoding. The output fused features from the two paths (64D each) are concatenated with the original composition features after dimensionality unification, forming a 128D multi-feature fusion vector. This vector is then input into a 6-layer Transformer encoder for deep encoding, with each layer containing an 8-head self-attention mechanism and a feedforward neural network (feedforward dimension 2048). Through multi-layer self-attention and residual connections, the model gradually aggregates local features and global semantics, ultimately encoding the 128D input features into a 512D global sentiment feature vector, providing a high-dimensional, structured semantic representation for subsequent sentiment classification tasks.

[0101] This design achieves dynamic association of multi-dimensional features through a cross-attention mechanism. Combined with the sequence modeling capabilities of Transformer, it effectively captures the emotional synergy of color, brushstrokes and composition in hand-drawn images, providing a robust feature foundation for emotion recognition tasks.

[0102] The fourth step in model training involves using ensemble learning as the decision-making unit and employing a stacking strategy to fuse the outputs of a Convolutional Neural Network (CNN) and a Transformer base model. The specific structure of the base model layer is as follows: CNN base model: Utilizing an improved VGG16 architecture (removing the last three fully connected layers and adding a global average pooling layer and a new fully connected layer), it focuses on extracting local brushstroke textures and refined patterns from the fused sentiment features. Its final layer output serves as the sentiment probability. Transformer base model: Employs the Vision Transformer (ViT-Base) architecture (12-layer encoder, 12-head attention), leveraging its powerful global attention mechanism to capture long-range dependencies between multi-dimensional features such as color, brushstrokes, and composition, outputting sentiment probabilities. The meta-learner integrates the output probabilities of two base models through a logistic regression model. The final output is the sentiment classification probability. It enables the identification of both positive and negative emotions.

[0103] This technical solution significantly improves the accuracy and generalization ability of hand-drawn image emotion recognition by organically fusing multiple features and using complementary decision-making of integrated models, covering the entire technical process from preprocessing to classification output.

[0104] The training parameters of the ensemble classification model need to be refined at both the base model (CNN and Transformer) and meta-learner levels, while a global training strategy ensures model convergence and performance optimization. The following explains the parameter settings of each component and the overall training process: The CNN model uses an improved VGG16 architecture, optimized for extracting local brushstroke features from hand-drawn images: the input size is fixed at 224×224 pixels, the original fully connected layers are removed and global average pooling is added, and a Dropout layer (rate=0.3) is introduced into the network to suppress overfitting. This configuration focuses on extracting local brushstroke texture features, effectively controlling model complexity while preserving key emotional expression information. The Transformer model is built on the ViT-Base architecture and processes 512D emotional feature vectors: it includes a 12-layer encoder and a 12-head self-attention mechanism. By projecting the 512D feature vectors into semantic sequences, it uses a multi-head attention mechanism to model deep semantic relationships between multi-dimensional features such as color, brushstrokes, and composition. The optimizer used is AdamW, with specific parameters set as β1=0.9 (momentum parameter) and β2=0.999 (adaptive learning rate parameter), and weight decay. (Regularization strength) This configuration accelerates convergence while reducing the risk of weight oscillation.

[0105] The following is an exemplary description of the "experimental results analysis" step involved in this experimental example.

[0106] In terms of experimental results analysis, the effectiveness of the technical solution was verified through feature combination comparison, model comparison, and ablation experiments. Firstly, the feature combination comparison experiment results show that fusing color, brushstrokes, and structure... Figure 3The average accuracy of multi-dimensional features reached 89.2%, significantly outperforming single color features (79.5%), brushstroke features (83.8%), and composition features (72.1%), as shown in Table 1. This indicates that multi-dimensional features can more comprehensively capture the emotional information in hand-drawn works, with each feature dimension forming a complementary effect. Secondly, the comparison of classification performance between different models shows that the proposed ensemble model (CNN+Transformer) outperforms single models in all metrics, achieving an accuracy of 93.5%, precision of 0.931, recall of 0.925, and F1 score of 0.928. This represents a 6.3% improvement compared to the CNN model (87.2%) and a 3.4% improvement compared to the Transformer model (90.1%), validating the superiority of the Stacking ensemble strategy. Ablation experiments further showed that the cross-attention fusion module had the greatest impact on model performance, with accuracy decreasing by 4.8% after removal, mainly affecting the recognition of hand-drawn works; accuracy decreased by 3.9% after the Transformer encoding layer was removed, indicating the importance of global feature modeling for complex compositions; and accuracy decreased by 2.1% after the SMOTE sampling strategy (to solve class imbalance), mainly affecting the recognition stability of minority classes (negative emotions).

[0107] Table 1. Results of the comparative experiment on feature combinations

[0108] The practical application effect of this application is further verified through a typical case analysis: taking a hand-drawn artwork labeled as "negative" by experts as an example. The feature extraction results are as follows: Overall warmth / coolness value Ta = 45 (cool tone), warm color area accounts for 18%, cool color area accounts for 72%, weighted average saturation 0.35 (low saturation). This combination of features clearly indicates a negative, depressed emotional tendency. Pen pressure fluctuation value P=0.65 (high pressure fluctuation, representing emotional excitement), curvature frequency... (High-frequency transitions represent tension and anxiety). The values ​​of each dimension of its 60-dimensional spatial distribution vector are relatively small and evenly distributed. From an information theory perspective, the theoretical entropy value corresponding to this distribution pattern is high, indicating that the layout of elements in the image is scattered and disordered, which is highly correlated with anxiety. During the recognition process, the CNN base model (weight 0.52) focuses on the high-frequency pressure changes in dense brushstroke areas; the Transformer base model (weight 0.48) comprehensively analyzes the color distribution dominated by cool colors and the highly complex spatial layout. The meta-learner integrates the outputs of both and calculates the probability of negative emotion as 0.916. According to the binary classification decision rule (with a threshold of 0.5), the work was successfully identified as having "negative" emotion, which is completely consistent with the expert annotation results, verifying the adaptability of this application to works with complex styles and the effectiveness of feature fusion.

[0109] Therefore, based on the above experimental examples, it can be seen that on a dataset containing 1200 hand-drawn works (800 public works + 400 self-built works), the sentiment classification accuracy reached 90.5%, with an F1 score of 0.892, which is 5.9% higher than a single CNN model (84.6%) and 3.3% higher than a single Transformer model (87.2%), significantly improving the recognition accuracy.

[0110] The processing time for a single image is 2.8 seconds (including preprocessing, feature extraction, and classification), and it supports full-process management of experiments (experiment creation, data collection, and analysis report export). It can meet the batch data processing and visualization needs of emotion quantification analysis scenarios, and achieves optimization of efficiency and scenario adaptability.

[0111] The following describes the hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning provided in this application. The hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning described below can be referred to in correspondence with the hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning described above.

[0112] Figure 6 This is a schematic diagram of the structure of the hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning provided in the embodiments of this application, as shown below. Figure 6 As shown, the system includes: an image preprocessing module 10, a multi-dimensional feature extraction module 20, a cross-attention fusion encoding module 30, and a sentiment classification module 40. Among them: Image preprocessing module 10 is used to perform image preprocessing based on the hand-drawn image to obtain the preprocessed hand-drawn image. Image preprocessing is used to remove noise from the image and enhance the image. The multi-dimensional feature extraction module 20 is used to extract multi-dimensional features based on the preprocessed hand-drawn image, and obtain color temperature features, brushstroke dynamics features and composition space distribution features. The cross-attention fusion encoding module 30 is used to perform cross-attention fusion encoding based on color temperature features, brush stroke dynamics features, and compositional spatial distribution features to obtain emotional feature vectors. The cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and to fuse and encode the extracted attention vectors. Each attention path is used to process two of the color temperature features, brush stroke dynamics features, and compositional spatial distribution features through the cross-attention mechanism. The sentiment classification module 40 is used to predict the sentiment classification of hand-drawn images based on sentiment feature vectors through ensemble learning.

[0113] The hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning provided in this application is applicable to the entire process management of emotion quantification analysis.

[0114] Figure 7 This is a schematic diagram of the system layered architecture provided in the embodiments of this application, such as... Figure 7 As shown, the system of this application can adopt a layered design, including: a data acquisition layer, a preprocessing layer, a feature extraction layer, an emotion recognition layer, and a visualization layer.

[0115] Figures 8-11 This is a schematic diagram of a hand-drawn emotion recognition feature visualization example provided in the embodiments of this application, such as... Figures 8-11 As shown, based on the Stacking ensemble strategy (CNN weight 0.52, Transformer weight 0.48), the sentiment tendency and confidence scores are given (e.g., Figure 8 The confidence level for positive sentiment was 87.5%, highlighting key influencing factors such as warm color tone and low curvature frequency; for example... Figure 9 As shown, a visualization interface for color temperature characteristics displays the temperature values ​​calculated by the color temperature formula and the proportions of warm and cool tones (65% warm, 35% cool), combined with visual attention weighting to highlight areas of significant emotion; for example... Figure 10 As shown, the visualization interface of brushstroke dynamics (or brush force dynamics) features displays the parameters calculated by the fluctuation and curvature frequency formulas (such as brush force fluctuation of 12.8 and curvature frequency of 0.32) and the spatial grid distribution, reflecting the fluctuations in the drawing emotion; for example... Figure 11 As shown, the interface displays the entropy value calculated based on the composition entropy formula through a visualization interface of the spatial distribution characteristics of the composition, linking the complexity of the image distribution with emotional tension. The interface transforms the complex algorithm into a visualization result using the logic of "conclusion-feature-basis".

[0116] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0117] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0118] Based on the methods in the above embodiments, this application provides an electronic device. Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 12As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0119] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0120] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0121] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0122] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0123] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0124] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0125] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0126] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning, characterized in that, include: Based on the hand-drawn image, image preprocessing is performed to obtain the preprocessed hand-drawn image. Image preprocessing is used to remove noise from the image and enhance the image. Based on the preprocessed hand-drawn image, multi-dimensional feature extraction is performed to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features. Based on color temperature features, brushstroke dynamics features, and compositional spatial distribution features, cross-attention fusion encoding is performed to obtain emotional feature vectors. Cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and to fuse and encode the extracted attention vectors. Each attention path is used to process two of the color temperature features, brushstroke dynamics features, and compositional spatial distribution features through a cross-attention mechanism. Based on sentiment feature vectors, the sentiment classification of hand-drawn images is predicted through ensemble learning. The preprocessed hand-drawn image undergoes multi-dimensional feature extraction to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features, including: Based on the preprocessed hand-drawn image, the color temperature characteristics are obtained by statistically analyzing the difference between the red and green channels and the saturation value of the pixels using a weighted average method. Based on a preset grid, the preprocessed hand-drawn image is divided into multiple cells, and the brushstroke force fluctuation and curvature are statistically analyzed for each cell to obtain brushstroke dynamics features. The preprocessed hand-drawn image is divided into multiple regions according to a specified regional scale, and the effective pixel ratio of each region is calculated. Based on the effective pixel ratio of each region at multiple regional scales, the spatial distribution characteristics of the composition are obtained. The process of statistically analyzing the pen pressure fluctuation and curvature of each cell includes calculating the pen pressure fluctuation value and curvature frequency for each cell using the following formulas: ； ； in, This indicates the fluctuation value of pen strokes. This represents the number of stroke sampling points in the cell. For the cell's first The stroke width of each stroke sampling point Indicates the curvature frequency. This represents the number of pen stroke sampling points in the cell whose curvature value is greater than the preset curvature. The dynamic characteristics of the stroke include the stroke force fluctuation value and curvature frequency corresponding to each cell; Multiple attention paths include color-brushstroke paths and brushstroke-composition paths. The process involves cross-attention fusion encoding based on color temperature features, brushstroke dynamics features, and compositional spatial distribution features to obtain an emotional feature vector, including: Input color temperature features and brush stroke dynamics features into the color-brush path, and obtain the attention vector output by the color-brush path; Input the brushstroke dynamics features and composition space distribution features into the brushstroke-composition path, and obtain the attention vector output by the brushstroke-composition path; The attention vectors corresponding to the color-brushstroke path and the attention vectors corresponding to the brushstroke-composition path are concatenated to obtain the concatenated features. Based on the concatenated features, a multi-layer Transformer encoder is used to encode and obtain the sentiment feature vector. Among them, the color-brushstroke path is used to take the color temperature feature as the query vector and the brushstroke dynamics feature as the key vector and value vector, and extract the attention vector corresponding to this path by calculating the attention. The brushstroke-composition path uses brushstroke dynamics features as query vectors and composition space distribution features as key and value vectors. By calculating attention, the attention vector corresponding to this path is extracted.

2. The hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning according to claim 1, characterized in that, The preprocessed hand-drawn image is used to statistically analyze the difference between the red and green channels and the saturation value of each pixel using a weighted average method to obtain color temperature features, including obtaining the color temperature features using the following formula: ； ； ； ； in, Indicates the overall temperature value. Indicates the proportion of warm colors in the area. Indicates the percentage of cool colors in the area. Indicates average saturation; Indicates the first The red channel value of each pixel Indicates the first The green channel value of each pixel Indicates the first The saturation value of each pixel in the HSV color space Indicates the first Visual attention weight per pixel; Color temperature characteristics include , , and .

3. The hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning according to claim 2, characterized in that, Based on the saliency graph The saliency map is obtained by taking the values ​​of each pixel and using the Itti-Koch model. In the linear superposition stage of synthesizing saliency maps using the Itti-Koch model, the weight corresponding to brightness contrast is greater than the weight corresponding to color contrast, and the weight corresponding to color contrast is greater than the weight corresponding to orientation contrast.

4. The hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning according to claim 1, characterized in that, The method of predicting the emotion classification of hand-drawn images based on emotion feature vectors through ensemble learning includes: Based on sentiment feature vectors, the probability of sentiment classification is predicted through multiple base models; Based on the sentiment classification probabilities predicted by each base model, an ensemble decision is made through a meta-learner to obtain the sentiment classification of the hand-drawn image. Several base models include: convolutional neural networks and Transformer.

5. The hand-drawn emotion recognition method based on multi-feature fusion and ensemble learning according to claim 1, characterized in that, The image preprocessing includes: median filtering and contrast-limited adaptive histogram equalization (CLAHE).

6. A hand-drawn emotion recognition system based on multi-feature fusion and ensemble learning, characterized in that, include: The image preprocessing module is used to perform image preprocessing based on the hand-drawn image, and obtain the preprocessed hand-drawn image. Image preprocessing is used to remove noise from the image and enhance the image. The multi-dimensional feature extraction module is used to extract multi-dimensional features based on the preprocessed hand-drawn image, and obtain color temperature features, brushstroke dynamics features and compositional spatial distribution features. The cross-attention fusion encoding module is used to perform cross-attention fusion encoding based on color temperature features, brush stroke dynamics features, and compositional spatial distribution features to obtain emotional feature vectors. The cross-attention fusion encoding is used to extract attention vectors through multiple attention paths and to fuse and encode the extracted attention vectors. Each attention path is used to process two of the color temperature features, brush stroke dynamics features, and compositional spatial distribution features through the cross-attention mechanism. The sentiment classification module is used to predict the sentiment classification of hand-drawn images based on sentiment feature vectors through ensemble learning; The preprocessed hand-drawn image undergoes multi-dimensional feature extraction to obtain color temperature features, brushstroke dynamics features, and compositional spatial distribution features, including: Based on the preprocessed hand-drawn image, the color temperature characteristics are obtained by statistically analyzing the difference between the red and green channels and the saturation value of the pixels using a weighted average method. Based on a preset grid, the preprocessed hand-drawn image is divided into multiple cells, and the brushstroke force fluctuation and curvature are statistically analyzed for each cell to obtain brushstroke dynamics features. The preprocessed hand-drawn image is divided into multiple regions according to a specified regional scale, and the effective pixel ratio of each region is calculated. Based on the effective pixel ratio of each region at multiple regional scales, the spatial distribution characteristics of the composition are obtained. The process of statistically analyzing the pen pressure fluctuation and curvature of each cell includes calculating the pen pressure fluctuation value and curvature frequency for each cell using the following formulas: ； ； in, This indicates the fluctuation value of pen strokes. This represents the number of stroke sampling points in the cell. For the cell's first The stroke width of each stroke sampling point Indicates the curvature frequency. This represents the number of pen stroke sampling points in the cell whose curvature value is greater than the preset curvature. The dynamic characteristics of the stroke include the stroke force fluctuation value and curvature frequency corresponding to each cell; Multiple attention paths include color-brushstroke paths and brushstroke-composition paths. The process involves cross-attention fusion encoding based on color temperature features, brushstroke dynamics features, and compositional spatial distribution features to obtain an emotional feature vector, including: Input color temperature features and brush stroke dynamics features into the color-brush path, and obtain the attention vector output by the color-brush path; Input the brushstroke dynamics features and composition space distribution features into the brushstroke-composition path, and obtain the attention vector output by the brushstroke-composition path; The attention vectors corresponding to the color-brushstroke path and the attention vectors corresponding to the brushstroke-composition path are concatenated to obtain the concatenated features. Based on the concatenated features, a multi-layer Transformer encoder is used to encode and obtain the sentiment feature vector. Among them, the color-brushstroke path is used to take the color temperature feature as the query vector and the brushstroke dynamics feature as the key vector and value vector, and extract the attention vector corresponding to this path by calculating the attention. The brushstroke-composition path uses brushstroke dynamics features as query vectors and composition space distribution features as key and value vectors. By calculating attention, the attention vector corresponding to this path is extracted.

7. An electronic device, characterized in that, include: Memory and one or more processors; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; The one or more processors invoke the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-5.

Images