Intelligent photographic theme matching algorithm based on emotion recognition and teaching application

Abstract

The invention discloses a photography theme intelligent matching algorithm based on emotion recognition and a teaching application system. According to the algorithm, emotion features of photography works are extracted through deep learning and computer vision technologies, emotion is classified and quantified, the emotion features are matched with a database containing rich photography themes and emotion features thereof, and appropriate photography themes are recommended for users. The teaching application system provides sentiment analysis, personalized theme recommendation, teaching resource pushing and other services for students based on the algorithm, assists teachers in teaching evaluation and feedback, optimizes the teaching process, improves the photography teaching quality, and promotes the improvement of the photography creation ability and the sentiment expression level of the students.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of photography and sentiment analysis, and particularly relates to a photography theme intelligent matching algorithm based on sentiment recognition and its application system in photography teaching, which can accurately match appropriate photography themes according to the emotions expressed by photography works, and assist in photography teaching activities. BACKGROUND

[0002] In existing photography practice, photographers often rely on personal experience, intuition or reference to others' works when determining photography themes, lacking a scientific, accurate and efficient emotion and theme matching method. This traditional theme selection method may result in insufficient expression of emotions or inconsistency with the intended theme, affecting the artistic effect and appeal of the work. In the field of photography teaching, teachers usually adopt a unified teaching mode and theme arrangement, which is difficult to meet the emotional expression needs and individualized development of different students. Students may have difficulty expressing emotions and cannot find a suitable creative direction, thereby limiting their photography creation ability and artistic accomplishment. Although there are some image recognition and analysis technologies, the application of intelligent matching of photography work emotions and themes serving teaching has not been fully developed and effectively solved. SUMMARY

[0003] (I) Photography theme intelligent matching algorithm based on sentiment recognition

[0004] The emotion feature extraction module deep learning algorithm and computer vision technology are the key to realizing emotion feature extraction. Specifically, the present application uses a convolutional neural network (CNN) to extract features from images. This network is trained on a large number of image data labeled with emotion features, and can automatically learn the visual patterns and feature representations related to emotions in images. For example, by analyzing the color distribution of an image, CNN can identify that images with bright and vibrant colors are more likely to be associated with positive emotions such as happiness and vitality, while images with dark and low saturation colors may suggest negative emotions such as sadness and depression.

[0005] The emotion classification and quantification module uses a multi-label classification approach to classify image emotions into multiple dimensions, such as happiness, sadness, anger, fear, surprise, disgust, and more specific emotion categories such as tranquility, romance, and mystery. Each emotion dimension corresponds to a 0-1 score. To achieve this goal, the present application uses a multi-task learning method in deep learning to simultaneously predict scores for multiple emotion dimensions. By training on a large number of image data labeled with emotion scores, the model can learn the manifestations of different emotion dimensions in image features, thereby accurately classifying and quantifying the emotions of new images.

[0006] The photography theme database construction module collects photography theme samples that not only include common natural scenery, urban buildings, and portrait photos, but also cover more specific and unique themes such as architectural details, street culture moments, and natural ecological microcosms. For the emotional feature vector labeling of each theme, the present application adopts a combination of expert labeling and crowdsourcing labeling. First, professional photographers and photography critics label the theme samples emotionally to provide authoritative emotional feature vectors. Then, the labeling opinions of a large number of ordinary photography enthusiasts are collected through a crowdsourcing platform to supplement and verify the expert labeling, so that the emotional feature vectors are closer to the public aesthetic and emotional cognition.

[0007] In the theme matching algorithm module, when calculating the similarity, the present application not only considers the similarity of the emotional feature vectors, but also takes other related information of the theme (such as the popularity and difficulty level of the theme) as auxiliary reference factors for weighted fusion calculation of the comprehensive score. For example, for a case where an emotional feature is highly similar to multiple themes but the difficulty difference is large, the comprehensive score will be appropriately tilted towards the theme with moderate difficulty to better meet the actual creation needs of users. The themes are ranked according to the comprehensive score, and recommended to the user according to certain rules (such as returning the top K themes).

[0008] (II) Teaching application system of photography theme intelligent matching algorithm based on emotion recognition

[0009] After the students submit their photography works, the system analyzes them through the emotion recognition algorithm. In addition to generating an emotional analysis report, it also conducts comparative analysis with the typical emotional expression targets of the students' learning stage. For example, in the early stage of learning portrait photography emotional expression, if a student's work is lower than the target value in terms of emotional intensity and accuracy, the system will prompt the student to strengthen the practice of related emotional expression skills. At the same time, the system will compare the student's emotional analysis results with the emotional features of outstanding works in the same stage, and display the differences in a visual way to help students more intuitively understand their shortcomings.

[0010] The personalized theme recommendation subsystem considers multiple dimensions of information such as the students' emotional analysis results, learning progress, and skill level. For students with single emotional expression, the system will recommend themes that can guide them to expand their emotional expression range; for students with high skill level, the system will recommend more challenging themes that are consistent with their emotional expression characteristics. The theme recommendation results not only include basic information about the theme, but also provide theme-related creative idea guidance, common emotional expression skill prompts, and other content to help students better understand and grasp the theme creation direction.

[0011] The teaching resource pushing subsystem automatically pushes relevant photography courses, skill articles, excellent work cases, and other teaching resources according to the recommended photography theme. The present application plans a personalized learning path for students according to their learning progress and recommended theme by constructing an intelligent learning path planning function. For example, for a student who is recommended the theme of “natural landscape photography”, the system will push teaching resources starting from basic landscape photography composition skill courses, gradually to advanced light and shadow use skills, post-processing methods, and other teaching resources, and will push them to the student according to a certain order and time arrangement, guiding the student to learn step by step.

[0012] The teaching evaluation and feedback subsystem allows teachers to view the students' photography works, emotional analysis results, and theme matching conditions, and to make more accurate evaluation and guidance to the students. Teachers can make comments and scores on the students' works in the system and provide targeted guidance. The system will automatically collect the feedback of the teachers, combine the usage data of the students (such as work submission frequency, acceptance of recommended theme, browsing of teaching resources, etc.), and use data mining and machine learning algorithms to optimize and adjust the emotional recognition algorithm and theme recommendation strategy. For example, if multiple teachers give similar adjustment suggestions on a certain theme recommendation result, the system will automatically optimize the recommendation algorithm parameters of the theme to improve the accuracy and effectiveness of the recommendation.

[0013] Technical effects:

[0014] Teaching closed-loop optimization model: a four-order teaching model of “emotional analysis-theme adaptation-resource navigation-evaluation feedback” is proposed, and the adaptive evolution of the recommendation strategy is realized through Bayesian optimization.

[0015] Dynamic weight recommendation algorithm: innovatively, the theme difficulty, popularity, and other scene factors are included in the similarity calculation, and through online learning of the alpha / beta / gamma parameters, the recommendation result is more in line with the teaching practice needs, and the accuracy is improved by 29% compared with the traditional cosine similarity algorithm. DETAILED DESCRIPTION

[0016] The present application will be further described in detail below in combination with specific embodiments.

[0017] In the preprocessing stage, in addition to adjusting the image size and normalizing the pixel value, the implementation of the emotional feature extraction module also adopts image enhancement techniques such as adaptive contrast enhancement to improve image quality and make subsequent emotional feature extraction more accurate. For color feature extraction, in addition to color histogram and hue distribution, a color space conversion-based analysis method is introduced. The image is converted from RGB color space to Lab color space to more accurately analyze the brightness, hue, and saturation information of the color, as well as a composition feature analysis method based on local feature extraction. At the same time, the Scale-Invariant Feature Transform (SIFT) algorithm is used to detect key points and extract descriptors from the image, identify local composition elements such as lines, shapes, and textures in the image, and analyze their distribution and combination in the picture to further enrich the dimension of composition features.

[0018] Explicit convolutional neural network architecture: ResNet-50 is used as the base network, and the ImageNet pre-trained model is fine-tuned for emotional features. Three new fully connected layers (with 1024, 512, and 128 neurons respectively) are added for emotional feature mapping.

[0019] Micro-expression analysis implementation: The Dlib library is integrated for face key point detection (68 landmark points), the AU (Action Unit) coding system is used to analyze eye and mouth muscle movement features, and the LightGBM model is used to map the AU combination to emotions.

[0020] Composition feature quantization method: The GIST visual semantic feature extraction algorithm is introduced, the image is divided into a 4x4 grid, the histogram of oriented gradients (HOG) feature of each grid is calculated, and a 1024-dimensional composition feature vector is constructed.

[0021] Implementation of the emotional classification and quantization module: In the data preprocessing stage, strict quality control is performed on the labeled data, and samples with inaccurate or ambiguous labels are removed. In the training process, cross-validation is used to divide the dataset into multiple subsets, and training and validation are performed in turn to ensure the stability and generalization ability of the model. At the same time, in order to improve the classification and quantization accuracy of different emotional dimensions, class imbalance processing techniques are used. For some emotional dimensions with fewer labeled samples, more positive samples are generated through data augmentation methods such as image rotation, translation, and cropping to balance the class distribution of the dataset.

[0022] Emotional dimension expansion: In addition to the original 8 basic emotions, 3 common artistic photography emotions, "nostalgia", "shock", and "healing", are added to form an 11-dimensional emotional space.

[0023] Quantitative standard specification: Develop a "Photography Emotion Quantification Manual" to define visual feature mapping rules for each emotion dimension (e.g., "Romance" corresponds to warm color tone proportion > 60%, Gaussian blur radius > 3.0px soft focus effect).

[0024] The implementation of the photography theme database construction module focuses on the diversity and representativeness of the samples when collecting photography theme samples. It collects widely from different photography styles, shooting scenes, cultural backgrounds, etc. For the expert annotation process, strict annotation specifications and procedures are established to ensure the consistency and accuracy of expert annotation. After expert annotation, the annotation results are checked and corrected through internal audit and cross-validation mechanism. In the crowd annotation stage, the qualification of the annotators is audited and trained to improve the annotation quality. Incentive mechanism is adopted to reward the annotators with high annotation accuracy to ensure the smooth completion of the crowd annotation task.

[0025] Annotation conflict resolution: Establish a three-level audit mechanism - crowd annotation (threshold > 70% consistency) → expert review → cross-validation (Kappa coefficient > 0.85), use voting weighted algorithm (expert weight 70% + public weight 30%) to determine the final annotation for controversial samples.

[0026] Theme feature vector structure: Each theme contains (emotion feature vector 11 dimensions + style label 5 dimensions + difficulty coefficient 1 dimension + popular index 1 dimension), where the difficulty coefficient is divided into 5 levels (L1-L5) by expert evaluation, and the popular index is calculated based on the TF-IDF value of the works in the past year on platforms such as Meitu / 500px.

[0027] The implementation of the theme matching algorithm module normalizes the emotion feature vector before calculating the cosine similarity to eliminate the influence of different emotion dimension scales, making the similarity calculation more accurate. At the same time, considering the different importance of different emotion dimensions in theme matching, weight coefficients are introduced to weight and adjust the similarity calculation. The determination of weight coefficients is obtained by analyzing the creative habits and theme selection preferences of a large number of photography experts, and combining feature importance evaluation methods in machine learning. For auxiliary reference factors such as popularity and difficulty level, establish the corresponding evaluation index system. For example, the popularity can be determined by analyzing the attention, number of works, etc. of the theme in the photography community; the difficulty level is evaluated by experts according to the shooting skills, post-processing difficulty, etc. involved in the theme. After quantifying these auxiliary reference factors, they are combined with the emotional similarity to form a comprehensive score calculation model. The parameters of the model are optimized through machine learning methods to improve the accuracy of theme recommendation.

[0028] Similarity weighting formula:

[0029] S = a * cosine(Vimg, Vtheme) + b * (1 - diff(difficulty)) + g * popularity

[0030] (S: Subjective comprehensive matching score (range 0-1)

[0031] Vimg: 11-dimensional emotional feature vector of the image to be matched (e.g. [happy, sad,..., healing])

[0032] Vtheme: Emotional feature vector in the photography theme database

[0033] cosine(Vimg, Vtheme): Cosine similarity of emotional feature vectors (measure of directional consistency)

[0034] diff(difficulty): Normalized difference between image emotional matching degree and theme difficulty

[0035] popularity: Theme popularity (0-1 standardized value)

[0036] a = 0.6, b = 0.3, g = 0.1: Weight coefficients )

[0038] Dynamic weight optimization: Use Bayesian optimization algorithm to take the actual theme adoption rate of users as the objective function, and automatically update a / b / g parameters every 1000 user feedbacks collected.

[0039] The implementation of each subsystem of the teaching application system is as follows: The student emotion analysis subsystem provides multiple visualization methods for students to choose from when displaying emotion analysis reports. In addition to radar charts and column charts, heat maps and other visualization forms are added to more intuitively display the distribution of emotions in the image. For example, a heat map can show the emotional intensity of different areas in the image, helping students understand which elements in the picture play a key role in emotional expression. The personalized theme recommendation subsystem will further filter and sort the recommended themes based on the student's creative habits and interest preferences. If the student often takes black and white portrait works, the system will preferentially recommend themes related to black and white portraits or with similar emotional expression styles. The teaching resource pushing subsystem will push teaching resources based on the student's reading duration, collection status, and other behavior data to determine the student's interest in the resources and dynamically adjust the pushing strategy. For teaching resources that students are interested in, increase the pushing frequency and preferentially recommend associated content; for content that students are not interested in, reduce the pushing weight. The teaching evaluation and feedback subsystem regularly collects feedback from teachers and classifies, organizes, and analyzes the feedback. For some common problems, such as the lack of accuracy of the emotion analysis algorithm in specific scenarios, the deviation between the theme recommendation results and the student's actual level is large, etc., a special technical team is organized to conduct concentrated research and optimization. The personalized theme recommendation subsystem calls the matching algorithm of the present application, and the input parameters include the emotion vector, the learning stage label, and the historical work theme distribution.

[0040] (II) Detailed implementation examples

[0041] Example 1: Portrait photography emotion matching scene

[0042] Input image: indoor portrait, warm yellow tone, model smiling (AU6+AU12 activated), background blurred

[0043] Feature extraction results:

[0044] Color feature: L* = 65, a* = 12, b* = 18 in Lab space (warm tone)

[0045] Micro-expression: smile intensity 0.82 (AU6 = 0.78, AU12 = 0.86)

[0046] Composition feature: center composition, 60% of the figure, Gaussian blur radius 4.5px

[0047] Emotion quantification results: happiness 0.85, romance 0.62, tranquility 0.58

[0048] Database matching: cosine similarity 0.91 with "Warm Family Portrait" theme (emotion vector [0.9, 0.6, 0.6]), combined with difficulty coefficient L2 (moderate), comprehensive score 0.89, recommended as the top theme.

[0049] Example 2: Teaching Application Scenario

[0050] Student Stage: Elementary Portrait Class (Week 3, Target: Mastering "Joy" Emotional Expression)

[0051] Work Analysis: Emotional Intensity 0.45 (0.65 below class average), Insufficient Color Saturation (S = 0.31 in HSV), Incomplete Facial Expression Capture (Only AU12 Detected, AU6 Not Identified)

[0052] System Feedback:

[0053] Recommended Theme: "Children's Smile Snapshots" (Difficulty L1, Emotional Matching 0.78)

[0054] Resource Push: "Portrait Photography Color Adjustment Tutorial" "5 Tips for Capturing Micro Expressions"

[0055] Training Suggestions: Set Specialized Exercises (Take 20 Smile Photos Daily, System Automatically Labels AU Activation)

[0056] The above examples are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the above examples, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the above examples, or make equivalent substitutions for part of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent photographic theme matching algorithm based on emotion recognition, characterized in that: The following steps are involved: Extract emotional features from photographic images. The extracted features include visual elements such as image color, composition, light and shadow, facial expressions, and body posture. Based on the extracted emotional features, the emotions expressed by the image are classified and quantified into scores on multiple emotional dimensions; Construct a photography theme database containing various photography themes and their typical emotional feature vectors; Calculate the similarity between the emotional feature vector of the photographic image to be matched and the emotional feature vectors of each theme in the photographic theme database; Sort photographic themes by similarity and recommend the ones that best match the emotional requirements of the image to be matched.

2. The photography theme intelligent matching algorithm based on emotion recognition according to claim 1 is characterized in that: The emotional feature extraction adopts deep learning algorithms and computer vision technology, including using convolutional neural networks to extract features from images and perform micro-expression analysis on human expressions.

3. The photography theme intelligent matching algorithm based on emotion recognition according to claim 1 is characterized in that: The sentiment classification adopts a machine learning classifier, including a support vector machine, a random forest, etc., and the score range of the sentiment quantification is between 0 and 1.

4. The photography theme intelligent matching algorithm based on emotion recognition according to claim 1 is characterized in that: The similarity calculation method includes cosine similarity, Euclidean distance, etc.

5. A teaching application system for a photography theme intelligent matching algorithm based on emotion recognition, characterized in that: include: Student sentiment analysis subsystem, used to analyze the emotional characteristics of student photographs and generate sentiment analysis reports; Personalized theme recommendation subsystem, which recommends suitable photography themes based on students' sentiment analysis results and learning status; Teaching resource push subsystem pushes relevant teaching resources for recommended photography topics; The teaching evaluation and feedback subsystem is used for teachers to evaluate and guide students and for the system to optimize its own algorithms and strategies.

6. The teaching application system according to claim 5, characterized in that: The student sentiment analysis subsystem performs sentiment analysis on the student's photographic works using the photographic theme intelligent matching algorithm based on sentiment recognition as described in any one of claims 1-4.

7. The teaching application system according to claim 5, characterized in that: The personalized theme recommendation subsystem uses a theme matching algorithm to recommend photography themes based on the student's personal learning profile and sentiment analysis results. The recommendation results include the theme's name, introduction, example works, and matching degree description.

8. The teaching application system according to claim 5, characterized in that: The teaching resources pushed by the teaching resource push subsystem include photography tutorials, skill articles, excellent work cases, etc., and the push methods include online links, attachment downloads or embedded players, etc.

9. The teaching application system according to claim 5, characterized in that: The teaching evaluation and feedback subsystem can collect teachers' feedback and optimize and adjust the emotion recognition algorithm and topic recommendation strategy in combination with students' usage data.