Smile image recognition system

Abstract

The invention provides a smile image recognition system, which comprises a smile display module, and the smile display module comprises a smile acquisition sub-module, a smile recognition sub-module, a smile evaluation sub-module and a display sub-module. Wherein the smile acquisition sub-module is configured to acquire a target object; the smile recognition sub-module is configured to recognize the collected smile image so as to obtain expression features in the smile image; the smile evaluation module is configured to perform scoring test on each smile image according to a preset method; and the display sub-module is configured to provide the scored smile image pair to the display device according to a preset rule. The smiling condition of the student can be displayed, so that the psychological condition of the student is reflected laterally, a school can observe the smiling display condition of the student in the system through the system, psychological counseling can be carried out in a targeted mode, and the psychological health level of the student is promoted to be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interviews, and in particular to a smile image recognition method and system. Background Art

[0002] With the rapid development of the times, primary and secondary school students are under increasing pressure from all sides. Students' mental health problems are more prominent. Psychological problems such as depression, anxiety, and self-harm occur frequently. The incidence of mental illness and emotional disorders continues to rise. It is urgent to offer mental health education courses.

[0003] Face recognition is a biometric technology that identifies people based on their facial features. At present, the face recognition industry chain can be roughly divided into three parts: the basic layer, the technical layer and the application layer. The basic layer is located in the upstream of the industry chain, providing hardware, algorithms and data support for the industry. The technical layer is located in the middle of the industry chain, mainly including related face recognition technology and system integration, including video face recognition, image face recognition and database comparison and inspection technologies. The application layer is located in the downstream of the industry chain, specifically including smart security, smart finance, smart transportation, mobile payment, medical care, government functions and other fields.

[0004] With the development of artificial intelligence and the need for security defense, the face recognition market continues to expand and the technology level continues to improve. The following is the latest status of face recognition research at home and abroad.

[0005] In China, the current research status of face recognition is as follows: face recognition technology based on deep learning, which mainly includes face recognition technology based on convolutional neural network (CNN), face dynamic recognition technology based on recurrent neural network (RNN), etc.; face recognition technology based on three-dimensional face reconstruction, which mainly obtains more face information by establishing a three-dimensional model to achieve the purpose of improving the accuracy and robustness of face recognition; face recognition technology based on multimodal fusion, which mainly improves the accuracy and robustness of face recognition by fusing multiple modal information such as images, videos, and sounds.

[0006] Abroad, the current research status of face recognition is as follows: face recognition technology based on transfer learning, which mainly transfers existing knowledge to new tasks through pre-trained models, thereby improving the accuracy and robustness of face recognition; face recognition technology based on adversarial samples, which mainly attacks the face recognition system by generating adversarial samples, thereby improving the robustness of the face recognition system; face recognition technology based on deep generative models, which mainly generates realistic face images through deep generative models, thereby improving the robustness and accuracy of the face recognition system.

[0007] At present, there are several technologies and patents similar to the present invention in the domestic market.

[0008] The Chinese invention patent application with publication number CN114360012A and name “Interview Smile Evaluation Method and System” pre-trains a face model with a neural network, obtains a picture of the interviewee, and then imports the picture of the interviewee into the model, uses the previously trained model to extract the positions of the facial features from the picture of the interviewee, and then compares the extracted facial features data with the data of the standard model and scores them according to the similarity.

[0009] The Chinese invention patent application with publication number CN106548160A, titled A method for detecting facial smile, mainly uses an algorithm to extract the smile part (mouth) from the acquired image, and then determines whether the face is smiling based on the mouth feature information.

[0010] The Chinese invention patent application with publication number CN109472198A and name is "A posture-robust video smile recognition method". The application extracts image frames, identifies faces in image frames, calibrates facial feature parts based on faces, matches smile feature models based on the feature parts, and then only detects a small area around the smile feature parts in subsequent frame detections. If subsequent motions are the same, a smile is output.

[0011] At present, there are no specialized products based on smile recognition technology in the market. It is only used as a special video effect when shooting videos on Douyin. However, the application is immature and has not been specifically used in psychological auxiliary education in primary and secondary schools. Summary of the invention

[0012] In view of the shortcomings of the above-mentioned prior art, the main purpose of the present invention is to optimize the core technology in a targeted manner on the basis of the prior art, reasonably adjust the test time interval, and improve the core technology scoring and judgment criteria, so as to guide students to realize the positive impact of smiles on the development of physical and mental health, so that students can express their emotions more actively through smiles, show a friendly state; relieve their own tension, anxiety and other negative emotions, develop a positive and optimistic inner attitude, and obtain a healthy psychological state. And use the present invention as an entry point to open up the application market of psychological auxiliary teaching in primary and secondary schools.

[0013] To this end, some embodiments of the present application provide a smile image recognition system, characterized in that: it includes a smile display module and a smile knowledge popularization module, wherein it also includes a display device as a display carrier of the smile display module and the smile knowledge popularization module; the image comes from an image acquisition device; the smile display module includes a smile acquisition submodule, a smile recognition submodule, a smile scoring submodule and a display submodule; wherein the smile acquisition submodule is configured to acquire a target object; the smile recognition submodule is configured to identify the acquired smile image to obtain expression features in the smile image; the smile scoring submodule is configured to perform a scoring test on each of the smile images according to a preset method; the display submodule is configured to provide the scored smile images to the display device according to preset rules; the smile knowledge popularization module is configured to display content related to the content displayed on the smile display wall in the smile and mental health knowledge display part of the display device.

[0014] In some embodiments, the smile display module is configured to execute continuously, thereby continuously collecting new smile images, and updating the image of the smile display wall portion of the display device after being processed by the smile recognition submodule, the smile scoring submodule and the display submodule.

[0015] In some embodiments, the display submodule further includes a score ranking real-time update unit for providing an updated ranking of the scores of the smile images.

[0016] In some embodiments, the display submodule further includes a recent rating display unit for providing ratings of recent smile images.

[0017] In some embodiments, a smile image recognition method is performed, wherein the smile image acquisition step performed by the smile acquisition submodule includes the following sub-steps: continuously monitoring whether there is a user facing the camera, and when it is detected that a user is smiling at a acquisition device, such as a camera, acquiring the image of the user; if not, continuing monitoring; after the image is acquired, recognizing the image of the user.

[0018] In some embodiments, the smile image recognition method further includes determining whether the face in the collected image of the user is complete. If it is incomplete, a prompt is issued to let the user adjust the angle, and the method returns to the initial step to execute the above steps until the complete face of the user is identified; if the complete face of the user is identified, the method will continue to detect whether multiple faces of the user are captured. If multiple faces of the user are identified, a prompt is issued that the number of people is more than one, and the method returns to the initial step and executes the above steps until a complete user image with only one face is identified; if it is determined that only one user face is identified, it will be used as a smile image for subsequent evaluation.

[0019] In some embodiments, the smile recognition submodule is also configured to determine whether the smile image is recognized for the first time; if it is the first time, the smile image is recorded in the smile image recognition system set by the present invention; if it is not the first time, the smile image is sent to the smile scoring submodule for scoring.

[0020] In some embodiments, the smile scoring submodule is configured to score the smile image based on the identified smile features, and then determine whether the score exceeds a preset threshold; if the score is lower than the preset threshold, a prompt will be issued to retest, and then the operation will be terminated; if it is higher than the preset threshold, the smile image including the face is sent to the display submodule, and displayed on the display device by the display submodule.

[0021] In some embodiments, the smile scoring submodule is configured to score the smile image based on the identified smile features, and then determine whether the score exceeds a preset threshold; if the score is lower than the preset threshold, the score will be sent to the display submodule, which will display it on the display device, and then the operation will be terminated; if it is higher than the preset threshold, a selection interface for whether to display is provided and the user's selection is received. If the received user selection is to display, the smile image including the face will be sent to the display submodule, which will display it on the display device. If the received user selection is not to display, the score will be discarded and the operation will be terminated.

[0022] In some embodiments, the smile knowledge popularization module includes an emotion recognition and self-cognition submodule, which is used to associate smile images with different scores with possible emotion descriptions and display the association results on the display device.

[0023] In some embodiments, the smile knowledge popularization module includes an individual psychological state monitoring and recognition subsystem, which is used to analyze multiple smile images belonging to the same user, and associate the characteristics of the multiple smile images with possible emotions, so as to identify the individual psychological state of the user, and display the identified psychological state results through a display device.

[0024] In some embodiments, the smile knowledge popularization module also includes an interpersonal relationship psychology course sub-module, which uses the smile image scoring sub-module to score the smile image, identify the user's individual positive expression images and negative expression images, and associate the positive expression images and negative expression images with the corresponding interpersonal relationship course content, thereby displaying the corresponding interpersonal relationship course content on the display device according to the results of the user's individual positive expression images and negative expression images.

[0025] In some embodiments, the smile knowledge popularization module further includes an emotion recognition guidance submodule, which is configured to provide an emotion recognition course, that is, to present an emotion recognition course starting from a smile image.

[0026] In some embodiments, the smile knowledge popularization module further includes an emotion vocabulary matching submodule configured to introduce emotion vocabulary and match the smile image with the corresponding emotion vocabulary.

[0027] In some embodiments, the smile knowledge popularization module also includes a smile diary sub-module, which is configured to guide students to check in on the smile image scoring system at a certain frequency, record the student's smile image, score the smile image, and form a smile diary.

[0028] In some embodiments, the smile knowledge popularization module also includes an emotion expression practice sub-module, which is configured to display the real-time collected student smile images on a display device, allowing students to try different degrees of smiles in front of a mirror and feel the changes in their facial muscles. At the same time, the emotions corresponding to different degrees of smiles are displayed in the smile and mental health knowledge display part, and think about the expression effects of different smiles in different situations.

[0029] In some embodiments, the smile knowledge popularization module also includes a problem discovery and response sub-module, which is configured to monitor the changes in students' smile scores by monitoring their smile images at a certain frequency; if it is monitored that a student's smile score has significantly decreased over a period of time, a prompt is generated that the student may be experiencing psychological stress or emotional problems, which is then displayed on the display device.

[0030] The present invention can guide students to smile more in daily life and be willing to try to adjust their emotions with a sincere smile by making them smile at the smile scoring system. This can promote them to smile from the heart frequently, maintain an optimistic and positive mental state, promote the benign development of interpersonal relationships, positive and healthy psychological development and sound personality development, and improve their ability to resist frustration. At the same time, the present invention also plays a role in helping schools to carry out psychological counseling.

[0031] The present invention can display the smiling conditions of students, thereby indirectly reflecting the psychological conditions of students. Schools can observe the smiling conditions of students in the system through the present invention, carry out targeted psychological counseling, and thus promote the improvement of students' mental health levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 FIG. 4 is a schematic diagram of a smile image recognition system according to an embodiment of the present invention.

[0033] Figure 2 FIG. 4 is a schematic diagram of a display device of a smile image recognition system according to an embodiment of the present invention.

[0034] Figure 3 4 is a flow chart of a smile image recognition method executed by a smile image recognition system according to an embodiment of the present invention.

[0035] Figure 4 Detailed flowchart of the image acquisition steps of the smile image recognition method executed by the smile image recognition system according to an embodiment of the present invention.

[0036] Figure 5 4 is a schematic diagram of a detailed flow of image preprocessing of a smile image recognition method performed by a smile image recognition system according to an embodiment of the present invention.

[0037] Figure 6 4 is a detailed flowchart of feature extraction based on HOG of a smile image recognition method performed by a smile image recognition system according to an embodiment of the present invention.

[0038] Figure 7 Detailed flowchart of the feature extraction step of the smile image recognition method executed by the smile image recognition system according to an embodiment of the present invention.

[0039] Figure 8 4 is a detailed flowchart of the scoring steps of the smile image recognition method executed by the smile image recognition system according to an embodiment of the present invention.

[0040] Fig. 9It is a model schematic diagram of a combination of a Histogram of Oriented Gradients (HOG) model and a Convolutional Neural Network (CNN) model of a smile image recognition method executed by a smile image recognition system according to an embodiment of the present invention.

[0041] Fig.10 4 is a schematic diagram of a histogram of a HOG model of a smile image recognition method executed by a smile image recognition system according to an embodiment of the present invention.

[0042] Fig.11A is a schematic diagram of a rectangular interval (R-HOG) of a HOG model of a smile image recognition method performed by a smile image recognition system according to an embodiment of the present invention.

[0043] Fig. 11B is a schematic diagram of a circular interval (C-HOG) of a HOG model of a smile image recognition method performed by a smile image recognition system according to an embodiment of the present invention.

[0044] Fig. 12A Schematic diagram of the training accuracy and verification accuracy of the HOG model of the smile image recognition method executed by the smile image recognition system according to an embodiment of the present invention.

[0045] Fig. 12B Schematic diagram of the training loss rate and the validation loss rate of the HOG model of the smile image recognition method executed by the smile image recognition system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Reference throughout the specification to "one embodiment" means that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of the present invention. Thus, various occurrences of the phrase "in one embodiment" throughout this specification do not necessarily refer to the same embodiment. In addition, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] like Figure 1 As shown, a smile image recognition system of an embodiment of the present invention is mainly composed of two modules: a smile display module 10 and a smile knowledge popularization module 20; wherein, the carriers displayed by these two modules are a display device D, such as a large LED screen, and an image acquisition device, such as a camera.

[0049] The smile display module 10 further includes a smile collection submodule 110, a smile recognition submodule 120, a smile scoring submodule 130 and a display submodule 140. The smile collection submodule 110 is configured to collect smile images of target objects, such as students, and the specific collection methods available will be described later; the smile recognition submodule 120 is configured to identify the collected smile images to obtain the expression features in the smile images, and the specific recognition methods available will be described later; the smile scoring submodule 130 is configured to perform a scoring test on each smile image according to a preset method; the display submodule 140 is configured to provide the scored smile images to the display device according to preset rules, such as according to a preset score. If the smile score exceeds the preset score, the smile image is displayed on the display device D, such as a part of the LED large screen, the smile display wall part D1.

[0050] In addition, the smile display module 10 can be configured to execute continuously, thereby continuously collecting new smile images, and updating the image of the smile display wall part D1 of the LED large screen after being processed by the smile recognition submodule 120, the smile scoring submodule 130 and the display submodule 140.

[0051] The display submodule 140 also includes a rating ranking real-time update unit for providing an updated ranking of the ratings of the smile images. The display submodule 140 also includes a recent rating display unit for providing the ratings of recent smile images. Thus, corresponding displays can be made on the real-time update ranking part D2 and the recent rating display part D3 on the LED screen.

[0052] like Figure 1 , Figure 2 As shown, the smile knowledge popularization module 20 is configured to display content related to the content displayed on the smile display wall part D1 on the display device D, such as the smile and mental health knowledge display part D4 of the LED large screen.

[0053] The smile and mental health knowledge display part D4, for example, can display the knowledge about smile and mental health, including the connotation, function, expression state of a true smile and other knowledge about the positive effects of smile, deepen the user's understanding of smile, guide them to smile more in daily life, further relieve the user's anxiety, tension, uneasiness and other negative emotions, and guide their psychology towards healthy development.

[0054] The present invention adopts the technology of identifying and scoring the smile features in the smile image, and uses it in psychology-assisted teaching.

[0055] The overall process of the smile image recognition method according to the embodiment of the present invention is as follows: Figure 3 shown.

[0056] The smile image acquisition step S110 performed by the smile acquisition submodule 110 is as follows: Figure 4 The following sub-steps may be included as shown: continuously monitoring whether there is a user facing the camera, step S1101; when it is detected that a user is smiling at the acquisition device, such as the camera, the image of the user is acquired, step S1102; if not, the monitoring is continued. After the image is acquired, the image of the user is recognized, step S1103. In an optional embodiment, it is determined whether the face in the acquired image of the user is complete, step S1104; if it is incomplete, a prompt is issued to let the user adjust the angle, step S1105, and the process returns to step S1101 and the above steps are executed until the complete face of the user is recognized. In an optional embodiment, if the complete face of the user is recognized, the process will continue to detect whether multiple user faces are captured, step S1106. If multiple user faces are recognized, a prompt is issued that the number of applicable persons is more than one, step S1107, and the process returns to step S1101 and executes the above steps until a complete user image with only one face is recognized; if it is determined that only one user face is recognized, it will be used as a smile image for subsequent evaluation, step S1108.

[0057] The smile recognition submodule 120 is configured to perform step S120 of recognizing and extracting features of the smile image based on a model combining a histogram of oriented gradients (HOG) and a convolutional neural network (CNN).

[0058] The model combining the histogram of oriented gradients (HOG) and the convolutional neural network (CNN) can be as follows: Fig. 9 As shown, it includes a HOG model part 1201 and a convolutional neural network convolutional neural network model part 1202, a fully connected layer (FC) 1203, and an output layer 1203.

[0059] The step S120 of identifying and extracting smile features of the smile image based on the HOG model part 1201 may include:

[0060] First, image preprocessing settings 1210 are performed, such as Figure 5 As shown, including:

[0061] Step S1211, size standardization, that is, standardizing the size of the collected smile images, such as standardizing them to preset pixel width and height, so that the faces in the smile images are at the same spatial scale in subsequent analysis, which is convenient for feature extraction and comparison.

[0062] Step S1212, grayscale conversion, that is, converting the color smile image into a grayscale image, reduces the amount of data and computational complexity while highlighting key information such as facial contour and texture, because the change in grayscale value can better reflect the ups and downs of facial muscles and changes in expression.

[0063] Step S1213, illumination correction, i.e., illumination compensation or normalization processing is performed on the smile image to reduce the influence of different illumination conditions on the smile feature extraction, such as by adjusting the brightness and contrast of the smile image to make the illumination of the facial area more uniform, to avoid partial smile features being covered or misjudged due to excessive brightness or darkness.

[0064] The order of the above steps S1211, S1212, and S1213 can be adjusted arbitrarily without affecting the effect of the preprocessing.

[0065] Then, the face detection and positioning step S1220 is performed on the pre-processed smile image. Figure 5 As shown, including:

[0066] Step S1221, using an accurate smile detection algorithm model to find the position and range of the face, using the HOG model to accurately find the position and range of the face in the preprocessed smile image, eliminating background and other interference factors.

[0067] Step S1222, key point positioning, that is, determining the key point positions of the mouth in the face, accurately analyzing the changes in facial expressions, such as the degree of upward movement of the corners of the mouth, the degree of mouth opening, the degree of grinning of the corners of the mouth, etc., to provide a basis for the accurate collection of smile features.

[0068] Finally, the geometric feature extraction is set, step S1230, geometric feature extraction, that is, measuring the relative position and shape changes of the face, such as calculating the distance between the corners of the mouth and the eyes, the ratio of the width and height of the mouth, etc. These geometric features will have obvious differences under different smile levels, and can be used as an important basis for judging the intensity of the smile.

[0069] Specifically, in this embodiment, the HOG algorithm is used to implement the geometric feature setting step S1221 of the student's smile image. Figure 6 As shown, it may further include:

[0070] Step S12211, color and gamma normalization.

[0071] In order to reduce the influence of lighting factors, the entire image needs to be normalized first. In the texture intensity of the image, the local surface exposure contributes a large proportion, so this compression process can effectively reduce the local shadows and lighting changes in the image.

[0072] Step S12212, calculate the image gradient.

[0073] Calculate the gradient of the image in the horizontal and vertical directions, and calculate the gradient direction value of each pixel position accordingly; the derivative operation can not only capture contours, human figures and some texture information, but also further weaken the influence of lighting. The most commonly used method is to simply use a one-dimensional discrete differential template to process the image in one direction or in both horizontal and vertical directions. More precisely, this method requires the use of a filter kernel to filter out the color or drastically changing data in the image. The image is composed of pixels, and the image can be regarded as a two-dimensional discrete function f(x,y). The gradient G(x,y) of the image function f(x,y) at point (x,y) is a vector with size and direction, set to Gx and Gy, representing the gradient in the horizontal (x) direction and vertical (y) direction respectively.

[0074] GG x (i,j)=I(i+1,j)-I(i,j)

[0075] G y (i,j)=I(i,j+1)-I(i,j)

[0076] G(i,j)=G x (i,j)+G y (i,j)

[0077] θ(i,j)=tan -1 (G x (i,j) / G y (i,j))

[0078] Among them, I(i,j) represents the pixel value of the image at the pixel point (i,j), and θ(i,j) represents the direction of the gradient of the pixel point (i,j).

[0079] Step S12213, construct a direction histogram.

[0080] Each pixel in the cell unit votes for a direction-based histogram channel. Voting is weighted voting, that is, each vote has a weight, and this weight is calculated based on the gradient amplitude of the pixel. The weight can be represented by the amplitude itself or its function. Actual tests show that using the amplitude to represent the weight can achieve the best effect. Of course, you can also choose to use the function of the amplitude to represent it, such as the square root of the amplitude, the square of the amplitude, the truncated form of the amplitude, etc. The cell unit can be rectangular or star-shaped. Histogram, such as Fig.10As shown, the channels are evenly distributed in the range of 0-1800 (undirected) or 0-3600 (directed). It is found that the use of undirected gradient and 9 histogram channels can achieve the best results in pedestrian detection experiments.

[0081] Step S12214, combine the cell units into large intervals.

[0082] Due to the change of local illumination and the change of foreground-background contrast, the gradient intensity varies greatly. This requires normalization of the gradient intensity. Normalization can further compress illumination, shadows and edges.

[0083] The approach is to combine the individual cells into large, spatially connected intervals. In this way, the HOG descriptor becomes a vector consisting of the histogram components of all the cells in each interval. These intervals overlap, which means that the output of each cell acts on the final descriptor multiple times.

[0084]

[0085] There are two main geometric shapes of intervals - rectangular intervals (R-HOG) and circular intervals (C-HOG), as shown in Figure 11. Fig. 11B As shown in Figure 2, R-HOG intervals are generally square grids, which can be characterized by three parameters: the number of cell units in each interval, the number of pixels in each cell unit, and the number of histogram channels for each cell.

[0086] Step S12215, collecting HOG features.

[0087] The smile image dataset used in the embodiment of the present application can be the Dlib dataset. The extracted HOG features are input into the SVM classifier to find an optimal hyperplane as the decision function. The training accuracy and verification accuracy in an embodiment of the present application are as follows: Fig. 12A As shown, the training loss rate and the validation loss rate in one embodiment of the present application are as follows: Fig. 12B shown.

[0088] Based on the convolutional neural network model part 1202, smile features can be automatically extracted, and rich hidden features can be extracted. The extracted hidden features can better reflect the changes in facial expressions, which can effectively improve the accuracy of facial expression recognition and increase the robustness of the system.

[0089] The features extracted from both can be introduced into the fully connected layer for full connection operation S1250, and then output through the output layer for output operation S1260. The output result includes geometric features and hidden features, so as to better express the features of the input image P.

[0090] In an optional embodiment, if Figure 8 As shown, the smile recognition submodule 120 is further configured to determine whether the smile image is recognized for the first time, step S1241; if it is the first time, the smile image is recorded in the smile image recognition system set by the present invention, step S1242; if it is not the first time, the smile image is sent to the smile scoring submodule 130 for scoring, step S1243, Figure 7 shown.

[0091] In some embodiments, the smile scoring submodule 130 of the present application is configured to score the smile image according to the identified smile features, and then determine whether the score exceeds a preset threshold, step S131; if the score is lower than the preset threshold, such as 60 points, a prompt will be issued, step S132; for example, a prompt that the smile score is too low, please retest, and then the present invention ends; if the score is higher than the preset threshold, the smile image including the face is sent to the display submodule 140, step S133, and the display submodule 140 displays it on the smile display wall D1 of the display device D. In some embodiments, if the score is lower than the preset threshold, such as 60 points, the score is sent to the display submodule 140, and the display submodule 140 displays it on the smile display wall D1 of the display device D. In some embodiments, if the score is higher than the preset threshold, a selection interface is provided to receive the user's selection. If the user's selection is to display, the smile image including the face is sent to the display submodule 140, step S133, and displayed by the display submodule 140 on the smile display wall D1 of the display device D; if the user's selection is not to display, the score is discarded and the operation ends.

[0092] In the present invention, the smile images that have passed the rating are applied to the psychological auxiliary teaching of primary and secondary school students through the following process. As mentioned above, the smile knowledge popularization module 20 is configured to generate smile knowledge content and display it on the display device D, such as the smile and mental health knowledge display part D4 of the LED large screen.

[0093] Specifically, the smile knowledge popularization module 20 may include an emotion recognition and self-cognition submodule 210 for associating smile images with different scores, i.e., different characteristics, with possible emotion descriptions, and displaying the association results in the smile and mental health knowledge display part D4.

[0094] The smile knowledge popularization module 20 may also include an individual psychological state monitoring and identification subsystem 220, which is used to analyze multiple smile images belonging to the same user, such as those within a preset time period, such as multiple smile images on different dates or multiple smile images in different time periods on the same day, and associate their features with possible emotions, thereby identifying the individual psychological state of the user, and displaying the identified psychological state results through the smile and mental health knowledge display part D4.

[0095] The smile knowledge popularization module 20 may also include an interpersonal relationship psychology course submodule 230. The interpersonal relationship psychology course submodule 230 utilizes the smile image scoring submodule 130 to identify the individual user's positive expression images and negative expression images, and associates the positive expression images and negative expression images with the corresponding interpersonal relationship course content, thereby displaying the corresponding interpersonal relationship course content in the smile and mental health knowledge display part D4 according to the results of the individual user's positive expression images and negative expression images.

[0096] In addition, the smile scoring system of the present invention uses the collected smile image of the corresponding user as an entry point to guide students to conduct psychological education counseling in a targeted manner through the following submodules, specifically including:

[0097] Emotional cognition guidance submodule 240. The emotional cognition guidance submodule is configured to provide an emotion recognition course, that is, to present the emotion recognition course starting from a smile image. For example, various smile images can be displayed in the classroom through the display device D of the smile scoring system of the present application, including laughing, smiling, polite smile, etc., while also showing some non-smiling expressions, such as frowning, crying, etc. Guide students to observe these expressions carefully and ask them to describe the characteristic differences between different smiles and other expressions, such as the curvature characteristics of the corners of the mouth, the state characteristics of the eyes, etc. For example, a truly happy smile usually makes the eyes squint and eye wrinkles appear, while a polite smile may just raise the corners of the mouth without much change in the eyes.

[0098] Emotion vocabulary matching submodule 250. The emotion vocabulary matching subsystem 250 is configured to introduce emotion vocabulary and match the smile image with the corresponding emotion vocabulary; the emotion vocabulary can be such as happy, joyful, friendly, etc., and provide a comparison of emotion vocabulary corresponding to non-smiley expressions such as sadness, anger, anxiety, etc. Let students understand that facial expressions are the external manifestations of emotions, and smiles often represent positive emotions. Later, through group discussions, students can share under what circumstances they will show different types of smiles, and the real emotional feelings behind these smiles.

[0099] Smile diary submodule 260. The smile diary submodule 260 can realize self-emotional awareness and expression. The smile diary submodule is configured to guide students to check in on the smile scoring system at a certain frequency, such as every day; record the student's smile image; score the smile image; and form a smile diary. Displaying the smile diary on the display device D of the smile scoring system can help students be more aware of their emotions and learn to use smiles to mark the generation of positive emotions.

[0100] Emotional expression practice submodule 270. The emotional expression practice submodule 270 is configured to display the real-time collected student smile images on the display device D, allowing students to try different degrees of smiles in front of the mirror and feel the changes in their facial muscles. At the same time, the emotions corresponding to different degrees of smiles are displayed in the smile and mental health knowledge display part D4, and think about the expression effects of different smiles in different situations. For example, when expressing welcome to friends, it may be a warm laugh, while in front of elders who meet for the first time, it may be a gentle smile. Through this practice, students' ability to control emotional expression is improved, allowing them to convey their emotions more accurately.

[0101] Problem discovery and response submodule 280. The problem discovery and response submodule 280 is configured to monitor the change of students' smile scores by monitoring students' smile images at a certain frequency; if it is monitored that the smile score of a student decreases significantly over a period of time, a prompt is generated that the student may be experiencing psychological pressure or emotional problems, and then displayed on the display device D, such as the smile and mental health knowledge display part D4.

[0102] The smile image scoring system of the present invention can also be used for individual psychological status monitoring to identify students with potential psychological problems.

[0103] On the one hand, the smile scoring system can be used for emotion recognition and self-cognition as an aid to psychology classes. For example, at the beginning of a psychology class, teachers can export the smile images of different degrees of smiles (such as high-scoring smiles, low-scoring smiles, and neutral expressions) that have been scored and stored in the smile scoring system. Let students observe these smile images and guess the possible emotions behind these expressions. This helps students improve their emotion recognition ability and learn to distinguish different emotional states, such as real happiness, forced smiles, or negative emotions that may be hidden behind a blank expression.

[0104] On the one hand, this smile scoring system can be used for self-emotional assessment. For example, students can be asked to use the smile scoring system for test assessment regularly, and these assessment results can be fed back to students so that they can reflect on their inner feelings. For example, some students may smile when taking photos, but they do not feel happy, which can trigger their thinking about the difference between their emotional expression and their true inner feelings. For example, organize an activity of "I know my emotions". A self-emotional assessment test can be conducted, in which students take a photo every day for a week, and then record their emotions in a psychological diary based on the results of the smile scoring system and their own experiences of the day. Teachers can guide students to think accordingly.

[0105] On the one hand, the smile scoring system in the present invention can collect the scoring scores of smile images of student groups and identify those students whose smile images have low scores for a long time, that is, students with long-term negative expressions. These students may have potential psychological pressure or emotional problems, such as depression, anxiety, etc. Teachers and psychological counselors can focus on these students and conduct further psychological assessments to understand whether they have encountered difficulties in learning, life or interpersonal relationships. Track the effect of psychological intervention. During the process of intervention treatment (such as psychological counseling, group psychological counseling, etc.) for students with psychological problems, the smile scoring system is used to continuously monitor the changes in students' expressions. The effectiveness of psychological intervention is evaluated by comparing the scores of smile images before and after the intervention. For example, for a student who is depressed due to family changes, after a period of individual psychological counseling, observe whether his smile score has an upward trend. If the smile score gradually increases, it means that the psychological intervention may have played a positive role; otherwise, the intervention strategy may need to be adjusted.

[0106] On the one hand, the smile scoring system of the present invention can be used to provide interpersonal relationship courses for improving social interaction and interpersonal relationships.

[0107] In the interpersonal psychology course, we use the positive and negative expression images identified by the smile scoring system to explain the impact of different expressions on interpersonal relationships. We help students understand the importance of positive expressions such as smiles in social interactions, encourage them to show more positive emotions in daily interactions, and improve their interpersonal relationships.

[0108] Compared with the prior art, the invention is characterized by using facial feature point recognition technology to extract feature points for smile scoring and applying them to psychological auxiliary teaching for primary and secondary school students. Taking smile as the entry point, students can be guided to receive psychological education and counseling in a targeted manner, so that students' psychological pressure can be released.

[0109] This application uses facial feature point recognition for smile scoring, allowing students to adjust their mental state during use. The collected scoring data can help teachers understand students' mental state.

[0110] For some embodiments, smiles and other facial expressions are used for annotation on personal computers, documents, videos to establish interesting entry points or tags in the video. This PC application can be used for a camera installed in the display of a personal computer, for example.

[0111] According to certain embodiments, data processing uses a digital processing system (DPS). The DPS may be configured to store, process, and communicate a variety of digital information of various types, including digital images and videos.

[0112] As discussed above, embodiments of the present invention may utilize a DPS or device with digital processing capabilities. Example components of such a system include a central processing unit (CPU) and a signal processor coupled to a main memory, a static memory, and a mass storage device. The main memory may store various applications for implementing the operation of the present invention, while the mass storage device may store various digital content.

[0113] The DPS may also be coupled to input / output (I / O) devices and audio / video devices. The CPU may be used to process information and / or signals for processing the system. The main memory may be a random access memory (RAM) or some other dynamic storage device for storing information or instructions (program code) used by the CPU. The static memory may be a read-only memory (ROM) and / or other static storage device for storing information or instructions that may also be used by the CPU. The mass storage device may be, for example, a hard disk drive, an optical disk drive, or firmware for storing information or instructions for the processing system.

[0114] Embodiments of the present invention have been described as including various operations. Many of the processes are described in their most basic form, but operations may be added to or deleted from any process without departing from the scope of the present invention.

[0115] The operation of the present invention can be performed by hardware components or can be implemented in the following machine executable instructions, which can be used to make a general or special processor or logic circuit programmed with these instructions operate. Alternatively, the steps can be performed by a combination of hardware and software. The present invention can be provided as the following computer program product, which can include a machine-readable medium with the following instructions stored thereon, which can be used to program a computer (or other electronic device) to perform a process according to the present invention. The machine-readable medium can include, but is not limited to, a floppy disk, an optical disk, a CD-ROM and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory or other types of media / machine-readable media suitable for storing electronic instructions. In addition, the present invention can also be downloaded as a computer program product, wherein the program can be transmitted from a remote computer to a requesting computer by a data signal implemented in a carrier or via other propagation media (e.g., a modem or a network connection) of a communication cell. All operations can be performed at the same central site, or one or more operations can be performed elsewhere instead.

Claims

1. A smile image recognition system, characterized in that: It comprises a smile display module (10) and a smile knowledge popularization module (20), wherein it also comprises a display device as a display carrier of the smile display module (10) and the smile knowledge popularization module (20); the image comes from an image acquisition device; The smile display module (10) comprises a smile collection submodule (110), a smile recognition submodule (120), a smile scoring submodule (130) and a display submodule (140); wherein the smile collection submodule (110) is configured to collect target objects; the smile recognition submodule (120) is configured to recognize the collected smile images to obtain expression features in the smile images; the smile scoring submodule (130) is configured to perform a scoring test on each of the smile images according to a preset method; and the display submodule (140) is configured to provide the scored smile images to the display device according to preset rules; The smile knowledge popularization module (20) is configured to display content related to the content displayed in the smile display wall part (D1) on the smile and mental health knowledge display part (D4) of the display device (D).

2. The smile image recognition system according to claim 1, characterized in that: The smile display module (10) is configured to be continuously executed, thereby continuously collecting new smile images, and updating the image of the smile display wall part (D1) of the display device after being processed by the smile recognition submodule (120), the smile scoring submodule (130) and the display submodule (140).

3. The smile image recognition system according to claim 1, characterized in that: The display submodule (140) also includes a rating ranking real-time updating unit (D2) for providing an updated ranking of the ratings of the smile images.

4. The smile image recognition system according to claim 1, characterized in that: The display submodule (140) further comprises a recent rating display unit (D3) for providing ratings of recent smile images.

5. The smile image recognition system according to claim 1, characterized in that: A smile image recognition method is executed, wherein the smile image acquisition step executed by the smile acquisition submodule (110) includes the following sub-steps: continuously monitoring whether there is a user facing a camera, and when it is detected that a user is smiling at an acquisition device, such as a camera, acquiring the image of the user; if not, continuing monitoring; and after an image is acquired, recognizing the image of the user.

6. The smile image recognition system according to claim 5, characterized in that: The smile image recognition method further includes determining whether the face of the user in the collected image is complete, and if it is incomplete, issuing a prompt to allow the user to adjust the angle, and returning to the initial step to perform the above steps until the complete face of the user is recognized; If the complete face of the user is recognized, the system will continue to detect whether multiple faces of the user are captured. If multiple faces of the user are recognized, a prompt will be issued to indicate that there are more than one person, and the system will return to the initial step and execute the above steps until a complete user image with only one face is recognized; if it is determined that only one face of the user is recognized, it will be used as a smile image for subsequent evaluation.

7. The smile image recognition system according to claim 6, characterized in that: The smile recognition submodule (120) is also configured to determine whether the smile image is recognized for the first time; if it is the first time, the smile image is recorded in the smile image recognition system set by the present invention; if it is not the first time, the smile image is sent to the smile scoring submodule (130) for scoring.

8. The smile image recognition system according to claim 7, characterized in that: The smile scoring submodule (130) is configured to score the smile image according to the identified smile features, and then determine whether the score exceeds a preset threshold; if the score is lower than the preset threshold, a prompt is issued to retest, and then the operation is terminated; if the score is higher than the preset threshold, the smile image including the face is sent to the display submodule (140), and the display submodule (140) displays it on the display device (D); or if the score is lower than the preset threshold, the score is sent to the display submodule (140), and the display submodule (140) displays it on the display device, and then the operation is terminated; if the score is higher than the preset threshold, a selection interface for whether to display is provided and a user's selection is received; if the received user selection is to display, the smile image including the face is sent to the display submodule (140), and the display submodule (140) displays it on the display device (D); if the received user selection is not to display, the score is discarded and the operation is terminated.

9. The smile image recognition system according to claim 1, characterized in that: The smile knowledge popularization module (20) includes an emotion recognition and self-cognition submodule (210) for associating smile images with different scores with possible emotion descriptions and displaying the association results on the display device (D); and / or the smile knowledge popularization module (20) includes an individual psychological state monitoring and identification subsystem (220) for analyzing multiple smile images belonging to the same user and associating the characteristics of the multiple smile images with possible emotions, thereby identifying the psychological state of the individual user and displaying the identified psychological state results through the display device (D); and / or the smile knowledge popularization module (20) also includes an interpersonal relationship psychology course submodule (230), the interpersonal relationship psychology course submodule (230) uses the scores of the smile images by the smile image scoring submodule (130) to identify the positive expression images and negative expression images of the individual user, and associates the positive expression images and negative expression images with corresponding interpersonal relationship course content, thereby displaying the corresponding interpersonal relationship course content on the display device according to the results of the positive expression images and negative expression images of the individual user.

10. The smile image recognition system according to claim 1, characterized in that: The smile knowledge popularization module (20) further includes: and / or An emotion recognition guidance submodule (240) is configured to provide an emotion recognition course, that is, to present an emotion recognition course starting from a smile image; and / or An emotion vocabulary matching submodule (250) configured to introduce emotion vocabulary and match the smile image with the corresponding emotion vocabulary; and / or The smile diary submodule (260) is configured to guide the student to check in on the smile image scoring system at a certain frequency, record the student's smile image, score the smile image, and form a smile diary; and / or The emotion expression practice submodule (270) is configured to display the real-time collected student smile images on the display device (D), allowing the student to try different degrees of smiles in front of the mirror and feel the changes in their facial muscles. At the same time, the emotions corresponding to different degrees of smiles are displayed in the smile and mental health knowledge display part (D4), and the student thinks about the expression effects of different smiles in different situations; and / or The problem discovery and response submodule (280) is configured to monitor the smile images of students at a certain frequency to monitor the changes in the smile scores of students; if it is monitored that the smile score of a student decreases significantly over a period of time, a prompt is generated indicating that the student may be experiencing psychological pressure or emotional problems, and then displayed on the display device (D).

Images