Emotion analysis method and device based on intelligent fitness mirror

Abstract

The invention discloses an emotion analysis method and device based on an intelligent fitness mirror, belongs to the technical field of intelligent fitness mirrors, integrates an emotion monitoring function, and judges the emotion state of a user by analyzing the facial expression of the user through a camera and monitoring the heart rate through a bracelet. According to the emotional state, the light bar embedded in the mirror frame can give out light feedback of the corresponding color (green represents happiness, blue represents difficulty, and yellow represents quietness). In addition, the emotion analysis method and device based on the intelligent fitness mirror can recommend appropriate fitness content and music according to the mood and emotion of the user.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart fitness mirrors, and in particular relates to a method and device for analyzing emotions based on smart fitness mirrors. Background Art

[0002] Smart fitness mirror is a weight loss product. It uses advanced far-infrared cold light technology and intelligent technology to help the body consume excess calories through aerobic and anaerobic exercise to achieve the effect of weight loss. The market positioning of smart fitness mirror is mainly focused on consumers who pursue efficient and convenient fitness methods, such as urban white-collar workers, home fitness users and gym members. These users generally have a certain fitness awareness and hope to achieve goals such as body shaping, muscle gain or fat loss through scientific fitness methods. Smart fitness mirrors meet the diverse needs of these users with their integrated functions and personalized services. In the field of population segmentation: it can be divided into three major groups: young people, middle-aged people and the elderly. Young people are usually curious about emerging technology products and pursue personalized fitness methods. Smart fitness mirrors can provide a variety of fitness courses to meet the diverse fitness needs of young people; middle-aged people may not have time to go to the gym due to work, family and other reasons. Smart fitness mirrors can provide professional fitness guidance in a home environment to help them stay healthy; the elderly may pay more attention to health and wellness. The gentle courses of smart fitness mirrors are suitable for their physical condition and can also help them stay energetic.

[0003] At present, the smart fitness mirror with the patent publication number of "CN214315387U" is often used. However, the smart fitness mirror currently used has the following defects:

[0004] The user's emotional state cannot be reflected on the smart fitness mirror, and the smart fitness mirror cannot recommend appropriate fitness content and music based on the user's emotional state. Summary of the invention

[0005] In order to solve the defects in the prior art, the present invention proposes an emotion analysis method and device based on a smart fitness mirror, which integrates an emotion monitoring function and judges the user's emotional state by analyzing the user's facial expressions through a camera and monitoring the heart rate through a bracelet. According to the emotional state, the light bar embedded in the frame will emit light feedback of corresponding colors (green represents happiness, blue represents sadness, and yellow represents calmness). In addition, the emotion analysis method and device based on the smart fitness mirror will also recommend appropriate fitness content and music according to the user's mood.

[0006] The present invention uses the following technical solutions.

[0007] A sentiment analysis method based on a smart fitness mirror, comprising:

[0008] Step 1: Data collection of smart wearable bracelet;

[0009] Step 2: Camera data collection;

[0010] Step 3: Data is reported to the backend server.

[0011] Preferably, in step 1, the backend server uses Python's Bluetooth library to establish a connection with the smart wearable bracelet and receives the user's heart rate data and pulse data acquired by the controller in the smart wearable bracelet.

[0012] Preferably, the backend server uses the Python Bluetooth library to establish a connection with the smart wearable bracelet and receives the user's heart rate data and pulse data obtained from the controller in the smart wearable bracelet, specifically including:

[0013] Step 1-1: Install PyBluez on the backend server;

[0014] Step 1-2: Connect the wireless communication module of the smart wearable bracelet and receive the user's heart rate data and pulse data transmitted through the controller.

[0015] Preferably, step 1-2 specifically includes: using the PyBluez library to search for and connect to the wireless communication module 1 of the smart wearable bracelet, and then receiving the user's heart rate data and pulse data transmitted through the controller 1.

[0016] Preferably, step 2 specifically includes:

[0017] Step 2-1: Install OpenCV on the backend server;

[0018] Step 2-2: The backend server uses OpenCV to capture and process the video.

[0019] Preferably, step 2-2 specifically includes: using the VideoCapture class of OpenCV to capture a video, and then obtaining facial expressions from the video, wherein the video includes a facial expression image of the user.

[0020] Preferably, step 3 specifically includes:

[0021] Step 3-1: Install requests on controller 1 and controller 2;

[0022] Step 3-2: Controller 1 and controller 2 send data to the server.

[0023] Preferably, step 3-2 specifically includes: controller 1 and controller 2 package the collected user's heart rate data, pulse data, facial expression images into JSON or other formats, and send them to the backend server via HTTP request.

[0024] Preferably, the emotion analysis method based on the smart fitness mirror further includes:

[0025] Step 4: Smart exercise recommendations.

[0026] Preferably, the user's mood is judged based on the user's heart rate data, pulse data collected by the smart wearable bracelet and facial expressions captured by the camera, and suitable exercise is recommended based on the mood.

[0027] An emotion analysis device based on a smart fitness mirror, comprising:

[0028] Smart wearable bracelets, cameras and smart fitness mirrors;

[0029] The smart wearable bracelet is used to be worn on the user's wrist to collect the user's pulse and heart rate. The lower end surface of the bracelet body of the smart wearable bracelet is provided with a heart rate monitoring sensor and a pulse sensor. The heart rate monitoring sensor and the pulse sensor are both connected to a controller 1 provided in the smart wearable bracelet, and the controller 1 is also connected to a wireless communication module 1;

[0030] The smart fitness mirror comprises a controller 2, a camera, a wireless communication module 2, an LED light and a display screen arranged on the frame of the smart fitness mirror. The wireless communication module 2, the camera, the LED light and the display screen are all connected to the controller 2. The wireless communication module 2 and the wireless communication module 1 are both connected to the back-end server in the wireless network. The camera is used to collect the user's facial expression image and transmit it to the controller 2.

[0031] The emotion analysis device based on the smart fitness mirror also includes:

[0032] Collection module 1, which is used for data collection of smart wearable bracelets;

[0033] Collection module 2, which is used for camera data collection;

[0034] Reporting module, which is used to report data to the backend server;

[0035] Recommendation module, which is used for intelligent sports recommendation.

[0036] Preferably, the heart rate monitoring sensor and the pulse sensor are used to collect the user's heart rate data and pulse data respectively and transmit them to the controller.

[0037] The beneficial effects of the present invention are that, compared with the prior art, the technical effects of the present invention include:

[0038] The emotion analysis method and device based on the smart fitness mirror integrates the emotion monitoring function, and judges the user's emotional state by analyzing the user's facial expressions through the camera and monitoring the heart rate through the bracelet. According to the emotional state, the light bar embedded in the frame will emit light feedback of the corresponding color (green represents happiness, blue represents sadness, and yellow represents calmness). In addition, the emotion analysis method and device based on the smart fitness mirror will also recommend appropriate fitness content and music according to the user's mood. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the emotion analysis method based on the smart fitness mirror described in the present invention;

[0040] Figure 2 It is a partial structural diagram of the emotion analysis device based on the smart fitness mirror described in the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the following will be combined with the drawings in the embodiments of the present invention to clearly and completely express the technical solution of the present invention. The embodiments expressed in the present invention are only some embodiments of the present invention, not all embodiments. According to the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0042] like Figure 1 As shown, the emotion analysis method based on the smart fitness mirror described in the present invention includes:

[0043] Based on the original functions of the smart fitness mirror, by investigating the sales data and consumer preferences of the site, combined with relevant research data, it is found that different moods should be matched with corresponding music and sports, which is very helpful for physical and mental health and improving mood. Emotion monitoring requires collecting data (such as pulse and heart rate indicators) through smart wearable bracelets, collecting facial expressions through the camera of the smart fitness mirror, and reporting these data to the backend server through the Bluetooth module. Several key steps need to be completed.

[0044] Step 1: Data collection of smart wearable bracelet;

[0045] In a preferred but non-limiting embodiment of the present invention, in step 1, the smart wearable bracelet usually communicates with a mobile phone or other device through a wireless communication module 1. The back-end server is required to use a Python Bluetooth library (such as PyBluez) to establish a connection with the smart wearable bracelet and receive the user's heart rate data and pulse data obtained from the controller 1 in the smart wearable bracelet.

[0046] In a preferred but non-limiting embodiment of the present invention, the back-end server uses a Python Bluetooth library (such as PyBluez) to establish a connection with the smart wearable bracelet and receive the user's heart rate data and pulse data obtained from the controller in the smart wearable bracelet, specifically including:

[0047] Step 1-1: Install PyBluez on the backend server;

[0048] Step 1-1 specifically includes: You can use pip to install PyBluez as follows:

[0049] pip install pybluez

[0050] Step 1-2: Connect the wireless communication module of the smart wearable bracelet and receive the user's heart rate data and pulse data transmitted through the controller.

[0051] In a preferred but non-limiting embodiment of the present invention, steps 1-2 specifically include: using the PyBluez library to search for and connect to the wireless communication module 1 of the smart wearable bracelet, and then receiving the user's heart rate data and pulse data transmitted through the controller 1.

[0052] Step 2: Camera data collection;

[0053] In a preferred but non-limiting embodiment of the present invention, in step 2, for camera data acquisition, the OpenCV library can be used to capture the video stream and extract facial expressions from it. It specifically includes:

[0054] Step 2-1: Install OpenCV on the backend server;

[0055] Step 2-1 specifically includes: Install OpenCV using pip as follows:

[0056] pip install opencv-python

[0057] Step 2-2: The backend server uses OpenCV to capture and process the video.

[0058] In a preferred but non-limiting embodiment of the present invention, step 2-2 specifically includes: using the VideoCapture class of OpenCV to capture a video, and then using an image processing method to identify and analyze the video to obtain a facial expression and / or posture image, wherein the video includes a facial expression image and / or posture image of the user. Subsequently, the patent application number "CN202111285381.9" and the patent name "Emotion Recognition Method for Face and Human Body Posture Based on Video Images" can be used to identify and analyze happy expressions, neutral expressions, and sad expressions of facial expressions.

[0059] Step 3: Data is reported to the backend server.

[0060] In a preferred but non-limiting embodiment of the present invention, step 3 specifically includes: using Python's requests library or socket library to send data to the backend server, depending on how the backend server is set up.

[0061] Step 3-1: Install requests on controller 1 and controller 2;

[0062] Step 3-1 specifically includes: If you choose to use the requests library, you can install it through pip, including:

[0063] pip install requests

[0064] Step 3-2: Controller 1 and controller 2 send data to the server.

[0065] In a preferred but non-limiting embodiment of the present invention, step 3-2 specifically includes: controller 1 and controller 2 package the collected user's heart rate data, pulse data, facial expression images and / or posture images into JSON or other formats, and send them to the back-end server via an HTTP request.

[0066] The code implementation process of controller 1 and controller 2 sending data to the server is as follows:

[0067] import cv2

[0068] import bluetooth

[0069] import requests

[0070] import json

[0071] #Function to connect Bluetooth to the bracelet and receive data

[0072] def connect_and_receive_from_bracelet():

[0073] #Use PyBluez to connect to the bracelet and receive data

[0074] pass

[0075] #Function to capture facial expressions using OpenCV

[0076] def capture_facial_expression():

[0077] cap = cv2.VideoCapture(0) # Open the camera

[0078] #Capture video frames and process them to recognize facial expressions

[0079] pass

[0080] #Function to report data to the server

[0081] def report_to_server(data):

[0082] url = 'your server URL'

[0083] headers={'Content-Type':'application / json'}

[0084] response=requests.post(url,data=json.dumps(data),headers=headers)

[0085] #Processing responses, etc.

[0086] #Main program

[0087] def mPythonn():

[0088] bracelet_data=connect_and_receive_from_bracelet()

[0089] facial_expression_data=capture_facial_expression()

[0090] data_to_send = {

[0091] 'bracelet_data':bracelet_data,

[0092] 'facial_expression':facial_expression_data

[0093] }

[0094] report_to_server(data_to_send)

[0095] if__name__=='__mPythonn__':

[0096] mPythonn()

[0097] In a preferred but non-limiting embodiment of the present invention, the emotion analysis method based on the smart fitness mirror further includes:

[0098] Step 4: Smart exercise recommendations.

[0099] In a preferred but non-limiting embodiment of the present invention, the user's mood is judged based on the user's heart rate data, pulse data and facial expressions captured by the camera collected by the smart wearable bracelet, and suitable exercise is recommended based on the mood. Taking a bad mood indicator as an example, the data flow diagram is as follows:

[0100] Implementing the above process involves multiple components and steps, including Bluetooth communication, image processing, data analysis, and Web application development. The following is a conceptual implementation framework and some key codes:

[0101] (1) Bluetooth communication

[0102] Use the PyBluez library to communicate with the bracelet via Bluetooth and receive data.

[0103] import bluetooth

[0104] def receive_data_from_bracelet(address):

[0105] #The address here is the Bluetooth address of the bracelet

[0106] sock=bluetooth.BluetoothSocket(bluetooth.RFCOMM)

[0107] sock.connect((address,1))#Assume that the channel used by the bracelet is 1

[0108] while True:

[0109] data = sock.recv(1024) # receive data

[0110] #Process the received data, such as parsing it into heartbeat and pulse indicators

[0111] process_bracelet_data(data)

[0112] (2) Image processing

[0113] Capture and process camera data using OpenCV to recognize facial expressions.

[0114] import cv2

[0115] def capture_and_analyze_face_expression():

[0116] cap = cv2.VideoCapture(0) #0 is usually the ID of the computer's default camera

[0117] while True:

[0118] ret,frame=cap.read()

[0119] if not ret:

[0120] break

[0121] #Here you can add the code for facial expression recognition

[0122] #For example, use OpenCV's Haar Cascades or deep learning models for face detection and expression recognition

[0123] expression = detect_expression(frame) #Assume this function can detect facial expressions

[0124] process_expression_data(expression)

[0125] cap.release()

[0126] (3) Data processing and exercise recommendation

[0127] Recommend exercises based on the data collected.

[0128] def process_bracelet_data(data):

[0129] #Analyze the bracelet data and extract heartbeat and pulse information

[0130] heart_rate,pulse_rate=parse_bracelet_data(data)

[0131] recommend_exercise(heart_rate,pulse_rate)

[0132] def process_expression_data(expression):

[0133] #Recommend exercise based on facial expressions

[0134] recommend_exercise(None,None,expression)

[0135] def recommend_exercise(heart_rate=None,pulse_rate=None,expression=None):

[0136] #Recommend exercise based on heartbeat, pulse and facial expressions

[0137] if heart_rate is not None and pulse_rate is not None:

[0138] if heart_rate>100and pulse_rate>80:#Example threshold

[0139] print("Aerobic exercise, such as jogging, is recommended.")

[0140] if expression=="sad":

[0141] print("Recommend yoga or meditation.")

[0142] #Other conditions and recommendation logic...

[0143] (4) Web application development

[0144] In order to build a complete system, you may also need a web application to display data and recommendation results. This can be achieved using a web framework such as Flask.

[0145] from flask import Flask,render_template,request

[0146] app = Flask(__name__)

[0147] @app.route(' / ')

[0148] def index():

[0149] return render_template('index.html')#Display the front-end interface

[0150] @app.route(' / receive_data',methods=['POST'])

[0151] def receive_data():

[0152] data = request.json #Assume that the front end sends JSON data via AJAX

[0153] heart_rate=data.get('heart_rate')

[0154] pulse_rate=data.get('pulse_rate')

[0155] expression=data.get('expression')

[0156] recommend_exercise(heart_rate,pulse_rate,expression)

[0157] #Return the recommended results to the front end for display

[0158] return{'recommended_exercise':'Recommended exercise'}#The actual return value should be determined according to the result of recommend_exercise.

[0159] like Figure 2 As shown, the emotion analysis device based on the smart fitness mirror of the present invention comprises:

[0160] Smart wearable bracelets, cameras and smart fitness mirrors;

[0161] The smart wearable bracelet is used to be worn on the user's wrist to collect the user's pulse and heart rate. The lower end surface of the bracelet body of the smart wearable bracelet is provided with a heart rate monitoring sensor and a pulse sensor. The heart rate monitoring sensor and the pulse sensor are both connected to a controller 1 provided in the smart wearable bracelet, and the controller 1 is also connected to a wireless communication module 1;

[0162] The smart fitness mirror comprises a controller 2, a camera, a wireless communication module 2, an LED light and a display screen arranged on the frame of the smart fitness mirror, the wireless communication module 2, the camera, the LED light and the display screen are all connected to the controller 2, the wireless communication module 2 and the wireless communication module 1 are both connected to the back-end server in the wireless network, and the camera is used to collect the user's facial expression image and / or posture image and transmit it to the controller 2;

[0163] The emotion analysis device based on the smart fitness mirror also includes:

[0164] Collection module 1, which is used for data collection of smart wearable bracelets;

[0165] Collection module 2, which is used for camera data collection;

[0166] Reporting module, which is used to report data to the backend server;

[0167] The recommendation module is used for intelligent sports recommendation. The controller 1 and the controller 2 can be single chip microcomputers, MCU or PLC. The wireless communication module 2 and the wireless communication module 1 can be Bluetooth modules, and the wireless network can be a Bluetooth network.

[0168] In a preferred but non-limiting embodiment of the present invention, the heart rate monitoring sensor and the pulse sensor are used to collect the user's heart rate data and pulse data respectively and transmit them to the controller.

[0169] At the beginning of the design, the appearance of the smart wearable bracelet paired with the smart fitness mirror will mainly follow the existing related bracelet products on the domestic market. The material of the bracelet strap was unanimously selected after trying many materials of straps. The material used is fluorosilicone. Compared with leather and metal materials, this material can give users a soft touch when it fits the skin of the hand. At the same time, it is not easy to sweat compared to leather straps. For people who often wear watches, sweating will make users feel very uncomfortable, and exercise is the easiest to sweat. Therefore, the use of this fluorosilicone bracelet strap will provide a good experience for users with more developed sweat glands during exercise.

[0170] At the same time, compared with metal straps, this material provides convenience in terms of weight and wearing method. First of all, the weight of fluorosilicone is much lower than that of metal straps of the same length and width, which can reduce the user's weight during exercise. At the same time, the wearing of metal straps needs to be adjusted according to the different wrist lengths of different users. Metal straps need to be installed using a strap remover and four pins, which causes unnecessary complexity. Therefore, using a fluorosilicone strap, which can be worn according to the size of the holes, will achieve the benefits of killing two birds with one stone.

[0171] The color of the bracelet strap is matched according to the frame color of the smart fitness mirror selected by the user. There are lake blue, light pink, beige and obsidian color matching corresponding to the frame color.

[0172] The beneficial effects of the present invention are that, compared with the prior art, the technical effects of the present invention include:

[0173] The emotion analysis method and device based on the smart fitness mirror integrates the emotion monitoring function, and judges the user's emotional state by analyzing the user's facial expressions through the camera and monitoring the heart rate through the bracelet. According to the emotional state, the light bar embedded in the frame will emit light feedback of the corresponding color (green represents happiness, blue represents sadness, and yellow represents calmness). In addition, the emotion analysis method and device based on the smart fitness mirror will also recommend appropriate fitness content and music according to the user's mood.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not deviate from the spirit and scope of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for emotion analysis based on a smart fitness mirror, characterized in that: include: Step 1: Data collection of smart wearable bracelet; Step 2: Camera data collection; Step 3: Data is reported to the backend server.

2. The emotion analysis method based on the smart fitness mirror according to claim 1, characterized in that: In step 1, the backend server uses the Python Bluetooth library to establish a connection with the smart wearable bracelet and receives the user's heart rate data and pulse data obtained from the controller in the smart wearable bracelet.

3. The emotion analysis method based on the smart fitness mirror according to claim 2 is characterized in that: The backend server uses the Python Bluetooth library to establish a connection with the smart wearable bracelet and receive the user's heart rate data and pulse data transmitted by the controller in the smart wearable bracelet, specifically including: Step 1-1: Install PyBluez on the backend server; Step 1-2: Connect the wireless communication module of the smart wearable bracelet and receive the user's heart rate data and pulse data transmitted through the controller.

4. The emotion analysis method based on the smart fitness mirror according to claim 3 is characterized in that: Step 1-2 specifically includes: using the PyBluez library to search for and connect to the wireless communication module 1 of the smart wearable bracelet, and then receiving the user's heart rate data and pulse data transmitted through the controller 1.

5. The emotion analysis method based on the smart fitness mirror according to claim 4 is characterized in that: Step 2 specifically includes: Step 2-1: Install OpenCV on the backend server; Step 2-2: The backend server uses OpenCV to capture and process the video.

6. The emotion analysis method based on the smart fitness mirror according to claim 5, characterized in that: Step 2-2 specifically includes: using the VideoCapture class of OpenCV to capture a video, and then obtaining facial expressions from the video, where the video includes a facial expression image of the user.

7. The emotion analysis method based on the smart fitness mirror according to claim 6, characterized in that: Step 3 specifically includes: Step 3-1: Install requests on controller 1 and controller 2; Step 3-2: Controller 1 and controller 2 send data to the server; Step 3-2 specifically includes: Controller 1 and Controller 2 package the collected user's heart rate data, pulse data, and facial expression images into JSON or other formats, and send them to the backend server via HTTP request.

8. The emotion analysis method based on the smart fitness mirror according to claim 7, characterized in that: The emotion analysis method based on the smart fitness mirror also includes: Step 4: Intelligent exercise recommendation; The user's mood is judged based on the user's heart rate data, pulse data collected by the smart wearable bracelet, and facial expressions captured by the camera, and suitable exercises are recommended based on the mood.

9. An emotion analysis device based on a smart fitness mirror, characterized in that: include: Smart wearable bracelets, cameras and smart fitness mirrors; The smart wearable bracelet is used to be worn on the user's wrist to collect the user's pulse and heart rate. The lower end surface of the bracelet body of the smart wearable bracelet is provided with a heart rate monitoring sensor and a pulse sensor. The heart rate monitoring sensor and the pulse sensor are both connected to a controller 1 provided in the smart wearable bracelet, and the controller 1 is also connected to a wireless communication module 1; The smart fitness mirror comprises a controller 2, a camera, a wireless communication module 2, an LED light and a display screen arranged on the frame of the smart fitness mirror. The wireless communication module 2, the camera, the LED light and the display screen are all connected to the controller 2. The wireless communication module 2 and the wireless communication module 1 are both connected to the back-end server in the wireless network. The camera is used to collect the user's facial expression image and transmit it to the controller 2. The emotion analysis device based on the smart fitness mirror also includes: Collection module 1, which is used for data collection of smart wearable bracelets; Collection module 2, which is used for camera data collection; Reporting module, which is used to report data to the backend server; Recommendation module, which is used for intelligent sports recommendation.

10. The emotion analysis device based on the smart fitness mirror according to claim 1, characterized in that: The heart rate monitoring sensor and the pulse sensor are used to collect the user's heart rate data and pulse data respectively and transmit them to the controller.

Images