Electronic system and physiological monitoring method

Abstract

The invention provides an electronic system and a physiological monitoring method. The electronic system comprises a camera, an auxiliary feature sensor and an arithmetic unit. The camera is used for shooting a face image of a user. The auxiliary feature sensor is used for sensing an auxiliary feature of a user. The arithmetic unit is used for calculating the emotional state of the user according to the face image, judging the emotional level of the user in the emotional state by analyzing the auxiliary features when judging that the emotional state accords with a preset emotional state, and judging whether to send out a prompt signal according to the emotional level.

Description

Technical Field

[0001] The present invention relates to an electronic system and a physiological monitoring method, and more particularly to an electronic system and a physiological monitoring method for monitoring a health state. Background Art

[0002] In modern life, people are often under the living conditions of high pressure and long working hours. In this case, how to appropriately remind users so that they can make timely responses or improvements is one of the quite important topics. Summary of the Invention

[0003] The present invention provides an electronic system and a physiological monitoring method, which can monitor the physiological state of a user and issue a prompt when needed.

[0004] The electronic system of the present invention includes a camera, an auxiliary feature sensor, and an arithmetic unit. The camera is used to capture a face image of the user. The auxiliary feature sensor is used to sense the auxiliary features of the user. The arithmetic unit is used to calculate the emotional state of the user based on the face image, and when it is determined that the emotional state meets a preset emotional state, it determines the emotional level of the user in the emotional state by analyzing the auxiliary features, and accordingly determines whether to issue a prompt signal.

[0005] The physiological monitoring method of the present invention includes: capturing a face image of the user; sensing the auxiliary features of the user; and calculating the emotional state of the user based on the face image, and when it is determined that the emotional state meets a preset emotional state, determining the emotional level of the user in the emotional state by analyzing the auxiliary features, and accordingly determining whether to issue a prompt signal.

[0006] Based on the above, the electronic system and the physiological monitoring method of the present invention can continuously monitor the emotions and physiological states of the user, and accordingly determine whether to issue a prompt signal. Brief Description of the Drawings

[0007] Figure 1 A block diagram showing an electronic system 1 according to Embodiment 1 of the present invention is illustrated.

[0008] Figure 2 A flowchart showing the physiological monitoring method according to Embodiment 1 of the present invention is illustrated.

[0009] Figure 3 It is a flowchart of the physiological monitoring method according to Embodiment 1 of the present invention.

[0010] Among them, the reference numerals are explained as follows:

[0011] 1: Electronic system

[0012] 10: Camera

[0013] 11: Auxiliary feature sensor

[0014] 12: Arithmetic unit

[0015] S20 - S22, S30 - S37: Steps Detailed implementation manners

[0016] Figure 1 The block diagram of the electronic system 1 according to the first embodiment of the present invention is shown. As Figure 1 shown, the electronic system 1 includes a camera 10, an auxiliary feature sensor 11, and an arithmetic unit 12. The camera 10 can be used to capture the facial image of the user. The auxiliary feature sensor 11 can be used to sense the auxiliary features of the same user. The arithmetic unit 12 is coupled to the camera 10 and the auxiliary feature sensor 11. The arithmetic unit 12 can be used to calculate the emotional state of the user based on the facial image, and when it is determined that the emotional state meets the preset emotional state, the emotional level of the user in this emotional state is judged by analyzing the auxiliary features, and based on this, it is judged whether to issue a prompt signal.

[0017] Briefly, in some embodiments, the arithmetic unit 12 of the electronic system 1 can first judge the emotional state of the user based on the facial image, and then use the auxiliary features to assist in discriminating the emotional level of the user, and judge whether to issue a prompt signal based on the emotional level in this emotional state.

[0018] Figure 2 The flowchart of the physiological monitoring method according to the first embodiment of the present invention is shown. Figure 2 The shown physiological monitoring method can be applied to Figure 1 the electronic system 1 and be executed by it. Specifically, Figure 2 the physiological monitoring method includes steps S20 - S22.

[0019] In step S20, the electronic system 1 can capture the facial image of the user through the camera 10 and provide it to the arithmetic unit 12. In step S21, the auxiliary feature sensor 11 can sense the auxiliary features of the user. Finally, in step S22, the arithmetic unit 12 of the electronic system 1 can calculate the emotional state of the user based on the facial image, and when it is determined that the emotional state meets the preset emotional state, the emotional level of the user in this emotional state is judged by analyzing the auxiliary features, and based on this, it is judged whether to issue a prompt signal.

[0020] Specifically, in step S22, after receiving the facial image, the arithmetic unit 12 can determine the user's emotional state through image recognition. In some embodiments, the arithmetic unit 12 can pre-store the image features of at least one preset emotional state, and by recognizing whether the facial image conforms to any of the image features, to determine whether the user's facial image belongs to a specific preset emotional state. For example, in order to monitor the user's health status, the image features stored by the arithmetic unit 12 can be, for example, those belonging to states of excitement, overwork, anxiety, anger, etc., or other more agitated emotional states. The arithmetic unit 12 can determine whether the user's emotion falls into these more agitated emotional states by recognizing the facial image.

[0021] Furthermore, when the arithmetic unit 12 determines that the user falls into a preset emotional state, the arithmetic unit 12 can further obtain the auxiliary features sensed by the auxiliary feature sensor 11, so as to determine the emotional level of the user in this emotional state. In some embodiments, the auxiliary feature sensor 11 can be, for example, an infrared camera, used to sense auxiliary features such as facial temperature change signals. Or, the auxiliary feature sensor 11 can be, for example, other wearable or non-wearable sensing devices, which can be used to sense auxiliary features such as at least one of blood pressure, blood oxygen, pulse, body temperature or other similar parameters.

[0022] Taking the facial temperature change as an example, when the arithmetic unit 12 determines based on the facial image that the user is highly likely to be in an excited state, the arithmetic unit 12 can further refer to the facial temperature change signal to determine whether the user's facial temperature change is too high, and then determine whether the user's emotional level in the excited emotion is too high, that is, an overly excited emotional state. Specifically, the arithmetic unit 12 can sense the temperature change of part or all of the user's face, such as the temperature change of areas such as the forehead and cheeks, or sense the average temperature change of the whole face. In some embodiments, when the arithmetic unit 12 determines based on the facial image that the user's emotional state changes to a more agitated emotional state such as excitement or anger, generally the user will have a higher facial temperature, so the arithmetic unit 12 can determine whether the upward change in the user's facial temperature is still within the normal range through the facial temperature change signal. On the contrary, when the arithmetic unit 12 determines that the user's emotional state changes to a more negative emotional state such as depression or overwork, generally the user will have a lower facial temperature, and the arithmetic unit 12 can determine whether the downward change in the user's facial temperature is within the normal range through the facial temperature change signal.

[0023] In some embodiments, the normal range of the above-mentioned facial temperature change signal may have a fixed or variable temperature threshold. For example, the variable temperature threshold may take into account the change in the reference room temperature or other variables. For example, the arithmetic unit 12 may also refer to the rise or fall of the change in the room temperature, and after deducting the change in the facial temperature caused by the change in the room temperature, determine the emotion level based on the deducted facial temperature change, so as to increase the accuracy of determining the emotion level. In this way, the arithmetic unit 12 can determine the emotional state of the user based on the facial image and determine the emotion level of the user based on the auxiliary features.

[0024] In some embodiments, the auxiliary feature used by the arithmetic unit 12 to determine the emotion level may also be one of the physiological change signals such as blood pressure, blood oxygen, pulse, body temperature, etc. Specifically, similar to the process of determining the emotion level based on the facial temperature change described above, the arithmetic unit 12 can determine the emotion level of the user based on whether the physiological change signal exceeds the preset range. For example, the normal range of the user's pulse rate should fall between 60 and 100 beats per minute. The arithmetic unit 12 can determine the emotion level of the user by judging whether the user's pulse exceeds the normal range, or whether the change in the pulse rate increases or decreases too much.

[0025] Next, when the arithmetic unit 12 determines that the emotion of the user is too high or too low based on the auxiliary features, the arithmetic unit can determine and send a prompt signal accordingly. In some embodiments, the arithmetic unit 12 can remind the user in the form of, for example, image display, sound emission, vibration emission, etc. Or, the prompt signal can be sent by the electronic device 1 to the user's emergency contact through the network. In this way, based on the calculated emotional state and the corresponding emotional degree, the electronic device 1 can determine whether the physiological state of the user endangers health and can send a corresponding prompt signal according to the critical degree.

[0026] In some embodiments, the arithmetic unit 12 may be disposed, for example, integrally on the motherboard, or separately in a keyboard, a mouse, a connector, or a peripheral device. In some embodiments, the arithmetic unit 12 may be, for example, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Arithmetic Logic Unit (ALU), a Field Programmable Gate Array (FPGA), any other type of integrated circuit, a state machine, a processor based on an Advanced RISC Machine (ARM), or other similar components or a combination of the above components. A pre-trained artificial intelligence (AI) model may be stored in the arithmetic unit 12. The artificial intelligence model may be trained by a data set formed by multiple pre-labeled facial images of emotional states, so that the artificial intelligence model can classify subsequent input facial images to determine whether the facial image falls into any of the preset emotional states. The arithmetic unit 12 may perform image recognition on the facial image by executing the artificial intelligence model to determine the emotional state of the user. For example, the artificial intelligence model stored in the arithmetic unit 12 may be, for example, a machine learning model of supervised or unsupervised learning, or a deep learning model under the above classification, such as a Region-based Convolutional Neural Networks (RCNN) series, such as RCNN, Fast RCNN, Mask RCNN, and Faster RCNN, SSD, and YOLO (You only look once, YOLO) series (e.g., YOLO, YOLOv2, YOLO 9000, and YOLOv3), etc. Further, the above artificial intelligence model may be constructed based on a decision tree, logistic regression, SVM (support vector machine), naive Bayes, kNN, or other suitable algorithms. In addition, for the aforementioned deep learning model, the model architecture can also be spliced, increased, decreased, adjusted, and modified to become a custom deep learning model by referring to its image preprocessing, model architecture, model depth, convolutional layer parameters, pooling layer, activation function, loss function, specific architecture of each model, design concept, accuracy, and execution speed, etc.

[0027] In some embodiments, the auxiliary feature sensor 11 may have one or more sensors according to the requirements of the electronic system 1. For example, when the arithmetic unit 12 needs to evaluate the emotion level based on the facial temperature change signal, the auxiliary feature sensor 11 may include, for example, an infrared camera for capturing and presenting the temperature change of the user's face. When the arithmetic unit 12 determines the emotion level based on physiological change signals such as blood pressure, blood oxygen, pulse, body temperature, or other similar signals, the auxiliary feature sensor 11 may be a wearable sensing device such as a watch or a bracelet, or a non-wearable sensing device such as a mouse, a mouse pad, or a headset.

[0028] Figure 3 FIG. is a flowchart of a physiological monitoring method according to an embodiment of the present invention. Figure 3 The physiological monitoring method shown can be applied to Figure 1 the electronic system 1 and is executed by the same. Specifically, Figure 3 the physiological monitoring method includes steps S30 to S37.

[0029] In step S30, the camera 10 may capture the facial image of the user and provide it to the arithmetic unit 12. In step S31, the auxiliary feature sensor 11 may be, for example, an infrared temperature sensor, which can be used to sense the facial temperature of the user and provide a facial temperature change signal to the arithmetic unit 12 accordingly. In step S32, in addition to the above-mentioned infrared temperature sensor, the auxiliary feature sensor 11 may also have other wearable or non-wearable sensing instruments for sensing physiological change signals of at least one of the user's physiological parameters such as blood pressure, blood oxygen, pulse, body temperature, or other similar parameters, and providing them to the arithmetic unit 12.

[0030] In step S33, the arithmetic unit 12 may determine the emotional state of the user based on the facial image. For example, the arithmetic unit 12 may identify the lines in the facial image by means of image recognition to determine the emotional state of the user, whether it is in an excited, overworked, anxious, angry, or other more agitated emotional state, or in a depressed, overworked, or other more negative emotional state. In these emotional states, the arithmetic unit 12 may further determine the emotion level of the user in that emotion based on the facial temperature change signal and the physiological change signal.

[0031] In step S34, the arithmetic unit 12 evaluates whether the user's emotional state and emotional level reach the first prompt level or the second prompt level, and accordingly determines whether to enter step S35 to issue a first prompt signal, or enter steps S36 to S37 to issue a second prompt signal, or continue to monitor the user's emotional state, facial temperature change, and physiological condition when neither prompt level is reached. Specifically, the arithmetic unit 12 can judge the user's emotional state based on the facial image, and judge the user's emotional level in this emotional state based on the facial temperature change signal. When the arithmetic unit 12 determines that the user's emotional level falls within the first range, which may include a relatively high emotional level (for example, higher than the first upper limit) or a relatively low emotional level (for example, lower than the first lower limit), the arithmetic unit 12 can determine that the first prompt level is reached and enter step S35 to instruct the issuance of a first prompt signal.

[0032] Furthermore, the arithmetic unit 12 can also use the physiological change signal as an auxiliary to the facial temperature change signal to further confirm whether the user's physiological state reaches the relatively urgent second prompt state. For example, when the arithmetic unit 12 determines based on the facial temperature change signal that the user's emotional level falls within the second range, which may include an overly high emotional level (for example, higher than the second upper limit, and the second upper limit is greater than the first upper limit) or an overly low emotional level (for example, lower than the second lower limit, and the second lower limit is less than the first lower limit), the arithmetic unit 12 can further refer to whether the physiological change signal exceeds the preset range. When the arithmetic unit 12 determines that the user's emotional level is too high or too low, and the physiological change signal exceeds the preset range, the arithmetic unit 12 can accordingly determine that the user's emotional level can reach the second prompt level, indicating that the user's physiological state reaches a relatively critical situation, and accordingly enter step S36 to attempt to issue a second prompt signal.

[0033] In step S36, the electronic system 1 can evaluate whether its network function is enabled or whether it has the permission to set the network function to be enabled, and when the network function or the permission to enable the network function exists, enter step S37 to send a second prompt signal to the user's emergency contact through the network. When the evaluation result is negative, the arithmetic unit 12 abandons sending the second prompt signal.

[0034] In summary, the electronic system and physiological monitoring method of the present invention can instantaneously monitor the user's physiological state, issue a prompt signal when the user's physiological state is critical, effectively issue a prompt for the critical state caused by the user's overwork, excessive use of electronic devices, or stress, so that the user can make corresponding emergency responses.

Claims

1. An electronic system, characterized in that, Comprising: A camera for capturing a facial image of a user; An auxiliary feature sensor for sensing an auxiliary feature of the user; And An arithmetic unit for calculating an emotional state of the user based on the facial image, and when it is determined that the emotional state conforms to a preset emotional state, judging an emotional level of the user in the emotional state by analyzing the auxiliary feature, and accordingly judging whether to issue a prompt signal.

2. The electronic system according to claim 1, characterized in that The arithmetic unit identifies the facial image through an image recognition technology to judge the emotional state of the user.

3. The electronic system according to claim 2, characterized in that, The arithmetic unit performs the image recognition technology on the facial image by executing an artificial intelligence model to judge the emotional state of the user.

4. The electronic system according to claim 1, wherein The auxiliary feature includes a facial temperature change signal, and the arithmetic unit judges the emotional level of the user in the emotional state by analyzing the facial temperature change signal of the user.

5. The electronic system according to claim 1, characterized in that, The auxiliary feature includes a physiological change signal, and the physiological change signal includes at least one of blood pressure, blood oxygen, pulse, and body temperature. The arithmetic unit judges the emotional level of the user in the emotional state by analyzing the physiological change signal.

6. The electronic system according to claim 4, wherein The auxiliary feature further includes a physiological change signal. After the arithmetic unit judges the emotional level of the user by analyzing the facial temperature change signal, it further analyzes the physiological change signal to verify the emotional level.

7. The electronic system according to claim 1, wherein When the arithmetic unit judges that the emotional level of the user falls into a first range, the arithmetic unit instructs a first prompt device to issue a first prompt signal. Wherein when the arithmetic unit judges that the emotional level of the user falls into a second range, after confirming that a network setting of the electronic system is turned on, the arithmetic unit issues a second prompt signal through the network.

8. The electronic system according to claim 1, wherein The arithmetic unit is disposed on a main board or a connector, or is disposed in an edge device connected through a connection terminal or a transmission line.

9. A physiological monitoring method, characterized in that, Comprising: Capturing a facial image of a user; Sensing an auxiliary feature of the user; And Calculating an emotional state of the user based on the facial image, and when it is determined that the emotional state conforms to a preset emotional state, judging an emotional level of the user in the emotional state by analyzing the auxiliary feature, and accordingly judging whether to issue a prompt signal.

10. The physiological monitoring method according to claim 9, wherein Comprising: Identifying the facial image through an image recognition technology to judge the emotional state of the user.

11. The physiological monitoring method according to claim 10, wherein, Comprising: Performing the image recognition technology on the facial image by executing an artificial intelligence model to judge the emotional state of the user.

12. The physiological monitoring method according to claim 9, wherein The auxiliary feature includes a facial temperature change signal, and the physiological monitoring method includes: Judging the emotional level of the user in the emotional state by analyzing the facial temperature change signal of the user.

13. The physiological monitoring method according to claim 9, characterized in that, The auxiliary feature includes a physiological change signal, and the physiological change signal includes at least one of blood pressure, blood oxygen, pulse, and body temperature. The physiological monitoring method includes: Judging the emotional level of the user in the emotional state by analyzing the physiological change signal.

14. The physiological monitoring method according to claim 12, wherein The auxiliary feature further includes a physiological change signal, and the physiological monitoring method includes: After analyzing the facial temperature change signal to determine the user's emotional level, then analyze the physiological change signal to verify the emotional level.

15. The physiological monitoring method according to claim 9, wherein, Including: When it is determined that the user's emotional level falls within a first range, instruct to emit a first prompt signal. When it is determined that the user's emotional level falls within a second range, after confirming that a network setting is enabled, send a second prompt signal through the network.

Images