Method, apparatus and system for adaptively regulating surrounding temperature of area using facial expression recognition

Abstract

The present disclosure provides a method, an apparatus and a system for adaptively regulating a surrounding temperature of an area using facial expression recognition, the method comprising: computing a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level comprises a valence level and an arousal level of the person; determining if the core affective state level matches a target core affective state level set for the area; and in response to determining that the core affective state level does not match the target core affective state level set for the area, deriving an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level.

Description

METHOD, APPARATUS AND SYSTEM FOR ADAPTIVELY REGULATING SURROUNDING TEMPERATURE OF AREA USING FACIAL EXPRESSION RECOGNITION

[0001] The present invention relates broadly, but not exclusively, to a method and an apparatus for adaptively regulating a surrounding temperature of an area using facial expression recognition.

[0002] Sufficient temperature regulation is essential to ensure the thermal comfort of employees, which, in turn, has a significant impact on both their well-being and productivity. According to guidelines from the US Occupational Safety and Health Administration (OSHA), it is recommended that employers maintain the thermostat within the range of 20°C to 24°C. Research has indicated that individual preferences for thermal comfort can vary based on factors such as age, gender, cultural background, the nature of the activity, clothing choices, and more, all of which contribute to a person's perception of comfort.

[0003] Research has also indicated that the core affect or emotional state of an individual can be influenced by thermal comfort, potentially influencing their overall work productivity and well-being. Additionally, thermal sensation can be influenced by various factors, including temperature and the surrounding environment. Currently, office spaces are equipped with heating, ventilation, and air conditioning (HVAC) systems but they rely on manual control and monitoring by occupants or persons occupying an area to control room / surrounding temperature of the area, as well as lighting, humidity, and fan speed, all in pursuit of achieving an optimal level of comfort.

[0004] Therefore, there is a need for a method, an apparatus and a system for adaptively regulating a surrounding temperature of an area, which includes some form of feedback mechanism that determines the thermal comfort and emotional / mental states currently experienced by each occupant in the area and adjusts the surrounding temperature to an ideal temperature for improved thermal comfort and emotional / mental states, which in turn, improves productivity and well-being of the occupants.

[0005] Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

[0006] In a first aspect, the present disclosure provides a method for adaptively regulating a surrounding temperature of an area, comprising: computing a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level is represented by a valence level and an arousal level of the person; determining if the core affective state level matches a target core affective state level set for the area; and in response to determining that the core affective state level does not match the target core affective state level set for the area, deriving an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level.

[0007] In a second aspect, the present disclosure provides an apparatus for adaptively regulating a surrounding temperature of an area, the apparatus comprising: at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: compute a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level is represented by a valence level and an arousal level of the person; determine if the core affective state level matches a target core affective state level set for the area; and in response to determining that the core affective state level does not match the target core affective state level set for the area: derive an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level.

[0008] In a third aspect, the present disclosure provides a system for adaptively regulating a surrounding temperature of an area, the system comprises the apparatus according to the second aspect and an image capturing apparatus for detecting a facial expression of a person.

[0009] Additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. The benefits and / or advantages may be individually obtained by the various embodiments and features of the specification and drawings, which need not all be provided in order to obtain one or more of such benefits and / or advantages.

[0010] The accompanying Figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to illustrate various embodiments and to explain various principles and advantages in accordance with a present embodiment, by way of non-limiting example only.

[0011] Embodiments of the invention will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:Fig. 1 shows a block diagram illustrating a system which includes an apparatus for adaptively regulating a surrounding temperature of an area and an image capturing device for detecting a facial expression of a person according to various embodiments of the present disclosure.Fig. 2 shows a flow chart illustrating a method for adaptively regulating a surrounding temperature of an area according to various embodiments of the present disclosure.Fig. 3 shows a flow diagram illustrating an overview of the process for adaptively regulating a surrounding temperature of an area according to various embodiments of the present disclosure.Fig. 4 shows a flow diagram illustrating the process of Fig. 3 in detail.Fig. 5 shows a schematic diagram illustrating a camera configured to carry facial expression recognition (FER) to generate various FER outputs according to an embodiment of the present disclosure.Fig. 6 shows a graph 600 illustrating an exemplary valence-arousal space and an exemplary core affective state level in the valence-arousal space.Fig. 7A shows a graph illustrating an exemplary valence-arousal space and exemplary reference core affective state levels of seven basic emotions in the valence-arousal space.Fig. 7B shows a graph illustrating an exemplary valence-arousal space divided into twelve reference core affective state levels in the valence-arousal space.Fig. 8 shows a flow diagram illustrating a process carried out by a person role detection unit according to an embodiment of the present disclosure.Fig. 9 shows a flow diagram illustrating a process carried out by an average weightage score computation unit according to an embodiment of the present disclosure.Fig. 10A shows a bar chart illustrating a target concentration level and a target satisfaction level according to an embodiment of the present disclosure.Fig. 10B shows a graph illustrating a target core affective state level in a valence-arousal vector space according to an embodiment of the present disclosure.Fig. 11 shows a flow diagram illustrating a process carried out by a core affect, concentration and satisfaction adjustment unit according to an embodiment of the present disclosure.Fig. 12A shows a graph illustrating an exemplary valence-arousal space according to an embodiment of the present disclosure.Fig. 12B shows a graph illustrating another exemplary valence-arousal space according to an embodiment of the present disclosure.Fig. 13 show an exemplary current core affective state level (emotion status), concentration level and satisfaction level of a person in an area obtained from a FER preprocessor and an exemplary target core affective state level, concentration level and satisfaction level set for the area obtained from an area / activity type emotion profile, respectively, according to an embodiment of the present disclosure.Fig. 14 show an exemplary current core affective state level (emotion status), concentration level and satisfaction level of a person in an area obtained from a FER preprocessor and an exemplary target core affective state level, concentration level and satisfaction level set for the area obtained from an area / activity type emotion profile, respectively, according to an embodiment of the present disclosure.Fig. 15 shows a flow diagram illustrating a process carried out by a stepwise temperature adjustment unit to effect core affect adjustment, according to an embodiment of the present disclosure.Fig. 16 shows a flow diagram illustrating a process carried out by a stepwise temperature adjustment unit to effect concentration / satisfaction affect adjustment, according to an embodiment of the present disclosure.Fig. 17 shows a bar chart illustrating a target concentration level, a target satisfaction level and a graph illustrating a core affective state level in a valence-arousal vector space for a serious discussion in a collaboration space use case according to an embodiment of the present disclosure.Fig. 18 shows a bar chart illustrating a target concentration level, a target satisfaction level and a graph illustrating a core affective state level in a valence-arousal vector space for classroom use case according to an embodiment of the present disclosure.Fig. 19 shows a bar chart illustrating a target concentration level, a target satisfaction level and a graph illustrating a core affective state level in a valence-arousal vector space for a celebratory event in a conference use case according to an embodiment of the present disclosure.Fig. 20 shows a bar chart illustrating a target concentration level, a target satisfaction level and a graph illustrating a core affective state level in a valence-arousal vector space for a relaxation space use case according to an embodiment of the present disclosure.Fig. 21 shows a bar chart illustrating a target concentration level, a target satisfaction level and a graph illustrating a core affective state level in a valence-arousal vector space for a long road trip in an in-car use case according to an embodiment of the present disclosure.Fig. 22 shows a schematic diagram of an exemplary computing device suitable for use to execute the method in Fig. 2 and implement the apparatus in Fig. 1.

[0012] Embodiments of the present invention will be described, by way of example only, with reference to the drawings. Like reference numerals and characters in the drawings refer to like elements or equivalents.

[0013] Some portions of the description which follows are explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to convey most effectively the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0014] Unless specifically stated otherwise, and as apparent from the following, it will be appreciated that throughout the present specification, discussions utilizing terms such as "receiving", "calculating", "determining", "updating", "generating", "initializing", "outputting", "retrieving", "identifying", "dispersing", "authenticating" or the like, refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system or other information storage, transmission or display devices.

[0015] The present specification also discloses apparatus for performing the operations of the methods. Such apparatus may be specially constructed for the required purposes, or may comprise a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various machines may be used with programs in accordance with the teachings herein. Alternatively, the construction of more specialized apparatus to perform the required method steps may be appropriate. The structure of a computer will appear from the description below.

[0016] In addition, the present specification also implicitly discloses a computer program, in that it would be apparent to the person skilled in the art that the individual steps of the method described herein may be put into effect by computer code. The computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be used to implement the teachings of the disclosure contained herein. Moreover, the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows without departing from the spirit or scope of the invention.

[0017] Furthermore, one or more of the steps of the computer program may be performed in parallel rather than sequentially. Such a computer program may be stored on any computer readable medium. The computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a computer. The computer readable medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM mobile telephone system. The computer program when loaded and executed on such a computer effectively results in an apparatus that implements the steps of the preferred method.

[0018] Various embodiments of the present disclosure relate to a method, an apparatus and a system for adaptively regulating a surrounding temperature of an area using facial expression recognition.

[0019] In various embodiments below, the term "core affective state level" may be used to generally describe an emotion or an emotional state of a person, while the term "core affect" in the present disclosure is understood to mean emotion accordingly. The terms "core affect" and "emotion" may be used interchangeably. A core affective state level is detected through facial expression recognition and may be represented by a valence level and an arousal level. The valence level represents the extent to which an emotion is positive or negative, ranging from pleasant to unpleasant, and the arousal level represents the state of heightened physiological activity, ranging from activation to deactivation. In various embodiments of the present disclosures, such core affective state level may be represented as a vector in a valence-arousal vector space 600, as illustrated in Fig. 6, where the valence level represents the horizontal axis of the space and the arousal level represents the vertical axis of the space and such vector space may be known as Circumplex Model. A core affective state level in the space is used to represent an emotion of a person with a mix of a certain level of pleasantness / unpleasantness (valence level) of the person and a certain level of activation / deactivation (arousal level) of the person.

[0020] The term "concentration level" may be referred to as "concentration score"; and the term "satisfaction level" may be referred to as "satisfaction score". In the present disclosure, a person is detected through a facial recognition while the emotional state, concentration level and satisfaction level are detected through facial expression recognition.

[0021] The term "area" in which the surrounding temperature is regulated, and a person is located or detected for adaptively regulating the surrounding temperature may be referred "area of interest".

[0022] Fig. 1 shows a block diagram 100 illustrating a system which includes an apparatus 104 for adaptively regulating a surrounding temperature of an area and an image capturing device 102 for detecting a facial expression of a person according to various embodiments of the present disclosure.

[0023] The managing of image or video input is performed by at least one image capturing device 102 and an apparatus 104. For the sake of simplicity, only one image capturing device 102 is illustrated. The system 100 comprises an image capturing device 102 in communication with the apparatus 104. In an implementation, the apparatus 104 may be generally described as a physical device comprising at least one processor 106 and at least one memory 108 including computer program code. The at least one memory 108 and the computer program code are configured to, with the at least one processor 106, cause the physical device to perform the operations described in Fig. 2. The processor 106 is configured to receive one or more images or videos from the image capturing device 102 or retrieve one or more images or videos from a database. Alternatively or additionally, the one or more images or videos captured by the image capturing device 102 is stored in a database 110, and the processor 106 is configured to retrieve the one or more images or videos from the database 110. It should be appreciated that the image capturing device 102 may be a part of the apparatus 104, forming a system 100 to perform operations described in Fig. 2.

[0024] The image capturing device 102 may be a device such as a mobile phone camera which provides a variety of data such as data relating to a facial and / or body feature and / or a movement of the facial and / or body feature of a person. In an implementation, appearance data derived from the video capturing device 102 may be stored in memory 108 of the apparatus 104 or a database 110 accessible by the apparatus 104. The data may include (i) facial and body feature data such as relative position, size, shape and / or contour of eyes, nose, cheekbones, jaw, chin, neck, shoulder, arm, iris pattern, skin colour, hair colour or a combination thereof, (ii) physical characteristic data such as height, body size, body ratio, shoulder width, distance between two facial and body features, length of limbs, hair colour, skin colour, apparels, belongings, equipment, other similar characteristics or combinations, and (iii) behavioral characteristic data such as movement, position of limbs, position of apparel / belonging / equipment, direction of movement, differential in movement direction, moving speed, frequency, movement patterns, the way or the time period a person or his / her facial and body feature stay stills or moves, other similar characteristics or combinations. In various embodiments of the present disclosure, the data may be used to perform facial expression recognition (FER) to obtain emotion scores (hereinafter may refer to "FER for emotion" or "emotion FER"), concentration scores (hereinafter may refer to "FER for concentration" or "concentration FER"), and satisfaction scores (hereinafter may refer to "FER for satisfaction" or "satisfaction FER") based on the facial expressions and facial recognition (FR) to identify a known person.

[0025] In an implementation, camera data such as location and resolution, and / or time data which includes a timestamp at which the person or his / her facial or body feature is identified may also be derived from the image capturing device 102. The camera data and / or time data may be stored in memory 108 of the apparatus 104 or a database 110 accessible by the apparatus 104 and the processor 106 is configured to identify and retrieve data, image or video based on the time data. Additionally, other data such as location type (area type) and a type of activity (activity type) may also be stored in memory 108 of the apparatus 104 or a database 110 accessible by the apparatus 104, and the processor 106 is configured to identify and retrieve such data for adaptively regulating the surrounding temperature. It is appreciated that such data relating to an area type and an activity type may be derived from the image capturing device 102. It should be appreciated that the database 110 may be a part of the apparatus 104.

[0026] The apparatus 104 may be configured to communicate with the image capturing device 102 and the database 110. In an example, the apparatus 304 may receive, from the image capturing device 102, or retrieve from the database 110, one or more images of a person in a location (area) or the data associated with the images.

[0027] As shown in the exemplified method for adaptively regulating a surrounding temperature of an area in Fig. 2, the memory 108 and the computer program code stored therein are configured to, with the processor 106 cause the apparatus 104 to configured to perform the following steps: - Step 202: computing a core affective state of a person in an area based on a detected facial expression of the person, wherein the core affective state is represented by a valence level and an arousal level of the person; - Step 204: determining if the core affective state level matches a target core affective state level set for the area; and - Step 206: deriving an adjustment to a surrounding temperature of the area based on a first adjustment required to match the core affective state level to the target core affective state level.

[0028] In one embodiment, the memory 108 and the computer program code stored therein are configured to, with the processor 106 further cause the apparatus 104 to: in step 202, obtain a mental state level of the person based on the detected facial expression of the person, wherein the mental state level comprises one of a concentration level and a satisfaction level of the person; in step 204, determining if the mental state level matches a target mental state level set for the area in response to determining that the core affective state level matches the target core affective state level set for the area; and, in step 206, in response to determining that the mental state level does not match a target mental state level set for the area: calculating the adjustment to the surrounding temperature based on a second adjustment required to match the mental state level to the target mental state level and core affective state level to the target core affective state level.

[0029] It is noted that such core affective state level may be an average core affective state level of a plurality of core affective state levels computed based on a plurality of detected facial expressions of the person over a time period. Similarly, the mental state level may also be an average mental state level of a plurality of mental state levels obtained based on the plurality of detected facial expressions of the person over the time period.

[0030] In one embodiment, in step 204, the memory 108 and the computer program code stored therein are configured to, with the processor 106 cause the apparatus 104 to further identify one of an area type and an activity type of the area, and setting one of the target core affective state level and the target mental state level for the area based on the one of the area type and the activity type of the area.

[0031] Additionally, in step 204, the memory 108 and the computer program code stored therein are configured to, with the processor 106 cause the apparatus 104 to further detect a role identifier relating to the person based on the detected facial feature of the person, and apply a weightage corresponding to the role identifier to one of the core affective state level and the mental state level of the person, wherein the determination of the one of the core affective state level and the mental state level is based on the weighted one of the core affective state level and the mental state level.

[0032] In another embodiment, in step 206, the memory 108 and the computer program code stored therein are configured to, with the processor 106 cause the apparatus 104 to further classify one of the first adjustment and the second adjustment, and derive the adjustment to the surrounding temperature based on a result of the classification.

[0033] Yet in another embodiment, in step 206, the memory 108 and the computer program code stored therein are configured to, with the processor 106 cause the apparatus 104 to further set a number of adjustments of and an interval between each adjustment step to effect the adjustment to the surrounding temperature. Additionally or alternatively, the memory 108 and the computer program code stored therein are configured to, with the processor 106 cause the apparatus 104 to further classify (i) the surrounding temperature, and / or (ii) the adjustment to the surrounding temperature, and set the number of adjustment steps and the interval between the each adjustment step based on a result of the classification(s).

[0034] Fig. 3 shows a flow diagram 300 illustrating an overview of the process for adaptively regulating a surrounding temperature of an area using facial expression recognition according to various embodiments of the present disclosure. Fig. 4 shows a flow diagram 400 illustrating the process of Fig. 3 in detail.

[0035] According to the present disclosure, an apparatus for adaptively regulating a surrounding temperature may comprise a facial expression recognition (FER) preprocessor 302, 402 and a feedback controller 304, 404. The FER preprocessor 302, 402 is configured to compute an average weightage value from intensity scores (emotion, concentration, satisfaction) derived from FER and FR of different persons / occupants in the area of interest while the feedback controller 304, 404 is configured to load the preset targeted values for the emotion, concentration and satisfaction scores to generate the optimal temperature to be adjusted.

[0036] In particular, the FER preprocessor 302, 402 may be configured to receive outputs from FER for emotions 412, FER for concentration 416 and FER for satisfaction 418 and facial recognition (FR) 414 and provide an output including a core affective state level of a person(s) in an area, for example, his / her weighted average core affective state level, and his / her mental state level, for example, weighted average satisfaction score and concentration score, to the feedback controller 304, 404. The feedback controller receiving the output from the FER preprocessor 302, 402 is configured to (i) load an area / activity profile, for example, from a database 436, which includes an area type and an activity type of the area and their associated target core affective / mental state levels, and (ii) set a temperature, i.e., derive an adjustment to a surrounding temperature based on an adjustment required to match the current core affective / mental state levels of the person in the area with the target core affective / mental state levels. The HVAC 306, 406 is then configured to regulate and make the adjustment to the surrounding temperature. In an embodiment, the processor 106 in the apparatus 102 may be configured to function as the FER preprocessor 302, 402 and the feedback controller 304, 404.

[0037] Fig. 5 shows a schematic diagram 500 illustrating a camera 510 configured to carry FER 512, 516, 518 to generate various FER outputs 512a, 516a, 518a according to an embodiment of the present disclosure. In particular, the camera 310, 510 may be configured to carry out FER for emotion 512 to generate an emotion FER output 512a, FER for concentration 516 to generate a concentration FER output 516a and FER for satisfaction 518 to generate a satisfaction FER output 518.

[0038] The FER output is provided for each detected face in the camera image as intensity score from 0 to 1. Result is an array of FER scores for emotion, concentration and satisfaction. Exemplary FER outputs for emotional, concentration and satisfaction are shown in blocks 512a, 516a, 518a, respectively. For example, the emotion FER output 512a is {'calm':0.86} indicating an emotion score or intensity of 0.86 and an associated emotion of 'calm'; the concentration FER output 516a is a concentration score of 0.42; the satisfaction FER output 518 is a satisfaction score of 0.63. Such FER outputs 512a, 516a, 518a will then be used by FER preprocessor 302, 402 for further processing.

[0039] The FER output for emotion 512a from each facial expression of a person is then used to compute a core affective state level represented by two primary dimensions, i.e., a valence level and an arousal level, for example, according to "The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology" authored by James A. Russell et. al. published in 2005. The valence level represents the extent to which an emotion is positive or negative, ranging from pleasant to unpleasant, and the arousal level represents the state of heightened physiological activity, ranging from activation to deactivation. In various embodiments of the present disclosures, such core affective state level may be represented as a vector in a valence-arousal vector space 600, as illustrated in Fig. 6, where the valence level represents the horizontal axis of the space and the arousal level represents the vertical axis of the space and such vector space may be known as Circumplex Model.

[0040] The valence level and arousal level of the facial expression can be computed by a valence and arousal computation unit 422 of the FER preprocessor 302, 402 using the emotion score and identified emotion from the FER output based on equations (1) and (2) below, respectively, and a reference angle associated with each emotion. According to Universal Facial Expressions of Emotion authored by Ekman P. published in 1970, six basic human emotions -- Happy, Sad, Fear, Anger, Disgust and Surprised, were identified; additionally, there can be a 7themotional state, which is a neutral or calm state that does not fall into any of the categories mentioned above. In one example, seven basic emotions -- Happy, Sad, Fear, Anger, Disgust, Surprised and Calm (Neutral) are being categorized, and each basic emotion may be associated with certain valence and arousal levels. In the form of the vector space, each basic emotion is also associated with a reference vector (or segment) having a reference angle θ in the valence-arousal vector space 700, as illustrated in Fig. 7A. If a basic emotion is identified from the FER output, its reference angle will be applied to equations (1) and (2) to compute the valence and arousal levels.

[0041]

[0042] For example, if the identified emotion from a facial expression is 'Calm' with an intensity score of 0.86. Its reference angle, for example θ=300°according to Fig. 7A, will be applied. In this case, the valence level is computed to be 0.86×cos300° or 0.43 and the arousal level is computed to be 0.86×sin300° or -0.74.

[0043] Alternatively, as illustrated in Fig. 7B, the vector space 710 may be divided into 12 equal segments, each representing a different core affect classification associated with different valence (pleasant) level and arousal (activation) level and a reference angle θ in the vector space 710.

[0044] Fig. 8 shows a flow diagram 800 illustrating a process carried out by a person role detection unit, for example, the person role detection unit 424 in Fig. 4, according to an embodiment of the present disclosure. The person role detection unit 424 may receive, from the valence and arousal computation unit 402, an input of core affective state levels (valence and arousal levels) (herein referred as emotion FER input) as well as an input of person ID derived from FR output 414 (herein referred as "FR input"). Each emotion FER input corresponds to a FR input as they originated from the same detection. In step 802, the person role detection unit 424 may retrieve an area type and an activity type associated with the area from the area / activity type emotion profile database 436 to determine the roles involved. In step 804, the person role detection unit 424 may look up the person profile database 432 and retrieve a role or role identifier associated with the person ID according to the area type and the activity type, and tag the emotion FER input with the role of the person involved in this area / activity type. In step 806, a step of determining whether there is any more emotion FER input (valence and arousal levels). If there is more emotion FER input, step 808 is carried out where the next FER emotion input and its corresponding FR input are obtained and step 804 is repeated based on the next FER emotion and FR input. If there is no more emotion FER input, the process may end and the emotion FER input, i.e., core affective state levels, tagged with person role identifiers will then be output for further processing.

[0045] Table 1 shows an example of area type, activity type and roles stored in the area / activity type emotion profile database 436.

[0046] (Table 1)

[0047] Table 2 shows an example person ID and its assigned role stored in the person profile database 432.

[0048] (Table 2)

[0049] Table 3 shows a sample output from the person role detection unit 424.

[0050] (Table 3)

[0051] Fig. 9 shows a flow diagram 900 illustrating a process carried out by an average weightage score computation unit, for example, the average weightage score computation unit 426 in Fig. 4, according to an embodiment of the present disclosure. In an example, valence, arousal, concentration and satisfaction levels across all faces of a same person detected over a time period are used to calculate respective average levels or mean levels (scores) for the valence, arousal, concentration and satisfaction levels obtained over the time period, and such average levels or mean levels will then output for further processing. This averaging process extends over a period of time to enhance the reliability of the scores because core affective states, Satisfaction, and Concentration require time to reach a stable condition. Also, through the averaging procedure, momentary emotions that may not precisely represent the stable core affective state will gradually fade or become less significant over time. For instance, sporadic stimuli such as someone telling a joke constitute only temporary emotional fluctuations.

[0052] Additionally, a weightage assigned to a role may be applied to valence, arousal, concentration and satisfaction levels to obtain respective weighted valence, arousal, concentration and satisfaction levels (or scores) respectively. The thermal comfort of a smaller group of occupants who play a more crucial role in the event holds greater significance than that of a larger number of other occupants. Higher weightage will be given to more important roles when averaging. For example, in a classroom, a teacher will have precedence over students. In a conference, the panel of discussion participants have precedence over audience.

[0053] Returning to Fig. 9, the average weightage score computation unit 426 may receive an input of core affective state levels with valence and arousal levels tagged with role identifiers from person role detection unit 424 and an input of concentration scores and satisfaction scores from concentration FER 416 and satisfaction FER 418. In step 902, the FER record containing valence and arousal levels and concentration and satisfaction scores obtained from the same FER are collated. In step 904, the average weight score computation unit 426 may retrieve weightages from the role weights storage 434 and assign the weightages to the FER record input according to the tagged role. In step 906, it is determined if there are any more FER records. If there are more FER records, step 908 where the average weightage score computation unit 426 obtains the next FER record and repeat step 902. If there is no more FER record, step 910 is carried out, where it is determined whether the data has been collated for at least 2 minutes. If the data has been collated for at least 2 minutes, step 912 is carried out where the average weight score is computed. In one example, the average weightage score computation unit 426 may calculate the average valence levels, average arousal levels, average concentration scores and average satisfaction scores from various valence levels, arousal levels, concentration scores and satisfaction scores obtained from FER of a same person / role over a time period using equations in Table 5; and then apply weightages to the average levels and scores using equations (3)-(6). In step 914, it is determined whether there are any more roles in the records. If there are more roles in the records, the next FER record is obtained in step 916; otherwise the average weightage score computation unit 426 may output the average weight scores for further processing. It is appreciated that the 2 minutes time in the determination step 910 is a pre-configurable time interval, and other time interval such as 1 minute, 20 seconds or 3 hours may be used instead. The purpose of the time interval is to batch up enough data before computing the average weight scores so as to enhance reliability.

[0054] Table 4 shows a sample FER record of the average weightage score computation unit 426 where the weightage of each role is retrieved and assigned. In this case, the supervisor is assigned 50% weightage and the worker is assigned 50% weightage.

[0055] (Table 4)

[0056] Table 5 shows the equations to calculate an average valence level, an average arousal level, an average concentration level and an average satisfaction level according to an embodiment of the present disclosure.

[0057] (Table 5)

[0058]

[0059] Returning to Fig. 4, the feedback controller unit 404 comprises an emotion profile loading unit 442 configured to load emotion profile comprising the target core affective state level (e.g., target valence level, target arousal level) and target mental state level (e.g., target satisfaction level, target concentration level) from the area / activity type emotion profiles corresponding to the area / activity type of the area of interest. Such target levels are then output and then used by core affect, concentration and satisfaction adjustment unit 444 to determine the adjustments required to match the core affective state level to the target core affective state level set for the area and the mental state level to the target mental state level set for the area.

[0060] Fig. 10A shows a bar chart 1000 illustrating a target concentration level and a target satisfaction level according to an embodiment of the present disclosure. Fig. 10B shows a graph 1010 illustrating a target core affective state level in a valence-arousal vector space according to an embodiment of the present disclosure. In this embodiment, the target ranges are categorized into Low, Medium, and High ranges. If the goal is to achieve a high level of satisfaction, a target level may be set at 0.83, positioned at the midpoint of the high range. For an effective office environment, the predominant emotion of Calm is preferred, with a notable presence of Happy states. The target core affective state level(s) fall under or within the ranges of core affective state levels for deactivated pleasure emotion and pleasant deactivation emotion, or in the form of circumplex model, in the target area 1012 within the vector space around core affective segments 4 and 5 (see shaded area in the figure) corresponding to deactivated pleasure emotion and pleasant deactivation emotion respectively. In this case, a target core affective level may be set at 315-degree mark (θ = 315o), positioned at the midpoint of the target area 1012.

[0061] Fig. 11 shows a flow diagram 1100 illustrating a process carried out by a core affect, concentration and satisfaction adjustment unit, for example, the core affect, concentration and satisfaction adjustment unit 444 in Fig. 4, according to an embodiment of the present disclosure. The core affect, concentration and satisfaction adjustment unit 444 may receive a target core affective state level, for example, in the form of a target segment in a valence-arousal vector space, and a target mental state level (herein may collectively referred to as "preferred target segments") from the emotion profile loading unit 442 and an average weighted core affective state level and an average mental state level (herein referred to as "weighted scores") from the average weighted score computation unit 426. In step 1102, the core affect, concentration and satisfaction adjustment unit 444 may evaluate the weighted scores against the preferred target segments.

[0062] According to the present disclosure, step 1104 is carried out where it is determined whether the weighted emotion score (weighted core affective state level) is within the target segment. If the weighted emotion score is not within the target segment, step 1106 is carried out where the adjustment required to adjust the weighted emotion score to match the target segment (herein referred to as "core affect adjustment") is calculated; otherwise step 1108 is carried out. Such adjustment can be an increment or decrement in the core affective state level, valence level or arousal level, or in the form of a degree of rotation of the core affective state level to match the target core affective state level in a circumplex model. In this example, a core affect adjustment of 45oanti-clockwise is calculated.

[0063] Fig. 12A illustrates an effect of temperature adjustment to a core affective state level in a valence-arousal vector space according to an embodiment of the present disclosure. Such circumplex structure of core affect is similar to that illustrated in the article entitled "A 12-point Circumplex Structure of Core Affect", published in 2011 by Michelle Yik. In one embodiment, following the study done by Francisco Barbosa Escobar et. al. (titled "The temperature of emotions") and Maria Sol Soria et. al. (titled "Fuzzy Control of Temperature on SACI Based on the Emotion Recognition") published in 2021 and 2020 respectively, it can be dedued that an increase of emotion affect (increase in a core affective state level) may correspond to a counterclockwise rotation in the valence-arousal vector space, as illustrated using arrow 1202. It is found that such adjustment can be achieved through an increase in surrounding temperature of an area. Conversely, a decrease of emotion affect (decrease in a core affective state level) may correspond to a clockwise rotation in the valence-arousal vector space, as illustrated using arrow 1204. It is found that such adjustment can be achieved through a decrease in surrounding temperature of an area.

[0064] Based on the circumplex model in "The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology" authored by James A. Russell et. al. published in 2005, it can be deduced that the highest satisfaction is generally observed at a core affective state level in a segment corresponds to deactivated pleasure emotion at a valence-arousal vector space while the highest concentration is generally observed at a core affection state level in a segment corresponds to pleasant deactivation emotion at a valence-arousal vector space, as shown in arrows 1212 and 1214 in Fig. 12B, respectively. This result is consistent with the study done by Hassan II University of Casablanca entitled "Determine the Level of Concentration of Students in Real Time from their Facial Expressions" authored by Bouhlal Meriem et. al. published in 2011. These findings provide some guidance on the adjustment required, including temperature adjustment, to achieve the desired level of satisfaction and concentration levels.

[0065] Fig. 13 show an exemplary current core affective state level (emotion status), concentration level and satisfaction level of a person in an area 1300 obtained from a FER preprocessor 302, 402 and an exemplary target core affective state level, concentration level and satisfaction level set for the area 1310 obtained from an area / activity type emotion profile 436, respectively, according to an embodiment of the present disclosure. In this example, a weighted core affective state level at 262o, broadly categorized under deactivated emotion under circumplex model is computed based on the emotion FER, and a low concentration level and a medium satisfaction level are obtained from the concentration FER and satisfaction FER respectively. Similar to Figs. 10A and 10B, the target core affective state level for this area / activity type in this example is set at 315-degree mark (θ = 315o), positioned at the midpoint of target area around deactivated pleasure segment and pleasant deactivation segment on the circumplex model, while the target concentration level and satisfaction level are set at 0.83, positioned at the midpoint of the high range. Carrying out steps 1104 and 1106, it is determined that the current weighted core affective state level does not fall within the target area 1212 (or does not match the target core affective state level at 315-degree mark (θ = 315o)) and a rotational adjustment of 53oanti-clockwise direction in the vector space is required to match the weighted core affective state to the target core affective state level at 315-degree mark (θ = 315o).

[0066] Returning to Fig. 11, if it is determined that the weighted emotion score is within the target segment in step 1104, step 1108 is carried out, where it is determined whether the weighted concentration and satisfaction scores are within their target levels / ranges. If any of the weighted concentration and satisfaction scores is not within its target level / range, step 1110 is carried out, where the adjustment(s) required to adjust the weighted concentration and / or satisfaction scores to match the target levels / ranges (herein referred to as "concentration / satisfaction affect adjustment") is calculated. In some cases, such concentration / satisfaction affect adjustment is effected through minor adjustment on the core affective state level while keeping the core affective state level within the target segments / ranges. In this example shown in Fig. 11, it is calculated that an increment of 0.2 in the concentration level intensity and 0.1 in the satisfaction level intensity is required, and such increment can be achieved through a minor rotational adjustment in the clockwise direction in the vector space.

[0067] Fig. 14 show an exemplary current core affective state level (emotion status), concentration level and satisfaction level of a person in an area 1400 obtained from a FER preprocessor 302, 402 and an exemplary target core affective state level, concentration level and satisfaction level set for the area 1410 obtained from an area / activity type emotion profile 436, respectively, according to an embodiment of the present disclosure. In this example, after step 1104 is carried out to determine that the current weighted emotion score falls within the target segment 1412, step 1108 is carried out to determine whether the weighted concentration and satisfaction scores are within their target levels / ranges. In this case, it is determined that the current weighted concentration score of 0.53 and weighed satisfaction score of 0.63 do not fall within their target levels / ranges of 0.83 and step 1110 is carried out, and a 0.3 increment in the concentration score and 0.2 increment in the satisfaction score are required, and to achieve the increment, a rotational adjustment on the core affective state level in the clockwise direction in the vector space is required while keeping the core affective state level within the target segment.

[0068] In an embodiment, using the emotion adjustment data from the previous embodiment, a step control method is applied to progressively modify the temperature, aiming to improve thermal comfort and avoid abrupt surrounding temperature change that may affect the thermal comfort of persons in the area of interest. To achieve that, the stepwise temperature adjustment unit 446 may implement a stepwise adjustment of the HVAC's set temperature output at regular intervals so as to ensure a gradual change in temperature rather than an instantaneous one. This approach offers two benefits: (i) enhanced thermal comfort for occupants, and (ii) mitigation of emotional fluctuations among occupants, which could otherwise complicate control.

[0069] A quicker adjustment may be used when a larger adjustment is required in core affect, aiming to reach the target emotion segment more rapidly. Two temperature adjustment parameters may be used (i) a temperature adjustment per step, for example, 0.5 °C from current ambience temperature per adjustment; and (ii) Time step (TS) per control interval, for example, at 2-min interval. If a smaller adjustment is required, a multiplier will be applied to the control Interval to result in a more gradual change.

[0070] Fig. 15 shows a flow diagram 1500 illustrating a process carried out by a stepwise temperature adjustment unit, for example, the stepwise temperature adjustment unit 446 in Fig. 4, to effect core affect adjustment, according to an embodiment of the present disclosure. In step 1502, the stepwise temperature adjustment unit 446 may detect or receive an indication on a current surrounding temperature (herein referred to as "ambience temperature"). In this case, the ambience temperature is 21°C. The stepwise temperature adjustment unit 446 may look up the ambience temperature range and classify the ambience temperature range in which the ambience temperature falls under based on ambience temperature classifications (e.g., those in Table 6) retrieved from a database 1504. In this case, the ambience temperature falls under the ambience temperature range classified as "cool". Similarly, in step 1506, the stepwise temperature adjustment unit 446 also receive the required core affect adjustment, for example a 45oanti-clockwise rotation in the valence-arousal vector space from the core affect, concentration and satisfaction adjustment unit 444, and look up the required core affect adjustment, in this case, the core affect adjustment on circumplex model and classify the extent or range of core affect adjustment required based on core affect adjustment classifications (e.g., those in Table 7) retrieved from a database 1508. In this case, the core affect adjustment required is classified as "slight increase". In step 1510, the stepwise temperature adjustment unit 446 may look up temperature adjustment rule set (e.g., those in Table 8) to determine a classification of temperature adjustment from a database 1512 based on the ambience temperature and core affect classifications obtained from steps 1502 and 1506. In this case, a class of "slightly warmer" for the required temperature adjustment is identified. In step 1514, the stepwise temperature adjustment unit 446 may again and look up for temperature adjustment per step and time interval each step (e.g., those in Table 9) according to the temperature adjustment classification obtained from step 1510 based on a database 1516. In this case, a temperature adjustment of 0.5°Cand a time interval of 4 minutes are obtained. In step 1518, it is determined whether the time interval obtained from step 1514 has reached. If the time interval has reached, step 1520 is carried out. In step 1520, the stepwise temperature adjustment unit 446 may compute the set temperature for HVAC by adding the temperature adjustment per step (0.5°C) obtained from step 1514 and the ambience temperature (21°C). In this case, a set temperature of 21.5 °Cis output.

[0071] Table 6 shows example classifications for various temperature range according to an embodiment of the present disclosure.

[0072] (Table 6)

[0073] Table 7 shows example classifications for various core affect adjustments on circumplex model according to an embodiment of the present disclosure.

[0074] (Table 7)

[0075] Table 8 shows example temperature adjustment classification rule according to an embodiment of the present disclosure. In this case, the rules are based on ambience temperature and core affect obtained from steps 1402 and 1406.

[0076] (Table 8)

[0077] Table 9 shows example temperature adjustment classifications and their corresponding temperature adjustment per step and time interval for each control step according to an embodiment of the present disclosure.

[0078] (Table 9)

[0079] Fig. 16 shows a flow diagram 1600 illustrating a process carried out by a stepwise temperature adjustment unit, for example, the stepwise temperature adjustment unit 446 in Fig. 4, to effect concentration / satisfaction affect adjustment, according to an embodiment of the present disclosure. In step 1602, the stepwise temperature adjustment unit 446 may receive the required concentration and satisfaction adjustments, for example a 0.3 increment in the concentration score and 0.2 increment in the satisfaction score as well as a rotational adjustment on the core affective state level in the clockwise direction from the core affect, concentration and satisfaction adjustment unit 444, and look up the intensity adjustments for satisfaction and concentration based on concentration adjustment classifications and satisfaction adjustment classifications (e.g., those in Tables 10 and 11) retrieved from respective databases 1604, 1606. In this case, the concentration adjustment required is classified as "slight increase" and the satisfaction adjustment required is also classified as "slight increase".

[0080] In step 1608, the stepwise temperature adjustment unit 446 may look up temperature adjustment rule set (e.g., those in Table 12) to determine a classification of temperature adjustment from a database 1610 based on the core affect adjustment and concentration / satisfaction classifications obtained from step 1602. In this case, both classifications lead to a class of temperature adjustment of "slightly cooler". In step 1612, the stepwise temperature adjustment unit 446 may again and look up for temperature adjustment per step and time interval each step (e.g., those in Table 9) according to the temperature adjustment classification obtained from step 1608 based on a database 1614. In this case, a temperature adjustment of -0.5°Cand a time interval of 4 minutes are obtained. In step 1616, it is determined whether the time interval obtained from step 1618 has reached. If the time interval has reached, step 1620 is carried out. In step 1620, the stepwise temperature adjustment unit 446 may compute the set temperature for HVAC by adding the temperature adjustment per step (-0.5°C) obtained from step 1614 and the ambience temperature (22.5 °C). In this case, a set temperature of 22 °C is output.

[0081] Table 10 shows example classifications for various concentration intensity adjustments according to an embodiment of the present disclosure.

[0082] (Table 10)

[0083] Table 11 shows example classifications for various satisfaction intensity adjustments according to an embodiment of the present disclosure.

[0084] (Table 11)

[0085] Table 12 shows example temperature adjustment classification rule according to an embodiment of the present disclosure. In this case, the rules are based on core affect adjustment and satisfaction / concentration adjustment classification obtained from step 1502.

[0086] (Table 12)

[0087] An alternative approach, such as employing fuzzy logic, can replace the stepwise temperature adjustment process carried out by the stepwise temperature adjustment unit. In this scenario, the classification databases would be substituted with fuzzy sets.

[0088] According to the present disclosure, the method, apparatus and system for adaptive regulating a surrounding temperature of an area can be applied to and cater for various area types and activity types. Table 13 shows a list of spaces (areas) and activity types where the method, apparatus and system can be applied to regulating their surrounding temperature.

[0089] (Table 13)

[0090] Returning to Fig. 3, the camera 310 will continue to perform FER and FR to detect new facial expression of the persons in the area and generate new FER output to compute new core affective state level and mental state level of the persons. Such new core affective and mental state levels may be used to monitor the effect of the change in temperature in the persons' core affective and mental state levels, and further adjustment may be made to complete the feedback loop and to ensure the core affective and mental state levels of the persons in the area are kept within the target core affective and mental state levels.

[0091] It is noted that the emotion profile (i.e., the target core affective state levels / ranges and mental state levels / ranges) is configured differently for each area type and activity type to realize different needs, for example, to achieve different level of productivity and well-being of the persons occupying the space and area. Such emotion profile may also be configured differently based on own business use case. Figs. 16-20 illustrate exemplary emotion profiles (i.e., target core affective state levels / ranges and mental state levels / ranges) configured for five different spaces for different space use case.

[0092] Fig. 17 shows a bar chart 1700 illustrating a target concentration level, a target satisfaction level and a graph 1710 illustrating a core affective state level in a valence-arousal vector space for a serious discussion in a collaboration space use case according to an embodiment of the present disclosure. In such use case, a group of co-workers having a serious discussion in a collaboration workspace. The person (role) types are co-workers, each may be assigned equal weightage. The target core affective state level is set to be around segment 5 in valence-arousal vector space 1710 (see shaded area in the figure), i.e., pleasant deactivation emotion with high target concentration level and medium target satisfaction level to encourage profound discussions.

[0093] Fig. 18 shows a bar chart 1800 illustrating a target concentration level, a target satisfaction level and a graph 1710 illustrating a core affective state level in a valence-arousal vector space for a classroom use case according to an embodiment of the present disclosure. Such use case includes a case where a teacher is conducting lesson with 20 students in a classroom. There are two different person (role) types in this use case: teacher (x1) and student (x20). The teacher's core affective and mental state levels are more important under such setting, therefore a weightage of 50% is assigned while the remaining 50% weightage is distributed among 20 students' core affective and mental state levels. The ideal classroom setting should predominantly foster feelings of happiness and calmness so the target emotion state level is set to be around segments 3 and 4 in valence-arousal vector space 1810 (see shaded area in the figure), i.e., pleasure and deactivated pleasure emotions with medium target concentration level and medium target satisfaction level.

[0094] Fig. 19 shows a bar chart 1900 illustrating a target concentration level, a target satisfaction level and a graph 1910 illustrating a core affective state level in a valence-arousal vector space for a celebratory event in a conference use case according to an embodiment of the present disclosure. Such use case includes a case where an emcee is presenting over a celebratory event with 20 attendees in a conference room. Similarly to the classroom use case, there are two different person (role) types in this use case: emcee (x1) and participants (x20). The emcee's core affective and mental state levels are more important under such setting, therefore a weightage of 50% is assigned while the remaining 50% weightage is distributed among 20 participants' core affective and mental state levels. The goal is to elicit intense feelings of positivity (high Valence) and excitement (high Arousal) among participants, while also aim for a sustained attention span accompanied by moments of joy. As such, the target emotion is around segments 1 to 2 in valence-arousal vector space 1910 (see shaded area in the figure), i.e., pleasant activation to activated pleasure emotions. Concentration and satisfaction are not relevant factors for this activity as the event is active and not in a sedentary environment., therefore there is no target for concentration and satisfaction levels.

[0095] Fig. 20 shows a bar chart 2000 illustrating a target concentration level, a target satisfaction level and a graph 2010 illustrating a core affective state level in a valence-arousal vector space for a relaxation space use case according to an embodiment of the present disclosure. An example relaxation space use case is two co-workers having a casual Chat Over Coffee at a relaxation corner. The person (role) types are co-workers, each may be assigned equal weightage. The goal is to foster a laid-back atmosphere for casual discussions, expecting an overall sense of calmness with occasional moments of joy. The targeted emotion range is segments 3 to 4 in valence-arousal vector space 2010 (see shaded area in the figure), i.e., pleasant deactivation emotion, with low target concentration level and high target satisfaction level to reduce stress.

[0096] Fig. 21 shows a bar chart 2100 illustrating a target concentration level, a target satisfaction level and a graph 2110 illustrating a core affective state level in a valence-arousal vector space for a long road trip in an in-car use case according to an embodiment of the present disclosure. Such use case includes a driver with three passengers on an extended road trip for a weekend getaway. There are two different person (role) types in this use case: driver (x1) and passenger (x3). The driver's core affective and mental state levels are more important under such setting, thereby therefore a weightage of 80% is assigned, while the remaining 20% weightage is distributed among three passengers' core affective and mental state levels. The primary focus for the emotion target is on the driver, where the prevailing emotion is anticipated to be calmness. The target is set at segment 4 in valence-arousal vector space 2110 (see shaded area in the figure), i.e., deactivated pleasure, with high target satisfaction level and medium concentration level for safe driving.

[0097] Drowsiness as a sensing input parameter can be applied in detecting expressions among construction workers and heavy vehicle drivers, particularly in contexts where safety risks are elevated.

[0098] Recognizing facial expressions through still images may lack timeliness, given the necessity for averaging. In contrast, Facial Video offers a more immediate and precise assessment, enabling the capture of details such as eyelid blink rates, openness, and gaze.

[0099] Many care centers equip patients with wearable devices designed to measure both Heart Rate and Heart Rate Variability. Utilizing these parameters, it becomes possible to estimate the core affective / emotional states of the patients. Research indicates a correlation between heart rate and emotions. Since the normal heartbeat rate varies for each individual, establishing emotion states requires profiling and personalization. Research also demonstrates that Heart Rate Variability (HRV), which refers to the fluctuation of intervals between adjacent heartbeats, is linked to emotions.

[0100] Dynamic LED lighting allows for dynamic control of illumination in office environments. Different lighting settings, such as warm, cool, and daylight (with daylight having the highest temperature), result in varying light temperatures. Research indicates that lighting has an impact on emotions. Other research shows that lighting affects concentration and satisfaction levels. A systematic review on the effects of light on attention and reaction time show that shorter wavelengths, higher intensity and higher color temperature lead to increased attention and faster reaction time. Natural sunlight is most effective in improving mental mood, attention and cognitive function. Experimental study was also conducted on the effect of color temperature and illuminance on psychology, physiology, and productivity. It concludes that warm correlated color temperature (CCT) and higher illuminance made subjects more comfortable. Subjects preferred intermediate CCT and bright illumination after self-adjustment.

[0101] The nature of activities will also dictate the appropriate lighting conditions. For instance, a discussion area would ideally benefit from higher color temperature and illuminance, whereas a relaxation area should feature warmer lighting with lower illuminance.

[0102] Studies have shown that higher levels of CO2lower mental performance. Higher Levels of CO2is associated with drowsiness. According to the ASHRAE 62.1 standard, the acceptable indoor CO2concentration ranges from 1000 to 1200 ppm. Maintaining a lower CO2concentration level would be ideal but will incur a higher ventilation cost.

[0103] The requirement for CO2concentration levels may vary based on the nature of activities. Specifically, a discussion area would ideally necessitate maintaining a lower CO2concentration level, while a relaxation area may tolerate a slightly higher CO2concentration level.

[0104] While the CO2 concentration may comply with the levels stipulated by the ASHRAE 62.1 standard, there might be a preference to augment ventilation if occupants are observed to be fatigued or experiencing drowsiness.

[0105] The introduction of fragrance stimuli elevates arousal levels, subsequently enhancing intellectual productivity. Studies indicate that applying fragrance stimuli at appropriate moments and intervals can effectively boost arousal level.

[0106] The invention can be extended by incorporating additional environmental parameters and utilizing alternative feedback parameters. The sensing inputs can be extended to include drowsiness, heart rate and heart rate variability. The feedback method can also apply to lighting, CO2concentration, ventilation and aroma.

[0107] In particular, the present disclosure may also provide a method, an apparatus and a system for regulating a surrounding temperature of an area using additional input such as a drowsiness level of the person(s) in the area from FER output and / or a heartbeat input such as heart rate and heart rate. In addition, the feedback controller (e.g., feedback controller 304 in Fig. 3) also monitors ambience temperature, lighting, air concentration (e.g., CO2concentration) and aroma and generates not only a set temperature but also set light operating parameter (e.g., light intensity, color temperature), set ventilation operating parameter (e.g., fan speed), set aroma diffusion operating parameter (e.g., type of aroma, aroma diffusion rate), so that the HVAC (e.g., HVAC 306 in Fig. 3) is then configured to regulate the surrounding temperature, lighting, ventilation and / or aroma in the area.

[0108] For example, the apparatus may be configured to include a stepwise lighting adjustment unit to calculate the required adjustment to the lighting parameter to effect the core affect, concentration and satisfaction adjustment. If it is computed that an adjustment in the core affecitve state level in a counterclockwise direction in the valence-arousal vector space is required, then an increase in illumination and / or light temperature may also be additionally, or alternatively used to effect the adjustment in the core affective state level.

[0109] For example, the apparatus may be configured to include a stepwise ventilation adjustment unit to calculate the required adjustment to the ventilation operating parameter to effect the emotion affect, concentration and satisfaction adjustment. If it is computed that an adjustment in the core affective state level in a counterclockwise direction in the valence-arousal vector space is required, then an increase in ventilation (e.g., fan speed) may also be additionally, or alternatively used to effect the adjustment in the core affective state level.

[0110] Fig. 22 depicts an exemplary computing device 2200, hereinafter interchangeably referred to as a computer system 2200, where one or more such computing devices 2200 may be used to execute the method of Fig. 2. The exemplary computing device 2200 can be used to implement the apparatus 104 shown in Fig. 1. The following description of the computing device 2200 is provided by way of example only and is not intended to be limiting.

[0111] As shown in Fig. 22, the example computing device 2200 includes a processor 2204 for executing software routines. Although a single processor is shown for the sake of clarity, the computing device 2200 may also include a multi-processor system. The processor 2204 is connected to a communication infrastructure 2206 for communication with other components of the computing device 2200. The communication infrastructure 2206 may include, for example, a communications bus, cross-bar, or network.

[0112] The computing device 2200 further includes a main memory 2208, such as a random access memory (RAM), and a secondary memory 2210. The secondary memory 2210 may include, for example, a storage drive 2212, which may be a hard disk drive, a solid state drive or a hybrid drive and / or a removable storage drive 2214, which may include a magnetic tape drive, an optical disk drive, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), or the like. The removable storage drive 2214 reads from and / or writes to a removable storage medium 2218 in a well-known manner. The removable storage medium 2218 may include magnetic tape, optical disk, non-volatile memory storage medium, or the like, which is read by and written to by removable storage drive 2214. As will be appreciated by persons skilled in the relevant art(s), the removable storage medium 2218 includes a computer readable storage medium having stored therein computer executable program code instructions and / or data.

[0113] In an alternative implementation, the secondary memory 2210 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device 2200. Such means can include, for example, a removable storage unit 2222 and an interface 2220. Examples of a removable storage unit 2222 and interface 2220 include a program cartridge and cartridge interface (such as that found in video game console devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a removable solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), and other removable storage units 2222 and interfaces 2220 which allow software and data to be transferred from the removable storage unit 2222 to the computer system 2200.

[0114] The computing device 2200 also includes at least one communication interface 2224. The communication interface 2224 allows software and data to be transferred between computing device 2200 and external devices (e.g., an image capturing apparatus and / or a HVAC system) via a communication path 2226. In various embodiments of the inventions, the communication interface 2224 permits data to be transferred between the computing device 2200 and a data communication network, such as a public data or private data communication network. The communication interface 2224 may be used to exchange data between different computing devices 600 which such computing devices 2200 form part an interconnected computer network. Examples of a communication interface 2224 can include a modem, a network interface (such as an Ethernet card), a communication port (such as a serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), an antenna with associated circuitry and the like. The communication interface 2224 may be wired or may be wireless. Software and data transferred via the communication interface 2224 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communication interface 2224. These signals are provided to the communication interface via the communication path 2226.

[0115] As shown in Fig. 22, the computing device 2200 further includes a display interface 2202 which performs operations for rendering images to an associated display 2230 and an audio interface 2232 for performing operations for playing audio content via associated speaker(s) 2234.

[0116] As used herein, the term "computer program product" may refer, in part, to removable storage medium 2218, removable storage unit 2222, a hard disk installed in storage drive 2212, or a carrier wave carrying software over communication path 2226 (wireless link or cable) to communication interface 2224. Computer readable storage media refers to any non-transitory, non-volatile tangible storage medium that provides recorded instructions and / or data to the computing device 2200 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray Disc, a hard disk drive, a ROM or integrated circuit, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), a hybrid drive, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computing device 2200. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the computing device 2200 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0117] The computer programs (also called computer program code) are stored in main memory 2208 and / or secondary memory 2210. Computer programs can also be received via the communication interface 2224. Such computer programs, when executed, enable the computing device 2200 to perform one or more features of embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor 1207 to perform features of the above-described embodiments. Accordingly, such computer programs represent controllers of the computer system 2200.

[0118] Software may be stored in a computer program product and loaded into the computing device 2200 using the removable storage drive 2214, the storage drive 2212, or the interface 2220. The computer program product may be a non-transitory computer readable medium. Alternatively, the computer program product may be downloaded to the computer system 2200 over the communications path 2226. The software, when executed by the processor 2204, causes the computing device 2200 to perform the necessary operations to execute the method as shown in Fig. 2.

[0119] It is to be understood that the embodiment of Fig. 22 is presented merely by way of example to explain the operation and structure of the apparatus 104. Therefore, in some embodiments one or more features of the computing device 2200 may be omitted. Also, in some embodiments, one or more features of the computing device 2200 may be combined together. Additionally, in some embodiments, one or more features of the computing device 2200 may be split into one or more component parts.

[0120] It will be appreciated by a person skilled in the art that numerous variations and / or modifications may be made to the present invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

[0121] Each of the drawings or figures is merely an example to illustrate one or more example embodiments. Each figure may not be associated with only one particular example embodiment, but may be associated with one or more other example embodiments. As those of ordinary skill in the art will understand, various features or steps described with reference to any one of the figures can be combined with features or steps illustrated in one or more other figures, for example, to produce example embodiments that are not explicitly illustrated or described. Not all of the features or steps illustrated in any one of the figures to describe an example embodiment are necessarily essential, and some features or steps may be omitted. The order of the steps described in any of the figures may be changed as appropriate.

[0122] The whole or part of the example embodiments disclosed above can be described as, but not limited to, the following supplementary notes. (Supplementary Note 1) 　　A method for adaptively regulating a surrounding temperature of an area using facial expression recognition, comprising: 　　computing a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level comprises a valence level and an arousal level of the person; 　　determining if the core affective state level matches a target core affective state level set for the area; and 　　in response to determining that the core affective state level does not match the target core affective state level set for the area, deriving an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level. (Supplementary Note 2) 　　The method according to supplementary note 1, further comprising: 　　obtaining a mental state level of the person based on the detected facial expression of the person, wherein the mental state level comprises one of a concentration level and a satisfaction level of the person; 　　determining if the mental state level matches a target mental state level set for the area in response to determining that the core affective state level matches the target core affective state level set for the area; and 　　in response to determining that the mental state level does not match a target mental state level set for the area: 　　　　calculating the adjustment to the surrounding temperature based on a second adjustment required to match the mental state level to the target mental state level and core affective state level to the target core affective state level. (Supplementary Note 3) 　　The method according to supplementary note 2, further comprising: 　　identifying one of an area type and an activity type of the area; and 　　setting one of the target core affective state level and the target mental state level for the area based on the one of the area type and the activity type of the area. (Supplementary Note 4) 　　The method according to supplementary note 3, further comprising: 　　detecting a role identifier relating to the person based on the detected facial feature of the person; and 　　applying a weightage corresponding to the role identifier to one of the core affective state level and the mental state level of the person, wherein the determination of the one of the core affective state level and the mental state level is based on the weighted one of the core affective state level and the mental state level. (Supplementary Note 5) 　　The method according to any one of supplementary notes 2 to 4, wherein the core affective state level is an average core affective state level of a plurality of core affective state levels computed, and the mental state level is an average mental state level of a plurality of mental state levels obtained based on a plurality of detected facial expressions of the person over a time period. (Supplementary Note 6) 　　The method according to any one of supplementary notes 2 to 5, further comprising: 　　classifying one of the first adjustment and the second adjustment; wherein the adjustment to the surrounding temperature is derived based on a result of the classification. (Supplementary Note 7) 　　The method according to any one of supplementary notes 1 to 6, further comprising: 　　setting a number of adjustment steps and an interval between each adjustment step to effect the adjustment to the surrounding temperature. (Supplementary Note 8) 　　The method according to supplementary note 7, further comprising: 　　classifying the adjustment to the surrounding temperature; wherein the number of adjustment steps and the interval between the each adjustment step is based on a result of the classification of the adjustment to the surrounding temperature. (Supplementary Note 9) 　　The method according to supplementary note 7 or 8, further comprising: 　　classifying the surrounding temperature, wherein the number of adjustment steps and the interval between the each adjustment step is further based on a result of the classification of the surrounding temperature. (Supplementary Note 10) 　　An apparatus for adaptively regulating a surrounding temperature of an area, the apparatus comprising: 　　at least one processor; and 　　at least one memory including computer program code; 　　the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　　　compute a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level comprises a valence level and an arousal level of the person; 　　　　determine if the core affective state level matches a target core affective state level set for the area; and 　　　　in response to determining that the core affective state level does not match the target core affective state level set for the area: 　　　　　　derive an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level. (Supplementary Note 11) 　　The apparatus according to supplementary note 10, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　obtain a mental state level of the person based on the detected facial expression of the person, wherein the mental state level comprises one of a concentration level and a satisfaction level of the person; 　　determine if the mental state level matches a target mental state level set for the area in response to determining that the core affective state level matches the target core affective state level set for the area; and 　　in response to determining that the mental state level does not match a target mental state level set for the area: 　　　　calculate the adjustment to the surrounding temperature based on a second adjustment required to match the mental state level to the target mental state level and core affective state level to the target core affective state level. (Supplementary Note 12) 　　The apparatus according to supplementary note 11, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　identify one of an area type and an activity type of the area; and 　　set one of the target core affective state level and the target mental state level for the area based on the one of the area type and the activity type of the area. (Supplementary Note 13) 　　The apparatus according to supplementary note 12, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　detect a role identifier relating to the person based on the detected facial feature of the person; 　　apply a weightage corresponding to the role identifier to one of the core affective state level and the mental state level of the person, the determination of the one of the core affective state level and the mental state level is based on the weighted one of the core affective state level and the mental state level. (Supplementary Note 14) 　　The apparatus according to any one of supplementary notes 11 to 13, wherein the core affective state level is an average core affective state level of a plurality of core affective state levels computed and the mental state level is an average mental state level of a plurality of mental state levels obtained based on a plurality of detected facial expressions of the person over a time period. (Supplementary Note 15) 　　The apparatus according to any one of supplementary notes 11 to 14, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　classify one of the first adjustment and the second adjustment; and 　　derive the adjustment to the surrounding temperature based on a result of the classification. (Supplementary Note 16) 　　The apparatus according to any one of supplementary notes 10 to 15, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　set a number of adjustment steps and an interval between each adjustment step to effect the adjustment to the surrounding temperature. (Supplementary Note 17) 　　The apparatus according to supplementary note 16, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　classify the adjustment to the surrounding temperature; and 　　set the number of adjustment steps and the interval between the each adjustment step based on a result of the classification of the adjustment to the surrounding temperature. (Supplementary Note 18) 　　The apparatus according to supplementary note 16 or 17, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　classify the surrounding temperature; and 　　set the number of adjustment steps and the interval between the each adjustment step further based on a result of the classification of the surrounding temperature. (Supplementary Note 19) 　　A system for adaptively regulating a surrounding temperature of an area using facial expression recognition, the system comprises the apparatus according to any one of supplementary notes 10 to 18 and an image capturing apparatus for detecting a facial expression of a person.

[0123] Some or all of elements specified in any of Supplementary Notes may be applied to various types of hardware, software, and recording means for recording software, systems, and methods.

[0124] This application is based upon and claims the benefit of priority from Singaporean patent application No. 10202400174W, filed on January 19, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0125] 100 System 102 Image capturing device 104 Apparatus 106 Processor 108 Memory 110 Database 302 FER preprocessor 304 Feedback controller 306 HVAC 310 Camera 402 FER preprocessor 404 Feedback controller 406 HVAC 422 Arousal computation unit 424 Person role detection unit 426 Average weightage score computation unit 432 Person profile database 434 Role weights storage 436 Area / activity type emotion profile database 442 Emotion profile loading unit 444 Core affect, concentration and satisfaction adjustment unit 446 Stepwise temperature adjustment unit 510 Camera 600 Valence-arousal vector space 700 Valence-arousal vector space 710 Vector space 2200 Computing device 2202 Display interface 2204 Processor 2206 Communication infrastructure 2208 Main memory 2210 Secondary memory 2212 Storage drive 2214 Removable storage drive 2218 Removable storage medium 2220 Interface 2222 Removable storage unit 2224 Communication interface 2226 Communication path 2230 Display 2232 Audio interface 2234 Speaker

Claims

1. A method for adaptively regulating a surrounding temperature of an area using facial expression recognition, comprising: 　　computing a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level comprises a valence level and an arousal level of the person; 　　determining if the core affective state level matches a target core affective state level set for the area; and 　　in response to determining that the core affective state level does not match the target core affective state level set for the area, deriving an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level.

2. The method according to claim 1, further comprising: 　　obtaining a mental state level of the person based on the detected facial expression of the person, wherein the mental state level comprises one of a concentration level and a satisfaction level of the person; 　　determining if the mental state level matches a target mental state level set for the area in response to determining that the core affective state level matches the target core affective state level set for the area; and 　　in response to determining that the mental state level does not match a target mental state level set for the area: 　　 calculating the adjustment to the surrounding temperature based on a second adjustment required to match the mental state level to the target mental state level and core affective state level to the target core affective state level.

3. The method according to claim 2, further comprising: 　　identifying one of an area type and an activity type of the area; and 　　setting one of the target core affective state level and the target mental state level for the area based on the one of the area type and the activity type of the area.

4. The method according to claim 3, further comprising: 　　detecting a role identifier relating to the person based on the detected facial feature of the person; and 　　applying a weightage corresponding to the role identifier to one of the core affective state level and the mental state level of the person, wherein the determination of the one of the core affective state level and the mental state level is based on the weighted one of the core affective state level and the mental state level.

5. The method according to any one of claims 2 to 4, wherein the core affective state level is an average core affective state level of a plurality of core affective state levels computed, and the mental state level is an average mental state level of a plurality of mental state levels obtained based on a plurality of detected facial expressions of the person over a time period.

6. The method according to any one of claims 2 to 5, further comprising: 　　classifying one of the first adjustment and the second adjustment; wherein the adjustment to the surrounding temperature is derived based on a result of the classification.

7. The method according to any one of claims 1 to 6, further comprising: 　　setting a number of adjustment steps and an interval between each adjustment step to effect the adjustment to the surrounding temperature.

8. The method according to claim 7, further comprising: 　　classifying the adjustment to the surrounding temperature; wherein the number of adjustment steps and the interval between the each adjustment step is based on a result of the classification of the adjustment to the surrounding temperature.

9. The method according to claim 7 or 8, further comprising: 　　classifying the surrounding temperature, wherein the number of adjustment steps and the interval between the each adjustment step is further based on a result of the classification of the surrounding temperature.

10. An apparatus for adaptively regulating a surrounding temperature of an area, the apparatus comprising: 　　at least one processor; and 　　at least one memory including computer program code; 　　the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　　　compute a core affective state level of a person in the area based on a detected facial expression of the person, wherein the core affective state level comprises a valence level and an arousal level of the person; 　　　　determine if the core affective state level matches a target core affective state level set for the area; and 　　　　in response to determining that the core affective state level does not match the target core affective state level set for the area: 　　　　　　derive an adjustment to the surrounding temperature based on a first adjustment required to match the core affective state level to the target core affective state level.

11. The apparatus according to claim 10, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　obtain a mental state level of the person based on the detected facial expression of the person, wherein the mental state level comprises one of a concentration level and a satisfaction level of the person; 　　determine if the mental state level matches a target mental state level set for the area in response to determining that the core affective state level matches the target core affective state level set for the area; and 　　in response to determining that the mental state level does not match a target mental state level set for the area: 　　　　calculate the adjustment to the surrounding temperature based on a second adjustment required to match the mental state level to the target mental state level and core affective state level to the target core affective state level.

12. The apparatus according to claim 11, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　identify one of an area type and an activity type of the area; and 　　set one of the target core affective state level and the target mental state level for the area based on the one of the area type and the activity type of the area.

13. The apparatus according to claim 12, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　detect a role identifier relating to the person based on the detected facial feature of the person; 　　apply a weightage corresponding to the role identifier to one of the core affective state level and the mental state level of the person, the determination of the one of the core affective state level and the mental state level is based on the weighted one of the core affective state level and the mental state level.

14. The apparatus according to any one of claims 11 to 13, wherein the core affective state level is an average core affective state level of a plurality of core affective state levels computed and the mental state level is an average mental state level of a plurality of mental state levels obtained based on a plurality of detected facial expressions of the person over a time period.

15. The apparatus according to any one of claims 11 to 14, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　classify one of the first adjustment and the second adjustment; and 　　derive the adjustment to the surrounding temperature based on a result of the classification.

16. The apparatus according to any one of claims 10 to 15, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　set a number of adjustment steps and an interval between each adjustment step to effect the adjustment to the surrounding temperature.

17. The apparatus according to claim 16, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　classify the adjustment to the surrounding temperature; and 　　set the number of adjustment steps and the interval between the each adjustment step based on a result of the classification of the adjustment to the surrounding temperature.

18. The apparatus according to claim 16 or 17, the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　classify the surrounding temperature; and 　　set the number of adjustment steps and the interval between the each adjustment step further based on a result of the classification of the surrounding temperature.

19. A system for adaptively regulating a surrounding temperature of an area using facial expression recognition, the system comprises the apparatus according to any one of claims 10 to 18 and an image capturing apparatus for detecting a facial expression of a person.

Images