Employee status monitoring method and system based on deep learning of facial multimodal information

Abstract

The present invention provides an employee status monitoring method and system based on deep learning of facial multimodal information, relating to the technical field of employee status monitoring. The system includes a facial acquisition module, a facial processing module, a deep learning module and an emotion assessment module; the facial acquisition module is used to collect basic facial information and work facial information of employees; the facial processing module is used to analyze the collected data; the deep learning module is used to construct a convolutional neural network; the emotion assessment module evaluates the work facial information of employees based on the convolutional neural network to obtain the real-time status of employees; the present invention obtains facial data of employees in basic status and work status, compares and analyzes the facial data of employees in different states, and obtains the real-time status of employees, so as to solve the problem that the existing employee monitoring system does not obtain and analyze the facial status of employees, resulting in the inability to judge the fatigue status of employees.

Description

Technical Field

[0001] The present invention relates to the technical field of employee status monitoring, and in particular to an employee status monitoring method and system based on deep learning of facial multimodal information. Background Art

[0002] With the rapid development of science and technology, employee status monitoring technology plays an increasingly important role in enterprise management. By comprehensively monitoring and analyzing employees' physiological indicators, psychological state, work efficiency, etc., enterprises can more accurately grasp the work status of employees and thus formulate more effective management strategies.

[0003] In the prior art, the work status of employees is often judged by monitoring their work results and background processes. For example, the Chinese patent application with publication number CN116629651A discloses a work status process monitoring method, device and storage medium, which includes: obtaining the work status data of the target monitoring personnel within the unit working time; calculating the first work seriousness coefficient value, the second work seriousness coefficient value and the third work seriousness coefficient value according to the obtained work status data; performing weighted calculation on the first work seriousness coefficient value, the second work seriousness coefficient value and the third work seriousness coefficient value to obtain the total work seriousness coefficient value; presetting the total work seriousness coefficient threshold value to judge the work status of the target monitoring personnel within the unit working time. Whether the work status is serious; this method is to monitor the employee's background process, without obtaining and analyzing the employee's mental state. The existing method of monitoring the employee status through video usually analyzes whether the employee is on the job. For example, the Chinese patent application with publication number CN114663809A discloses an employee status recognition method, device, electronic equipment and medium. This method determines the positional relationship between the target position and the reference position. If the target position is within the reference position, it determines that the workstation to be identified is in the employee's on-the-job status. It can automatically identify the on-the-job status based on video surveillance data. However, it lacks the recognition of the employee's real-time working status, and there will be a problem of being unable to judge the employee's fatigue status. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide an employee status monitoring method and system based on deep learning of facial multimodal information. By obtaining facial data of employees in basic and working states, the facial data of employees in different states are compared, analyzed and processed to obtain the real-time status of employees, so as to solve the problem that the existing employee monitoring system does not obtain and analyze the facial status of employees, resulting in the inability to judge the fatigue status of employees.

[0005] In order to achieve the above-mentioned object, the present invention is implemented through the following technical solutions: In a first aspect, the present application provides an employee status monitoring system based on deep learning of facial multimodal information, including a facial acquisition module, a facial processing module, a deep learning module and an emotion assessment module;

[0006] The facial collection module is used to collect basic facial information and work facial information of employees;

[0007] The facial processing module includes a basic processing unit and a working processing unit, wherein the basic processing unit is used to divide and extract the basic facial image to obtain basic state data;

[0008] The work processing unit is used to divide and extract the work facial image to obtain work status data;

[0009] The deep learning module is used to construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network;

[0010] The emotion assessment module evaluates the employee's work facial information based on a convolutional neural network to obtain the employee's real-time status.

[0011] Furthermore, the facial acquisition module includes a working facial information acquisition unit and a basic facial information acquisition unit, wherein the working facial information acquisition unit is used to acquire working facial information, and the working facial information includes working facial images and working thermal imaging data;

[0012] Collecting work facial information includes: using an image acquisition device to collect a work area scene image, locating a face collection area from the work area scene image, and capturing an image of an employee's face within the face collection area to obtain a work facial image; dividing the work area scene image into a plurality of workstation areas and other areas, analyzing each workstation area separately, obtaining the orientation of the computer screen of the employee in each workstation area, and setting an area within the first work area in front of the computer screen orientation as a face collection area; using an infrared sensor device to obtain thermal imaging data of the face collection area, and setting the data as work thermal imaging data;

[0013] The basic facial information acquisition unit is used to collect basic facial information, and the basic facial information includes a basic facial image and basic thermal imaging data; the basic facial information acquisition unit is configured with a basic image acquisition strategy, and the basic image acquisition strategy includes using a photographic device to collect a frontal photo of the employee and setting the photo as the basic facial image; using an infrared sensing device to obtain the frontal thermal imaging data of the employee and setting it as the basic thermal imaging data.

[0014] Furthermore, the basic processing unit is configured with a basic image processing strategy, which includes obtaining a basic facial image, dividing the basic facial image into a basic eye area, a basic brow area, and a basic lip area; performing grayscale processing on the basic eye area, the basic brow area, and the basic lip area, converting the grayscale image into a binary image, setting pixels with grayscale values ​​less than or equal to a first threshold to 0, and setting pixels with grayscale values ​​greater than the first threshold to 255; and obtaining a basic eye binary image, a basic brow binary image, and a basic lip binary image;

[0015] Obtain a basic eye binary image, set a pixel region with a grayscale value of 255 in the basic eye binary image as a first region, and set a pixel region with a grayscale value of 0 in the basic eye binary image as a second region; obtain the areas of the first region and the second region, and set the area ratio of the first region to the second region as the basic image fatigue level;

[0016] Obtaining the region outline of the pixel region with a grayscale value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the region outline, and setting them as the basic eyebrow horizontal length and basic eyebrow vertical length;

[0017] The vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the basic lip binary image are obtained, and the height difference between the vertical highest point and the vertical lowest point is set as the basic image pleasantness.

[0018] Furthermore, the basic processing unit is further configured with a basic infrared processing strategy, which includes acquiring basic thermal imaging data to establish a basic three-dimensional facial model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance;

[0019] Obtain the leftmost point, rightmost point, highest point, and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height;

[0020] The leftmost point, rightmost point and lowest point of the lip are obtained from the basic three-dimensional facial model. The line connecting the leftmost point and the lowest point of the lip is set as the slant line of the corner of the mouth, the line connecting the leftmost point and the rightmost point of the lip is set as the horizontal line of the lip, and the angle between the slant line of the corner of the mouth and the horizontal line of the lip is set as the basic infrared pleasantness.

[0021] Furthermore, the work processing unit is configured with a work image processing strategy, which includes acquiring a work facial image, extracting a work eye area, a work brow area, and a work lip area from the work facial image; comparing the total area of ​​the work eye area, the work brow area, and the work lip area with the total area of ​​the basic eye area, the basic brow area, and the basic lip area, respectively; and when the total area of ​​the work eye area, the work brow area, and the work lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area, marking the employee as leaving the job;

[0022] When the total area of ​​the working eye region, the working brow region, and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region, and the basic lip region, marking the employee as being in a working state;

[0023] When an employee is in a working state, the working eye area, the working brow area, and the working lip area are grayscaled, the grayscale image is converted into a binary image, the pixel points with a grayscale value less than or equal to a first threshold are set to 0, and the pixel points with a grayscale value greater than the first threshold are set to 255; the working eye binary image, the working brow binary image, and the working lip binary image are obtained.

[0024] Furthermore, the work processing unit is further configured with a working eye area analysis strategy, which includes obtaining a working eye binary image, setting a pixel area with a grayscale value of 255 in the working eye binary image as a third area, and setting a pixel area with a grayscale value of 0 in the working eye binary image as a fourth area; obtaining the areas of the third area and the fourth area, and setting the area ratio of the third area to the fourth area as the working image fatigue degree;

[0025] The work processing unit is further configured with a work eyebrow region analysis strategy, which includes obtaining a region outline of a pixel region with a grayscale value of 255 in a work eyebrow binary image, obtaining a horizontal length and a vertical length of the region outline, and setting them as the horizontal length and vertical length of the work eyebrow;

[0026] The working processing unit is also configured with a working lip area analysis strategy, which includes obtaining the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the working lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the working image pleasantness.

[0027] Furthermore, the work processing unit is further configured with a work infrared processing strategy, which includes acquiring work thermal imaging data to establish a facial work three-dimensional model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial work three-dimensional model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the work fatigue distance;

[0028] Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the working 3D facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height;

[0029] The leftmost point, rightmost point and lowest point of the lip are obtained from the working three-dimensional facial model. The line connecting the leftmost point and the lowest point of the lip is set as the working mouth corner oblique line, the line connecting the leftmost point and the rightmost point of the lip is set as the working lip horizontal line, and the angle between the working mouth corner oblique line and the working lip horizontal line is set as the working infrared pleasantness.

[0030] Furthermore, the deep learning module is configured with a deep learning strategy, which includes establishing a convolutional neural network, wherein the input layer of the convolutional neural network is used to input the basic facial information and work facial information of the employee;

[0031] The convolutional layer of the convolutional neural network is used to convolve the basic facial information and the work facial information of the employee to obtain multiple feature contours and multiple feature points;

[0032] The fully connected layer of the convolutional neural network is used to calculate basic state data and working state data according to the basic feature profile, the working feature profile, the basic feature points and the working feature points, the basic state data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working state data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness;

[0033] Substitute the basic state data into the basic state formula , the basic state value is calculated; among them, T0 is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, a1+a2=1, and a1 and a2 are both greater than zero, b is the fatigue weight value, and c is the pleasure weight value;

[0034] Substitute the working status data into the working status formula , the working state value is calculated; where T is the working state value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, and Y′ is the working infrared pleasure;

[0035] The value of the working state value minus the basic state value is set as the mood value.

[0036] Furthermore, the emotion assessment module includes an emotion assessment unit, and the emotion assessment unit is configured with an emotion assessment strategy. The emotion assessment strategy includes obtaining the employee's working facial information in real time, substituting the employee's working facial information into a convolutional neural network to obtain an emotion value. When the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state; when the employee's emotion value is negative, the employee is judged to be in a fatigue state.

[0037] The emotion assessment module further includes an emotion correction unit configured with an emotion correction strategy. The emotion correction strategy includes: collecting historical status statistics of a first number of employees, setting a ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state percentage, obtaining a median of the fatigue state percentages of the first number of employees, and setting the median as a fatigue state percentage threshold;

[0038] When the fatigue state percentage of any employee is greater than or equal to the fatigue state percentage threshold, the employee's mood will be modified;

[0039] The emotion correction includes obtaining the employee's historical work status data, calculating the average value of the employee's historical work status data, and correcting the employee's basic status data to the average value of the employee's historical work status data.

[0040] In a second aspect, the present application provides an employee status monitoring method based on deep learning of facial multimodal information, the method comprising the following steps:

[0041] Step S1: Collecting basic facial information and work facial information of employees;

[0042] Step S2: dividing and extracting the basic facial image to obtain basic state data, wherein the basic state data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height, and basic infrared pleasure;

[0043] Step S3: dividing and extracting the working facial image to obtain working status data, wherein the working status data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasure, working fatigue distance, working eyebrow length, working eyebrow height, and working infrared pleasure;

[0044] Step S4: constructing a convolutional neural network, and training the constructed convolutional neural network using the basic state data and the working state data;

[0045] Step S5: Evaluate the employee's work facial information based on the convolutional neural network to obtain the employee's real-time status.

[0046] Beneficial effects of the present invention: The present invention first collects the basic status of employees to obtain basic facial information of employees, and then collects the work status of employees to obtain work facial information of employees. By comparing and calibrating the basic facial information and work facial information of employees, the accuracy of real-time status analysis of employees can be improved;

[0047] The facial acquisition method of the present invention includes image acquisition and infrared acquisition. The two methods are used to collect employee facial information, which solves the problem of image acquisition errors caused by obstructions or low infrared acquisition accuracy caused by other heat sources in the environment when using a single acquisition method.

[0048] The present invention also convolves the collected basic facial information and working facial information by constructing a convolutional neural network, which can automatically extract useful feature representations from the original data without the need for manual feature engineering, greatly reducing the workload and improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0050] Figure 1 is a functional block diagram of the system of the present invention;

[0051] Figure 2 A side view of the first region of the present invention;

[0052] Figure 3 This is a schematic diagram of the plane rectangular coordinate system of the present invention;

[0053] Figure 4 Flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0055] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0056] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0057] Example 1, please refer to Figure 1 As shown, the employee status monitoring system based on deep learning of facial multimodal information includes a facial acquisition module, a facial processing module, a deep learning module, and an emotion assessment module; the facial acquisition module is used to collect basic facial information and work facial information of employees; the facial acquisition module includes a work facial information acquisition unit and a basic facial information acquisition unit, and the work facial information acquisition unit is used to collect work facial information, and the work facial information includes work facial images and work thermal imaging data;

[0058] The work face information acquisition unit is configured with a work image capture strategy, which includes using an image acquisition device to acquire a work area scene image, locating a face acquisition area from the work area scene image, and capturing an image of an employee's face within the face acquisition area to obtain a work face image;

[0059] See also Figure 2As shown, the work facial information collection unit is further configured with a face collection area positioning strategy, which includes dividing the work area scene image into several workstation areas and other areas, analyzing each workstation area separately, obtaining the computer screen orientation of the employees in each workstation area, and setting the area in the first work area in front of the computer screen orientation as the face collection area. In the specific implementation process, when the employees use the computer screen, their faces will move within a certain range, but will remain within a range of 1 meter horizontally from the computer screen. Therefore, the first work area is set to a rectangular range with a horizontal distance of 1 meter from the computer screen, a length of the computer screen, and a width of the computer screen. This can ensure that the employees' faces always remain within the collection area when they are working, avoiding the problem of not collecting the employees' faces due to the collection area being too small or collecting other employees' faces due to the collection area being too large.

[0060] The working facial information collection unit is also equipped with a working infrared collection strategy, which includes using infrared sensing equipment to obtain thermal imaging data of the face collection area, which is set as working thermal imaging data:

[0061] The basic facial information acquisition unit is used to acquire basic facial information, which includes a basic facial image and basic thermal imaging data;

[0062] The basic facial information acquisition unit is configured with a basic image acquisition strategy, which includes using a photographic device to capture a frontal photo of the employee and setting the photo as a basic facial image;

[0063] The basic facial information collection unit is also equipped with a basic infrared collection strategy, which includes using infrared sensing equipment to obtain frontal thermal imaging data of employees, which is set as basic thermal imaging data. In the specific implementation process, when using photographic equipment to collect frontal photos of employees, the frontal photos of employees may be incomplete due to obstruction by items such as clothing and glasses. When using infrared sensing equipment to collect frontal thermal imaging data of employees, the frontal thermal imaging data may be affected by the heat of other equipment, resulting in inaccurate frontal thermal imaging data. Therefore, using photographic equipment and infrared sensing equipment to obtain frontal data of employees at the same time can avoid errors caused by single collection.

[0064] The facial processing module includes a basic processing unit and a working processing unit. The basic status data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasure;

[0065] The basic processing unit is configured with a basic image processing strategy, which includes obtaining a basic facial image, dividing the basic facial image into a basic eye area, a basic brow area and a basic lip area; grayscale processing is performed on the basic eye area, the basic brow area and the basic lip area, and the grayscale image is converted into a binary image, and the pixel points with grayscale values ​​less than or equal to the first threshold are set to 0 (black), and the pixel points with grayscale values ​​greater than the first threshold are set to 255 (white); a basic eye binary image, a basic brow binary image and a basic lip binary image are obtained; in the specific implementation process, the method for obtaining the first threshold includes extracting the grayscale histogram of the image, and selecting the lowest point between the two peaks in the histogram as the first threshold. The foreground and background of the basic facial image have obvious differences in grayscale, so the two peaks correspond to the grayscale value distributions of the background and foreground pixels respectively, so selecting the lowest point between the two peaks as the first threshold can achieve better results.

[0066] The basic processing unit is also equipped with a basic eye processing strategy, which includes obtaining a basic eye binary image, setting the pixel area with a grayscale value of 255 in the basic eye binary image as the first area, and setting the pixel area with a grayscale value of 0 in the basic eye binary image as the second area; obtaining the area of ​​the first area and the second area, and setting the area ratio of the first area to the second area as the basic image fatigue; in the specific implementation process, the first area is the surrounding skin and pupil area, and the second area is the white of the eye area. When the white of the eye area is relatively smaller, the employee's eye opening range is smaller and the fatigue level is higher. For example, the area of ​​the first area is 3cm 2 , the area of ​​the second region is 5cm 2 , then the basic image fatigue is 0.6;

[0067] See also Figure 3 As shown, the basic processing unit is further configured with a basic eyebrow processing strategy, which includes obtaining a regional outline of a pixel area with a grayscale value of 255 in a basic eyebrow binary image, obtaining a horizontal length and a vertical length of the regional outline, and setting them as the basic eyebrow horizontal length and the basic eyebrow vertical length. In a specific implementation process, a plane rectangular coordinate system is established in the eyebrow binary image, and the lowest point coordinate, the highest point coordinate, the leftmost coordinate, and the rightmost coordinate of each regional outline are obtained. The difference between the vertical coordinates of the highest point coordinate and the lowest point coordinate of the regional outline is set as the vertical length of the eyebrow, and the difference between the horizontal coordinates of the leftmost coordinate and the rightmost coordinate of the regional outline is set as the horizontal length of the eyebrow. In a specific implementation process, for example, if the lowest point coordinate is (9, 5), the highest point coordinate is (10, 6), the leftmost coordinate is (2, 5.2), and the rightmost coordinate is (12, 5), then the vertical length of the eyebrow is 1, and the horizontal length of the eyebrow is 10.

[0068] The basic processing unit is also configured with a basic lip processing strategy, which includes obtaining the vertical highest point and vertical lowest point of the pixel area with a grayscale value of 255 in the basic lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the basic image pleasantness; in the specific implementation process, a plane rectangular coordinate system is established in the lip binary image, and the highest point coordinate and the lowest point coordinate of the pixel area with a grayscale value of 255 on the vertical coordinate are obtained and set as the vertical highest point and the vertical lowest point. For example, if the lowest point coordinate is (2, 2) and the highest point coordinate is (7, 4), the basic image pleasantness is 2.

[0069] The basic processing unit is also configured with a basic infrared processing strategy, which includes acquiring basic thermal imaging data to establish a basic three-dimensional facial model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance. In a specific implementation, for example, if the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid is 1.5 cm, then the basic fatigue distance is 1.5;

[0070] Obtain the leftmost point, rightmost point, highest point, and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height. In a specific implementation, for example, if the vertical distance between the highest point and the lowest point of the eyebrow is 1 cm, the basic eyebrow height is 1;

[0071] The leftmost point, rightmost point and lowest point of the lip are obtained from the basic three-dimensional model of the face, the line connecting the leftmost point of the lip and the lowest point of the lip is set as the slant line of the corner of the mouth, the line connecting the leftmost point of the lip and the rightmost point of the lip is set as the horizontal line of the lip, and the value of the angle between the slant line of the corner of the mouth and the horizontal line of the lip is set as the basic infrared pleasantness. In the specific implementation process, for example, if the angle between the slant line of the corner of the mouth and the horizontal line of the lip is 15°, the basic infrared pleasantness is 15.

[0072] The work status data include work image fatigue, work eyebrow horizontal length, work eyebrow vertical length, work image pleasure, work fatigue distance, work eyebrow length, work eyebrow height and work infrared pleasure;

[0073] The work processing unit is configured with a work image processing strategy, which includes obtaining a work facial image, extracting a work eye area, a work brow area, and a work lip area from the work facial image; comparing the total area of ​​the work eye area, the work brow area, and the work lip area with the total area of ​​the basic eye area, the basic brow area, and the basic lip area, respectively; when the total area of ​​the work eye area, the work brow area, and the work lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area, the employee is marked as leaving the job. In a specific implementation process, the first ratio is set to 50%. When the ratio of the total area of ​​the work eye area, the work brow area, and the work lip area to the total area of ​​the basic eye area, the basic brow area, and the basic lip area is larger, the area of ​​the employee's face facing the computer screen is larger;

[0074] When the total area of ​​the working eye region, the working brow region, and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region, and the basic lip region, marking the employee as being in a working state;

[0075] When the employee is in a working state, the working eye area, the working eyebrow area, and the working lip area are grayscaled, the grayscale image is converted into a binary image, the pixel points with a grayscale value less than or equal to the first threshold are set to 0 (black), and the pixel points with a grayscale value greater than the first threshold are set to 255 (white); the working eye binary image, the working eyebrow binary image, and the working lip binary image are obtained.

[0076] The working processing unit is also configured with a working eye area analysis strategy, which includes obtaining a working eye binary image, setting the pixel area with a grayscale value of 255 in the working eye binary image as the third area, and setting the pixel area with a grayscale value of 0 in the working eye binary image as the fourth area; obtaining the areas of the third area and the fourth area, and setting the area ratio of the third area to the fourth area as the working image fatigue degree. For example, the area of ​​the third area is 2cm 2 , the area of ​​the fourth region is 4cm 2 , then the working image fatigue is 0.5;

[0077] The work processing unit is also configured with a work eyebrow region analysis strategy, which includes obtaining a regional outline of a pixel region with a grayscale value of 255 in the work eyebrow binary image, obtaining the horizontal length and vertical length of the regional outline, and setting them as the horizontal length and vertical length of the work eyebrow; in a specific implementation process, for example, the lowest point coordinates are (9.5, 5.5), the highest point coordinates are (10, 6), the leftmost coordinates are (3, 5.2), and the rightmost coordinates are (11, 2), then the vertical length of the work eyebrow is 0.5, and the horizontal length of the work eyebrow is 8;

[0078] The working processing unit is also configured with a working lip area analysis strategy, which includes obtaining the vertical highest point and vertical lowest point of the pixel area with a grayscale value of 255 in the working lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the working image pleasantness; in the specific implementation process, for example, if the coordinates of the lowest point are (2, 2) and the coordinates of the highest point are (7, 3), then the working image pleasantness is 1.

[0079] The work processing unit is further configured with a work infrared processing strategy, which includes acquiring work thermal imaging data to establish a facial work three-dimensional model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial work three-dimensional model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the work fatigue distance. In a specific implementation process, for example, if the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid is 1.2 cm, then the work fatigue distance is 1.2;

[0080] Obtain the leftmost point, rightmost point, highest point, and lowest point of the eyebrow from the working three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height. In a specific implementation, for example, if the vertical distance between the highest point and the lowest point of the eyebrow is 1.5 cm, then the working eyebrow height is 1.5;

[0081] Obtain the leftmost point, rightmost point and lowest point of the lip from the working three-dimensional facial model, set the line connecting the leftmost point of the lip and the lowest point of the lip as the working mouth corner oblique line, set the line connecting the leftmost point of the lip and the rightmost point of the lip as the working lip horizontal line, and set the value of the angle between the working mouth corner oblique line and the working lip horizontal line as the working infrared pleasantness. In the specific implementation process, for example, if the angle between the working mouth corner oblique line and the working lip horizontal line is 10°, then the working infrared pleasantness is 10.

[0082] The deep learning module is used to construct a convolutional neural network and train the constructed convolutional neural network using the basic status data and the working status data. The deep learning module is configured with a deep learning strategy, which includes establishing a convolutional neural network. The input layer of the convolutional neural network is used to input the basic facial information and the working facial information of the employee.

[0083] The convolutional layer of the convolutional neural network is used to convolve the basic facial information and work facial information of the employee to obtain multiple feature contours and multiple feature points;

[0084] The convolution process includes setting the basic eye area, the basic eyebrow area and the basic lip area as the basic areas to be extracted in the convolution layer;

[0085] Dividing the working facial image in the input working facial information into a working eye area, a working eyebrow area, and a working lip area, and setting them as working areas to be extracted;

[0086] Performing contour extraction on the basic area to be extracted and the working area to be extracted to obtain a basic feature contour and a working feature contour; the basic feature contour includes a basic eye feature contour, a basic eyebrow feature contour, and a basic lip feature contour, and the working feature contour includes a working eye feature contour, a working eyebrow feature contour, and a working lip feature contour;

[0087] A basic three-dimensional facial model is established using the basic thermal imaging data in the basic facial information. The highest point of the upper eyelid, the lowest point of the lower eyelid, the leftmost point of the eyebrow, the rightmost point of the eyebrow, the highest point of the eyebrow, the lowest point of the eyebrow, the leftmost point of the lip, the rightmost point of the lip, and the lowest point of the lip are obtained from the basic three-dimensional facial model and set as basic feature points;

[0088] Using the working thermal imaging data in the working facial information, a facial working 3D model is established. The highest point of the upper eyelid, the lowest point of the lower eyelid, the leftmost point of the eyebrow, the rightmost point of the eyebrow, the highest point of the eyebrow, the lowest point of the eyebrow, the leftmost point of the lip, the rightmost point of the lip, and the lowest point of the lip are obtained from the facial working 3D model and set as working feature points.

[0089] The contour extraction comprises: performing grayscale processing on the area to be extracted, converting the grayscale image into a binary image, setting pixels with grayscale values ​​less than or equal to a first threshold value to 0, and setting pixels with grayscale values ​​greater than the first threshold value to 255; obtaining a basic binary image to be extracted and a working binary image of the area to be extracted, and extracting contours of the basic binary image to be extracted and the working binary image of the area to be extracted using an edge detection algorithm, and recording them as a basic feature contour and a working feature contour;

[0090] The fully connected layer of the convolutional neural network is used to calculate basic state data and working state data according to the basic feature profile, the working feature profile, the basic feature points and the working feature points, the basic state data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working state data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness;

[0091] Substitute the basic state data into the basic state formula In the calculation, the basic state value is obtained; wherein, T. is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value; a1+a2=1, and a1 and a2 are both greater than zero; in the specific implementation process, the visible light image can provide the basic texture and color information of the face, while the infrared image can capture the thermal radiation information of the face, and can identify camouflage or facial features hidden under items such as hats and scarves, and is not affected by obstructions. Therefore, the image weight value a1 is set to 0.4, and the infrared weight value a2 is set to 0.6;

[0092] b is the fatigue weight value, and c is the pleasure weight value. In the specific implementation process, the basic image fatigue and basic fatigue distance of 100 groups of different employees are collected. The ratio of the basic fatigue distance to the basic image fatigue of the same employee is set as the fatigue ratio. The average of the fatigue ratios of the 100 groups of employees is obtained and set as the fatigue weight value.

[0093] Collect the basic image pleasure and basic infrared pleasure of 100 different groups of employees, set the ratio of the basic image pleasure to the basic infrared pleasure of the same employee as the pleasure ratio, and obtain the average of the pleasure ratios of the 100 groups of employees and set it as the pleasure weight value;

[0094] Substitute the work status data into the work status formula , the working status value is calculated; where T is the working status value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, Y′ is the working infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, b is the fatigue weight value, and c is the pleasure weight value;

[0095] The value of the working state value minus the basic state value is set as the mood value.

[0096] The emotion assessment module evaluates the employee's work facial information based on a convolutional neural network to obtain the employee's real-time status; the emotion assessment module includes an emotion assessment unit, and the emotion assessment unit is configured with an emotion assessment strategy. The emotion assessment strategy includes obtaining the employee's work facial information in real time, substituting the employee's work facial information into the convolutional neural network, and obtaining an emotion value. When the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state; when the employee's emotion value is negative, the employee is judged to be in a fatigue state; in a specific implementation process, the employee's emotion value is a relative value based on the baseline state. When the employee's emotion value is 0 or a positive number, it indicates that the employee's real-time emotion is positive relative to the baseline state, and the employee is in a normal state; when the employee's emotion is negative, it indicates that the employee's real-time emotion is negative relative to the baseline state, and the employee is in a fatigued state.

[0097] The emotion assessment module further includes an emotion correction unit configured with an emotion correction strategy. The emotion correction strategy includes: collecting historical status statistics of a first number of employees, setting a ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state percentage, obtaining a median of the fatigue state percentages of the first number of employees, and setting the median as a fatigue state percentage threshold;

[0098] When the fatigue state percentage of any employee is greater than or equal to the fatigue state percentage threshold, the employee's mood will be modified;

[0099] Emotion correction includes obtaining the employee's historical work status data, calculating the average value of the employee's historical work status data, and correcting the employee's basic status data to the average value of the employee's historical work status data. In the specific implementation process, when the fatigue state ratio is greater than or equal to the fatigue state ratio threshold, it indicates that the employee's fatigue state is abnormally high, so the employee's basic status data needs to be corrected to avoid inaccurate judgment results due to collection errors.

[0100] Example 2, please refer to Figure 4 As shown, the present application also provides an employee status monitoring method based on deep learning of facial multimodal information, the method comprising the following steps:

[0101] Step S1: Collecting basic facial information and work facial information of employees;

[0102] Step S1 includes the following sub-steps: Step S101: using an image acquisition device to acquire a work area scene image, locating a face acquisition area from the work area scene image, dividing the work area scene image into a plurality of workstation areas and other areas, performing a separate analysis on each workstation area, obtaining the computer screen orientation of employees in each workstation area, setting an area within the first work area in front of the computer screen orientation as a face acquisition area, and capturing the faces of employees in the face acquisition area to obtain work face images;

[0103] Step S102: using an infrared sensor device to obtain thermal imaging data of the face collection area, and setting it as working thermal imaging data;

[0104] Step S103: using a photographic device to capture a frontal photo of the employee, and setting the photo as a basic facial image;

[0105] Step S104: using infrared sensing equipment to obtain front-facing thermal imaging data of the employee, and setting it as basic thermal imaging data;

[0106] Step S2: Step S1: Collect the employee's basic facial information and work facial information;

[0107] Step S2: dividing and extracting the basic facial image to obtain basic state data, wherein the basic state data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height, and basic infrared pleasure;

[0108] Step S2 includes the following sub-steps:

[0109] Step S201: obtaining a basic facial image, dividing the basic facial image into a basic eye region, a basic eyebrow region, and a basic lip region, and processing the basic eye region, the basic eyebrow region, and the basic lip region to obtain a basic image fatigue degree, a basic eyebrow horizontal length, a basic eyebrow vertical length, and a basic image pleasure degree;

[0110] Step S201 includes the following sub-steps:

[0111] Step S20101: Grayscale processing is performed on the basic eye region, the basic brow region, and the basic lip region, converting the grayscale image into a binary image, setting pixels with grayscale values ​​less than or equal to a first threshold to 0 (black), and setting pixels with grayscale values ​​greater than the first threshold to 255 (white); obtaining a basic eye binary image, a basic brow binary image, and a basic lip binary image;

[0112] Step S20102: Obtain a basic eye binary image, set a pixel region with a grayscale value of 255 in the basic eye binary image as a first region, and set a pixel region with a grayscale value of 0 in the basic eye binary image as a second region; obtain the areas of the first region and the second region, and set the area ratio of the first region to the second region as the basic image fatigue level;

[0113] Step S20103: obtaining the area outline of the pixel area with a grayscale value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the area outline, and setting them as the basic eyebrow horizontal length and basic eyebrow vertical length;

[0114] Step S20104: obtaining the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the basic lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the basic image pleasantness;

[0115] Step S202: Acquire basic thermal imaging data to establish a basic three-dimensional facial model, process the basic three-dimensional facial model, and obtain a basic fatigue distance, a basic eyebrow length, a basic eyebrow height, and a basic infrared pleasantness. Step S202 includes the following sub-steps:

[0116] Step S2201: obtaining the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance;

[0117] Step S20202: Obtain the leftmost point, rightmost point, highest point, and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height;

[0118] Step S20203: Obtain the leftmost point, rightmost point, and lowest point of the lip from the basic 3D facial model, set the line connecting the leftmost point and the lowest point of the lip as the slant line of the mouth corner, set the line connecting the leftmost point and the rightmost point of the lip as the horizontal line of the lip, and set the angle between the slant line of the mouth corner and the horizontal line of the lip as the basic infrared pleasantness;

[0119] Step S3: dividing and extracting the working facial image to obtain working status data, wherein the working status data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasure, working fatigue distance, working eyebrow length, working eyebrow height, and working infrared pleasure;

[0120] Step S3 includes the following sub-steps:

[0121] Step S301: Acquire a working facial image, and extract a working eye region, a working eyebrow region, and a working lip region from the working facial image;

[0122] Step S302: Compare the total area of ​​the working eye area, the working brow area, and the working lip area with the total area of ​​the basic eye area, the basic brow area, and the basic lip area, respectively. When the total area of ​​the working eye area, the working brow area, and the working lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area, mark the employee as leaving the job.

[0123] Step S303: When the total area of ​​the working eye area, the working brow area, and the working lip area is greater than a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area, the employee is marked as being in a working state;

[0124] Step S304: When the employee is in a working state, the working eye area, the working eyebrow area, and the working lip area are processed to obtain the work image fatigue degree, the work eyebrow horizontal length, the work eyebrow vertical length, and the work image pleasure degree;

[0125] Step S304 includes the following sub-steps:

[0126] Step S30401: Grayscale processing is performed on the working eye area, the working brow area, and the working lip area, and the grayscale image is converted into a binary image. Pixels with grayscale values ​​less than or equal to a first threshold are set to 0 (black), and pixels with grayscale values ​​greater than the first threshold are set to 255 (white); thus, a working eye binary image, a working brow binary image, and a working lip binary image are obtained.

[0127] Step S30402: Obtain a binary image of the working eye, set a pixel region with a grayscale value of 255 in the binary image of the working eye as a third region, and set a pixel region with a grayscale value of 0 in the binary image of the working eye as a fourth region; obtain the areas of the third region and the fourth region, and set the ratio of the areas of the third region to the fourth region as the working image fatigue level;

[0128] Step S30403: Obtain the area outline of the pixel area with a grayscale value of 255 in the binary image of the working eyebrow, obtain the horizontal length and vertical length of the area outline, and set them as the horizontal length and vertical length of the working eyebrow;

[0129] Step S30404: Obtain the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the binary image of the working lip, and set the height difference between the vertical highest point and the vertical lowest point as the pleasantness of the working image;

[0130] Step S305: Acquire work thermal imaging data to establish a facial work 3D model, process the facial work 3D model, and obtain work fatigue distance, work eyebrow length, work eyebrow height, and work infrared pleasure. Step S305 includes the following sub-steps:

[0131] Step S30501: obtaining the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial working three-dimensional model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the working fatigue distance;

[0132] Step S30502: Obtain the leftmost point, rightmost point, highest point, and lowest point of the eyebrow from the working three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height;

[0133] Step S30503: Obtain the leftmost point of the lip, the rightmost point of the lip and the lowest point of the lip from the working three-dimensional facial model, set the line connecting the leftmost point of the lip and the lowest point of the lip as the working mouth corner oblique line, set the line connecting the leftmost point of the lip and the rightmost point of the lip as the working lip horizontal line, and set the value of the angle between the working mouth corner oblique line and the working lip horizontal line as the working infrared pleasantness.

[0134] Step S4: Construct a convolutional neural network and train the constructed convolutional neural network using the basic state data and the working state data; Step S3 includes the following sub-steps:

[0135] Step S401: establishing a convolutional neural network;

[0136] Step S402: The input layer of the convolutional neural network is used to input the basic facial information and work facial information of the employee;

[0137] Step S403: The convolution layer convolves the employee's basic facial information and work facial information to obtain multiple feature contours and feature points;

[0138] Step S404: The fully connected layer of the convolutional neural network calculates basic state data and working state data based on multiple feature contours and feature points, where the basic state data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height, and basic infrared pleasantness; and the working state data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height, and working infrared pleasantness;

[0139] Substitute the basic state data into the basic state formula , the basic state value is calculated; among them, T0 is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, b is the fatigue weight value, and c is the pleasure weight value:

[0140] Substitute the work status data into the work status formula , the working status value is calculated; where T is the working status value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, Y′ is the working infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, b is the fatigue weight value, and c is the pleasure weight value;

[0141] The value of the working state value minus the basic state value is set as the mood value.

[0142] Step S5: Evaluate the employee's work facial information based on the convolutional neural network to obtain the employee's real-time status;

[0143] Step S5 includes the following sub-steps:

[0144] Step S501: Obtaining the employee's working facial information, substituting the employee's working facial information into a convolutional neural network to obtain an emotion value. When the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state; when the employee's emotion value is negative, the employee is judged to be in a fatigue state;

[0145] Step S502: Counting the historical statuses of a first number of employees, setting the ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state percentage, obtaining the median fatigue state percentage of the first number of employees, and setting the median fatigue state percentage as a fatigue state percentage threshold;

[0146] Step S503: When the fatigue state ratio of any employee is greater than or equal to the fatigue state ratio threshold, the employee's mood is modified;

[0147] The emotion correction includes obtaining the historical work status data of the employee, calculating the average value of the historical work status data of the employee, and correcting the basic status data of the employee to the average value of the historical work status data of the employee.

[0148] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] The above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An employee status monitoring system based on deep learning of facial multimodal information, characterized by: Includes facial acquisition module, facial processing module, deep learning module and emotion assessment module; The facial collection module is used to collect basic facial information and work facial information of employees; The facial processing module includes a basic processing unit and a working processing unit, wherein the basic processing unit is used to divide and extract the basic facial image to obtain basic state data; The work processing unit is used to divide and extract the work facial image to obtain work status data; The deep learning module is used to construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network; The emotion assessment module evaluates the employee's working facial information based on a convolutional neural network to obtain the employee's real-time status; The basic processing unit is configured with a basic image processing strategy, the basic image processing strategy comprising acquiring a basic facial image, dividing the basic facial image into a basic eye region, a basic eyebrow region, and a basic lip region; Grayscale processing is performed on the basic eye area, the basic eyebrow area, and the basic lip area, converting the grayscale image into a binary image, setting the pixel points with grayscale values ​​less than or equal to a first threshold to 0, and setting the pixel points with grayscale values ​​greater than the first threshold to 255; Obtaining a basic eye binary image, a basic eyebrow binary image, and a basic lip binary image; Obtain a basic eye binary image, set a pixel region with a grayscale value of 255 in the basic eye binary image as a first region, and set a pixel region with a grayscale value of 0 in the basic eye binary image as a second region; obtain the areas of the first region and the second region, and set the area ratio of the first region to the second region as the basic image fatigue level; Obtaining the region outline of the pixel region with a grayscale value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the region outline, and setting them as the basic eyebrow horizontal length and basic eyebrow vertical length; Obtain the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the basic lip binary image, and set the height difference between the vertical highest point and the vertical lowest point as the basic image pleasantness; The basic processing unit is further configured with a basic infrared processing strategy, which includes acquiring basic thermal imaging data to establish a basic three-dimensional facial model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance; Obtain the leftmost point, rightmost point, highest point, and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height; Obtain the leftmost point, rightmost point, and lowest point of the lip from the basic 3D facial model, set the line connecting the leftmost point and the lowest point of the lip as the slant line of the mouth corner, set the line connecting the leftmost point and the rightmost point of the lip as the horizontal line of the lip, and set the angle between the slant line of the mouth corner and the horizontal line of the lip as the basic infrared pleasantness; The work processing unit is configured with a work image processing strategy, which includes obtaining a work facial image, extracting a work eye area, a work brow area, and a work lip area from the work facial image; comparing the total area of ​​the work eye area, the work brow area, and the work lip area with the total area of ​​the basic eye area, the basic brow area, and the basic lip area, respectively; and marking the employee as leaving the job when the total area of ​​the work eye area, the work brow area, and the work lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area; When the total area of ​​the working eye region, the working brow region, and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region, and the basic lip region, marking the employee as being in a working state; When the employee is in a working state, grayscale processing is performed on the working eye area, the working eyebrow area, and the working lip area, and the grayscale image is converted into a binary image. Pixels with grayscale values ​​less than or equal to a first threshold are set to 0, and pixels with grayscale values ​​greater than the first threshold are set to 255; thus, a working eye binary image, a working eyebrow binary image, and a working lip binary image are obtained; The working processing unit is further configured with a working eye region analysis strategy, which includes obtaining a working eye binary image, setting a pixel region with a grayscale value of 255 in the working eye binary image as a third region, and setting a pixel region with a grayscale value of 0 in the working eye binary image as a fourth region; obtaining the areas of the third region and the fourth region, and setting the area ratio of the third region to the fourth region as the working image fatigue degree; The work processing unit is further configured with a work eyebrow region analysis strategy, which includes obtaining a region outline of a pixel region with a grayscale value of 255 in a work eyebrow binary image, obtaining a horizontal length and a vertical length of the region outline, and setting them as the horizontal length and vertical length of the work eyebrow; The working processing unit is further configured with a working lip area analysis strategy, which includes obtaining the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the working lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the working image pleasantness; The work processing unit is further configured with a work infrared processing strategy, which includes acquiring work thermal imaging data to establish a facial work three-dimensional model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial work three-dimensional model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the work fatigue distance; Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the working 3D facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height; The leftmost point, rightmost point and lowest point of the lip are obtained from the working three-dimensional facial model. The line connecting the leftmost point and the lowest point of the lip is set as the working mouth corner oblique line, the line connecting the leftmost point and the rightmost point of the lip is set as the working lip horizontal line, and the angle between the working mouth corner oblique line and the working lip horizontal line is set as the working infrared pleasantness.

2. The employee status monitoring system based on deep learning of facial multimodal information according to claim 1 is characterized in that: The facial acquisition module includes a working facial information acquisition unit and a basic facial information acquisition unit, wherein the working facial information acquisition unit is used to acquire working facial information, and the working facial information includes working facial images and working thermal imaging data; Collecting work facial information includes: using an image acquisition device to collect a work area scene image, locating a face collection area from the work area scene image, and capturing an image of an employee's face within the face collection area to obtain a work facial image; dividing the work area scene image into a plurality of workstation areas and other areas, analyzing each workstation area separately, obtaining the orientation of the computer screen of the employee in each workstation area, and setting an area within the first work area in front of the computer screen orientation as a face collection area; using an infrared sensor device to obtain thermal imaging data of the face collection area, and setting the data as work thermal imaging data; The basic facial information acquisition unit is used to acquire basic facial information, wherein the basic facial information includes a basic facial image and basic thermal imaging data; The basic facial information acquisition unit is configured with a basic image acquisition strategy, which includes using a photographic device to capture a frontal photo of the employee and setting the photo as a basic facial image; and using an infrared sensing device to obtain the frontal thermal imaging data of the employee and setting it as the basic thermal imaging data.

3. The employee status monitoring system based on deep learning of facial multimodal information according to claim 2 is characterized in that: The deep learning module is configured with a deep learning strategy, wherein the deep learning strategy includes establishing a convolutional neural network, wherein the input layer of the convolutional neural network is used to input the basic facial information and the working facial information of the employee; The convolutional layer of the convolutional neural network is used to convolve the basic facial information and the work facial information of the employee to obtain multiple feature contours and multiple feature points; The fully connected layer of the convolutional neural network is used to calculate basic state data and working state data according to the basic feature profile, the working feature profile, the basic feature points and the working feature points, the basic state data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working state data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness; Substitute the basic state data into the basic state formula , calculate the basic state value; among them, T0 is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0' is the basic eyebrow length, Dv0' is the basic eyebrow height, Y0' is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, a1+a2=1, and a1 and a2 are both greater than zero, b is the fatigue weight value, and c is the pleasure weight value; Substitute the working status data into the working status formula , the working state value is calculated; where T is the working state value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh' is the working eyebrow length, Dv' is the working eyebrow height, and Y' is the working infrared pleasure; The value of the working state value minus the basic state value is set as the mood value.

4. The employee status monitoring system based on deep learning of facial multimodal information according to claim 3 is characterized in that: The emotion assessment module includes an emotion assessment unit, which is configured with an emotion assessment strategy. The emotion assessment strategy includes obtaining the employee's working facial information in real time, substituting the employee's working facial information into a convolutional neural network to obtain an emotion value. When the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state; when the employee's emotion value is negative, the employee is judged to be in a fatigue state. The emotion assessment module further includes an emotion correction unit configured with an emotion correction strategy. The emotion correction strategy includes: collecting historical status statistics of a first number of employees, setting a ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state percentage, obtaining a median of the fatigue state percentages of the first number of employees, and setting the median as a fatigue state percentage threshold; When the fatigue state percentage of any employee is greater than or equal to the fatigue state percentage threshold, the employee's mood will be modified; The emotion correction includes obtaining the employee's historical work status data, calculating the average value of the employee's historical work status data, and correcting the employee's basic status data to the average value of the employee's historical work status data.

5. An employee status monitoring method based on deep learning of facial multimodal information, applicable to an employee status monitoring system based on deep learning of facial multimodal information according to any one of claims 1 to 4, characterized in that: The method comprises the following steps: Step S1: Collecting basic facial information and work facial information of employees; Step S2: dividing and extracting the basic facial image to obtain basic state data, wherein the basic state data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height, and basic infrared pleasure; Step S3: dividing and extracting the working facial image to obtain working status data, wherein the working status data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasure, working fatigue distance, working eyebrow length, working eyebrow height, and working infrared pleasure; Step S4: constructing a convolutional neural network, and training the constructed convolutional neural network using the basic state data and the working state data; Step S5: Evaluate the employee's work facial information based on the convolutional neural network to obtain the employee's real-time status.

Images