Employee state monitoring method and system based on facial multi-modal information deep learning

Abstract

The invention provides an employee state monitoring method and system based on face multi-modal information deep learning, and relates to the technical field of employee state monitoring, and the system comprises a face collection module, a face processing module, a deep learning module and an emotion evaluation module. The face acquisition module is used for acquiring basic face information and work face information of employees; the face processing module is used for analyzing the collected data; the deep learning module is used for constructing a convolutional neural network; the emotion evaluation module evaluates the work face information of the employee based on a convolutional neural network to obtain the real-time state of the employee; according to the invention, the face data of the employees in the basic state and the working state are obtained, and the face data of the employees in different states are compared, analyzed and processed to obtain the real-time states of the employees, so that the problem that the fatigue states of the employees cannot be judged due to the fact that the existing employee monitoring system does not obtain and analyze the face states of the employees is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of employee status monitoring, and in particular to an employee status monitoring method and system based on deep learning of facial multimodal information. Background Art

[0002] With the rapid development of science and technology, employee status monitoring technology plays an increasingly important role in enterprise management. Through comprehensive monitoring and analysis of employees' physiological indicators, psychological state, work efficiency, etc., enterprises can more accurately grasp the work status of employees and thus formulate more effective management strategies.

[0003] In the prior art, the work status of employees is often judged by monitoring their work results and background processes. For example, a Chinese patent application with publication number CN116629651A discloses a work status process monitoring method, device and storage medium. The method includes: obtaining work status data of a target monitoring person within a unit working time; calculating a first work seriousness coefficient value, a second work seriousness coefficient value and a third work seriousness coefficient value according to the obtained work status data; performing weighted calculation on the first work seriousness coefficient value, the second work seriousness coefficient value and the third work seriousness coefficient value to obtain a total work seriousness coefficient value; presetting a total work seriousness coefficient threshold to judge the work seriousness of the target monitoring person within the unit working time. Whether the work status is serious; this method is to monitor the background process of the employee, without obtaining and analyzing the mental state of the employee. The existing method of monitoring the employee status through video usually analyzes whether the employee is on the job. For example, Chinese patent application with publication number CN114663809A discloses an employee status recognition method, device, electronic equipment and medium. This method determines the positional relationship between the target position and the reference position. If the target position is within the reference position, it determines that the workstation to be identified is in the on-the-job status of the employee. The on-the-job status can be automatically identified according to the video surveillance data. However, the real-time working status of the employee is lacking, and the fatigue status of the employee cannot be judged. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an employee status monitoring method and system based on deep learning of facial multimodal information. By acquiring facial data of employees in a basic state and a working state, the facial data of employees in different states are compared, analyzed and processed to obtain the real-time status of the employees, so as to solve the problem that the existing employee monitoring system does not acquire and analyze the facial status of employees, resulting in the inability to judge the fatigue status of employees.

[0005] In order to achieve the above-mentioned object, the present invention is implemented through the following technical solutions: In a first aspect, the present application provides an employee status monitoring system based on deep learning of facial multimodal information, including a facial acquisition module, a facial processing module, a deep learning module and an emotion assessment module;

[0006] The facial collection module is used to collect basic facial information and work facial information of employees;

[0007] The facial processing module includes a basic processing unit and a working processing unit, wherein the basic processing unit is used to divide and extract the basic facial image to obtain basic state data;

[0008] The work processing unit is used to divide and extract the work facial image to obtain work status data;

[0009] The deep learning module is used to construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network;

[0010] The emotion assessment module assesses the employee's work facial information based on a convolutional neural network to obtain the employee's real-time status.

[0011] Further, the facial acquisition module includes a working facial information acquisition unit and a basic facial information acquisition unit, the working facial information acquisition unit is used to acquire working facial information, the working facial information includes working facial images and working thermal imaging data;

[0012] Collecting working facial information includes: using an image acquisition device to collect a working area scene image, locating a face acquisition area from the working area scene image, and capturing an image of an employee's face in the face acquisition area to obtain a working facial image; dividing the working area scene image into a number of workstation areas and other areas, analyzing each workstation area separately, obtaining the computer screen orientation of employees in each workstation area, and setting an area in the first working area in front of the computer screen orientation as a face acquisition area; using an infrared sensor device to obtain thermal imaging data of the face acquisition area, and setting it as working thermal imaging data;

[0013] The basic facial information acquisition unit is used to acquire basic facial information, and the basic facial information includes a basic facial image and basic thermal imaging data; the basic facial information acquisition unit is configured with a basic image acquisition strategy, and the basic image acquisition strategy includes using a photographic device to acquire a frontal photo of an employee and setting the photo as a basic facial image; and using an infrared sensing device to acquire the frontal thermal imaging data of the employee and setting it as basic thermal imaging data.

[0014] Further, the basic processing unit is configured with a basic image processing strategy, which includes obtaining a basic facial image, dividing the basic facial image into a basic eye area, a basic eyebrow area, and a basic lip area; graying the basic eye area, the basic eyebrow area, and the basic lip area, converting the grayscale image into a binary image, setting the pixel points whose grayscale value is less than or equal to a first threshold value to 0, and setting the pixel points whose grayscale value is greater than the first threshold value to 255; obtaining a basic eye binary image, a basic eyebrow binary image, and a basic lip binary image;

[0015] Obtain a basic eye binary image, set a pixel area with a gray value of 255 in the basic eye binary image as a first area, and set a pixel area with a gray value of 0 in the basic eye binary image as a second area; obtain the areas of the first area and the second area, and set the area ratio of the first area to the second area as the basic image fatigue degree;

[0016] Obtaining the area contour of the pixel area with a gray value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the area contour, and setting them as the basic eyebrow horizontal length and the basic eyebrow vertical length;

[0017] The vertical highest point and the vertical lowest point of the pixel area with a gray value of 255 in the basic lip binary image are obtained, and the height difference between the vertical highest point and the vertical lowest point is set as the basic image pleasantness.

[0018] Furthermore, the basic processing unit is also configured with a basic infrared processing strategy, which includes acquiring basic thermal imaging data to establish a basic three-dimensional facial model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the value of the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance;

[0019] Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height;

[0020] The leftmost point, rightmost point and lowest point of the lip are obtained from the basic three-dimensional facial model, the line connecting the leftmost point of the lip and the lowest point of the lip is set as the slant line of the corner of the mouth, the line connecting the leftmost point of the lip and the rightmost point of the lip is set as the horizontal line of the lip, and the value of the angle between the slant line of the corner of the mouth and the horizontal line of the lip is set as the basic infrared pleasantness.

[0021] Further, the work processing unit is configured with a work image processing strategy, which includes acquiring a work facial image, extracting a work eye area, a work brow area, and a work lip area from the work facial image; comparing the total area of ​​the work eye area, the work brow area, and the work lip area with the total area of ​​the basic eye area, the basic brow area, and the basic lip area, respectively, and when the total area of ​​the work eye area, the work brow area, and the work lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area, marking the employee as leaving the job;

[0022] When the total area of ​​the working eye region, the working brow region, and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region, and the basic lip region, marking the employee as being in a working state;

[0023] When the employee is in a working state, the working eye area, the working eyebrow area and the working lip area are grayed, the grayscale image is converted into a binary image, the pixel points with a grayscale value less than or equal to the first threshold are set to 0, and the pixel points with a grayscale value greater than the first threshold are set to 255; a working eye binary image, a working eyebrow binary image and a working lip binary image are obtained.

[0024] Furthermore, the working processing unit is also configured with a working eye area analysis strategy, which includes obtaining a working eye binary image, setting a pixel area with a gray value of 255 in the working eye binary image as a third area, and setting a pixel area with a gray value of 0 in the working eye binary image as a fourth area; obtaining the areas of the third area and the fourth area, and setting the area ratio of the third area to the fourth area as the working image fatigue degree;

[0025] The work processing unit is also configured with a work eyebrow region analysis strategy, which includes obtaining a region contour of a pixel region with a gray value of 255 in a work eyebrow binary image, obtaining a horizontal length and a vertical length of the region contour, and setting them as a work eyebrow horizontal length and a work eyebrow vertical length;

[0026] The working processing unit is also configured with a working lip area analysis strategy, which includes obtaining the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the working lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the working image pleasantness.

[0027] Furthermore, the work processing unit is also configured with a work infrared processing strategy, which includes acquiring work thermal imaging data to establish a facial work three-dimensional model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial work three-dimensional model, and setting the value of the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the work fatigue distance;

[0028] Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the facial working three-dimensional model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height;

[0029] The leftmost point, rightmost point and lowest point of the lip are obtained from the working three-dimensional facial model, the line connecting the leftmost point of the lip and the lowest point of the lip is set as the working mouth corner oblique line, the line connecting the leftmost point of the lip and the rightmost point of the lip is set as the working lip horizontal line, and the value of the angle between the working mouth corner oblique line and the working lip horizontal line is set as the working infrared pleasantness.

[0030] Further, the deep learning module is configured with a deep learning strategy, and the deep learning strategy includes establishing a convolutional neural network, and the input layer of the convolutional neural network is used to input the basic facial information and work facial information of the employee;

[0031] The convolution layer of the convolutional neural network is used to convolve the basic facial information and the working facial information of the employee to obtain multiple feature contours and multiple feature points;

[0032] The fully connected layer of the convolutional neural network is used to calculate basic status data and working status data according to the basic feature contour, the working feature contour, the basic feature points and the working feature points, the basic status data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working status data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness;

[0033] Substitute the basic state data into the basic state formula , the basic state value is calculated; wherein T0 is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, a1+a2=1, and a1 and a2 are both greater than zero, b is the fatigue weight value, and c is the pleasure weight value;

[0034] Substitute the working status data into the working status formula , the working state value is calculated; where T is the working state value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, and Y′ is the working infrared pleasure;

[0035] The value of the working state value minus the basic state value is set as the emotion value.

[0036] Further, the emotion assessment module includes an emotion assessment unit, the emotion assessment unit is configured with an emotion assessment strategy, the emotion assessment strategy includes obtaining the employee's work facial information in real time, substituting the employee's work facial information into a convolutional neural network to obtain an emotion value, when the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state, and when the employee's emotion value is a negative number, the employee is judged to be in a fatigue state;

[0037] The emotion assessment module further includes an emotion correction unit, wherein the emotion correction unit is configured with an emotion correction strategy, wherein the emotion correction strategy includes counting the historical status of a first number of employees, setting the ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state proportion, obtaining the median of the fatigue state proportion of the first number of employees, and setting the median as a fatigue state proportion threshold;

[0038] When the fatigue state percentage of any employee is greater than or equal to the fatigue state percentage threshold, the employee's mood will be modified;

[0039] The emotion correction includes obtaining the historical work status data of the employee, calculating the average value of the historical work status data of the employee, and correcting the basic status data of the employee to the average value of the historical work status data of the employee.

[0040] In a second aspect, the present application provides an employee status monitoring method based on deep learning of facial multimodal information, the method comprising the following steps:

[0041] Step S1: Collecting basic facial information and work facial information of employees;

[0042] Step S2: dividing and extracting the basic facial image to obtain basic status data, wherein the basic status data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasure;

[0043] Step S3: dividing and extracting the working facial image to obtain working status data, wherein the working status data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasure, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasure;

[0044] Step S4: construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network;

[0045] Step S5: Evaluate the employee's work facial information based on the convolutional neural network to obtain the employee's real-time status.

[0046] Beneficial effects of the present invention: The present invention first collects the basic status of employees to obtain the basic facial information of employees, and then collects the working status of employees to obtain the working facial information of employees. By comparing and calibrating the basic facial information and the working facial information of employees, the accuracy of real-time status analysis of employees can be improved;

[0047] The facial acquisition of the present invention includes image acquisition and infrared acquisition. The facial information of employees is collected by using image acquisition and infrared acquisition in two ways, which solves the problem that in the case of a single acquisition method, image acquisition errors are caused by obstruction of objects or the infrared acquisition accuracy is low due to other heat sources in the environment.

[0048] The present invention also convolves the collected basic facial information and working facial information by constructing a convolutional neural network, which can automatically extract useful feature representations from the original data without the need for manual feature engineering, greatly reducing the workload and improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0050] Figure 1 is a functional block diagram of the system of the present invention;

[0051] Figure 2 A side view of the first region of the present invention;

[0052] Figure 3 It is a schematic diagram of marking the plane rectangular coordinate system of the present invention;

[0053] Figure 4 The figure is a flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0055] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0056] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0057] Example 1, please refer to Figure 1 As shown, an employee status monitoring system based on deep learning of facial multimodal information includes a facial acquisition module, a facial processing module, a deep learning module and an emotion assessment module; the facial acquisition module is used to collect basic facial information and work facial information of employees; the facial acquisition module includes a work facial information acquisition unit and a basic facial information acquisition unit, the work facial information acquisition unit is used to collect work facial information, and the work facial information includes work facial images and work thermal imaging data;

[0058] The work face information acquisition unit is configured with a work image capture strategy, which includes using an image acquisition device to acquire a work area scene image, locating a face acquisition area from the work area scene image, and capturing an image of an employee's face in the face acquisition area to obtain a work face image;

[0059] See also Figure 2As shown, the work facial information collection unit is also configured with a face collection area positioning strategy, which includes dividing the work area scene image, dividing the work area scene image into several workstation areas and other areas, analyzing each workstation area separately, obtaining the computer screen orientation of the employees in each workstation area, and setting the area in the first work area in front of the computer screen orientation as the face collection area. In the specific implementation process, when the employees use the computer screen, their faces will move within a certain range, but will remain within a range of 1 meter horizontally from the computer screen. Therefore, the first work area is set to a rectangular range with a horizontal distance of 1 meter from the computer screen, a length of the computer screen, and a width of the computer screen. This can ensure that the employees' faces are always within the collection area when they are in the working state, avoiding the problem that the employee's face cannot be collected due to the collection area being too small or the face of other employees is collected due to the collection area being too large;

[0060] The working facial information collection unit is also configured with a working infrared collection strategy, which includes using infrared sensing equipment to obtain thermal imaging data of the face collection area, which is set as working thermal imaging data:

[0061] The basic facial information acquisition unit is used to acquire basic facial information, the basic facial information including basic facial images and basic thermal imaging data;

[0062] The basic facial information acquisition unit is configured with a basic image acquisition strategy, which includes using a photographic device to acquire a frontal photo of the employee and setting the photo as a basic facial image;

[0063] The basic facial information collection unit is also configured with a basic infrared collection strategy, which includes using infrared sensing equipment to obtain the frontal thermal imaging data of employees, which is set as basic thermal imaging data; in the specific implementation process, when using photographic equipment to collect frontal photos of employees, the frontal photos of employees collected may be incomplete due to the obstruction of items such as clothing and glasses, and when using infrared sensing equipment to obtain frontal thermal imaging data of employees, the frontal thermal imaging data collected may be inaccurate due to the heat of other equipment. Therefore, using photographic equipment and infrared sensing equipment to obtain frontal data of employees at the same time can avoid errors caused by single collection;

[0064] The facial processing module includes a basic processing unit and a working processing unit, and the basic status data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness;

[0065] The basic processing unit is configured with a basic image processing strategy, which includes obtaining a basic facial image, dividing the basic facial image into a basic eye area, a basic eyebrow area and a basic lip area; graying the basic eye area, the basic eyebrow area and the basic lip area, converting the grayscale image into a binary image, setting the pixel points whose grayscale value is less than or equal to the first threshold to 0 (black), and setting the pixel points whose grayscale value is greater than the first threshold to 255 (white); obtaining a basic eye binary image, a basic eyebrow binary image and a basic lip binary image; in a specific implementation process, the method for obtaining the first threshold includes extracting a grayscale histogram of the image, selecting the lowest point between two peaks in the histogram as the first threshold, the foreground and background of the facial basic image have obvious differences in grayscale, so the two peaks correspond to the grayscale value distribution of the background and foreground pixels respectively, so selecting the lowest point between the two peaks as the first threshold can achieve better results.

[0066] The basic processing unit is also equipped with a basic eye processing strategy, which includes obtaining a basic eye binary image, setting a pixel area with a gray value of 255 in the basic eye binary image as a first area, and setting a pixel area with a gray value of 0 in the basic eye binary image as a second area; obtaining the areas of the first area and the second area, and setting the area ratio of the first area to the second area as the basic image fatigue; in the specific implementation process, the first area is the surrounding skin and pupil area, and the second area is the white of the eye area. When the white of the eye area is relatively small, the employee's eye opening range is smaller and the fatigue level is higher. For example, the area of ​​the first area is 3cm 2 , the area of ​​the second region is 5cm 2 , then the basic image fatigue is 0.6;

[0067] See also Figure 3 As shown, the basic processing unit is also configured with a basic eyebrow processing strategy, which includes obtaining the regional contour of the pixel area with a grayscale value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the regional contour, and setting them as the basic eyebrow horizontal length and the basic eyebrow vertical length. In the specific implementation process, a plane rectangular coordinate system is established in the eyebrow binary image, the lowest point coordinate, the highest point coordinate, the leftmost coordinate and the rightmost coordinate of each regional contour are obtained, and the difference between the ordinate of the highest point coordinate and the lowest point coordinate of the regional contour is set as the eyebrow vertical length, and the difference between the abscissa of the leftmost coordinate and the rightmost coordinate of the regional contour is set as the eyebrow horizontal length; in the specific implementation process, for example, the lowest point coordinate is (9, 5), the highest point coordinate is (10, 6), the leftmost coordinate is (2, 5.2), and the rightmost coordinate is (12, 5), then the eyebrow vertical length is 1, and the eyebrow horizontal length is 10;

[0068] The basic processing unit is also configured with a basic lip processing strategy, which includes obtaining the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the basic lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the basic image pleasantness; in the specific implementation process, a plane rectangular coordinate system is established in the lip binary image, and the highest point coordinate and the lowest point coordinate of the pixel area with a grayscale value of 255 on the vertical coordinate are obtained and set as the vertical highest point and the vertical lowest point. For example, if the lowest point coordinate is (2, 2) and the highest point coordinate is (7, 4), then the basic image pleasantness is 2.

[0069] The basic processing unit is also configured with a basic infrared processing strategy, which includes acquiring basic thermal imaging data to establish a basic three-dimensional facial model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance. In the specific implementation process, for example, if the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid is 1.5 cm, the basic fatigue distance is 1.5;

[0070] The leftmost point, rightmost point, highest point and lowest point of the eyebrow are obtained from the basic three-dimensional facial model, and the horizontal distance between the leftmost point and the rightmost point of the eyebrow is set as the basic eyebrow length, and the vertical distance between the highest point and the lowest point of the eyebrow is set as the basic eyebrow height. In the specific implementation process, for example, if the vertical distance between the highest point and the lowest point of the eyebrow is 1 cm, the basic eyebrow height is 1;

[0071] The leftmost point, rightmost point and lowest point of the lips are obtained from the basic three-dimensional model of the face, the line connecting the leftmost point of the lips and the lowest point of the lips is set as the slant line of the corner of the mouth, the line connecting the leftmost point of the lips and the rightmost point of the lips is set as the horizontal line of the lips, and the value of the angle between the slant line of the corner of the mouth and the horizontal line of the lips is set as the basic infrared pleasantness. In the specific implementation process, for example, if the angle between the slant line of the corner of the mouth and the horizontal line of the lips is 15°, the basic infrared pleasantness is 15.

[0072] The work status data include work image fatigue, work eyebrow horizontal length, work eyebrow vertical length, work image pleasure, work fatigue distance, work eyebrow length, work eyebrow height and work infrared pleasure;

[0073] The work processing unit is configured with a work image processing strategy, which includes obtaining a work facial image, extracting a work eye area, a work eyebrow area, and a work lip area from the work facial image; comparing the total area of ​​the work eye area, the work eyebrow area, and the work lip area with the total area of ​​the basic eye area, the basic eyebrow area, and the basic lip area, respectively; when the total area of ​​the work eye area, the work eyebrow area, and the work lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic eyebrow area, and the basic lip area, marking the employee as leaving the job position; in a specific implementation process, the first ratio is set to 50%, and when the ratio of the total area of ​​the work eye area, the work eyebrow area, and the work lip area to the total area of ​​the basic eye area, the basic eyebrow area, and the basic lip area is larger, the area of ​​the employee's face facing the computer screen is larger;

[0074] When the total area of ​​the working eye region, the working brow region, and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region, and the basic lip region, marking the employee as being in a working state;

[0075] When the employee is in a working state, the working eye area, the working eyebrow area and the working lip area are grayed, the grayscale image is converted into a binary image, the pixel points with a grayscale value less than or equal to the first threshold are set to 0 (black), and the pixel points with a grayscale value greater than the first threshold are set to 255 (white); a working eye binary image, a working eyebrow binary image and a working lip binary image are obtained.

[0076] The working processing unit is also configured with a working eye area analysis strategy, which includes obtaining a working eye binary image, setting a pixel area with a gray value of 255 in the working eye binary image as a third area, and setting a pixel area with a gray value of 0 in the working eye binary image as a fourth area; obtaining the areas of the third area and the fourth area, and setting the area ratio of the third area to the fourth area as the working image fatigue degree, for example, the area of ​​the third area is 2cm 2 , the area of ​​the fourth region is 4cm 2 , then the working image fatigue is 0.5;

[0077] The working processing unit is also configured with a working eyebrow region analysis strategy, which includes obtaining a region contour of a pixel region with a grayscale value of 255 in a working eyebrow binary image, obtaining a horizontal length and a vertical length of the region contour, and setting them as the horizontal length and the vertical length of the working eyebrow; in a specific implementation process, for example, the coordinates of the lowest point are (9.5, 5.5), the coordinates of the highest point are (10, 6), the leftmost coordinates are (3, 5.2), and the rightmost coordinates are (11, 2), then the vertical length of the working eyebrow is 0.5, and the horizontal length of the working eyebrow is 8;

[0078] The working processing unit is also configured with a working lip area analysis strategy, which includes obtaining the vertical highest point and vertical lowest point of the pixel area with a grayscale value of 255 in the working lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the working image pleasantness; in the specific implementation process, for example, if the coordinates of the lowest point are (2, 2) and the coordinates of the highest point are (7, 3), then the working image pleasantness is 1.

[0079] The work processing unit is also configured with a work infrared processing strategy, which includes acquiring work thermal imaging data to establish a facial work three-dimensional model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial work three-dimensional model, and setting the value of the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the work fatigue distance. In the specific implementation process, for example, if the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid is 1.2 cm, the work fatigue distance is 1.2;

[0080] The leftmost point, rightmost point, highest point and lowest point of the eyebrow are obtained from the facial working three-dimensional model, the horizontal distance between the leftmost point and the rightmost point of the eyebrow is set as the working eyebrow length, and the vertical distance between the highest point and the lowest point of the eyebrow is set as the working eyebrow height. In the specific implementation process, for example, if the vertical distance between the highest point and the lowest point of the eyebrow is 1.5 cm, the working eyebrow height is 1.5;

[0081] The leftmost point of the lip, the rightmost point of the lip and the lowest point of the lip are obtained from the working three-dimensional facial model, the line connecting the leftmost point of the lip and the lowest point of the lip is set as the working mouth corner oblique line, the line connecting the leftmost point of the lip and the rightmost point of the lip is set as the working lip horizontal line, and the value of the angle between the working mouth corner oblique line and the working lip horizontal line is set as the working infrared pleasantness. In the specific implementation process, for example, if the angle between the working mouth corner oblique line and the working lip horizontal line is 10°, then the working infrared pleasantness is 10.

[0082] The deep learning module is used to construct a convolutional neural network, and the basic status data and the working status data are used to train the constructed convolutional neural network; the deep learning module is configured with a deep learning strategy, and the deep learning strategy includes establishing a convolutional neural network, and the input layer of the convolutional neural network is used to input the basic facial information and working facial information of the employee;

[0083] The convolutional layer of the convolutional neural network is used to convolve the basic facial information and work facial information of the employee to obtain multiple feature contours and multiple feature points;

[0084] The convolution process includes setting the basic eye area, the basic eyebrow area and the basic lip area as the basic areas to be extracted in the convolution layer;

[0085] The working face image in the input working face information is divided into a working eye area, a working eyebrow area and a working lip area, which are set as working areas to be extracted;

[0086] Performing contour extraction on the basic area to be extracted and the working area to be extracted to obtain a basic feature contour and a working feature contour; the basic feature contour includes a basic eye feature contour, a basic eyebrow feature contour and a basic lip feature contour, and the working feature contour includes a working eye feature contour, a working eyebrow feature contour and a working lip feature contour;

[0087] Use the basic thermal imaging data in the basic facial information to establish a basic three-dimensional facial model, obtain the highest point of the upper eyelid, the lowest point of the lower eyelid, the leftmost point of the eyebrow, the rightmost point of the eyebrow, the highest point of the eyebrow, the lowest point of the eyebrow, the leftmost point of the lip, the rightmost point of the lip and the lowest point of the lip in the basic three-dimensional facial model, and set them as basic feature points;

[0088] Using the working thermal imaging data in the working facial information, a facial working three-dimensional model is established, and the highest point of the upper eyelid, the lowest point of the lower eyelid, the leftmost point of the eyebrow, the rightmost point of the eyebrow, the highest point of the eyebrow, the lowest point of the eyebrow, the leftmost point of the lip, the rightmost point of the lip and the lowest point of the lip are obtained in the facial working three-dimensional model and set as working feature points;

[0089] The contour extraction comprises: graying the area to be extracted, converting the gray image into a binary image, setting the pixel points whose gray value is less than or equal to the first threshold value to 0, and setting the pixel points whose gray value is greater than the first threshold value to 255; obtaining a basic binary image to be extracted and a working binary image of the area to be extracted, and extracting the contours of the basic binary image to be extracted and the working binary image of the area to be extracted by using an edge detection algorithm, and recording them as a basic feature contour and a working feature contour;

[0090] The fully connected layer of the convolutional neural network is used to calculate basic status data and working status data according to the basic feature contour, the working feature contour, the basic feature points and the working feature points, the basic status data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working status data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness;

[0091] Substitute the basic state data into the basic state formula In the calculation, the basic state value is obtained; where T. is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, and a2 is the infrared weight value; a1+a2=1, and a1 and a2 are both greater than zero; in the specific implementation process, the visible light image can provide the basic texture and color information of the face, while the infrared image can capture the thermal radiation information of the face, and can identify camouflage or facial features hidden under items such as hats and scarves, and is not affected by occlusions. Therefore, the image weight value a1 is set to 0.4, and the infrared weight value a2 is set to 0.6;

[0092] b is the fatigue weight value, and c is the pleasure weight value. In the specific implementation process, the basic image fatigue and basic fatigue distance of 100 groups of different employees are collected, and the ratio of the basic fatigue distance and basic image fatigue of the same employee is set as the fatigue ratio. The average value of the fatigue ratio of the 100 groups of employees is obtained and set as the fatigue weight value.

[0093] Collect the basic image pleasure and basic infrared pleasure of 100 groups of different employees, set the ratio of the basic image pleasure and basic infrared pleasure of the same employee as the pleasure ratio, obtain the average of the pleasure ratios of the 100 groups of employees, and set it as the pleasure weight value;

[0094] Substitute the work status data into the work status formula , the working state value is calculated; where T is the working state value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, Y′ is the working infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, b is the fatigue weight value, and c is the pleasure weight value;

[0095] The value of the working state value minus the basic state value is set as the emotion value.

[0096] The emotion evaluation module evaluates the employee's work facial information based on a convolutional neural network to obtain the employee's real-time status; the emotion evaluation module includes an emotion evaluation unit, and the emotion evaluation unit is configured with an emotion evaluation strategy, and the emotion evaluation strategy includes obtaining the employee's work facial information in real time, substituting the employee's work facial information into the convolutional neural network, and obtaining an emotion value. When the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state, and when the employee's emotion value is negative, the employee is judged to be in a fatigue state; in the specific implementation process, the employee's emotion value is a relative value based on the basic state. When the employee's emotion value is 0 or a positive number, it means that the employee's real-time emotion is positive relative to the basic state, and the employee is in a normal state; when the employee's emotion is negative, it means that the employee's real-time emotion is negative relative to the basic state, and the employee is in a fatigue state;

[0097] The emotion assessment module further includes an emotion correction unit, which is configured with an emotion correction strategy. The emotion correction strategy includes counting the historical status of a first number of employees, setting the ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state proportion, obtaining the median of the fatigue state proportion of the first number of employees, and setting the median as a fatigue state proportion threshold;

[0098] When the fatigue state percentage of any employee is greater than or equal to the fatigue state percentage threshold, the employee's mood will be modified;

[0099] Emotion correction includes obtaining the employee's historical work status data, calculating the average value of the employee's historical work status data, and correcting the employee's basic status data to the average value of the employee's historical work status data. In the specific implementation process, when the fatigue state ratio is greater than or equal to the fatigue state ratio threshold, it indicates that the employee's fatigue state is abnormally high. Therefore, the employee's basic status data needs to be corrected to avoid inaccurate judgment results due to collection errors.

[0100] Example 2, please refer to Figure 4 As shown, the present application also provides an employee status monitoring method based on deep learning of facial multimodal information, the method comprising the following steps:

[0101] Step S1: Collecting basic facial information and work facial information of employees;

[0102] Step S1 includes the following sub-steps: Step S101: Use an image acquisition device to acquire a scene image of a work area, locate a face acquisition area from the scene image of the work area, divide the scene image of the work area, divide the scene image of the work area into a plurality of workstation areas and other areas, analyze each workstation area separately, obtain the computer screen orientation of employees in each workstation area, set the area in the first work area in front of the computer screen orientation as a face acquisition area, and capture the faces of employees in the face acquisition area to obtain a work face image;

[0103] Step S102: using an infrared sensor device to obtain thermal imaging data of a face collection area, and setting the data as working thermal imaging data;

[0104] Step S103: using a photographic device to capture a frontal photo of the employee, and setting the photo as a basic facial image;

[0105] Step S104: using infrared sensing equipment to obtain front thermal imaging data of the employee, and setting it as basic thermal imaging data;

[0106] Step S2: Step S1: Collecting basic facial information and work facial information of employees;

[0107] Step S2: dividing and extracting the basic facial image to obtain basic status data, wherein the basic status data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasure;

[0108] Step S2 includes the following sub-steps:

[0109] Step S201: obtaining a basic facial image, dividing the basic facial image into a basic eye area, a basic eyebrow area and a basic lip area, processing the basic eye area, the basic eyebrow area and the basic lip area to obtain a basic image fatigue degree, a basic eyebrow horizontal length, a basic eyebrow vertical length and a basic image pleasure degree;

[0110] Step S201 includes the following sub-steps:

[0111] Step S20101: grayscale the basic eye region, the basic eyebrow region, and the basic lip region, convert the grayscale image into a binary image, set the pixel points whose grayscale value is less than or equal to the first threshold value to 0 (black), and set the pixel points whose grayscale value is greater than the first threshold value to 255 (white); obtain a basic eye binary image, a basic eyebrow binary image, and a basic lip binary image;

[0112] Step S20102: Obtain a basic eye binary image, set a pixel area with a gray value of 255 in the basic eye binary image as a first area, and set a pixel area with a gray value of 0 in the basic eye binary image as a second area; obtain the areas of the first area and the second area, and set the area ratio of the first area to the second area as the basic image fatigue degree;

[0113] Step S20103: obtaining the area contour of the pixel area with a gray value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the area contour, and setting them as the basic eyebrow horizontal length and the basic eyebrow vertical length;

[0114] Step S20104: obtaining the vertical highest point and the vertical lowest point of the pixel area with a gray value of 255 in the basic lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the basic image pleasantness;

[0115] Step S202: Acquire basic thermal imaging data to establish a basic three-dimensional facial model, process the basic three-dimensional facial model, and obtain a basic fatigue distance, a basic eyebrow length, a basic eyebrow height, and a basic infrared pleasantness; Step S202 includes the following sub-steps:

[0116] Step S20201: obtaining the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the value of the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance;

[0117] Step S20202: Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height;

[0118] Step S20203: Obtain the leftmost point of the lip, the rightmost point of the lip and the lowest point of the lip from the basic three-dimensional facial model, set the line connecting the leftmost point of the lip and the lowest point of the lip as the slant line of the corner of the mouth, set the line connecting the leftmost point of the lip and the rightmost point of the lip as the horizontal line of the lip, and set the value of the angle between the slant line of the corner of the mouth and the horizontal line of the lip as the basic infrared pleasantness;

[0119] Step S3: dividing and extracting the working facial image to obtain working status data, wherein the working status data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasure, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasure;

[0120] Step S3 includes the following sub-steps:

[0121] Step S301: obtaining a working facial image, and extracting a working eye area, a working eyebrow area, and a working lip area from the working facial image;

[0122] Step S302: comparing the total area of ​​the working eye area, the working eyebrow area and the working lip area with the total area of ​​the basic eye area, the basic eyebrow area and the basic lip area respectively; when the total area of ​​the working eye area, the working eyebrow area and the working lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic eyebrow area and the basic lip area, marking the employee as leaving the job;

[0123] Step S303: When the total area of ​​the working eye region, the working brow region and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region and the basic lip region, marking the employee as being in working state;

[0124] Step S304: When the employee is in a working state, the working eye area, the working eyebrow area and the working lip area are processed to obtain the working image fatigue degree, the working eyebrow horizontal length, the working eyebrow vertical length and the working image pleasure degree;

[0125] Step S304 includes the following sub-steps:

[0126] Step S30401: grayscale the working eye area, the working eyebrow area, and the working lip area, convert the grayscale image into a binary image, set the pixel points whose grayscale value is less than or equal to the first threshold value to 0 (black), and set the pixel points whose grayscale value is greater than the first threshold value to 255 (white); obtain the working eye binary image, the working eyebrow binary image, and the working lip binary image;

[0127] Step S30402: Obtain a working eye binary image, set a pixel area with a gray value of 255 in the working eye binary image as the third area, and set a pixel area with a gray value of 0 in the working eye binary image as the fourth area; obtain the areas of the third area and the fourth area, and set the area ratio of the third area to the fourth area as the working image fatigue degree;

[0128] Step S30403: obtaining the area contour of the pixel area with a gray value of 255 in the binary image of the working eyebrow, obtaining the horizontal length and vertical length of the area contour, and setting them as the horizontal length and vertical length of the working eyebrow;

[0129] Step S30404: obtaining the vertical highest point and the vertical lowest point of the pixel area with a gray value of 255 in the binary image of the working lip, and setting the height difference between the vertical highest point and the vertical lowest point as the pleasantness of the working image;

[0130] Step S305: Acquire the work thermal imaging data to establish a facial work three-dimensional model, process the facial work three-dimensional model, and obtain the work fatigue distance, work eyebrow length, work eyebrow height and work infrared pleasure; Step S305 includes the following sub-steps:

[0131] Step S30501: obtaining the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial working three-dimensional model, and setting the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the working fatigue distance;

[0132] Step S30502: Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the facial working three-dimensional model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height;

[0133] Step S30503: Obtain the leftmost point of the lip, the rightmost point of the lip and the lowest point of the lip from the facial working three-dimensional model, set the line connecting the leftmost point of the lip and the lowest point of the lip as the working mouth corner oblique line, set the line connecting the leftmost point of the lip and the rightmost point of the lip as the working lip horizontal line, and set the value of the angle between the working mouth corner oblique line and the working lip horizontal line as the working infrared pleasantness.

[0134] Step S4: construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network; Step S3 includes the following sub-steps:

[0135] Step S401: establishing a convolutional neural network;

[0136] Step S402: The input layer of the convolutional neural network is used to input the basic facial information and work facial information of the employee;

[0137] Step S403: The convolution layer convolves the basic facial information and the working facial information of the employee to obtain a plurality of feature contours and feature points;

[0138] Step S404: The fully connected layer of the convolutional neural network calculates basic state data and working state data according to multiple feature contours and feature points, the basic state data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working state data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness;

[0139] Substitute the basic state data into the basic state formula The basic state value is calculated; among them, T0 is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, b is the fatigue weight value, and c is the pleasure weight value:

[0140] Substitute the work status data into the work status formula , the working state value is calculated; where T is the working state value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, Y′ is the working infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, b is the fatigue weight value, and c is the pleasure weight value;

[0141] The value of the working state value minus the basic state value is set as the emotion value.

[0142] Step S5: Evaluate the employee's work facial information based on the convolutional neural network to obtain the employee's real-time status;

[0143] Step S5 includes the following sub-steps:

[0144] Step S501: Obtain the employee's working facial information, substitute the employee's working facial information into the convolutional neural network, and obtain an emotion value. When the employee's emotion value is 0 or a positive number, the employee is judged to be in a normal state; when the employee's emotion value is a negative number, the employee is judged to be in a fatigue state;

[0145] Step S502: Count the historical statuses of the first number of employees, set the ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as the fatigue state proportion, obtain the median of the fatigue state proportions of the first number of employees, and set it as the fatigue state proportion threshold;

[0146] Step S503: When the fatigue state ratio of any employee is greater than or equal to the fatigue state ratio threshold, the employee's mood is modified;

[0147] The emotion correction includes obtaining the historical work status data of the employee, calculating the average value of the historical work status data of the employee, and correcting the basic status data of the employee to the average value of the historical work status data of the employee.

[0148] It should be understood by those skilled in the art that the embodiments of the present invention can be provided as methods, systems or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] The above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An employee status monitoring system based on deep learning of facial multimodal information, characterized in that: It includes face acquisition module, face processing module, deep learning module and emotion assessment module; The facial collection module is used to collect basic facial information and work facial information of employees; The facial processing module includes a basic processing unit and a working processing unit, wherein the basic processing unit is used to divide and extract the basic facial image to obtain basic state data; The work processing unit is used to divide and extract the work facial image to obtain work status data; The deep learning module is used to construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network; The emotion assessment module assesses the employee's work facial information based on a convolutional neural network to obtain the employee's real-time status.

2. The employee status monitoring system based on deep learning of facial multimodal information according to claim 1 is characterized in that: The facial acquisition module includes a working facial information acquisition unit and a basic facial information acquisition unit, wherein the working facial information acquisition unit is used to acquire working facial information, wherein the working facial information includes a working facial image and working thermal imaging data; Collecting working facial information includes: using an image acquisition device to collect a working area scene image, locating a face acquisition area from the working area scene image, and capturing an image of an employee's face in the face acquisition area to obtain a working facial image; dividing the working area scene image into a number of workstation areas and other areas, analyzing each workstation area separately, obtaining the computer screen orientation of employees in each workstation area, and setting an area in the first working area in front of the computer screen orientation as a face acquisition area; using an infrared sensor device to obtain thermal imaging data of the face acquisition area, and setting it as working thermal imaging data; The basic facial information acquisition unit is used to acquire basic facial information, wherein the basic facial information includes a basic facial image and basic thermal imaging data; The basic facial information acquisition unit is configured with a basic image acquisition strategy, which includes using a photographic device to capture a frontal photo of the employee and setting the photo as a basic facial image; using an infrared sensor device to obtain frontal thermal imaging data of the employee and setting it as basic thermal imaging data.

3. The employee status monitoring system based on deep learning of facial multimodal information according to claim 2 is characterized in that: The basic processing unit is configured with a basic image processing strategy, the basic image processing strategy comprising acquiring a basic facial image, dividing the basic facial image into a basic eye region, a basic eyebrow region and a basic lip region; Grayscale processing is performed on the basic eye area, the basic eyebrow area, and the basic lip area, and the grayscale image is converted into a binary image, and the pixel points whose grayscale values ​​are less than or equal to the first threshold are set to 0, and the pixel points whose grayscale values ​​are greater than the first threshold are set to 255; Obtaining a basic eye binary image, a basic eyebrow binary image, and a basic lip binary image; Obtain a basic eye binary image, set a pixel area with a gray value of 255 in the basic eye binary image as a first area, and set a pixel area with a gray value of 0 in the basic eye binary image as a second area; obtain the areas of the first area and the second area, and set the area ratio of the first area to the second area as the basic image fatigue degree; Obtaining the area contour of the pixel area with a gray value of 255 in the basic eyebrow binary image, obtaining the horizontal length and vertical length of the area contour, and setting them as the basic eyebrow horizontal length and the basic eyebrow vertical length; The vertical highest point and the vertical lowest point of the pixel area with a gray value of 255 in the basic lip binary image are obtained, and the height difference between the vertical highest point and the vertical lowest point is set as the basic image pleasantness.

4. The employee status monitoring system based on deep learning of facial multimodal information according to claim 3 is characterized in that: The basic processing unit is also configured with a basic infrared processing strategy, which includes acquiring basic thermal imaging data to establish a basic three-dimensional facial model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the basic three-dimensional facial model, and setting the value of the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the basic fatigue distance; Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the basic three-dimensional facial model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the basic eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the basic eyebrow height; The leftmost point, rightmost point and lowest point of the lip are obtained from the basic three-dimensional facial model, the line connecting the leftmost point of the lip and the lowest point of the lip is set as the slant line of the corner of the mouth, the line connecting the leftmost point of the lip and the rightmost point of the lip is set as the horizontal line of the lip, and the value of the angle between the slant line of the corner of the mouth and the horizontal line of the lip is set as the basic infrared pleasantness.

5. The employee status monitoring system based on deep learning of facial multimodal information according to claim 4 is characterized in that: The work processing unit is configured with a work image processing strategy, which includes acquiring a work facial image, extracting a work eye area, a work brow area, and a work lip area from the work facial image; comparing the total area of ​​the work eye area, the work brow area, and the work lip area with the total area of ​​the basic eye area, the basic brow area, and the basic lip area, respectively, and marking the employee as leaving the job when the total area of ​​the work eye area, the work brow area, and the work lip area is less than or equal to a first ratio of the total area of ​​the basic eye area, the basic brow area, and the basic lip area; When the total area of ​​the working eye region, the working brow region, and the working lip region is greater than a first ratio of the total area of ​​the basic eye region, the basic brow region, and the basic lip region, marking the employee as being in a working state; When the employee is in a working state, the working eye area, the working eyebrow area and the working lip area are grayed, the grayscale image is converted into a binary image, the pixel points with a grayscale value less than or equal to the first threshold are set to 0, and the pixel points with a grayscale value greater than the first threshold are set to 255; a working eye binary image, a working eyebrow binary image and a working lip binary image are obtained.

6. The employee status monitoring system based on deep learning of facial multimodal information according to claim 5 is characterized in that: The working processing unit is also configured with a working eye area analysis strategy, which includes obtaining a working eye binary image, setting a pixel area with a gray value of 255 in the working eye binary image as a third area, and setting a pixel area with a gray value of 0 in the working eye binary image as a fourth area; obtaining the areas of the third area and the fourth area, and setting the area ratio of the third area to the fourth area as the working image fatigue degree; The work processing unit is also configured with a work eyebrow region analysis strategy, which includes obtaining a region contour of a pixel region with a gray value of 255 in a work eyebrow binary image, obtaining a horizontal length and a vertical length of the region contour, and setting them as a work eyebrow horizontal length and a work eyebrow vertical length; The working processing unit is also configured with a working lip area analysis strategy, which includes obtaining the vertical highest point and the vertical lowest point of the pixel area with a grayscale value of 255 in the working lip binary image, and setting the height difference between the vertical highest point and the vertical lowest point as the working image pleasantness.

7. The employee status monitoring system based on deep learning of facial multimodal information according to claim 6 is characterized in that: The work processing unit is also configured with a work infrared processing strategy, which includes acquiring work thermal imaging data to establish a facial work three-dimensional model, acquiring the highest point of the upper eyelid and the lowest point of the lower eyelid in the facial work three-dimensional model, and setting the value of the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid as the work fatigue distance; Obtain the leftmost point, rightmost point, highest point and lowest point of the eyebrow from the facial working three-dimensional model, set the horizontal distance between the leftmost point and the rightmost point of the eyebrow as the working eyebrow length, and set the vertical distance between the highest point and the lowest point of the eyebrow as the working eyebrow height; The leftmost point, rightmost point and lowest point of the lip are obtained from the working three-dimensional facial model, the line connecting the leftmost point of the lip and the lowest point of the lip is set as the working mouth corner oblique line, the line connecting the leftmost point of the lip and the rightmost point of the lip is set as the working lip horizontal line, and the value of the angle between the working mouth corner oblique line and the working lip horizontal line is set as the working infrared pleasantness.

8. The employee status monitoring system based on deep learning of facial multimodal information according to claim 7 is characterized in that: The deep learning module is configured with a deep learning strategy, and the deep learning strategy includes establishing a convolutional neural network, and the input layer of the convolutional neural network is used to input basic facial information and work facial information of the employee; The convolution layer of the convolutional neural network is used to convolve the basic facial information and the working facial information of the employee to obtain multiple feature contours and multiple feature points; The fully connected layer of the convolutional neural network is used to calculate basic status data and working status data according to the basic feature contour, the working feature contour, the basic feature points and the working feature points, the basic status data including basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasantness, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasantness, and the working status data including working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasantness, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasantness; Substitute the basic state data into the basic state formula , the basic state value is calculated; wherein T0 is the basic state value, P0 is the basic image fatigue, Dh0 is the basic eyebrow horizontal length, Dv0 is the basic eyebrow vertical length, Y0 is the basic image pleasure, L0 is the basic fatigue distance, Dh0′ is the basic eyebrow length, Dv0′ is the basic eyebrow height, Y0′ is the basic infrared pleasure, a1 is the image weight value, a2 is the infrared weight value, a1+a2=1, and a1 and a2 are both greater than zero, b is the fatigue weight value, and c is the pleasure weight value; Substitute the working status data into the working status formula , the working state value is calculated; where T is the working state value, P is the working image fatigue, Dh is the horizontal length of the working eyebrow, Dv is the vertical length of the working eyebrow, Y is the working image pleasure, L is the working fatigue distance, Dh′ is the working eyebrow length, Dv′ is the working eyebrow height, and Y′ is the working infrared pleasure; The value of the working state value minus the basic state value is set as the emotion value.

9. The employee status monitoring system based on deep learning of facial multimodal information according to claim 8 is characterized in that: The emotion assessment module includes an emotion assessment unit, and the emotion assessment unit is configured with an emotion assessment strategy, and the emotion assessment strategy includes obtaining the working facial information of the employee in real time, substituting the working facial information of the employee into a convolutional neural network, and obtaining an emotion value, and when the emotion value of the employee is 0 or a positive number, the employee is judged to be in a normal state, and when the emotion value of the employee is a negative number, the employee is judged to be in a fatigue state; The emotion assessment module further includes an emotion correction unit, wherein the emotion correction unit is configured with an emotion correction strategy, wherein the emotion correction strategy includes counting the historical status of a first number of employees, setting the ratio of the duration of the employees being in a fatigue state to the duration of the employees being in a working state as a fatigue state proportion, obtaining the median of the fatigue state proportion of the first number of employees, and setting the median as a fatigue state proportion threshold; When the fatigue state percentage of any employee is greater than or equal to the fatigue state percentage threshold, the employee's mood will be modified; The emotion correction includes obtaining the historical work status data of the employee, calculating the average value of the historical work status data of the employee, and correcting the basic status data of the employee to the average value of the historical work status data of the employee.

10. An employee status monitoring method based on deep learning of facial multimodal information, applicable to an employee status monitoring system based on deep learning of facial multimodal information according to any one of claims 1 to 9, characterized in that: The method comprises the following steps: Step S1: Collecting basic facial information and work facial information of employees; Step S2: dividing and extracting the basic facial image to obtain basic status data, wherein the basic status data includes basic image fatigue, basic eyebrow horizontal length, basic eyebrow vertical length, basic image pleasure, basic fatigue distance, basic eyebrow length, basic eyebrow height and basic infrared pleasure; Step S3: dividing and extracting the working facial image to obtain working status data, wherein the working status data includes working image fatigue, working eyebrow horizontal length, working eyebrow vertical length, working image pleasure, working fatigue distance, working eyebrow length, working eyebrow height and working infrared pleasure; Step S4: construct a convolutional neural network, and use the basic state data and the working state data to train the constructed convolutional neural network; Step S5: Evaluate the employee's work facial information based on the convolutional neural network to obtain the employee's real-time status.

Images