Method and apparatus for determining working state, electronic device, and storage medium

Abstract

The application provides a working state determination method, device, electronic equipment and storage medium. The method comprises: collecting video information of a target object; extracting a target frame sequence in the video information, determining a face feature image sequence and a limb feature image sequence based on the target frame sequence; determining action labels corresponding to the limb feature image sequence and first image sequence labels corresponding to each action label, and determining expression labels corresponding to the face feature image sequence and second image sequence labels corresponding to each expression label; determining target expression labels corresponding to each action label from the expression labels corresponding to each action label based on each first image sequence label, each second image sequence label; and determining the working state of the target object based on each action label, the first image sequence label corresponding to each action label and the target expression label corresponding to each action label.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a working state determination method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the aspect of manufacturing industry production management in an industrial scene, the traditional way of monitoring the work efficiency and working state of employees is mostly a method of supervising by video monitoring through manual work. This traditional way not only causes waste of personnel and heavy work burden of the supervising personnel, but also causes supervising failure and inaccurate working state judgment because the personnel are prone to fatigue by staring at the monitoring video for a long time. SUMMARY

[0003] In view of the problems in the related art, the present application provides a working state determination method and device, electronic equipment and storage medium. The video information of a target object is acquired. The target frame sequence in the video information is extracted, and the face feature image sequence and the body feature image sequence are determined based on the target frame sequence. The action label corresponding to the body feature image sequence and the first image sequence label corresponding to each action label are determined, and the expression label corresponding to the face feature image sequence and the second image sequence label corresponding to each expression label are determined. The target expression label corresponding to each action label is determined from the expression label corresponding to each action label based on each first image sequence label and each second image sequence label. The working state of the target object is determined based on the first image sequence label corresponding to each action label and the target expression label corresponding to each action label. In this way, automatic supervision can be realized, and the working state judgment is more accurate, thereby effectively reducing the work burden of the supervising personnel.

[0004] The present application provides a working state determination method, which comprises:

[0005] Acquiring the video information of a target object;

[0006] Extracting the target frame sequence in the video information, and determining the face feature image sequence and the body feature image sequence based on the target frame sequence;

[0007] Determining the action label corresponding to the body feature image sequence and the first image sequence label corresponding to each action label, and determining the expression label corresponding to the face feature image sequence and the second image sequence label corresponding to each expression label;

[0008] Determining the target expression label corresponding to each action label from the expression label corresponding to each action label based on each first image sequence label and each second image sequence label;

[0009] determine the working state of the target object based on the action label, the first image sequence label corresponding to the action label, and the target expression label corresponding to the action label.

[0010] In some embodiments, the determining the working state of the target object based on the action label, the first image sequence label corresponding to the action label, and the target expression label corresponding to the action label comprises:

[0011] determining a matrix value of an initial matrix based on the action label and the target expression label corresponding to the action label, to obtain a target matrix, wherein a column of the initial matrix is an expression label, a row of the initial matrix is an action label, and a corresponding position in the initial matrix is represented by a matrix value 1 when the action label has a corresponding expression label;

[0012] determining a coordinate point of each emotional element state in a three-dimensional model based on the target matrix and the number of first image sequence labels corresponding to each action label, wherein the Z-axis in the three-dimensional model represents the number of first image sequence labels, the X-axis represents an expression label, and the Y-axis represents an action label;

[0013] determining an emotional element state vector corresponding to each action label based on the coordinate point of each emotional element state corresponding to each action label;

[0014] determining a fusion emotional element state vector corresponding to each action label based on the emotional element state vector corresponding to each action label;

[0015] determining the working state of the target object based on the fusion emotional element state vector corresponding to each action label.

[0016] In some embodiments, the determining the emotional element state vector corresponding to each action label based on the coordinate point of each emotional element state corresponding to each action label comprises:

[0017] subtracting a 0 vector from the coordinate point of each emotional element state corresponding to each action label to obtain the emotional element state vector corresponding to each action label;

[0018] the determining the fusion emotional element state vector corresponding to each action label based on the emotional element state vector corresponding to each action label comprises:

[0019] summing the emotional element state vector corresponding to each action label to obtain the fusion emotional element state vector corresponding to each action label;

[0020] the determining the working state of the target object based on the fusion emotional element state vector corresponding to each action label comprises:

[0021] The working state of the target object is calculated by using a clustering algorithm based on the fusion emotional element state vectors corresponding to each action label.

[0022] In some embodiments, the target expression label corresponding to each action label is determined from the expression labels corresponding to each action label based on each first image sequence label and each second image sequence label.

[0023] The first number of first image sequence labels corresponding to each action label is determined, and the second number of second image sequence labels corresponding to each expression label is determined.

[0024] The ratio value between the second number and the first number within the time range corresponding to each action label is determined.

[0025] The target expression label corresponding to each action label is determined from the expression labels corresponding to each action label based on each ratio value.

[0026] In some embodiments, the target frame sequence in the video information is extracted, and the face feature image sequence and the body feature image sequence are determined based on the target frame sequence, including:

[0027] The video information is split to obtain a video frame sequence of the video information.

[0028] The target frame sequence is extracted from the video frame sequence based on a preset frame extraction rate.

[0029] The target frame sequence is input into a third network model to determine a face feature image sequence, and the target frame is input into a fourth network model to determine a body feature image sequence.

[0030] In some embodiments, the action label corresponding to the body feature image sequence and the first image sequence label corresponding to each action label are determined, and the expression label corresponding to the face feature image sequence and the second image sequence label corresponding to each expression label are determined, including:

[0031] The body feature image sequence is input into a pre-trained first network model to determine each action label and the first image sequence label corresponding to each action label.

[0032] The face feature image sequence is input into a pre-trained second network model to determine each expression label and the second image sequence label corresponding to each expression label.

[0033] In some embodiments, the method further comprises:

[0034] Based on the working state, a management suggestion corresponding to the working state is determined.

[0035] output the management suggestion to a manager, so that the manager manages based on the management suggestion.

[0036] The embodiment of the application provides a working state determination device, comprising:

[0037] A collection module is configured to collect video information of a target object.

[0038] An extraction module is configured to extract a target frame sequence from the video information, and determine a face feature image sequence and a limb feature image sequence based on the target frame sequence.

[0039] A first determination module is configured to determine action labels corresponding to the limb feature image sequence and first image sequence labels corresponding to each action label, and determine expression labels corresponding to the face feature image sequence and second image sequence labels corresponding to each expression label.

[0040] A second determination module is configured to determine target expression labels corresponding to each action label from expression labels corresponding to each action label based on each first image sequence label and each second image sequence label.

[0041] A third determination module is configured to determine a working state of the target object based on each action label and the target expression labels corresponding to each action label.

[0042] The embodiment of the application provides an electronic device, comprising a memory and a processor, and the memory stores a computer program, which is executed by the processor to execute any one of the working state determination methods.

[0043] The embodiment of the application provides a storage medium, which stores a computer program capable of being executed by one or more processors and capable of being used to implement any one of the working state determination methods.

[0044] This application provides a method, apparatus, electronic device, and storage medium for determining the working status of a target object. The method involves: acquiring video information of a target object; extracting target frame sequences from the video information; determining facial feature image sequences and limb feature image sequences based on the target frame sequences; determining action tags corresponding to the limb feature image sequences and first image sequence numbers corresponding to each action tag; determining expression tags corresponding to the facial feature image sequences and second image sequence numbers corresponding to each expression tag; determining target expression tags corresponding to each action tag from the expression tags corresponding to each action tag based on each first image sequence number and each second image sequence number; and determining the working status of the target object based on the first image sequence numbers and target expression tags corresponding to each action tag. This enables automated monitoring and more accurate judgment of the working status, thereby effectively reducing the workload of monitoring personnel. Attached Figure Description

[0045] The present application will be described in more detail below based on embodiments and with reference to the accompanying drawings.

[0046] Figure 1 A schematic diagram illustrating the implementation flow of a method for determining a working state provided in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram illustrating the implementation process of determining the working state of a target object based on each action tag, the first image sequence number corresponding to each action tag, and the target expression tag corresponding to each action tag, as provided in an embodiment of this application.

[0048] Figure 3 A schematic diagram of the structure of a working state determination system provided in an embodiment of this application;

[0049] Figure 4 A schematic diagram of the structure of a device for determining a working state provided in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application.

[0051] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets as each other and as other subsets of all possible embodiments, and can be combined with each other and with other subsets of all possible embodiments without conflict.

[0054] If there are similar descriptions of "first\second\third" in the application file, the following explanations are added. In the following description, the terms "first\second\third" referred to are only to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

[0056] Based on the problems in the related art, the embodiments of the present application provide a working state determination method. The method is applied to an electronic device, which can be a mobile terminal, a computer, etc. The working state determination method provided by the embodiments of the present application can realize the functions by calling program codes by a processor of the electronic device, wherein the program codes can be saved in a computer storage medium.

[0057] The embodiments of the present application provide a working state determination method, Figure 1 As shown in the implementation flowchart of the working state determination method provided by the embodiments of the present application, Figure 1 As shown in the implementation flowchart of the working state determination method provided by the embodiments of the present application,

[0058] Step S1, video information of a target object is collected.

[0059] In the embodiments of the present application, the electronic device can include a collection module, and the video information of the target object is collected through the collection module. The collection module can be a camera, and the target object can be a worker working in a manufacturing enterprise. In some embodiments, the electronic device and the collection module can be two independent devices, and the electronic device is in communication connection with the collection module.

[0060] Step S2, a target frame sequence in the video information is extracted, and the target frame sequence is input into a neural network model to determine a face feature image sequence and a limb feature image sequence.

[0061] In the embodiments of the present application, the video information includes a plurality of image frames. After the electronic device obtains the video information, the video information is split to obtain a video frame sequence of the video information. Based on a preset frame extraction rate, a target frame sequence is extracted from the video frame sequence. The target frame sequence is a plurality of image frames sorted by time. When frame extraction is performed, the distance between two extracted image frames is equal, and the balance of frame extraction is ensured as much as possible, and the video information can be covered to the maximum extent. Since most normal video files have 25 image frames per second, feature extraction and training can be performed without using too many image frames during identification. Therefore, frame extraction operation is required here. The frame extraction rate can be evaluated according to the action duration. The frame extraction rate can be selected as appropriate according to the computing performance of the device. The greater the frame extraction rate, the more the number of feature sequence frames, and the more accurate the prediction result. However, in terms of computing performance, the longer the time consumption is. For example, for a manufacturing enterprise production line action with an average duration of 5 seconds, there are about 5*25=125 frames for an average action. If the 12th frame and the 25th frame are selected from 1 second (25 frames), a total of two frames are extracted. After frame extraction, a total of about 5s*2 frames / s=10 frames of production line action frames are extracted, and the frame rate is 10 / 125=0.08. The video information includes 750 frames, and the sequence number of the plurality of image frames in the video information is [1, 2, 3, 4, …… 749, 750]. The extracted target frame has 60 frames, and the sequence number of the target sequence can be represented as: [1, 2, 3, 4, 5, …… 59, 60].

[0062] In the embodiments of the present application, after the target frame sequence is determined, the target frame sequence is input into the third network model to determine the face feature image sequence, and the target frame sequence is input into the fourth network model to determine the body feature image sequence. In the embodiments of the present application, the third neural network model and the fourth neural network model can be a convolutional neural network model. The number of the face feature image sequence is less than the number of the body feature image sequence. For example, the number of the face feature image sequence is 5, and the number of the body feature image sequence is 25.

[0063] In step S3, the action label corresponding to the body feature image sequence and the first image sequence number corresponding to each action label are determined, and the expression label corresponding to the face feature image sequence and the second image sequence number corresponding to each expression label are determined.

[0064] In the embodiments of the present application, the body feature image sequence can be input into the first network model trained in advance to determine each action label and the first image sequence number corresponding to each action label. The face feature image sequence can be input into the second network model trained in advance to determine each expression label and the second image sequence number corresponding to each expression label.

[0065] In the embodiments of the present application, the first network model and the second network model are trained by using a convolutional neural network combined with a long short-term memory network, and the training samples of the first network model and the second network model are different.

[0066] In the embodiments of the present application, each action label and the first image sequence label corresponding to each action label can be represented as [action label][first image sequence label], and the first image sequence label corresponding to each action label can include multiple, for example, [turn around][6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. In the embodiments of the present application, the action label can represent the category of action, and the action category can include: valid process (some conventional production line process operations belong to the valid process label), turn around, bend down, both hands free, play mobile phone, walk, and the like. In the embodiments of the present application, the number of first image sequence labels represents how many frames the action category corresponding to the action label lasts.

[0067] In the embodiments of the present application, each expression label and the second image sequence label corresponding to each expression label can be represented as [expression label][second image sequence label], and the second image sequence label corresponding to each expression label can include multiple, for example, [smile][4, 5, 6, 7, 8, 9]. The expression label is used to represent the category of expression, and the category of expression can include: normal, sad, angry, smile, happy, and the like.

[0068] In step S4, the target expression label corresponding to each action label is determined from the expression label corresponding to each action label based on each first image sequence label and each second image sequence label.

[0069] In the embodiments of the present application, when the electronic device obtains each first image sequence label and each second image sequence label, the target expression label corresponding to each action label can be determined from the expression label corresponding to each action label. In the embodiments of the present application, the number of target expression labels corresponding to each action label can be equal to the number of expression labels corresponding to each action label, and the number of target expression labels corresponding to each action label can be less than the number of expression labels corresponding to each action label.

[0070] In the embodiments of the present application, the number of target expression labels is not limited, and can be multiple, 1, or 0.

[0071] In the embodiments of the present application, the step of determining the target expression label corresponding to each action label from the expression labels corresponding to each action label based on the respective first image sequence labels and the respective second image sequence labels can be implemented in the following manner: determining the first number of the first image sequence labels corresponding to each action label, and determining the second number of the second image sequence labels corresponding to each expression label; determining the ratio value between the second number and the first number within the time range corresponding to each action label; and determining the target expression label corresponding to each action label from the expression labels corresponding to each action label based on the respective ratio values. In the embodiments of the present application, the expression label with the largest ratio value is selected as the target expression label corresponding to each action label.

[0072] Within the time range corresponding to one action label, there are multiple expression labels, and the second numbers of the multiple expression labels are equal. In the case that the ratio value between the second number and the first number of the image sequence label corresponding to the action label is the largest, the multiple expression labels are all target expression labels. For example, taking walking as an example of one action label, the expression label during walking is [normal], the expression label during the later stage of walking is [smile], and the turning around while smiling, then within the action sequence of [walking], there are two expression labels of [normal] and [smile], and the numbers of the two expression labels of [normal] and [smile] are equal and the largest, so the two expression labels of [normal] and [smile] are both target expression labels.

[0073] In step S5, the working state of the target object is determined based on the respective action labels, the first image sequence labels corresponding to each action label, and the target expression labels corresponding to each action label.

[0074] In the embodiments of the present application, each emotional element state coordinate point in the three-dimensional model can be converted based on the respective action labels, the first image sequence labels corresponding to each action label, and the target expression labels corresponding to each action label, so as to determine the respective emotional element state vectors based on the respective emotional element coordinate points, and further determine the fusion emotional element state vectors corresponding to each action label, so as to calculate the working state of the target object based on the respective fusion emotional element state vectors.

[0075] In the embodiments of the present application, the working state can include positive, normal, negative, and working redundancy.

[0076] The application provides a working state determination method. The method comprises the following steps: obtaining video information of a target object; extracting a target frame sequence in the video information, determining a face feature image sequence and a limb feature image sequence based on the target frame sequence; determining action labels corresponding to the limb feature image sequence and first image sequence labels corresponding to each action label, and determining expression labels corresponding to the face feature image sequence and second image sequence labels corresponding to each expression label; determining target expression labels corresponding to each action label from the expression labels corresponding to each action label based on each first image sequence label, each second image sequence label; and determining the working state of the target object based on each action label, the first image sequence label corresponding to each action label and the target expression label corresponding to each action label. In this way, automatic supervision can be realized, the working state determination is more accurate, and the working burden of supervisors is effectively reduced.

[0077] In some embodiments, the step S5 of determining the working state of the target object based on each action label, the first image sequence label corresponding to each action label and the target expression label corresponding to each action label can be implemented in the following way, Figure 2 An implementation flow diagram for determining the working state of the target object based on each action label, the first image sequence label corresponding to each action label and the target expression label corresponding to each action label is provided in the embodiments of the application, as shown in Figure 2 The implementation flow diagram comprises the following steps:

[0078] In step S51, the matrix value of an initial matrix is determined based on each action label and the target expression label corresponding to each action label to obtain a target matrix, wherein the vertical column of the initial matrix is an expression label, the horizontal column of the initial matrix is an action label, and when each action label has a corresponding expression label, the corresponding position in the initial matrix is represented by a matrix value 1.

[0079] For example, there are six action labels, including effective process, turning around, bending, both hands idle, playing a mobile phone and walking, and there are five expression labels, including normal, sad, angry, smiling and happy. An initial matrix can be established in advance, wherein the vertical column of the initial matrix is an expression label, and the horizontal column of the initial matrix is an action label. The initial matrix is a 5*6 matrix, that is, the initial matrix is an expression label quantity*action label quantity matrix. In the embodiments of the application, when an action label is an effective process and the corresponding expression labels include a normal expression and an angry expression, the corresponding position in the initial matrix can be filled with a matrix value 1. Table 1 is a schematic table of a target matrix provided in the embodiments of the application, as shown in Table 1,

[0080] Table 1 is a schematic table of a target matrix provided in the embodiments of the application

[0081]

[0082] In step S52, the emotion element state coordinate points in the three-dimensional model are determined based on the target matrix and the number of first image sequence labels corresponding to each action label, wherein the Z axis in the three-dimensional model represents the number of first image sequence labels, the X axis represents the expression label, and the Y axis represents the action label.

[0083] In the embodiment of the present application, after the target matrix is determined, the target matrix is a two-dimensional matrix, the time dimension can be determined based on the first image sequence labels corresponding to each action label, and then a three-dimensional model is established through the time dimension and the two dimensions of the matrix. In the three-dimensional model, the Z axis represents the number of first image sequence labels, the X axis represents the expression label, and the Y axis represents the action label.

[0084] In the embodiment of the present application, the Z axis is the number of first image sequence labels corresponding to the action label, and different numbers of first image sequence labels corresponding to the action label represent different durations of the action label.

[0085] In the embodiment of the present application, in the three-dimensional model, the XOY plane horizontal and vertical coordinates are constituted by the expression label and the action label. Taking the above example, the x axis represents five expressions, and the x value ranges from 1 to 5, for example, normal 1, sad 2, angry 3, smile 4, and happy 5. The y axis represents six action labels, and the y value ranges from 1 to 6, for example, effective process 1, turning around 2, bending 3, both hands idle 4, playing a mobile phone 5, and walking 6. The values of X and Y are integers, one value of X corresponds to one expression label, and one value of Y corresponds to one action label.

[0086] In the embodiment of the present application, the matrix has corresponding coordinate points in the XOY plane of the three-dimensional model, and the number of time sequences on the Z axis corresponding to each action label is added, thereby constructing the three-dimensional cleaning element state coordinate points.

[0087] In step S53, the emotion element state vector corresponding to each action label is determined based on the emotion element state coordinate points corresponding to each action label.

[0088] In the embodiment of the present application, the emotion element state vector corresponding to each action label is obtained by subtracting the 0 vector from the emotion element state coordinate points corresponding to each action label.

[0089] In step S54, the fusion emotion element state vector corresponding to each action label is determined based on the emotion element state vector corresponding to each action label.

[0090] In the embodiments of the present application, the fusion emotional element state vector corresponding to each action label can be obtained by summing each emotional element state vector corresponding to each action label.

[0091] In step S55, the working state of the target object is determined based on the fusion emotional element state vector corresponding to each action label.

[0092] In the embodiments of the present application, the working state of the target object is calculated based on the fusion emotional element state vector corresponding to each action label.

[0093] The method for determining the working state provided in the embodiments of the present application mathematically models the three dimensions of expression, action and action duration, respectively maps them to the x, y and z axes of a three-dimensional rectangular coordinate system, marks different emotional attributes by the combination of spatial vectors and the spatial coordinate relationship, and comprehensively constructs a three-dimensional spatial vector representing the working state of the employee, i.e., an emotional element state vector. Then, the working state is classified by using the clustering analysis method, so that the determined working state is more accurate.

[0094] In some embodiments, after step S5, the method further includes:

[0095] In step S6, the management suggestion corresponding to the working state is determined based on the working state.

[0096] In the embodiments of the present application, the corresponding relationship between the working state and the management establishment can be established in advance. When the working state is determined, the management suggestion can be determined based on the corresponding relationship. The management establishment can be psychological counseling or process improvement suggestion.

[0097] For example, the working state of an employee performing a certain process action is "working redundancy", and the suggestion "xx process action can be improved, please click the video to play for details" is proposed. The manager can click the button to jump out the process video of the corresponding action sequence for the manager to view.

[0098] In step S7, the management suggestion is output to the manager, so that the manager manages based on the management suggestion.

[0099] In the embodiments of the present application, the management suggestion is output to the manager, which can effectively reduce the work burden of the supervisor, increase efficiency by reducing staff, and improve production management efficiency.

[0100] In some embodiments, the employees can also be classified, and an employee portrait based on the working state is constructed. When a certain amount of employee data is accumulated, the employees are given targeted psychological counseling and suggestions according to the employee portrait, so as to improve the happiness of the employees and thus improve the productivity of the enterprise and promote the development of the enterprise.

[0101] Based on the foregoing various embodiments, the embodiments of the present application further provide a working state determination method, which is applied to a working state determination system, Figure 3 A structural schematic diagram of a working state determination system provided by the embodiments of the present application is shown in Figure 3 The system comprises a feature extraction module, an action recognition module, an expression recognition module, a working state judgment module, and a management suggestion output module.

[0102] In the embodiments of the present application, the input of the feature extraction module is a video file, and the output is a feature sequence. The output contains two kinds of feature image sequences, one is a face feature image sequence, and the other is a body feature image sequence. The feature extraction module splits video image frames from the video file and extracts a frame feature sequence (the same as the target frame sequence in the above-mentioned embodiments), and then inputs them to the rear functional modules. Since most normal video files have 25 image frames per second, feature extraction and training can be performed without using too many image frames during recognition, so frame extraction operation is required. The frame extraction parameters can be evaluated according to the action duration. Generally, the average duration of a production line action is 5 seconds, so there are about 5*25=125 frames in an average action. Selecting 1s (25 frames) to extract the 12th frame and the 25th frame, a total of two frames, after frame extraction, a total of about 5s*2 frames / s=10 frames of production line action frames are obtained, and the frame rate is 10 / 125=0.08. After frame extraction, the frame sequence is input into two trained convolutional neural networks (the two trained convolutional neural network models are the third network model and the fourth network model in the above-mentioned embodiments), one neural network model outputs five face feature key point images (the same as the face feature image sequence in the above-mentioned embodiments), and the other neural network model outputs 25 body feature key point image sequences (the same as the body feature image sequence in the above-mentioned embodiments) which are input into different functional modules.

[0103] The input of the action recognition module is an image sequence containing 25 body feature key points, and the output is a pair of action labels and corresponding image sequence labels. This module contains a trained network model (the training data is the body feature image sequence and the corresponding action label), and the network structure is a common convolutional neural network (CNN) + long short-term memory network (LSTM). The action categories include: valid process (some conventional production line process operations belong to the valid process label), turning around, bending, both hands idle, playing mobile phone, and walking.

[0104] The expression recognition module input is an image sequence containing 5 facial feature key points, and the output is a pair of expression labels and corresponding image sequence labels. The module contains a trained network model (the training data is a labeled feature image sequence and corresponding expression label), and the network structure is a common convolutional neural network. The expression categories are: normal, sad, angry, smile, and happy.

[0105] The work state judgment module mainly combines the action sequence and the expression sequence to jointly make decision and judge the work state of the production line staff. Since the action usually lasts for a long time, the number of action label sequences is greater than the number of expression label sequences, so the action sequence is used as the work state alignment judgment basis. The proportion of the expression sequence in the action sequence range is calculated, and the expression label with the maximum proportion is selected to represent the work state. The combination of 5 expressions and 6 actions of a certain work state is abstracted into a 5*6 matrix. In the embodiment of the present application, if several expression labels in the range have the same proportion, the corresponding position values of the matrix in the following text are all 1.

[0106] The action sequence is used as the work state alignment judgment basis, and the sequence dimension (i.e. time dimension) is increased to construct a three-dimensional model. The z-axis is the number of a certain action sequence (different sequence numbers represent different durations), and the xoy plane is the expression and action (x-axis represents five expressions, x value range 1-5, y-axis represents six actions, y value range 1-6). The matrix described in the foregoing has corresponding coordinate points in the xoy plane, and the time sequence number is added to the z-axis, thereby constructing a three-dimensional emotional element state coordinate point. The emotional element state vector is calculated by subtracting the zero vector from the point. When there are multiple expression labels in a certain action sequence (there are multiple coordinate points in the xoy plane), the sum of the multiple emotional element state vectors represents a unique work state. The method of clustering (such as K-means clustering) is used to analyze the emotional element state vector, and finally the classification of the work state is calculated. The work state is divided into: positive, normal, negative, and work redundancy.

[0107] The management suggestion output module gives corresponding process improvement suggestions according to the work state output by the work state judgment module, which can effectively reduce the work burden of supervisors, reduce staff and increase efficiency, and improve production management efficiency. For example, if the work state of a certain employee performing a certain process action is "work redundancy", the suggestion "xx process action can be improved, please click the video to play for details" is proposed. Clicking the button will jump out the process video of the corresponding action sequence for the supervisor to view. This module can also classify employees based on work state to construct an employee portrait based on work state. When a certain amount of employee data is accumulated, targeted psychological counseling and suggestions are given to employees according to the employee portrait, which can improve the happiness of employees and thus improve the productivity of enterprises and promote the development of enterprises.

[0108] The embodiment of the application provides a working state determination method, which comprises the following steps: acquiring video information of a target object; extracting a target frame sequence from the video information; determining a face feature image sequence and a limb feature image sequence based on the target frame sequence; determining an action label corresponding to the target frame sequence and a first image sequence label corresponding to each action label; determining an expression label corresponding to the target frame sequence and a second image sequence label corresponding to each expression label; determining a target expression label corresponding to each action label from the expression label corresponding to each action label based on each first image sequence label and each second image sequence label; performing clustering analysis on the target expression label corresponding to each action label to obtain a three-dimensional space vector representing a working state of the target object; and performing intelligent decision-making on the working state, completing automatic decision-making of process action management and optimization, and finally outputting corresponding management suggestions.

[0109] Based on the foregoing embodiment, the embodiment of the application provides a detection device, each module included in the device and each unit included in each module can be realized by a processor in a computer device; of course, it can also be realized by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0110] The embodiment of the application provides a working state determination device, Figure 4 The structure diagram of the working state determination device provided by the embodiment of the application is shown in Figure 4 The working state determination device 400 comprises:

[0111] The acquisition module 401 is configured to acquire video information of a target object;

[0112] The extraction module 402 is configured to extract a target frame sequence from the video information and determine a face feature image sequence and a limb feature image sequence based on the target frame sequence;

[0113] The first determination module 403 is configured to determine an action label corresponding to the target frame sequence and a first image sequence label corresponding to each action label, and determine an expression label corresponding to the target frame sequence and a second image sequence label corresponding to each expression label;

[0114] The second determination module 404 is configured to determine a target expression label corresponding to each action label from the expression label corresponding to each action label based on each first image sequence label and each second image sequence label;

[0115] The third determining module 405 is configured to determine the working state of the target object based on the respective action label, the first image sequence label corresponding to the respective action label, and the target expression label corresponding to the respective action label.

[0116] In some embodiments, the third determining module 405 includes:

[0117] The first determining unit is configured to determine the matrix value of an initial matrix based on the respective action label and the target expression label corresponding to the respective action label, to obtain a target matrix, wherein the columns of the initial matrix are expression labels, the rows of the initial matrix are action labels, and the corresponding positions in the initial matrix are represented by a matrix value 1 when the respective action label has a corresponding expression label.

[0118] The second determining unit is configured to determine the respective emotional element state coordinate point in a three-dimensional model based on the target matrix and the number of first image sequence labels corresponding to the respective action label, wherein the Z axis of the three-dimensional model represents the number of first image sequence labels, the X axis represents the expression label, and the Y axis represents the action label.

[0119] The third determining unit is configured to determine the emotional element state vector corresponding to the respective action label based on the respective emotional element state coordinate point corresponding to the respective action label.

[0120] The fourth determining unit is configured to determine the fusion emotional element state vector corresponding to the respective action label based on the emotional element state vector corresponding to the respective action label.

[0121] The fifth determining unit is configured to determine the working state of the target object based on the fusion emotional element state vector corresponding to the respective action label.

[0122] In some embodiments, the third determining unit includes:

[0123] The first determining sub-unit is configured to subtract a 0 vector from the respective emotional element state coordinate point corresponding to the respective action label, to obtain the emotional element state vector corresponding to the respective action label.

[0124] The fourth determining unit includes:

[0125] The second determining sub-unit is configured to sum the respective emotional element state vector corresponding to the respective action label, to obtain the fusion emotional element state vector corresponding to the respective action label.

[0126] The fifth determining unit includes:

[0127] The third determining sub-unit is configured to determine the working state of the target object based on the fusion emotional element state vector corresponding to the respective action label, by using a clustering algorithm.

[0128] In some embodiments, the second determining module comprises:

[0129] determining a first number of image sequence labels corresponding to each action label, and determining a second number of image sequence labels corresponding to each expression label;

[0130] The sixth determining unit is configured to determine a ratio value between each second number and first number within a time range corresponding to each action label.

[0131] The seventh determining unit is configured to determine a target expression label corresponding to each action label from expression labels corresponding to each action label based on each ratio value.

[0132] In some embodiments, the extracting module comprises:

[0133] The splitting unit is configured to split the video information to obtain a video frame sequence of the video information.

[0134] The extracting unit is configured to extract a target frame sequence from the video frame sequence based on a preset frame extraction rate.

[0135] The input unit is configured to input the target frame sequence into a third network model to determine a face feature image sequence, and input the target frame into a fourth network model to determine a body feature image sequence.

[0136] In some embodiments, the first determining module comprises:

[0137] The eighth determining unit is configured to input the body feature image sequence into a pre-trained first network model to determine each action label and a first image sequence label corresponding to each action label.

[0138] The ninth determining unit is configured to input the face feature image sequence into a pre-trained second network model to determine each expression label and a second image sequence label corresponding to each expression label.

[0139] In some embodiments, the working state determining apparatus 400 further comprises:

[0140] The fourth determining module is configured to determine a management suggestion corresponding to the working state based on the working state.

[0141] The output module is configured to output the management suggestion to a manager, so that the manager manages based on the management suggestion.

[0142] It should be noted that, in the embodiments of the present application, if the determination of the working state is implemented in the form of a software function module and is sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0143] Correspondingly, the embodiments of the present application provide a storage medium having a computer program stored thereon, characterized by comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the steps of the determination of the working state provided in the above embodiments are implemented.

[0144] The embodiments of the present application provide an electronic device; Figure 5 The electronic device provided by the embodiments of the present application has a component structure diagram as shown in Figure 5 The electronic device 500 includes a processor 501, at least one communication bus 502, a user interface 503, at least one external communication interface 504, and a memory 505. The communication bus 502 is configured to realize the connection and communication between the components. The user interface 503 can include a display screen, and the external communication interface 504 can include a standard wired interface and a wireless interface. The processor 501 is configured to execute the program of the determination of the working state stored in the memory, so as to implement the steps of the determination of the working state provided in the above embodiments.

[0145] The descriptions of the above electronic device and storage medium embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the computer device and storage medium embodiments of the present application, please refer to the description of the method embodiments of the present application.

[0146] It should be noted that the descriptions of the above storage medium and electronic device and garbage can embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application.

[0147] It should be understood that every feature, structure, or characteristic described herein is within a preferred embodiment of the present application. Thus, it is meant that the features, structures, or characteristics can be combined with each other in any manner within a preferred embodiment of the present application. In addition, it is contemplated that each feature, structure, or characteristic can be implemented in hardware, software, or a combination thereof.

[0148] It should be noted that, as used herein, the articles "a", "an", "the", and "at least one" are intended to mean that there is one or more of the elements in the preceding descriptions. The articles "a" (or "an"), as well as the first article "the" and "at least one" do not denote a limitation of quantity, and are used with their plain, ordinary meaning. Thus, these articles should be interpreted in the manner it is employed by those having ordinary skill in the art to indicate that the accompanying recited elements can be included either singly or in multiple number, that the referenced member or elements can be selected from the group consisting of one or more member or elements.

[0149] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described above as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units; they can be located in one place or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0151] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or hardware plus software functional unit.

[0152] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the method embodiments when executed.

[0153] Alternatively, the integrated units of the present application can be stored in a computer readable storage medium if they are realized in the form of software function modules and sold or used as independent products. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions for causing a controller to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, magnetic discs or optical discs, and various media that can store program codes.

[0154] The above is only an embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of determining an operating state, characterized by The method comprises: collecting video information of a target object; extracting a target frame sequence from the video information, and determining a face feature image sequence and a limb feature image sequence based on the target frame sequence; determining action labels corresponding to the limb feature image sequence and first image sequence labels corresponding to each action label, and determining expression labels corresponding to the face feature image sequence and second image sequence labels corresponding to each expression label; determining target expression labels corresponding to each action label from the expression labels corresponding to each action label based on each first image sequence label and each second image sequence label; determining a working state of the target object based on each action label, the first image sequence label corresponding to each action label, and the target expression label corresponding to each action label; the determination of the working state of the target object based on each action label, the first image sequence label corresponding to each action label, and the target expression label corresponding to each action label comprises: determining matrix values of an initial matrix based on each action label and each target expression label corresponding to each action label to obtain a target matrix, wherein the columns of the initial matrix are expression labels, the rows of the initial matrix are action labels, and when each action label has a corresponding expression label, the corresponding position in the initial matrix is represented by a matrix value 1; determining each emotional element state coordinate point in a three-dimensional model based on the target matrix and the number of first image sequence labels corresponding to each action label, wherein the Z-axis in the three-dimensional model represents the number of first image sequence labels, the X-axis represents expression labels, and the Y-axis represents action labels; determining an emotional element state vector corresponding to each action label based on each emotional element state coordinate point corresponding to each action label, wherein each emotional element state coordinate point minus a zero vector obtains the emotional element state vector; determining a fused emotional element state vector corresponding to each action label based on the emotional element state vector corresponding to each action label; determining the working state of the target object based on the fused emotional element state vector corresponding to each action label.

2. The method of claim 1, wherein, the determination of the emotional element state vector corresponding to each action label based on each emotional element state coordinate point corresponding to each action label comprises: subtracting a zero vector from each emotional element state coordinate point corresponding to each action label to obtain the emotional element state vector corresponding to each action label; the determination of the fused emotional element state vector corresponding to each action label based on the emotional element state vector corresponding to each action label comprises: summing each emotional element state vector corresponding to each action label to obtain the fused emotional element state vector corresponding to each action label; the determination of the working state of the target object based on the fused emotional element state vector corresponding to each action label comprises: adopting a clustering algorithm to calculate the working state of the target object based on the fused emotional element state vector corresponding to each action label.

3. The method of claim 1, wherein, The determining the target expression label corresponding to each action label from the expression labels corresponding to each action label based on each first image sequence label and each second image sequence label comprises: determining a first number of first image sequence labels corresponding to each action label, and determining a second number of second image sequence labels corresponding to each expression label; determining a ratio value between each second number and the first number within a time range corresponding to each action label; determining the target expression label corresponding to each action label from the expression labels corresponding to each action label based on each ratio value.

4. The method of claim 1, wherein, The extracting the target frame sequence in the video information, and determining a face feature image sequence and a body feature image sequence based on the target frame sequence comprises: splitting the video information to obtain a video frame sequence of the video information; extracting a target frame sequence from the video frame sequence based on a preset frame extraction rate; inputting the target frame sequence into a third network model to determine the face feature image sequence, and inputting the target frame into a fourth network model to determine the body feature image sequence.

5. The method of claim 1, wherein, The determining the action label corresponding to the body feature image sequence and the first image sequence label corresponding to each action label, and the determining the expression label corresponding to the face feature image sequence and the second image sequence label corresponding to each expression label comprises: inputting the body feature image sequence into a pre-trained first network model to determine each action label and the first image sequence label corresponding to each action label; inputting the face feature image sequence into a pre-trained second network model to determine each expression label and the second image sequence label corresponding to each expression label.

6. The method of claim 1, wherein, The method further comprises: determining a management suggestion corresponding to the working state based on the working state; outputting the management suggestion to a manager, so that the manager manages based on the management suggestion.

7. A device for determining a state of operation, characterized in that comprise: a collection module configured to collect video information of a target object; an extraction module configured to extract a target frame sequence in the video information, and determine a face feature image sequence and a body feature image sequence based on the target frame sequence; a first determination module configured to determine an action label corresponding to the target frame sequence and a first image sequence label corresponding to each action label, and determine an expression label corresponding to the target frame sequence and a second image sequence label corresponding to each expression label; a second determination module configured to determine a target expression label corresponding to each action label from expression labels corresponding to each action label based on each first image sequence label and each second image sequence label; a third determination module configured to determine a working state of the target object based on each action label, the first image sequence label corresponding to each action label, and the target expression label corresponding to each action label. The third determination module comprises: The first determining unit is configured to determine a matrix value of an initial matrix based on each action label and a target expression label corresponding to each action label to obtain a target matrix, wherein a column of the initial matrix is an expression label, a row of the initial matrix is an action label, and when each action label has a corresponding expression label, a corresponding position in the initial matrix is represented by a matrix value 1; The second determining unit is configured to determine each emotional element state coordinate point in a three-dimensional model based on the target matrix and a number of first image sequence labels corresponding to each action label, wherein a Z axis in the three-dimensional model represents the number of first image sequence labels, an X axis represents an expression label, and a Y axis represents an action label; The third determining unit is configured to determine an emotional element state vector corresponding to each action label based on each emotional element state coordinate point corresponding to each action label, wherein each emotional element state coordinate point minus a zero vector is the emotional element state vector; The fourth determining unit is configured to determine a fusion emotional element state vector corresponding to each action label based on the emotional element state vector corresponding to each action label; The fifth determining unit is configured to determine a working state of the target object based on the fusion emotional element state vector corresponding to each action label.

8. An electronic device, comprising: The memory and the processor, the memory has a computer program stored thereon, and the computer program is executed by the processor to perform the working state determination method of any one of claims 1 to 6. The computer program stored in the storage medium can be executed by one or more processors, and can be used to implement the working state determination method of any one of claims 1 to 6.

9. A storage medium, characterized by ​

Images