A method for intelligently identifying the work content of barbershop employees

Abstract

The present invention relates to a method for intelligently identifying the work content of barbershop employees, which belongs to the field of machine vision technology and includes the following steps: S1: setting the installation position and installation conditions of the acquisition equipment; S2: establishing a label library of barbershop employee faces and customer faces, and training a face recognition model; S3: establishing an action label library related to objects, tools and people, and training a barbershop action behavior recognition model; S4: using the trained face recognition model and barbershop action behavior recognition model to perform action behavior recognition in actual barbershop service scenarios. Construct an "action pair" behavior sequence of several elements, including customers, employees and actions; S5: establishing a work content recognition label, and constructing a deep neural network model for work content recognition, so as to determine the service work content of barbershop employees to customers. The present invention assists in realizing effective intelligent management of barbershops through an intelligent method for identifying the work content of barbershop employees.

Description

Technical Field

[0001] The invention belongs to the technical field of machine vision and relates to a method for intelligently identifying the work content of barber shop employees. Background Art

[0002] At present, with the increasing demand for barber service items, large barber shop chains are gradually becoming a trend. When the number of barber shop employees is large and the service items are varied, it will bring difficulties to the daily management of the barber shop, such as: the authenticity of the work content of the employee service, the length of time for providing services, the materials used and their corresponding quotations, etc. At present, there is no intelligent system that can automatically identify, monitor and manage the above work content. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method for intelligently identifying the work content of barber shop employees, and to perform real-time behavior recognition on barber shop employees in actual scenarios, so as to realize effective intelligent management of barber shops.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A method for intelligently identifying the work content of barbershop employees, comprising the following steps:

[0006] S1: Specify the installation location and hardware conditions of the collection equipment to identify barber employees and customers;

[0007] S2: Establish a label library of barbershop staff and customer faces, and train a face recognition model;

[0008] S3: Establish an action label library related to objects, tools and people, and train a barbershop action behavior recognition model, which involves the recognition of multiple action behaviors in three categories: human actions related to objects, tools and actions in service work, object operation interactions, and human-to-human interactions;

[0009] S4: Use the trained face recognition model and barbershop action recognition model to perform action recognition in the actual barber service scene; construct the action sequence of "action pairs" of customers, employees, and action elements;

[0010] S5: Establish standard key behavior sequences for different types of hairdressing service work content as work content identification labels; and build a deep neural network model for work content identification based on the "action pair" behavior sequence to determine the service work content of barber shop employees for customers.

[0011] Furthermore, the step S1 specifically includes: capturing video frames of barber employees and customers in a barber shop scene, and performing employee ID and customer identity confirmation as well as detection and recognition of objects, tools, and behavioral actions.

[0012] Further, the step S3 specifically includes the following steps:

[0013] S31: According to the labeling rules of the AVA (aomic visual actions) dataset, we build a label library of three categories of human actions, object operation interactions, and human-to-human interactions related to objects, tools, and people in the service process, as well as the action behavior labels involved therein;

[0014] S32: Build an action behavior label library and train a barber shop action behavior recognition model;

[0015] S33: Establish a set of key activities for barber shop action behaviors.

[0016] Further, step S31 specifically includes the following steps:

[0017] S311: First, the original collected action videos are analyzed in 15-minute segments, and the 15-minute videos are uniformly divided into 300 non-overlapping 3-second segments; the video sampling follows the strategy of maintaining the time order of the action sequence;

[0018] S312: Then, the bounding box of the person in the middle frame of each 3-second segment is manually labeled using the LabelImg labeling tool;

[0019] S313: For each person in the marked frame, select an appropriate label from a pre-made action category table to describe the character action; the character actions are divided into the following three categories of labels: human body posture / displacement action, human / object / human interaction action, and human / human interaction action;

[0020] S314: Finally, all video clips are labeled to establish a haircut action video training label library.

[0021] Further, in step S32, a SlowFast model based on a 3D-Resnet50 network is used to perform action behavior recognition, wherein the SlowFast model consists of a Slow branch and a Fast branch;

[0022] First, the input video is sampled at intervals of Stride=16 frames and input into the 3D-Resnet50 backbone network to extract the environmental feature information during haircut;

[0023] Secondly, we sample from the input video frames with Stride = 2 frames as the interval, and set the number of channels to 1 / 8 of the Slow branch, and input them into the network to extract the temporal action feature information during haircutting;

[0024] Then, the Res_conv3_1 and Res_conv4_1 layers of the 3D-Resnet50 backbone are connected horizontally to integrate the temporal action information features into the environmental features;

[0025] Finally, the fused feature information after the Slow branch and the Fast branch is used in the fully connected layer to classify and predict the haircutting action.

[0026] Furthermore, in step S33, according to the constructed action behavior tag library and combined with the actual application scenario of the barber shop, all action behaviors are divided into two categories:

[0027] Key action behavior activity set: including hair cutting, hair curling, hair dyeing, hair perming, etc.; the key action behavior activity set is represented by KeyAct = {KeyAct1, ..., KeyAct i ,…,KeyAct n}, where KeyAct i is the i-th key action behavior, i=1,…,n, n is the number of key action behaviors;

[0028] Normal action behavior set: including communication, standing, sitting, walking, etc.; the normal action behavior set is represented by NormalAct = {NormalAct1, ..., NormalAct i1 ,…,NormalAct n1}, where NormalAct i1 is the i1th common action behavior, i1=1,…,n1, n1 is the number of common action behaviors.

[0029] Further, the S4 includes the following steps:

[0030] S41: Sampling images and video frames according to certain rules for face recognition and action recognition;

[0031] S42: Based on the confirmation of personal identity through face recognition and action behavior recognition in the video sequence recognition process, as well as the recognition results of various behaviors, the corresponding relationship between the customer and employee identities and the specific "action pair" behavior sequence in the service process are established, and the "action pair" relationship between the customer and the employee in the video sequence is recorded.

[0032] Further, step S41 specifically includes the following steps:

[0033] S411: During the haircut service process, samples are collected according to the average time of the service items, and the sampled images and video frames are input into the model for detection;

[0034] S412: Input the image obtained according to the sampling rule into the face recognition model to determine the customer membership and employee identity information; establish the association relationship between the customer, employee and the work content corresponding to the service action behavior at the workstation according to the workstation location and the camera index information corresponding to the workstation;

[0035] S413: Associating the human body region detected by the trained SlowFast model in a certain frame with the face recognition result of the face frame in the same frame and the same human body region in S412, for subsequent tracking of the person's identity when the face is not recognized;

[0036] S414: Use the SlowFast model trained in step S32 to perform action recognition of barber staff and customers, including: human body posture / displacement movements of customers and staff, objects and tools used by staff during the service process and customer service interaction behaviors, and interaction behavior recognition between staff and customers.

[0037] Further, step S42 specifically includes the following steps:

[0038] S421: During a service process, when a customer and an employee enter a station k In the camera range, the facial recognition model trained in step S2 is used to simultaneously complete the facial recognition of customers and employees to activate the station establishment k Customer and employee service <Cid p , Eid q >, Cid p represents the customer set, p = 1,…,m; Eid q represents the set of employees, q=1,…,m1, m and m1 represent the number of customers and employees respectively;

[0039] S422: At the workstation k In the camera range, the barber shop action recognition model trained in step S32 is used to perform action recognition of the real-time video sequence. At time t, customer Cid p The set of recognized actions is Actp t ={Actp t,1 ,…,Actp t,k};Employee Eid q The set of recognized actions is Actq t ={Actq t,1 ,…,Actq t,k1}, where k and k1 are the number of actions recognized by customers and employees at time t, and Actp t,k and Actq t,k1 For the identified action behavior:

[0040]

[0041] S423: At time t, customer Cid p Eid with employees q Forming an "action pair" <Actp t , Actq t >, and construct an "action pair" matrix;

[0042] Actp t and Actq t Each action in the set Actp t,k and Actq t,k1 Sort the probability values, take the first f actions, and form a vector for each action:

[0043] Matrixp t,f =[Actp t,f ,Actp t,f The probability value of

[0044] or

[0045] Matrixq t,f =[Actq t,f ,Actq t,f The probability value of

[0046] If Actp t or Actq t The set does not have f actions, and the actions and their probability values ​​in the vector are filled with 0 values; if the employee is waiting during the service process, he may not be in the service area, Actq t,f The actions and their probability values ​​in the vector are filled with zero values.

[0047] This will make the "action pair" <Actp t , Actq t >Constructed as a 2f*2 matrix Act <p，q>,t =[Matrixp t,1 ,…,Matrixp t,f ,Matrixq t,1 ,…,Matrixq t,f ]; Throughout the service process, for customer Cid p , based on the video frame sequence, establish a matrix based on Act <p，q>,t The time series S of “action pairs” p =[Act <p，q>,1 ,…,Act <p，q>,t ].

[0048] Further, the step S5 comprises the following steps:

[0049] S51: Based on the key action activity set KeyAct obtained in step S33, a standard key action sequence of different types of hairdressing service work contents is established as a work content identification label, represented as S k =[KeyAct k,1 ,…,KeyAct k,i ], where KeyAct k,i represents the i-th action in the k-th category of work content, KeyAct k,i ∈KeyAct; the number of key behaviors in the longest standard key behavior sequence shall prevail. If the number of other key behaviors is insufficient, all the insufficient dimensions shall be filled with 0;

[0050] S52: The customer Cid obtained in step S42 p The time series S of “action pairs” p =[Act <p，q>,1 ,…,Act <p，q>,t ] is preprocessed, and the preprocessing method is:

[0051] S521: Traverse S p The "action pair" matrix on the , using the matrix cosine similarity to calculate S p The approximation of the matrices of two adjacent "action pairs" above;

[0052] S522: If the similarity between two adjacent “action pair” matrices is greater than a threshold, the latter “action pair” matrix is ​​removed, indicating that the action behaviors at two adjacent moments are repeated;

[0053] S523: Continue to traverse all S p The “action pair” matrix on , until the end of the time series;

[0054] S p After preprocessing, the value probability column of each action is removed, and the "action pair" matrix becomes Act' <p，q>,t =[Actp t,1 ,…,Actp t,f ,Actq t,1 ,…,Actq t,f ], the "action pair" matrix Act' <p，q>,t , get the timing S p '=[Act' <p，q>,1 ,…,Act' <p，q>,t ], each remaining action in the sequence represents an “action pair” with a certain degree of difference;

[0055] S53: Step S53 of the plurality of customer service processes obtained from S52 p' and the corresponding work content identification labels, establish a training data set, build a deep neural network model for work content identification, input the training data set, and p ' and its corresponding work content labels, train the deep neural network model so that each customer's S p 'The work content sequence vector obtained by the deep neural network model has the smallest loss with its corresponding work content label, which specifically includes the following steps:

[0056] S531: Construct a training data set, collect videos and process them as described above, or obtain multiple S p ' and the corresponding work content identification label, with S p 'The maximum sequence length is used as the basis, and the insufficient sequence is padded with 0;

[0057] S532: The method for constructing a deep neural network model for work content recognition is as follows: assuming that the maximum sequence length is ActNum, p Each behavior in ' is converted into a vector with a dimension of (n+n1); the padded S p 'The dimension is (2f×(n+n1))×ActNum, where n is the number of key actions and n1 is the number of common actions;

[0058] The deep neural network model for work content recognition performs the following steps:

[0059] S after conversion by padding and action vector p 'As input, first pass through the first neural network module to convert the (2f×(n+n1))×ActNum-dimensional input data into n×ActNum-dimensional features;

[0060] Then, through the second neural network module, the n×ActNum-dimensional features are converted into n×MaxKeyActNum-dimensional features; MaxKeyActNum is the number of key behaviors in the largest standard key behavior sequence in different work contents;

[0061] Finally, the n×MaxKeyActNum-dimensional features are input into the Transformer network, where the positoin tags of the n×MaxKeyActNum-dimensional feature sequence are divided into each behavior and enter the Transformer network for positoin Embedding. The final output is MaxKeyActNum key behavior vectors, which are mapped to the corresponding standard key behavior sequence of the work content.

[0062] The beneficial effects of the present invention are as follows: the present invention establishes the behavior association sequence of employees and customers through video image identity recognition and behavior action recognition, and during the service time, through the constructed work content recognition deep neural network model, the behavior association sequence of employees and customers is mapped to their corresponding work content standard key behavior sequence, so as to achieve the purpose of identifying and outputting the work content, and assist in realizing effective intelligent management of the barber shop.

[0063] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0065] Figure 1 This is a flow chart of the method for intelligently identifying the content of customer service work by barbershop employees according to the present invention;

[0066] Figure 2 is the “action pair” time series based on the customer and employee “action pair” matrix in the present invention;

[0067] Figure 3 A structural diagram of a deep neural network model for identifying the working content in the present invention. DETAILED DESCRIPTION

[0068] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0069] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0070] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0071] like Figure 1-3 As shown, the present invention provides an intelligent detection and identification method for public service haircut behavior, comprising the following steps:

[0072] S1: Setting of installation location and installation conditions of acquisition equipment;

[0073] S2: Establish a label library of barbershop staff and customer faces, and train a face recognition model;

[0074] S3: Establish an action label library related to objects, tools and people, and train a barbershop action behavior recognition model, which involves the recognition of multiple action behaviors in three categories: human actions related to objects, tools and actions in service work, object operation interactions, and human-to-human interactions;

[0075] S4: Use the trained face recognition model and barbershop action recognition model to perform action recognition in actual barber service scenarios. Construct the action sequence of "action pairs" of customers, employees, and actions;

[0076] S5: Establish standard key behavior sequences for different types of barber service work content as work content identification labels. And based on the "action pair" behavior sequence established by S4, build a deep neural network model for work content identification to determine the service work content of barber shop employees for customers.

[0077] The S1 includes the installation of equipment and the setting of installation conditions:

[0078] S11: Specify the installation location of the acquisition device. Install the camera at each barber station to shoot the video of the barber service in real time, collect the facial images of customers and employees at the barber station, and capture the barber's haircutting actions at the barber station to establish a mapping relationship between the facial identity information and the actions during the haircut service process.

[0079] S12: Installation of hardware equipment conditions. The cameras deployed on site are required to have a refresh rate of no less than 30fps to ensure the real-time requirements of the slowfast action recognition model for high frame rates; at the same time, the camera resolution is required to be no less than 1080P to meet the requirements of face recognition detection. It can also transmit and process real-time video information for subsequent calculations.

[0080] The S2 comprises the following steps:

[0081] S21: Establish a scene face label library. Use the face images of employees and customers, and crop the image size to 224*224 pixels. Use the labeling tool labellmg to mark the position of the face, that is, manually draw a frame to mark the face part. Save the position coordinates of each labeled frame [x1, y1, x2, y2], where (x1, y1) represents the coordinates of the upper left point of the face label box, and (x2, y2) represents the coordinates of the lower right point of the face label box. Label the face image id identity number and establish the employee and customer face label library.

[0082] S22: Use the employee and customer face label library established in S21 to train the face recognition model. Preprocess the image, align the face part using the face detection algorithm, and uniformly crop it into 224*224 pixels. During the training process of the face recognition model, the input batch size batch_size is 64.

[0083] Optionally, the face recognition model uses FaceNet based on a deep convolutional neural network.

[0084] The S3 comprises the following steps:

[0085] S31: Establish a barber action label library. The steps for establishing a barber action video label library are established according to the labeling rules of the Google AVA (aomic visual actions) dataset. Specifically, it includes: first, the initially collected behavior action video is analyzed for 15 minutes, and the 15-minute video is uniformly divided into 300 non-overlapping 3-second segments. The sampling follows the strategy of maintaining the time order of the action sequence. Then, the person in the middle frame of each 3-second segment is manually annotated with the person boundary box using LabelImg, and for each person in the annotated box, an appropriate label is selected from the pre-made action category table to describe the person's action. These actions are divided into three types: human posture / displacement actions (sitting, standing, bending, etc.), human / object / human interaction actions (using a hair dye brush to dye hair, using a clipper to shave, using scissors to trim, etc.), and human / human interaction actions (chatting with customers, etc.). Finally, all video clips are fully annotated to establish a barber action behavior video training label library.

[0086] S32: Use the haircut action label library established in S31 to train the action recognition model. In the training phase, a set of video data is input cyclically from the established action recognition training set and a clip (64 frames) is randomly sampled. Then, 4 frames and 16 frames are input to the Slow branch path and the Fast branch path respectively, and the original video frames are preprocessed (scaled, randomly cropped to 224*224 size video frames, and horizontally flipped). The training cycle epoch is set to 20 times.

[0087] Optionally, the SlowFast action recognition model based on the 3D-Resnet50 convolutional neural network is used for action behavior recognition. The SlowFast model consists of a Slow branch and a Fast branch. According to the low frame rate characteristics of the Slow branch, the input video is sampled at intervals of Stride=16 frames and input into the 3D-Resnet50 backbone network to extract the environmental feature information during haircutting; according to the high frame rate and low channel characteristics of the Fast branch, the input video frame is sampled at intervals of Stride=2 frames, and the number of channels is set to 1 / 8 of the Slow branch, and input into the network to extract the temporal action feature information during haircutting; and the Res_conv3_1 and Res_conv4_1 layers of the 3D-Resnet50 backbone are horizontally connected to integrate the temporal action information features into the environmental features, and finally the fused feature information after the Slow branch and the Fast branch is used in the fully connected layer to classify and predict the haircutting action.

[0088] S33: Establish a key action set of barbershop actions. Based on the constructed action tag library and the actual application scenarios of barbershops, all actions are divided into two sets: key action set, such as hair cutting, hair curling, hair dyeing, hair perming, etc.; and common action set, such as communication, standing, sitting, walking, etc. The key action set is represented by KeyAct = {KeyAct1, ..., KeyAct2, ..., KeyAct3, ..., KeyAct4, ..., KeyAct5, ..., KeyAct6, ..., KeyAct7, ..., KeyAct8, ..., KeyAct9, ..., KeyAct10, ..., KeyAct11, ..., KeyAct12, ..., KeyAct13, ..., KeyAct14, ..., KeyAct15, ..., KeyAct16, ..., KeyAct17, ..., KeyAct18, ..., KeyAct19, ..., KeyAct20, ..., KeyAct21 i ,…,KeyAct n}, where KeyAct i is the i-th key action behavior, i = 1, ..., n, n is the number of key actions; the normal action behavior activity set is represented by NormalAct = {NormalAct1, ..., NormalAct i1 ,…,NormalAct n1}, where NormalAct i1 is the i1th common action behavior, i1=1,…,n1, n1 is the number of common action behaviors.

[0089] The S4 comprises the following steps:

[0090] S41: Sample images and video frames according to certain rules for face recognition and action recognition. The purpose of sampling is to reduce the frequency of face recognition and action recognition, reduce the recognition of repeated identities and actions, reduce the model computing power overhead, and ensure that key action behaviors can be recognized and obtained. By reducing the recognition calculation frequency and time overhead, the real-time performance of smart devices can be improved.

[0091] S411: During the hairdressing service, samples are collected based on the average time of services such as hair cutting, hair dyeing, and perming. Taking hair cutting as an example, when an employee is identified to start cutting hair for a customer with scissors, samples are collected at a reference interval of about one tenth of the average time of hair cutting, and the sampled images and video frames are input into the model for detection.

[0092] S412: First, face recognition is performed using the face recognition model trained in step S22. The image obtained according to the sampling rule is input to the face recognition model to determine the customer membership and employee identity information. According to the workstation location and the camera index information corresponding to the workstation, the association relationship between the customer, employee and the work content corresponding to the service action behavior at the workstation is established.

[0093] S413: While the SlowFast model is performing action recognition, it will also perform human detection. The human body area detected by the trained SlowFast model in a certain frame is associated with the face recognition result of the face frame in the same frame and the same human body area in S412, and is used for subsequent person identity tracking when the face is not recognized.

[0094] S414: At the same time, the SlowFast model trained in step S32 is used to identify the actions of barber staff and customers, including: the body posture / displacement actions of customers and staff, the objects and tools used by staff during the service process and the customer service interaction behavior, and the interaction behavior between staff and customers.

[0095] S42: Based on the confirmation of personal identity through face recognition and action behavior recognition in the video sequence recognition process, as well as the recognition results of various behaviors, the corresponding relationship between the customer and employee identities and the specific "action pair" behavior sequence in the service process are established, and the "action pair" relationship between the customer and the employee in the video sequence is recorded.

[0096] S421: During a service process, when a customer and an employee enter a station k The S22 trained face recognition model is activated within the camera range. The customer and employee face recognition is completed at the same time to activate the station. kCustomer and employee service <Cid p , Eid q >, Cid p represents the customer set, p = 1,…,m; Eid q represents the set of employees, q=1,…,m1. m and m1 represent the number of customers and employees respectively.

[0097] S422: At the workstation k In the camera range, the barber shop action recognition model trained by S32 is started. In the action recognition process of the real-time video sequence, at a certain time t, customer Cid p The set of recognized actions is Actp t ={Actp t,1 ,…,Actp t,k}, for example: sitting, talking to people, etc.; staff Eid q The set of recognized actions is Actq t ={Actq t,1 ,…,Actq t,k1}, for example: standing, cutting hair, talking to people, etc. Where k and k1 are the number of actions recognized by customers and employees at time t, and Actp t,k and Actq t,k1 For the identified action behavior:

[0098]

[0099] S423: At time t, customer Cid p Eid with employees q Forming an "action pair" <Actp t , Actq t >, and construct an "action pair" matrix.

[0100] Actp t and Actq t Each action in the set Actp t,k and Actq t,k1 Sort the probability values ​​and take the first f actions. Each action is formed into a vector:

[0101] Matrixp t,f =[Actp t,f ,Actp t,f The probability value of

[0102] or

[0103] Matrixq t,f =[Actq t,f ,Actq t,fThe probability value of

[0104] If Actp t or Actq t The set does not have f actions, and the actions and their probability values ​​in the vector are filled with 0 values; if the waiting time during the service process, such as the waiting time during the hair dyeing process, the employee may not be in the service area, Actq t,f The actions and their probability values ​​in the vector are filled with zero values.

[0105] Therefore, the "action pair" <Actp t , Actq t >Can be constructed as a 2f*2 matrix Act <p，q>,t =[Matrixp t,1 ,…,Matrixp t,f ,Matrixq t,1 ,…,Matrixq t,f ]. During the entire service process, p , based on the video frame sequence, establish a matrix based on Act <p，q>,t The time series S of “action pairs” p =[Act <p，q>,1 ,…,Act <p，q>,t ].

[0106] Optionally, the value of f can be set to 3.

[0107] The S5 includes the following steps:

[0108] S51: Based on the key action activity set KeyAct obtained in S33, a standard key action sequence of different types of hairdressing service work contents is established as a work content identification label. k =[KeyAct k,1 ,…,KeyAct k,i ], where KeyAct k,i represents the i-th action in the k-th category of work content, KeyAct k,i ∈KeyAct. At the same time, in order to make all label sequence dimensions uniform, the number of key behaviors in the longest standard key behavior sequence shall be used as the standard. For other key behaviors with insufficient number, all the insufficient dimensions shall be filled with 0 to facilitate calculation.

[0109] S52: The customer Cid obtained from S42 p The time series S of “action pairs” p =[Act <p，q>,1 ,…,Act <p，q>,t] to preprocess and remove the repeated "action pair" matrices with high similarity. The preprocessing method is:

[0110] (1) Traverse S p The "action pair" matrix on , calculate S p The similarity of the matrices of two adjacent "action pairs" above. Optionally, the method for calculating the similarity of the "action pair" matrices is matrix cosine similarity.

[0111] (2) If the similarity between two adjacent “action pair” matrices is greater than a certain threshold, the latter similar “action pair” matrix is ​​removed, indicating that the action behaviors at two adjacent moments are repeated.

[0112] (3) Continue to traverse all S p The “action pair” matrix on , the direct time series ends (i.e. the service ends).

[0113] S p After preprocessing, the value probability column of each action is removed, and the "action pair" matrix becomes Act' <p，q>,t =[Actp t,1 ,…,Actp t,f ,Actq t,1 ,…,Actq t,f ], the "action pair" matrix Act' <p，q>,t , get the timing S p '=[Act' <p，q>,1 ,…,Act' <p，q>,t ], each remaining action in the sequence represents an “action pair” with a certain degree of difference.

[0114] S53: S of multiple customer service processes obtained from S52 p ' and the corresponding work content identification labels to establish a training data set. At the same time, a deep neural network model for work content identification is constructed, and the training data set is input. p ' and its corresponding work content labels, train the deep neural network model so that each customer's S p 'The work content sequence vector obtained by the deep neural network model has the smallest loss with its corresponding work content label.

[0115] S531: Construct a training data set. Collect videos and process them as described above, or obtain multiple S p ' and the corresponding work content identification label. Since each S p ' has different sequence lengths, the maximum sequence length shall prevail, and the insufficient sequence shall be padded with 0.

[0116] S532: The method for constructing a deep neural network model for work content recognition is as follows: assuming that the maximum sequence length is ActNum, p Each line in ' is converted into a vector with a dimension of (n+n1). p 'The dimension is (2f×(n+n1))×ActNum. n is the number of key actions; n1 is the number of common actions.

[0117] The entire deep neural network model for work content recognition consists of the following parts:

[0118] (1) S after conversion from padding and action vector p ' as input. First, the (2f×(n+n1))×ActNum dimensional input data is converted into n×ActNum dimensional features through the first neural network module.

[0119] (2) Then, through the second neural network module, the n×ActNum-dimensional features are converted into n×MaxKeyActNum-dimensional features. MaxKeyActNum is the number of key behaviors in the maximum standard key behavior sequence in different work contents.

[0120] (3) Finally, the n×MaxKeyActNum-dimensional features are input into the Transformer network, where the positoin tags of the n×MaxKeyActNum-dimensional feature sequence are divided into each behavior and enter the Transformer network for positoin embedding. The final output is MaxKeyActNum key behavior vectors. They are mapped to the corresponding standard key behavior sequence of the work content to achieve the purpose of identifying and outputting the work content.

[0121] Optionally, the first neural network module and the second neural network module in the entire model can be different structural templates such as DNN or CNN.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for intelligently identifying the work content of barbershop employees, characterized in that: The following steps are involved: S1: Specify the installation location and hardware conditions of the collection equipment to identify barbers and customers; S2: Establish a label library of barbershop staff and customer faces, and train a face recognition model; S3: Establish an action label library related to objects, tools and people, and train a barbershop action behavior recognition model, which involves the recognition of multiple action behaviors in three categories: human actions related to objects, tools and actions in service work, object operation interaction, and human-to-human interaction; S4: using the trained face recognition model and barbershop action recognition model to perform action recognition on the actual barber service scene; constructing the "action pair" action sequence of customers, employees, and action elements; the step S4 specifically includes the following steps: S41: Sampling images and video frames according to certain rules for face recognition and action recognition; S42: According to the confirmation of the identity of the person through face recognition and action behavior recognition in the video time sequence recognition process, as well as the recognition results of various behaviors, the corresponding relationship between the identity of the customer and the employee, as well as the specific "action pair" behavior time sequence in the service process, are established, and the "action pair" relationship between the customer and the employee in the video time sequence is recorded; step S42 specifically includes the following steps: S421: During a service process, when a customer and an employee enter a station k In the camera range, the facial recognition model trained in step S2 is used to simultaneously complete the facial recognition of customers and employees to activate the station establishment k Customer and employee service <Cid p , Eid q >, Cid p represents the customer set, p = 1,…,m; Eid q represents the set of employees, q=1,…,m1, m and m1 represent the number of customers and employees respectively; S422: At the workstation k In the camera range, the barber shop action recognition model trained in step S32 is used to perform action recognition of the real-time video sequence. At time t, customer Cid p The set of recognized actions is Actp t ={Actp t,1 ,…,Actp t,k };Employee Eid q The set of recognized actions is Actq t ={Actq t,1 ,…,Actq t,k1 }, where k and k1 are the number of actions recognized by customers and employees at time t, and Actp t,k and Actq t,k1 For the recognized action behavior: Actp t,k or S423: At time t, customer Cid p Eid with employees q Form an "action pair" <Actp t , Actq t >, and construct the "action pair" matrix; Actp t and Actq t Each action in the set Actp t,k and Actq t,k1 Sort the probability values, take the first f actions, and form a vector for each action: Matrixp t,f =[Actp t,f ,Actp t,f The probability value of or Matrixq t,f =[Actq t,f ,Actq t,f The probability value of If Actp t or Actq t There are no f actions in the set, and the actions and their probability values ​​in the vector are filled with 0 values; Thus, the "action pair" <Actp t , Actq t >Constructed as a 2f*2 matrix Act <p，q>,t =[Matrixp t,1 ,…,Matrixp t,f ,Matrixq t,1 ,…,Matrixq t,f ]; Throughout the service process, for customer Cid p , according to the video frame sequence, a matrix based on Act <p，q>,t The time series S of "action pairs" p =[Act <p，q>,1 ,…,Act <p，q>,t ]; S5: Establish standard key behavior sequences for different types of hairdressing service work content as work content identification labels; and build a deep neural network model for work content identification based on the "action pair" behavior sequence to determine the service work content of barber shop employees for customers.

2. The method for intelligently identifying the work content of barbershop employees according to claim 1, characterized in that: The step S1 specifically includes: capturing video frames of barber employees and customers in a barber shop scene, and performing employee ID and customer identity confirmation as well as object, tool, and behavior detection and recognition.

3. The method for intelligently identifying the work content of barbershop employees according to claim 1, characterized in that: The step S3 specifically comprises the following steps: S31: According to the labeling rules of the AVA dataset, a label library of three categories of human actions, object operation interactions, and human-to-human interactions related to objects, tools, and people in the service process and the action behavior involved is constructed; S32: Build an action behavior label library and train a barber shop action behavior recognition model; S33: Establish a set of key activities for barber shop action behaviors.

4. The method for intelligently identifying the work content of barbershop employees according to claim 3 is characterized in that: Step S31 specifically includes the following steps: S311: First, the original collected action videos are analyzed in 15-minute segments, and the 15-minute videos are uniformly divided into 300 non-overlapping 3-second segments; the video sampling follows the strategy of maintaining the time order of the action sequence; S312: Then, the bounding box of the person in the middle frame of each 3-second segment is manually labeled using the LabelImg labeling tool; S313: For each person in the marked frame, select an appropriate label from a pre-made action category table to describe the character action; the character actions are divided into the following three categories of labels: human body posture / displacement action, human / object / human interaction action, and human / human interaction action; S314: Finally, all video clips are labeled to establish a haircut action video training label library.

5. The method for intelligently identifying the work content of barbershop employees according to claim 3 is characterized in that: In the step S32, a SlowFast model based on a 3D-Resnet50 network is used to perform action behavior recognition, wherein the SlowFast model consists of a Slow branch and a Fast branch; First, the input video is sampled at intervals of Stride=16 frames and input into the 3D-Resnet50 backbone network to extract the environmental feature information during haircut; Secondly, we sample from the input video frames with a step length of 2 frames, and set the number of channels to 1 / 8 of the Slow branch, and input them into the network to extract the temporal action feature information during haircutting; Then, the Res_conv3_1 and Res_conv4_1 layers of the 3D-Resnet50 backbone are connected horizontally to integrate the temporal action information features into the environmental features; Finally, the fused feature information after the Slow branch and the Fast branch is used in the fully connected layer to classify and predict the haircutting action.

6. The method for intelligently identifying the work content of barbershop employees according to claim 3, characterized in that: In step S33, according to the constructed action behavior tag library and combined with the actual application scenario of the barber shop, all action behaviors are divided into two categories: Key action behavior activity set: including hair cutting, hair curling, hair dyeing, and hair perming; the key action behavior activity set is represented by KeyAct = {KeyAct1, ..., KeyAct i ,…,KeyAct n }, where KeyAct i is the i-th key action behavior, i=1,…,n, n is the number of key action behaviors; Normal action behavior set: including communication, standing, sitting, and walking; the normal action behavior set is represented by NormalAct = {NormalAct1, ..., NormalAct i1 ,…,NormalAct n1 }, where NormalAct i1 is the i1th common action behavior, i1=1,…,n1, n1 is the number of common action behaviors.

7. The method for intelligently identifying the work content of barbershop employees according to claim 1, characterized in that: Step S41 specifically includes the following steps: S411: During the haircut service process, samples are collected according to the average time of the service items, and the sampled images and video frames are input into the model for detection; S412: Input the image obtained according to the sampling rule into the face recognition model to determine the customer membership and employee identity information; establish the association relationship between the customer, employee and the work content corresponding to the service action behavior at the workstation according to the workstation location and the camera index information corresponding to the workstation; S413: Associating the human body region detected by the trained SlowFast model in a certain frame with the face recognition result of the face frame in the same frame and the same human body region in S412, for subsequent tracking of the person's identity when the face is not recognized; S414: Use the SlowFast model trained in step S32 to perform action recognition of barber staff and customers, including: human body posture / displacement movements of customers and staff, objects and tools used by staff during the service process and customer service interaction behaviors, and interaction behavior recognition between staff and customers.

8. The method for intelligently identifying the work content of barbershop employees according to claim 1, characterized in that: The step S5 comprises the following steps: S51: Based on the key action activity set KeyAct obtained in step S33, a standard key action sequence of different types of hairdressing service work contents is established as a work content identification label, represented as S k =[KeyAct k,1 ,…,KeyAct k,i ], where KeyAct k,i represents the i-th action in the k-th category of work content, KeyAct k,i ∈KeyAct; the number of key behaviors in the longest standard key behavior sequence shall prevail. If the number of other key behaviors is insufficient, all the insufficient dimensions shall be filled with 0; S52: The customer Cid obtained in step S42 p The time series S of "action pairs" p =[Act <p，q>,1 ,…,Act <p，q>,t ] is preprocessed, and the preprocessing method is: S521: Traverse S p The "action pair" matrix on the , using the matrix cosine similarity to calculate S p The approximation of the matrices of two adjacent "action pairs" above; S522: If the similarity between two adjacent "action pair" matrices is greater than a threshold, the latter "action pair" matrix is ​​removed, indicating that the action behaviors at two adjacent moments are repeated; S523: Continue to traverse all S p The "action pair" matrix on , until the end of the time series; S p After preprocessing, the value probability column of each action is removed, and the "action pair" matrix becomes Act' <p，q>,t =[Actp t,1 ,…,Actp t,f ,Actq t,1 ,…,Actq t,f ], the "action pair" matrix Act' <p，q>,t , get the timing S p '=[Act' <p，q>,1 ,…,Act' <p，q>,t ], each remaining action in the sequence represents an "action pair" with a certain degree of difference; S53: Step S53 of the plurality of customer service processes obtained from S52 p ' and the corresponding work content identification labels, establish a training data set, build a deep neural network model for work content identification, input the training data set, and p ' and its corresponding work content labels, train the deep neural network model so that each customer's S p 'The work content sequence vector obtained by the deep neural network model has the smallest loss with its corresponding work content label, which specifically includes the following steps: S531: Construct a training data set, collect videos and process them as described above, or obtain multiple S p ' and the corresponding work content identification label, with S p 'The maximum sequence length is used as the basis, and the insufficient sequence is padded with 0; S532: The method for constructing a deep neural network model for work content recognition is as follows: assuming that the maximum sequence length is ActNum, p Each behavior in ' is converted into a vector with a dimension of (n+n1); the padded S p 'The dimension is (2f×(n+n1))×ActNum, where n is the number of key actions and n1 is the number of common actions; The deep neural network model for work content recognition performs the following steps: S after conversion by padding and action vector p 'As input, first pass through the first neural network module to convert the (2f×(n+n1))×ActNum-dimensional input data into n×ActNum-dimensional features; Then, through the second neural network module, the n×ActNum-dimensional features are converted into n×MaxKeyActNum-dimensional features; MaxKeyActNum is the number of key behaviors in the largest standard key behavior sequence in different work contents; Finally, the n×MaxKeyActNum-dimensional features are input into the Transformer network, where the position tags of the n×MaxKeyActNum-dimensional feature sequence are divided according to each behavior and enter the Transformer network for position embedding. The final output is MaxKeyActNum key behavior vectors, which are mapped to the corresponding standard key behavior sequence of the work content.

Images