A behavior time sequence-oriented workflow identification method

Abstract

The application relates to a behavior time sequence-oriented work flow identification method and belongs to the technical field of computer vision. A work behavior action label library for different application scenarios is established, and behavior identification is carried out; a light-weight multi-target tracking model is constructed, and character identity identification is carried out to track characters in a video; the identity and behavior of a person are bound to obtain a behavior sequence after identity binding; a work content training label library is established, a time sequence model is used to check the work content of the behavior sequence, so that the work content of the behavior sequence is judged; a work flow check library is established, the behavior sequence with the determined work content is matched with a work flow sequence belonging to the same work content in the work flow check library, and whether the work flow of the behavior sequence conforms to standard norms is judged. The application is used for checking and judging the work flow normativeness of workers in a complex video scene by establishing a relationship chain of different targets and behaviors.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a workflow recognition method oriented towards behavioral timing. Background Technology

[0002] In recent years, workflow standardization has become increasingly important for industry applications. Workflow standardization can effectively improve a company's work efficiency, thereby enhancing its competitiveness. Simultaneously, workflow standardization can improve employee professionalism and promote the standardization and normalization of employee work. In the process of standardization and normalization testing, automated testing methods can be combined to achieve the monitoring of workers' workflows.

[0003] Currently, research methods for workflows both domestically and internationally can be divided into two types: one is based on computer vision, and the other is based on sensors and voice.

[0004] While there are many voice-based solutions available, they are inherently sensitive to privacy concerns and are easily affected by environmental factors such as noise, which limits their application scenarios.

[0005] Computer vision-based methods can accurately simulate real-world situations using models, resulting in relatively accurate evaluations. However, while offering numerous advantages, their limitations are becoming increasingly apparent. As the degrees of freedom increase, the model complexity grows exponentially, leading to a surge in computational demands during the evaluation process. Therefore, these methods present a significant barrier to entry and lack universality and convenience. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a time-series-oriented workflow recognition method to monitor the work process of staff; and to design a workflow detection and recognition method that combines video and text.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A workflow recognition method oriented towards behavioral timing, mainly for checking the workflow of workers in multi-objective scenarios, includes the following steps:

[0009] S1: Based on the different requirements of workflow standardization in different application scenarios, a work behavior action label library is established using the captured video frames, and a behavior recognition network model is used to perform behavior recognition on the video frames.

[0010] S2: Build a lightweight multi-target tracking model to track people in videos; establish a face database to identify the identities of people in order to obtain the identities of the tracking targets;

[0011] S3: By measuring the overlap between the person tracking box and the behavior recognition box, the person's identity and behavior are bound together to obtain the behavior sequence after identity binding;

[0012] S4: Establish a work content training label library, and use a time series model to check the work content of the behavior sequence to determine the work content of the behavior sequence after identity binding.

[0013] S5: Establish a workflow check library, match the behavior sequence of the identified work content with the workflow sequence of the same work content in the workflow check library, and determine whether the workflow of the behavior sequence conforms to the standard specifications.

[0014] Furthermore, in step S2, the construction steps of the lightweight multi-target tracking network model are as follows:

[0015] S21: Based on the ShuffleNetV2 network structure, the backbone network C3Net of Yolov5s is replaced to obtain a lightweight Yolov5s detection network;

[0016] S22: Replace the DeepSORT detection network with the lightweight Yolov5s detection network described in S21 to form a lightweight improved DeepSORT network, thereby improving the real-time performance of detection and tracking. Feed the video after frame extraction from action recognition in step S1 into the lightweight improved DeepSORT network to track different targets in the video and obtain the tracking box information for each frame.

[0017] S23: After the target tracking calculation in step S22, the tracking video frames are sent to the face recognition model to perform face recognition on each frame and obtain the identity information of the person in the tracking box.

[0018] Furthermore, step S21 specifically includes:

[0019] S211: Input the video frames sequentially. The detection network needs to resize each frame. Then, use the Focus method to process the frame images into features. Figure 1 ;

[0020] S212: Perform feature extraction using convolutions with stride=1 and kernel_size=1*1, and output the convolutional features. Figure 2 ;

[0021] S213: First stage: The SH-b module performs downsampling on the spatial dimension to extract deep features. Sampling is performed using a DWConv module with stride=2 and kernel size=3*3, outputting a feature map. Then, the feature map obtained from SH-b is used as input to the SH-a feature extraction module, which extracts features using a DWConv module with stride=1 and kernel size=3*3, outputting the feature map. Figure 4 ;

[0022] S214: Second Stage: Feature Figure 4 As input, the SH-b module is used to downsample the spatial dimension, sampling with a stride of 2 and a kernel size of 3*3 using DWConv. After the second stage of SH-b, the features are obtained. Figure 5 ; Perform three consecutive SH-a operations, each with a stride of 1 and a kernel size of 3*3 for feature extraction using DWConv, and obtain the feature map 6 after the second stage;

[0023] S215: Third stage: First, feature map 6 obtained in the second stage is used as the input feature map for the third stage. Feature map 6 is downsampled in spatial dimension by SH-b to reduce the resolution of the image. DWConv with stride=2 and kernel_size=3*3 is used for sampling, and feature map 7 is output. Then, SH-a is used three times in succession for feature extraction. DWConv with stride=1 and kernel_size=3*3 is used for feature extraction, and feature map 8 is output.

[0024] S216: Fourth stage: First, the output feature map 8 obtained after the third stage is used as the input feature map of the fourth stage. The input feature map 8 is downsampled by SH-b, and sampled by DWConv with stride=2 and kernel_size=3*3 to reduce the resolution of the image and increase the receptive field. Finally, the feature map 9 is output.

[0025] S217: Using the SPP method, three different MaxPools are performed according to different pooling kernel sizes, and feature map 10 is output.

[0026] S218: Feature extraction is performed using SH-a, with DWConv having stride=1 and kernel_size=3*3, and the feature map 11 is output.

[0027] S219: Finally, feature maps 6, 8, and 11 are used to complete the fusion and detection.

[0028] Furthermore, step S22 specifically includes:

[0029] The video after action recognition in S1 is fed into the lightweight DeepSort model. For each frame, it will output a nob*5 dimensional matrix:

[0030] {(id1,position1),(id2,position2),...,(id j ,position j ),...,(id nob ,position nob )}

[0031] position j =(x j ,y j ,w j ,h j )

[0032] Where nob represents the number of tracked targets in the current image frame, j represents the j-th target in the current frame, j = 1…nob. id j The position represents the ID of the j-th target in the current frame image. j These represent the location information of the j-th target, including: the horizontal and vertical coordinates of the top left corner of the tracking box and the width and height of the tracking box;

[0033] Based on the lightweight and improved DeepSORT network, each tracked target is assigned an ID, set ID = {ID1, ..., IDq}, representing that a total of q targets were tracked in the video. Finally, the lightweight and improved DeepSORT network obtains the tracking bounding box information of the video, denoted as A. n :

[0034]

[0035] Where n represents the number of frames in the video. This represents the tracking box information in the i-th frame of the video, where i = 1, ..., n.

[0036]

[0037]

[0038] Where nobi represents the number of tracked targets in the i-th frame, z i This indicates that the current tracking box information belongs to the i-th frame. This represents the ID of the j-th target in the i-th frame. These represent the location information of the j-th target in the i-th frame, including the coordinates of the top-left corner of the tracking box and the width and height of the tracking box. This represents the confidence level of the object contained in the j-th target tracking box in the i-th frame, where j = 1, ..., nobi, and nobi represents the number of tracked targets in the i-th frame.

[0039] Furthermore, in step S23, the person tracking box identification method specifically includes:

[0040] Let personID be the set of all target person identities in the videos, where personID = {personID1, ..., personID}. m}, where m represents the number of people in the video library, and personID m This represents the corresponding person's identity information. Using a face recognition model, the face detection bounding boxes and their identity information for each frame of the video in S22 are obtained, denoted as... This represents the identity information of the k-th target in the i-th frame, where i = 1, ..., n. in This represents the identity information of the k-th person in the i-th frame. z i Indicates the i-th frame. It is the face localization information of the kth face in the i-th frame obtained by the face recognition model, including: the coordinates of the top left corner of the detection box and the width and height information of the detection box.

[0041] By using the face detection bounding box location information and the tracking bounding box location information of the same frame, the Euclidean distance between the top left corners of the two types of boxes and the intersection of the two types of boxes in each frame are compared to obtain the person identity information of the tracking bounding box: when the face location information of the kth person is matched in a certain frame, such as the i-th frame. Location information of the j-th target Within an approximate range, then obtain corresponding and corresponding And iterate through A n All of the above IDs with the same value will Give to these The identities of the people in these tracking frames are determined based on the identity information obtained from facial recognition.

[0042] The approximate matching method for two target boxes on the same frame is as follows: First, calculate the Euclidean distance between the top-left corner coordinates of all face detection boxes and target tracking boxes on the same frame. dist person A value less than a certain threshold D is considered to be close to or overlapped between the face detection bounding box and the target tracking bounding box. The ratio of the intersection of close to or overlapping face detection bounding boxes and target tracking boxes to the face recognition bounding box is calculated. When I is greater than or equal to a certain threshold E, it means that the current detection box and the target tracking box are within an approximate range.

[0043] Furthermore, in step S3, the behavior sequence with identity binding is obtained, which specifically includes the following steps:

[0044] S31: Use the behavior recognition model from step S1 to recognize the behavior of people in the video and obtain behavior recognition box sequence information. in

[0045]

[0046] This represents the behavior recognition information of the i-th frame; z i This indicates that the current action box appears in frame i; This indicates the identity of the person in the l-th action recognition box of the i-th frame. Before further operation, this identity is unknown. This represents the location information of the l-th action recognition box in the i-th frame, including: the coordinates of the top left corner of the action recognition box and the width and height of the action recognition box; This represents the content of the l-th action in the i-th frame. Where l = 1, ..., acti, and represents the number of action recognition boxes in the i-th frame.

[0047] S32: Behavior recognition bounding box sequence information B n The tracking box information A obtained in step S2 n Perform overlap calculations to obtain behavioral sequences with identity bindings;

[0048] All tracking boxes in each frame of the video Each with all behavior recognition boxes IOU is calculated as the ratio of the overlap between two regions to the union of the two regions. Will Corresponding If a tracking box is identified as a possible match for binding the identity of a person in the l-th action recognition box of the i-th frame, then the identity of the person in the l-th action recognition box of the i-th frame... The value assigned is Because in S23 The person's identity has been assigned through facial recognition. Then you can Person identification Association and binding. Furthermore, due to... Then it can be determined To identify a person in the personID set.

[0049] Traverse An and B n By performing IOU calculations between the target tracking bounding box and the behavior recognition bounding box in all video frames, the identity information of the person in each behavior bounding box in each video frame can be obtained.

[0050] Using the bound behavior recognition bounding box sequence information B containing personal identification information n The same identity information By video frame z i Arrange the information in ascending order; then organize the sorted information. For personID in the personID set m Those with the same person ID m The character target's corresponding behavioral information is obtained in frame order. Construct the behavioral sequence (ActP) of the target character in the video. m , in Represents personID m The initial behavior of the behavior sequence in this video, Indicates the end of the action.

[0051] Furthermore, in step S4, a time-series model is used to examine the work content of the behavioral sequence, specifically including the following steps:

[0052] S41: First, preprocess each behavior sequence obtained after step S32 to remove special characters from the data; simultaneously, only one instance of consecutively repeated behavior information is retained, resulting in a preprocessed behavior sequence with personnel identification. personID m This indicates the identity of the person in the sequence of actions. It is about personID m of The behavioral sequence after predictive processing;

[0053] S42: Obtain the behavior sequence after processing in step S41. Represent each behavior in the behavior sequence directly using word vectors. Let... There are t word vectors in total, and the word vector for each action is denoted as x. t ,but It can be represented as Set the dimension of the word vector matrix of the longest behavior sequence in the current scenario to the standard matrix dimension of the input to the subsequent content inspection model. If the dimension of the word vector matrix of a behavior sequence is insufficient, pad it with zero values.

[0054] S43: After processing all video information in the video library using the above process, we obtain behavior sequences and their corresponding person identities. We then represent these behavior sequences using word vectors to obtain a standard behavior sequence word vector matrix. Simultaneously, we label each behavior sequence word vector matrix, marking the work content of each matrix, thus constructing a work content training dataset. Based on different work contents in different application scenarios, we perform one-hot encoding on the dataset labels. Assuming there are M types of work content in a certain work scenario, plus category labels that do not conform to any work content, the one-hot encoding dimension for one type of work content is M+1.

[0055] S44: An LSTM time-series model is constructed, with each action word vector input into an LSTM unit. A softmax multi-class activation function is used as the activation function for job content inspection, classifying the job content. A cross-entropy loss function is employed to ensure that the predicted classification data distribution learned from the training data is as close as possible to the true classification data distribution. Softmax yields an M+1 dimensional vector y, where each value ranges from [0,1], representing the probability that a sequence of actions belongs to a certain job content. max =argmax(y) can be used to obtain the detection of the working content of the behavior sequence.

[0056] Furthermore, in step S5, a workflow timing check module is established to perform workflow timing checks, specifically including the following steps:

[0057] S51: Construct a workflow retention bag-of-words. Retain all action words contained in all work content and unify them into a workflow retention bag-of-words. The bag-of-words S contains r action words, S = {s1, s2, ..., sr}. r The workflow retains the bag-of-words approach to remove behavior words from the behavior sequence that are not included in the bag-of-words, i.e., noisy behaviors;

[0058] S52: Build a workflow check library, where the standard workflow behavior sequence for each type of work content is as follows: cM represents the cMth type of work content. cK represents the number of actions in the cM-th workflow. A linked list is used to store the standard workflow action sequence corresponding to each job content, and this sequence serves as a template for workflow inspection. Each P... cM The action words in the text are linked into a list according to their temporal order.

[0059] S53: The workflow sequence after step S41. Using the bag-of-words approach in step S51 to remove noisy action words and retain valid action words, All action words in the sequence are matched against action words in the bag of words. If an action word exists in the bag of words, it is retained; otherwise, it is considered a noise action word and is removed. This compresses the sequence from step S41, transforming a noisy long sequence into a noise-free short sequence.

[0060]

[0061] S54: Process the sequence ActPB' from step S53. m The input is fed into the time series model to obtain the categories of work content. This will be combined with... Standard workflow behavior sequence P with the same work content cM Compare them.

[0062] Use linked lists to store the behavior sequence Link the actions in sequence into a linked list. Lp m Each behavior in P cM Each action word in the list is treated as a storage unit in a linked list. Using the KMP algorithm for string pattern matching, Lp... m As the main string, P cM As a pattern string, the matching unit is an action word, which is used to match the action sequence string to achieve fast matching and checking in Lp. m Does it include a complete and correct standard procedure P? cM A matching unit is a single action word, which is a storage unit in a linked list. If a match is successful, it means ActPB'... m The workflow sequence and the corresponding staff members who complete the work content in the process conform to the standard process specifications.

[0063] The beneficial effects of this invention are as follows:

[0064] This patent proposes a workflow recognition method, primarily applied to check the workflow compliance of each worker in multi-target scenarios. This invention establishes different targets and their behavioral sequences, utilizing image and text information to check the workflow. It is used to detect and identify the compliance of workers' workflows in video information within complex scenarios, enabling monitoring, reporting, and early warning.

[0065] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0066] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0067] Figure 1 This is a flowchart of a behavior-time-oriented workflow recognition method disclosed in an embodiment of the present invention;

[0068] Figure 2 This is a diagram of the lightweight target detection network structure disclosed in an embodiment of the present invention;

[0069] Figure 3 This is an overall flowchart of the workflow inspection model disclosed in the embodiments of the present invention;

[0070] Figure 4 This is a flowchart of a single-frame multi-target behavior binding method disclosed in an embodiment of the present invention;

[0071] Figure 5 The diagram shows the network structure of the timing module disclosed in the embodiment of the present invention, where (a) is the timing model and (b) is the LSTM module of the timing model. Detailed Implementation

[0072] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0073] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0074] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0075] like Figure 1 As shown, the present invention provides a time-series-oriented workflow identification method, comprising the following steps:

[0076] S1: Utilize video frames captured by factory surveillance cameras to establish different behavioral sequences corresponding to different tasks in the granular pharmaceutical manufacturing process, creating a work behavior action label library. Train a behavior recognition network using data from this library. Establish a face database, with each face corresponding to an employee in the factory, for identity verification.

[0077] S11: To ensure that no target is lost in the video of the current scene, a panoramic camera must be used for video capture, and the camera's frame rate should be greater than 128fps. The camera should be mounted at a high point indoors to cover as much of the scene as possible.

[0078] S12: Based on the different workflow requirements of various application scenarios, a work behavior action label library is established. In the granule drug manufacturing workshop, the main work behavior action labels include operations such as material handling, weighing, and stirring. Relevant behavioral videos are collected to construct the work behavior action label library. The data from this library is then input into a SlowFast network for training. The label creation process mainly involves segmenting the filtered videos into non-overlapping 3-second segments. During video segmentation, the temporal characteristics between video frames within a segment and between video segments themselves must be preserved. Then, for each 0.5-second segment, the bounding boxes of the characters in the middle frames are manually labeled using LabelImg software. Whether each person's behavior is labeled and what label to use should meet the workflow inspection requirements of the current work scenario. Finally, after labeling all segments, the work behavior action label library is completed. During training, a set of video data is first iteratively input from the training set of the action label library, and a segment is randomly sampled. Then, preprocessing operations such as horizontal flipping are performed on the original input video frames. The processed video frame segments are then input into the SlowFast network, and the dataset is trained for 50 epochs with a learning rate of 0.000125 and SGD as the optimizer. Finally, the training phase begins, where classification and loss calculation are performed, and the specified number of training epochs are executed to complete the training of the SlowFast action recognition model. In use, videos are input into the SlowFast network to identify human actions in the videos and obtain information. , in This represents the behavior recognition information of the i-th frame; This indicates the identity of the person in the l-th action recognition box of the i-th frame. Before further operation, this identity is unknown. This represents the location information of the l-th action recognition box in the i-th frame, including: the coordinates of the top left corner of the action recognition box and the width and height of the action recognition box; This represents the content of the l-th action in the i-th frame. Where l = 1, ..., acti, representing the number of action recognition boxes in the i-th frame. The action information obtained by SlowFast is recorded in text in chronological order, with one piece of information recorded for each action box. The video after action recognition is sampled frame by frame every 64 frames, and concatenated into a new video sequence. This video sequence is then applied to the target tracking network.

[0079] S2: As Figures 2-3 As shown, a lightweight multi-target tracking network model is constructed for tracking employees in a factory. A face recognition model is used to identify the tracked target's face, and the target's identity is determined by using a factory face recognition database.

[0080] S21: Because the detection module of multi-object tracking networks has a slow detection speed and a large model size, it cannot meet the requirements of real-time performance. Therefore, this invention constructs a lightweight multi-object tracking network based on DeepSORT, replacing the DeepSORT detection network with a lightweight Yolov5s detection network. The main idea of ​​the lightweight Yolov5s detection network is to replace the Yolov5s backbone network C3Net based on the ShuffleNetV2 network structure.

[0081] A lightweight Yolov5s network is constructed, taking one frame of the video extracted by S1 as input each time. The model automatically resizes each frame to 640*640 pixels. The Focus method is used to process the image into a 320*320*12 feature map, where 320*320 represents the feature map size and 12 represents the number of channels. Next, feature extraction is performed using convolutions with stride=1 and kernel_size=1*1, outputting a 320*320*32 feature map. In the next stage, the SH-b module is used to downsample the spatial dimensions, extracting deeper features. This is achieved using DWConv with stride=2 and kernel_size=3*3, resulting in a 160*160*64 feature map. Then, the feature map obtained by SH-b is used as the input of the SH-a feature extraction module. Features are extracted by DWConv with stride=1 and kernel_size=3*3, resulting in an output feature map of 160*160*64. In the second stage, the 160*160*64 feature map is used as input, and the SH-b module is used to downsample the spatial dimension. Sampling is performed by DWConv with stride=2 and kernel_size=3*3. The output feature map obtained by SH-b in the second stage is an 80*80*128 feature map. Three consecutive SH-a operations are performed, each with a stride of 1 and a kernel size of 3*3 using DWConv for feature extraction, resulting in an 80*80*128 output feature map. In the third stage, the 80*80*128 feature map obtained in the second stage is used as the input feature map. This 80*80*128 feature map is then downsampled using SH-b to reduce the image resolution, using a stride of 2 and a kernel size of 3*3 using DWConv, resulting in a 40*40*256 feature map. Subsequently, three consecutive SH-a operations are performed for feature extraction, each with a stride of 1 and a kernel size of 3*3 using DWConv, resulting in another 40*40*256 feature map. The fourth stage first uses the 40*40*256 feature map obtained in the third stage as the input feature map. This input 40*40*256 feature map is then downsampled using SH-b with a stride of 2 and a kernel size of 3*3 (DWConv) to reduce image resolution and increase the receptive field. The final output feature map is then 20*20*512.Furthermore, the SPP method is employed, using three different MaxPools with pooling kernels of 5*5, 9*9, and 13*13 to increase the receptive field of the feature map and enhance high-level semantic information. The output feature map is 20*20*512. Finally, feature extraction is performed using SH-a, with DWConv using a stride of 1 and a kernel size of 3*3. The output feature map is also 20*20*512.

[0082] S22: During use, the video frames after behavior recognition in S1 are fed into a lightweight DeepSORT network to track different targets in the video frames. The core of this method is that the detection network combines Kalman filtering and the Hungarian algorithm. First, the detector obtains the target bounding box, while the Kalman filter predicts the trajectory of the target in the current frame. Then, the target bounding box obtained by the detector and the trajectory obtained by the Kalman filter are matched by IOU. When the target bounding box and the trajectory cannot match, it means that the match has failed. If the match fails three times in a row, the target is deleted from the image. If no trajectory can match the target bounding box of the detector, it means that the target has appeared, so a new trajectory is assigned to it and recorded. When the target bounding box of the detector matches the trajectory, it means that the information in the target bounding box can be transmitted. Therefore, only the identity of the person and the location information of the bounding box need to be recorded in the latter two cases. Finally, each frame of the image will output a nob*5-dimensional matrix:

[0083] {(id1,position1),(id2,position2),...,(id j ,position j ),...,(id nob ,position nob )},

[0084] position j =(x j ,y j ,w j ,h j ),

[0085] Where nob represents the number of tracked targets in the current image frame, j represents the j-th target in the current frame, j = 1…nob. id j The position represents the ID of the j-th target in the current frame image. j These represent the location information of the j-th target, including: the horizontal and vertical coordinates of the top left corner of the tracking box and the width and height of the tracking box;

[0086] Based on the lightweight and improved DeepSORT network, each tracked target is assigned an ID, set ID = {ID1, ..., IDq}, representing that a total of q targets were tracked in the video. Finally, the lightweight and improved DeepSORT network obtains the tracking bounding box information of the video, denoted as A. n :

[0087]

[0088] Where n represents the number of frames in the video. This represents the tracking box information in the i-th frame of the video, where i = 1, ..., n.

[0089]

[0090]

[0091] Where nobi represents the number of tracked targets in the i-th frame, z i This indicates that the current tracking box information belongs to the i-th frame. This represents the ID of the j-th target in the i-th frame. These represent the location information of the j-th target in the i-th frame, including the coordinates of the top-left corner of the tracking box and the width and height of the tracking box. This represents the confidence level of the object contained in the j-th target tracking box in the i-th frame, where j = 1, ..., nobi, and nobi represents the number of tracked targets in the i-th frame.

[0092] S23: Establish a face database by collecting facial images of all employees in the factory workshop. The background should be as uniform as possible, and the facial images should be clear. The collected facial images should include three angles for each person: frontal, left 45° angle, and right 45° angle. During the labeling process, each image is associated with an employee's name. During training, the data from the face database is fed into the FaceNet face recognition network for training. The dataset is trained for 50 epochs with a learning rate of 1e-2, and SGD is chosen as the optimizer. During use, the video frames after target tracking in S22 are fed into the trained face recognition model to perform face recognition on each frame, identifying each person's identity information. The set of all video target person identities is defined as personID, where personID = {personID1, ..., personID...}. m}, where m represents the number of people in the video library, and personID m This represents the corresponding person's identity information. Tracking was performed using a face recognition model to obtain the face detection bounding boxes and their identity information for each frame of the video in S22, denoted as... This represents the identity information of the k-th target in the i-th frame, where i = 1, ..., n. in This represents the identity information of the k-th person in the i-th frame. z i Indicates the i-th frame. This refers to the face localization information of the k-th person in the i-th frame obtained by the face recognition model, including the coordinates of the top-left corner of the detection box and the width and height information of the detection box. Using the face detection box localization information and the tracking box localization information in the same frame, the Euclidean distance between the top-left corners of the two types of boxes and the intersection of the two types of boxes in each frame are compared to obtain the person's identity information from the tracking box: when the k-th face localization information is matched in a certain frame, such as the i-th frame... Location information of the j-th target Within an approximate range, then obtain corresponding and corresponding And iterate through A n All of the above IDs with the same value will Give to these The identities of the people targeted in these tracking frames can be determined based on facial recognition.

[0093] The approximate matching method for two target boxes on the same frame is as follows: First, calculate the Euclidean distance between the top-left corner coordinates of all face detection boxes and target tracking boxes on the same frame. dist person A value less than 0.3 indicates that the face detection bounding box and the target tracking bounding box are adjacent or overlap. The ratio of the intersection of adjacent or overlapping face detection bounding boxes and target tracking boxes to the face recognition bounding box is calculated. When I > 0.9, it means that the current detection box and the target tracking box are within an approximate range.

[0094] S3: Obtain the tracking box information and behavior box information of the person. By the overlap between the tracking box and the behavior box, bind the person's identity and behavior to obtain the behavior sequence with identity binding.

[0095] S31: Use the behavior recognition model from step S1 to recognize the behavior of people in the video and obtain behavior recognition box sequence information. in

[0096]

[0097] This represents the behavior recognition information of the i-th frame; This indicates the identity of the person in the l-th action recognition box of the i-th frame. Before further operation, this identity is unknown. This represents the location information of the l-th action recognition box in the i-th frame, including: the coordinates of the top left corner of the action recognition box and the width and height of the action recognition box; This represents the content of the l-th action in the i-th frame. Where l = 1, ..., acti, and represents the number of action recognition boxes in the i-th frame.

[0098] S32: As Figure 4 As shown, using the behavior recognition bounding box sequence information B n The tracking frame information A obtained in step S2 n Perform overlap calculations to obtain the behavior sequence with identity binding; extract all tracking boxes from each frame of the video. Each with all behavior recognition boxes IOU is calculated as the ratio of the overlap between two regions to the sum of the values ​​of the two regions. Will Corresponding If a tracking box is identified as a possible match for binding the identity of a person in the l-th action recognition box of the i-th frame, then the identity of the person in the l-th action recognition box of the i-th frame... The value assigned is Because in S23 The person's identity has been identified through facial recognition. Then you can Person identification Association and binding. Furthermore, due to... Then it can be determined To identify a person in the personID set.

[0099] Traverse A n and B n By performing IOU calculations between the target tracking bounding box and the behavior recognition bounding box in all video frames, the identity information of the person in each behavior bounding box in each video frame can be obtained.

[0100] B, which contains the bound behavior recognition box sequence information with the person's identity information n The same identity information By video frame z i Arrange the information in descending order of size; then organize the sorted information, assuming... For personID in the personID set m Those with the same person ID m The character target's corresponding behavioral information is obtained in frame order. Construct the behavioral sequence (ActP) of the target character in the video. m ActP m ={personID m pst ,...,p ed}, where p st p represents the beginning of a sequence of actions. ed This indicates the end of the action.

[0101] S4: As Figure 5 As shown in (a), a training label library for work content is established, and a time series model is used to check the work content of the behavior sequence; the behavior sequence with identity binding obtained in step S3 is used as the input of the time series model to determine the work content of the sequence.

[0102] S41: First, the data processed in S32 is preprocessed using functions from the zhon library in Python to remove punctuation marks and other special characters. The behavioral sequences and human identity information obtained in S32 are complex and contain a lot of redundant information. Therefore, only one instance of consecutively repeated behavioral information is retained, resulting in the preprocessed behavioral sequences with human identity information. personID m This indicates the identity of the person in the sequence of actions. It is about personID m of The behavioral sequence after predictive processing;

[0103] S42: Obtain the behavior sequence after processing in step S41. Represent each behavior in the behavior sequence directly using word vectors. Let... There are t word vectors in total, and the word vector for each action is denoted as x. t ,but It can be represented as Set the dimension of the word vector matrix of the longest behavior sequence in the current scenario to the standard matrix dimension of the input to the subsequent content inspection model. If the dimension of the word vector matrix of a behavior sequence is insufficient, pad it with zero values.

[0104] S43: After processing all video information in the video library using the above process, we obtain behavior sequences and their corresponding person identities. We then represent the behavior sequences using word vectors to obtain a standard behavior sequence word vector matrix. Simultaneously, we label each behavior sequence word vector matrix, marking the work content of each matrix, thus constructing a work content training dataset. Based on different work contents in different application scenarios, we perform one-hot encoding on the dataset labels. In a granule pharmaceutical manufacturing plant, there are 9 types of work contents, plus category labels that do not match any work content. One type of work content has a one-hot encoding dimension of 9+1.

[0105] S44: As Figure 5As shown in (b), an LSTM is used to construct the temporal model, with each action word vector input into an LSTM unit. A granule pharmaceutical manufacturing plant involves various work processes, including dispensing, soft material preparation, granulation and drying, sizing and mixing, packaging, etc. Therefore, the softmax multi-class activation function can be used as the temporal activation function to classify the work processes. The cross-entropy loss function can be chosen to ensure that the distribution of predicted classification data learned from the training data is as close as possible to the distribution of the true classification data. The softmax function will yield a 9+1 dimensional vector y, where each value ranges from [0,1], representing the probability that a sequence of actions belongs to a certain work process. max =argmax(y) can be used to obtain the detection of the working content of the behavior sequence.

[0106] S5: Establish a workflow inspection library. Compare the behavior sequence identified in step S4 with the workflow sequence of the same work content in the workflow inspection library to determine whether the workflow conforms to the standard process.

[0107] S51: Construct a workflow retention bag-of-words. Retain all action words contained in all work content and unify them into a workflow retention bag-of-words. The bag-of-words S contains r action words, S = {s1, s2, ..., sr}. r The workflow retains the bag-of-words approach to remove behavior words from the behavior sequence that are not included in the bag-of-words, i.e., noisy behaviors;

[0108] S52: As Figure 3 As shown, a workflow check library is constructed, using a linked list to store the standard workflow behavior sequence corresponding to each job. The workflow check library includes all job contents, such as drug preparation, soft material preparation, granulation and drying, granulation and mixing, dispensing and packaging, etc. Each job content has one and only one corresponding standard behavior sequence. The standard workflow behavior sequence corresponding to each job content is as follows: cM represents the cMth type of work content, and cM can be 9. cK indicates that there are cK actions in the cMth workflow. The workflow inspection library is used as a template library for workflow inspection.

[0109] S53: The workflow sequence after step S41. Using the bag-of-words approach in step S51 to remove noisy action words and retain valid action words, All action words in the sequence are matched against action words in the bag of words. If an action word exists in the bag of words, it is retained; otherwise, it is considered a noise action word and is removed. This compresses the sequence from step S41, transforming a noisy long sequence into a noise-free short sequence.

[0110]

[0111] S54: Process the sequence ActPB' from step S53. m The input is fed into the time series model to obtain the categories of work content. This will be combined with... Standard workflow behavior sequence P with the same work content cM Compare them.

[0112] Use linked lists to store the behavior sequence Link the actions in sequence into a linked list. Lp m Each behavior in P cM Each action word in the list is treated as a storage unit in a linked list. Using the KMP algorithm for string pattern matching, Lp... m As the main string, P cM As a pattern string, the matching unit is an action word, which is used to match the action sequence string to achieve fast matching and checking in Lp. m Does it include a complete and correct standard procedure P? cM A matching unit is a single action word, which is a storage unit in a linked list. If a match is successful, it means ActPB'... m The workflow sequence and the corresponding staff members who complete the work content in the process conform to the standard process specifications.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A workflow recognition method oriented towards behavioral timing, mainly for checking the workflow of workers in multi-objective scenarios, including the following steps: S1: Based on the different requirements of workflow standardization in different application scenarios, a work behavior action label library is established using the captured video frames, and a behavior recognition network model is used to perform behavior recognition on the video frames. S2: Build a lightweight multi-target tracking model to track people in the video; A face database is established to identify individuals and obtain the identities of tracking targets. This includes: constructing a lightweight multi-target tracking network based on DeepSORT, and replacing the DeepSORT detection network with a lightweight Yolov5s detection network; the lightweight Yolov5s detection network is based on the ShuffleNetV2 network structure, replacing the Yolov5s backbone network C3Net; during use, video frames after behavior recognition in S1 are fed into the lightweight DeepSORT network to track different targets in the video frames; the detection network combines Kalman filtering and the Hungarian algorithm, first obtaining the target bounding box through the detector, and simultaneously predicting the trajectory of the target in the current frame using Kalman filtering. The detector obtains a bounding box, and then performs IOU matching between the target bounding box and the current frame trajectory obtained by Kalman filtering. If the bounding box and trajectory cannot match, the match fails, and if the match fails three times in a row, the target is removed from the image. If no trajectory can match the detector's bounding box, it means that the target has appeared, and a new trajectory is assigned to it and recorded. When the detector's bounding box and trajectory match successfully, it means that the information in the bounding box can be transmitted. The identity of the person and the location information of the bounding box are recorded in the latter two cases. Finally, each frame of the image outputs a nob*5 dimensional matrix. According to the lightweight improved DeepSORT network, each tracked target is assigned an ID, set ID={ID1,…, IDq}, indicating that a total of q targets were tracked in the video. Finally, the lightweight improved DeepSORT network records the tracking box information of the video as follows: ; S3: By measuring the overlap between the person tracking box and the behavior recognition box, the person's identity and behavior are bound together to obtain the behavior sequence after identity binding; S4: Establish a work content training label library, and use a time series model to check the work content of the behavior sequence to determine the work content of the behavior sequence after identity binding. S5: Establish a workflow check library, match the behavior sequence of the identified work content with the workflow sequence of the same work content in the workflow check library, and determine whether the workflow of the behavior sequence conforms to the standard specifications.

2. The workflow identification method based on behavior and timing according to claim 1, characterized in that: In step S21, the lightweight Yolov5s detection network specifically includes: S211: Input the video frames sequentially, and the detection network needs to resize each frame image; and process the frame image into feature map 1 using the Focus method; S212: Perform feature extraction using convolution with stride=1 and kernel_size=1*1, and output the convolutional feature map 2. S213: First stage: The SH-b module is used to downsample the spatial dimension to extract deep features. The feature map 3 is output by sampling with stride=2 and kernel_size=3*3 using DWConv. Then, the feature map obtained by SH-b is used as the input of the SH-a feature extraction module. The feature map 4 is output by sampling with stride=1 and kernel_size=3*3 using DWConv. S214: Second stage: Using feature map 4 as input, the SH-b module is used to downsample the spatial dimension. The sampling is performed by DWConv with stride=2 and kernel_size=3*3. After the second stage of SH-b, feature map 5 is obtained. Then, SH-a operation is performed three times in succession. Each time, feature extraction is performed by DWConv with stride=1 and kernel_size=3*3. Feature map 6 is obtained after the second stage. S215: Third stage: First, feature map 6 obtained in the second stage is used as the input feature map for the third stage. Feature map 6 is downsampled in spatial dimension by SH-b to reduce the resolution of the image. DWConv with stride=2 and kernel_size=3*3 is used for sampling, and feature map 7 is output. Then, SH-a is used three times in succession for feature extraction. DWConv with stride=1 and kernel_size=3*3 is used for feature extraction, and feature map 8 is output. S216: Fourth stage: First, the output feature map 8 obtained after the third stage is used as the input feature map of the fourth stage. The input feature map 8 is downsampled by SH-b, and sampled by DWConv with stride=2 and kernel_size=3*3 to reduce the resolution of the image and increase the receptive field. Finally, the feature map 9 is output. S217: Using the SPP method, three different MaxPools are performed according to different pooling kernel sizes, and feature map 10 is output. S218: Feature extraction is performed using SH-a, with DWConv having stride=1 and kernel_size=3*3, and the feature map 11 is output. S219: Finally, feature maps 6, 8, and 11 are used to complete the fusion and detection.

3. The workflow identification method based on behavior and timing according to claim 2, characterized in that: Step S22 specifically includes: The video from action recognition in S1 is fed into the lightweight DeepSort model, which outputs a nob*5 dimensional matrix for each frame: , , Where nob represents the number of tracked targets in the current image frame, j represents the j-th target in the current frame, j=1…nob; This represents the ID of the j-th target in the current frame image. These represent the location information of the j-th target, including: the horizontal and vertical coordinates of the top left corner of the tracking box and the width and height of the tracking box; Based on the lightweight and improved DeepSORT network, each tracked target is assigned an ID, set ID={ID1,…,IDq}, representing that a total of q targets were tracked in the video; finally, the lightweight and improved DeepSORT network obtains the tracking bounding box information of the video, denoted as... : Where n represents the number of frames in the video. This represents the tracking box information in the i-th frame of the video, where i=1,…,n. , in This represents the number of targets in the i-th frame. This indicates that the current tracking box information belongs to the i-th frame. This represents the ID of the j-th target in the i-th frame. ∈ID, These represent the location information of the j-th target in the i-th frame, including the coordinates of the top-left corner of the tracking box and the width and height of the tracking box. This represents the confidence level of the object contained in the j-th target tracking box in the i-th frame, where j=1,…, , This represents the number of tracked targets in the i-th frame.

4. The workflow identification method based on behavior and timing according to claim 3, characterized in that: In step S23, the person tracking box identification method specifically includes: Set the set of identities of all target individuals in the videos as ,gather , m represents the number of people in the video library, This represents the corresponding person's identity information; using a face recognition model, the face detection bounding box and its identity information for each frame of the video in S22 are obtained, denoted as... , represents the identity information of the k-th target in the i-th frame, i=1,…,n; , in This represents the identity information of the k-th person in the i-th frame. ∈ , Indicates the i-th frame. It is the face localization information of the kth face in the i-th frame obtained by the face recognition model, including: the coordinates of the top left corner of the detection box and the width and height information of the detection box; Using the face detection bounding box location information and the tracking bounding box location information of the same frame, the Euclidean distance between the top left corners of the two types of boxes and the intersection of the two types of boxes in each frame are compared to obtain the person identity information of the tracking bounding box: when the face location information of the kth person is matched in the i-th frame... Location information of the j-th target Within an approximate range, then obtain corresponding and corresponding and iterate through All of the above IDs with the same value will Give to these To identify the individuals targeted in these tracking frames based on facial recognition. The approximate matching method for two target boxes on the same frame is as follows: First, calculate the Euclidean distance between the top-left corner coordinates of all face detection boxes and target tracking boxes on the same frame. ,Bundle If the value is less than a certain threshold D, it is determined that the face detection box and the target tracking box are adjacent or overlap; the ratio of the intersection of adjacent or overlapping face detection boxes and target tracking boxes to the face recognition box is calculated. ,when I Greater than or equal to a certain threshold E When the current detection box and the target tracking box are within an approximate range.

5. The workflow identification method based on behavioral timing according to claim 1, characterized in that: In step S3, the behavior sequence with identity binding is obtained, which specifically includes the following steps: S31: Use the behavior recognition model from step S1 to recognize the behavior of people in the video and obtain behavior recognition box sequence information. ,in This represents the behavior recognition information of the i-th frame; Indicates the i-th frame. l The identity of the person identified by the behavior bounding box is unknown until further action is taken; , Indicates the i-th frame. l The location information of each behavior recognition bounding box includes: the coordinates of the top left corner of the behavior recognition bounding box and the width and height of the behavior recognition bounding box; Indicates the i-th frame. l One behavior, among which l =1,…, , representing the number of action recognition boxes in the i-th frame; S32: Behavior recognition bounding box sequence information B n The tracking box information obtained in step S2 Perform overlap calculations to obtain behavioral sequences with identity bindings; All tracking boxes in each frame of the video Each with all behavior recognition boxes Perform IOU calculation: Will Corresponding It is determined to be usable in the i-th frame. l If a tracking box is bound to a person's identity in a behavior recognition box, then the tracking box in the i-th frame... l Individual Behavior Recognition Box for Person Identity The value assigned is ;Will Person identification Association and binding; for The identity of a specific person in the group is confirmed; Traversal and In all video frames, the target tracking box is compared with the behavior recognition box to calculate the IOU, and the person's identity information in all behavior boxes in each video frame is obtained. Using the bound action recognition bounding box sequence information containing personal identification information The same identity information By video frame Arrange the information in ascending order; organize the sorted information, assuming... for In the set They will have the same identity The character target's corresponding behavioral information is obtained in frame order. Construct the behavioral sequence of the target character in the video. , ,in express The initial behavior of the behavior sequence in this video, Indicates the end of the action.

6. The workflow identification method based on behavior and timing according to claim 5, characterized in that: In step S4, a time-series model is used to examine the work content of the behavioral sequence, specifically including the following steps: S41: First, preprocess each behavior sequence obtained after step S32 to remove special characters from the data; retain only one instance of consecutively repeated behavior information to obtain the preprocessed behavior sequence with personnel identity. , This indicates the identity of the person in the sequence of actions. Yes of The behavioral sequence after predictive processing; S42: Obtain the behavior sequence after processing in step S41; represent each behavior in the behavior sequence directly using word vectors; let... There are t word vectors in total, and the word vector for each action is denoted as . ,but Represented as Set the dimension of the word vector matrix of the longest behavior sequence in the current scenario to the standard matrix dimension of the input of the subsequent content inspection model; if the dimension of the word vector matrix of a behavior sequence is insufficient, fill it with zero values; S43: After processing all video information in the video library through steps S41-S42, obtain the behavior sequences and their corresponding person identities, and represent the behavior sequences using word vectors to obtain a standard behavior sequence word vector matrix; label each behavior sequence word vector matrix to mark the work content of each behavior sequence word vector matrix, and construct a work content training dataset; according to different work contents in different application scenarios, perform one-hot encoding on the labels of the dataset; assuming there are M types of work contents in a certain work scenario, plus the category label that does not conform to any work content, the one-hot encoding dimension of one type of work content is M+1; S44: A temporal model is constructed using LSTM, with each action word vector input into an LSTM unit; The multi-class activation function is used as the activation function for job content inspection, and is used to classify the job content; the cross-entropy loss function is adopted so that the distribution of predicted classification data learned on the training data is as close as possible to the distribution of real classification data; This will result in an M+1 dimensional vector y, where each value in y ranges from [0,1], representing the probability that a sequence of actions belongs to a certain type of work content. This allows for the detection of the working content of the behavioral sequence.

7. The workflow identification method based on behavior and timing according to claim 6, characterized in that: In step S5, a workflow sequence check module is established to perform workflow standardization checks, specifically including the following steps: S51: Construct a workflow retention bag-of-words, retaining all behavioral terms contained in all work content and forming a unified workflow retention bag-of-words, S containing r behavioral terms. ;Utilize the workflow to retain the bag-of-words and remove behavior words from the behavior sequence that are not included in the bag-of-words; S52: Build a workflow check library, where the standard workflow behavior sequence for each type of work content is as follows: cM=1,…,M, where cM represents the cM-th type of work content. K=1,…,cK, where cK represents the number of actions in the cM-th workflow; a linked list is used to store the standard workflow action sequence corresponding to each work content, and this sequence is used as a template for workflow inspection; each The action words in the text are linked into a list according to their temporal order. ; S53: The workflow sequence after step S41. The noisy action words are removed using the bag-of-words method in step S51, and the valid action words are retained. All action words in the sequence are matched against action words in the bag of words. If an action word exists in the bag of words, it is retained; otherwise, it is considered a noise action word and is removed. This process compresses the sequence from step S41, transforming a noisy long sequence into a noise-free short sequence. S54: Process the sequence after step S53 The input is fed into the time series model to obtain the category of the work content; this will be combined with... Standard workflow behavior sequence with the same job content Perform a comparison; Use linked lists to store the behavior sequence The actions are linked together in chronological order to form a linked list. ;Will Each action word in the text is related to... Each action word in the list is considered a storage unit in a linked list; the KMP algorithm for string pattern matching is used to... As the main string, As a pattern string, it is used for matching behavioral temporal sequences. The matching and checking are performed within... Does it include complete and correct standard procedures? The matching unit is a single word, which is a storage unit in the linked list; if the match is successful, it means... The workflow sequence and the corresponding staff members who complete the work content in the process conform to the standard process specifications.

Images