Suspected sleep recognition method based on action recognition

Abstract

The invention discloses a suspected sleep recognition method based on action recognition. The method comprises the following steps: detecting a human body in a real-time image frame; the detected human body frame is sent to a ByteTrack algorithm to be tracked; cutting a tracking result into a single human body image, zooming the single human body image to a uniform size, and sending the single human body image to an HRNet algorithm for human body skeleton key point prediction; taking human skeleton key points of the same target extracted from the same real-time image frame as a sequence of the target, and continuously obtaining N sequences of the same target; and inputting the N sequences of the same target into the trained and improved STGCN model for action recognition, and generating a suspected sleep recognition result. According to the method, the suspected sleep state of the sailor can be detected in time by analyzing the body movement and the behavior mode of the sailor. According to the method, the monitoring accuracy and timeliness can be improved, the burden of manual monitoring can be relieved, and the safety in the navigation process is ensured.

Description

Technical Field

[0001] The present invention relates to the field of sleeping behavior recognition, and in particular to a suspected sleeping recognition method based on action recognition. Background Art

[0002] With the rapid development of the ocean transportation industry, the time and distance that large ships spend on the sea are increasing. In order to ensure navigation safety, the mental state and work efficiency of the crew are crucial. However, long hours of sea operations and monotonous working environment can easily lead to fatigue of the crew, which may in turn cause sleep problems. In this case, if not discovered and handled in time, it may lead to serious navigation accidents, threatening the safety of the ship and the crew.

[0003] Traditional crew monitoring methods mainly rely on manual duty and monitoring, but this method is not only inefficient but also easily affected by human factors. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a suspected sleeping identification method based on motion recognition, which can monitor the sleeping of the target person in real time and ensure safe operation.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A method for identifying suspected sleeping based on motion recognition is provided, which comprises the following steps:

[0007] S1. Hard decode the camera through the streaming program to obtain real-time image frames;

[0008] S2, detecting the human body in the real-time image frame by Yolov8 algorithm, if no human body frame is detected, returning to step S1; otherwise, proceeding to step S3;

[0009] S3, sending the human body frame detected by the Yolov8 algorithm to the ByteTrack algorithm for tracking. If no target is tracked, return to step S1; otherwise, obtain the tracking result and enter step S4;

[0010] S4, cropping the tracking results into a single human body image and scaling it to a uniform size, and sending it to the HRNet algorithm for human skeleton key point prediction to obtain a set of human skeleton key point coordinates;

[0011] S5, taking the human skeleton key points of the same target extracted from the same real-time image frame as a sequence of the target, and continuously acquiring N sequences of the same target;

[0012] S6. Input N sequences of the same target into the trained and improved STGCN model for action recognition and generate suspected sleeping recognition results.

[0013] Furthermore, in step S1, real-time image frames are acquired at a speed of 10 frames per second.

[0014] Further, in step S4, the width and height of the uniform size are 192 and 256 pixels respectively.

[0015] Furthermore, the key points of the human skeleton include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and are denoted as Q = {q i (x,y)│i=1,2,…,17}, where Q represents the coordinate set of the key points of the human skeleton, q i (x,y) is the coordinate of the i-th human skeleton key point.

[0016] Furthermore, the STGCN model includes a BN layer, a feature extraction module, a pooling layer, and a fully connected layer connected in sequence; wherein:

[0017] The BN layer is used to standardize the data input into the STGCN model and output the standardized data;

[0018] Feature extraction module, used to extract time series features and spatial features of standardized data;

[0019] The pooling layer is used to pool the time series features and spatial features of the standardized data;

[0020] The fully connected layer is used to perform action recognition on the output of the pooling layer and generate suspected sleeping recognition results.

[0021] Furthermore, the feature extraction module includes a first ATT module, a first DGCN module, a first TCN module, a first MATT module, a second ATT module, a second DGCN module, a second TCN module and a second MATT module which are connected in sequence.

[0022] Furthermore, the action recognition result of the fully connected layer is the probability value of sleeping. When the probability value of sleeping recognized by the fully connected layer is greater than 0.9, the final recognition result is suspected sleeping; when the probability value of sleeping recognized by the fully connected layer is less than or equal to 0.9, the final recognition result is normal.

[0023] Furthermore, when the final recognition result is suspected sleeping, based on the time when the recognition result is obtained, the streaming media is used to push back 3 minutes and forward 2 minutes to form a 5-minute video to push sleeping warning data.

[0024] The beneficial effects of the present invention are:

[0025] 1. This method can timely detect the suspected sleeping state of the crew by analyzing the body movements and behavior patterns of the crew, and promptly issue an alarm when the crew appears suspected of sleeping behavior. This method can not only improve the accuracy and timeliness of monitoring, but also reduce the burden of manual monitoring and ensure safety during navigation.

[0026] 2. This method is not only applicable to the cabins of large ships such as commercial cargo ships and cruise ships, but also can be adapted to various environments and scenarios, whether indoors or outdoors, and can effectively identify sleep. It can be seamlessly integrated with existing monitoring systems and intelligent analysis platforms to provide an enhanced behavior recognition capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram of the process of this method;

[0028] Figure 2 Schematic diagram of the key points of the human skeleton. DETAILED DESCRIPTION

[0029] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0030] like Figure 1 As shown, the suspected sleeping identification method based on action recognition includes the following steps:

[0031] S1. Hard decode the camera through the streaming program to obtain real-time image frames;

[0032] S2, detecting the human body in the real-time image frame by Yolov8 algorithm, if no human body frame is detected, returning to step S1; otherwise, proceeding to step S3;

[0033] S3, sending the human body frame detected by the Yolov8 algorithm to the ByteTrack algorithm for tracking. If no target is tracked, return to step S1; otherwise, obtain the tracking result and enter step S4;

[0034] The tracking result can be expressed as O[j] = {o1, i2, o3, …, o j}, where j is the target ID of the current frame, indicating that the identity information of each target is numbered;

[0035] S4, cropping the tracking results into a single human body image and scaling it to a uniform size, and sending it to the HRNet algorithm for human skeleton key point prediction to obtain a set of human skeleton key point coordinates;

[0036] S5, taking the human skeleton key points of the same target extracted from the same real-time image frame as a sequence of the target, and continuously acquiring N sequences of the same target;

[0037] S6. Input N sequences of the same target into the trained and improved STGCN model for action recognition and generate suspected sleeping recognition results.

[0038] In step S1 , real-time image frames are acquired at a speed of 10 frames per second.

[0039] In step S4, the width and height of the uniform size are 192 and 256 pixels respectively.

[0040] In step S5, the value of N is 150, and the N sequences can be represented as {p1, p2, p3, …, p 150}.

[0041] like Figure 2 As shown, the key points of the human skeleton include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and are denoted as Q = {q i (x,y)│i=1,2,…,17}, where Q represents the coordinate set of the key points of the human skeleton, q i (x,y) is the coordinate of the i-th human skeleton key point.

[0042] The STGCN model includes a BN layer, a feature extraction module, a pooling layer, and a fully connected layer connected in sequence; among them:

[0043] The BN layer is used to standardize the data input into the STGCN model and output the standardized data;

[0044] Feature extraction module, used to extract time series features and spatial features of standardized data;

[0045] The pooling layer is used to pool the time series features and spatial features of the standardized data;

[0046] The fully connected layer is used to perform action recognition on the output of the pooling layer and generate suspected sleeping recognition results.

[0047] The feature extraction module includes a first ATT module, a first DGCN module, a first TCN module, a first MATT module, a second ATT module, a second DGCN module, a second TCN module and a second MATT module connected in sequence.

[0048] When a person is sleeping, the body generally does not move at all or only moves slightly for a long period of time. The above characteristics show that sleeping behavior is a temporal and spatial problem. The trained and improved STGCN model used in this embodiment can combine temporal and spatial information very well; the connection between the 17 key points of the human skeleton represents spatial information, and the human body motion sequence N of the previous and next frames represents temporal information. The combination of the two can extract features from the dimensions of time and space, thereby improving the recognition ability and reliability of this method.

[0049] The improved STGCN model after training used in this embodiment can be obtained by collecting business data and self-collected data for training; and in order to enhance the accuracy and robustness of this model, it is achieved in this embodiment by enhancing the feature extraction capabilities in time and space. The existing STGCN model extracts time series features through the TCN module and spatial features through the GCN module, and then judges the action behavior by combining the time and space of TCN and GCN; this embodiment retains part of the structure of the STGCN algorithm model in this scheme, and improves part of the structure, that is, the GCN module of the original STGCN model is replaced with a DGCN module, and the MATT multi-head attention mechanism is added. The specific improvement plan is as follows:

[0050] The CeN module in the DGCN module is a lightweight context encoding network that can learn the topological information of the key points of the human skeleton in the global context. According to the characteristics of human motion, the movement of the key points of the human skeleton is relatively small during deep sleep; under other behaviors, the movement amplitude of some key points of the human skeleton is large; at the same time, the key points of the skeleton at each position will convey very different information. For example, if it is a hand joint, the action is likely to be raising the hand, waving the hand, etc.; if it is a foot joint, the action may be kicking, walking, etc.; in view of the above characteristics, in this solution, the algorithm capability is improved by strengthening the feature extraction of the key points of the human skeleton. At the same time, the CeN module uses a 1×1 low-computation convolution layer to enhance the feature extraction capability in a very short time.

[0051] The MATT module (MATT multi-head attention mechanism) is used to enhance the ability of the improved STGCN model to extract features in time and space, which is more conducive to the network extracting useful information such as the motion changes between key points and the relationship between timelines, suppressing useless information, and making the network model more focused; the specific formula of the MATT module is as follows:

[0052] Assume we have an input data matrix X = [x1, x2, x3, …, x i ], weight matrix W Q ,W K ,W V , where X is the data input to the network, i represents the number of data elements, and W Q ,W K ,W V is the weight matrix obtained through network learning, and is the parameter matrix obtained through training; for each input data x i , we can calculate:

[0053] Q=W Q *x i

[0054] K=W K *x i

[0055] V=W V *x i

[0056] Where Q, K, V are vector matrices named query, key, and value.

[0057] Next, calculate the single-head attention (ATT) weight matrix. The calculation method is as follows:

[0058]

[0059] d k is the dimension of the K vector; softmax is an existing function; after passing the softmax function, we get a matrix att consisting of numbers between 0 and 1, and the sum of this att is 1; multiply the obtained weight matrix with the value vector C, and perform weighted summation. The formula is as follows:

[0060]

[0061] where att(Q,K,V) i is the attention output vector corresponding to the i-th query, V j refers to the j-th column vector of the matrix, a ij is the weight of the i-th query to the j-th value in the attention score matrix.

[0062] The calculation of the multi-head attention (MATT) weight matrix is ​​as follows:

[0063] In the multi-head attention mechanism, the input data is divided into multiple heads, each head is calculated according to the above single-head calculation method, and finally the results are spliced ​​together to form the final output, which is expressed as:

[0064] Z=W0*Concat(att1,att2,att3,...,att n )

[0065] Where W O It is an output weight matrix that can be trained by the network. Concat means concatenating the outputs of all attention heads together, and n represents multiple heads.

[0066] In this embodiment, the action recognition result of the fully connected layer is the probability value of sleeping. When the probability value of sleeping recognized by the fully connected layer is greater than 0.9, the final recognition result is suspected sleeping; when the probability value of sleeping recognized by the fully connected layer is less than or equal to 0.9, the final recognition result is normal.

[0067] When the final recognition result is suspected sleeping, based on the time of obtaining the recognition result, the streaming media is used to push back 3 minutes and forward 2 minutes to form a 5-minute video to push sleeping warning data.

[0068] In summary, the improved STGCN model after training can better model in time and space. Compared with other solutions, this method can continuously and efficiently extract the changes in human body movements, and through the learning of a large amount of data, it can have stronger generalization ability and can be used in other environments. This method can not only accurately identify the suspected sleeping behavior of crew members, but also provide managers with a friendly monitoring platform and provide more comprehensive and worry-free monitoring information.

Claims

1. A method for identifying suspected sleep based on motion recognition, characterized in that: The following steps are involved: S1. Hard decode the camera through the streaming program to obtain real-time image frames; S2, detecting the human body in the real-time image frame by Yolov8 algorithm, if no human body frame is detected, returning to step S1; otherwise, proceeding to step S3; S3, sending the human body frame detected by the Yolov8 algorithm to the ByteTrack algorithm for tracking. If no target is tracked, return to step S1; otherwise, obtain the tracking result and enter step S4; S4, cropping the tracking results into a single human body image and scaling it to a uniform size, and sending it to the HRNet algorithm for human skeleton key point prediction to obtain a set of human skeleton key point coordinates; S5, taking the human skeleton key points of the same target extracted from the same real-time image frame as a sequence of the target, and continuously acquiring N sequences of the same target; S6. Input N sequences of the same target into the trained and improved STGCN model for action recognition and generate suspected sleeping recognition results.

2. The method for identifying suspected sleep based on motion recognition according to claim 1, characterized in that: In step S1 , real-time image frames are acquired at a speed of 10 frames per second.

3. The method for identifying suspected sleep based on motion recognition according to claim 1, characterized in that: In step S4, the width and height of the uniform size are 192 and 256 pixels respectively.

4. The method for identifying suspected sleep based on motion recognition according to claim 1, characterized in that: The key points of the human skeleton include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and are denoted as Q = {q i (x,y)│i=1,2,…,17}, where Q represents the coordinate set of the key points of the human skeleton, q i (x,y) is the coordinate of the i-th human skeleton key point.

5. The method for identifying suspected sleep based on motion recognition according to claim 1, characterized in that: The STGCN model includes a BN layer, a feature extraction module, a pooling layer, and a fully connected layer connected in sequence; among them: The BN layer is used to standardize the data input into the STGCN model and output the standardized data; Feature extraction module, used to extract time series features and spatial features of standardized data; The pooling layer is used to pool the time series features and spatial features of the standardized data; The fully connected layer is used to perform action recognition on the output of the pooling layer and generate suspected sleeping recognition results.

6. The method for identifying suspected sleep based on motion recognition according to claim 5, characterized in that: The feature extraction module includes a first ATT module, a first DGCN module, a first TCN module, a first MATT module, a second ATT module, a second DGCN module, a second TCN module and a second MATT module connected in sequence.

7. The method for identifying suspected sleep based on motion recognition according to claim 5, characterized in that: The action recognition result of the fully connected layer is the probability value of sleeping. When the probability value of sleeping recognized by the fully connected layer is greater than 0.9, the final recognition result is suspected sleeping; when the probability value of sleeping recognized by the fully connected layer is less than or equal to 0.9, the final recognition result is normal.

8. The method for identifying suspected sleep based on motion recognition according to claim 7, characterized in that: When the final recognition result is suspected sleeping, based on the time of obtaining the recognition result, the streaming media is used to push back 3 minutes and forward 2 minutes to form a 5-minute video to push sleeping warning data.

Images