Feature pyramid-based suspected sleep identification method
Through the identification method of feature pyramids, combined with Yolov8 and ByteTrack algorithms, the image differential method is used to identify sleeping behavior, which solves the sleep monitoring problem of ship duty personnel, and achieves efficient and accurate sleep recognition, adapts to complex environments, and reduces the risk of accidents.
Patent Information
- Application Number
- CN202410996015.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
Ship duty personnel find it difficult to monitor sleeping behavior in real time during navigation, resulting in an increase in the risk of potential collision accidents. It is difficult for the existing technology to effectively identify sleeping behavior, especially in complex environments.
A suspected sleep recognition method based on feature pyramids is used, human body detection is performed through the Yolov8 algorithm, the ByteTrack algorithm is tracked, the image difference method is used to calculate the motion change rate, and the motion change rate of multiple slicing methods is used to determine the suspected sleep behavior.
It realizes efficient and accurate recognition of sleeping behavior in various environments, reduces misjudgment, adapts to different scenarios, and can seamlessly integrate with existing monitoring systems, improving recognition accuracy and robustness.
Smart Images

Figure CN118942154A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of sleeping behavior recognition, and in particular to a suspected sleeping recognition method based on a feature pyramid. Background Art
[0002] During the voyage of a ship, the staff on duty need to observe the sea area at all times to ensure the safety of the ship. If the staff falls asleep during the duty time, once an obstacle appears in front, a serious accident may occur, causing harm to the staff on board and significant economic losses to the shipping company. Therefore, the risk of collision can be reduced by identifying the sleeping staff on board. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a suspected sleeping identification method based on feature pyramid, which can monitor the sleeping of the target person in real time and ensure safe operation.
[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:
[0005] A method for identifying suspected sleeping based on a feature pyramid is provided, which comprises the following steps:
[0006] S1, obtain real-time images;
[0007] S2, perform human body detection on the real-time image using the Yolov8 algorithm. If a human body is detected, add a human body frame and proceed to step S3; otherwise, return to step S1;
[0008] S3, tracking the human frame by ByteTrack algorithm, if the human target is not tracked, returning to step S1; if the human target is tracked, assigning the same tracking ID to the same target and proceeding to step S4;
[0009] S4. For each tracking ID, save the tracking results of the past K frames, so that each tracking ID corresponds to a basic data set for sleeping recognition;
[0010] S5, converting the frame images in the sleeping recognition basic data set into grayscale images after unifying the size to obtain a grayscale image set;
[0011] S6, dividing the same grayscale image into 4 regions, 16 regions and 64 regions respectively;
[0012] S7, for the grayscale image before segmentation and the regional image under the same segmentation method, respectively calculate the change matrix of the previous and next frames by image difference method, and obtain the motion change rate corresponding to different segmentation methods;
[0013] S8, taking the average of the motion change rates of the four different segmentation methods corresponding to the same frame image as the final motion change rate of the frame image; wherein the motion change rate corresponding to the grayscale image before segmentation is the motion change rate corresponding to one segmentation method;
[0014] S9. The variance of the final motion change rate of the K-frame tracking results of the same tracking ID is used as an identification parameter. If the identification parameter is less than a threshold, it is determined that the tracking ID has a suspected sleeping behavior; otherwise, it is determined that the tracking ID has not had a sleeping behavior.
[0015] Furthermore, the specific method for obtaining the real-time image is:
[0016] The camera is hard decoded through the streaming program to obtain real-time images at 10 frames per second.
[0017] Further, the value of K is 900.
[0018] Furthermore, the frame image size in the sleeping recognition basic dataset is unified as 240×240.
[0019] Furthermore, for the grayscale image before segmentation, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0020] D 1,1 =|p k ―p k―1 |
[0021] B 1,1 =binarize(D 1,1 )
[0022] A 1,1 =contour_area(B 1,1 )
[0023]
[0024] Where D 1,1 Represents the k-th grayscale image p before segmentation k and the k-1th frame grayscale image p before segmentation k―1 The difference result of B is the change matrix; 1,1 is the binarization result; binarize(.) is the binarization calculation function of opencv; A 1,1 is the contour area of the changing region; contour_area(.) is the contour calculation function of opencv; p k,1 is the motion change rate of the k-th frame image.
[0025] Furthermore, for the regional image divided into 4 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0026] D 2,2 (i4)=|p k (i4)―p k―1 (i4)|i4=1,2,3,4
[0027] B 2,2 (i4)=binarize(D 2,2 (i4))
[0028] A 2,2 (i4) = contour_area(B 2,2 (i4))
[0029]
[0030]
[0031] Where D 2,2 (i4) indicates that when the segmentation method is 4 regions, the grayscale image p of the kth frame k The i4th region image and the k-1th frame grayscale image p k―1 The difference result of the image of the i4th region in B is the change matrix; 2,2 (i4) is the binarization result of the i4th region image when the segmentation method is 4 regions; A 2,2 (i4) is the contour area of the changed region of the i4th region image when the segmentation method is 4 regions; p k,2 (i4) is the motion change rate of the image in the i4th region when the segmentation method is 4 regions; p k,2 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 4 regions.
[0032] Furthermore, for the regional image divided into 16 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0033] D 4,4 (i 16 )=|p k (i 16 )―p k―1 (i 16 )|i 16 =1,2,…,16
[0034] B 4,4 (i 16 )=binarize(D 4,4 (i16 ))
[0035] A 4,4 (i 16 )=contour_area(B 4,4 (i 16 ))
[0036]
[0037]
[0038] Where D 4,4 (i 16 ) indicates that when the segmentation method is 16 regions, the grayscale image p of the kth frame k No. i 16 The region image and the k-1th frame grayscale image p k―1 No. i 16 The difference result of the regional image is the change matrix; B 4,4 (i 16 ) is divided into 16 regions, the i-th 16 The binarization result of the region image; A 4,4 (i 16 ) is divided into 16 regions, the i-th 16 The contour area of the changing region of the region image; p k,3 (i 16 ) is divided into 16 regions, the i-th 16 The motion change rate of the image in the region; p k,3 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 16 regions.
[0039] Furthermore, for the regional image divided into 64 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0040] D 8,8 (i 64 )=|p k (i 64 )―p k―1 (i 64 )|i 64 =1,2,…,64
[0041] B 8,8 (i 64 )=binarize(d 8,8 (i 64 ))
[0042] A 8,8 (i 64)=contour_area(B 8,8 (i 64 ))
[0043]
[0044]
[0045] Where D 8,8 (i 64 ) indicates that when the segmentation method is 64 regions, the grayscale image p of the kth frame k No. i 64 The region image and the k-1th frame grayscale image p k―1 No. i 64 The difference result of the regional image is the change matrix; B 8,8 (i 64 ) is divided into 64 regions, the i-th 64 The binarization result of the region image; A 8,8 (i 64 ) is divided into 64 regions, the i-th 64 The contour area of the changing region of the region image; p k,4 (i 64 ) is divided into 64 regions, the i-th 64 The motion change rate of the image in the region; p k,4 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 64 regions.
[0046] Furthermore, the calculation expression for obtaining the final motion change rate of a frame of image is:
[0047]
[0048] where p avgk is the final motion change rate of the k-th frame image.
[0049] Furthermore, the expression of taking the average value of the final motion change rate of the K-frame tracking results of the same tracking ID as the identification parameter is:
[0050]
[0051]
[0052] Where P a is the average value of the final motion change rate of the K-frame tracking results of the same tracking ID; value is the variance of the final motion change rate of the K-frame tracking results of the same tracking ID; the threshold in step S9 is 0.25.
[0053] The beneficial effects of the present invention are:
[0054] 1. This method can adapt to various environments and scenarios, and can effectively identify sleep both indoors and outdoors; it can be seamlessly integrated with existing monitoring systems and intelligent analysis platforms to provide an enhanced behavior recognition capability and achieve efficient, accurate and robust sleep behavior recognition.
[0055] 2. This method converts the frame image into a grayscale image after unifying the size. It can process the image data without changing the overall shape of the human body frame, which is convenient for subsequent motion change rate calculation. It can also reduce the influence of factors such as light and improve the recognition accuracy of this method.
[0056] 3. This method models the global to local features of human motion and uses four segmentation methods. As the area is continuously subdivided, more detailed motion features can be captured. Combined with the previous global features, it can not only directly exclude sleeping behavior, but also capture small movements of the human body to avoid being misjudged as sleeping behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 Schematic diagram of the process of this method. DETAILED DESCRIPTION
[0058] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0059] like Figure 1 As shown, the suspected sleeping recognition method based on feature pyramid includes the following steps:
[0060] S1, obtain real-time images;
[0061] S2, perform human body detection on the real-time image using the Yolov8 algorithm. If a human body is detected, add a human body frame and proceed to step S3; otherwise, return to step S1;
[0062] S3, tracking the human frame by ByteTrack algorithm, if the human target is not tracked, returning to step S1; if the human target is tracked, assigning the same tracking ID to the same target and proceeding to step S4;
[0063] S4. For each tracking ID, save the tracking results of the past K frames, so that each tracking ID corresponds to a basic data set for sleeping recognition;
[0064] S5, converting the frame images in the sleeping recognition basic data set into grayscale images after unifying the size to obtain a grayscale image set;
[0065] S6, dividing the same grayscale image into 4 regions, 16 regions and 64 regions respectively;
[0066] S7, for the grayscale image before segmentation and the regional image under the same segmentation method, respectively calculate the change matrix of the previous and next frames by image difference method, and obtain the motion change rate corresponding to the different segmentation methods;
[0067] S8, taking the average of the motion change rates of the four different segmentation methods corresponding to the same frame image as the final motion change rate of the frame image; wherein the motion change rate corresponding to the grayscale image before segmentation is the motion change rate corresponding to one segmentation method;
[0068] S9. The variance of the final motion change rate of the K-frame tracking results of the same tracking ID is used as an identification parameter. If the identification parameter is less than a threshold, it is determined that the tracking ID has a suspected sleeping behavior; otherwise, it is determined that the tracking ID has not had a sleeping behavior.
[0069] The specific method of obtaining real-time images is: hard decoding the camera through the streaming program, and obtaining real-time images in a manner of 10 frames per second.
[0070] In this embodiment, the value of K is 900. The frame image size in the sleeping recognition basic data set is unified into 240×240 by using the ratio of width to height.
[0071] For the grayscale image before segmentation, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0072] D 1,1 =|p k ―p k―1 |
[0073] B 1,1 =binarize(D 1,1 )
[0074] A 1,1 =contour_area(B 1,1 )
[0075]
[0076] Where D 1,1 Represents the k-th grayscale image p before segmentation k and the k-1th frame grayscale image p before segmentation k―1 The difference result of B is the change matrix; 1,1is the binarization result; binarize(.) is the binarization calculation function of opencv; A 1,1 is the contour area of the changing region; contour_area(.) is the contour calculation function of opencv; p k,1 is the motion change rate of the k-th frame image.
[0077] For the regional image divided into 4 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0078] D 2,2 (i4)=|p k (i4)―p k―1 (i4)|i4=1,2,3,4
[0079] B 2,2 (i4)=binarize(D 2,2 (i4))
[0080] A 2,2 (i4) = contour_area(B 2,2 (i4))
[0081]
[0082]
[0083] Where D 2,2 (i4) indicates that when the segmentation method is 4 regions, the grayscale image p of the kth frame k The i4th region image and the k-1th frame grayscale image p k―1 The difference result of the image of the i4th region in B is the change matrix; 2,2 (i4) is the binarization result of the i4th region image when the segmentation method is 4 regions. Binarization is used to highlight the contour of the moving image. A 2,2 (i4) is the contour area of the changed region of the i4th region image when the segmentation method is 4 regions; p k,2 (i4) is the motion change rate of the i4th region image when the segmentation method is 4 regions, which is used to measure the proportion of the motion region in the single region image; p k,2 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 4 regions.
[0084] For the regional image divided into 16 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0085] D 4,4 (i 16 )=|pk (i 16 )―p k―1 (i 16 )|i 16 =1,2,…,16
[0086] B 4,4 (i 16 )=binarize(D 4,4 (i 16 ))
[0087] A 4,4 (i 16 )=contour_area(B 4,4 (i 16 ))
[0088]
[0089]
[0090] Where D 4,4 (i 16 ) indicates that when the segmentation method is 16 regions, the grayscale image p of the kth frame k No. i 16 The region image and the k-1th frame grayscale image p k―1 No. i 16 The difference result of the regional image is the change matrix; B 4,4 (i 16 ) is divided into 16 regions, the i-th 16 The binarization result of the region image; A 4,4 (i 16 ) is divided into 16 regions, the i-th 16 The contour area of the changing region of the region image; p k,3 (i 16 ) is divided into 16 regions, the i-th 16 The motion change rate of the image in the region; p k,3 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 16 regions.
[0091] For the regional image divided into 64 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows:
[0092] D 8,8 (i 64 )=|p k (i 64 )―p k―1 (i 64 )|i 64=1,2,…,64
[0093] B 8,8 (i 64 )=binarize(D 8,8 (i 64 ))
[0094] A 8,8 (i 64 )=contour_area(B 8,8 (i 64 ))
[0095]
[0096]
[0097] Where D 8,8 (i 64 ) indicates that when the segmentation method is 64 regions, the grayscale image p of the kth frame k No. i 64 The region image and the k-1th frame grayscale image p k―1 No. i 64 The difference result of the regional image is the change matrix; B 8,8 (i 64 ) is divided into 64 regions, the i-th 64 The binarization result of the region image; A 8,8 (i 64 ) is divided into 64 regions, the i-th 64 The contour area of the changing region of the region image; p k,4 (i 64 ) is divided into 64 regions, the i-th 64 The motion change rate of the image in the region; p k,4 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 64 regions.
[0098] The calculation expression for obtaining the final motion change rate of a frame of image is:
[0099]
[0100] where p avgk is the final motion change rate of the k-th frame image.
[0101] The expression of taking the average value of the final motion change rate of the K-frame tracking results of the same tracking ID as the identification parameter is:
[0102]
[0103]
[0104] Where P a is the average value of the final motion change rate of the K-frame tracking results of the same tracking ID; value is the variance of the final motion change rate of the K-frame tracking results of the same tracking ID; the threshold in step S9 is 0.25.
[0105] In the specific implementation process, it takes 90 seconds to obtain 900 frames of images using the method of 10 frames per second, but not every frame of the 900 frames can identify the human body (the person moves out of the camera range). Therefore, when a person is identified as suspected of sleeping, the suspected sleeping time is greater than or equal to 90 seconds. To be on the safe side, a 5-minute video can be formed by streaming media by pushing back 3 minutes and forward 2 minutes, and the warning information is pushed to the alarm end, which will perform subsequent processing.
[0106] In this embodiment, the motion change rate obtained from the unsegmented image is a global feature. The global feature is a large-amplitude motion feature, which can directly exclude sleeping behavior. However, if the human body has a small-amplitude motion, the global feature cannot be captured, which can easily lead to misjudgment. This method introduces local tiny features (motion change rates obtained by different segmentation methods) in a pyramidal manner to supplement it, and can effectively identify non-moving features (such as occlusion). For features of small-scale motion (such as playing with a mobile phone), the motion amplitude is small except for the hands. Capturing such local features can avoid misjudgment as sleeping behavior. Therefore, this method has a good recognition ability for sleeping behavior.
[0107] In summary, the present invention can adapt to various environments and scenarios, and can effectively perform sleep recognition no matter indoors or outdoors.
Claims
1. A method for identifying suspected sleeping based on feature pyramid, characterized in that: The following steps are involved: S1, obtain real-time images; S2, perform human body detection on the real-time image using the Yolov8 algorithm. If a human body is detected, add a human body frame and proceed to step S3; otherwise, return to step S1; S3, tracking the human frame by ByteTrack algorithm, if the human target is not tracked, returning to step S1; if the human target is tracked, assigning the same target the same tracking ID and proceeding to step S4; S4. For each tracking ID, save the tracking results of the past K frames, so that each tracking ID corresponds to a basic data set for sleeping recognition; S5, converting the frame images in the sleeping recognition basic data set into grayscale images after unifying the size to obtain a grayscale image set; S6, dividing the same grayscale image into 4 regions, 16 regions and 64 regions respectively; S7, for the grayscale image before segmentation and the regional image under the same segmentation method, respectively calculate the change matrix of the previous and next frames by image difference method, and obtain the motion change rate corresponding to the different segmentation methods; S8, taking the average of the motion change rates of the four different segmentation methods corresponding to the same frame image as the final motion change rate of the frame image; wherein the motion change rate corresponding to the grayscale image before segmentation is the motion change rate corresponding to one segmentation method; S9, taking the variance of the final motion change rate of the K-frame tracking results of the same tracking ID as an identification parameter, if the identification parameter is less than a threshold, it is determined that the tracking ID has a suspected sleeping behavior; Otherwise, it is determined that the tracking ID has no sleeping behavior.
2. The method for identifying suspected sleeping status based on feature pyramid according to claim 1, characterized in that: The specific method to obtain real-time images is: The camera is hard decoded through the streaming program to obtain real-time images at 10 frames per second.
3. The method for identifying suspected sleeping status based on feature pyramid according to claim 2, characterized in that: The value of K is 900.
4. The method for identifying suspected sleeping status based on feature pyramid according to claim 3, characterized in that: The frame image size in the sleeping recognition basic dataset is unified as 240×240.
5. The method for identifying suspected sleeping status based on feature pyramid according to claim 4, characterized in that: For the grayscale image before segmentation, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows: D 1,1 =|p k ―p k―1 | B 1,1 =binarize(D 1,1 ) A 1,1 =contour_area(B 1,1 ) Where D 1,1 Represents the k-th grayscale image p before segmentation k and the k-1th frame grayscale image p before segmentation k―1 The difference result of B is the change matrix; 1,1 is the binarization result; binarize(.) is the binarization calculation function of opencv; A 1,1 is the contour area of the changing region; contour_area(.) is the contour calculation function of opencv; p k,1 is the motion change rate of the k-th frame image.
6. The method for identifying suspected sleeping status based on feature pyramid according to claim 5, characterized in that: For the regional image divided into 4 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows: D 2,2 (i4)=|p k (i4)―p k―1 (i4)|i4=1,2,3,4 B 2,2 (i4)=binarize(D 2,2 (i4)) A 2,2 (i4)=contour_area(B 2,2 (i4)) Where D 2,2 (i4) indicates that when the segmentation method is 4 regions, the grayscale image p of the kth frame k The i4th region image and the k-1th frame grayscale image p k―1 The difference result of the image of the i4th region in B is the change matrix; 2,2 (i4) is the binarization result of the i4th region image when the segmentation method is 4 regions; A 2,2 (i4) is the contour area of the changed region of the i4th region image when the segmentation method is 4 regions; p k,2 (i4) is the motion change rate of the image in the i4th region when the segmentation method is 4 regions; p k,2 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 4 regions.
7. The method for identifying suspected sleeping status based on feature pyramid according to claim 6, characterized in that: For the regional image divided into 16 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows: D 4,4 (i 16 )=|p k (i 16 )―p k―1 (i 16 )|i 16 =1,2,…,16 B 4,4 (i 16 )=binarize(D 4,4 (i 16 )) A 4,4 (i 16 )=contour_area(B 4,4 (i 16 )) Where D 4,4 (i 16 ) indicates that when the segmentation method is 16 regions, the grayscale image p of the kth frame k No. i 16 The region image and the k-1th frame grayscale image p k―1 No. i 16 The difference result of the regional image is the change matrix; B 4,4 (i 16 ) is divided into 16 regions, the i-th 16 The binarization result of the region image; A 4,4 (i 16 ) is divided into 16 regions, the i-th 16 The contour area of the changing region of the region image; p k,3 (i 16 ) is divided into 16 regions, the i-th 16 The motion change rate of the image in the region; p k,3 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 16 regions.
8. The method for identifying suspected sleeping status based on feature pyramid according to claim 7, characterized in that: For the regional image divided into 64 regions, the change matrix of the previous and next frames is calculated by the image difference method, and the expression of the corresponding motion change rate is obtained as follows: D 8,8 (i 64 )=|p k (i 64 )―p k―1 (i 64 )|i 64 =1,2,…,64 B 8,8 (i 64 )=binarize(D 8,8 (i 64 )) A 8,8 (i 64 )=contour_area(B 8,8 (i 64 )) Where D 8,8 (i 64 ) indicates that when the segmentation method is 64 regions, the grayscale image p of the kth frame k No. i 64 The region image and the k-1th frame grayscale image p k―1 No. i 64 The difference result of the regional image is the change matrix; B 8,8 (i 64 ) is divided into 64 regions, the i-th 64 The binarization result of the region image; A 8,8 (i 64 ) is divided into 64 regions, the i-th 64 The contour area of the changing region of the region image; p k,4 (i 64 ) is divided into 64 regions, the i-th 64 The motion change rate of the image in the region; p k,4 It is the motion change rate corresponding to the k-th frame grayscale image when the segmentation method is 64 regions.
9. The method for identifying suspected sleeping persons based on feature pyramid according to claim 8, characterized in that: The calculation expression for obtaining the final motion change rate of a frame of image is: where p avgk is the final motion change rate of the k-th frame image.
10. The method for identifying suspected sleeping status based on feature pyramid according to claim 9, characterized in that: The expression of taking the average value of the final motion change rate of the K-frame tracking results of the same tracking ID as the identification parameter is: Where P a is the average value of the final motion change rate of the K-frame tracking results of the same tracking ID; value is the variance of the final motion change rate of the K-frame tracking results of the same tracking ID; The threshold value in step S9 is 0.25.
Citation Information
Patent Citations
Video detection method and device, electronic equipment and storage medium
CN113569770A
Sleep detection method and device based on front-end camera
CN114299606A
Sleep behavior detection method and system
CN115346169A
Sleep behavior detection method and device and readable storage medium
CN117789241A
Posture Detection Apparatus, Posture Detection Method, and Sleeping Posture Determination Method
US20230368408A1
