Personnel hand-held mobile phone identification method based on video feature judgment

Abstract

The invention relates to the technical field of computer vision, intelligent monitoring and behavior analysis, in particular to a method for identifying a person holding a mobile phone based on video feature judgment. The method comprises the following steps: detecting and tracking a person, and performing identity recognition; analyzing the hand area of the person, and judging whether the person has the behavior of holding the mobile phone or not; when a detector holds the mobile phone, starting a timer to continuously track the time length of the mobile phone holding behavior; through the above mode, accurate identification and continuous tracking of the behavior of holding a mobile phone by a person can be realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, intelligent monitoring and behavior analysis, and in particular to a method for identifying people holding mobile phones based on video feature judgment. Background Technology

[0002] In applications such as security monitoring, public place management, and behavior analysis, accurately identifying whether a person is holding a mobile phone is of great significance. Traditional mobile phone usage detection methods mainly rely on manual monitoring or simple motion detection technology.

[0003] However, existing mobile phone detection methods are unable to accurately identify and continuously track people holding mobile phones in complex scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method for identifying people holding mobile phones based on video feature judgment, which aims to solve the technical problem that existing mobile phone detection methods cannot accurately identify and continuously track people holding mobile phones in complex scenarios.

[0005] To achieve the above objectives, the present invention employs a method for identifying people holding mobile phones based on video feature judgment, comprising the following steps:

[0006] Personnel are monitored, tracked, and identified.

[0007] Analyze the area around the person's hands to determine if they are holding a mobile phone.

[0008] When the inspector holds the phone, a timer is started to continuously track the duration of the phone-holding behavior.

[0009] Among these steps are personnel detection, tracking, and identification:

[0010] Using video processing technology, the camera captures real-time images at a frequency of at least 5 frames per second across the entire image.

[0011] Preprocess the acquired images;

[0012] Utilize deep learning models to detect people in images and continuously track their movement trajectories;

[0013] After detecting personnel, further personnel characteristics are extracted;

[0014] Perform cosine similarity calculations with a pre-calculated and pre-saved database of staff features.

[0015] In the step of analyzing the personnel's hand area to determine whether the personnel are holding a mobile phone:

[0016] Recognize the shape and color of the mobile phone held by the person;

[0017] The way analysts hold their phones.

[0018] The process involves identifying whether a person is a staff member or a stranger based on a similarity score and a pre-set threshold. If the person is identified as a stranger, an alarm mechanism is triggered.

[0019] The system continuously judges the behavior of holding a mobile phone based on consecutive frames to ensure the continuity and accuracy of the behavior. If no handheld mobile phone behavior is detected within a set time, the timer stops.

[0020] Based on a preset time threshold, it is determined whether the act of holding a mobile phone has exceeded the time limit. If it does, an early warning mechanism is triggered to notify relevant personnel to handle the situation.

[0021] Among the steps involved in preprocessing the acquired images:

[0022] Image quality can be improved by denoising, and possible errors in subsequent processing can be reduced by rotation.

[0023] In the step of using a deep learning model to detect people in the image and continuously track their movement trajectory:

[0024] The system is based on a self-trained deep learning model, which includes an input segment, a backbone network, a neck network, and a detection head. It consists of multiple convolutional neural network layers and is trained using a mixture of publicly available network datasets and real-world scene datasets. The total dataset exceeds 100,000 images. During training, a subset training method is used to gradually expand the model from a small number to a larger number to obtain stage models. In the training process, subsequent training is continuously adjusted based on the problems shown by the stage models to obtain the optimal model.

[0025] Among the steps, after detecting personnel, the next step is to extract personnel features:

[0026] Personnel characteristics include facial features, clothing color, and body shape. Facial features were extracted using the FaceNet network, including key facial features such as eyes, nose, and mouth. Clothing color features were extracted using color histograms and average color values. Body shape features were extracted using the Canny operator to detect contours and calculate gradients and integrals to obtain body shape characteristics.

[0027] In the step of calculating cosine similarity with the pre-calculated and pre-saved staff feature database:

[0028] The system uses similarity scores and pre-set thresholds to distinguish whether the person is a staff member or a stranger. If the person is identified as a stranger, an alarm mechanism is triggered.

[0029] Among the steps, the following steps involve recognizing the shape and color of the mobile phone held by a person:

[0030] The hand contour features are obtained by detecting the hand contour using the Canny operator and calculating the gradient and integral.

[0031] The hand color characteristics are obtained by calculating the color histogram and average color value;

[0032] Texture features are extracted by calculating the gray-level co-occurrence matrix of the hand region;

[0033] The system performs cosine similarity calculations with preset mobile phone templates, and determines whether a mobile phone exists based on the similarity score and a pre-set threshold.

[0034] Among the steps involved in analyzing the posture of the person holding the phone:

[0035] The IOU (Interval of Units) is used to calculate coordinates and analyze whether there is an overlap between the positions of the hand and the phone.

[0036] The hand contour obtained using the Canny operator;

[0037] ORB is used to calculate the positions of key points of the hand based on the hand contour, and the positions of each key point are connected to obtain the hand pose;

[0038] The system calculates cosine similarity with a preset mobile phone holding posture template, and determines whether a person is holding a mobile phone based on the similarity score and a preset threshold.

[0039] This invention discloses a method for identifying people holding mobile phones based on video feature judgment. It detects and tracks people, performs identity recognition, distinguishes between staff and strangers, and achieves intelligent control over mobile phone holding behavior. By analyzing the person's hand area to determine whether the person is holding a mobile phone, and starting a timer to continuously track the duration of the hand-holding behavior when it is detected, the method achieves accurate detection and continuous timing of people holding mobile phones, significantly improving the accuracy and intelligence of monitoring. Through this method, it enables accurate identification and continuous tracking of people holding mobile phones. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1This is a flowchart of the steps of the method for identifying people holding mobile phones based on video feature judgment according to the present invention.

[0042] Figure 2 This is a flowchart of steps S100 of the present invention.

[0043] Figure 3 This is a flowchart of steps S200 of the present invention.

[0044] Figure 4 This is a flowchart of steps S300 of the present invention.

[0045] Figure 5 This is a flowchart of steps S201 of the present invention.

[0046] Figure 6 This is a flowchart of steps S202 of the present invention. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0048] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0049] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0050] Please see Figures 1-6 This invention provides a method for identifying people holding mobile phones based on video feature judgment, comprising the following steps:

[0051] S100: Personnel detection and tracking, and identification.

[0052] In this embodiment, personnel detection and tracking, as well as identity verification, are performed. The specific process is as follows:

[0053] S101: Through video processing technology, it captures real-time camera footage at a frequency of at least 5 frames per second;

[0054] S102: Preprocess the acquired images;

[0055] S103: Utilize a deep learning model to detect people in the scene and continuously track their movement trajectory;

[0056] S104: After detecting personnel, further extract personnel characteristics;

[0057] S105: Calculate cosine similarity with the pre-calculated and pre-saved staff feature database.

[0058] In the aforementioned process, firstly, video processing technology is used to capture real-time images from the camera at a frequency of at least 5 frames per second. Then, the captured images undergo preprocessing, including image denoising to improve image quality and rotation to reduce potential errors in subsequent processing. Next, a deep learning model is used to detect people in the image and continuously track their motion trajectories. The system is based on a self-trained deep learning model, which includes an input segment, a backbone network, a neck network, and a detection head. This model consists of multiple convolutional neural network layers and is trained using a mixture of publicly available network datasets and real-world scene datasets. The total dataset exceeds 100,000 images. During training, a subset training method is used to gradually expand the model from a small set to a larger set, resulting in phased models. The training process is then used to demonstrate the performance of each phased model. The model is continuously adjusted during subsequent training to obtain the optimal model. After detecting a person, its features are further extracted, including facial features, clothing color, and body shape. Facial features are extracted using the FaceNet network, including key facial features such as eyes, nose, and mouth. Clothing color features are extracted using color histograms and average color values. Body shape features are extracted using the Canny operator to detect contours and calculate gradients and integrals to obtain body shape features. Then, cosine similarity is calculated with a pre-calculated and pre-saved database of staff features. Based on the similarity score and a pre-set threshold, it is determined whether the current person is a staff member or a stranger. If a stranger is identified, an alarm mechanism is triggered.

[0059] S200: Analyzes the area around a person's hands to determine if the person is holding a mobile phone.

[0060] In this embodiment, the analysis of the person's hand area determines whether the person is holding a mobile phone. The specific process is as follows:

[0061] S201: Recognize the shape and color of a person's handheld mobile phone;

[0062] S202: The way analysts hold their phones.

[0063] In the above process, the shape and color of the mobile phone held by the person are identified, and then the posture of the person holding the mobile phone is analyzed.

[0064] S300: When the inspector holds the mobile phone, a timer is started to continuously track the duration of the handheld mobile phone behavior.

[0065] In this embodiment, when the testing personnel hold the mobile phone, a timer is started to continuously track the duration of the handheld mobile phone behavior. The specific process is as follows:

[0066] S301: Continuously determine the behavior of the handheld mobile phone based on consecutive frames to ensure the continuity and accuracy of the behavior. If no handheld mobile phone behavior is detected within a set time, the timer stops.

[0067] S302: Based on a preset time threshold, determine whether the handheld mobile phone behavior has exceeded the time limit. If it has, trigger an early warning mechanism to notify relevant personnel to handle the situation.

[0068] During the above process, the system continuously judges the handheld phone behavior based on consecutive frames to ensure the continuity and accuracy of the behavior. If no handheld phone behavior is detected within the set time, the timer stops. Then, the system judges whether the handheld phone behavior has timed out according to the preset time threshold. If it has timed out, the system triggers the warning mechanism and notifies relevant personnel to handle the situation.

[0069] S201: Recognize the shape and color of a person's handheld mobile phone.

[0070] In this embodiment, the shape and color of the mobile phone held by the person are identified. The specific process is as follows:

[0071] S2011: Hand contour features are obtained by detecting the hand contour using the Canny operator and calculating the gradient and integral.

[0072] S2012: Hand color characteristics are derived through the calculation of color histogram and average color value;

[0073] S2013: Extract texture features by calculating the gray-level co-occurrence matrix of the hand region;

[0074] S2014: Perform cosine similarity calculation with the preset mobile phone template, and determine whether a mobile phone exists based on the similarity score and the preset threshold.

[0075] In the above process, the hand contour is detected by the Canny operator and the gradient and integral are calculated to obtain the hand contour features. Then, the hand color features are obtained by calculating the color histogram and the average color value. The texture features are then extracted by calculating the gray-level co-occurrence matrix of the hand region. Subsequently, the cosine similarity is calculated with the preset mobile phone template. Based on the similarity score and the preset threshold, it is determined whether a mobile phone exists.

[0076] S202: The way analysts hold their phones.

[0077] In this embodiment, the specific process of the analyst holding the mobile phone is as follows:

[0078] S2021: Analyze whether there is an overlap between the hand and the phone position by calculating coordinates using IOU;

[0079] S2022: Hand contour obtained using the Canny operator;

[0080] S2023: Use ORB to calculate the positions of key points of the hand based on the hand contour, and connect the positions of each key point to obtain the hand pose;

[0081] S2024: Calculate the cosine similarity with the preset mobile phone holding posture template, and determine whether the person is holding a mobile phone based on the similarity score and the preset threshold.

[0082] In the above process, the system calculates coordinates using IOU to analyze whether the positions of the hand and the phone overlap. Then, the hand contour is obtained using the Canny operator, and ORB is used to calculate the positions of key points of the hand (including elbow, wrist, and finger joints) based on the hand contour. The positions of each key point are connected to obtain the hand posture. Then, cosine similarity is calculated with a preset phone holding posture template. Based on the similarity score and a preset threshold, the system determines whether the person is holding the phone. The system performs multiple confirmations based on continuous frames to reduce the possibility of false alarms, thus finally recognizing the action of holding the phone.

[0083] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0084] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for identifying people holding mobile phones based on video feature judgment, characterized in that, Includes the following steps: Personnel are monitored, tracked, and identified. Analyze the area around the person's hands to determine if they are holding a mobile phone. When the inspector holds the phone, a timer is started to continuously track the duration of the phone-holding behavior.

2. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 1, characterized in that, In the steps of personnel detection, tracking, and identification: Using video processing technology, the camera captures real-time images at a frequency of at least 5 frames per second across the entire image. Preprocess the acquired images; Utilize deep learning models to detect people in images and continuously track their movement trajectories; After detecting personnel, further personnel characteristics are extracted; Perform cosine similarity calculations with a pre-calculated and pre-saved database of staff features.

3. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 1, characterized in that, In the process of analyzing the area around a person's hands to determine whether they are holding a mobile phone: Recognize the shape and color of the mobile phone held by the person; The way analysts hold their phones.

4. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 1, characterized in that, The process involves identifying whether a person is a staff member or a stranger based on a similarity score and a pre-set threshold. If the person is identified as a stranger, an alarm mechanism is triggered. The system continuously judges the behavior of holding a mobile phone based on consecutive frames to ensure the continuity and accuracy of the behavior. If no handheld mobile phone behavior is detected within a set time, the timer stops. Based on a preset time threshold, it is determined whether the act of holding a mobile phone has exceeded the time limit. If it does, an early warning mechanism is triggered to notify relevant personnel to handle the situation.

5. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 2, characterized in that, In the preprocessing steps of the acquired images: Image quality can be improved by denoising, and possible errors in subsequent processing can be reduced by rotation.

6. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 5, characterized in that, In the process of using a deep learning model to detect people in an image and continuously track their movement trajectory: The system is based on a self-trained deep learning model, which includes an input segment, a backbone network, a neck network, and a detection head. It consists of multiple convolutional neural network layers and is trained using a mixture of publicly available network datasets and real-world scene datasets. The total dataset exceeds 100,000 images. During training, a subset training method is used to gradually expand the model from a small number to a larger number to obtain stage models. In the training process, subsequent training is continuously adjusted based on the problems shown by the stage models to obtain the optimal model.

7. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 6, characterized in that, After detecting personnel, the next step is to extract their characteristics: Personnel characteristics include facial features, clothing color, and body shape. Facial features were extracted using the FaceNet network, including key facial features such as eyes, nose, and mouth. Clothing color features were extracted using color histograms and average color values. Body shape features were extracted using the Canny operator to detect contours and calculate gradients and integrals to obtain body shape characteristics.

8. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 7, characterized in that, In the step of calculating cosine similarity with a pre-computed and pre-saved database of staff features: The system uses similarity scores and pre-set thresholds to distinguish whether the person is a staff member or a stranger. If the person is identified as a stranger, an alarm mechanism is triggered.

9. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 1, characterized in that, In the step of recognizing the shape and color of a person's handheld mobile phone: The hand contour features are obtained by detecting the hand contour using the Canny operator and calculating the gradient and integral. The hand color characteristics are obtained by calculating the color histogram and average color value; Texture features are extracted by calculating the gray-level co-occurrence matrix of the hand region; The system performs cosine similarity calculations with preset mobile phone templates, and determines whether a mobile phone exists based on the similarity score and a pre-set threshold.

10. The method for identifying a person holding a mobile phone based on video feature judgment as described in claim 9, characterized in that, In the steps of analyzing the posture of the person holding the phone: The IOU (Interval of Units) is used to calculate coordinates and analyze whether there is an overlap between the positions of the hand and the phone. The hand contour obtained using the Canny operator; ORB is used to calculate the positions of key points of the hand based on the hand contour, and the positions of each key point are connected to obtain the hand pose; The system calculates cosine similarity with a preset mobile phone holding posture template, and determines whether a person is holding a mobile phone based on the similarity score and a preset threshold.

Images