A method for detecting illegal use of mobile phones

Abstract

The present invention relates to a method for detecting illegal use of mobile phones, and belongs to the field of information confidentiality protection technology. The present invention obtains pictures containing mobile phones in a public environment, simulates a scene where the use of mobile phones is prohibited, and respectively shoots videos of making and receiving calls on mobile phones; the collected materials are first automatically labeled using the YOLOv5 official model, and the data set is data enhanced; the enhanced data set is used to train the YOLOv7 target detection model and the VGG-16 classification model respectively; the camera video stream is obtained, and according to the frame skipping setting, a single frame image is obtained therein, and the image size is transformed to the YOLOv7 model input size. The position coordinates of the mobile phone in the image are obtained by training the YOLOv7 model. An image of the mobile phone area is obtained, and the image is classified using VGG-16 to determine whether the mobile phone is currently held; and whether the behavior is illegal. The present invention determines whether there is a violation based on the requirements for mobile phone control in the application site and combines detection and classification. The detection accuracy of the present invention is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of information confidentiality protection, and in particular relates to a method for detecting illegal use of a mobile phone. Background Art

[0002] With the increasing power and widespread use of mobile phones, information can be transmitted through phone calls, voice, video, and photos. Therefore, preventing information leaks caused by illegal mobile phone use has become a key priority for government, military, enterprises, and public institutions. In practice, monitoring personnel face dozens of surveillance screens, and the monotonous viewing can easily lead to fatigue, making it easy to miss violations. Even post-event inspections using surveillance videos require significant manpower and time. Preventing information leaks and promptly identifying potential leaks can expedite remediation and minimize the damage caused by such leaks.

[0003] In addition, different scenarios require different levels of standards for detecting mobile phones. For example, in important places, mobile phones are strictly prohibited. In places such as museums and art exhibitions, the use of mobile phones for taking photos is restricted, but carrying mobile phones is not restricted.

[0004] Existing methods to prevent information leakage caused by illegal use of mobile phones mainly include the following:

[0005] (1) Detecting mobile phone signals through analog signal base stations to detect mobile phones in communication mode. This method only prevents mobile phone communication functions and cannot detect illegal activities such as recording or taking photos with mobile phones. It is also susceptible to interference from other mobile phones in non-sensitive information areas. In addition, this method is expensive and takes up additional space.

[0006] (2) By having people walk through a special X-ray mobile phone detection gate to detect mobile phones, thus preventing mobile phones from entering sensitive areas. This solution also has the disadvantage of being expensive. In addition, placing the security gate outside the archive room and conference room door will destroy the original architectural design aesthetics and slow down the speed of people entering and exiting.

[0007] (3) Based on deep learning visual detection technology, this method can protect against information leakage caused by illegal use of mobile phones without any sense, but there is a problem that the mobile phone target is small and easily missed when obscured.

[0008] Moreover, current methods almost all rely on detecting whether a mobile phone exists in the picture. Depending on the strictness of the regulations for various scenarios, a more precise judgment is needed on whether a mobile phone is used or just exists in the scene. Summary of the Invention

[0009] (1) Technical issues to be resolved

[0010] The technical problem to be solved by the present invention is how to provide a method for detecting illegal use of mobile phones to solve the problem of fatigue omissions that easily occur in manual monitoring; the problem that the method of detecting mobile phone signals cannot prevent mobile phones from taking photos, recording, and storing sensitive information; the X-ray mobile phone detection door is expensive and affects the original environment; the method based on deep learning technology has poor detection effect when the mobile phone is partially blocked, and the detection result is unclear whether the person is using the mobile phone.

[0011] (2) Technical solution

[0012] In order to solve the above technical problems, the present invention proposes a method for detecting illegal use of mobile phones, which includes the following steps:

[0013] S1. Create a dataset

[0014] S11. Obtain images of various mobile phone categories as dataset 1 through public datasets or commercial datasets;

[0015] S12. Simulate a typical application scenario and organize personnel to shoot videos of making and receiving phone calls. Then capture pictures of the videos as Dataset 2.

[0016] S13, automatically annotate the image materials collected in Dataset 1 and Dataset 2 using the YOLOv5 model, annotate the mobile phone categories, and generate annotated Dataset 1 and Dataset 2, which will be used to train the YOLOv7 model;

[0017] S14. For the images labeled with the mobile phone category, expand the length and width of the annotation box by 1.2 times based on the center of the annotation box, crop the image, and classify the images into two categories: holding a mobile phone and not holding a mobile phone. This creates a classification dataset of images with and without mobile phones, which is referred to as Dataset 3.

[0018] S15, use rotation, symmetry, cropping, and noise addition to enhance the data, and use mosaic data enhancement to enhance the data;

[0019] S2. Training model

[0020] S21. Use the labeled dataset 1 and dataset 2 to train the YOLOv7 model for target detection.

[0021] S22, train the VGG-16 model using dataset 3 with and without mobile phones;

[0022] S3. Detecting mobile phone usage behavior

[0023] S31. Obtain the video stream to be detected, parse the video stream, obtain a single frame image according to the pre-set frame skipping parameters, and convert the size to the input size specified by the model;

[0024] S32. Input the image into the YOLOv7 model to detect the location of the mobile phone in the image, filter out detection results with a confidence level less than 0.5, and return the category, detection box coordinates, and confidence level.

[0025] S33. For a detection result where the detection category is mobile phone, obtain the coordinates of the mobile phone, proportionally expand the detection frame to 1.2 times the original value with the center point of the mobile phone detection frame as the center, convert the image size, and classify the image as holding a mobile phone or not holding a mobile phone using the VGG-16 model;

[0026] S4. Determine whether the behavior is illegal

[0027] After obtaining the classification results of all mobile phones in the picture, a logical OR operation is performed, that is, if a mobile phone is held once, it is considered that the current video frame is holding a mobile phone; if the classification result shows that a mobile phone is held, it is considered that the mobile phone is used; if the classification result does not show that a mobile phone is held, it is considered that a mobile phone is present; if no mobile phone is detected, it is considered that there is no mobile phone; based on the pre-set rules prohibiting the use of mobile phones or prohibiting the appearance of mobile phones, it is determined whether the behavior is in violation of the rules.

[0028] Furthermore, step S12 specifically includes: common environments where mobile phone use is prohibited include: indoor office areas, factory workshops and outdoor venues. A 4K high-definition camera is fixed on a tripod in each environment to simulate the picture conditions presented under the camera monitoring angle; the data set production takes into account the influence of various factors including: light intensity, mobile phone holding posture, traffic flow and mobile phone type. In the above environments, videos are shot considering various factors, and images with mobile phones in the picture are captured as data set 2.

[0029] Furthermore, step S13 specifically includes: performing target detection on dataset 1 and dataset 2 respectively using the YOLOv5 model yolov5l.pt, detecting the coordinates and categories of the mobile phones, generating a labeling file, and setting the parameters as follows: confidence conf-thres is 0.5, intersection-over-union threshold iou-thres is 0.45, and detection category classes is: 0,67; then correcting the labeling results through labeling software. If the labeling results are inaccurate, correct some of the inaccurately labeled pictures for training through transfer learning.

[0030] Furthermore, step S14 specifically includes: enlarging the detection frame of the pictures detected as mobile phone categories in data set 1 and data set 2 to 1.2 times the original size according to the center of the detection frame; if the enlarged area exceeds the image range at this time, the out-of-range area is supplemented with 0 for all three RGB channels, and the enlarged picture is intercepted and marked as holding a mobile phone and not holding a mobile phone, as data set 3.

[0031] Furthermore, the step S15 specifically includes: performing data enhancement by rotation, symmetry, cropping, and adding noise, for a total of 11 times, randomly extracting 5% of the pictures in the data set each time, and using an enhancement method, including: rotating 45°, rotating 90°, rotating 180°, left-right axis symmetry, upper-lower axis symmetry, center symmetry, cropping 20% ​​of the bottom picture, cropping 20% ​​of the left picture, cropping 20% ​​of the right picture, cropping 20% ​​of the top picture, adding Gaussian noise, and then recording the annotation of the picture after data enhancement; randomly extracting 20% ​​of the pictures in the data set and using a mosaic data enhancement method to increase the detection effect for small targets, each four pictures are grouped into a group, and the four pictures are spliced. Each picture has its corresponding frame. After splicing the four pictures, a new picture is obtained, and the frame corresponding to this picture is also obtained.

[0032] Furthermore, step S21 specifically includes: first, using data set 1 to train a YOLOv7 model for detecting mobile phones, with the training set and test set divided in a ratio of 8:2, 16 pictures as a batch, using the SGD optimizer method, training 200 rounds, and setting the values ​​of hyperparameters: the network depth parameter is 0.75, the network width parameter is 1, the learning rate learning_rate is 0.01, the stochastic gradient descent momentum parameter momentum is 0.99, and the weight decay is 0.00025; then, using data set 2 for transfer learning, using the YOLOv7 model trained in the previous step, fixing the network parameters of the first 50 layers of the model without changing, and then training on data set 2, still detecting one category of mobile phones, and the hyperparameter settings are the same as before.

[0033] Furthermore, step S22 specifically includes: using dataset 3 to train the VGG-16 model, with the training set and test set divided in a ratio of 8:2, 16 images as a batch, using the SGD optimizer method, training for 300 rounds, setting the hyperparameter momentum to 0.937, and the weight decay to 0.0005.

[0034] Furthermore, the step S31 specifically includes: parsing the RTSP video stream through opencv, using frame skipping mode, taking one frame out of every three frames for analysis, the parsed video frame is in BGR format, converting it to RGB format and using the letterbox method to scale the size to 640*480 pixels.

[0035] Furthermore, step S32 specifically includes: inputting the image scaled to 640*480 pixels into the YOLOv7 model for inference, returning the detected object category, coordinates, and confidence, detecting one category of mobile phones, and discarding detection results with a confidence level lower than 0.5.

[0036] Furthermore, step S33 specifically includes: for the detection result of the detection category being a mobile phone, obtaining the coordinates of the mobile phone, and proportionally expanding the detection frame to 1.2 times the original size according to the center of the detection frame. If the expanded area exceeds the image range at this time, the out-of-range area is supplemented with 0 for all three RGB channels, and the current image is captured. The captured image is converted to a size of 224*224 using the letterbox method, and is input into the VGG-16 model for classifying whether a mobile phone is held, and a classification result of holding a mobile phone or not is obtained.

[0037] (3) Beneficial effects

[0038] The present invention proposes a method for detecting illegal use of mobile phones. The present invention determines whether a violation has occurred based on the requirements for mobile phone control in the application site and combines detection and classification. The present invention utilizes machine vision technology and adopts deep learning methods to detect mobile phones and human actions in images, thereby determining whether illegal use of mobile phones has occurred. The present invention overcomes the problems of detecting mobile phone signals and X-ray mobile phone detection gates in the prior art through a series of image processing and the design of an automatic detection model. It also overcomes the problem of poor detection effect of deep learning technology methods. The present invention has a high detection accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the process of the present invention;

[0040] Figure 2 To create a dataset sub-flowchart;

[0041] Figure 3 This is the sub-flowchart for training the model;

[0042] Figure 4 To detect the use of mobile phone behavior sub-flowchart;

[0043] Figure 5 A sub-flowchart for determining whether an action is in violation of regulations. DETAILED DESCRIPTION

[0044] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0045] The present invention belongs to the field of information confidentiality protection technology, and specifically relates to a method of using machine vision technology and a deep learning method to detect the movements of mobile phones and people in images in scenarios where the use of mobile phones is prohibited, thereby determining whether there is any violation of the use of mobile phones.

[0046] To achieve the above object, the present invention provides a method for detecting illegal use of a mobile phone, comprising:

[0047] S1. Create a dataset

[0048] S11. Obtain images of various mobile phone categories as dataset 1 through public datasets or commercial datasets.

[0049] S12. Simulate typical application scenarios such as workshops, offices, and training grounds. Organize personnel to shoot videos of making and receiving phone calls, and capture pictures of making and receiving phone calls from the videos as Dataset 2.

[0050] S13. Automatically label the images collected in Dataset 1 and Dataset 2 using the YOLOv5 official model. Label the mobile phone categories and generate labeled Dataset 1 and Dataset 2, which will be used to train the YOLOv7 model.

[0051] S14. For each category labeled as a mobile phone, expand the box by 1.2 times its center, crop the image, and categorize it into two groups: holding a mobile phone and not holding a mobile phone. Use annotation tools to check the accuracy of the annotations and manually correct any missed or incorrect labels. If the annotation accuracy is low, use a small batch of annotated images for transfer learning based on the YOLOv5 model to increase the accuracy of automatic annotation. Create a classification dataset of images with and without mobile phones as Dataset 3.

[0052] S15. Data enhancement can be performed by rotation, symmetry, cropping, and adding noise. Mosaic data enhancement is also used for data enhancement.

[0053] S2. Training model

[0054] S21. Use the labeled dataset 1 and dataset 2 to train the YOLOv7 model for target detection.

[0055] S22. Use dataset 3 with and without mobile phones to train the VGG-16 model.

[0056] S3. Detecting mobile phone usage behavior

[0057] S31. Obtain the video stream to be tested via methods including but not limited to RTSP, parse the video stream, obtain a single frame image according to pre-set frame skipping parameters, and convert the image size to the model's specified input size. Frame skipping is set to take one frame every three frames, and the image size is converted to 640*480 pixels.

[0058] S32. Input the image into the YOLOv7 model to detect the location of the mobile phone in the image, filter out detection results with a confidence level less than 0.5, and return the category, detection box coordinates, and confidence level.

[0059] S33. For the detection result whose detection category is mobile phone, obtain the coordinates of the mobile phone, proportionally expand the detection frame to 1.2 times the original size with the center point of the mobile phone detection frame as the center, convert the image size, and classify the image as holding a mobile phone or not holding a mobile phone through the VGG-16 model.

[0060] S4. Determine whether the behavior is illegal

[0061] Depending on the application scenario, you can choose to prohibit the use of mobile phones in the scene or prohibit the presence of mobile phones. After obtaining the classification results of all mobile phones in the frame, a logical OR operation is performed. That is, if a mobile phone is held once, it is considered to be in the current video frame. If the classification result shows that a mobile phone is held, it is considered to be in use. If the classification result does not show that a mobile phone is held, it is considered to be a mobile phone presence. If no mobile phone is detected, it is considered to be a mobile phone absence. The pre-set rules for prohibiting the use of mobile phones in the scene or prohibiting the presence of mobile phones are used to determine whether the behavior violates the rules.

[0062] Example 1:

[0063] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0064] (1) Create a dataset

[0065] (11) In a public environment, you can use public datasets, such as the COCO2017 object detection dataset, or purchase commercial datasets as dataset 1.

[0066] (12) Common environments where mobile phone use is prohibited can be divided into indoor office areas, factory workshops, and outdoor venues. In each environment, a 4K high-definition camera is fixed on a tripod to simulate the image presented by the camera monitoring angle. The dataset production takes into account the influence of various factors including light intensity, phone holding posture, traffic flow, and phone type. Videos are shot in the above environments taking various factors into consideration, and images with mobile phones in the screen are captured as Dataset 2.

[0067] (13) Datasets 1 and 2 were each subjected to target detection using the official YOLOv5 model yolov5l.pt. The coordinates and categories of the mobile phones were detected and a labeling file was generated. The parameters were set to: confidence level conf-thres = 0.5, intersection-over-union threshold iou-thres = 0.45, and detection categories classes = 0.67. The labeling results were then corrected using labeling software such as labelimg. If the labeling results are too inaccurate, transfer learning can be used to correct some of the inaccurately labeled images for training.

[0068] (14) The categories detected as mobile phones in datasets 1 and 2 are enlarged to 1.2 times the original detection frame in proportion to the center of the detection frame. If the enlarged area exceeds the image range, the out-of-range area is supplemented with 0 for all three RGB channels. The enlarged image is intercepted and marked as holding a mobile phone and not holding a mobile phone, which is used as dataset 3.

[0069] (16) Data augmentation can be performed by rotation, symmetry, cropping, and adding noise. This is done 11 times in total, randomly selecting 5% of the images in the dataset each time and using one of the following augmentation methods: rotation 45°, rotation 90°, rotation 180°, left-right axis symmetry, top-bottom axis symmetry, center symmetry, cropping the bottom 20% of the image, cropping the left 20% of the image, cropping the right 20% of the image, cropping the top 20% of the image, and adding Gaussian noise. The annotations of the images after data augmentation are then recorded.

[0070] We randomly select 20% of the images in the dataset and use the mosaic data augmentation method to improve the detection of small objects. We group four images into a group and stitch them together. Each image has its own corresponding frame. After stitching the four images together, we get a new image and the corresponding frame for that image.

[0071] (2) Training model

[0072] (21) Training of the target detection model: First, the YOLOv7 target detection model for mobile phone detection was trained using Dataset 1. The training set and test set were divided into a ratio of 8:2, 16 images were used as a batch, and the SGD optimizer method was used. Training was repeated for 200 rounds, and the hyperparameter values ​​were set as follows: network depth parameter was 0.75, network width parameter was 1, learning rate learning_rate was 0.01, stochastic gradient descent momentum parameter momentum was 0.99, and weight decay was 0.00025. Then, transfer learning was performed using Dataset 2. The YOLOv7 model trained in the previous step was used, and the parameters of the first 50 layers of the model were fixed. Training was then performed on Dataset 2, still detecting a single category of mobile phones, with the same hyperparameter settings as before.

[0073] (22) Classification model training: The VGG-16 image classification model was trained using dataset 3. The training set and test set were divided into a ratio of 8:2, 16 images were used as a batch, and the SGD optimizer method was used. The training was repeated for 300 rounds, and the hyperparameter momentum was set to 0.937 and the weight decay was set to 0.0005.

[0074] (3) Detecting mobile phone usage behavior

[0075] (31) Using OpenCV to parse the RTSP video stream, considering the need to improve the efficiency of multi-channel video analysis, a frame skipping method can be used, generally taking one frame out of every three frames for analysis. The parsed video frame is in BGR format, converted to RGB format and scaled to 640*480 pixels using the letterbox method. The letterbox method maintains the aspect ratio of the image and scales the image until the length or width after scaling is exactly equal to the target size, and the image pixels are less than or equal to the target image pixels. The blank part of the RGB channels is filled with 0 respectively.

[0076] (32) Input the image scaled to 640*480 pixels into the YOLOv7 model for inference, and return the detected object category, coordinates, and confidence. Detect only one category of mobile phones, and discard the detection results with a confidence lower than 0.5.

[0077] (33) For the detection result of the detection category being mobile phone, obtain the coordinates of the mobile phone, and expand the detection frame proportionally to 1.2 times the original size according to the center of the detection frame. If the expanded area exceeds the image range at this time, the area beyond the range is supplemented with 0 for all three RGB channels, and the current image is captured. The captured image is converted to 224*224 using the letterbox method and input into the VGG-16 model for classifying whether a mobile phone is held, and the classification result of whether a mobile phone is held or not is obtained.

[0078] (4) Determining whether the behavior is illegal

[0079] Depending on the application scenario, you can choose to prohibit the use of mobile phones in the scene or prohibit the presence of mobile phones. After obtaining the classification results of all mobile phones in the frame, a logical OR operation is performed. That is, if a mobile phone is held once, it is considered to be in the current video frame. If the classification result shows that a mobile phone is held, it is considered to be in use. If the classification result does not show that a mobile phone is held, it is considered to be a mobile phone presence. If no mobile phone is detected, it is considered to be a mobile phone absence. The pre-set rules for prohibiting the use of mobile phones in the scene or prohibiting the presence of mobile phones are used to determine whether the behavior violates the rules.

[0080] Example 2:

[0081] A method for detecting illegal use of a mobile phone, characterized in that the method comprises:

[0082] (1) Create a dataset: obtain and annotate the dataset, and perform data enhancement;

[0083] (2) Model training: Use the dataset to train the YOLOv7 detection model and the VGG-16 classification model respectively;

[0084] (3) Detecting mobile phone usage behavior: reading video data for detection and classification;

[0085] (4) Determine whether the behavior is illegal: Based on the different requirements of different scenarios, make a personalized judgment on whether there is illegal use of mobile phones.

[0086] Furthermore, we collected the COCO2017 object detection public dataset;

[0087] Create application environment dataset;

[0088] Use the YOLOv5 model for automatic labeling;

[0089] Perform data augmentation on the data.

[0090] Furthermore, in the step of creating a data set,

[0091] Three scenarios were simulated: a workshop, an office, and a training ground. The dataset was refined by adjusting lighting intensity, phone holding posture, crowd flow, and phone types to enhance its comprehensiveness. Light intensity was set to 1000lx, 500lx, and 100lx; phone holding postures varied from one hand to the ear for making a call to two handsets for taking a photo; and crowd flow varied from an average of 1, 5, and 20 people per frame. A total of 40 phone types were used, including white, red, black, green, and blue.

[0092] Furthermore, the dataset was annotated using the YOLOv5 model, with a detection confidence threshold of 0.5 and an intersection-over-union ratio of 0.45. 5% of the images in the dataset were manually judged and inaccurate annotations were manually corrected. The proportion of manually corrected images in the dataset was calculated. If this percentage exceeded 10%, transfer learning was performed. This manually annotated set of images served as the dataset for training the YOLOv5 model used for annotation. The trained model was then used to automatically annotate the dataset.

[0093] Furthermore, a total of 11 data augmentation runs were performed, each time randomly selecting 5% of the images in the dataset and applying a different augmentation method, including: 45° rotation, 90° rotation, 180° rotation, left-right axis symmetry, top-bottom axis symmetry, center symmetry, cropping the bottom 20% of the image, cropping the left 20% of the image, cropping the right 20% of the image, cropping the top 20% of the image, and adding Gaussian noise. The annotations of the images after data augmentation were then recorded. A mosaic data augmentation method was applied to 20% of the images randomly selected from the dataset to improve the detection of small targets. Each group of four images was spliced ​​together, each with its corresponding frame. After splicing the four images together, a new image was obtained, along with the corresponding frame for that image.

[0094] Furthermore, we first trained on the COCO2017 object detection public dataset, setting the following hyperparameters: network depth to 0.75, network width to 1, learning rate to 0.01, stochastic gradient descent momentum to 0.99, and weight decay to 0.00025. We then kept the parameters of the first 50 layers of the YOLOv7 model unchanged and used the same hyperparameter settings for further training on the dataset to improve the recognition accuracy of mobile phone usage behavior.

[0095] Furthermore, the video data is read from the camera and the size scaling and frame skipping operations are performed;

[0096] Use the YOLOv7 model to detect a category of mobile phones and return the detection results;

[0097] For the detected mobile phone class, VGG-16 is used for classification.

[0098] Furthermore, one of the video's length and width is scaled to the target size of 640*480, while the other is smaller than or equal to the target size. The portion of the video missing from the target size is then filled with black, i.e., RGB(0,0,0). The video frames are skipped, taking one out of every three frames, to speed up analysis.

[0099] Furthermore, the detected mobile phone category is classified using VGG-16, including: for the detection result of the detection category being mobile phone, its coordinates are obtained, and the detection frame is proportionally enlarged to 1.2 times the original size according to the center of the detection frame. If the enlarged area exceeds the image range at this time, the out-of-range area is supplemented with 0 for all three RGB channels, the current image is captured, the image is scaled by a ratio of the longest side to 224, and then 0 pixels are added in the short side direction, and the image size is converted to 224*224. Finally, it is input into the VGG-16 model for classifying whether a mobile phone is held, and a classification result of holding a mobile phone or not is obtained.

[0100] Furthermore, in the step of determining whether the behavior is in violation, the categories of all mobile phones in the image are detected. After obtaining the classification results of all mobile phones in the image, a logical OR operation is performed. If the classification result shows that a mobile phone is being held, it is considered that the mobile phone is in use. If the classification result shows that no mobile phone is being held, it is considered that a mobile phone is present. If no mobile phone is detected, it is considered that no mobile phone is present. The result of whether to issue an alarm is output according to pre-set standards.

[0101] The present invention discloses a method for detecting illegal use of mobile phones. The method of the present invention comprises: (1) creating a data set. Obtain pictures containing mobile phones in a public environment through a public data set or a commercial data set, simulate a scene where the use of mobile phones is prohibited, and shoot videos of making and receiving mobile phone calls respectively. The collected materials are first automatically labeled using the YOLOv5 official model, and the missed labels and wrong labels are manually corrected, and then the data set is enhanced. (2) training a model. Using the data-enhanced data set, the YOLOv7 target detection model and the VGG-16 classification model are trained respectively. (3) detecting the use of mobile phones. Obtain a camera video stream, and according to the frame skipping setting, obtain a single frame image therein, and transform the image size to the YOLOv7 model input size. The position coordinates of the mobile phone in the image are obtained by training the YOLOv7 model. Obtain an image of the mobile phone area, use VGG-16 to classify the image, and determine whether the current user is holding a mobile phone. (4) determine whether the behavior is illegal.

[0102] The present invention combines detection and classification based on the requirements for mobile phone control in the application area to determine whether there is a violation. The present invention uses machine vision technology and a deep learning method to detect the mobile phone and the actions of the person in the image, thereby determining whether there is any violation of the mobile phone use. The present invention overcomes the problems of detecting mobile phone signals and X-ray mobile phone detection in the prior art through a series of image processing and the design of an automatic detection model. It also overcomes the problem of poor detection effect of deep learning technology methods. The present invention has a high detection accuracy.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting illegal use of mobile phones, characterized in that: The method comprises the following steps: S1. Create a dataset S11. Obtain images of various mobile phone categories as dataset 1 through public datasets or commercial datasets; S12. Simulate a typical application scenario and organize personnel to shoot videos of making and receiving phone calls. Then capture pictures of the videos as Dataset 2. S13, automatically annotate the image materials collected in Dataset 1 and Dataset 2 using the YOLOv5 model, annotate the mobile phone categories, and generate annotated Dataset 1 and Dataset 2, which will be used to train the YOLOv7 model; S14. For the images labeled with the mobile phone category, expand the length and width of the annotation box by 1.2 times based on the center of the annotation box, crop the image, and classify the images into two categories: holding a mobile phone and not holding a mobile phone. This creates a classification dataset of images with and without mobile phones, which is referred to as Dataset 3. S15, use rotation, symmetry, cropping, and noise addition to enhance the data, and use mosaic data enhancement to enhance the data; S2. Training model S21. Use the labeled dataset 1 and dataset 2 to train the YOLOv7 model for target detection. S22, train the VGG-16 model using dataset 3 with and without mobile phones; S3. Detecting mobile phone usage behavior S31. Obtain the video stream to be detected, parse the video stream, obtain a single frame image according to the pre-set frame skipping parameters, and convert the size to the input size specified by the model; S32. Input the image into the YOLOv7 model to detect the location of the mobile phone in the image, filter out detection results with a confidence level less than 0.5, and return the category, detection box coordinates, and confidence level. S33. For a detection result where the detection category is mobile phone, obtain the coordinates of the mobile phone, proportionally expand the detection frame to 1.2 times the original value with the center point of the mobile phone detection frame as the center, convert the image size, and classify the image as holding a mobile phone or not holding a mobile phone using the VGG-16 model; S4. Determine whether the behavior is illegal After obtaining the classification results of all mobile phones in the picture, a logical OR operation is performed, that is, if a mobile phone is held once, it is considered that the current video frame is holding a mobile phone; if the classification result shows that a mobile phone is held, it is considered that the mobile phone is used; if the classification result does not show that a mobile phone is held, it is considered that a mobile phone is present; if no mobile phone is detected, it is considered that there is no mobile phone; based on the pre-set rules prohibiting the use of mobile phones or prohibiting the appearance of mobile phones, it is determined whether the behavior is in violation of the rules.

2. The method for detecting illegal use of a mobile phone according to claim 1, wherein: The step S12 specifically includes: common environments where the use of mobile phones is prohibited include: indoor office areas, factory workshops and outdoor venues. A 4K high-definition camera is fixed on a tripod in each environment to simulate the picture conditions presented by the camera monitoring angle; the dataset production takes into account the influence of various factors including: light intensity, phone holding posture, traffic flow and phone type. Videos are shot in the above environments taking various factors into consideration, and images of mobile phones in the picture are captured as dataset 2.

3. The method for detecting illegal use of a mobile phone according to claim 1, wherein: The step S13 specifically includes: using the YOLOv5 model yolov5l.pt to perform target detection on dataset 1 and dataset 2 respectively, detecting the coordinates and categories of the mobile phones, generating a labeling file, and setting the parameters as follows: confidence conf-thres is 0.5, intersection-over-union threshold iou-thres is 0.45, and detection category classes is: 0,67; then correcting the labeling results through labeling software. If the labeling results are too inaccurate, correct some of the inaccurately labeled pictures for training through transfer learning.

4. The method for detecting illegal use of a mobile phone according to claim 3, wherein: Step S14 specifically includes: enlarging the detection frame of the images detected as mobile phones in datasets 1 and 2 to 1.2 times the original size according to the center of the detection frame; if the enlarged area exceeds the image range at this time, the out-of-range area is supplemented with 0 for all three RGB channels; the enlarged image is intercepted, and the image is marked as holding a mobile phone and not holding a mobile phone, as dataset 3.

5. The method for detecting illegal use of a mobile phone according to claim 4, wherein: The step S15 specifically includes: performing data enhancement by means of rotation, symmetry, cropping, and adding noise, which is performed 11 times in total, and randomly extracting 5% of the pictures in the data set each time, and using an enhancement method including: rotating 45°, rotating 90°, rotating 180°, left-right axis symmetry, top-bottom axis symmetry, center symmetry, cropping 20% ​​of the bottom picture, cropping 20% ​​of the left picture, cropping 20% ​​of the right picture, cropping 20% ​​of the top picture, adding Gaussian noise, and then recording the annotations of the pictures after data enhancement; randomly extracting 20% ​​of the pictures in the data set and using a mosaic data enhancement method to increase the detection effect for small targets, and splicing four pictures into a group, each of which has its corresponding frame. After splicing the four pictures, a new picture is obtained, and the frame corresponding to this picture is also obtained.

6. The method for detecting illegal use of a mobile phone according to any one of claims 1 to 5, characterized in that: The step S21 specifically includes: first, using data set 1 to train the YOLOv7 model for detecting mobile phones, with the training set and test set divided in a ratio of 8:2, 16 pictures as a batch, using the SGD optimizer method, training for 200 rounds, and setting the values ​​of hyperparameters: the network depth parameter is 0.75, the network width parameter is 1, the learning rate learning_rate is 0.01, the stochastic gradient descent momentum parameter momentum is 0.99, and the weight decay is 0.00025; then, using data set 2 for transfer learning, using the YOLOv7 model trained in the previous step, fixing the network parameters of the first 50 layers of the model without changing, and then training on data set 2, still detecting one category of mobile phones, and the hyperparameter settings are the same as before.

7. The method for detecting illegal use of a mobile phone according to claim 6, wherein: The step S22 specifically includes: using dataset 3 to train the VGG-16 model, with the training set and test set divided in a ratio of 8:2, 16 images as a batch, using the SGD optimizer method, training for 300 rounds, setting the hyperparameter momentum to 0.937, and the weight decay to 0.0005.

8. The method for detecting illegal use of a mobile phone according to claim 1, wherein: The step S31 specifically includes: parsing the RTSP video stream through opencv, using a frame skipping method, taking one frame out of every three frames for analysis, and converting the parsed video frame into BGR format, and using the letterbox method to scale the size to 640*480 pixels.

9. The method for detecting illegal use of a mobile phone according to claim 8, wherein: The step S32 specifically includes: inputting the image scaled to 640*480 pixels into the YOLOv7 model for inference, returning the detected object category, coordinates, and confidence, detecting one category of mobile phones, and discarding detection results with a confidence level lower than 0.

5.

10. The method for detecting illegal use of a mobile phone according to claim 9, wherein: The step S33 specifically includes: for the detection result of the detection category being a mobile phone, obtaining the coordinates of the mobile phone, proportionally expanding the detection frame to 1.2 times the original size according to the center of the detection frame, if the expanded area exceeds the image range at this time, supplementing the out-of-range area with 0 for all three RGB channels, intercepting the current image, converting the intercepted image to 224*224 using the letterbox method, and inputting it into the VGG-16 model for classifying whether the mobile phone is held, thereby obtaining a classification result of whether the mobile phone is held or not.

Images