A method for detecting a mobile phone use behavior while riding a bicycle

Abstract

The application provides a detection method for mobile phone use behavior of cyclists, which is based on images collected by existing monitoring equipment, selects front driving images that can be used for behavior analysis, intercepts images of non-motor vehicles and cyclists, corrects the direction, and forms a video to be analyzed, effectively improving the identifiable behavior characteristics of the vehicle driver in the video, and then performing behavior recognition through a three-dimensional convolutional neural network to ensure that the mobile phone use behavior of non-motor vehicle cyclists in the video can be effectively found; the behavior recognition through the three-dimensional convolutional neural network can effectively reduce the misrecognition rate; the method does not need to add new hardware, and the mobile phone use behavior can be detected based on ordinary resolution video data at a low cost.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically a method for detecting the use of a mobile phone while riding a bicycle. Background Technology

[0002] Common two-wheeled vehicles include bicycles, electric bicycles, and motorcycles. The act of using a mobile phone while riding a two-wheeled vehicle poses a significant danger. The rider's attention is diverted, and their reaction time to their surroundings is sluggish, greatly increasing the risk of traffic accidents. Furthermore, the dangers of using a mobile phone while riding are often difficult to detect. Some cyclists may have already developed this bad habit without experiencing serious consequences, thus ignoring its potential risks. Because mobile phones are relatively small targets with fewer pixels in video images, existing target detection algorithms can only support the detection of mobile phone use while riding in some high-definition videos, and are insufficient to detect mobile phone use in videos with normal resolution. Summary of the Invention

[0003] To address the difficulty in detecting mobile phone use while cycling in videos with standard resolution, this invention provides a method for detecting mobile phone use while cycling, which can effectively detect mobile phone use by non-motorized vehicle riders in videos.

[0004] The technical solution of the present invention is as follows: a method for detecting mobile phone use while cycling, characterized by comprising the following steps:

[0005] S1: Construct an object detection model and a frontal portrait recognition and classification model;

[0006] Based on the target detection model, the image region where the non-motorized vehicle is located is located in the image to be detected;

[0007] Based on the aforementioned facial recognition model, the facial images in the image are classified and recognized from the front view.

[0008] S2: Use the entire image region as the region to be detected;

[0009] S3: Acquire the image collected by the monitoring equipment, denoted as: image to be monitored;

[0010] S4: Based on the target detection model, locate the non-motorized vehicle in the region to be detected in the image to be detected, and crop out the image region where the non-motorized vehicle and the driver are located, denoted as: target image region; for the human figure part in the target image region, identify whether it is a frontal human figure.

[0011] If so, it indicates that the non-motorized vehicle in the image to be monitored is traveling in front, and is denoted as: image to be analyzed; proceed to step S4;

[0012] Otherwise, discard the image to be monitored, and execute steps S2-S3 in a loop until all images to be monitored are processed;

[0013] S5: Based on the image to be analyzed, continuously accept images collected by the monitoring device, find all images with the same vehicle and driver in them, and construct a set of images to be analyzed;

[0014] S6: Perform direction correction on the images of non-motor vehicles and cyclists in the target image area in each frame of the set of images to be analyzed;

[0015] S7: Combine all corrected images in the set of images to be analyzed as a video to be analyzed;

[0016] S8: Construct a mobile phone behavior detection model based on a three-dimensional convolutional neural network;

[0017] The mobile phone behavior detection model is used to determine whether the cyclist has a mobile phone behavior;

[0018] The mobile phone behavior includes: calling on a bicycle, playing a mobile phone with one hand, operating a mobile phone on a support with one hand, and playing a mobile phone with both hands;

[0019] Based on historical data, construct a training data set and a validation set, train the mobile phone behavior detection model, and obtain a trained mobile phone behavior detection model;

[0020] S9: Input the video to be analyzed into the trained mobile phone behavior detection model to determine whether the cyclist has a mobile phone behavior.

[0021] Further features include:

[0022] The method for constructing the set of images to be analyzed includes the following steps:

[0023] a1: Initialize the set of images to be analyzed;

[0024] Put the image to be analyzed as the first frame image into the set of images to be analyzed;

[0025] a2: Construct a rectangular frame for the target image area in the image to be analyzed, denoted as: current image area rectangular frame; Based on the position and size of the current image area rectangular frame, enlarge the current image area rectangular frame to construct the detection area frame of the next frame image;

[0026] a3: Obtain the detection image of the next frame;

[0027] a4: based on the rectangular frame in which the target image region is located, the side length is enlarged by a preset multiple N, and a sliding scanning frame is established;

[0028] wherein the area of the sliding scanning frame is smaller than the area of the to-be-detected region;

[0029] a5: in the to-be-detected region frame of the to-be-detected image, the sliding scanning frame is slid according to a preset step length, and M groups of image hash values in the sliding scanning frames are obtained;

[0030] a6: calculate the hash value of the image in the current image region rectangular frame in the to-be-analyzed image, denoted as: comparison hash value;

[0031] calculate the hash value similarity between the M groups of rectangular frame image hash values and the comparison hash value, denoted as: hash value similarity;

[0032] a7: compare the M groups of hash value similarities with a preset difference threshold;

[0033] If all the hash value similarities are greater than the difference threshold, it indicates that there is no image similar to the image in the target image region in the M groups of sliding scanning frames; step a8 is executed;

[0034] If there is a group of hash value similarities less than the difference threshold, it indicates that the image in the corresponding sliding scanning frame is most similar to the image in the target image region, denoted as: candidate image, and step a9 is executed;

[0035] If there are multiple groups of hash value similarities less than the difference threshold, find the hash value similarity with the largest absolute value, and the image in the corresponding sliding scanning frame is most similar to the image in the target image region, denoted as: candidate image; step a9 is executed;

[0036] a8: discard the image, and execute steps a2-a7 until the vehicle leaves the collection range of the monitoring device;

[0037] a9: sort the candidate images according to time and put them into a to-be-analyzed image set;

[0038] a10: record the candidate image as: to-be-analyzed image, and cyclically execute steps a2-a9;

[0039] In step a8, based on a preset driving-off threshold NP, if NP consecutive frames of the to-be-detected image are discarded, it is judged that the vehicle has left the collection range of the monitoring device;

[0040] The construction method of the to-be-detected region frame comprises the following steps:

[0041] b1: construct a coordinate system on the to-be-monitored image, taking the upper left corner as the coordinate origin;

[0042] b2: The center of the current image region rectangle is used as the center of the region to be detected;

[0043] b3: Get the side lengths of the current image region rectangle, denoted as: horizontal side and vertical side;

[0044] Obtain the preset magnification factor CN, where CN>1;

[0045] The horizontal side length of the detection area box is CN times the horizontal side length.

[0046] The vertical side length of the region to be detected is CN times the vertical side length.

[0047] b4: In the next frame image, construct the detection region bounding box based on the center position of the region bounding box;

[0048] In step S6, the orientation correction method includes the following steps:

[0049] c1: Obtain each frame of the image set to be analyzed one by one, denoted as: image to be corrected;

[0050] c2: Based on the direction of the road in the image to be corrected, the image of the road in the image to be corrected is corrected to the angle and direction of the frontal shot to obtain the correction matrix;

[0051] c3: Based on the correction matrix, the images of non-motorized vehicles and cyclists in the target image area of ​​the image are oriented to obtain a frontal view image;

[0052] c4: Store the corrected images of non-motorized vehicles and cyclists in the target image region to replace the original images, and obtain a set of corrected images to be analyzed;

[0053] In step S7, all corrected images in the set of images to be analyzed are collectively referred to as the video to be analyzed, which specifically includes the following steps:

[0054] d1: Adjust all images in the set of images to be analyzed to a fixed size of the specified dimensions using image scaling;

[0055] d2: Combine all the resized images into a video clip to obtain the video to be analyzed.

[0056] This application provides a method for detecting mobile phone use while cycling. Based on images collected by existing monitoring equipment, it selects frontal driving images suitable for behavior analysis, extracts images of non-motorized vehicles and cyclists, corrects their orientation, and constructs the video to be analyzed. This effectively improves the identifiable behavioral features of vehicle drivers in the video. The method then uses a 3D convolutional neural network for behavior recognition, ensuring effective detection of mobile phone use by non-motorized vehicles, cyclists, and other users in the video. Using a 3D convolutional neural network for behavior recognition effectively reduces the false recognition rate of cyclist behavior. This method requires no additional hardware and can detect mobile phone use based on ordinary resolution video data at a low cost. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method for detecting mobile phone use while cycling in this application;

[0058] Figure 2 Example of the bounding box for the region to be detected;

[0059] Figure 3 This is an example of orientation correction. Detailed Implementation

[0060] like Figure 1 As shown, the present invention includes a method for detecting the behavior of using a mobile phone while riding a bicycle, which includes the following steps.

[0061] S1: Construct an object detection model and a frontal portrait recognition and classification model;

[0062] Based on the target detection model, the image region where the non-motorized vehicle is located is located in the image to be detected;

[0063] Based on the human face recognition model, the human face portion of the image is classified and recognized from the front view.

[0064] In practical implementation, the object detection model is based on various existing non-motorized vehicle recognition models, such as those built using neural networks like CNN, YOLO, and GAN. The facial recognition model is built using existing machine learning classification models, such as support vector machines, decision trees, or random forests. A labeled training dataset is built based on historical data to train the model, and then a validation set is built based on historical data to validate the model, until a well-trained model is obtained.

[0065] S2: Use the entire image region as the region to be detected. When detecting the first image, use the entire image region as the region to be detected.

[0066] S3: Acquire images collected by monitoring equipment, denoted as: images to be monitored.

[0067] In this method, video data acquired by a standard-resolution monitoring device is used as the data source. The data is decomposed into image frames, and the resulting image frames are denoted as the images to be monitored. Alternatively, data from image acquisition devices installed on a drive test can be used as the data source, with the images acquired by these devices serving as the images to be monitored. This method does not require changing the resolution of existing monitoring equipment; behavior detection can be achieved using existing monitoring devices.

[0068] S4: Based on the target detection model, locate the non-motorized vehicle in the region to be detected in the image to be detected, and crop out the image region where the non-motorized vehicle and the driver are located, which is denoted as: target image region; for the human figure part in the target image region, identify whether it is a frontal human figure based on the human figure recognition model;

[0069] If so, it means that the non-motorized vehicle in the image to be monitored is traveling in front, and is recorded as: image to be analyzed; proceed to step S4;

[0070] Otherwise, discard the image to be monitored and repeat steps S2 to S3 until all images to be monitored have been processed.

[0071] The target image region construction method in this method is as follows:

[0072] Construct coordinate axes in the image to be detected, with the top left corner as the origin. For example... Figure 3 As shown. Find the point corresponding to the minimum x and y values ​​in the image pixel coordinates of non-motorized vehicles and drivers, and use it as the top left corner. Use the point corresponding to the maximum x and y values ​​as the bottom right corner to construct a rectangle, thus obtaining the bounding box corresponding to the target image region.

[0073] In this method, the recognition area of ​​the human face recognition model is set in the target image area, rather than the area to be detected, which can effectively reduce the amount of computation and improve the computational efficiency.

[0074] S5: Based on the images to be analyzed, continuously receive images collected by monitoring equipment, find all images with the same vehicle and driver, and construct a set of images to be analyzed.

[0075] The method for constructing the image set to be analyzed includes the following steps.

[0076] a1: Initialize the set of images to be analyzed;

[0077] The image to be analyzed is used as the first frame image and added to the image set to be analyzed.

[0078] a2: Construct a bounding box for the target image region in the image to be analyzed, denoted as: current image region bounding box; based on the position and size of the current image region bounding box, enlarge the current image region bounding box and construct the detection region bounding box for the next frame image;

[0079] a3: Obtain the image to be detected for the next frame.

[0080] a4: Based on the rectangle of the target image area, magnify the side length by a preset multiple N to establish a sliding scanning frame; where, since the condition needs to be satisfied: the area of the sliding scanning frame is smaller than the area of the area to be detected. In this actual example, the area to be detected frame is magnified by 1.2 (CN) times based on the side length of the rectangle of the target image area, then the value of N is set as: 0.9 < N < CN; the specific value is set according to the calculation accuracy requirement. When the value of N is set small, the calculation amount is large and the accuracy is high. When the value of N is set large, the calculation amount is small and the accuracy is low. In this embodiment, the value of N is 0.9 to 1.1 times.

[0081] a5: In the area to be detected frame of the image to be detected, slide the sliding scanning frame according to a preset step size to obtain the image hash values within M groups of sliding scanning frames.

[0082] The image hash value is to convert the image into a unique numerical identifier through a specific algorithm, and is often used for image similarity comparison. The image hash algorithm measures the similarity of images by converting the image into a hash value of a fixed length and comparing the Hamming distance of these hash values. The specific hash value calculation method and hash value similarity algorithm are implemented based on the existing technology.

[0083] a6: Calculate the hash value of the image inside the rectangle of the current image area in the image to be analyzed, denoted as: the hash value for comparison;

[0084] Calculate the similarity between the image hash values within the M groups of rectangles and the hash value for comparison respectively, denoted as: the hash value similarity.

[0085] a7: Compare the hash value similarities of the M groups with a preset difference threshold;

[0086] In this application, the similarity accuracy of two images compared with each other is controlled based on the difference threshold, and the specific value is set according to the actual accuracy requirement.

[0087] If all the hash value similarities are greater than the difference threshold, it means that there is no image similar to the image in the target image area within the M groups of sliding scanning frames; execute step a8;

[0088] If there is a group of hash value similarities less than the difference threshold, it means that the image within the corresponding sliding scanning frame is the most similar to the image in the target image area, denoted as: the alternative image, and execute step a9;

[0089] If multiple sets of hash values ​​have similarity less than the difference threshold, find the hash value with the largest absolute value. The image within the corresponding sliding scan box is most similar to the image in the target image region, and is denoted as: candidate image; proceed to step a9.

[0090] a8: Discard the image and execute steps a2~a7 until the vehicle leaves the monitoring equipment's acquisition range.

[0091] a9: Sort the candidate images by time and add them to the set of images to be analyzed.

[0092] a10: Record the candidate image as: image to be analyzed, and repeat steps a2~a9.

[0093] In this application, through the construction process of the set of images to be analyzed, video frames in ordinary resolution videos that cannot be used for recognition due to low resolution or because the driver is not a frontal image are removed, and all sufficiently clear images are retained based on hash value comparison, which effectively improves the accuracy of subsequent 3D convolutional neural network for behavior recognition.

[0094] In step a8, based on the preset departure threshold NP, if consecutive NP frames of the image to be detected are discarded, it is determined that the vehicle has left the collection range of the monitoring device; in this embodiment, the value of NP is 5, that is, if there is no target in 5 consecutive frames of the image to be detected, it is determined that the vehicle has left.

[0095] like Figure 2 As shown, the method for constructing the bounding box of the region to be detected includes the following steps:

[0096] b1: Construct a coordinate system on the image to be monitored, with the upper left corner as the origin.

[0097] b2: Use the center of the current image region rectangle as the center of the region to be detected.

[0098] b3: Get the side lengths of the current image region rectangle, denoted as: horizontal side and vertical side;

[0099] A preset magnification factor CN is obtained, where CN > 1. In this method, the target detection range is limited by the area to be detected to reduce the computational load. By setting CN to a value greater than 1, the bounding box of the area to be detected is gradually magnified based on the area where the non-motorized vehicle image is located in the previous frame image, in order to adapt to the increasingly larger changes in the image area where the non-motorized vehicle is located in the forward-moving video, ensuring that this application is more practical. In this embodiment, CN is set to 1.2.

[0100] The horizontal side length of the detection area is CN times the horizontal side length.

[0101] The vertical side length of the detection area is CN times the vertical side length.

[0102] b4: In the next frame, construct the detection region bounding box based on the center position of the region bounding box.

[0103] S6: Based on the fact that "vehicles travel along the road", the images of non-motorized vehicles and cyclists in the target image region of each frame in the image set to be analyzed are oriented according to the direction of travel of non-motorized vehicles.

[0104] In step S6, the orientation correction method includes the following steps:

[0105] c1: Obtain each frame of the image set to be analyzed one by one, denoted as: the image to be corrected;

[0106] c2: Based on the direction of the road in the image to be corrected, the image of the road in the image to be corrected is corrected to the angle and direction of the frontal shot to obtain the correction matrix;

[0107] c3: Based on the correction matrix, the images of non-motorized vehicles and cyclists in the target image area of ​​the image are oriented to obtain a frontal view image;

[0108] c4: Replace the original images with the images of non-motorized vehicles and cyclists in the corrected target image region to obtain a set of corrected images to be analyzed.

[0109] like Figure 3 As shown, in the frontal image, the road direction r is perpendicular to the x-axis from top to bottom.

[0110] The road direction r' in the image to be corrected has an angle α with the direction r. Therefore, the road direction r' in the image to be corrected is first corrected to the direction r, which yields the corresponding correction matrix. Then, based on this correction matrix, the images of non-motorized vehicles and cyclists in the target image region of the image to be corrected are corrected to obtain the corrected frontal view image. The angle α can be obtained from the parameters of the image acquisition equipment during installation, or it can be calculated based on existing image recognition technology.

[0111] S7: Combine all corrected images in the set of images to be analyzed into a video to be analyzed.

[0112] In step S7, all corrected images in the set of images to be analyzed are collectively referred to as the video to be analyzed, which specifically includes the following steps:

[0113] d1: Adjust all images in the set of images to be analyzed to a fixed size of a specified dimension by image scaling; in this embodiment, the images are adjusted to a fixed size of 320*320 by image scaling.

[0114] d2: Combine all the resized images into a video clip to obtain the video to be analyzed.

[0115] The image frame values ​​in the video to be analyzed in this method include corrected images of non-motorized vehicles and cyclists, which can more effectively provide features for the detection model to recognize. Moreover, compared with methods that detect based on the entire image, it effectively reduces the amount of computation and improves computational efficiency.

[0116] S8: A mobile phone behavior detection model built based on a 3D convolutional neural network;

[0117] A mobile phone behavior detection model is used to determine whether a cyclist is using a mobile phone.

[0118] Mobile phone usage behaviors include: making phone calls while riding a bicycle, using a mobile phone with one hand, operating a mobile phone on a stand with one hand, and using a mobile phone with both hands.

[0119] Training and validation datasets are constructed based on historical data, and the mobile phone behavior detection model is trained to obtain a well-trained mobile phone behavior detection model.

[0120] S9: Input the video to be analyzed into the trained mobile phone use behavior detection model to determine whether the cyclist is using a mobile phone.

[0121] After using the technical solution of the present invention, based on existing equipment, it is also possible to effectively detect four types of mobile phone use behaviors while riding a bicycle: making a phone call while riding a bicycle, playing with a mobile phone with one hand, operating a mobile phone on a bracket with one hand, and playing with a mobile phone with both hands from ordinary resolution videos.

Claims

1. A method for detecting mobile phone use while cycling, characterized in that, It includes the following steps: S1: Construct an object detection model and a frontal portrait recognition and classification model; Based on the target detection model, the image region where the non-motorized vehicle is located is located in the image to be detected; Based on the aforementioned human image recognition and classification model, for the human image portion of the image, the human image facing the front is classified and recognized; S2: Use the entire image region as the region to be detected; S3: Acquire the image collected by the monitoring equipment, denoted as: image to be monitored; S4: Based on the target detection model, locate the non-motorized vehicle in the region to be detected in the image to be detected, and crop out the image region where the non-motorized vehicle and the driver are located, denoted as: target image region; for the human figure part in the target image region, identify whether it is a frontal human figure. If so, it indicates that the non-motorized vehicle in the image to be monitored is traveling in front, and is denoted as: image to be analyzed; proceed to step S4; Otherwise, discard the image to be monitored and repeat steps S2 to S3 until all the images to be monitored have been processed. S5: Based on the image to be analyzed, continuously receive images collected by the monitoring equipment, find all images with the same vehicle and driver, and construct a set of images to be analyzed; S6: Perform orientation correction on the images of non-motorized vehicles and cyclists in the target image region of each frame in the image set to be analyzed; S7: All corrected images in the set of images to be analyzed are collectively referred to as the video to be analyzed; S8: A mobile phone behavior detection model built based on a 3D convolutional neural network; The mobile phone usage behavior detection model is used to determine whether the cyclist is using a mobile phone. The aforementioned mobile phone usage behaviors include: making phone calls while riding a bicycle, using a mobile phone with one hand, operating a mobile phone on a stand with one hand, and using a mobile phone with both hands; A training dataset and a validation dataset are constructed based on historical data. The mobile phone usage behavior detection model is then trained to obtain a well-trained mobile phone usage behavior detection model. S9: Input the video to be analyzed into the trained mobile phone use behavior detection model to determine whether the cyclist is using a mobile phone.

2. The method for detecting mobile phone use while cycling according to claim 1, characterized in that: The method for constructing the set of images to be analyzed includes the following steps: a1: Initialize the set of images to be analyzed; The image to be analyzed is used as the first frame image and placed into the set of images to be analyzed. a2: Construct a rectangular bounding box for the target image region in the image to be analyzed, denoted as: current image region bounding box; based on the position and size of the current image region bounding box, enlarge the current image region bounding box and construct the detection region bounding box for the next frame image; a3: Obtain the image to be detected in the next frame; a4: Based on the rectangle containing the target image region, the side length is enlarged by a preset multiple N to create a sliding scanning box; In this case, the area of ​​the sliding scanning frame is smaller than the area of ​​the region to be detected; a5: In the detection area box of the image to be detected, slide the sliding scanning box according to a preset step size to obtain M sets of image hash values ​​within the sliding scanning box; a6: Calculate the hash value of the image inside the current image region rectangle in the image to be analyzed, denoted as: comparison hash value; The similarity between the hash values ​​of the images within the M groups of rectangles and the hash values ​​used for comparison is calculated and denoted as: hash value similarity; a7: Compare the similarity of the hash values ​​in group M with a preset difference threshold; If the similarity of all the hash values ​​is greater than the difference threshold, it means that there is no image similar to the image in the target image region within the M group of sliding scan boxes; proceed to step a8; If there exists a set of hash values ​​with a similarity less than the difference threshold, it means that the image within the corresponding sliding scan box is most similar to the image within the target image region, and is denoted as: candidate image, and step a9 is executed; If multiple sets of hash value similarities are less than the difference threshold, find the hash value similarity with the largest absolute value. The image within the corresponding sliding scan box is most similar to the image in the target image region, and is denoted as: candidate image; proceed to step a9; a8: Discard the image and execute steps a2~a7 until the vehicle leaves the monitoring equipment's acquisition range; a9: Sort the candidate images according to time and put them into the image set to be analyzed; a10: The candidate image is denoted as: image to be analyzed, and steps a2 to a9 are executed cyclically.

3. The method for detecting mobile phone use while cycling according to claim 2, characterized in that: In step a8, based on the preset departure threshold NP, if the images to be detected in consecutive NP frames are all discarded, it is determined that the vehicle has left the collection range of the monitoring equipment.

4. The method for detecting mobile phone use while cycling according to claim 2, characterized in that: The method for constructing the bounding box of the region to be detected includes the following steps: b1: Construct a coordinate system on the image to be monitored, with the upper left corner as the origin; b2: The center of the current image region rectangle is used as the center of the region to be detected; b3: Get the side lengths of the current image region rectangle, denoted as: horizontal side and vertical side; Obtain the preset magnification factor CN, where CN>1; The horizontal side length of the detection area box is CN times the horizontal side length. The vertical side length of the region to be detected is CN times the vertical side length. b4: In the next frame image, construct the detection region bounding box based on the center position of the region bounding box.

5. The method for detecting mobile phone use while cycling according to claim 1, characterized in that: In step S6, the orientation correction method includes the following steps: c1: Obtain each frame of the image set to be analyzed one by one, denoted as: image to be corrected; c2: Based on the direction of the road in the image to be corrected, correct the image of the road in the image to the angle and direction of the frontal shot to obtain the correction matrix; c3: Based on the correction matrix, the images of non-motorized vehicles and cyclists in the target image area of ​​the image are oriented to obtain a frontal view image; c4: Replace the original images with the corrected images of non-motorized vehicles and cyclists in the target image region to obtain a set of corrected images to be analyzed.

6. The method for detecting mobile phone use while cycling according to claim 1, characterized in that: In step S7, all corrected images in the set of images to be analyzed are collectively referred to as the video to be analyzed, which specifically includes the following steps: d1: Adjusts all images in the set of images to be analyzed to a fixed size of the specified dimensions using image scaling; d2: Combine all the resized images into a video clip to obtain the video to be analyzed.

Images