Method and device for detecting driver's in-car phone calls based on multi-module face recognition

Abstract

A method and device for detecting a driver's in-car phone calls based on multi-module face recognition, the method includes: obtaining an image data set, real-time recording with an in-car camera or other visual device, transmitting the video stream to an embedded device or to the cloud, etc., directly using the received video as the input of a trained target detection algorithm for real-time detection; the target detection algorithm performs real-time detection on the mobile phone and the driver, uses the prediction frame recognized by the target detection algorithm as input, and performs real-time tracking on the target object; uses a twin fully connected neural network SiameseFC target tracking algorithm, and uses the target tracking algorithm to count and track mobile phones; the target detection algorithm marks the driver and tracks him in real time; identifies key points of the face for detection; and determines the operation of making or receiving a mobile phone. The present invention reduces the amount of calculation, improves the detection performance when the driver and the mobile phone overlap or are blocked, and the small target detection performance of the mobile phone.

Description

Technical Field

[0001] The present invention relates to the field of detection methods for drivers making or receiving phone calls in a car, and in particular to a detection method and device for making or receiving phone calls based on face recognition. Background Art

[0002] At present, most traffic law enforcement departments use cameras on gantry at intersections to monitor drivers’ phone calls and use image recognition algorithms to identify and detect phone calls, but there are still some problems. 1) When the traffic volume is large or the speed is fast, the phone itself is small in size, and the user usually holds it in his hand, with a large occlusion rate, so there is a high missed detection rate. 2) Some customers only pick up or move the phone without answering the phone, which results in a certain false detection rate. 3) Some algorithms use gesture detection methods, which have a large false detection rate for actions similar to raising your hand to answer or make phone calls, such as wiping your mouth. In response to the above problems, and in the face of the development trend of abundant visual equipment inside and outside the future intelligent connected cars, there is an urgent need for a method to detect phone calls inside the car to improve the small target detection of mobile phones and the precise matching of driver operations. Summary of the invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a method and device for detecting a driver making or receiving a mobile phone call based on multi-module face recognition in a vehicle.

[0004] The method for detecting making and receiving calls of the present invention includes a target detection algorithm, a target tracking algorithm and a facial key point detection algorithm, and improves the target detection algorithm and the facial key point detection algorithm, and reduces the overall model parameter amount and the amount of calculation, making it more suitable for the application of embedded devices. Under the premise of ensuring the detection speed, the small target detection of the mobile phone and the overlap detection of the human face mouth area and the mobile phone are achieved.

[0005] The technical solution of the present invention is:

[0006] In the detection method of the driver's in-car phone call based on face recognition, the target detection algorithm and the face key point detection algorithm are improved to reduce the number of parameters and calculations of the two algorithms, making them more suitable for embedded device applications, and improving the accuracy of small target detection for mobile phones, as well as reducing the missed detection rate under occlusion. Since the target tracking algorithm is only used to locate and track the detected targets, no changes are made.

[0007] The method for detecting a driver making or receiving a mobile phone call based on multi-module face recognition in a vehicle specifically comprises the following steps:

[0008] Step 1: Obtain an image dataset. The in-car camera or other visual device records the video in real time, transmits the video stream to an embedded device or to the cloud, and uses the received video directly as the input of the trained target detection algorithm for real-time detection. The process is as follows:

[0009] Step 1.1: Change the feature extraction algorithm of the bottleneck layer of the target detection algorithm backbone network to a lightweight network structure. The present invention adopts GhostBottle composed of the new end-side neural network architecture GhostNet proposed by Huawei. Under the premise of meeting the same number of channels, the feature extraction capability is enhanced, which greatly reduces the calculation amount and parameter amount of the overall model, which is conducive to the use of the detection algorithm in embedded devices and improves the detection speed.

[0010] Step 1.2: Add an attention mechanism algorithm to the end of the backbone network of the target detection algorithm, and strengthen feature extraction by assigning weights to each channel. Adaptive attention distribution can be learned from different feature spaces, which enhances the feature extraction capability for occluded targets and reduces the missed detection rate.

[0011] Step 1.3: Multiple downsampling will cause the features of small targets in the image to disappear, resulting in the model network being unable to learn the corresponding features, and small targets such as mobile phones are prone to missed detection. The target detection algorithm retains the original detection dimension, does not reduce the original scale feature learning ability, adds connection layers, convolution layers, and bottleneck layers, and performs upsampling and downsampling again. The same-scale feature layer that retains more small target feature information in the backbone network is input into the connection layer for feature fusion operations. The newly added detection layer is more sensitive to small target features, which is conducive to the detection of small targets such as mobile phones. Due to the increase in the detection scale diversity of the detection layer, a genetic algorithm is used to analyze and calculate the custom data set to obtain the predicted anchor boxes required for each scale of the data set.

[0012] Step 1.4: Use the concat connection layer to fuse feature layers of different levels but the same scale to reduce feature loss, and input the multi-layer calculation results of the target detection algorithm's backbone network into the enhanced feature extraction network once or multiple times. The enhanced feature extraction network is divided into two parts, the bottom-up structure and the top-down structure. The bottom-up part fuses two features of the same scale; when it comes to the top-down structure, three features of the same scale are fused using jump connections, and output after passing through the bottleneck layer to predict multiple targets of different scales. The use of a bidirectional feature pyramid network structure makes the prediction network more sensitive to targets of different sizes, improves the overall model detection capability, and reduces the missed detection rate and false detection rate.

[0013] Step 2: The target detection algorithm performs real-time detection on the mobile phone and the driver. The target tracking algorithm uses the prediction box identified by the target detection algorithm as input to track the target object in real time. The present invention adopts the SiameseFC (Siamese Fully-Connected Neural Network) target tracking algorithm to count and track mobile phones. The process is as follows:

[0014] Step 2.1: When a mobile phone is detected, the target tracking algorithm is first used to mark, count and distinguish the mobile phones, such as mobile phone-1. At the same time, a single frame image of the first marked target object is captured and retained as a panoramic image to facilitate subsequent manual identification.

[0015] Step 2.2: Output the real-time tracking to obtain the mobile phone’s tag value and the real-time parameters of the detection box (x c ,y c ,w c ,h c ), x c ,y c Indicates the center point of the mobile phone detection frame, w c ,h c Indicates the width and height of the phone detection frame.

[0016] Step 2.3: If the real-time parameters of the mobile phone detection frame do not change within a certain period of time, the target object is deemed to be abandoned or unused, and the mobile phone detection frame mark value is retained, but it is not used as a comparison target in steps 3.2 and 5 until the real-time parameters of the target object change.

[0017] Step 3: Driver detection and technology. The process is as follows:

[0018] Step 3.1: The target detection algorithm simultaneously marks the driver and tracks the driver in real time. After outputting the real-time tracking, the driver’s marking value and the real-time parameters of the detection box (x R ,y R ,w R ,h R ), x R ,y R represents the center point of the driver detection box, w R ,h R Indicates the width and height of the driver detection box.

[0019] Step 3.2: When the mobile phone detection frame overlaps with the driver detection frame, the overlapping driver detection frame is marked with a count to distinguish, for example, driver-1.

[0020] Step 3.3: Simultaneously return the driver's tag value and the driver's detection frame real-time parameters (x R ,yR ,w R ,h R ). When the marked mobile phone detection frame ends overlapping with the driver detection frame, the driver detection frame mark is discarded and the real-time parameters are stopped from being returned.

[0021] Step 4: Face key point detection and recognition. The process is as follows:

[0022] Step 4.1: Change the feature extraction algorithm of the bottleneck layer of the backbone network of the face key point detection algorithm to a lightweight network structure. The present invention adopts the GhostBottle network structure and adjusts the number of features of each layer. After the adjustment, the overall calculation amount and parameter amount of the model are smaller, which is more conducive to the use of the detection algorithm in embedded devices and improves the detection speed. Compared with the image recognition algorithm, the face key point algorithm is more accurate for face recognition. High-precision key point detection can be performed when there is partial occlusion on the driver's face or the driver's side face, head up, head down, etc., and the overlap degree of the detection frame after subsequent mobile phone recognition is compared.

[0023] Step 4.2: Use the target image with the driver detection frame mark in step 3.3 as the input of the face key point detection algorithm, perform the driver's face key point detection, retain the mouth key points, connect the key points, and form an accurate mouth area. When the key points are missing during the detection, such as the driver's side face, the formed mouth area is enlarged with its center point as the scaling origin.

[0024] Step 5: Determine the operation of answering or making a phone call. Compare the mouth area detected in step 4.2 with the real-time parameters of the marked phone detection frame. If overlap occurs and the overlap time exceeds the set threshold, the driver is determined to be the one who answers or makes a phone call, and the detection frame image at the time of judgment is captured for manual judgment. The driver's face image is also captured to facilitate subsequent combination with other equipment for driver face comparison.

[0025] The second aspect of the present invention relates to a device for detecting whether a driver receives or makes a phone call in a vehicle based on multi-module facial recognition, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for detecting whether a driver receives or makes a phone call in a vehicle based on multi-module facial recognition of the present invention.

[0026] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method of detecting a driver making or receiving a mobile phone call based on multi-module face recognition in a vehicle of the present invention.

[0027] The panoramic images, images of the driver answering or making phone calls, and images of the driver's face captured by the driver's phone call detection algorithm can all be transmitted to the monitoring platform, replacing manual 24-hour uninterrupted detection. Manual work only needs to make a secondary judgment based on the detected images, locate the person answering or making phone calls in a timely manner, and impose penalties or persuasion.

[0028] The working principle of the present invention is as follows: the target detection algorithm, the target tracking algorithm, and the face key point detection algorithm cooperate with each other. The target detection algorithm completes the detection of mobile phones and driver faces; the target tracking algorithm tracks the detected targets in real time. When a mobile phone is detected, a mark count distinction is made, and a mark count distinction is made for the driver detection frame that overlaps with the detection frame, thereby narrowing the confirmation range for answering and making phone calls. The face key point detection algorithm performs driver face key point detection on the driver detection frame containing the mark count, determines the mouth area, and compares the overlap with the mobile phone detection frame. When the overlap is completed and the overlap time exceeds the set threshold, it is determined to be the person who answers or makes phone calls. Under the premise of ensuring the detection speed, the detection of small targets such as mobile phones and the overlap detection of the mouth area of ​​the face and the mobile phone are completed, which greatly reduces the labor cost and only requires a secondary judgment on the detected image content.

[0029] The advantages of the present invention are as follows: an improved method is proposed for the initial target detection algorithm and the face key point detection algorithm, thereby reducing the number of parameters and the amount of calculation of the overall detection algorithm, and at the same time improving the detection performance when the driver and the mobile phone overlap or are blocked, and the small target detection performance of the mobile phone; by recognizing the mouth area of ​​the driver when answering or making a call on the mobile phone and calculating the overlap degree of the mobile phone target, the driver's action of answering or making a call on the mobile phone is confirmed, thereby improving the detection accuracy of the person who makes a call on the mobile phone. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the YOLOv5 model structure of the present invention;

[0031] Figure 2 It is a schematic diagram of the network structure of the YOLOv5 model of the present invention;

[0032] Figure 3 It is a schematic diagram of the Ghostnet structure of the present invention;

[0033] Figure 4 It is the basic framework structure of the SiameseFC target tracking algorithm of the present invention;

[0034] Figure 5 is a schematic diagram of the overlap of the mobile phone detection frame and the driver detection frame of the present invention;

[0035] Figure 6 It is a schematic diagram of the improved PFLD network structure of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0037] Example 1

[0038] The method for detecting the driver's in-car phone calls based on multi-module face recognition improves the target detection algorithm and the face key point detection algorithm, reduces the number of parameters and calculations of the two algorithms, is more suitable for the application of embedded devices, and improves the detection accuracy of small mobile phone targets, and reduces the missed detection rate under occlusion. Since the target tracking algorithm is only used to locate and track the detected targets, it does not change. The specific steps include the following:

[0039] Step 1: Obtain an image dataset. The in-car camera or other visual device records the video in real time, transmits the video stream to an embedded device or to the cloud, and directly uses the received video as the input of the trained target detection algorithm for real-time detection. The target detection algorithm uses the YOLOv5 (You Only Look Once v5) target detection algorithm, and improves the model such as Figure 1 As shown in the figure, the model network structure is Figure 2 As shown. It can be divided into the backbone network (Backbone), enhanced feature extraction network (Neck), and detection network (Head). Figure 1 Trans represents the Transformer algorithm structure (attention mechanism algorithm); Ghost_N is the GhostBottle layer, which means that this layer contains N Ghostnet algorithm structures. The remaining structure representation methods are the same as the original YOLOv5. The specific process is as follows:

[0040] Step 1.1: Change the feature extraction algorithm of the bottleneck layer of the target detection algorithm backbone network to a lightweight network structure. The present invention adopts the Ghostnet network structure. The Ghostnet structure is as follows: Figure 3 As shown. Figure 3 It can be seen that Ghostnet first performs ordinary convolution on the input image;

[0041] The calculation formula for full convolution is:

[0042]

[0043] Then perform linear transformation (φ);

[0044] The calculation formula for the output vector of the linear transformation is:

[0045] y=W*x+b (2)

[0046] Where x is the input vector, W is the weight matrix, and b is the bias term.

[0047] Finally, the Ghost feature map obtained by linear transformation is concatenated with the feature map after the initial convolution (identity operation) to obtain the final result.

[0048] Theoretical speedup refers to the theoretical change in the running time of a computer program or algorithm when using multiple processors or computing resources. Theoretical compression refers to the ratio of the model storage space occupied by reducing the number of network parameters.

[0049] The theoretical speed of ratio calculation formula is:

[0050]

[0051] The theoretical compression ratio is calculated as follows:

[0052]

[0053] The present invention adopts the improved YOLOv5 target detection algorithm to make the algorithm more scalable during execution, improve the operation speed, and have a suitable compression ratio according to the model size and accuracy.

[0054] The calculation formula for the number of parameters of each convolutional layer is:

[0055] params=C o *(k w *k h *C i +1) (5)

[0056] The calculation formula for the amount of computation of each convolutional layer is:

[0057] FLOPs = [(C i *k w *k h )+(C i *k w *k h -1)+1]*C0*W*H (6)

[0058] In the formula, C0 represents the number of output channels, C i Indicates the number of input channels, k w represents the convolution kernel width, k h Represents the height of the convolution kernel, W and H represent the length and width of the feature map respectively.

[0059] After the Ghostnet network structure is introduced into the present invention, the feature extraction capability is enhanced while satisfying the same number of channels, which greatly reduces the amount of calculation and parameters of the overall model, facilitates the use of the detection algorithm in embedded devices, and improves the detection speed.

[0060] Step 1.2: Add the Transformer attention mechanism algorithm to the end of the backbone network of the target detection algorithm. Figure 1 It can be seen that the attention mechanism algorithm is added at the end of the backbone network.

[0061] The core of Transformer is the self-attention mechanism, in which each element in the input sequence is input and a weight is calculated for each element. This weight measures the relevance of the current element to other elements. That is, for each element in the input sequence, its dot product with other elements is calculated, and then normalized with the softmax function to obtain a weight, which represents the relative importance of each element to other elements. Finally, the normalized weight is multiplied by the corresponding value vector, and the new element is obtained by summing all elements.

[0062] Q=W Q X, K = W K X, V = W V X (7)

[0063] Where Q, K, and V are the query, key, and value obtained by linear transformation of X, respectively, and the weight matrix W Q ,W K ,W V Can be adjusted dynamically through attention.

[0064] The calculation formula of the attention output vector is:

[0065]

[0066] Written in matrix form:

[0067] The Transformer attention mechanism algorithm strengthens feature extraction by assigning weights to each channel and calculating the importance of each position. It can learn adaptive attention distribution from different feature spaces, enhance the feature extraction capability for occluded objects, and reduce the missed detection rate.

[0068] Step 1.3: Since multiple downsampling will produce multiple convolution processes, the image will continue to shrink, causing the small target features in the image to disappear, resulting in the model network being unable to learn the corresponding features, and small targets such as mobile phones are prone to missed detection; and upsampling is a deconvolution process, which is to enlarge the image. The target detection algorithm retains the original detection dimension, does not reduce the original scale feature learning ability, adds connection layers, convolution layers, bottleneck layers, and performs upsampling and downsampling again, and inputs the same-scale feature layer that retains more small target feature information in the backbone network into the connection layer for feature fusion operations. The newly added detection layer is more sensitive to small target features, which is beneficial for small target detection on mobile phones. Due to the increase in the detection scale diversity of the detection layer, the present invention uses a genetic algorithm to analyze and calculate the custom data set to obtain the predicted anchor frames required for each scale of the data set.

[0069] Step 1.4: Use the concat connection layer to fuse feature layers of different levels but the same scale to reduce feature loss, and input the multi-layer calculation results of the target detection algorithm's backbone network into the enhanced feature extraction network once or multiple times. The enhanced feature extraction network is divided into two parts, the bottom-up structure and the top-down structure. The bottom-up part fuses two features of the same scale; when it comes to the top-down structure, three features of the same scale are fused using a jump connection method, and output after passing through the bottleneck layer to predict multiple targets of different scales. The use of a bidirectional feature pyramid network structure makes the prediction network more sensitive to targets of different sizes, improves the overall model detection capability, and reduces the missed detection rate and false detection rate.

[0070] Step 2: The target detection algorithm detects the mobile phone and the driver's face in real time. The target tracking algorithm uses the prediction box recognized by the target detection algorithm as input to track the target in real time. The present invention adopts the SiameseFC (Siamese Fully-Connected Neural Network) target tracking algorithm, such as Figure 4 , the input is the template image z and the search image x, and the output of the convolutional neural network is and After cross-correlation calculation*, we get the score map, which reflects and The similarity relationship of each corresponding part in the image is the same. The greater the similarity, the more likely it is that they are the same object. In essence, it is to find the position of the template image z on the search image x to achieve single target tracking. The target tracking algorithm is used to count and track mobile phones. The process is as follows:

[0071] Step 2.1: When a mobile phone is detected, the target tracking algorithm is first used to distinguish the mobile phone mark count, such as mobile phone-1, and a single frame image of the first marked target is captured as a panoramic image for subsequent manual identification.

[0072] Step 2.2: Output the real-time tracking to obtain the mobile phone’s tag value and the real-time parameters of the detection box (x c ,y c ,w c ,h c ), x c ,y c Indicates the center point of the mobile phone detection frame, w c ,h c Indicates the width and height of the mobile phone detection frame. If the real-time parameters of the mobile phone detection frame do not change within a certain period of time, the target object is considered to be abandoned or unused, and the mobile phone detection frame mark value is retained, but it is not used as a comparison target in steps 3.2 and 5 until the real-time parameters of the target object change.

[0073] Step 3: Driver detection and technology. The process is as follows:

[0074] Step 3.1: The target detection algorithm simultaneously marks the driver and tracks the driver in real time. After outputting the real-time tracking, the driver’s marking value and the real-time parameters of the detection box (x R ,y R ,w R ,h R ), x R ,y R represents the center point of the driver detection box, w R ,h R Indicates the width and height of the driver detection box.

[0075] Step 3.2: When the phone detection frame overlaps with the driver detection frame, Figure 5 As shown, overlapping driver detection box marker counts are distinguished, such as driver-1.

[0076] Step 3.3: Simultaneously return the driver's mark value and the real-time parameters of the detection box (x R ,y R ,w R ,h R ), x R ,y R represents the center point of the driver detection box, w R ,h R Indicates the width and height of the driver detection frame. When the marked mobile phone detection frame ends overlapping with the driver detection frame, the driver detection frame mark is discarded and the real-time parameters are stopped from being sent back.

[0077] Step 4: Face key point detection and recognition. The process is as follows:

[0078] Step 4.1: Face key point detection algorithm uses PFLD (A Pratical Facial Landmark Detector) face key point detection algorithm. The improved model is as follows Figure 6 As shown. The feature extraction algorithm of the bottleneck layer of the backbone network of the face key point detection algorithm is changed to a lightweight network structure. The present invention adopts the GhostBottle network structure, such as step 1.1, and adjusts the number of network structures for each layer. The overall calculation amount and parameter amount of the adjusted model are smaller, which is more conducive to the use of the detection algorithm in embedded devices, improving the detection speed and accuracy. Compared with the image recognition algorithm, the facial key point algorithm is more accurate for face recognition. High-precision key point detection can be performed when there is partial occlusion on the driver's face or the driver's side face, raising his head, lowering his head, etc., and the overlap with the detection frame after subsequent mobile phone recognition is compared.

[0079] Step 4.2: Use the target image with the driver detection frame mark in step 3.3 as the input of the face key point detection algorithm, perform the driver's face key point detection, retain the mouth key points, connect the key points, and form an accurate mouth area. When the key points are missing during the detection, such as the driver's side face, the mouth area formed is enlarged by 50% with its center point as the scaling origin.

[0080] Step 5: Determine whether the driver is making or receiving phone calls. The mouth area detected in step 4.2 is compared with the real-time parameters of the marked phone detection frame. If an overlap occurs and the overlap time exceeds a set threshold, it is determined that the driver is making or receiving phone calls.

[0081] Numerous studies have shown that the minimum time for looking at a mobile phone is usually 3 seconds, so the threshold is set as:

[0082] t max =3s(10)

[0083] At the same time, the detection frame image is captured for manual judgment. The driver's face image is also captured to facilitate subsequent combination with other equipment for driver face comparison.

[0084] The panoramic images, images of drivers making phone calls, and images of drivers' faces captured by the driver's phone call detection algorithm can all be transmitted to the monitoring platform, replacing manual 24-hour uninterrupted detection. Manual detection only needs to be performed based on the detected images to timely locate the driver making phone calls and impose penalties or persuasion. Compared with traditional algorithms, the detection of small targets is improved, the missed detection rate of target objects with occlusion is reduced, and the detection is more accurate.

[0085] Example 2

[0086] The present embodiment relates to a device for detecting whether a driver receives or makes a phone call in a vehicle based on multi-module facial recognition, comprising a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the method for detecting whether a driver receives or makes a phone call in a vehicle based on multi-module facial recognition of embodiment 1.

[0087] Example 3

[0088] The present embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for detecting a driver's in-vehicle phone call receiving or making a call based on multi-module face recognition of Embodiment 1 is implemented.

[0089] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0090] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting a driver's in-car phone call based on multi-module face recognition, specifically comprising the following steps: Step 1: Obtain an image dataset; use the in-vehicle camera or other visual device to record in real time, transmit the video stream to an embedded device or to the cloud, and use the received video directly as the input of the trained target detection algorithm for real-time detection; Step 2: The target detection algorithm performs real-time detection of mobile phones and drivers; The target tracking algorithm uses the predicted box identified by the target detection algorithm as input to track the target in real time. The SiameseFC target tracking algorithm is used to count and track mobile phones. Step 3: The target detection algorithm marks the driver and tracks him in real time. When the mobile phone detection frame overlaps with the driver detection frame, the overlapping driver detection frame mark counts are distinguished, and the driver mark value and the real-time parameters of the driver detection frame are returned. When the marked mobile phone detection frame ends overlapping with the driver detection frame, the driver detection frame mark is discarded and the real-time parameters are stopped from being returned. Step 4: Identify facial key point detection; Step 5: Determine the operation of making and receiving calls on the mobile phone; Step 1 specifically includes: Step 1.1: Change the feature extraction algorithm of the bottleneck layer of the target detection algorithm backbone network to a lightweight network structure, using GhostBottle composed of the end-side neural network architecture GhostNet; Under the premise of meeting the same number of channels, the feature extraction capability is enhanced, and the calculation amount and parameter amount of the overall model are greatly reduced, which is conducive to the use of the detection algorithm in embedded devices and improves the detection speed; Step 1.2: Add an attention mechanism algorithm to the end of the backbone network of the target detection algorithm to enhance feature extraction by assigning weights to each channel; learn adaptive attention distribution from different feature spaces; Step 1.3: The target detection algorithm retains the original detection dimension, does not reduce the original scale feature learning ability, adds connection layer, convolution layer, bottleneck layer, and performs upsampling and downsampling again, and inputs the same scale feature layer that retains more small target feature information in the backbone network into the connection layer for feature fusion operation; due to the increase in the detection scale diversity of the detection layer, a genetic algorithm is used to analyze and calculate the custom data set to obtain the predicted anchor boxes required for each scale of the data set; Step 1.4: Use the concat connection layer to fuse feature layers of different levels but the same scale to reduce feature loss, and input the multi-layer calculation results of the target detection algorithm's backbone network into the enhanced feature extraction network once or multiple times; the enhanced feature extraction network is divided into two parts, a bottom-up structure and a top-down structure; the bottom-up part fuses two features of the same scale; when it comes to the top-down structure, three features of the same scale are fused using a skip connection method, and are output after passing through the bottleneck layer to predict multiple targets of different scales; Step 2 specifically includes: Step 2.1: When a mobile phone is detected, the target tracking algorithm is first used to mark, count and distinguish the mobile phones. At the same time, a single frame image of the first marked target object is captured and retained as a panoramic image for subsequent manual identification; Step 2.2: Output the real-time tracking to obtain the mobile phone’s tag value and the real-time parameters of the detection box (x c ,y c ,w c ,h c ), x c ,y c Indicates the center point of the mobile phone detection frame, w c ,h c Indicates the width and height of the mobile phone detection frame; Step 2.3: If the real-time parameters of the mobile phone detection frame do not change within a certain period of time, the target object is deemed to be abandoned or unused, and the tag value of the mobile phone is retained, but it is not used as a comparison target in steps 3.2 and 5 until the real-time parameters of the target object change; Step 3 specifically includes: Step 3.1: The target detection algorithm simultaneously marks the driver and tracks the driver in real time. After outputting the real-time tracking, the driver’s marking value and the real-time parameters of the detection box (x R ,y R ,w R ,h R ), x R ,y R represents the center point of the driver detection box, w R ,h R Indicates the width and height of the driver detection box; Step 3.2: When the mobile phone detection frame overlaps with the driver detection frame, the overlapping driver detection frame mark counts are distinguished; Step 3.3: Simultaneously return the driver's tag value and the driver's detection frame real-time parameters (x R ,y R ,w R ,h R ); When the marked mobile phone detection frame ends overlapping with the driver detection frame, the driver detection frame mark is discarded and the real-time parameters are stopped from being sent back; Step 4 specifically includes: Step 4.1: Change the feature extraction algorithm of the bottleneck layer of the backbone network of the face key point detection algorithm to a lightweight network structure, adopt the GhostBottle network structure, and adjust the number of features of each layer; the face key point detection algorithm is more accurate for face recognition than the image recognition algorithm. High-precision key point detection can be performed when the driver's face is partially blocked or the driver's face is sideways, raised, or lowered, and the overlap with the detection frame after subsequent mobile phone recognition is used for comparison; Step 4.2: Use the target image with the driver detection frame mark in step 3.3 as the input of the face key point detection algorithm, detect the driver's face key points, retain the mouth key points, connect the key points to form an accurate mouth area; when the key points are missing during detection, the formed mouth area is enlarged with its center point as the scaling origin; Step 5 specifically includes: The mouth area detected in step 4.2 is compared with the real-time parameters of the marked mobile phone detection frame. If overlap occurs and the overlap time exceeds the set threshold, the driver is judged as the person who answers or makes a call on the phone, and the detection frame image at the time of judgment is captured for manual judgment; and the driver's face image is captured to facilitate subsequent combination with other equipment for driver face comparison.

2. A driver's mobile phone call detection device based on multi-module face recognition in a car, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for detecting the driver's in-car mobile phone reception and making calls based on multi-module face recognition as described in claim 1.

3. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by the processor, the method for detecting a driver's in-car phone call based on multi-module face recognition as described in claim 1 is implemented.

Images