Monitoring device for detecting driver mobile phone use behavior and control method thereof

Abstract

The present application relates to the technical field of driver driving monitoring, and particularly relates to a control method of a device for detecting driver mobile phone use behavior monitoring, the device is arranged in a vehicle, and the device comprises a camera unit, an image temporary storage unit, an image processing unit and an output unit, and the method comprises the following steps: acquiring a plurality of driver seat image frames of the vehicle through the camera unit. It is judged whether a suspected mobile phone use behavior of the driver is detected according to a first frame of the driver seat image frames through the image processing unit. When the image processing unit detects a suspected mobile phone use behavior from the first frame, the image processing unit traces back a plurality of second frames within 2 seconds before the first frame to judge whether the hand of the driver in the second frame contains a mobile phone, and the number of the second frames in which the hand of the driver contains the mobile phone is calculated. When it is judged that the number of the second frames is greater than a threshold frame value, the image processing unit outputs prompt information. Otherwise, the image processing unit does not output prompt information.

Description

Technical Field

[0001] This invention relates to the field of driver monitoring technology, and in particular to a monitoring device and control method for detecting a driver's mobile phone usage behavior. Background Technology

[0002] Existing driver monitoring systems detect driver behavior through cameras and sensors, but they are prone to misinterpreting certain actions, such as touching the head, reaching out, or shaving, as mobile phone use. Furthermore, existing systems lack high accuracy and cannot dynamically adjust to different driving environments, thus affecting the accuracy of data detection.

[0003] In view of this, the present invention relates to a monitoring device and control method that can improve the accuracy of detecting drivers' mobile phone use and can adapt to different driving environments. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention relates to a monitoring device for detecting driver mobile phone usage, applicable to a vehicle, comprising a camera unit, an image storage unit, an image processing unit, and an output unit. The camera unit captures multiple driver-side image frames of the vehicle. The image storage unit temporarily stores these driver-side image frames. The image processing unit determines whether suspected mobile phone usage by a driver is detected based on a first frame of the driver-side image frames. When the image processing unit detects suspected mobile phone usage from the first frame, it acquires multiple second frames of the driver-side image frames within two seconds prior to the first frame through the image storage unit. The image processing unit determines whether the driver's hand contains a mobile phone in each of the second frames and calculates the number of second frames in which the driver's hand contains the mobile phone. When the number of second frames in which the driver's hand contains the mobile phone exceeds a threshold frame value, the image processing unit outputs a prompt message. When the number of second frames in which the driver's hand contains the phone is determined to be no greater than the threshold frame value, the image processing unit does not output the prompt information.

[0005] Specifically, when the number of second frames acquired within the first 2 seconds is 10, the threshold frame value is set to one of 5, 6, 7 or 8.

[0006] Specifically, the image processing unit detects the position and pose of one hand in each of the second frames according to a convolutional neural network model, and generates hand feature data based on the hand position and the hand pose.

[0007] Specifically, the image processing unit calculates a mobile phone presence probability value based on the hand feature data of each of the second frames, where the mobile phone presence probability value is the probability that the hand feature data of the second frame contains the mobile phone. The image processing unit determines whether the mobile phone presence probability values ​​of the second frames are greater than a threshold mobile phone probability value to determine whether the driver's hand in each second frame contains the mobile phone. When the image processing unit determines that the mobile phone presence probability values ​​of the second frames are greater than the threshold mobile phone probability value, it determines that the driver's hand in the second frame contains the mobile phone. When the image processing unit determines that the mobile phone presence probability values ​​of the second frames are not greater than the threshold mobile phone probability value, it determines that the driver's hand in the second frame does not contain the mobile phone. Wherein, when the mobile phone presence probability value is 1, the threshold mobile phone probability value is set to one of 0.5, 0.6, or 0.7.

[0008] Specifically, the image processing unit adjusts the threshold frame value and the threshold mobile phone probability value according to the driver's driving environment, lighting conditions, and driver behavior patterns.

[0009] This invention also provides a control method for a monitoring device for detecting driver mobile phone usage behavior. The monitoring device is installed in a vehicle and includes a camera unit, an image storage unit, an image processing unit, and an output unit. The method includes: capturing multiple driver's seat image frames of the vehicle through the camera unit; determining, through the image processing unit, whether a driver's suspected mobile phone usage behavior is detected based on a first frame of one of the driver's seat image frames; when the image processing unit detects the suspected mobile phone usage behavior from the first frame, acquiring multiple second frames of the driver's seat image frames within 2 seconds prior to the first frame through the image storage unit; determining, in each of the second frames, whether the driver's hand contains a mobile phone, and calculating the number of second frames in which the driver's hand contains the mobile phone; when the number of second frames in which the driver's hand contains the mobile phone is determined to be greater than a threshold frame value, the image processing unit outputs a prompt message; when the number of second frames in which the driver's hand contains the mobile phone is determined to be less than the threshold frame value, the image processing unit does not output the prompt message.

[0010] Specifically, it also includes setting the threshold frame value to one of 5, 6, 7 or 8 when the number of second frames acquired within 2 seconds prior to the first frame is 10.

[0011] Specifically, it also includes: detecting the position and pose of one hand in each of the second frames using the image processing unit based on a convolutional neural network model, and generating hand feature data based on the hand position and the hand pose.

[0012] Specifically, it also includes: calculating a mobile phone presence probability value based on the hand feature data of each of the second frames using the image processing unit, wherein the mobile phone presence probability value is the probability that the hand feature data of the second frame contains the mobile phone. The image processing unit determines whether the mobile phone presence probability values ​​of the second frames are greater than a threshold mobile phone probability value to determine whether the driver's hand in each second frame contains the mobile phone. When the image processing unit determines that the mobile phone presence probability values ​​of the second frames are greater than the threshold mobile phone probability value, it determines that the driver's hand in the second frame contains the mobile phone. When the image processing unit determines that the mobile phone presence probability values ​​of the second frames are not greater than the threshold mobile phone probability value, it determines that the driver's hand in the second frame does not contain the mobile phone. Wherein, when the mobile phone presence probability value is 1, the threshold mobile phone probability value is set to one of 0.5, 0.6, or 0.7.

[0013] Specifically, it also includes adjusting the threshold frame value and the threshold mobile phone probability value through the image processing unit based on the driver's driving environment, lighting conditions and driver behavior patterns.

[0014] Compared with existing technologies, the monitoring device of the present invention, when determining whether the driver is suspected of using a mobile phone in the first frame of the driver's seat image frame through the image processing unit, analyzes the second frame within 2 seconds before the first frame to determine whether the driver's hand contains the mobile phone, and calculates the number of second frames in which the driver's hand contains the mobile phone. Simultaneously, it determines whether the number of such second frames exceeds a threshold frame value, and if so, the image processing unit outputs a prompt message. Through this method, it can confirm whether the suspected mobile phone use behavior in the first frame is a misidentification of the driver's actions or whether the driver is actually using a mobile phone. Attached Figure Description

[0015] Other features and effects of the present invention will be clearly presented in the embodiments with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram illustrating a monitoring device according to an embodiment of the present invention; Figure 2 This is a display of the present invention. Figure 1 A flowchart of the control method for the monitoring device in the embodiment. Detailed Implementation

[0016] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when appearing in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of the apparatus and methods within the scope of the present invention's patent application.

[0017] Figure 1 This is a schematic diagram of a monitoring device 10 for detecting driver mobile phone use according to an embodiment of the present invention. The monitoring device 10 of the present invention is applicable to vehicles. The monitoring device 10 according to an embodiment of the present invention includes a camera unit 110, an image storage unit 120, an image processing unit 130, and an output unit 140. The image processing unit 130 determines whether suspected mobile phone use by the driver is detected based on the first frame of the driver's seat image captured by the camera unit 110. The image processing unit 130 then analyzes multiple second frames within two seconds before the first frame to determine if the driver is indeed using a mobile phone, and outputs a prompt message.

[0018] In one embodiment of the present invention, the camera unit 110 may be a camera with video recording or audio recording functions. The setting angle or number of camera units 110 is not particularly limited, as long as it ensures that the camera unit 110 can capture multiple driver's seat image frames of the vehicle. Of course, the driver's seat image frames obtained above can be arbitrary and are not limited by any time or location. The camera unit 110 captures image frames from the driver's seat of the vehicle to capture the driver's hand operations and related data when the driver is driving, moving, or parking. Of course, in this embodiment, the camera unit 110 can also be appropriately adjusted in terms of its mounting height, mounting position, mounting angle, or number according to actual needs, depending on the driver's position in the driver's seat, to meet different requirements.

[0019] In one embodiment of the present invention, the image temporary storage unit 120 may be a memory or storage device with storage or backup functions. It is only necessary to ensure that the image temporary storage unit 120 can temporarily store the driver's seat image frame and that the image processing unit 130 can retrieve the driver's seat image frame from the image temporary storage unit 120, but it is not limited thereto.

[0020] In one embodiment of the present invention, the image processing unit 130 determines whether suspected mobile phone use by the driver is detected based on the first frame of the driver's seat image frame, and also obtains multiple second frames of the driver's seat image frame within 2 seconds before the first frame via the image storage unit 120. The first and second frames are arbitrarily selected; they are only ordered for ease of explanation in this embodiment and are not specifically designated. Other frame numbers can also be used, but this is not a limitation. It should be noted that in this embodiment, the selection interval between the first and second frames is within the first 2 seconds. If different monitoring devices 10 are used or different testing requirements are met, an appropriate selection interval between the first and second frames can be selected according to the specific situation, but this is not a limitation.

[0021] In one embodiment of the present invention, the image processing unit 130 further determines whether the driver's hand contains a mobile phone in a plurality of second frames, and calculates the number of second frames in which the driver's hand contains a mobile phone. When the number of second frames in which the driver's hand contains a mobile phone is determined to be greater than a threshold frame value, the image processing unit 130 outputs a prompt message. Otherwise, the image processing unit 130 does not output a prompt message, but this is not a limitation. In one embodiment of the present invention, when the number of second frames within 2 seconds before the first frame is 10, the threshold frame value is set to one of 5, 6, 7 or 8, but this is not a limitation.

[0022] In one embodiment of the present invention, the output unit 140 is used to notify or display prompt information to the driver for timely viewing. The output unit 140 may use text, audio, images, or other means to remind or inform the driver, but is not limited thereto. In one embodiment of the present invention, the output unit 140 may be a display screen or an in-vehicle terminal page, but is not limited thereto. It is sufficient that the image processing unit 130 outputs prompt information to the output unit 140 for display when the image processing unit 130 determines that the driver has suspected mobile phone use behavior and backtracks to confirm that the driver has used a mobile phone.

[0023] In detail, in one embodiment of the present invention, when the second frame of the driver's seat image frame within 2 seconds before the first frame is acquired through the image storage unit 120, the hand position and hand posture in the second frame are detected according to the convolutional neural network model to generate hand feature data. The probability value of the presence of a mobile phone is calculated based on the hand feature data of the second frame, wherein the probability value of the presence of a mobile phone is the probability that the hand feature data of the second frame contains a mobile phone. In other words, the image processing unit 130 needs to classify and detect the hand feature data in the second frame to determine whether the driver's hand in the second frame contains a mobile phone, and simultaneously determine whether the probability value of the presence of a mobile phone in the second frame is greater than a threshold mobile phone probability value. Wherein, when the probability value of the presence of a mobile phone is 1, the threshold mobile phone probability value is set to one of 0.5, 0.6, or 0.7, but is not limited to this. This can exclude certain actions of the driver (e.g., touching the head, shaving, snapping fingers, etc.) that might be mistakenly identified as the driver using a mobile phone, but is not limited to this.

[0024] In one embodiment of the present invention, when the image processing unit 130 determines that the probability value of a mobile phone presence in the second frame is greater than a threshold probability value, the image processing unit 130 determines that the driver's hand in the second frame contains a mobile phone, and calculates the number of second frames in which the driver's hand contains a mobile phone. When the number of second frames in which the driver's hand contains a mobile phone is greater than the threshold frame value, the image processing unit 130 outputs a prompt message. Conversely, when the number of second frames in which the driver's hand contains a mobile phone is not greater than the threshold frame value, the image processing unit 130 does not output a prompt message. When the image processing unit 130 determines that the probability value of a mobile phone presence in the second frame is not greater than the threshold probability value, the image processing unit 130 determines that the driver's hand in the second frame does not contain a mobile phone. Through the above operations, the image processing unit 130 can determine whether the driver has suspected mobile phone use behavior, and by reviewing previous frames, can more precisely determine whether the driver has definitely used a mobile phone, thereby confirming whether the driver has engaged in certain mobile phone use behaviors, and using this to prompt or inform the driver, but is not limited to this.

[0025] In one embodiment of the present invention, the image processing unit 130 further adjusts the threshold frame value and the threshold mobile phone probability value according to the driver's driving environment, lighting conditions and driver behavior pattern, but is not limited thereto.

[0026] The monitoring device 10 of the present invention, through the aforementioned operation process, uses the image processing unit 130 to determine, in the first frame of the driver's seat image frame, whether the driver is suspected of using a mobile phone. Then, the image processing unit 130 back-analyzes the second frame within two seconds prior to the first frame to determine if the driver's hand is holding a mobile phone, and calculates the number of second frames in which the driver's hand is holding a mobile phone. When the number of such second frames is greater than a threshold frame value, the image processing unit 130 outputs a prompt message. Conversely, when the number of such second frames is not greater than the threshold frame value, the image processing unit 130 does not output a prompt message. By back-analyzing the relevant data of the second frames and performing calculations and comparisons, it can confirm whether the suspected mobile phone use behavior in the first frame is a misidentification of certain actions or a situation where the driver is actually using a mobile phone, but it is not limited to this.

[0027] Please see Figure 2 According to the present invention Figure 1 The control method of the monitoring device 10 in this embodiment will be explained.

[0028] In step S1, the camera unit 110 captures multiple driver's seat image frames of the vehicle. Specifically, the camera unit 110 captures multiple driver's seat image frames of the vehicle.

[0029] In step S2, the image processing unit 130 determines whether it detects any suspected mobile phone use by the driver based on the first frame of the driver's seat image frame. Specifically, after executing step S1, if the image processing unit 130 determines that the first frame detects suspected mobile phone use by the driver, then step S3 is executed. If the image processing unit 130 determines that the first frame does not detect suspected mobile phone use by the driver, then step S1 is executed.

[0030] In step S3, the image processing unit 130 calculates the probability value of the presence of a mobile phone in the second frame. Specifically, after performing step S2, when the image processing unit 130 detects suspected mobile phone use by the driver from the first frame, it acquires the second frame of the driver's seat image frame within 2 seconds before the first frame through the image storage unit 120, and detects the hand position and hand posture in the second frame through a convolutional neural network model. Based on the hand position and hand posture, hand feature data is generated, and the probability value of the presence of a mobile phone is calculated based on the hand feature data of the second frame. The probability value of the presence of a mobile phone is the probability value that the hand feature data packet of the second frame contains a mobile phone.

[0031] In step S4, the image processing unit 130 determines whether the probability value of the mobile phone presence in the second frame is greater than the threshold probability value. Specifically, after executing step S3, if the image processing unit 130 determines that the probability value of the mobile phone presence in the second frame is greater than the threshold probability value, then step S5 is executed. If the image processing unit 130 determines that the probability value of the mobile phone presence in the second frame is not greater than the threshold probability value, then step S1 is executed. Wherein, when the mobile phone presence probability value is 1, the threshold probability value is set to one of 0.5, 0.6, or 0.7.

[0032] In step S5, the image processing unit 130 calculates the number of second frames in which the driver's hand contains a mobile phone. Specifically, after performing step S4, when the image processing unit 130 determines that the probability value of the presence of a mobile phone in the second frame is greater than the threshold probability value, the image processing unit 130 determines that the driver's hand in the second frame contains a mobile phone and calculates the number of second frames in which the driver's hand contains a mobile phone.

[0033] In step S6, the image processing unit 130 determines whether the number of the second frames is greater than the threshold frame value. Specifically, after executing step S5, if the image processing unit 130 determines that the number of second frames containing the mobile phone in the driver's hand is greater than the threshold frame value, then step S7 is executed. If the image processing unit 130 determines that the number of second frames containing the mobile phone in the driver's hand is not greater than the threshold frame value, then step S8 is executed. Wherein, when the number of second frames within the first 2 seconds of acquiring the first frame is 10, the threshold frame value is set to one of 5, 6, 7, or 8.

[0034] In step S7, the image processing unit 130 outputs prompt information. Specifically, after completing step S6, when the image processing unit 130 determines that the number of the second frames is greater than the threshold frame value, the image processing unit 130 outputs prompt information to the output unit 140 to remind or inform the driver.

[0035] In step S8, the image processing unit 130 does not output any prompt information. Specifically, after executing step S6, when the image processing unit 130 determines that the number of the second frames is not greater than the threshold frame value, the image processing unit 130 does not output any prompt information to the output unit 140.

[0036] It can be understood that the monitoring device 10 of the present invention, when the image processing unit 130 determines whether the driver has detected suspected mobile phone use behavior in the first frame of the driver's seat image frame, analyzes the second frame within 2 seconds before the first frame to determine whether the driver's hand contains the mobile phone, and calculates the number of second frames in which the driver's hand contains the mobile phone. Simultaneously, if the number of such second frames exceeds a threshold frame value, the image processing unit 130 outputs a prompt message. Through the above method, it is possible to confirm whether the suspected mobile phone use behavior in the first frame is a misidentification of certain actions or a situation where the driver is actually using the mobile phone.

[0037] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A monitoring device for detecting a driver's mobile phone usage behavior, applicable to a vehicle, characterized in that, include: A camera unit is used to capture multiple driver's seat image frames of the vehicle; An image storage unit is used to temporarily store these driver's seat image frames; An image processing unit; and One output unit; The image processing unit determines whether a driver's suspected mobile phone use behavior is detected based on the first frame of one of the driver's seat image frames. When the image processing unit detects the suspected mobile phone use behavior in the first frame, it obtains multiple second frames of the driver's seat image frames within 2 seconds before the first frame through the image storage unit. The image processing unit determines whether the driver's hand contains a mobile phone in each second frame and counts the number of second frames in which the driver's hand contains the mobile phone. When the number of second frames in which the driver's hand contains the phone exceeds a threshold frame value, the image processing unit outputs a prompt message; and When the number of second frames in which the driver's hand contains the phone is determined to be no greater than the threshold frame value, the image processing unit does not output the prompt information.

2. The monitoring device according to claim 1, characterized in that, When the number of second frames acquired within 2 seconds prior to the first frame is 10, the threshold frame value is set to one of 5, 6, 7, or 8.

3. The monitoring device according to claim 1, characterized in that, The image processing unit detects the position and posture of one hand in each of the second frames using a convolutional neural network model, and generates hand feature data based on the hand position and posture.

4. The monitoring device according to claim 3, characterized in that, The image processing unit calculates a probability value of the presence of a mobile phone based on the hand feature data of each of the second frames, wherein the probability value of the presence of a mobile phone is the probability value of the hand feature data packet of the second frame containing the mobile phone. The image processing unit determines whether the probability value of the presence of the mobile phone in the second frames is greater than a threshold mobile phone probability value, so as to determine whether the driver's hand in each second frame contains the mobile phone. When the image processing unit determines that the probability value of the mobile phones in the second frame is greater than the threshold probability value, it determines that the driver's hand in the second frame contains the mobile phone. When the image processing unit determines that the probability value of the presence of these mobile phones in the second frame is not greater than the threshold mobile phone probability value, it determines that the driver's hand in the second frame does not contain the mobile phone; and Specifically, when the probability of obtaining these mobile phones is 1, the probability value of the threshold mobile phone is set to one of 0.5, 0.6, or 0.

7.

5. The monitoring device according to claim 1, characterized in that, The image processing unit adjusts the threshold frame value and the threshold mobile phone probability value based on the driver's driving environment, lighting conditions, and driver behavior patterns.

6. A control method for a monitoring device for detecting driver mobile phone usage behavior, wherein the monitoring device is installed in a vehicle, and the monitoring device includes a camera unit, an image storage unit, an image processing unit, and an output unit, characterized in that, include: The camera unit captures multiple driver's seat image frames of the vehicle. The image processing unit determines whether a driver is suspected of using a mobile phone based on the first frame of one of the driver's seat image frames. When the image processing unit detects the suspected mobile phone use behavior in the first frame, it obtains multiple second frames of the driver's seat image frames within 2 seconds before the first frame through the image storage unit. The image processing unit determines whether the driver's hand contains a mobile phone in each second frame and counts the number of second frames in which the driver's hand contains the mobile phone. When the number of second frames in which the driver's hand contains the phone exceeds a threshold frame value, the image processing unit outputs a prompt message; and When the number of second frames in which the driver's hand contains the phone is determined to be no greater than the threshold frame value, the image processing unit does not output the prompt information.

7. The control method according to claim 6, characterized in that, Also includes: When the number of second frames acquired within 2 seconds prior to the first frame is 10, the threshold frame value is set to one of 5, 6, 7, or 8.

8. The control method according to claim 6, characterized in that, Also includes: The image processing unit detects the position and posture of one hand in each of the second frames using a convolutional neural network model, and generates hand feature data based on the hand position and posture.

9. The control method according to claim 8, characterized in that, Also includes: The image processing unit calculates the probability value of a mobile phone presence based on the hand feature data of each of the second frames, wherein the probability value of a mobile phone presence is the probability value that the hand feature data packet of the second frame contains the mobile phone. The image processing unit determines whether the probability value of the presence of the mobile phone in the second frame is greater than a threshold mobile phone probability value, so as to determine whether the driver's hand contains the mobile phone in each second frame. When the image processing unit determines that the probability value of the mobile phones in the second frame is greater than the threshold probability value, it determines that the driver's hand in the second frame contains the mobile phone. When the image processing unit determines that the probability value of the presence of these mobile phones in the second frame is not greater than the threshold mobile phone probability value, it determines that the driver's hand in the second frame does not contain the mobile phone; and Specifically, when the probability of obtaining these mobile phones is 1, the probability value of the threshold mobile phone is set to one of 0.5, 0.6, or 0.

7.

10. The control method according to claim 6, characterized in that, Also includes: The image processing unit adjusts the threshold frame value and the threshold mobile phone probability value based on the driver's driving environment, lighting conditions, and driver behavior patterns.

Images