Method and device for detecting the state of using a mobile terminal while driving

Abstract

The present invention provides a state detection method and device based on the use of a mobile terminal while driving, the method comprising: obtaining a driving image within a preset time period; obtaining a first angle of the mobile terminal relative to the driver's head and a gaze angle of the mobile terminal relative to the driver's line of sight based on each frame of the driving image; obtaining a behavior judgment result of each frame of the driving image based on the gaze angle and the first angle; and generating and sending an alarm message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold. The present invention obtains the first angle of the driver's head relative to the mobile terminal and the gaze angle of the driver's line of sight relative to the mobile terminal respectively, and judges whether the driver is looking at the mobile terminal in combination with the first angle and the gaze angle, thereby improving the detection accuracy of the terminal used while driving, and avoiding detection errors caused by detecting the driver's line of sight alone or detecting the direction of the head relative to the mobile terminal.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a state detection method and device based on using a mobile terminal while driving. Background Art

[0002] With the popularization of smart mobile terminals, people are becoming more and more dependent on mobile terminals. Whether it is work, study, or life, mobile terminals are indispensable. Many drivers use mobile terminals while driving and ignore the road conditions ahead, which leads to traffic accidents.

[0003] Currently, in order to realize the behavior detection of the driver, most of them obtain the facial image of the driver's face, and use the line of sight obtained based on facial image recognition to determine whether the driver is actually looking at the mobile terminal.

[0004] Due to this behavior detection method, whether the driver is looking at the mobile terminal is judged only based on the driver's line of sight. If the driver is only looking at the navigation or occasionally glances at the mobile terminal, it is easy to be misjudged as looking at the mobile terminal, resulting in misjudgment or wrong punishment of the driver. Summary of the invention

[0005] The present invention provides a state detection method and device based on using a mobile terminal while driving, so as to solve the defect in the prior art that the behavior detection accuracy is poor due to occasional gaze at the mobile terminal or mis-gaze of the mobile terminal, realize high-precision detection, and avoid false detection or missed detection.

[0006] The present invention provides a state detection method based on using a mobile terminal while driving, comprising: acquiring driving images within a preset time period; acquiring a first angle of a mobile terminal relative to a driver's head and a gaze angle of the mobile terminal relative to the driver's line of sight based on each frame of the driving image; obtaining a behavior judgment result of each frame of the driving image based on the gaze angle and the first angle; and generating and sending an alarm message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0007] According to a state detection method based on using a mobile terminal while driving provided by the present invention, the behavior judgment result of each frame of the driving image is obtained according to the gaze angle and the first angle, including: for each frame of the driving image, judging whether the deviation between the gaze angle and the first angle is less than a second preset threshold value; if so, judging that the current frame is playing with the mobile terminal, and recording the judgment result as 1; otherwise, judging that the current frame is not playing with the mobile terminal, and recording the judgment result as 0.

[0008] According to a state detection method based on using a mobile terminal while driving provided by the present invention, the first angle of the mobile terminal relative to the driver's head is obtained based on each frame of driving image, including: based on each frame of driving image, selecting the direction of the driver's line of sight facing forward as the reference line direction; based on pre-calibrated camera parameters, determining the first direction of the mobile terminal relative to the driver's head; and obtaining the first angle of the current frame of driving image according to the first direction and the reference line direction.

[0009] According to a state detection method based on using a mobile terminal while driving provided by the present invention, the obtaining of the gaze angle of the mobile terminal relative to the driver's line of sight includes: selecting the direction in which the driver is looking straight ahead as a reference line direction; determining the driver's gaze direction based on each frame of driving image; and obtaining the gaze angle of the current frame of driving image based on the pre-selected reference line direction and the gaze direction.

[0010] According to a state detection method based on using a mobile terminal while driving provided by the present invention, the method determines the gaze direction of the driver based on each frame of driving image, including: based on each frame of driving image, identifying facial features in the driving image, and obtaining eye sockets and pupils in the facial image; determining a first sight line direction according to the area of ​​the identified eye sockets; determining a second sight line direction according to the relative positions of the eye sockets and the pupils; and determining the gaze direction of the driver of the current frame of the driving image according to the first sight line direction and the second sight line direction.

[0011] According to a state detection method based on using a mobile terminal while driving provided by the present invention, the sum of the judgment results based on each frame of driving image is greater than a first preset threshold value, and an alarm message is generated and sent, including: determining an alarm message reminder mode based on the sum of the judgment results of each frame of driving image and a preset alarm message reminder mode; wherein the alarm message includes at least one of a voice reminder mode, an image reminder mode, a steering wheel prompt light flashing mode, a buzzer reminder mode and an alarm mode; generating alarm information according to the determined alarm message reminder mode, and sending the alarm information according to the alarm message reminder mode.

[0012] The present invention also provides a state detection device based on using a mobile terminal while driving, comprising: a data acquisition module, which acquires driving images within a preset time period; an angle acquisition module, which acquires a first angle of the mobile terminal relative to the driver's head and a gaze angle of the mobile terminal relative to the driver's line of sight based on each frame of the driving image; a detection module, which obtains a behavior judgment result for each frame of the driving image based on the gaze angle and the first angle; and a control module, which generates and sends an alarm message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for detecting the state of using a mobile terminal while driving are implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for detecting the state of using a mobile terminal while driving are implemented.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for detecting the state of using a mobile terminal while driving.

[0016] The state detection method and device based on the use of a mobile terminal while driving provided by the present invention respectively obtain a first angle of the driver's head relative to the mobile terminal and a gaze angle of the driver's line of sight relative to the mobile terminal, and judge whether the driver is looking at the mobile terminal in combination with the first angle and the gaze angle, thereby improving the detection accuracy of the terminal while driving, and avoiding detection errors caused by detecting the driver's line of sight alone or the direction of the head relative to the mobile terminal; in addition, by generating and sending an alarm message based on the sum of the judgment results, the driver's attention to the steering control of the vehicle can be avoided from being relaxed, thereby preventing the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a state detection method based on using a mobile terminal while driving provided by the present invention;

[0019] Figure 2 A schematic diagram of the structure of a state detection device based on using a mobile terminal while driving provided by the present invention;

[0020] Figure 3 A schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Figure 1 A flow chart of a method for detecting a state of using a mobile terminal while driving is shown, the method comprising:

[0023] S01, acquiring driving images within a preset time period;

[0024] S02, based on each frame of the driving image, obtaining a first angle of the mobile terminal relative to the driver's head, and obtaining a gaze angle of the mobile terminal relative to the driver's sight line;

[0025] S03, obtaining a behavior judgment result for each frame of the driving image according to the gaze angle and the first angle;

[0026] S04: Generate and send a warning message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0027] It should be noted that S0N in this specification does not represent the order of the state detection method based on using a mobile terminal while driving. The state detection method based on using a mobile terminal while driving of the present invention is described in detail below.

[0028] Step S01, obtaining driving images within a preset time period.

[0029] In this embodiment, obtaining driving images within a preset time period includes: obtaining a video stream of the driving area; extracting driving images within a preset time period based on the video stream. Specifically, extracting driving images within a preset time period based on the video stream includes: based on the video stream, collecting a certain number of driving images before and after the current frame driving image, for example, collecting a preset number of driving images before and after the current frame driving image, so as to avoid the situation where subsequent judgment based on a single frame driving image leads to poor detection results. The specific number of driving images obtained can be set according to actual detection needs and is not further limited here. For example, it can be five frames of driving images before and after the current frame driving image. In other embodiments, obtaining driving images within a preset time period includes: obtaining multiple frames of driving images taken continuously within the preset time period.

[0030] It should be noted that the driving image is an image captured based on the driving area within the vehicle. The vehicle can be a vehicle, ship or aircraft used to carry people or goods, where the vehicle can be a private car or an operating vehicle, such as a shared car, online car-hailing, taxi, bus, school bus, truck, coach, train, subway and tram.

[0031] In an optional embodiment, an overhead road camera and a face road camera may be used to obtain driving images of the driving area from different directions, wherein the overhead road camera is arranged above the main driving area to shoot the driving area from top to bottom; the face road camera is arranged in front of the main driving area to shoot the driving area in the direction of the face. It should be noted that the face road camera and / or the overhead road camera includes at least one of a high-resolution infrared camera, a DMS camera, an OMS camera, a video camera, a video recorder, and an electronic device with a camera, and the electronic device may be a mobile terminal device such as a mobile terminal, a computer, a camera, and a tablet computer.

[0032] Step S02: based on each frame of driving image, obtaining a first angle of the mobile terminal relative to the driver's head, and obtaining a gaze angle of the mobile terminal relative to the driver's line of sight.

[0033] It should be noted that the mobile terminal may be a mobile phone, tablet computer, laptop computer, PDA, vehicle-mounted electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc.

[0034] In this embodiment, based on each frame of driving image, obtaining the first angle of the mobile terminal relative to the driver's head includes: based on each frame of driving image, selecting the driver's sight line direction facing forward as the reference line direction; based on pre-calibrated camera parameters, determining the first direction of the mobile terminal relative to the driver's head; and obtaining the first angle of the current frame of driving image according to the first direction and the reference line direction. By obtaining the first angle, it is convenient to judge the relative position of the driver's head and the mobile terminal, so as to facilitate the preliminary judgment of whether the driver is facing the direction of the mobile terminal.

[0035] In an optional embodiment, obtaining the gaze angle of the mobile terminal relative to the driver's line of sight includes: selecting the direction in which the driver is looking straight ahead as the reference line direction; determining the driver's gaze direction based on each frame of the driving image; and obtaining the gaze angle of the current frame of the driving image according to the pre-selected reference line direction and gaze direction. By obtaining the gaze angle, it is convenient to judge whether the driver's line of sight is facing the direction of the mobile terminal, so as to preliminarily judge whether the driver is looking at the mobile terminal in the current frame. It should be noted that when obtaining the reference line direction selected when obtaining the gaze angle of the mobile terminal relative to the driver's line of sight, the reference line direction determined when obtaining the first angle of the mobile terminal relative to the driver's head can be directly selected to avoid repeated determination of the reference line direction and reduce the deviation error of the subsequent first angle and line of sight angle.

[0036] In an optional embodiment, based on each frame of driving image, determining the driver's gaze direction includes: based on each frame of driving image, identifying facial features in the driving image to obtain eye sockets and pupils in the facial image; determining a first sight line direction according to the area of ​​the identified eye sockets; determining a second sight line direction according to the relative positions of the eye sockets and pupils; and determining the driver's gaze direction of the current frame of driving image according to the first sight line direction and the second sight line direction. Specifically:

[0037] First, the first gaze direction is determined according to the area of ​​the identified eye socket, including: calculating the face area in the face image and calculating the eye socket area in the face image; judging whether the ratio of the eye socket area to the face area is less than a third preset threshold; if so, judging that the first gaze direction is downward; otherwise, judging whether the ratio of the distance from the pupil to the upper eye socket to the distance from the pupil to the lower eye socket in the face image is less than a fourth preset threshold; if so, judging that the first gaze direction is upward, otherwise judging that the first gaze direction is horizontal.

[0038] In addition, the second gaze direction is determined according to the relative positions of the pupil and the eye socket in the face image, including: judging whether the ratio of the distance from the pupil to the left edge of the eye socket to the distance from the pupil to the right edge of the eye socket in the face image is less than a fifth preset threshold; if so, judging that the second gaze direction is to the left; otherwise, judging whether the ratio of the distance from the pupil to the left edge of the eye socket to the distance from the pupil to the right edge of the eye socket in the face image is less than a sixth preset threshold, if so, judging that the second gaze direction is to the level, otherwise judging that the second gaze direction is to the right.

[0039] Secondly, the gaze direction of the driver of the current frame driving image is determined according to the first gaze direction and the second gaze direction. It should be noted that the sight direction determined by the first gaze direction is the longitudinal direction, and the sight direction determined by the second gaze direction is the lateral direction. The first gaze direction combined with the second gaze direction can determine the driver's gaze direction. For example, if the first gaze direction is upward and the second gaze direction is left, then the gaze direction is the upper left; for another example, if the first gaze direction is upward and the second gaze direction is horizontal, then the gaze direction is directly above; for another example, if the first gaze direction is horizontal and the second gaze direction is horizontal, then the gaze direction is horizontal.

[0040] In an optional embodiment, determining the driver's gaze direction based on each frame of driving image also includes: based on each frame of driving image, identifying facial features in the driving image to obtain eye sockets and pupils in the facial image; and determining the driver's gaze direction in the current frame of driving image based on the relative positions of the eye sockets and pupils.

[0041] Step S03, obtaining a behavior judgment result of each frame of the driving image according to the gaze angle and the first angle.

[0042] In this embodiment, the behavior judgment result of each frame of the driving image is obtained according to the gaze angle and the first angle, including: for each frame of the driving image, judging whether the deviation between the gaze angle and the first angle is less than the second preset threshold value, if so, judging the current frame as playing with the mobile terminal, and recording the judgment result as 1; otherwise, judging the current frame as not playing with the mobile terminal, and recording the judgment result as 0. It should be noted that by combining the gaze angle and the first angle, it is convenient to judge whether the driver in the current frame is looking at the mobile terminal in combination with the line of sight direction of the driver in the current frame and the direction of the head relative to the mobile terminal, thereby facilitating the improvement of the detection accuracy.

[0043] In addition, the second preset threshold value can be set to 5°, and the specific value can be set according to the actual allowable error, which is not further limited here. By judging whether the deviation between the first angle and the sight angle meets the preset threshold value, it is convenient to judge whether the driver is currently looking at the mobile terminal in combination with the first angle and the sight angle, thereby improving the accuracy of the state detection based on the use of the mobile terminal while driving, and avoiding the situation where the driver occasionally looks at the mobile terminal or mistakenly looks at the mobile terminal, resulting in a detection error.

[0044] Step S04: Generate and send a warning message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0045] In this embodiment, based on the sum of the judgment results of each frame of driving image being greater than a first preset threshold value, an alarm message is generated and sent, including: determining an alarm message reminder mode based on the sum of the judgment results of each frame of driving image and a preset alarm message reminder mode; wherein the alarm message includes at least one of a voice reminder mode, an image reminder mode, a steering wheel prompt light flashing mode, a buzzer reminder mode and an alarm mode; generating alarm information according to the determined alarm message reminder mode, and sending the alarm information according to the alarm message reminder mode.

[0046] It should be noted that the sum of the judgment results of each frame of the driving image can be understood as the length of time the driver stares at the mobile terminal within the preset time period. When the length of time exceeds the first preset threshold, it is determined that the driver is staring at the mobile terminal.

[0047] In addition, when the warning information reminder mode is pre-set, the corresponding warning information reminder mode can be set according to the sum of the judgment results of each frame of driving image, so as to remind the driver not to stare at the mobile terminal for a long time to avoid slacking off. It can be set specifically according to the actual reminder needs. For example, if the driver stares at the mobile terminal for more than the time specified by the state, the warning information reminder mode is set to the alarm mode accordingly, so as to facilitate the subsequent generation of alarm information according to the alarm mode and send the alarm information according to the alarm mode; for another example, if the driver stares at the mobile terminal for less than the time specified by the state and is greater than the first preset threshold, a specific warning information reminder mode is set according to the time the driver stares at the mobile terminal. If the driver stares at the mobile terminal for a short time, the warning information reminder mode can be set to the voice reminder mode, the image reminder mode, the steering wheel prompt light flashing mode or the buzzer reminder mode. If the driver stares at the mobile terminal for a long time, the warning information reminder mode can be set to at least two of the voice reminder mode, the image reminder mode, the steering wheel prompt light flashing mode and the buzzer reminder mode.

[0048] In summary, this embodiment obtains the first angle of the driver's head relative to the mobile terminal and the gaze angle of the driver's line of sight relative to the mobile terminal, respectively, and judges whether the driver is looking at the mobile terminal in combination with the first angle and the gaze angle, thereby improving the detection accuracy of the terminal when driving and avoiding detection errors caused by detecting the driver's line of sight alone or the direction of the head relative to the mobile terminal; in addition, by generating and sending an alarm message based on the sum of the judgment results, the driver can avoid relaxing his attention on the steering control of the vehicle and prevent traffic accidents.

[0049] The following is a description of a state detection device based on using a mobile terminal while driving provided by the present invention. The state detection device based on using a mobile terminal while driving described below and the state detection method based on using a mobile terminal while driving described above can be referred to each other.

[0050] Figure 2 A schematic diagram of a state detection device for using a mobile terminal while driving is shown, the device comprising:

[0051] A data acquisition module 21 is used to acquire driving images within a preset time period;

[0052] An angle acquisition module 22, based on each frame of driving image, acquires a first angle of the mobile terminal relative to the driver's head, and acquires a gaze angle of the mobile terminal relative to the driver's line of sight;

[0053] The detection module 23 obtains a behavior judgment result of each frame of the driving image according to the gaze angle and the first angle;

[0054] The control module 24 generates and sends a warning message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0055] In this embodiment, the data acquisition module 21 includes: a camera unit that acquires a video stream of the driving area; and an image extraction unit that extracts driving images within a preset time period based on the video stream. Specifically, the image extraction unit collects a certain number of driving images before and after the current frame driving image based on the video stream. For example, it collects five frames of driving images before and after the current frame driving image, so as to avoid the subsequent judgment results based on a single frame driving image and the situation where the detection results are poor. The specific number of driving images acquired can be set according to the actual detection needs and is not further limited here. In other embodiments, the data acquisition module 21 includes: a single frame image acquisition unit that acquires multiple frames of continuously shot driving images within a preset time period.

[0056] It should be noted that the driving image is an image captured based on the driving area within the vehicle. The vehicle can be a vehicle, ship or aircraft used to carry people or goods, where the vehicle can be a private car or an operating vehicle, such as a shared car, online car-hailing, taxi, bus, school bus, truck, coach, train, subway and tram.

[0057] In an optional embodiment, an overhead road camera and a face road camera may be used to obtain driving images of the driving area from different directions, wherein the overhead road camera is arranged above the main driving area to shoot the driving area from top to bottom; the face road camera is arranged in front of the main driving area to shoot the driving area in the direction of the face. It should be noted that the face road camera and / or the overhead road camera includes at least one of a high-resolution infrared camera, a DMS camera, an OMS camera, a video camera, a video recorder, and an electronic device with a camera, and the electronic device may be a mobile terminal device such as a mobile terminal, a computer, a camera, and a tablet computer.

[0058] The angle acquisition module 22 includes a first angle acquisition submodule and a gaze angle acquisition submodule, wherein:

[0059] The first angle acquisition submodule includes: a first reference line selection unit, which selects the driver's sight line direction facing forward as the reference line direction based on each frame of driving image; a first direction determination unit, which determines the first direction of the mobile terminal relative to the driver's head based on pre-calibrated camera parameters; and a first angle acquisition unit, which obtains the first angle of the current frame of driving image according to the first direction and the reference line direction. The first angle is obtained by the first angle acquisition unit to facilitate the determination of the relative position between the driver's head and the mobile terminal, thereby facilitating the preliminary determination of whether the driver is facing the direction of the mobile terminal.

[0060] In addition, the gaze angle acquisition submodule includes: a second reference line selection unit, which selects the direction in which the driver is looking straight ahead as the reference line direction; a second direction determination unit, which determines the driver's gaze direction based on each frame of the driving image; and a second angle acquisition unit, which obtains the gaze angle of the current frame of the driving image according to the pre-selected reference line direction and gaze direction. The gaze angle is acquired by the second angle acquisition unit to facilitate the determination of whether the driver's line of sight is directed toward the direction of the mobile terminal, so as to facilitate the preliminary determination of whether the driver is looking at the mobile terminal in the current frame.

[0061] Furthermore, the second direction determination unit includes: a first feature extraction subunit, which identifies the facial features in the driving image based on each frame of the driving image, and obtains the eye sockets and pupils in the facial image; a sight line determination subunit, which determines the first sight line direction according to the area of ​​the identified eye sockets, and determines the second sight line direction according to the relative positions of the eye sockets and the pupils; a first gaze direction acquisition subunit, which determines the gaze direction of the driver in the current frame of the driving image according to the first sight line direction and the second sight line direction. Specifically:

[0062] First, the line of sight is determined as a subunit, including: an area calculation subunit, which calculates the face area in the face image and the eye socket area in the face image; a first judgment subunit, which judges whether the ratio of the eye socket area to the face area is less than a third preset threshold; if so, the first gaze direction is determined to be downward; otherwise, a second judgment subunit, which judges whether the ratio of the distance from the pupil to the upper eye socket to the distance from the pupil to the lower eye socket in the face image is less than a fourth preset threshold; if so, the first gaze direction is determined to be upward, otherwise the first gaze direction is determined to be horizontal.

[0063] In addition, the line of sight determination subunit also includes: a third judgment subunit, which judges whether the ratio of the distance from the pupil to the left edge of the eye socket in the face image to the distance from the pupil to the right edge of the eye socket is less than a fifth preset threshold; if it is less than, the second gaze direction is determined to be to the left; otherwise, a fourth judgment subunit, which judges whether the ratio of the distance from the pupil to the left edge of the eye socket in the face image to the distance from the pupil to the right edge of the eye socket is less than a sixth preset threshold, if it is less than, the second gaze direction is determined to be level, otherwise the second gaze direction is determined to be to the right.

[0064] It should be noted that the sight direction determined by the first gaze direction is the longitudinal direction, and the sight direction determined by the second gaze direction is the transverse direction. The first gaze direction combined with the second gaze direction can determine the driver's gaze direction. For example, if the first gaze direction is upward and the second gaze direction is leftward, the gaze direction is the upper left; for another example, if the first gaze direction is upward and the second gaze direction is horizontal, the gaze direction is directly upward; for another example, if the first gaze direction is horizontal and the second gaze direction is horizontal, the gaze direction is horizontal.

[0065] In an optional embodiment, the second direction determination unit also includes: a second feature extraction subunit, which identifies facial features in the driving image based on each frame of the driving image, and obtains the eye sockets and pupils in the facial image; and a second gaze direction acquisition subunit, which determines the gaze direction of the driver in the current frame of the driving image according to the relative positions of the eye sockets and the pupils.

[0066] The detection module 23 includes: a judgment unit, for each frame of the driving image, judging whether the deviation between the gaze angle and the first angle is less than a second preset threshold value, if so, judging the current frame as playing with the mobile terminal, and recording the judgment result as 1; otherwise, judging the current frame as not playing with the mobile terminal, and recording the judgment result as 0. It should be noted that the judgment unit combines the gaze angle and the first angle, so as to judge whether the driver in the current frame is looking at the mobile terminal in combination with the line of sight of the driver in the current frame and the direction of the head relative to the mobile terminal, thereby facilitating improving the accuracy of the detection.

[0067] In addition, the second preset threshold value can be set to 5°, and the specific value can be set according to the actual allowable error, which is not further limited here. By judging whether the deviation between the first angle and the sight angle meets the preset threshold value, it is convenient to judge whether the driver is currently looking at the mobile terminal in combination with the first angle and the sight angle, thereby improving the accuracy of the state detection based on the use of the mobile terminal while driving, and avoiding the situation where the driver occasionally looks at the mobile terminal or mistakenly looks at the mobile terminal, resulting in a detection error.

[0068] The control module 24 includes: a mode selection unit, which determines the warning information reminder mode based on the sum of the judgment results of each frame of the driving image and the pre-set warning information reminder mode; wherein the warning information includes at least one of a voice reminder mode, an image reminder mode, a steering wheel prompt light flashing mode, a buzzer reminder mode and an alarm mode; an information prompt unit, which generates warning information according to the determined warning information reminder mode, and sends the warning information according to the warning information reminder mode.

[0069] In an optional embodiment, the device also includes a mode setting module for setting an alarm information reminder mode according to the length of time the driver stares at the mobile terminal, thereby reminding the driver to keep holding the steering wheel and avoid slacking off.

[0070] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 31, a communication interface 32, a memory 33 and a communication bus 34, wherein the processor 31, the communication interface 32 and the memory 33 communicate with each other through the communication bus 34. The processor 31 may call the logic instructions in the memory 33 to execute a state detection method based on driving and using a mobile terminal, the method comprising: obtaining a driving image within a preset time period; obtaining a first angle of the mobile terminal relative to the driver's head and a gaze angle of the mobile terminal relative to the driver's line of sight based on each frame of the driving image; obtaining a behavior judgment result of each frame of the driving image based on the gaze angle and the first angle; generating and sending an alarm message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0071] In addition, the logic instructions in the above-mentioned memory 33 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the state detection method based on the use of a mobile terminal while driving provided by the above methods. The method includes: obtaining driving images within a preset time period; based on each frame of the driving image, obtaining a first angle of the mobile terminal relative to the driver's head, and obtaining a gaze angle of the mobile terminal relative to the driver's line of sight; according to the gaze angle and the first angle, obtaining a behavior judgment result for each frame of the driving image; based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold, generating and sending an alarm message.

[0073] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the state detection method based on the use of a mobile terminal while driving provided by the above-mentioned methods, the method comprising: obtaining driving images within a preset time period; based on each frame of the driving image, obtaining a first angle of the mobile terminal relative to the driver's head, and obtaining a gaze angle of the mobile terminal relative to the driver's line of sight; obtaining a behavior judgment result for each frame of the driving image based on the gaze angle and the first angle; and generating and sending an alarm message based on the sum of the judgment results of each frame of the driving image being greater than a first preset threshold.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the state of using a mobile terminal while driving, characterized in that: include: Acquire driving images within a preset time period; Based on each frame of the driving image, obtaining a first angle of the mobile terminal relative to the driver's head, and obtaining a gaze angle of the mobile terminal relative to the driver's line of sight; Obtaining a behavior judgment result for each frame of the driving image according to the gaze angle and the first angle; Generate and send warning information based on the sum of the judgment results of each frame of driving image being greater than a first preset threshold; The obtaining of the gaze angle of the mobile terminal relative to the driver's line of sight includes: Selecting the direction in which the driver is looking forward as the reference line direction; Determining the driver's gaze direction based on each frame of the driving image; Obtaining a gaze angle of a current frame driving image according to a pre-selected reference line direction and the gaze direction; The determining the gaze direction of the driver based on each frame of the driving image comprises: Based on each frame of the driving image, identifying facial features in the driving image, and obtaining eye sockets and pupils in the facial image; Determine the first sight direction according to the identified area of ​​the eye socket; Determining a second sight line direction according to the relative positions of the eye socket and the pupil; Determining a gaze direction of a driver in a current frame of the driving image according to the first sight line direction and the second sight line direction; Determining the first sight direction according to the identified area of ​​the eye socket includes: Calculating the face area in the face image and calculating the eye socket area in the face image; Determine whether the ratio of the eye socket area to the face area is less than a third preset threshold; if so, determine that the first gaze direction is downward; Otherwise, determining whether the ratio of the distance from the pupil to the upper eye socket to the distance from the pupil to the lower eye socket in the face image is less than a fourth preset threshold; if so, determining that the first gaze direction is upward, otherwise determining that the first gaze direction is horizontal; Determining a second sight line direction according to the relative positions of the eye socket and the pupil includes: Determine whether the ratio of the distance from the pupil to the left edge of the eye socket to the distance from the pupil to the right edge of the eye socket in the face image is less than a fifth preset threshold; if so, determine that the second gaze direction is leftward; Otherwise, determining whether the ratio of the distance from the pupil to the left edge of the eye socket to the distance from the pupil to the right edge of the eye socket in the face image is less than a sixth preset threshold, if so, determining that the second gaze direction is horizontal, otherwise determining that the second gaze direction is rightward; The sight line direction determined by the first gaze direction is a longitudinal direction, and the sight line direction determined by the second gaze direction is a transverse direction.

2. The method for detecting the state of a mobile terminal when driving according to claim 1, characterized in that: The obtaining, according to the gaze angle and the first angle, a behavior judgment result of each frame of the driving image includes: For each frame of the driving image, determine whether the deviation between the gaze angle and the first angle is less than a second preset threshold value, and if so, determine that the current frame is a mobile terminal playing frame, and record the determination result as 1; Otherwise, it is determined that the current frame is not played by the mobile terminal, and the determination result is recorded as 0.

3. The method for detecting the state of a mobile terminal when driving according to claim 1, characterized in that: The obtaining, based on each frame of the driving image, a first angle of the mobile terminal relative to the driver's head includes: Based on each frame of driving image, selecting the sight line direction of the driver looking straight ahead as the reference line direction; Based on pre-calibrated camera parameters, determining a first direction of the mobile terminal relative to the driver's head; A first angle of the current frame driving image is obtained according to the first direction and the reference line direction.

4. The method for detecting the state of a mobile terminal when driving according to claim 1, characterized in that: The generating and sending of warning information based on the sum of the judgment results of each frame of driving image being greater than a first preset threshold value includes: Determine a warning information reminder mode based on the sum of the judgment results of each frame of the driving image and a preset warning information reminder mode; wherein the warning information includes at least one of a voice reminder mode, an image reminder mode, a steering wheel prompt light flashing mode, a buzzer reminder mode and an alarm mode; Generate alarm information according to the determined alarm information reminder mode, and send the alarm information according to the alarm information reminder mode.

5. A state detection device based on using a mobile terminal while driving, characterized in that: include: A data acquisition module, which acquires driving images within a preset time period; An angle acquisition module, based on each frame of driving image, acquires a first angle of the mobile terminal relative to the driver's head, and acquires a gaze angle of the mobile terminal relative to the driver's line of sight; A detection module, which obtains a behavior judgment result of each frame of the driving image according to the gaze angle and the first angle; A control module, based on the sum of the judgment results of each frame of driving image being greater than a first preset threshold, generates and sends a warning message; The angle acquisition module includes a gaze angle acquisition submodule, and the gaze angle acquisition submodule includes: A second reference line selection unit selects the direction in which the driver is looking straight ahead as the reference line direction; A second direction determination unit, which determines the gaze direction of the driver based on each frame of the driving image; A second angle acquisition unit, which obtains a gaze angle of the current frame driving image according to a pre-selected reference line direction and the gaze direction; The second direction determining unit includes: A first feature extraction subunit, based on each frame of the driving image, identifies the facial features in the driving image to obtain the eye sockets and pupils in the facial image; a sight line determination subunit, which determines a first sight line direction according to the identified area of ​​the eye socket, and determines a second sight line direction according to the relative position of the eye socket and the pupil; A first gaze direction acquisition subunit determines a gaze direction of a driver in a current frame driving image according to the first sight line direction and the second sight line direction; The line of sight determination subunit comprises: An area calculation unit is used to calculate the face area in the face image and the eye socket area in the face image; a first judgment unit, for judging whether the ratio of the eye socket area to the face area is less than a third preset threshold; if so, judging that the first gaze direction is downward; Otherwise, the second judgment unit determines whether the ratio of the distance from the pupil to the upper eye socket to the distance from the pupil to the lower eye socket in the face image is less than a fourth preset threshold; if so, the first gaze direction is determined to be upward, otherwise, the first gaze direction is determined to be horizontal; The sight line determination subunit further includes: a third judgment unit, for judging whether the ratio of the distance from the pupil to the left edge of the eye socket to the distance from the pupil to the right edge of the eye socket in the face image is less than a fifth preset threshold; if so, judging that the second gaze direction is leftward; Otherwise, the fourth judgment unit determines whether the ratio of the distance from the pupil to the left edge of the eye socket to the distance from the pupil to the right edge of the eye socket in the face image is less than a sixth preset threshold value, and if so, determines that the second gaze direction is horizontal, otherwise determines that the second gaze direction is rightward; The sight line direction determined by the first gaze direction is a longitudinal direction, and the sight line direction determined by the second gaze direction is a transverse direction.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for detecting the state of using a mobile terminal while driving as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting the state of using a mobile terminal while driving according to any one of claims 1 to 4 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for detecting the state of using a mobile terminal while driving according to any one of claims 1 to 4 are implemented.

Images