Sound prompting method and device for fatigue of sanitation vehicle driver

Abstract

The embodiment of the invention discloses a sound prompting method and device for fatigue of a driver of an environmental sanitation vehicle. A specific embodiment of the method comprises the following steps: acquiring a driver face image sequence and a vehicle control information sequence; performing eye detection on each driver face image in the driver face image sequence to obtain eye detection information; in response to the fact that the eye detection information meets a preset driver fatigue condition, generating a driver fatigue level identifier based on the vehicle control information sequence and the eye detection information; based on the driver fatigue level identifier, generating a corresponding driver fatigue prompt tone, and sending the fatigue prompt tone to an audio device for sound prompt, the driver fatigue prompt tone being a question prompt tone sent to the driver. According to the implementation mode, fatigue sound prompting can be more accurately carried out on the driver, and it is ensured that the sent prompting sound can achieve the sound prompting purpose.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of driver fatigue detection, and more particularly to a method and device for sound prompting of fatigue of sanitation vehicle drivers. Background Art

[0002] The sound prompt of driver fatigue of sanitation vehicles can be used as a technology to remind the driver when the driver is tired. At present, when used for the sound prompt of driver fatigue, the method usually adopted is: after detecting the driver's fatigue, a voice prompt is issued to the driver to remind the driver.

[0003] However, the inventors have found that when the above method is used for sound prompting, the following technical problems often occur:

[0004] Since sanitation vehicles often travel at night, drivers are prone to deep driving fatigue. However, the sound that prompts drivers to drive fatigue is relatively conventional, and there are cases where drivers are still in a state of driving fatigue after hearing the prompt sound. Therefore, the driver's state reaction after hearing the sound is not considered, so it is difficult to achieve the purpose of sound prompting the driver's fatigue, and thus, driving dangers are likely to occur.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a method and device for sound prompting of fatigue of sanitation vehicle drivers to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for sound prompting of fatigue of sanitation vehicle drivers, the method comprising: obtaining a facial image sequence of the sanitation vehicle driver through a built-in camera of the sanitation vehicle, and obtaining a vehicle control information sequence of a central control terminal of the sanitation vehicle; performing eye detection on each facial image of the sanitation vehicle driver in the above-mentioned facial image sequence of the sanitation vehicle driver to obtain eye detection information; in response to the above-mentioned eye detection information satisfying a preset driver fatigue condition, generating a driver fatigue level identification based on the above-mentioned vehicle control information sequence and the above-mentioned eye detection information; based on the above-mentioned driver fatigue level identification, generating a corresponding driver fatigue prompt sound, and sending the above-mentioned fatigue prompt sound to an audio device for sound prompting, wherein the above-mentioned driver fatigue prompt sound is a problem prompt sound issued to the driver in the sanitation vehicle.

[0009] In a second aspect, some embodiments of the present disclosure provide a sound prompt device for fatigue of a sanitation vehicle driver, the device comprising: an acquisition unit, configured to acquire a facial image sequence of a sanitation vehicle driver through a built-in camera of the sanitation vehicle, and to acquire a vehicle control information sequence of a central control terminal of the sanitation vehicle; an eye detection unit, configured to perform eye detection on each facial image of the sanitation vehicle driver in the above-mentioned facial image sequence of the sanitation vehicle driver to obtain eye detection information; a first generation unit, configured to generate a driver fatigue level identifier based on the above-mentioned vehicle control information sequence and the above-mentioned eye detection information in response to the above-mentioned vehicle control information sequence and the above-mentioned eye detection information satisfying a preset driver fatigue condition; a generation and sound prompt unit, configured to generate a corresponding driver fatigue prompt sound based on the above-mentioned driver fatigue level identifier, and to send the above-mentioned fatigue prompt sound to an audio device for sound prompting, wherein the above-mentioned driver fatigue prompt sound is a problem prompt sound issued to the driver in the sanitation vehicle.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the sound prompt method for sanitation vehicle driver fatigue of some embodiments of the present disclosure, the purpose of driver fatigue prompt can be quickly achieved. Specifically, the reason why it is difficult to achieve the purpose of the sound prompt for driver fatigue is that the prompt sound is relatively conventional, and there is a driver who is still in a fatigue driving state after hearing the prompt sound, thus, the driver's state reaction after hearing the sound is not considered. Based on this, the sound prompt method for sanitation vehicle driver fatigue of some embodiments of the present disclosure, first, through the built-in camera of the sanitation vehicle, obtains the sanitation vehicle driver face image sequence, and obtains the vehicle control information sequence of the central control terminal of the sanitation vehicle. Then, each sanitation vehicle driver face image in the above sanitation vehicle driver face image sequence is subjected to eye detection to obtain eye detection information. Through eye detection, it can be used to preliminarily judge the driver's fatigue state. Afterwards, in response to the above vehicle control information sequence and the above eye detection information satisfying the preset driver fatigue condition, a driver fatigue level identifier is generated based on the above vehicle control information sequence and the above eye detection information. By generating the driver fatigue level identifier, it can be used to further determine the driver's fatigue state and its level. Finally, based on the above-mentioned driver fatigue level identifier, a corresponding driver fatigue prompt sound is generated, and the above-mentioned fatigue prompt sound is sent to the audio device for sound prompting. Among them, the above-mentioned driver fatigue prompt sound is a problem prompt sound issued to the driver in the sanitation vehicle. By generating a driver fatigue prompt sound corresponding to the driver's fatigue level status, the driver can be given a sound prompt more effectively. At the same time, because the prompted driver fatigue prompt sound is a problem prompt sound, it can be used to determine the driver's fatigue state change according to the driver's voice feedback. Thereby, the driver can be given a fatigue sound prompt more accurately, and it can be ensured that the issued prompt sound can achieve the purpose of the sound prompt. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the method for sound prompting of fatigue of sanitation vehicle drivers according to the present disclosure;

[0015] Figure 2 is a frequency domain energy schematic diagram of some embodiments of the sound prompt method for sanitation vehicle driver fatigue according to the present disclosure;

[0016] Figure 3is a schematic structural diagram of some embodiments of the sound prompt device for sanitation vehicle driver fatigue according to the present disclosure;

[0017] Figure 4 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0018] Figure 5 is a face image of a sanitation vehicle driver captured by a built-in camera of a sanitation vehicle according to some embodiments of the method for sound prompting of fatigue of a sanitation vehicle driver disclosed in the present invention;

[0019] Figure 6 It is a schematic diagram of the interface of the intelligent supervision platform for sanitation operations according to some embodiments of the sound prompt method for sanitation vehicle driver fatigue disclosed in the present invention;

[0020] Figure 7 It is a schematic diagram of the vehicle status interface of the intelligent supervision platform for sanitation operations according to some embodiments of the method for sound prompting of fatigue of sanitation vehicle drivers disclosed in the present invention. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0023] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0026] With regard to the collection, storage, and use of driver personal information (such as driver images and driver historical behaviors) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill obligations including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Figure 1 A process 100 of some embodiments of the sound method for sanitation vehicle driver fatigue according to the present disclosure is shown. The sound method for sanitation vehicle driver fatigue comprises the following steps:

[0029] Step 101, obtaining a facial image sequence of a sanitation vehicle driver through a built-in camera of the sanitation vehicle, and obtaining a vehicle control information sequence of a central control terminal of the sanitation vehicle.

[0030] In some embodiments, the executor of the sound method for sanitation vehicle driver fatigue can obtain the facial image sequence of the sanitation vehicle driver and the vehicle control information sequence of the central control terminal of the sanitation vehicle through the built-in camera of the sanitation vehicle in a wired or wireless manner. Among them, each sanitation vehicle driver's facial image in the sanitation vehicle driver's facial image sequence can be an image of a continuous frame. The vehicle control information in the vehicle control information sequence can also be information of a continuous frame. The vehicle control information can be various control data of the sanitation vehicle. For example, the vehicle control information may include: vehicle speed value, vehicle steering wheel angle, lane number of the vehicle, vehicle driving status (such as lane keeping status, lane change status, etc.), vehicle throttle amplitude, etc.

[0031] As an example, the face image of a sanitation vehicle driver can be Figure 5 shown.

[0032] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0033] Step 102: perform eye detection on each driver face image in the sanitation vehicle driver face image sequence to obtain eye detection information.

[0034] In some embodiments, the above-mentioned execution entity can perform eye detection on each sanitation vehicle driver face image in the above-mentioned sanitation vehicle driver face image sequence to obtain eye detection information.

[0035] In some optional implementations of some embodiments, the execution subject performs eye detection on each sanitation vehicle driver face image in the sanitation vehicle driver face image sequence to obtain eye detection information, which may include the following steps:

[0036] The first step is to perform eye detection on each sanitation vehicle driver face image in the above-mentioned sanitation vehicle driver face image sequence to generate an eye state identification sequence. The eye state identification in the above-mentioned eye state identification sequence can indicate that the driver's eyes are in a normal state, a distracted state, or a fatigued state. Secondly, eye detection can be performed on each driver face image in the above-mentioned driver face image sequence using a preset eye detection algorithm to generate an eye state identification sequence.

[0037] As an example, the above-mentioned eye detection algorithm may include but is not limited to at least one of the following: Mobile Net (Deep Level Separable Convolutional Network) algorithm, Shuffle Net (An Extremely Efficient Convolutional Neural Network for Mobile Devices, an efficient convolutional neural network applied to mobile devices) algorithm or SqueezeNet (Deep Neural Network) algorithm, etc.

[0038] The second step is to generate eye detection information in response to determining that each eye state identifier in the above eye state identifier sequence meets the preset state condition. The above eye detection information is information that characterizes that the driver is in a normal driving state, a distracted state, or a driving fatigue state. The above preset state condition may be: the number of eye state identifiers in the eye state identifier sequence that characterize a distracted state or a fatigued state is greater than a preset number threshold. Here, each eye state identifier in the above eye state identifier sequence meets the preset state condition, which can characterize that the driver is distracted for a long time (such as looking at a mobile phone, looking in other directions, etc.), or a fatigued state (such as dozing off). Therefore, eye detection information that characterizes that the driver is in a distracted state or a driving fatigue state can be generated. In practice, eye detection information is a state identifier that describes different states through state identifiers.

[0039] Optionally, in response to determining that each eye state identifier in the above eye state identifier sequence does not satisfy the above state condition, eye detection information characterizing that the driver is in a normal driving state is generated.

[0040] Step 103 , in response to the vehicle control information sequence and the eye detection information satisfying a preset driver fatigue condition, generating a driver fatigue level identifier based on the vehicle control information sequence and the eye detection information.

[0041] In some embodiments, the execution subject may generate a driver fatigue level indicator based on the vehicle control information sequence and the eye detection information in response to the eye detection information satisfying a preset driver fatigue condition. The preset driver fatigue condition may be: the eye detection information is information indicating that the driver is in a distracted state or a driving fatigue state.

[0042] In practice, in the process of adopting technical solutions to solve the problems mentioned in the background technology, the following technical problem 2 often occurs: conventional sound prompts are difficult to attract the driver's attention in time, which makes it difficult for the driver to change from a fatigue driving state to a normal state in time, thereby reducing driving safety. In response to the above technical problem 2, the inventor decided to adopt the following solution.

[0043] In some optional implementations of some embodiments, the execution subject generates a driver fatigue level identifier based on the vehicle control information sequence and the eye detection information, which may include the following steps:

[0044] The first step is to obtain the driver's heart rate detection value detected by the smart watch carried by the driver in the sanitation vehicle, wherein the smart watch is pre-connected. In the process of fatigue detection of the driver, the smart watch carried by the driver can be used to assist in detecting whether the driver is in a state of fatigue driving.

[0045] The second step is to generate a driver fatigue level indicator representing the first-level driver fatigue in response to determining that the eye detection information is information representing the driver being in a distracted state. The driver fatigue level indicator representing the first-level driver fatigue may be that the driver is distracted but not driving fatigued. Therefore, a simple attention reminder can be performed.

[0046] The third step is to generate a driver fatigue level indicator representing the fatigue of the second-level driver in response to determining that the above-mentioned eye detection information is information representing that the driver is in a state of driving fatigue, and the above-mentioned driver heart rate detection value is not within the preset driver fatigue value interval. Among them, the above-mentioned driver fatigue value interval can be the heart rate interval of the driver during daily rest pre-detected by the above-mentioned smart watch. Therefore, the driver's heart rate detection value is not within the preset driver fatigue value interval, which can represent that the driver is not in a state of fatigue driving determined from the heart rate perspective. Therefore, the driver is in a state of mild fatigue. Thus, a driver fatigue level indicator representing the fatigue of the second-level driver can be generated to indicate that the driver is transitioning to a fatigue state. Therefore, for drivers in this state, the reminder intensity can be appropriately increased.

[0047] The fourth step is to generate a driver fatigue level mark representing the third level of driver fatigue in response to determining that the above eye detection information is information representing that the driver is in a state of driving fatigue, and the above driver heart rate detection value is within a preset driver fatigue value interval. Among them, the driver's heart rate detection value is within the preset driver fatigue value interval (that is, when the driver's heart rate is basically the same as the driver's daily sleeping heart rate), which can represent that the driver is in a state of fatigue driving determined from the perspective of heart rate. Therefore, a driver fatigue level mark representing the third level of driver fatigue is generated to indicate that the driver is in a drowsy state. Therefore, drivers in this state need to be prompted by a stronger reminder sound.

[0048] The fifth step is to generate a vehicle state conversion mark based on each vehicle control information in the vehicle control information sequence. Among them, the above-mentioned vehicle state conversion mark can be a mark representing the vehicle lane keeping state, the vehicle speed keeping state, the vehicle changing from the lane keeping state to the lane changing state, or the vehicle changing from the speed keeping state to the acceleration state. Secondly, if the vehicle state included in each vehicle control information in the above-mentioned vehicle control information sequence is the lane keeping state, a vehicle state conversion mark representing the vehicle lane keeping state is generated. If the vehicle speed state included in each vehicle control information in the above-mentioned vehicle control information sequence is the vehicle speed keeping state, a vehicle state conversion mark representing the vehicle speed keeping state is generated. If the vehicle state included in each vehicle control information in the above-mentioned vehicle control information sequence changes from the lane keeping state to the lane changing state, a vehicle state conversion mark representing the vehicle changing from the lane keeping state to the lane changing state is generated. If the vehicle speed state included in each vehicle control information in the above-mentioned vehicle control information sequence changes from the speed keeping state to the acceleration state, a vehicle state conversion mark representing the vehicle changing from the speed keeping state to the acceleration state is generated.

[0049] Step 6: In response to determining that the eye detection information is information indicating that the driver is in a state of driving fatigue, the driver's heart rate detection value is not within a preset driver fatigue value interval, and the vehicle state transition mark is an mark that the vehicle changes from a speed-maintaining state to an accelerating state, a driver fatigue level mark indicating level 4 driver fatigue is generated. Among them, the eye detection information is information indicating that the driver is in a state of driving fatigue, the driver's heart rate detection value is not within a preset driver fatigue value interval, and the vehicle state transition mark is an mark that the vehicle changes from a speed-maintaining state to an accelerating state, which can indicate that the driver's fatigue driving state has affected the normal driving of the vehicle, and even if the driver is not completely asleep, a severe voice prompt is urgently needed.

[0050] The seventh step is to generate a driver fatigue level indicator representing level 5 driver fatigue in response to determining that the above-mentioned eye detection information is information representing that the driver is in a state of driving fatigue, and the above-mentioned driver heart rate detection value is not within the preset driver fatigue value interval, and the above-mentioned vehicle state transition mark is an indicator that the vehicle changes from a lane keeping state to a lane changing state. Among them, the above-mentioned eye detection information is information representing that the driver is in a state of driving fatigue, and the above-mentioned driver heart rate detection value is not within the preset driver fatigue value interval, and the above-mentioned vehicle state transition mark is an indicator that the vehicle changes from a lane keeping state to a lane changing state, which can represent that the driver is completely in a state of fatigue driving and in a deep sleep state. At the same time, this state also affects the normal driving of the vehicle. Therefore, a driver fatigue level indicator representing level 5 driver fatigue is generated. In order to provide the driver with the strongest fatigue prompt.

[0051] Optionally, before generating a corresponding driver fatigue warning tone based on the driver fatigue level identifier, the execution subject may further include the following steps:

[0052] The first step is to send a vibration prompt command to the smart watch so that the smart watch can perform a vibration operation to prompt the driver. In the case of abnormal driving of the driver (i.e., including fatigue driving), the vibration of the smart bracelet can be used to prompt the driver.

[0053] The second step is to control the driver's seat of the sanitation vehicle to vibrate to remind the driver. The driver can be reminded of the safety driving by combining the vibration of the driver's seat with the vibration of the smart bracelet.

[0054] Optionally, before generating a corresponding driver fatigue warning tone based on the driver fatigue level identifier, the execution subject may further include the following steps:

[0055] The first step is to generate vehicle auxiliary control information based on the above-mentioned vehicle control information sequence in response to determining that the above-mentioned driver fatigue level identifier is an identifier that characterizes level 2 driver fatigue, level 3 driver fatigue, level 4 driver fatigue or level 5 driver fatigue. Among them, first, a road image taken by the vehicle's front-view camera can be obtained. Then, the vehicle auxiliary control information can be generated by using the above-mentioned vehicle control information sequence and the road image through a preset vehicle control model. Here, the vehicle auxiliary control information can be used to control the sanitation vehicle to stay in the lane and drive along the lane. If an obstacle is detected in front of the same lane from the road image, the vehicle auxiliary control information can also include a control signal for the vehicle to decelerate. In addition, the vehicle auxiliary control information for the identifier of level 3 driver fatigue, level 4 driver fatigue or level 5 driver fatigue can also include braking information. The braking information can be used to control the vehicle to perform a braking operation to wake up the driver.

[0056] The second step is to send the above vehicle auxiliary control information to the vehicle control terminal to perform auxiliary control operations on the vehicle. In particular, when the driver is in a state of fatigue driving (for example, dozing off), the vehicle auxiliary control information can help the driver control the vehicle to maintain lane driving. Avoid dangerous driving behaviors such as sudden lane changes and vehicle acceleration caused by fatigue driving.

[0057] Step 104 , based on the driver fatigue level identifier, generate a corresponding driver fatigue prompt tone, and send the fatigue prompt tone to an audio device for voice prompting.

[0058] In some embodiments, the execution subject may generate a corresponding driver fatigue reminder sound based on the driver fatigue level indicator, and send the driver fatigue reminder sound to an audio device for sound prompting. The driver fatigue reminder sound is a problem reminder sound issued to the sanitation vehicle driver.

[0059] In some optional implementations of some embodiments, the execution subject generates a corresponding driver fatigue prompt tone based on the driver fatigue level identifier, and sends the fatigue prompt tone to an audio device for sound prompting, including:

[0060] For the above driver fatigue level indicators, perform the following driver fatigue dialogue steps:

[0061] The first step is to generate corresponding driver fatigue dialogue information based on the driver fatigue level represented by the above-mentioned driver fatigue level identifier. Among them, if the driver fatigue level identifier represents the first-level driver fatigue, the driver fatigue dialogue information that prompts the driver to concentrate can be generated. For example, "Driver, please concentrate". If the driver fatigue level identifier represents the second-level driver fatigue, the driver fatigue dialogue information that prompts the driver's eye fatigue can be generated. For example, "Driver, your eyes are tired, are you dozing off?". If the driver fatigue level identifier represents the third-level driver fatigue, the driver fatigue dialogue information that prompts the driver's eye fatigue can be generated. For example, "Driver, your eyes are closed, are you asleep?". If the driver fatigue level identifier represents the fourth-level driver fatigue, the driver fatigue dialogue information that prompts the driver to drive with severe fatigue can be generated. For example, "Driver, you are driving with severe fatigue now, wake up quickly?". If the driver fatigue level identifier represents the fifth-level driver fatigue, the driver fatigue dialogue information that prompts the driver to drive with severe fatigue can be generated. For example, "Driver, the garbage has fallen, wake up quickly!".

[0062] The second step is to convert the driver fatigue dialogue information into a driver fatigue prompt tone based on the preset frequency band energy proportion corresponding to the driver fatigue level represented by the driver fatigue level mark, and send the fatigue prompt tone to the audio device for sound prompting. Among them, first, the driver fatigue dialogue information can be converted into audio according to the preset sound to obtain the dialogue audio. Then, the corresponding background prompt tone can be added to the dialogue audio. For example, if the driver fatigue level mark represents the fatigue of the first-level driver, the corresponding background prompt tone can be a more cheerful prompt tone (such as bird calls, running water, etc.). At the same time, 30% of the prompt tone can be a frequency band energy between 2 kilohertz and 4 kilohertz. If the driver fatigue level mark represents the fatigue of the second-level driver, the corresponding background prompt tone can be a more urgent prompt tone (such as trumpet sound, drum sound, rainstorm sound, thunder sound, etc.). At the same time, 40% of the prompt tone can be a frequency band energy between 2 kilohertz and 4 kilohertz. If the driver fatigue level mark represents the fatigue of the third-level driver, the corresponding background prompt sound can be a noisy prompt sound (such as the sound of a police car siren, a horn, etc.). At the same time, 50% of the prompt sounds can be the frequency band energy between 2 kilohertz and 4 kilohertz. If the driver fatigue level mark represents the fatigue of the fourth-level driver, the corresponding background prompt sound can be a sudden prompt sound (such as the sound of glass breaking, etc.). At the same time, 70% of the prompt sounds can be the frequency band energy between 2 kilohertz and 4 kilohertz. If the driver fatigue level mark represents the fatigue of the fifth-level driver, the corresponding background prompt sound can be a very sudden prompt sound (such as an explosion, a scream, etc.). At the same time, 80% of the prompt sounds can be the frequency band energy between 2 kilohertz and 4 kilohertz.

[0063] In practice, the volume of the prompt tone can be increased step by step with the increase of fatigue level, but the maximum volume should not exceed 85 decibels, otherwise it will cause damage to the driver's hearing. As the prompt level increases, the duration can also be increased. The total duration of a single prompt (including continuous playback, intermittent playback or repeated playback) should be within 0.5 to 60 seconds. Specifically combine the selection of the prompt tone and the distinction between two thousand and four thousand frequency energy ratios. The spectrum can be converted from the time domain information of the audio file to the frequency domain information through Fourier transform. Here, the frequency domain energy diagram can be as follows Figure 2 shown.

[0064] The third step is to send a vibration prompt termination instruction to the smart watch and terminate the vibration operation of the driver's seat in response to receiving the driver's answer audio to the above-mentioned driver fatigue prompt sound within a preset time period, and terminate the vibration operation of the driver's seat. Among them, receiving the driver's answer audio to the above-mentioned driver fatigue prompt sound within a preset time period (for example, 1 second) indicates that the driver has changed from a fatigue driving state to a normal state. Therefore, the vibration of the smart bracelet and the driver's seat can be stopped. Secondly, the driver's answer audio can be the driver's answer voice to the driver's fatigue prompt sound. In addition, the auxiliary control can be terminated and the driver can control the vehicle.

[0065] Optionally, the above execution entity may further perform the following steps:

[0066] The first step is to adjust the energy proportion of the preset frequency band in response to the failure to receive the driver's answer audio to the driver fatigue reminder sound within the preset time period. The failure to receive the driver's answer audio to the driver fatigue reminder sound within the preset time period indicates that the driver has not been awakened, so it is necessary to increase the volume of the driver fatigue reminder sound (i.e., adjust the energy proportion of the preset frequency band) to remind the driver of fatigue again.

[0067] The second step is to determine the license plate number, driver number, sanitation vehicle work route, and driver fatigue information of the sanitation vehicle as sanitation operation risk information. Among them, the sanitation coordinate risk information represents the risky driving situation of the driver during the sanitation operation. The driver fatigue information may include but is not limited to at least one of the following: the facial image sequence of the sanitation vehicle driver, the vehicle control information sequence, the eye detection information, the driver fatigue level indicator, and the adjusted energy ratio.

[0068] The third step is to send the above sanitation operation risk information to the sanitation operation intelligent supervision platform for displaying and supervising the sanitation operation process of the above sanitation vehicles, and to execute the above driver fatigue dialogue step again according to the above adjusted energy ratio.

[0069] As an example, the interface diagram of the smart supervision platform for sanitation operations can be as follows: Figure 6 shown.

[0070] As another example, Figure 7 As shown, the intelligent supervision platform for sanitation operations can also display the vehicle status of various special vehicles (such as sanitation vehicles, sprinkler vehicles, road construction vehicles, etc.).

[0071] The various embodiments of steps 103-104 and their related contents are an inventive point of the embodiments of the present disclosure, which solves the technical problem 2 mentioned in the background technology: "Conventional voice prompts are difficult to attract the driver's attention in time, thereby making it difficult for the driver to change from a fatigue driving state to a normal state in time, thereby reducing driving safety". The factors that make it difficult to prompt the driver's fatigue in time are often as follows: Conventional voice prompts are difficult to attract the driver's attention in time, thereby making it difficult for the driver to change from a fatigue driving state to a normal state in time. If the above factors are solved, the driver can be reminded in time and change from a fatigue driving state to a normal state. In order to achieve this effect, first, the fatigue level of the driver is determined according to the driver's eye state, the driver's heart rate and the vehicle control information. By dividing the fatigue level, it can be used to provide the driver with corresponding voice prompts. At the same time, when the driver is in a fatigue state, the driver in the fatigue state can be awakened by the vibration of the smart bracelet, the vibration of the vehicle seat and the brake operation. Then, for different fatigue levels, corresponding driver fatigue prompt sounds are also generated. Here, prompt sounds with different frequency band energy proportions are also added to the fatigue prompt sounds corresponding to different levels. At the same time, the driver is informed by voice in the prompt tone combined with intelligent voice questioning. Therefore, after receiving the driver's voice answer, it can be determined that the driver has changed from a fatigue driving state to a normal state. In this way, the purpose of prompting the driver's fatigue is achieved. Also, because the prompt tone with different frequency band energy ratios is added, more powerful voice prompts can be given to drivers with different fatigue levels, while avoiding harming the driver's hearing. Thus, the purpose of attracting the driver's attention in time is achieved.

[0072] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the sound prompt method for driver fatigue of some embodiments of the present disclosure, the purpose of prompting the driver's fatigue can be quickly achieved. Specifically, the reason why it is difficult to achieve the purpose of the sound prompt for driver fatigue is that the prompt sound is relatively conventional, and there is a driver who is still in a fatigue driving state after hearing the prompt sound, thus, the driver's state reaction after hearing the sound is not considered. Based on this, the sound prompt method for driver fatigue of some embodiments of the present disclosure, first, through the built-in camera of the sanitation vehicle, obtains the face image sequence of the sanitation vehicle driver, and obtains the vehicle control information sequence of the central control terminal of the sanitation vehicle. Then, each sanitation vehicle driver face image in the above sanitation vehicle driver face image sequence is subjected to eye detection to obtain eye detection information. Through eye detection, it can be used to preliminarily judge the driver's fatigue state. Afterwards, in response to the above vehicle control information sequence and the above eye detection information satisfying the preset driver fatigue condition, a driver fatigue level identifier is generated based on the above vehicle control information sequence and the above eye detection information. By generating the driver fatigue level identifier, it can be used to further determine the driver's fatigue state and its level. Finally, based on the above-mentioned driver fatigue level identifier, a corresponding driver fatigue prompt sound is generated, and the above-mentioned fatigue prompt sound is sent to the audio device for sound prompting. Among them, the above-mentioned driver fatigue prompt sound is a problem prompt sound issued to the driver in the sanitation vehicle. By generating a driver fatigue prompt sound corresponding to the driver's fatigue level status, the driver can be given a sound prompt more effectively. At the same time, because the prompted driver fatigue prompt sound is a problem prompt sound, it can be used to determine the driver's fatigue state change according to the driver's voice feedback. Thereby, the driver can be given a fatigue sound prompt more accurately, and it can be ensured that the issued prompt sound can achieve the purpose of the sound prompt.

[0073] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a sound device for sanitation vehicle driver fatigue. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the sound device for sanitation vehicle driver fatigue can be specifically applied to various electronic devices.

[0074] like Figure 3As shown, in some embodiments, the sound device 300 for driver fatigue of sanitation vehicles includes: an acquisition unit 301, an eye detection unit 302, a first generation unit 303, and a generation and sound prompt unit 304. The acquisition unit 301 is configured to acquire a driver face image sequence and a vehicle control information sequence; the eye detection unit 302 is configured to perform eye detection on each driver face image in the driver face image sequence to obtain eye detection information; the first generation unit 303 is configured to generate a driver fatigue level identifier based on the vehicle control information sequence and the eye detection information in response to the vehicle control information sequence and the eye detection information satisfying a preset driver fatigue condition; the generation and sound prompt unit 304 is configured to generate a corresponding driver fatigue prompt sound based on the driver fatigue level identifier, and send the fatigue prompt sound to an audio device for sound prompting, wherein the driver fatigue prompt sound is a problem prompt sound issued to the driver.

[0075] It is understood that the units described in the device 300 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units contained therein, and will not be described in detail here.

[0076] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0077] like Figure 4 As shown, the electronic device 400 may include a processing device 401 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0078] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0079] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0080] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0081] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0082] The computer-readable medium may be included in the device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a facial image sequence of the driver of the sanitation vehicle through the built-in camera of the sanitation vehicle, and obtains a vehicle control information sequence of the central control terminal of the sanitation vehicle; performs eye detection on each facial image of the driver of the sanitation vehicle in the facial image sequence of the sanitation vehicle driver to obtain eye detection information; in response to the eye detection information satisfying the preset driver fatigue condition, generates a driver fatigue level identification based on the vehicle control information sequence and the eye detection information; generates a corresponding driver fatigue prompt tone based on the driver fatigue level identification, and sends the fatigue prompt tone to an audio device for sound prompting, wherein the driver fatigue prompt tone is a problem prompt tone issued to the driver in the sanitation vehicle.

[0083] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, an eye detection unit, a first generation unit, and a generation and sound prompting unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring a driver's facial image sequence and a vehicle control information sequence".

[0086] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0087] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for providing a sound reminder of driver fatigue of a sanitation vehicle, comprising: Obtain a facial image sequence of the driver of the sanitation vehicle through the built-in camera of the sanitation vehicle, and obtain a vehicle control information sequence of the central control terminal of the sanitation vehicle; Performing eye detection on each sanitation vehicle driver face image in the sanitation vehicle driver face image sequence to obtain eye detection information; In response to the eye detection information satisfying a preset driver fatigue condition, generating a driver fatigue level indicator based on the vehicle control information sequence and the eye detection information; Based on the driver fatigue level identifier, a corresponding driver fatigue prompt sound is generated, and the fatigue prompt sound is sent to an audio device for sound prompting, wherein the driver fatigue prompt sound is a problem prompt sound issued to the driver in the sanitation vehicle.

2. The method according to claim 1, wherein: The step of performing eye detection on each sanitation vehicle driver face image in the sanitation vehicle driver face image sequence to obtain eye detection information includes: Performing eye detection on each sanitation vehicle driver face image in the sanitation vehicle driver face image sequence to generate an eye state identification sequence, wherein the eye state identification in the eye state identification sequence indicates that the driver's eyes are in a normal state, a distracted state, or a fatigued state; In response to determining that each eye state identifier in the eye state identifier sequence satisfies a preset state condition, eye detection information is generated, wherein the eye detection information is information characterizing that the driver is in a distracted state or a driving fatigue state.

3. The method according to claim 2, wherein: The generating a driver fatigue level indicator based on the vehicle control information sequence and the eye detection information includes: Obtaining a driver's heart rate detection value detected by a smart watch carried by the driver in the sanitation vehicle, wherein the smart watch is pre-connected; In response to determining that the eye detection information is information indicating that the driver is in a distracted state, generating a driver fatigue level indicator indicating primary driver fatigue; In response to determining that the eye detection information is information indicating that the driver is in a driving fatigue state, and the driver's heart rate detection value is not within a preset driver fatigue value interval, generating a driver fatigue level indicator indicating secondary driver fatigue; In response to determining that the eye detection information is information indicating that the driver is in a driving fatigue state, and the driver's heart rate detection value is within a preset driver fatigue value interval, generating a driver fatigue level indicator indicating third-level driver fatigue; Generate a vehicle state transition identifier based on each vehicle control information in the vehicle control information sequence, wherein the vehicle state transition identifier is an identifier representing a vehicle lane keeping state, a vehicle speed keeping state, a vehicle transition from a lane keeping state to a lane changing state, or a vehicle transition from a speed keeping state to an acceleration state; In response to determining that the eye detection information is information indicating that the driver is in a driving fatigue state, and the driver's heart rate detection value is not within a preset driver fatigue value interval, and the vehicle state transition identifier is an identifier that the vehicle changes from a speed maintaining state to an accelerating state, generating a driver fatigue level identifier indicating fourth level driver fatigue; In response to determining that the eye detection information is information characterizing that the driver is in a driving fatigue state, and the driver's heart rate detection value is not within a preset driver fatigue value range, and the vehicle state transition identifier is an identifier of the vehicle changing from a lane keeping state to a lane changing state, a driver fatigue level identifier characterizing level five driver fatigue is generated.

4. The method according to claim 3, wherein: Before generating a corresponding driver fatigue warning tone based on the driver fatigue level identifier, the method further includes: Sending a vibration prompt instruction to the smart watch so that the smart watch performs a vibration operation to prompt the driver; The driver's seat in the sanitation vehicle is controlled to vibrate to remind the driver.

5. The method according to claim 3, wherein: Before generating a corresponding driver fatigue warning tone based on the driver fatigue level identifier, the method further includes: In response to determining that the driver fatigue level indicator is an indicator indicative of level 2 driver fatigue, level 3 driver fatigue, level 4 driver fatigue, or level 5 driver fatigue, generating vehicle auxiliary control information based on the vehicle control information sequence; The vehicle auxiliary control information is sent to a vehicle control terminal to perform auxiliary control operations on the vehicle.

6. The method according to claim 5, wherein: The generating a corresponding driver fatigue prompt tone based on the driver fatigue level identifier, and sending the fatigue prompt tone to an audio device for sound prompting, comprises: For the driver fatigue level indicator, the following driver fatigue dialogue steps are performed: generating corresponding driver fatigue dialogue information based on the driver fatigue level represented by the driver fatigue level identifier; Based on the energy proportion of a preset frequency band corresponding to the driver fatigue level represented by the driver fatigue level identifier, converting the driver fatigue dialogue information into a driver fatigue prompt tone, and sending the fatigue prompt tone to an audio device for sound prompting; In response to receiving the driver's answer audio to the driver fatigue prompt sound within a preset time period, a vibration prompt termination instruction is sent to the smart watch, and the vibration operation of the driver's seat is terminated.

7. The method according to claim 6, wherein: The method further comprises: In response to not receiving a driver's answer audio to the driver fatigue warning tone within a preset time period, adjusting the energy proportion of the preset frequency band to obtain an adjusted energy proportion; The license plate number, driver number, working route and driver fatigue information of the sanitation vehicle are determined as sanitation operation risk information, wherein the sanitation coordinate risk information represents the risk driving situation of the driver during the sanitation operation, and the driver fatigue information includes at least one of the following: the facial image sequence of the sanitation vehicle driver, the vehicle control information sequence, the eye detection information, the driver fatigue level identification and the adjusted energy proportion; The sanitation operation risk information is sent to the sanitation operation intelligent supervision platform for displaying and supervising the sanitation operation process of the sanitation vehicle, and the driver fatigue dialogue step is executed again according to the adjusted energy proportion.

8. A sound prompt device for sanitation vehicle driver fatigue, comprising: An acquisition unit is configured to acquire a facial image sequence of a driver of the sanitation vehicle through a built-in camera of the sanitation vehicle, and to acquire a vehicle control information sequence of a central control terminal of the sanitation vehicle; an eye detection unit configured to perform eye detection on each sanitation vehicle driver face image in the sanitation vehicle driver face image sequence to obtain eye detection information; a first generating unit configured to generate a driver fatigue level indicator based on the vehicle control information sequence and the eye detection information in response to the vehicle control information sequence and the eye detection information satisfying a preset driver fatigue condition; A generation and sound prompt unit is configured to generate a corresponding driver fatigue prompt sound based on the driver fatigue level identifier, and send the fatigue prompt sound to an audio device for sound prompting, wherein the driver fatigue prompt sound is a problem prompt sound issued to the driver in the sanitation vehicle.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images