Driving behavior determination method and device, electronic equipment and storage medium

Abstract

The invention discloses a driving behavior determination method and device, electronic equipment and a storage medium, and relates to the intelligent driving technology, and the method comprises the steps: inputting a current frame image in a video stream into a lip segmentation model, and obtaining a lip segmentation result, the lip segmentation result being used for indicating whether the lip of a driver is visible in the current frame image; determining a target detection model according to the lip cutting result; and inputting the current frame image into the target detection model, and determining the current driving behavior of the driver. According to the method, only the detection model related to the lip cutting result is called at the current moment, so that the operation efficiency of the model can be improved, and the detection result can be quickly obtained; and the current driving behavior is specifically detected through the target detection model, so that the beneficial effect of improving the driving behavior detection accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a driving behavior determination method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of the automotive industry, safe driving assistance functions are receiving more and more attention. During vehicle driving, drivers generally have a variety of driving behaviors (such as smoking, drinking, and making phone calls, etc.). Therefore, how to accurately identify the current driving behavior and better serve users based on the safe driving assistance function under the current driving behavior is a key issue to be solved.

[0003] The existing method of determining driving behavior generally analyzes driving images through a trained detection model to obtain the driver's current driving behavior. Due to the diversity and complexity of users' driving behaviors, when a detection model is used to analyze multiple complex driving behaviors, there is a problem of long model detection time; and when the model is trained, when there are few training samples or the samples of each driving behavior are unbalanced, there is a problem of poor model detection accuracy. Summary of the invention

[0004] The present application provides a driving behavior determination method, device, electronic device and storage medium, which can improve the model operation efficiency and improve the detection accuracy of driving behavior.

[0005] In a first aspect, the present application provides a driving behavior determination method, comprising: inputting a current frame image in a video stream into a lip segmentation model to obtain a lip segmentation result, wherein the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image; determining a target detection model based on the lip segmentation result; inputting the current frame image into the target detection model to determine the driver's current driving behavior.

[0006] Optionally, when the lip segmentation result indicates that the driver's lips are visible in the current frame image, determining the target detection model based on the lip segmentation result includes: obtaining a lip binary image corresponding to the lip segmentation result; determining whether the lip binary image includes parallel straight line features; when the lip binary image includes the parallel straight line features, determining that the target detection model is a smoking detection model, and the smoking detection model is used to detect the driver's smoking behavior; when the lip binary image does not include the parallel straight line features, determining that the target detection model is a phone detection model, and the phone detection model is used to detect the driver's phone calling behavior.

[0007] Optionally, determining whether the lip binary image includes parallel straight line features includes: performing edge extraction on the lip binary image to obtain a lip edge contour; performing parallel straight line detection on the lip edge contour to determine whether the lip edge contour contains two parallel straight lines; if the lip edge contour contains two parallel straight lines, determining that the lip binary image includes parallel straight line features; if the lip edge contour does not contain two parallel straight lines, determining that the lip binary image does not include parallel straight line features.

[0008] Optionally, when the target detection model is the smoking detection model, inputting the current frame image into the target detection model to determine the current driving behavior of the driver includes: inputting the current frame image into the smoking detection model to obtain a smoking detection result, wherein the smoking detection result is used to indicate whether the current frame image includes a cigarette butt ring feature and / or a cigarette butt area feature; and determining whether the driver's current driving behavior is smoking driving based on the smoking detection result.

[0009] Optionally, when the target detection model is the phone detection model, inputting the current frame image into the target detection model to determine the driver's current driving behavior includes: inputting the current frame image into the phone detection model to obtain a phone detection result, wherein the phone detection result is used to indicate whether a phone contour feature is included in the current frame image; and determining whether the driver's current driving behavior is driving while making a phone call based on the phone detection result.

[0010] Optionally, when the lip segmentation result indicates that the driver's lips are not visible in the current frame image, the target detection model is a drinking detection model, and the drinking detection model is used to detect the driver's drinking behavior; inputting the current frame image into the target detection model to determine the driver's current driving behavior includes: inputting the current frame image into the drinking detection model to obtain a water bottle detection result, and the water bottle detection result is used to indicate whether the current frame image includes a water bottle contour feature; determining whether the driver's current driving behavior is drinking driving according to the water bottle detection result.

[0011] Optionally, after determining the current driving behavior of the driver, it also includes: determining at least one target hardware information to be linked based on the current driving behavior; calling at least one target hardware device corresponding to the target hardware information to provide the driver with corresponding in-vehicle linkage services based on at least one target hardware device.

[0012] In a second aspect, the present application provides a driving behavior determination device, the device comprising:

[0013] An image input module, used for inputting a current frame image in a video stream into a lip segmentation model to obtain a lip segmentation result, wherein the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image;

[0014] A model determination module, used to determine a target detection model according to the lip segmentation result;

[0015] The behavior determination module is used to input the current frame image into the target detection model to determine the current driving behavior of the driver.

[0016] In a third aspect, the present application further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the driving behavior determination method described in any embodiment of the present application.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the driving behavior determination method described in any embodiment of the present application when executed.

[0021] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the driving behavior determination method described in any embodiment of the present application.

[0022] The driving behavior determination solution provided in the embodiment of the present application analyzes the current frame image through a lip segmentation model to obtain a lip segmentation result, and the lip segmentation result is used to determine whether the driver's lips are visible during the current driving process; thereby determining the corresponding target detection model according to the lip segmentation result. Since only the detection model related to the lip segmentation result is retrieved at the current moment, the operating efficiency of the model can be improved, so that the detection result can be obtained quickly; and by performing targeted detection of the current driving behavior through the target detection model, the beneficial effect of improving the accuracy of driving behavior detection is achieved.

[0023] It should be noted that the above-mentioned computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged together with the processor of the driving behavior determination device, or may be packaged separately from the processor of the driving behavior determination device, and this application does not limit this.

[0024] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0026] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flow chart of a driving behavior determination method provided in an embodiment of the present application;

[0029] Figure 2a is another flowchart of the driving behavior determination method provided in an embodiment of the present application;

[0030] Figure 2b is a lip segmentation diagram of the smoking driving behavior provided in the embodiment of the present application;

[0031] Figure 3 is a structural schematic diagram of a driving behavior determination device provided in an embodiment of the present application;

[0032] Figure 4 It is a structural schematic diagram of an electronic device of a driving behavior determination method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the present application will be clearly and completely described below in conjunction with the drawings in the present embodiment. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0036] Figure 1 A flowchart of a driving behavior determination method provided in an embodiment of the present application, the present embodiment can be applied to the situation where the driving behavior of the user is determined during vehicle driving, the method can be executed by a driving behavior determination device, the driving behavior determination device can be implemented in the form of hardware and / or software, and integrated in an electronic device that executes the method. Preferably, the electronic device in the embodiment of the present application can be a server, or a computer device, etc.

[0037] refer to Figure 1 The driving behavior determination method provided in this embodiment includes but is not limited to the following steps:

[0038] S110: Input the current frame image in the video stream into the lip segmentation model to obtain a lip segmentation result.

[0039] Video stream, which means video data received by the server during the driving process of the vehicle. The server may receive the video stream by installing a high-resolution camera at an appropriate position inside the vehicle (such as the A-pillar or the steering wheel) to capture the upper body area of ​​the driver. The camera should be configured to continuously record the video stream and capture the possible driving behavior of the driver, so that the video stream captured by the camera is sent to the server, so that the received video stream is processed at the server, and at least one frame of image is extracted from the video stream, and then the driving behavior determination scheme provided in this embodiment is executed for each frame of image. In this embodiment, the driving behavior to be determined is the behavior that will distract the driver's driving experience or affect the driver's driving safety, such as making phone calls, smoking, or drinking water.

[0040] For the current frame image, this embodiment will input the current frame image into the lip segmentation model to obtain the lip segmentation result. Among them, the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image, so as to perform the subsequent operations of this embodiment according to the lip segmentation result. It is easy to understand that the lip segmentation result can include visible lips and invisible lips. Among them, visible lips indicate that the driver's lips are captured in the current frame image; invisible lips indicate that the driver's lips are not captured in the current frame image. Optionally, for the determination method of whether the lips are visible, if part of the lips are captured in the current frame image, the captured lip area can be determined as invisible lips if it is less than the preset value of the current frame image area; the captured lip area can be determined as visible lips if it is greater than the preset value of the current frame image area. Exemplarily, the current preset value can be 1%, or 2%, etc., which shall be subject to actual analysis.

[0041] Among them, the above-mentioned lip segmentation model is obtained based on deep learning algorithm training. Specifically, the present embodiment can obtain the lip segmentation model in the following manner: first, a large number of images containing different angles that may be generated by different drivers in the vehicle are collected; then, the lip area appearing in each image is accurately annotated, and the lip edge contour is annotated so that the model can recognize the lips based on the annotated lip edge contour; then, the annotated lip edge contour is learned through the image semantic segmentation model. Optionally, the current image semantic segmentation model can be implemented through the Unet model or the Hrnet model; finally, the model is trained using the annotated image data set, and when the training accuracy reaches convergence, the lip segmentation model is obtained so that it can recognize the lip segmentation result of each frame of the image.

[0042] S120: Determine a target detection model according to the lip segmentation result.

[0043] In this embodiment, the detection model may include at least two, which are used to determine the target detection model from the at least two detection models according to the lip segmentation result obtained in the current frame image. The purpose of setting at least two detection models in this embodiment is to call the corresponding detection model through the driving behavior that may exist in the current lip segmentation result, without requiring all the detection models to be in operation. When only the target detection model is called, only one model needs to be run at present, thereby improving the model operation efficiency and data processing efficiency.

[0044] When the lip segmentation result shows that the driver's lips are visible in the current frame image, the driver may be driving normally, or may be driving while making a phone call, or may be driving while smoking, etc. Therefore, in order to identify the current driving behavior, the target detection model can be determined to be a detection model trained based on the phone call behavior and the smoking behavior. Optionally, in order to further speed up the model detection efficiency, this embodiment can train a corresponding detection model for a driving behavior. For example, when the driver's lips are visible in the current frame image, the corresponding driving behavior may be driving while smoking, or driving while making a phone call. In order to perform accurate identification, this embodiment can further train a smoking detection model and a phone detection model to further analyze the lip image when the lips are visible. For example, when a foreign object is detected in the lip image, the target detection model to be called is determined to be a smoking detection model; when the lip area is detected to be complete and without any occlusion, the target detection model to be called is determined to be a phone detection model, etc. Specifically, there is no restriction on the possible driving behaviors contained in the current frame image when the driver's lips are visible, and the detection model corresponding to each driving behavior.

[0045] When the lip segmentation result shows that the driver's lips are not visible in the current frame image, the driver may be drinking water, or may be looking out the window with his head turned to the side, or may be turning his head to the back. In this case, the target detection model may be determined to be a drinking detection model trained based on drinking driving behavior to determine whether the driver is currently drinking water, etc.

[0046] In this embodiment, the target detection model determined by the lip segmentation result is a model that can accurately identify the driving behavior, which helps to improve the recognition accuracy of the driver's current driving behavior in subsequent steps.

[0047] S130: Input the current frame image into a target detection model to determine the driver's current driving behavior.

[0048] For the obtained target detection model, the current frame image needs to be re-analyzed to identify the current driving behavior of the driver from the current frame image. Exemplarily, when the lip segmentation result is determined to be visible, the target detection model is determined to be a detection model trained based on phone calls and smoking behaviors, then the current frame image is input into the current target detection model to identify that the current driving behavior includes smoking or phone calls based on the target detection model. If smoking or phone calls are not identified, then it can be considered that the user is driving normally; if smoking or phone calls are identified, then a voice reminder can be given to the driver to remind the driver to concentrate and drive safely; when the lips are not visible, the target detection model is a drinking detection model to analyze whether there is drinking behavior in the current frame image through the drinking detection model. If there is, it is determined that the driver's current driving behavior is drinking driving, and the driver can be reminded to concentrate and drive safely; if there is no drinking behavior, then it can be considered that the driver is driving normally.

[0049] The driving behavior determination method provided in the present embodiment analyzes the current frame image through a lip segmentation model to obtain a lip segmentation result, and the lip segmentation result is used to determine whether the driver's lips are visible during the current driving process; thereby determining the corresponding target detection model according to the lip segmentation result. Since only the detection model related to the lip segmentation result is retrieved at the current moment, the operation efficiency of the model can be improved, so that the detection result can be obtained quickly; and by performing targeted detection of driving behavior through the target detection model, the beneficial effect of improving the accuracy of driving behavior detection is achieved.

[0050] Figure 2a It is another flow chart of the driving behavior determination method provided in an embodiment of the present application. The embodiment of the present application is optimized on the basis of the above embodiments. The specific optimization is as follows: this embodiment explains in detail the process of determining the corresponding target detection model according to the lip segmentation results when the lip segmentation results are different.

[0051] See also Figure 2a The method of this embodiment includes but is not limited to the following steps:

[0052] S210: Input the current frame image in the video stream into the lip segmentation model to obtain a lip segmentation result.

[0053] The lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image.

[0054] S220: Determine whether the driver's lips are visible in the current frame image in the lip segmentation result.

[0055] If the driver's lips are visible in the current frame image, that is, yes, then execute step S230; if the driver's lips are not visible in the current frame image, that is, no, then execute step S260.

[0056] S230, obtaining a lip binary image corresponding to the lip segmentation result.

[0057] A binary image is a black-and-white image consisting of only black and white pixels, represented by 0 and 1. "1" represents white, which is used to indicate the recognized foreground image, and "0" represents black, which is used to indicate the background image. By obtaining a binary image of the lips, the recognized lip features can be accurately analyzed, which helps to determine the corresponding target detection model in the subsequent steps.

[0058] S231. Determine whether the lip binary image includes parallel straight line features.

[0059] In this embodiment, when the lip segmentation result is that the lips are visible, it indicates that the driver's current driving behavior may involve smoking driving or talking on the phone while driving. In order to further accurately distinguish between smoking driving and talking on the phone while driving and to determine the corresponding target detection model, in this embodiment, the target detection model is determined by determining whether the lip binary image includes parallel straight line features.

[0060] Specifically, in this embodiment, when the lip binary map includes parallel straight line features, that is, yes, then step S240 is executed, that is, the relevant operation of determining that the target detection model is a smoking detection model is executed; when the lip binary map does not include parallel straight line features, that is, no, then step S250 is executed, that is, the relevant operation of determining that the target detection model is a telephone detection model is executed.

[0061] For details, please refer to Figure 2b , Figure 2b is a lip segmentation diagram of the smoking driving behavior provided in the embodiment of the present application. Figure 2b It can be seen that when the user is smoking, the middle of the lips will be blocked by the cigarette rod, resulting in two parallel straight lines in the lip segmentation image. Therefore, based on whether the lip binary image includes parallel straight line features, it can be determined whether the target detection model to be called is a smoking detection model or a phone detection model.

[0062] In a preferred implementation, the above step S231 can be implemented by the following steps:

[0063] a) extracting the edge of the lip binary image to obtain the lip edge contour;

[0064] When the driver's lips are visible in the current frame image, the lip binary image obtained after segmentation by the lip segmentation model is gray-scale transformed, and the lip edge contour is further obtained based on the edge extraction algorithm. In this embodiment, the edge extraction algorithm can be implemented based on the Canny edge detection operator or the Sobel operator (Sobel for short).

[0065] b) Perform parallel straight line detection on the lip edge contour to determine whether the lip edge contour contains two parallel straight lines.

[0066] In this embodiment, the parallel straight line detection of the lip edge contour can be implemented based on the following method:

[0067] For each edge point in the lip edge contour, first express it in polar coordinate form as follows:

[0068] r=xcos(θ)+ysin(θ) (1)

[0069] In the above formula, r and θ are the parameters of the straight-line distance and the straight-line direction respectively.

[0070] Furthermore, for each edge point, according to its polar coordinate representation, add 1 to the corresponding (r, θ) position in the accumulator array; then find the peak value in the accumulator array, and the (r, θ) corresponding to the peak value is the parameter of the detected straight line; finally, analyze the detected straight line parameters, if there are two sets of parameters, so that the θ of these two sets of parameters are close and When it approaches a certain threshold, it is determined that there are parallel straight lines, wherein the current threshold may be 0.

[0071] c) If the lip edge contour contains two parallel straight lines, it is determined that the lip binary image includes parallel straight line features; if the lip edge contour does not contain two parallel straight lines, it is determined that the lip binary image does not include parallel straight line features.

[0072] S240: Determine that the target detection model is a smoking detection model.

[0073] In this embodiment, the smoking detection model is used to detect the smoking behavior of the driver. That is, when it is determined that the lip binary image includes parallel straight line features, it can only indicate that the user has a columnar object in his mouth, such as a lollipop, a straw, or a cigarette. Then, whether the current columnar object is a cigarette in a lit state needs to be further detected on the current frame image based on the smoking detection model.

[0074] S241. Input the current frame image into the smoking detection model to obtain the smoking detection result.

[0075] In this embodiment, the smoking detection model is obtained through training based on a deep learning algorithm. The training samples of the smoking detection model are multi-angle images of the driver smoking in the cockpit, and the ability of cigarette semantic segmentation is further constructed based on the deep learning algorithm, so as to train and obtain the smoking detection model. In this embodiment, the specific method of obtaining the smoking detection model is the same as the method of obtaining the lip segmentation model in the above embodiment, which will not be repeated here.

[0076] Furthermore, when the smoking detection model analyzes the current frame image, the features of the burning cigarette may be analyzed to obtain the corresponding smoking detection result, such as determining whether the smoking detection result includes the cigarette ring features and / or cigarette area features generated during the cigarette burning process.

[0077] Among them, the cigarette butt ring feature refers to a circle of obvious black or darker area formed near the cigarette butt when the cigarette is burning. It is mainly caused by physical and chemical processes in the combustion process (the part of the cigarette butt close to the cigarette butt is in close contact with the cigarette butt, and the oxygen supply is often insufficient. Tobacco combustion requires oxygen. Insufficient oxygen will lead to incomplete combustion, produce a large amount of carbonized substances, and form a black cigarette butt ring). It is an important feature of the cigarette butt burning process. Therefore, by determining whether there is a cigarette butt ring in the current frame image, it can be determined whether there is smoking behavior.

[0078] The feature of the cigarette butt area refers to the fact that due to the chemical reaction produced by combustion and the different chemical components in tobacco, the cigarette butt will appear black, white and gray after burning during smoking, mainly black and grayish white. The black part is mainly coke, which is composed of carbonized substances left after tobacco burning. When tobacco burns, the carbon compounds in tobacco undergo a combustion reaction to produce coke, which remains on the surface of the cigarette butt to form a black coke area. The grayish white part is mainly ash, which is composed of grayish white residual substances after tobacco burning. After tobacco burns, in addition to generating coke, it will also produce some grayish white gas and particulate matter. These substances condense in the air and deposit on the surface of the cigarette butt to form a grayish white ash area. Therefore, by determining whether a black coke area and a grayish white ash area are generated in the current frame image, it can be determined whether there is smoking behavior.

[0079] S242: Determine whether the driver's current driving behavior is smoking driving according to the smoking detection result.

[0080] When the smoking detection model detects that the current frame image includes cigarette butt ring features and / or cigarette butt area features, it can be determined that the driver's current driving behavior is smoking driving; when the smoking detection model does not detect that the current frame image includes cigarette butt ring features and cigarette butt area features, it indicates that the user has a cylindrical object in his mouth, such as a lollipop or a straw, and the current driving behavior is determined to be normal driving.

[0081] S250: Determine that the target detection model is a telephone detection model.

[0082] In this embodiment, the phone detection model is used to detect the driver's phone-making behavior, that is, when it is determined that the lip binary image does not include parallel straight line features, it indicates that the user's lip area is complete. When the lip area is complete, the driver's driving state may be making a phone call, driving normally and seriously, or talking to other passengers in the car. Then, in this embodiment, in order to detect the user's phone-making behavior, it is necessary to further detect whether the driver has made a phone call in the current frame image based on the phone detection model. In this embodiment, the specific method of obtaining the phone detection model is the same as the method of obtaining the lip segmentation model in the above embodiment, which will not be repeated here.

[0083] S251. Input the current frame image into the phone detection model to obtain the phone detection result.

[0084] In this embodiment, the phone detection model is obtained by training based on a deep learning algorithm. The training samples of the phone detection model are various gesture images of the driver holding the phone that may appear while driving, as well as multi-angle images of various phone specifications. The ability of phone semantic segmentation is further constructed based on the deep learning algorithm, so as to train and obtain the phone detection model.

[0085] In this embodiment, the phone detection result output by the phone detection model is used to indicate whether the phone contour feature is included in the current frame image.

[0086] S252: Determine whether the driver's current driving behavior is driving while talking on the phone based on the phone detection result.

[0087] When the phone detection result includes the phone contour feature, it indicates that the driver's current driving behavior is driving while making a phone call; when the phone detection result does not include the phone contour feature, it indicates that the driver's current driving behavior is normal driving.

[0088] S260: Determine that the target detection model is a drinking water detection model.

[0089] When the lip segmentation result obtained based on the lip segmentation model is that the driver's lips are invisible, the situation at this time may be that the driver drinks water and the water cup blocks the lips, resulting in invisible lips; it is also possible that the driver's head turns to the side or turns to the back, resulting in invisible lips. Based on this, when determining that the driver's lips are invisible in the current frame image, this embodiment determines that the target detection model is a drinking detection model, so as to detect whether the driver has drinking behavior in the current image frame based on the drinking detection model in the subsequent steps. In this embodiment, the specific method of obtaining the drinking detection model is the same as the method of obtaining the lip segmentation model in the above embodiment, and will not be repeated here.

[0090] S261. Input the current frame image into the drinking water detection model to obtain the water bottle detection result.

[0091] In this embodiment, the drinking water detection model is obtained through training based on a deep learning algorithm. The training samples of the drinking water detection model are various horizontal hand gesture images that may appear during driving, as well as multi-angle images of various horizontal specifications. The ability of semantic segmentation of water bottles is further constructed based on the deep learning algorithm, thereby training and obtaining the water bottle detection model.

[0092] In this embodiment, the water bottle detection result output by the horizontal detection model is used to indicate whether the water bottle contour feature is included in the current frame image.

[0093] S262: Determine whether the driver's current driving behavior is drinking water and driving according to the water bottle detection result.

[0094] When the water bottle detection result includes the water bottle contour feature, it indicates that the driver's current driving behavior is drinking and driving; when the water bottle detection result does not include the water bottle contour feature, it indicates that the driver's current driving behavior is normal driving.

[0095] In another preferred embodiment, the driving behavior determination solution provided in this embodiment can further perform the following operations after determining the current driving behavior of the driver:

[0096] Determine at least one target hardware information to be linked based on the current driving behavior. Call the target hardware device corresponding to the at least one target hardware information to provide the driver with a corresponding in-vehicle linkage service based on the at least one target hardware device. The purpose of this is to provide the driver with a corresponding in-vehicle linkage service for the current driving behavior of the driver through at least one target hardware device, so as to improve the driver's safe driving ability and improve the driver's experience.

[0097] Specifically, in this embodiment, each driving behavior and the corresponding at least one target hardware information to be linked can be pre-selected for mapping and association. For example, the at least one target hardware information that can be associated with the smoking driving behavior is the window information and the voice module information; the at least one target hardware information that can be associated with the phone call driving behavior is the music player information and the voice module information; the at least one target hardware information that can be associated with the drinking driving behavior is the voice module information and the drinking recording program identification information, etc. The specific at least one target hardware information to be linked corresponding to each driving behavior is not limited here.

[0098] Furthermore, when it is detected that the current driving behavior is smoking driving behavior, the in-vehicle linkage service provided to the driver may be: opening the window based on the window information, and reminding the user to drive safely based on the voice module; when it is detected that the current driving behavior is making a phone call while driving, the in-vehicle linkage service provided to the driver may be: lowering the volume of the music player, and reminding the user to drive safely based on the voice module; when it is detected that the current driving behavior is drinking water driving behavior, the in-vehicle linkage service provided to the driver may be: recording the driver's current drinking time and amount of water based on the drinking water recording program identification information to synchronize the driver's health data, and reminding the user to drive safely based on the voice module, etc.

[0099] The driving behavior determination method provided in this embodiment analyzes the current image frame through the lip segmentation model to obtain the lip segmentation result, so that when the lip segmentation result is that the lips are invisible, the target detection model is determined to be a water bottle detection model, and the water bottle detection model determines whether the driver's current driving behavior is drinking water driving by analyzing whether the water bottle contour is contained in the current frame image; when the lip segmentation result is that the lips are visible, further detect whether the lip binary image corresponding to the lip segmentation result contains parallel straight line features, so as to determine whether the target detection model is a smoking detection model or a telephone detection model according to whether the parallel straight line features are obtained; thereby, the driver's driving behavior can be accurately detected based on the corresponding detection model. In this embodiment, only one detection model needs to be called when identifying the driving behavior in each image frame, which avoids the resource consumption of running multiple programs at the same time, reduces the burden on the system, improves the model operation efficiency, and improves the model calculation speed; and in this embodiment, the target detection model is used to perform targeted identification of the driver's corresponding driving behavior, which can improve the accuracy of driving behavior identification. Further, by determining at least one target hardware information to be linked for each driving behavior, corresponding in-vehicle linkage services are provided for the driver's current driving behavior through at least one target hardware device, which can improve the driver's safe driving ability and enhance the driver's experience.

[0100] Figure 3 is a schematic diagram of a driving behavior determination device provided in an embodiment of the present application, and the device is suitable for executing the driving behavior determination method provided in an embodiment of the present application. Figure 3 As shown, the device may specifically include: an image input module 310, a model determination module 320 and a behavior determination module 330, wherein:

[0101] An image input module 310, used for inputting a current frame image in a video stream into a lip segmentation model to obtain a lip segmentation result, wherein the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image;

[0102] A model determination module 320, configured to determine a target detection model according to the lip segmentation result;

[0103] The behavior determination module 330 is used to input the current frame image into the target detection model to determine the current driving behavior of the driver.

[0104] The driving behavior determination device provided in the embodiment of the present application analyzes the current frame image through a lip segmentation model to obtain a lip segmentation result, and the lip segmentation result is used to determine whether the driver's lips are visible during the current driving process; thereby determining the corresponding target detection model according to the lip segmentation result. Since only the detection model related to the lip segmentation result is called at the current moment, the operation efficiency of the model can be improved, so that the detection result can be obtained quickly; and by performing targeted detection of the current driving behavior through the target detection model, the beneficial effect of improving the accuracy of driving behavior detection is achieved.

[0105] In one embodiment, when the lip segmentation result indicates that the driver's lips are visible in the current frame image, the model determination module 320 includes a binary image acquisition unit, a feature determination unit and a model determination unit, wherein:

[0106] A binary image acquisition unit, used for acquiring a lip binary image corresponding to the lip segmentation result;

[0107] a feature determination unit, used to determine whether the lip binary image includes parallel straight line features;

[0108] A model determination unit is used to determine that the target detection model is a smoking detection model when the lip binary map includes the parallel straight line features, and the smoking detection model is used to detect the smoking behavior of the driver; when the lip binary map does not include the parallel straight line features, determine that the target detection model is a phone detection model, and the phone detection model is used to detect the driver's phone calling behavior.

[0109] In one embodiment, the feature determination unit is specifically used to perform edge extraction on the lip binary image to obtain the lip edge contour; perform parallel straight line detection on the lip edge contour to determine whether the lip edge contour contains two parallel straight lines; if the lip edge contour contains two parallel straight lines, it is determined that the lip binary image includes a parallel straight line feature; if the lip edge contour does not contain two parallel straight lines, it is determined that the lip binary image does not include a parallel straight line feature.

[0110] In one embodiment, when the target detection model is the smoking detection model, the behavior determination module 330 is specifically used to input the current frame image into the smoking detection model to obtain a smoking detection result, wherein the smoking detection result is used to indicate whether the current frame image includes a cigarette butt ring feature and / or a cigarette butt area feature; and determine whether the driver's current driving behavior is smoking driving based on the smoking detection result.

[0111] In one embodiment, when the target detection model is the phone detection model, the behavior determination module 330 is further specifically used to input the current frame image into the phone detection model to obtain a phone detection result, wherein the phone detection result is used to indicate whether a phone contour feature is included in the current frame image; and determine whether the driver's current driving behavior is driving while making a phone call based on the phone detection result.

[0112] In one embodiment, when the lip segmentation result indicates that the driver's lips are not visible in the current frame image, the target detection model is a drinking detection model, and the drinking detection model is used to detect the driver's drinking behavior;

[0113] The behavior determination module 330 is further specifically used to input the current frame image into the drinking water detection model to obtain a water bottle detection result, wherein the water bottle detection result is used to indicate whether the current frame image includes a water bottle contour feature; and determine whether the driver's current driving behavior is drinking water driving based on the water bottle detection result.

[0114] In one embodiment, the device further includes a hardware linkage module and an in-vehicle service module, wherein:

[0115] A hardware linkage module, used to determine at least one target hardware information to be linked based on the current driving behavior;

[0116] The in-vehicle service module is used to call at least one target hardware device corresponding to the target hardware information to provide the driver with a corresponding in-vehicle linkage service based on at least one target hardware device.

[0117] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0118] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the driving behavior determination method described in any embodiment of the present application.

[0119] An embodiment of the present application further provides a computer-readable medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the driving behavior determination method described in any embodiment of the present application when executed.

[0120] Reference below Figure 4 , Figure 4 1 is a schematic diagram of the structure of an electronic device for a driving behavior determination method provided in an embodiment of the present application, and shows a schematic diagram of the structure of a computer system 500 suitable for implementing the electronic device in an embodiment of the present application. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] like Figure 4 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0122] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0123] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0124] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, etc., or any suitable combination of the above.

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The modules and / or units involved in the embodiments described in the present application may be implemented by software or hardware. The modules and / or units described may also be set in a processor, for example, it may be described as: a processor includes an image input module, a model determination module, and a behavior determination module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0127] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: inputting the current frame image in the video stream into the lip segmentation model to obtain a lip segmentation result, wherein the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image; determining a target detection model based on the lip segmentation result; inputting the current frame image into the target detection model to determine the driver's current driving behavior.

[0128] According to the technical solution of this embodiment, the current frame image is analyzed by a lip segmentation model to obtain a lip segmentation result, and the lip segmentation result is used to determine whether the driver's lips are visible during the current driving process; thereby, the corresponding target detection model is determined according to the lip segmentation result. Since only the detection model related to the lip segmentation result is called at the current moment, the operation efficiency of the model can be improved, so that the detection result can be obtained quickly; and the current driving behavior is targetedly detected by the target detection model, thereby achieving the beneficial effect of improving the accuracy of driving behavior detection.

[0129] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for determining driving behavior, characterized in that: include: Inputting a current frame image in a video stream into a lip segmentation model to obtain a lip segmentation result, wherein the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image; Determine a target detection model according to the lip segmentation result; The current frame image is input into the target detection model to determine the current driving behavior of the driver.

2. The driving behavior determination method according to claim 1, characterized in that: When the lip segmentation result indicates that the driver's lips are visible in the current frame image, determining the target detection model according to the lip segmentation result includes: Obtain a lip binary image corresponding to the lip segmentation result; Determining whether the lip binary image includes parallel straight line features; When the lip binary image includes the parallel straight line feature, determining that the target detection model is a smoking detection model, and the smoking detection model is used to detect the smoking behavior of the driver; When the lip binary image does not include the parallel straight line feature, the target detection model is determined to be a phone detection model, and the phone detection model is used to detect the driver's phone calling behavior.

3. The driving behavior determination method according to claim 2, characterized in that: The determining whether the lip binary image includes parallel straight line features comprises: Performing edge extraction on the lip binary image to obtain a lip edge contour; Performing parallel straight line detection on the lip edge contour to determine whether the lip edge contour contains two parallel straight lines; If the lip edge contour includes two parallel straight lines, it is determined that the lip binary image includes parallel straight line features; If the lip edge contour does not include two parallel straight lines, it is determined that the lip binary image does not include parallel straight line features.

4. The driving behavior determination method according to claim 2, characterized in that: When the target detection model is the smoking detection model, inputting the current frame image into the target detection model to determine the current driving behavior of the driver includes: Inputting the current frame image into the smoking detection model to obtain a smoking detection result, wherein the smoking detection result is used to indicate whether the current frame image includes a cigarette butt ring feature and / or a cigarette butt area feature; Determine whether the driver's current driving behavior is smoking driving according to the smoking detection result.

5. The driving behavior determination method according to claim 2, characterized in that: When the target detection model is the phone detection model, inputting the current frame image into the target detection model to determine the current driving behavior of the driver includes: Inputting the current frame image into the phone detection model to obtain a phone detection result, wherein the phone detection result is used to indicate whether the current frame image includes a phone outline feature; Determine whether the driver's current driving behavior is driving while making a phone call according to the phone detection result.

6. The driving behavior determination method according to claim 1, characterized in that: When the lip segmentation result indicates that the driver's lips are not visible in the current frame image, the target detection model is a drinking detection model, and the drinking detection model is used to detect the drinking behavior of the driver; The step of inputting the current frame image into the target detection model to determine the current driving behavior of the driver includes: Input the current frame image into the drinking water detection model to obtain a water bottle detection result, where the water bottle detection result is used to indicate whether the current frame image includes a water bottle contour feature; Determine whether the driver's current driving behavior is drinking water and driving according to the water bottle detection result.

7. The driving behavior determination method according to claim 1, characterized in that: After determining the current driving behavior of the driver, the method further includes: Determining at least one target hardware information to be linked based on the current driving behavior; At least one target hardware device corresponding to the target hardware information is called to provide the driver with a corresponding vehicle linkage service based on at least one target hardware device.

8. A driving behavior determination device, characterized in that: include: An image input module, used for inputting a current frame image in a video stream into a lip segmentation model to obtain a lip segmentation result, wherein the lip segmentation result is used to indicate whether the driver's lips are visible in the current frame image; A model determination module, used to determine a target detection model according to the lip segmentation result; The behavior determination module is used to input the current frame image into the target detection model to determine the current driving behavior of the driver.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the driving behavior determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the driving behavior determination method as described in any one of claims 1 to 7 is implemented.

Images