Abnormal water drinking behavior detection method and device and storage medium

Abstract

The invention discloses an abnormal water drinking behavior detection method and device, electronic equipment and a storage medium, and relates to the field of safe driving, and the method comprises the steps: obtaining a key point location of each video frame through a key point detection model; if the drinking start frame is obtained, adding the throat point location into the key point location, and obtaining a movement track of the throat point location; if the water drinking ending frame is obtained, the throat point is shifted out of the key point position, and the water drinking behavior frequency is obtained according to the moving track of the throat point position; and determining whether an abnormal water drinking behavior exists according to the water drinking behavior frequency. According to the technical scheme provided by the embodiment of the invention, the automatic detection of the abnormal water drinking behavior is realized, the effective monitoring of driving safety and human health is ensured, the acquisition difficulty of the key point location is reduced, the influence of objective environmental factors and user behavior habits on the detection result is avoided, and the detection efficiency of the abnormal water drinking behavior is improved.

Description

Technical Field

[0001] The present invention relates to the field of safe driving, and in particular to a method, device and storage medium for detecting abnormal drinking behavior. Background Art

[0002] During vehicle driving, monitoring the driver's physical condition is crucial to ensuring driving safety, and the driver's drinking behavior not only affects driving safety, but is also an important indicator of the driver's health status.

[0003] In the prior art, the detection of a driver's drinking behavior usually uses the line of sight of the human eye as the detection basis. By obtaining the direction of the line of sight of the human eye when drinking water, it is determined whether the line of sight of the human eye is out of the driving direction, thereby determining whether there is abnormal behavior that affects driving.

[0004] However, this detection method requires a complete facial image to obtain the line of sight of the human eye, which is easily affected by the external environment. In particular, when the angle of sunlight is different, the position of the human eye is often different. At the same time, when the user wears glasses or sunglasses, the line of sight direction is often not obtained or cannot be accurately obtained, which has great limitations. Summary of the invention

[0005] The present invention provides a method, device and storage medium for detecting abnormal drinking behavior, so as to solve the problem of high complexity in detecting abnormal drinking behavior.

[0006] According to one aspect of the present invention, a method for detecting abnormal drinking behavior is provided, comprising:

[0007] The key points of each video frame are obtained through a key point detection model; wherein the key points include mouth points and container points;

[0008] If the drinking start frame is obtained according to the horizontal angle and interval distance between the mouth point and the container point, the throat point is added to the key point, and the movement trajectory of the throat point is obtained;

[0009] If the drinking end frame is obtained based on the horizontal angle and interval distance between the mouth point and the container point, the throat point is moved out of the key point, and the drinking behavior frequency is obtained based on the movement trajectory of the throat point;

[0010] Determine whether there is abnormal drinking behavior based on the drinking behavior frequency and the preset frequency threshold.

[0011] According to another aspect of the present invention, there is provided a device for detecting abnormal drinking behavior, comprising:

[0012] A key point acquisition module is used to acquire key points of each video frame through a key point detection model; wherein the key points include mouth points and container points;

[0013] A movement trajectory acquisition module is used to acquire a drinking start frame according to the horizontal angle and interval distance between the mouth point and the container point, add the throat point to the key point, and acquire the movement trajectory of the throat point;

[0014] A behavior frequency acquisition module is used to obtain the drinking end frame according to the horizontal angle and interval distance between the mouth point and the container point, move the throat point out of the key point, and obtain the drinking behavior frequency according to the movement trajectory of the throat point;

[0015] The abnormal behavior acquisition module is used to determine whether there is abnormal drinking behavior based on the drinking behavior frequency and a preset frequency threshold.

[0016] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the abnormal drinking behavior detection method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the abnormal drinking behavior detection method described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the abnormal drinking behavior detection method according to any embodiment of the present invention is implemented.

[0022] The technical solution of the embodiment of the present invention obtains the key points of each video frame through a key point detection model, firstly, after obtaining the drinking start frame according to the key points, the throat points are added to the key points, and then after obtaining the drinking end frame according to the key points, the throat points are moved out of the key points, and the drinking behavior frequency is obtained according to the movement trajectory of the throat points, and then whether there is abnormal drinking behavior is determined based on the drinking behavior frequency. This not only realizes the automatic detection of abnormal drinking behavior and ensures the continuous monitoring of driving safety and personal health, but also reduces the difficulty of obtaining key points, avoids the influence of objective environmental factors and user behavior habits on the detection results, and improves the detection efficiency of abnormal drinking behavior.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a method for detecting abnormal drinking behavior provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a schematic diagram of the position of a key point provided according to the first embodiment of the present invention;

[0027] Figure 3 is a flow chart of another abnormal drinking behavior detection method provided according to Embodiment 2 of the present invention;

[0028] Figure 4 is a flow chart of another abnormal drinking behavior detection method provided according to Embodiment 3 of the present invention;

[0029] Figure 5 is another schematic diagram of the positions of key points provided according to the third embodiment of the present invention;

[0030] Figure 6 is a schematic diagram of the structure of an abnormal drinking behavior detection device provided according to Embodiment 4 of the present invention;

[0031] Figure 7 It is a structural schematic diagram of an electronic device for implementing the abnormal drinking behavior detection method of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 This is a flow chart of an abnormal drinking behavior detection method provided in the first embodiment of the present invention. This embodiment can be applied to obtain abnormal drinking behavior according to the positional relationship of different key points. The method can be executed by the abnormal drinking behavior detection device in any embodiment of the present invention. The abnormal drinking behavior detection device can be implemented in the form of hardware and / or software. The abnormal drinking behavior detection device can be configured in a vehicle, for example, in an electronic device such as a vehicle-mounted terminal device or a vehicle-mounted device. Figure 1 As shown, the method includes:

[0036] S101. Obtain key points of each video frame through a key point detection model; wherein the key points include mouth points and container points.

[0037] Key point detection, also known as key point positioning or key point alignment, is to detect and obtain the predefined key point positions in the image containing the target object; the key point detection model is a mathematical model used to detect the positions of key points in the image; the image input to the key point detection model can be a face image or a body image; the output result of the key point detection model is the position of key points such as facial features or body joints.

[0038] The key point detection model can be built based on the Regression method, which sends the input image to the convolutional network, processes it through the fully connected layer, and finally outputs the coordinates of the key points; it can also be built based on the heatmap method, which outputs the coordinates of the key points by extracting key points from an image containing multiple channels; for example, the key point detection model can include a CenterNet model and an HRnet model.

[0039] In addition, the key point detection model can be detected by a top-down detection method, that is, the specified area in the image is first cut out separately, and then the cut-out part is detected; it can also be detected by a bottom-up detection method, that is, the original image is directly detected; optionally, in the embodiment of the present invention, there is no specific limitation on the construction method and detection method of the key point detection model.

[0040] The camera used to detect the driver's body image in the embodiment of the present invention can be installed at the A-pillar or steering wheel position of the cab to capture the driver's upper body area; Figure 2 As shown, the key points include mouth points (for example, lower lip points of the mouth) and container points (i.e., points of the container holding water, for example, opening points of the container); multiple driver images obtained from different drivers and at different shooting angles are used as training samples, and the positions of the key points are pre-marked in each training sample, and the marked training samples are input into the constructed key point detection model, so that the key point detection model has the ability to output key points for the input image after the training is completed.

[0041] As described in the above technical solution, for each video frame in the monitoring video, it is only necessary to obtain the area below the mouth, which not only reduces the number of facial features obtained, but also avoids the identity identification of the driver, ensuring the driver's information security. After obtaining the lower lip point and the opening point, the interval distance between the lower lip point and the opening point is obtained; wherein, the interval distance is expressed in the form of pixel values; at the same time, the horizontal angle between the lower lip point and the opening point is obtained; wherein, the horizontal angle refers to the angle between the line between the lower lip point and the opening point and the lower lip plane (i.e., the mouth plane); after obtaining the horizontal angle and interval distance between the lower lip point and the opening point, the horizontal angle is compared with a pre-set angle threshold, and the interval distance is compared with a pre-set distance threshold.

[0042] S102: If the drinking start frame is obtained based on the horizontal angle and the interval distance between the mouth point and the container point, the throat point is added to the key point, and the movement trajectory of the throat point is obtained.

[0043] In normal driving or stationary state, there is obviously no water container. When the driver lifts the water container and places it near his mouth, it means that he is ready to start drinking water. Therefore, in the current video frame, if the horizontal angle between the mouth point and the container point is less than or equal to the preset angle threshold, and the interval distance between the mouth point and the container point is less than or equal to the preset interval threshold, then the current video frame is determined to be the drinking start frame.

[0044] After determining that the current video frame is the drinking start frame, the throat point is added to the key points, that is, in the video frame after the drinking start frame, the key points include the mouth point, the container point and the throat point; wherein, the throat point may include the Adam's apple point; during the swallowing process, the up and down movement of the Adam's apple will help open or close the esophageal entrance, which is also part of the normal swallowing behavior. Therefore, the Adam's apple is also required to participate in the user's drinking behavior. Therefore, after determining that the current video frame is the drinking start frame, the Adam's apple point is also used as a key point to be identified.

[0045] During swallowing, the Adam's apple first moves upward for a certain distance and then returns to its initial position, thereby completing a drinking and swallowing behavior. Therefore, the Adam's apple point in the video frame can be marked, and the movement distance of the Adam's apple point can be obtained based on the marked Adam's apple points in multiple consecutive video frames. When the upward movement distance of the Adam's apple exceeds a preset movement threshold (for example, a 15-pixel value), it returns to its initial position again, and a swallowing behavior is completed.

[0046] S103: If the drinking end frame is obtained based on the horizontal angle and the interval distance between the mouth point and the container point, the throat point is moved out of the key point, and the drinking behavior frequency is obtained based on the movement trajectory of the throat point.

[0047] After completing the drinking behavior, the driver will move the water container away from the mouth, that is, after obtaining the drinking start frame, if it is detected in a video frame that the horizontal angle between the mouth point and the container point is greater than the preset angle threshold, and the interval distance between the mouth point and the container point is greater than the preset interval threshold, then the video frame is determined to be the drinking end frame, then all video frames between the drinking start frame and the drinking end frame represent a complete drinking behavior.

[0048] At the same time, the frequency of drinking behavior can also be obtained based on the number of swallowing behaviors in the complete drinking behavior between the drinking start frame and the drinking end frame; and the ratio of the number of swallowing behaviors to the unit detection time (for example, 1 minute) is the drinking behavior frequency; in addition, after determining that the current video frame is the drinking end frame, the throat point is moved out of the key point, that is, in the video frame after the drinking end frame, the key points only need to include the mouth point and the container point to avoid the appearance of redundant key points.

[0049] S104: Determine whether there is abnormal drinking behavior based on the drinking behavior frequency and a preset frequency threshold.

[0050] Under normal circumstances, adults also swallow at rest. For example, the frequency of swallowing at rest may be 1 to 3 times per minute. When it comes to active swallowing such as drinking, the frequency of swallowing will increase. For example, in normal drinking state, the frequency of swallowing may reach 5 to 10 times, that is, the frequency of swallowing in the current unit detection time may reach 5 to 10 times. Therefore, different frequency thresholds can be set in advance and the frequency of drinking behavior can be compared with each frequency threshold.

[0051] For example, 1 time, 5 times and 10 times can be used as different frequency thresholds respectively. When the frequency of drinking behavior is less than 1 time or greater than 10 times, it indicates abnormal swallowing action, which may be associated with sudden or habitual neurological control problems or muscle dysfunction, and it is obviously abnormal drinking behavior; when the frequency of drinking behavior is greater than or equal to 1 time and less than or equal to 5 times, it can be considered that the process does not actually involve drinking water. Even if the presence of a water container is detected, the driver only picked up the water container and placed it near his mouth, but did not actually drink water; when the frequency of drinking behavior is greater than or equal to 5 times and less than or equal to 10 times, it is considered that the driver is in a drinking state.

[0052] Optionally, in an embodiment of the present invention, determining whether there is abnormal drinking behavior based on the frequency of drinking behavior and a preset frequency threshold includes: obtaining an abnormal correction coefficient based on road condition information and vehicle status information, and obtaining a corrected frequency threshold based on the abnormal correction coefficient and the preset frequency threshold; determining whether there is abnormal drinking behavior based on the frequency of drinking behavior and the corrected frequency threshold.

[0053] Specifically, when the vehicle is traveling at a high speed or the vehicle torque is large, the driver needs to concentrate more. When the vehicle is on an uphill section or a traffic jam, the driver also needs to concentrate. It is not recommended for the driver to drink water in the current scenario. Therefore, the abnormal correction coefficient in the above scenario will reduce the preset frequency threshold, that is, the corrected frequency threshold in the above scenario is actually smaller than the preset frequency threshold.

[0054] The frequency of drinking behavior is compared with the above-mentioned corrected frequency threshold. Since the corrected frequency threshold is lower than the preset frequency threshold, drinking behavior under normal road conditions or normal vehicle status may be identified as abnormal drinking behavior in the above-mentioned scenario, that is, it is not recommended for the driver to drink water in the current scenario. Therefore, based on different road condition information and vehicle status information, the corrected frequency threshold is obtained, thereby realizing abnormal drinking behavior detection in different driving scenarios and ensuring driving safety in different driving scenarios.

[0055] Optionally, in an embodiment of the present invention, after determining whether there is abnormal drinking behavior based on the drinking behavior frequency and the preset frequency threshold, it also includes at least one of the following: if it is determined that the drinking behavior frequency is less than the third frequency threshold, issuing a drinking behavior alarm through the vehicle-mounted speaker or the vehicle-mounted display; generating a health report based on the drinking behavior frequency, the number of drinking behaviors and the duration of drinking; displaying the drinking behavior data through the vehicle-mounted display, or playing the drinking behavior data through the vehicle-mounted speaker; if it is determined that the drinking behavior frequency is less than the fourth frequency threshold, sending a notification message to a backup contact, or performing deceleration processing through the vehicle control system; wherein the fourth frequency threshold is less than the third frequency threshold.

[0056] Specifically, if the driver's drinking behavior frequency is too low, that is, less than the third frequency threshold, a drinking behavior alarm can be issued through the vehicle-mounted speaker or the vehicle-mounted display to remind the driver that there may be health problems or improper drinking behavior, and the driver is advised to slow down or stop to rest; in addition, if no drinking behavior is detected for a long time, a drinking reminder can also be issued through the vehicle-mounted speaker or the vehicle-mounted display to help the driver maintain a good physiological state and avoid fatigue or decreased concentration due to dehydration.

[0057] A health report can be generated based on the driver's drinking data in the current driving behavior or historical driving behavior, such as the frequency of drinking behavior, the number of drinking behaviors and the duration of drinking behavior, and then a customized drinking plan can be provided to the driver based on the driver's driving habits and physiological state; the drinking behavior data can also be displayed on the vehicle display screen, or played on the vehicle speakers, so that the user can know the drinking behavior data in real time.

[0058] It is also possible to send a notification message to a backup contact or perform deceleration or parking processing through the vehicle control system when more serious abnormal drinking behavior is detected, for example, when the frequency of drinking behavior is less than the fourth frequency threshold (the fourth frequency threshold is less than the third frequency threshold). In this way, when the driver is unable to control the vehicle due to serious health problems, the corresponding emergency processing plan can be executed to ensure vehicle driving safety.

[0059] The technical solution of the embodiment of the present invention obtains the key points of each video frame through a key point detection model, firstly, after obtaining the drinking start frame according to the key points, the throat points are added to the key points, and then after obtaining the drinking end frame according to the key points, the throat points are moved out of the key points, and the drinking behavior frequency is obtained according to the movement trajectory of the throat points, and then whether there is abnormal drinking behavior is determined based on the drinking behavior frequency. This not only realizes the automatic detection of abnormal drinking behavior and ensures the continuous monitoring of driving safety and personal health, but also reduces the difficulty of obtaining key points, avoids the influence of objective environmental factors and user behavior habits on the detection results, and improves the detection efficiency of abnormal drinking behavior.

[0060] Embodiment 2

[0061] Figure 3 This is a flow chart of a method for detecting abnormal drinking behavior provided by Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiment is that different first angle thresholds are obtained according to the container type. Figure 3 As shown, the method includes:

[0062] S201, obtaining key points of each video frame through a key point detection model.

[0063] S202: Obtain a container type in a video frame, and obtain a matching first angle threshold according to the container type.

[0064] Since the key point detection model has the key point recognition function after pre-training, it is obvious that it must have the container type recognition function, so that the location of the container point on the container can be known based on this; when the driver drinks water, the horizontal angles between the opening point and the mouth point of different types of containers are different. For example, when the container is a straw, the line between the straw opening and the mouth point is usually tilted downward, and the corresponding horizontal angle is relatively small; when the container is a water bottle or a water cup, the line between the opening point and the mouth point is usually tilted upward, and the corresponding horizontal angle is relatively large. Therefore, different first angle thresholds can be configured according to the different types of water containers used by the driver when drinking water; if the container is a straw, the first angle threshold is configured to a smaller value, and if the container is a water cup or a water bottle, the first angle threshold is configured to a larger value. Therefore, different first angle thresholds are configured according to different types of water containers, ensuring accurate detection of the drinking start frame and the drinking end frame.

[0065] S203: If the horizontal angle between the mouth point and the container point is less than or equal to the first angle threshold, and the interval distance between the mouth point and the container point is less than or equal to the preset interval threshold, it is determined that the drinking start frame is obtained, the throat point is added to the key point, and the movement trajectory of the throat point is obtained.

[0066] S204: If the horizontal angle between the mouth point and the container point is greater than the first angle threshold, or the interval distance between the mouth point and the container point is greater than the preset interval threshold, it is determined that the drinking end frame is obtained, the throat point is moved out of the key point, and the drinking behavior frequency is obtained according to the movement trajectory of the throat point.

[0067] S205: Determine whether there is abnormal drinking behavior based on the drinking behavior frequency and a preset frequency threshold.

[0068] Optionally, in an embodiment of the present invention, obtaining the key points of each video frame through a key point detection model includes: obtaining the container type in the video frame, and obtaining a matching first frequency threshold based on the container type; determining whether there is abnormal drinking behavior based on the drinking behavior frequency and a preset frequency threshold includes: determining whether there is abnormal drinking behavior based on the drinking behavior frequency and the first frequency threshold.

[0069] Specifically, different types of water containers have significantly different water flow rates at their openings. For example, when a user uses a straw to drink water, since the opening of the straw is small, the water flow rate passing through it is small, and the user needs to suck multiple times to reach the required amount of water to drink, so correspondingly, the number of swallowings is more; when a user uses a cup to drink water, since the opening is large, the water flow rate passing through it is large, and the user only needs to suck fewer times to reach the required amount of water to drink, so correspondingly, the number of swallowings is less; and when a user uses a bottle to drink water, since the opening is between the straw and the cup, the number of swallowings is often between the straw and the cup, so different first frequency thresholds can be configured for the straw, bottle and cup, respectively, for example, the first frequency thresholds corresponding to the straw, bottle and cup decrease in sequence; thus, different first frequency thresholds are configured according to different types of water containers, thereby ensuring the accuracy of the detection results of abnormal drinking behavior.

[0070] The technical solution of the embodiment of the present invention is to obtain a first angle threshold for matching according to different container types after the container type is acquired through the video frame, thereby improving the accuracy of the results of obtaining the drinking start frame and the drinking end frame, and to obtain a first frequency threshold for matching according to different container types, thereby further improving the detection result of abnormal drinking behavior.

[0071] Embodiment 3

[0072] Figure 4 This is a flow chart of a method for detecting abnormal drinking behavior provided by Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that the key points also include the mandibular point. Figure 4 As shown, the method includes:

[0073] S301. Obtain key points of each video frame through a key point detection model; wherein the key points include mouth points and container points.

[0074] S302: If the drinking start frame is obtained based on the horizontal angle and interval distance between the mouth point and the container point, and the vertical angle between the mouth point and the mandibular point, the throat point is added to the key point, and the movement trajectory of the throat point is obtained.

[0075] S303: If the drinking end frame is obtained based on the horizontal angle and interval distance between the mouth point and the container point, and the vertical angle between the mouth point and the mandibular point, the throat point is moved out of the key point, and the drinking behavior frequency is obtained based on the movement trajectory of the throat point.

[0076] S304: Determine whether there is abnormal drinking behavior based on the drinking behavior frequency and a preset frequency threshold.

[0077] like Figure 5 As shown, the key points also include the mandibular point. When drinking water, the driver will naturally raise his head, that is, increase the vertical angle between the mouth point and the mandibular point; wherein, the vertical angle between the mouth point and the mandibular point refers to the angle between the line between the mouth point (for example, the lower lip point) and the mandibular point and the throat plane; usually, when the driver is in a normal driving posture, the throat plane is perpendicular to or nearly perpendicular to the bottom plane of the vehicle; therefore, after obtaining the key points of each video frame, in addition to obtaining the horizontal angle and interval distance between the mouth point and the container point, the vertical angle between the mouth point and the mandibular point can also be obtained.

[0078] If the horizontal angle between the mouth point and the container point is less than or equal to the preset angle threshold, and the interval distance between the mouth point and the container point is less than or equal to the preset interval threshold, and at the same time, if the vertical angle between the mouth point and the mandibular point is greater than or equal to the second angle threshold, the current video frame can be determined to be the drinking start frame; similarly, after obtaining the drinking start frame, if the horizontal angle between the mouth point and the container point is greater than the preset angle threshold, or the interval distance between the mouth point and the container point is greater than the preset interval threshold, or the vertical angle between the mouth point and the mandibular point is less than the second angle threshold, the current video frame can be determined to be the drinking end frame, thereby ensuring the detection accuracy of the drinking start frame and the drinking end frame, and further ensuring the accuracy of the obtained numerical value of the drinking behavior frequency.

[0079] Optionally, in an embodiment of the present invention, after obtaining the drinking start frame, it also includes: obtaining a matching second frequency threshold based on the vertical angle between the mouth point and the mandibular point; determining whether there is abnormal drinking behavior based on the drinking behavior frequency and the preset frequency threshold, including: determining whether there is abnormal drinking behavior based on the drinking behavior frequency and the second frequency threshold.

[0080] Specifically, due to the structural characteristics of the human body, when the head is looking up at a larger angle, the amount of water swallowed when drinking is smaller, so the same amount of water needs more swallowing times, that is, a larger swallowing frequency; and when the head is not looking up or the upward angle is small, the amount of water swallowed when drinking is larger, so the same amount of water needs fewer swallowing times, that is, a smaller swallowing frequency. Therefore, the matching second frequency threshold can be obtained according to the vertical angle between the mouth point and the mandibular point. When the vertical angle between the mouth point and the mandibular point is larger, that is, when the head is looking up at a larger angle, the second frequency threshold is configured to a larger value; when the vertical angle between the mouth point and the mandibular point is smaller, that is, when the head is raised at a lower angle, the second frequency threshold is configured to a smaller value; thus, different second frequency thresholds are configured at different vertical angles between the mouth point and the mandibular point, thereby avoiding the influence of the driver's personal drinking habits on the abnormal drinking behavior detection results, ensuring that an accurate second frequency threshold is obtained, and improving the accuracy of the abnormal drinking behavior detection results.

[0081] The technical solution of the embodiment of the present invention, after obtaining the vertical angle between the mouth point and the mandibular point, determines the drinking start frame and the drinking end frame according to the vertical angle, thereby improving the accuracy of the acquisition results of the drinking start frame and the drinking end frame, and obtains a matching second frequency threshold according to the vertical angle between the mouth point and the mandibular point, which further improves the detection result of abnormal drinking behavior.

[0082] Embodiment 4

[0083] Figure 6 : is a structural block diagram of an abnormal drinking behavior detection device provided by Embodiment 4 of the present invention, and the device specifically comprises:

[0084] The key point acquisition module 401 is used to acquire the key points of each video frame through a key point detection model; wherein the key points include mouth points and container points;

[0085] The movement trajectory acquisition module 402 is used to acquire the drinking start frame according to the horizontal angle and the interval distance between the mouth point and the container point, add the throat point to the key point, and acquire the movement trajectory of the throat point;

[0086] The behavior frequency acquisition module 403 is used to obtain the drinking end frame according to the horizontal angle and interval distance between the mouth point and the container point, move the throat point out of the key point, and obtain the drinking behavior frequency according to the movement trajectory of the throat point;

[0087] The abnormal behavior acquisition module 404 is used to determine whether there is abnormal drinking behavior according to the drinking behavior frequency and a preset frequency threshold.

[0088] The technical solution of the embodiment of the present invention obtains the key points of each video frame through a key point detection model, firstly, after obtaining the drinking start frame according to the key points, the throat points are added to the key points, and then after obtaining the drinking end frame according to the key points, the throat points are moved out of the key points, and the drinking behavior frequency is obtained according to the movement trajectory of the throat points, and then whether there is abnormal drinking behavior is determined based on the drinking behavior frequency. This not only realizes the automatic detection of abnormal drinking behavior and ensures the continuous monitoring of driving safety and personal health, but also reduces the difficulty of obtaining key points, avoids the influence of objective environmental factors and user behavior habits on the detection results, and improves the detection efficiency of abnormal drinking behavior.

[0089] Optionally, the key point acquisition module is further used to acquire the container type in the video frame, and acquire a matching first angle threshold according to the container type;

[0090] The movement trajectory acquisition module is specifically used to determine that the drinking start frame is acquired if the horizontal angle between the mouth point and the container point is less than or equal to the first angle threshold, and the interval between the mouth point and the container point is less than or equal to the preset interval threshold;

[0091] The behavior frequency acquisition module is specifically used to determine that a drinking end frame has been acquired if the horizontal angle between the mouth point and the container point is greater than the first angle threshold, or the interval distance between the mouth point and the container point is greater than a preset interval threshold.

[0092] Optionally, the key point acquisition module is further used to acquire the container type in the video frame, and acquire a matching first frequency threshold according to the container type;

[0093] The abnormal behavior acquisition module is specifically used to determine whether there is abnormal drinking behavior based on the drinking behavior frequency and the first frequency threshold.

[0094] Optionally, the key points include mandibular points;

[0095] The movement trajectory acquisition module is specifically used to acquire the drinking start frame according to the horizontal angle and interval distance between the mouth point and the container point, and the vertical angle between the mouth point and the mandibular point;

[0096] The behavior frequency acquisition module is specifically used to obtain the drinking end frame according to the horizontal angle and interval distance between the mouth point and the container point, and the vertical angle between the mouth point and the mandibular point.

[0097] Optionally, the abnormal drinking behavior detection device is also used to obtain a matching second frequency threshold based on a vertical angle between a mouth point and a mandibular point.

[0098] The abnormal behavior acquisition module is specifically used to determine whether there is abnormal drinking behavior according to the drinking behavior frequency and the second frequency threshold.

[0099] Optionally, the abnormal behavior acquisition module is also used to obtain an abnormal correction coefficient based on road condition information and vehicle status information, and to obtain a correction frequency threshold based on the abnormal correction coefficient and a preset frequency threshold; and to determine whether there is abnormal drinking behavior based on the drinking behavior frequency and the correction frequency threshold.

[0100] Optionally, the abnormal drinking behavior detection device is also used to issue a drinking behavior alarm through a vehicle-mounted speaker or a vehicle-mounted display screen if it is determined that the drinking behavior frequency is less than a third frequency threshold; or generate a health report based on the drinking behavior frequency, the number of drinking behaviors and the drinking duration; or display the drinking behavior data through the vehicle-mounted display screen, or play the drinking behavior data through the vehicle-mounted speaker; or if it is determined that the drinking behavior frequency is less than a fourth frequency threshold, send a notification message to a backup contact, or perform deceleration processing through the vehicle control system; wherein the fourth frequency threshold is less than the third frequency threshold.

[0101] The above device can execute the abnormal drinking behavior detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the abnormal drinking behavior detection method provided by any embodiment of the present invention.

[0102] Embodiment 5

[0103] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0104] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the abnormal drinking behavior detection method.

[0107] In some embodiments, the abnormal drinking behavior detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the abnormal drinking behavior detection method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the abnormal drinking behavior detection method in any other appropriate manner (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or electronic device.

[0110] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein may be implemented on a heterogeneous hardware accelerator having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the heterogeneous hardware accelerator. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data electronic device), or a computing system that includes middleware components (e.g., an application electronic device), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0113] The computing system may include a client and an electronic device. The client and the electronic device are generally remote from each other and usually interact through a communication network. The relationship between the client and the electronic device is generated by computer programs running on corresponding computers and having a client-electronic device relationship with each other. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal drinking behavior, characterized in that: include: The key points of each video frame are obtained through a key point detection model; wherein the key points include mouth points and container points; If the drinking start frame is obtained according to the horizontal angle and interval distance between the mouth point and the container point, the throat point is added to the key point, and the movement trajectory of the throat point is obtained; If the drinking end frame is obtained based on the horizontal angle and interval distance between the mouth point and the container point, the throat point is moved out of the key point, and the drinking behavior frequency is obtained based on the movement trajectory of the throat point; Determine whether there is abnormal drinking behavior based on the drinking behavior frequency and the preset frequency threshold.

2. The method according to claim 1, characterized in that The key point detection model is used to obtain the key points of each video frame, including: Obtaining a container type in a video frame, and obtaining a matching first angle threshold according to the container type; The method of obtaining the drinking start frame according to the horizontal angle and the interval distance between the mouth point and the container point includes: If the horizontal angle between the mouth point and the container point is less than or equal to the first angle threshold, and the interval between the mouth point and the container point is less than or equal to the preset interval threshold, it is determined that the drinking start frame is obtained; The method of obtaining the drinking end frame according to the horizontal angle and the interval distance between the mouth point and the container point includes: If the horizontal angle between the mouth point and the container point is greater than the first angle threshold, or the interval between the mouth point and the container point is greater than the preset interval threshold, it is determined that the drinking end frame is obtained.

3. The method according to claim 1, characterized in that The key point detection model is used to obtain the key points of each video frame, including: Obtaining a container type in a video frame, and obtaining a matching first frequency threshold according to the container type; The determining whether there is abnormal drinking behavior according to the drinking behavior frequency and the preset frequency threshold includes: Whether there is abnormal drinking behavior is determined according to the drinking behavior frequency and the first frequency threshold.

4. The method according to claim 1, characterized in that: The key points include the mandibular point; The method of obtaining the drinking start frame according to the horizontal angle and the interval distance between the mouth point and the container point includes: If the drinking start frame is obtained based on the horizontal angle and interval distance between the mouth point and the container point, and the vertical angle between the mouth point and the mandibular point; The method of obtaining the drinking end frame according to the horizontal angle and the interval distance between the mouth point and the container point includes: The drinking end frame is obtained based on the horizontal angle and interval distance between the mouth point and the container point, and the vertical angle between the mouth point and the mandibular point.

5. The method according to claim 4, characterized in that After obtaining the drinking start frame, it also includes: According to the vertical angle between the mouth point and the mandibular point, a matching second frequency threshold is obtained; The determining whether there is abnormal drinking behavior according to the drinking behavior frequency and the preset frequency threshold includes: Whether there is abnormal drinking behavior is determined according to the drinking behavior frequency and the second frequency threshold.

6. The method according to claim 1, characterized in that The determining whether there is abnormal drinking behavior according to the drinking behavior frequency and the preset frequency threshold includes: Obtaining an abnormality correction coefficient according to the road condition information and the vehicle state information, and obtaining a correction frequency threshold according to the abnormality correction coefficient and a preset frequency threshold; Whether there is abnormal drinking behavior is determined according to the drinking behavior frequency and the modified frequency threshold.

7. The method according to claim 1, characterized in that After determining whether there is abnormal drinking behavior according to the drinking behavior frequency and the preset frequency threshold, at least one of the following is further included: If it is determined that the drinking behavior frequency is less than the third frequency threshold, a drinking behavior alarm is issued through the vehicle speaker or the vehicle display; Generate health reports based on drinking behavior frequency, number of drinking behaviors and drinking duration; Display drinking behavior data through the vehicle display screen, or play drinking behavior data through the vehicle speakers; If it is determined that the frequency of drinking behavior is less than a fourth frequency threshold, a notification message is sent to the backup contact, or a deceleration process is performed through the vehicle control system; wherein the fourth frequency threshold is less than the third frequency threshold.

8. A device for detecting abnormal drinking behavior, characterized in that: include: A key point acquisition module is used to acquire key points of each video frame through a key point detection model; wherein the key points include mouth points and container points; A movement trajectory acquisition module is used to acquire a drinking start frame according to the horizontal angle and interval distance between the mouth point and the container point, add the throat point to the key point, and acquire the movement trajectory of the throat point; A behavior frequency acquisition module is used to obtain the drinking end frame according to the horizontal angle and interval distance between the mouth point and the container point, move the throat point out of the key point, and obtain the drinking behavior frequency according to the movement trajectory of the throat point; The abnormal behavior acquisition module is used to determine whether there is abnormal drinking behavior based on the drinking behavior frequency and a preset frequency threshold.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the abnormal drinking behavior detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the abnormal drinking behavior detection method according to any one of claims 1 to 7 when executed.

Images