Driver foot action recognition-based warning method and device, and electronic device

Abstract

The application discloses a driver foot action recognition-based warning method and device and electronic equipment, and the method comprises the following steps: acquiring a first image collected by an image collection device, wherein a brake pedal and an accelerator pedal of a vehicle are both within the field of view of the image collection device; recognizing a driver's foot action based on the first image, wherein the foot action comprises braking, accelerating, and placing feet in a foot rest area; acquiring map information corresponding to the current position of the vehicle, and determining a foot operation that the driver should perform currently based on the map information; and if the foot action recognized based on the first image does not match the foot operation, corresponding warning information is sent. The technical scheme provided by the application can determine the operation that the driver should perform at the current position of the vehicle in combination with the map information, and corresponding warning information is sent when the actual recognized foot operation of the driver does not match the operation that should be performed currently, so that the driving behavior of the driver can be regulated.

Description

Technical Field

[0001] This invention relates to the field of vehicle driving safety technology, specifically to an alarm method, device, and electronic device based on driver foot movement recognition. Background Technology

[0002] Driver behavior has a significant impact on safety. Many existing technologies focus on detecting and assessing the driver's upper body, including fatigue detection and whether hands are off the steering wheel. However, there are currently no effective solutions for monitoring the driver's foot movements, such as the operation of the accelerator and brake pedals. For example, the need to slow down in advance at intersections without traffic lights, and to slow down near schools or when turning, currently relies on the driver's self-awareness, without effective monitoring and reminders. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an alarm method, device, and electronic device based on driver foot movement recognition to solve the problem that there is currently no effective monitoring and reminder of driver foot operation behavior.

[0004] According to a first aspect, embodiments of the present invention provide an alarm method based on driver foot movement recognition, the method comprising:

[0005] The first image acquired by the image acquisition device shows that both the vehicle's brake pedal and accelerator pedal are within the field of view of the image acquisition device.

[0006] Based on the first image recognition, the driver's foot movements include pressing the brake, pressing the accelerator, and placing the feet in the rest area;

[0007] Obtain map information corresponding to the current location of the vehicle, and determine the foot operation that the driver should perform based on the map information;

[0008] If the foot movement identified based on the first image does not match the foot operation, a corresponding alarm message will be issued.

[0009] In some optional implementations, before the first image is acquired by the image acquisition device, the method further includes:

[0010] Acquire the second image obtained by the image acquisition device;

[0011] Determine whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image;

[0012] If the brake pedal is not in the first area, the accelerator pedal is not in the second area, or the rest area is not in the third area, a prompt message for adjusting the image acquisition device is issued.

[0013] In some alternative implementations,

[0014] After determining whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image, the method further includes:

[0015] If the brake pedal is in the first region, the accelerator pedal is in the second region, and the resting area is in the third region, then the brake pedal, the accelerator pedal, and the resting area are identified respectively.

[0016] The relative position information of the brake pedal, the accelerator pedal, and the foot rest area in the second image is acquired and saved respectively.

[0017] The step of recognizing the driver's foot movements based on the first image includes:

[0018] Based on the relative position information and the first image, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the resting area.

[0019] In some optional implementations, determining whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image includes:

[0020] Acquire information from an infrared sensor installed under the driver's seat of the vehicle;

[0021] Based on the information collected by the infrared sensor, it is determined whether there are feet or legs in the vehicle's foot space when the image acquisition device acquires the second image;

[0022] If it is determined that there are no feet or legs under the driver's seat, then it is determined whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image.

[0023] In some optional implementations, determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image includes:

[0024] The first image is processed by weak perspective transformation according to the first target mapping relationship to obtain the first sub-image after weak perspective transformation. The first target mapping relationship is obtained based on the relative position information of the four corner points of the brake pedal in the second image.

[0025] The first image is processed by weak perspective transformation according to the second target mapping relationship to obtain the second sub-image after weak perspective transformation. The second target mapping relationship is obtained based on the relative position information of the four corner points of the accelerator pedal in the second image.

[0026] The first image is subjected to weak perspective transformation according to the third target mapping relationship to obtain the third sub-image after weak perspective transformation. The third target mapping relationship is obtained based on the relative position information of the four corner points of the resting area in the second image.

[0027] The first sub-image, the second sub-image, and the third sub-image are input into the classification branch of the deep learning model;

[0028] Obtain the location probability output by the classification branch, the location probability including the probability that the driver's foot is on the brake pedal based on the first sub-image, the probability that the driver's foot is on the accelerator pedal based on the second sub-image, and the probability that the driver's foot is in the resting area based on the third sub-image.

[0029] Based on the position probability, it is determined whether the driver's foot is on the brake pedal, the accelerator pedal, or the resting area, thus obtaining a first judgment result.

[0030] In some optional implementations, the step of determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image further includes:

[0031] The first image is input into the semantic segmentation branch of the deep learning model;

[0032] Obtain the boundary information of the driver's feet output by the semantic segmentation branch;

[0033] Based on the boundary information and the relative position information, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the foot rest area, and a second determination result is obtained;

[0034] The final judgment result is obtained by combining the first judgment result and the second judgment result.

[0035] In some optional implementations, after the image acquisition device acquires the first image, it further includes:

[0036] The first image is sent to a display device and / or a storage device and / or a remote monitoring platform.

[0037] According to a second aspect, embodiments of the present invention provide an alarm device based on driver foot movement recognition, comprising:

[0038] The acquisition module is used to acquire the first image captured by the image acquisition device, wherein the vehicle's brake pedal and accelerator pedal are both within the field of view of the image acquisition device;

[0039] The recognition module is used to recognize the driver's foot movements based on the first image, including pressing the brake, pressing the accelerator, and placing the feet in the rest area;

[0040] The judgment module is used to obtain map information corresponding to the current location of the vehicle, and to determine the foot operation that the driver should perform based on the map information.

[0041] The alarm module is used to issue a corresponding alarm message when the foot movement identified based on the first image does not match the foot operation.

[0042] According to a third aspect, embodiments of the present invention provide an electronic device, comprising:

[0043] The system includes a memory and a processor, which are interconnected. The memory stores a computer program, which, when executed by the processor, implements any of the alarm methods based on driver foot movement recognition described in the first aspect above.

[0044] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements any of the alarm methods based on driver foot movement recognition described in the first aspect.

[0045] This invention provides a vision-based driver foot operation recognition scheme. The required image acquisition device and other equipment are simple to install, and the related technologies such as image acquisition, image processing, and image recognition are relatively mature. Moreover, the images can be stored as an intuitive basis for judgment. This invention also provides an alarm scheme based on driver foot operation recognition. This scheme combines map information to determine the operation that should be performed based on the vehicle's current position. When the actual recognized driver foot operation does not match the operation that should be performed, a corresponding alarm message is issued. This can regulate the driver's driving behavior. For example, when a bus is about to arrive at a stop, it needs to slow down in advance. If the image recognizes that the driver has not applied the brake pedal at a certain distance before the bus stop, an alarm message is issued. The same applies when the vehicle is about to enter a ramp, turning area, or near a school. If the image recognizes that the driver has not applied the brake pedal at a certain distance, an alarm message is issued, thus adding an extra layer of protection for traffic safety. Attached Figure Description

[0046] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0047] Figure 1 A flowchart illustrating an alarm method based on driver foot movement recognition provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram showing the relative positions of the first region, the second region, and the third region in the image according to an embodiment of the invention.

[0049] Figure 3 A schematic diagram of the structure of an alarm device based on driver foot movement recognition provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. In the following descriptions of embodiments, "a plurality of" means two or more, unless otherwise expressly specified.

[0053] Please see Figure 1 This invention provides an alarm method based on driver foot movement recognition. This method can be implemented by an electronic device installed in a vehicle. The method includes:

[0054] S101: Acquire the first image captured by the image acquisition device, wherein the vehicle's brake pedal and accelerator pedal are both within the field of view of the image acquisition device; the image acquisition device may include a camera, which may be mounted at a suitable position below the driver's seat, directly above the brake pedal and accelerator pedal. Additionally, the driver's footrest area may also be within the field of view of the image acquisition device.

[0055] S102: Based on the first image, the driver's foot movements are recognized, including pressing the brake, pressing the accelerator, and placing the foot in the resting area; if the driver's resting area is not within the field of view of the image acquisition device, then if it is determined based on the first image that the driver's foot is not on the brake pedal or the accelerator pedal, then it is considered that the driver's foot is in the resting area.

[0056] S103: Obtain map information corresponding to the current location of the vehicle, and determine the foot operation that the driver should perform based on the map information; Of course, when determining the foot operation that the driver should perform based on the map information, the foot operation that should be performed is not always obtained, and except for some specific areas, most areas may not restrict the driver's operation.

[0057] S104: If the foot movement identified based on the first image does not match the foot operation, a corresponding alarm message is issued. If, when determining the foot operation that the driver should currently perform based on the map information, no foot operation is determined, it means that there are no restrictions on the driver's foot operation at this time. In this case, regardless of the driver's foot operation, no alarm message needs to be issued. The alarm message can be used to display on the screen, or to drive the vehicle's audio playback device to play a corresponding sound, or both.

[0058] This invention provides a vision-based method for recognizing driver foot movements. The required image acquisition device and other equipment are simple to install, and the related technologies such as image acquisition, image processing, and image recognition are relatively mature. Furthermore, the images can be stored as a direct basis for judgment. This invention also provides an alarm method based on driver foot movement recognition. This method combines map information to determine the appropriate action based on the vehicle's current location. When the actual recognized driver foot movement does not match the appropriate action, a corresponding alarm is issued. This can regulate driver behavior. For example, when a bus is about to arrive at a stop, it needs to slow down in advance. If, based on image recognition, the driver is not applying the brake pedal at a certain distance before the bus stop, an alarm is issued. The same applies when the vehicle is about to enter a ramp, turning area, or near a school. If, based on image recognition, the driver is not applying the brake pedal at a certain distance, an alarm is issued, thus adding an extra layer of protection for traffic safety.

[0059] In some specific embodiments, before the first image is acquired by the image acquisition device, the following steps are also included:

[0060] Acquire the second image obtained by the image acquisition device;

[0061] The system determines whether the brake pedal is located in the first region of the second image, whether the accelerator pedal is located in the second region of the second image, and whether the driver's foot rest area is located in the third region of the second image. The first, second, and third regions are defined based on the following principles: 1. Reasonableness; 2. Facilitating subsequent identification of whether the driver's foot intersects with the accelerator pedal region, the brake pedal region, or the foot rest area. Specifically, the first, second, and third regions can be defined as follows: Figure 2As shown; specifically, determining whether the brake pedal is in the first area can be done by determining whether the entire brake pedal is in the first area, or by determining whether most of the brake pedal (specifically, it can be determined according to proportion, for example, if it is greater than the preset proportion threshold of the entire brake pedal, then it is considered to be most of the brake pedal) is in the first area. The same applies to determining whether the accelerator pedal is in the second area and whether the resting area is in the third area.

[0062] If the brake pedal is not in the first area, the accelerator pedal is not in the second area, or the footrest area is not in the third area, a prompt message for adjusting the image acquisition device is issued. Adjusting the image acquisition device can specifically involve adjusting its installation position or the camera's orientation. The ultimate goal is to ensure that the image acquired by the image acquisition device simultaneously includes the brake pedal, accelerator pedal, and footrest area, and that their positions within the image are reasonable for subsequent image recognition.

[0063] Because the size and shape of the foot movement area vary significantly among different vehicle models, a single installation method cannot be used for the image acquisition device. Otherwise, the first image acquired would not simultaneously include the brake pedal, accelerator pedal, and the driver's footrest area, leading to inaccurate driver foot movement recognition. Therefore, in this embodiment of the invention, before acquiring the first image used for driver foot movement recognition, the installation position of the image acquisition device and / or the orientation of the camera need to be adjusted to ensure that the image acquired by the image acquisition device simultaneously includes the brake pedal, accelerator pedal, and footrest area, and that their positions within the image are reasonable.

[0064] In addition, since the image acquisition device may be installed at different locations or the camera may be facing different directions after installation, the above steps can be performed periodically (e.g., at a certain cycle) or irregularly during subsequent use.

[0065] In other optional embodiments, after acquiring the second image obtained by the image acquisition device, it is only necessary to determine whether the brake pedal is in the first region of the second image and whether the accelerator pedal is in the second region of the second image.

[0066] If the brake pedal is not in the first area or the accelerator pedal is not in the second area, a prompt message is issued to adjust the image acquisition device.

[0067] In some specific embodiments, after determining whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image, the method further includes:

[0068] If the brake pedal is in the first region, the accelerator pedal is in the second region, and the resting area is in the third region, then the brake pedal, the accelerator pedal, and the resting area are identified respectively.

[0069] The relative position information of the brake pedal, the accelerator pedal, and the foot rest area in the second image is acquired and saved respectively.

[0070] The step of recognizing the driver's foot movements based on the first image includes:

[0071] Based on the relative position information and the first image, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the resting area.

[0072] In this embodiment of the invention, the brake pedal is in the first region of the second image, the accelerator pedal is in the second region, and the footrest area is in the third region. This indicates that the relative positions of the brake pedal, accelerator pedal, and footrest area in the image acquired by the image acquisition device are relatively reasonable. At this time, the specific relative position information of the brake pedal, accelerator pedal, and footrest area in the image acquired by the image acquisition device is acquired and saved respectively. This makes it convenient to determine the image areas that the brake pedal, accelerator pedal, and footrest area should occupy in the first image when recognizing the driver's foot movements based on the first image. Then, it is determined whether the driver's foot area recognized from the first image intersects with the image area that the brake pedal should occupy in the first image, or with the image area that the accelerator pedal should occupy in the first image, or with the image area that the footrest area should occupy in the first image, thereby facilitating the judgment of the driver's foot movements.

[0073] In other optional embodiments, after determining whether the brake pedal is in the first region of the second image and whether the accelerator pedal is in the second region of the second image, the method further includes:

[0074] If the brake pedal is in the first region and the accelerator pedal is in the second region, then the brake pedal and the accelerator pedal are identified respectively.

[0075] The relative position information of the brake pedal and the accelerator pedal in the second image is acquired and saved respectively;

[0076] The step of recognizing the driver's foot movements based on the first image includes:

[0077] Based on the relative position information and the first image, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the resting area.

[0078] In some specific embodiments, determining whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image includes:

[0079] Acquire information from an infrared sensor installed under the driver's seat of the vehicle;

[0080] Based on the information collected by the infrared sensor, it is determined whether there are feet or legs in the vehicle's foot space when the image acquisition device acquires the second image;

[0081] If it is determined that there are no feet or legs under the driver's seat, then it is determined whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image.

[0082] Because the driver's feet or legs can obstruct the brake pedal, accelerator pedal, and / or footrest area in the field of view of the image acquisition device, the presence of feet or legs in the foot space under the driver's seat will obscure the brake pedal, accelerator pedal, and / or footrest area in the acquired second image, thus affecting the recognition of the brake pedal, accelerator pedal, and / or footrest area. To solve this problem, this embodiment of the invention installs an infrared sensor under the driver's seat and uses information collected by the infrared sensor to determine whether there are feet or legs in the vehicle's foot space. This ensures that the second image is acquired when there are no feet or legs in the vehicle's foot space, thereby correctly determining the relative position information of the brake pedal, accelerator pedal, and footrest area in the second image.

[0083] In other optional implementations, a smartphone can be used to adjust the position and angle of the image acquisition device during installation and subsequent use. Specifically, the smartphone can communicate with the image acquisition device to acquire images in real time. The smartphone has an installation marking function software installed, which displays the image acquired from the image acquisition device (similar to the second image mentioned above). This software also displays a first region, a second region, and a third region on the image. The positions of these three regions are fixed and do not change with the image content. If the installer finds that the brake pedal is not in the first region, the accelerator pedal is not in the second region, or the footrest area is not in the third region, they can adjust the installation position and / or angle of the image acquisition device until the brake pedal is in the first region, the accelerator pedal is in the second region, and the footrest area is in the third region. At this time, the smartphone also acquires the relative position information of the brake pedal, the accelerator pedal, and the footrest area in the second image. The smartphone can send the relative position information to the electronic device executing the method, so that the electronic device can determine whether the driver's foot is on the brake pedal, the accelerator pedal, or the rest area based on the relative position information and the newly acquired image by the image acquisition device.

[0084] In some specific implementations, determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image includes:

[0085] The first image is processed by weak perspective transformation according to the first target mapping relationship to obtain the first sub-image after weak perspective transformation. The first target mapping relationship is obtained based on the relative position information of the four corner points of the brake pedal in the second image.

[0086] The first image is processed by weak perspective transformation according to the second target mapping relationship to obtain the second sub-image after weak perspective transformation. The second target mapping relationship is obtained based on the relative position information of the four corner points of the accelerator pedal in the second image.

[0087] The first image is subjected to weak perspective transformation according to the third target mapping relationship to obtain the third sub-image after weak perspective transformation. The third target mapping relationship is obtained based on the relative position information of the four corner points of the resting area in the second image.

[0088] The first sub-image, the second sub-image, and the third sub-image are input into the classification branch of the deep learning model;

[0089] Obtain the location probability output by the classification branch, the location probability including the probability that the driver's foot is on the brake pedal based on the first sub-image, the probability that the driver's foot is on the accelerator pedal based on the second sub-image, and the probability that the driver's foot is in the resting area based on the third sub-image.

[0090] Based on the position probability, it is determined whether the driver's foot is on the brake pedal, the accelerator pedal, or the resting area, thus obtaining a first judgment result.

[0091] Specifically, the first target mapping relationship is determined based on the following two parameters: the relative position information of the four corner points of the brake pedal in the second image; and the relative position information of the four corner points of the brake pedal in the second image, determined based on empirical values ​​after excluding the factor of near objects appearing larger and far objects appearing smaller. The first sub-image mainly includes the brake pedal or the foot on the brake pedal (and may also include the portion of the brake pedal not obscured by the foot), with minimal background content.

[0092] The second target mapping relationship is determined based on the following two parameters: the relative position information of the four corner points of the accelerator pedal in the second image; and the relative position information of the four corner points of the accelerator pedal in the second image, determined based on empirical values ​​after excluding the near-large and far-small factor. The second sub-image mainly includes the accelerator pedal or the foot located on the accelerator pedal (and may also include the part of the accelerator pedal not obscured by the foot), with little other background content.

[0093] The third target mapping relationship is determined based on the following two parameters: the relative position information of the four corner points of the resting area in the second image; and the relative position information of the four corner points of the resting area in the second image, determined based on empirical values ​​after excluding the near-large and far-small factors. The third sub-image mainly includes the resting area or the foot located on the resting area (and may also include the resting area portion not obscured by the foot), with relatively little other background content.

[0094] In this embodiment of the invention, the position of the foot can be determined based on the probability of the brake pedal, accelerator pedal, and rest area being stepped on, as output by the classification branch of the deep learning model. Specifically, the foot is considered to be on the pedal with the highest probability of being stepped on. Furthermore, applying a weak perspective transformation to the first image can increase the robustness of the classification branch recognition.

[0095] In other specific embodiments, the step of determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image further includes:

[0096] The first image is input into the semantic segmentation branch of the deep learning model;

[0097] Obtain the boundary information of the driver's feet output by the semantic segmentation branch;

[0098] Based on the boundary information and the relative position information, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the foot rest area, and a second determination result is obtained;

[0099] The final judgment result is obtained by combining the first judgment result and the second judgment result. Specifically, the driver's foot is pressing the pedal based on the confidence levels of the first and second judgment results, as well as the prior knowledge that one foot will not press two pedals at the same time.

[0100] In other optional embodiments, determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image includes:

[0101] Based on the first image, the boundary information of the driver's feet was identified;

[0102] Based on the relative position information, the boundary information of the brake pedal, the boundary information of the accelerator pedal, and the boundary information of the rest area in the first image are estimated respectively.

[0103] Based on the boundary information of the driver's foot, as well as the estimated boundary information of the brake pedal, accelerator pedal, and rest area in the first image, it is determined whether the driver's foot overlaps with the brake pedal, accelerator pedal, or rest area. If the driver's foot overlaps with the brake pedal, the driver's foot action is determined to be braking; if the driver's foot overlaps with the accelerator pedal, the driver's foot action is determined to be accelerating; and if the driver's foot overlaps with the rest area, the driver's foot action is determined to be placing the foot in the rest area.

[0104] In some specific embodiments, the relative position information includes the relative position information of the corner points of the brake pedal and the second image, the relative position information of the corner points of the accelerator pedal and the second image, and the relative position information of the corner points of the rest area and the second image; or,

[0105] The relative position information includes the relative position information of the brake pedal boundary with the second image, the relative position information of the accelerator pedal boundary with the second image, and the relative position information of the rest area boundary with the second image.

[0106] In some specific embodiments, after the image acquisition device acquires the first image, it further includes:

[0107] The first image is sent to a display device and / or a storage device and / or a remote monitoring platform.

[0108] In this embodiment of the invention, the first image can be sent in real time to a display device on the vehicle (e.g., the center console display screen) to show an image of the driver's feet. This allows the driver to observe their driving habits and adjust their behavior according to warnings to meet safe driving requirements. Alternatively, the first image can be sent to a storage device and / or a remote monitoring platform for storage. Furthermore, the identified driver's foot movements and warning information can also be sent to a storage device and / or a remote monitoring platform for storage.

[0109] Accordingly, please refer to Figure 3 This invention provides an alarm device based on driver foot movement recognition, the device comprising:

[0110] The acquisition module 301 is used to acquire the first image acquired by the image acquisition device, wherein the vehicle's brake pedal and accelerator pedal are both within the field of view of the image acquisition device;

[0111] The recognition module 302 is used to recognize the driver's foot movements based on the first image, including pressing the brake, pressing the accelerator, and placing the foot in the rest area;

[0112] The judgment module 303 is used to obtain map information corresponding to the current location of the vehicle, and to determine the foot operation that the driver should perform based on the map information.

[0113] The alarm module 304 is used to issue a corresponding alarm message when the foot movement identified based on the first image does not match the foot operation.

[0114] This invention provides a vision-based driver foot operation recognition solution. The required image acquisition device and other equipment are simple to install, and the related technologies such as image acquisition, image processing, and image recognition are relatively mature. Moreover, the images can be stored as an intuitive basis for judgment. This invention also provides an alarm device based on driver foot operation recognition. This device combines map information to determine the operation that should be performed based on the vehicle's current position. When the actual recognized driver foot operation does not match the operation that should be performed, a corresponding alarm message is issued. This can regulate the driver's driving behavior. For example, when a bus is about to arrive at a stop, it needs to slow down in advance. If the image recognizes that the driver has not applied the brake pedal at a certain distance before the bus stop, an alarm message is issued. The same applies when the vehicle is about to enter a ramp, turning area, or near a school. If the image recognizes that the driver has not applied the brake pedal at a certain distance, an alarm message is issued, thus adding an extra layer of protection for traffic safety.

[0115] In some specific embodiments, the apparatus further includes:

[0116] The second image acquisition module is used to acquire the second image acquired by the image acquisition device;

[0117] The first judgment module is used to determine whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's foot rest area is in the third region of the second image;

[0118] The prompting module is used to issue a prompt message to adjust the image acquisition device when the brake pedal is not in the first area, the accelerator pedal is not in the second area, or the rest area is not in the third area.

[0119] In some specific embodiments, the apparatus further includes:

[0120] The first identification module is used to identify the brake pedal, the accelerator pedal, and the resting area respectively when the brake pedal is in the first area, the accelerator pedal is in the second area, and the resting area is in the third area;

[0121] The relative position information acquisition module is used to acquire and save the relative position information of the brake pedal, the accelerator pedal and the foot rest area in the second image, respectively.

[0122] The recognition module 302 is used to determine, based on the relative position information and the first image, whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area.

[0123] In some specific implementations, the first determination module includes:

[0124] The information acquisition unit is used to acquire information collected by an infrared sensor installed under the driver's seat of the vehicle.

[0125] The first judgment unit is used to determine, based on the information collected by the infrared sensor, whether there are feet or legs in the foot space of the vehicle when the image acquisition device acquires the second image;

[0126] The second judgment unit is used to determine whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image when it is determined that there are no feet or legs under the driver's seat.

[0127] In some specific embodiments, the identification module 302 includes:

[0128] The first image processing unit is configured to perform weak perspective transformation processing on the first image according to the first target mapping relationship to obtain the first sub-image after weak perspective transformation. The first target mapping relationship is obtained based on the relative position information of the four corner points of the brake pedal in the second image.

[0129] The second image processing unit is used to perform weak perspective transformation processing on the first image according to the second target mapping relationship to obtain a second sub-image after weak perspective transformation. The second target mapping relationship is obtained based on the relative position information of the four corner points of the accelerator pedal in the second image.

[0130] The third image processing unit is used to perform weak perspective transformation on the first image according to the third target mapping relationship to obtain a third sub-image after weak perspective transformation. The third target mapping relationship is obtained based on the relative position information of the four corner points of the resting area in the second image.

[0131] The first input unit is used to input the first sub-image, the second sub-image, and the third sub-image into the classification branch of the deep learning model;

[0132] The first model output acquisition unit is used to acquire the position probability of the classification branch output. The position probability includes the probability that the driver's foot is on the brake pedal based on the first sub-image, the probability that the driver's foot is on the accelerator pedal based on the second sub-image, and the probability that the driver's foot is in the resting area based on the third sub-image.

[0133] The first judgment unit is used to determine, based on the position probability, whether the driver's foot is located on the brake pedal, the accelerator pedal, or the resting area, and obtain a first judgment result.

[0134] In other specific embodiments, the identification module 302 further includes:

[0135] The second input unit is used to input the first image into the semantic segmentation branch of the deep learning model;

[0136] The second model output acquisition unit is used to acquire the boundary information of the driver's feet output by the semantic segmentation branch;

[0137] The second judgment unit is used to determine, based on the boundary information and the relative position information, whether the driver's foot is located on the brake pedal, on the accelerator pedal, or in the resting area, and obtain a second judgment result.

[0138] The comprehensive judgment unit is used to combine the first judgment result and the second judgment result to obtain the final judgment result.

[0139] In some specific embodiments, the relative position information includes the relative position information of the corner points of the brake pedal and the second image, the relative position information of the corner points of the accelerator pedal and the second image, and the relative position information of the corner points of the rest area and the second image; or,

[0140] The relative position information includes the relative position information of the brake pedal boundary with the second image, the relative position information of the accelerator pedal boundary with the second image, and the relative position information of the rest area boundary with the second image.

[0141] In some specific embodiments, the apparatus further includes:

[0142] The sending module is used to send the first image to a display device and / or a storage device and / or a remote monitoring platform.

[0143] The embodiments of the present invention are device embodiments based on the same inventive concept as the method embodiments described above. Therefore, for specific technical details and corresponding technical effects, please refer to the method embodiments described above, and they will not be repeated here.

[0144] This invention also provides an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 41 and a memory 42, wherein the processor 41 and the memory 42 can communicate with each other via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0145] Processor 41 can be a central processing unit (CPU). Processor 41 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0146] Memory 42, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the alarm method based on driver foot movement recognition in this embodiment of the invention (e.g., Figure 3 The first acquisition module 301, identification module 302, judgment module 303, and alarm module 304 are shown. The processor 41 executes various functional applications and data processing by running non-transitory software programs, instructions, and modules stored in the memory 42, thereby realizing the alarm method based on driver foot movement recognition in the above method embodiment.

[0147] The memory 42 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 41, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 42 may optionally include memory remotely located relative to the processor 41, and these remote memories may be connected to the processor 41 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] The one or more modules are stored in the memory 42, and when executed by the processor 41, they execute the alarm method based on driver foot movement recognition in the above method embodiment.

[0149] The specific details of the aforementioned electronic device can be understood by referring to the relevant descriptions and effects in the above method embodiments, and will not be repeated here.

[0150] Accordingly, this embodiment of the invention also provides a computer-readable storage medium for storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described alarm method embodiment based on driver foot movement recognition and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0151] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0152] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0153] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An alarm method based on driver foot movement recognition, characterized in that, The method includes: The first image acquired by the image acquisition device shows that both the vehicle's brake pedal and accelerator pedal are within the field of view of the image acquisition device. The driver's foot movements are identified based on the first image. These foot movements include pressing the brake, pressing the accelerator, and placing the foot in the resting area. Specifically, the driver's foot movements are determined by judging whether the area of ​​the driver's foot identified from the first image intersects with the area of ​​the brake pedal, the area of ​​the accelerator pedal, or the area of ​​the resting area in the first image. Obtain map information corresponding to the current location of the vehicle, and determine the foot operation that the driver should perform based on the map information; If the foot movement identified based on the first image does not match the foot operation, a corresponding alarm message will be issued; Before the first image is acquired by the image acquisition device, the method further includes: Acquire the second image obtained by the image acquisition device; Determine whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image; If the brake pedal is not in the first area, the accelerator pedal is not in the second area, or the rest area is not in the third area, a prompt message for adjusting the image acquisition device is issued; The step of determining whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image includes: Acquire information from an infrared sensor installed under the driver's seat of the vehicle; Based on the information collected by the infrared sensor, it is determined whether there are feet or legs in the vehicle's foot space when the image acquisition device acquires the second image; If it is determined that there are no feet or legs under the driver's seat, then it is determined whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image.

2. The method according to claim 1, characterized in that, After determining whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image, the method further includes: If the brake pedal is in the first region, the accelerator pedal is in the second region, and the resting area is in the third region, then the brake pedal, the accelerator pedal, and the resting area are identified respectively. The relative position information of the brake pedal, the accelerator pedal, and the foot rest area in the second image is acquired and saved respectively. The step of recognizing the driver's foot movements based on the first image includes: Based on the relative position information and the first image, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the resting area.

3. The method according to claim 2, characterized in that, The step of determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image includes: The first image is processed by weak perspective transformation according to the first target mapping relationship to obtain the first sub-image after weak perspective transformation. The first target mapping relationship is obtained based on the relative position information of the four corner points of the brake pedal in the second image. The first image is processed by weak perspective transformation according to the second target mapping relationship to obtain the second sub-image after weak perspective transformation. The second target mapping relationship is obtained based on the relative position information of the four corner points of the accelerator pedal in the second image. The first image is subjected to weak perspective transformation according to the third target mapping relationship to obtain the third sub-image after weak perspective transformation. The third target mapping relationship is obtained based on the relative position information of the four corner points of the resting area in the second image. The first sub-image, the second sub-image, and the third sub-image are input into the classification branch of the deep learning model; Obtain the location probability output by the classification branch, the location probability including the probability that the driver's foot is on the brake pedal based on the first sub-image, the probability that the driver's foot is on the accelerator pedal based on the second sub-image, and the probability that the driver's foot is in the resting area based on the third sub-image. Based on the position probability, it is determined whether the driver's foot is on the brake pedal, the accelerator pedal, or the resting area, thus obtaining a first judgment result.

4. The method according to claim 3, characterized in that, The step of determining whether the driver's foot is located on the brake pedal, the accelerator pedal, or the footrest area based on the relative position information and the first image also includes: The first image is input into the semantic segmentation branch of the deep learning model; Obtain the boundary information of the driver's feet output by the semantic segmentation branch; Based on the boundary information and the relative position information, it is determined whether the driver's foot is located on the brake pedal, the accelerator pedal, or the foot rest area, and a second determination result is obtained; The final judgment result is obtained by combining the first judgment result and the second judgment result.

5. The method according to claim 1, characterized in that, After the first image is acquired by the image acquisition device, the method further includes: The first image is sent to a display device and / or a storage device and / or a remote monitoring platform.

6. An alarm device based on driver foot movement recognition, characterized in that, include: The acquisition module is used to acquire the first image captured by the image acquisition device, wherein the vehicle's brake pedal and accelerator pedal are both within the field of view of the image acquisition device; The recognition module is used to recognize the driver's foot movements based on the first image, including pressing the brake, pressing the accelerator, and placing the feet in the rest area; Specifically, the driver's foot movement is determined by judging whether the area of ​​the driver's foot identified from the first image intersects with the area of ​​the brake pedal, the area of ​​the accelerator pedal, or the area of ​​the footrest in the first image. The judgment module is used to obtain map information corresponding to the current location of the vehicle, and to determine the foot operation that the driver should perform based on the map information. The alarm module is used to issue a corresponding alarm message when the foot movement identified based on the first image does not match the foot operation. The device further includes: The second image acquisition module is used to acquire the second image acquired by the image acquisition device; The first judgment module is used to determine whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's foot rest area is in the third region of the second image; The prompting module is used to issue a prompt message to adjust the image acquisition device when the brake pedal is not in the first area, the accelerator pedal is not in the second area, or the rest area is not in the third area; The first determination module includes: The information acquisition unit is used to acquire information collected by an infrared sensor installed under the driver's seat of the vehicle. The first judgment unit is used to determine, based on the information collected by the infrared sensor, whether there are feet or legs in the foot space of the vehicle when the image acquisition device acquires the second image; The second judgment unit is used to determine whether the brake pedal is in the first region of the second image, whether the accelerator pedal is in the second region of the second image, and whether the driver's footrest area is in the third region of the second image when it is determined that there are no feet or legs under the driver's seat.

7. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores a computer program, which, when executed by the processor, implements the alarm method based on driver foot movement recognition as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a processor, implements the alarm method based on driver foot movement recognition as described in any one of claims 1 to 5.

Images