Behavior recognition method and device, electronic device, and vehicle

Abstract

The present disclosure relates to a behavior recognition method and device, electronic equipment and vehicle. The method comprises: obtaining a first image set and a second image set of a driver; obtaining a first optical flow vector field according to the first image set and a second optical flow vector field according to the second image set; obtaining a vector numerical difference value of the first optical flow vector field and the second optical flow vector field; and determining whether the driver has dangerous driving behavior according to the vector numerical difference value. The method provided in the present disclosure can analyze the change of the optical flow vector field based on image analysis to recognize whether the driver has dangerous driving behavior, and can monitor in real time whether the driver has dangerous driving behavior to avoid safety hazards in time.

Description

Technical Field

[0001] This disclosure relates to the field of driving safety protection technology, specifically to a behavior recognition method, device, electronic device, and vehicle. Background Technology

[0002] Currently, there are two main methods for monitoring drivers' driving behavior: one is to install surveillance cameras to monitor drivers' behavior, and then have staff on the monitoring platform alert the driver when dangerous behavior is detected; the other is to use sensors to monitor the driver's heart rate or smell, which is prone to large errors and is susceptible to environmental interference. Therefore, existing technologies for identifying dangerous driving behaviors still pose significant driving safety risks.

[0003] Therefore, how to monitor whether drivers are engaging in dangerous driving behavior in real time is an urgent problem that needs to be solved. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a behavior recognition method, device, electronic device, and vehicle.

[0005] According to a first aspect of the present disclosure, a behavior recognition method is provided, comprising:

[0006] Obtain the first and second image sets of the driver;

[0007] A first optical flow vector field is obtained based on the first image set, and a second optical flow vector field is obtained based on the second image set;

[0008] Obtain the vector numerical difference between the first optical flow vector field and the second optical flow vector field;

[0009] The presence of dangerous driving behavior by the driver is determined based on the difference in the vector values.

[0010] Optionally, acquiring the first image set and the second image set of the driver includes:

[0011] The driver is acquired at least two frames within a first set time period as the first image set, and at least two frames within a second set time period as the second image set; wherein the first set time and the second set time period are consecutive.

[0012] Optionally, acquiring at least two frames of the driver within a first set of time as the first image set, and acquiring at least two frames of the driver within a second set of time as the second image set, includes:

[0013] Two consecutive frames of images of the driver are acquired as the first image set;

[0014] After obtaining the first image set, two consecutive frames of the driver's image are obtained as the second image set, wherein the two frames in the second image set are consecutive to the two frames in the first image set.

[0015] Optionally, obtaining the first optical flow vector field based on the first image set and obtaining the second optical flow vector field based on the second image set includes:

[0016] Based on the first image set and the second image set, the first optical flow vector field of the first image set and the second optical flow vector field of the second image set are determined using the Horn-Schunck algorithm.

[0017] Optionally, the first optical flow vector field includes optical flow vector fields at multiple locations, and the second optical flow vector field includes optical flow vector fields at multiple locations, wherein the multiple locations are multiple locations in any image of the first image set and the second image set, and obtaining the vector numerical difference between the first optical flow vector field and the second optical flow vector field includes:

[0018] Based on the vector values ​​at the same positions in the first optical flow vector field and the second optical flow vector field, the vector value difference at the same positions is obtained to obtain the vector value difference at each of the plurality of positions.

[0019] Optionally, determining whether the driver has engaged in dangerous driving behavior based on the vector numerical difference includes:

[0020] The absolute value of the vector numerical difference is compared with a preset threshold.

[0021] If the difference in the vector values ​​is greater than the preset threshold, it is determined that the driver has engaged in dangerous driving behavior.

[0022] If the difference in the vector values ​​is less than the preset threshold, it is determined that the driver is not engaging in dangerous driving behavior.

[0023] Optionally, determining whether the driver has engaged in dangerous driving behavior based on the vector numerical difference includes:

[0024] The absolute value of the vector numerical difference is compared with a preset threshold.

[0025] If the absolute value of the vector numerical difference is greater than or equal to the preset threshold, the steps of acquiring the first image set and the second image set of the driver to acquiring the vector numerical difference between the first optical flow vector field and the second optical flow vector field are repeated to monitor the vector numerical difference.

[0026] If the absolute value of the vector numerical difference is greater than or equal to the preset threshold, and the duration of the vector numerical difference being greater than or equal to the preset threshold is greater than a set duration, it is determined that the driver has engaged in dangerous driving behavior.

[0027] If the absolute value of the vector numerical difference is greater than or equal to the preset threshold within the set time period, and the direction of motion represented by the vector numerical difference is opposite to the direction of motion represented by the previously acquired vector numerical difference, it is determined that the driver does not engage in dangerous driving behavior.

[0028] Optionally, the method further includes:

[0029] When the driver engages in dangerous driving behavior, the data of the dangerous driving behavior is transmitted to the vehicle alarm system, and / or a voice alarm message is output.

[0030] Optionally, before acquiring the first and second image sets of the driver, the method further includes:

[0031] Get vehicle speed;

[0032] If the vehicle speed is greater than a set speed, it is determined that the vehicle is in motion.

[0033] In response to the driving state, the step of acquiring the first image set and the second image set of the driver is performed.

[0034] According to a second aspect of the present disclosure, a behavior recognition device is provided, comprising:

[0035] The first acquisition module is configured to acquire a first image set and a second image set of the driver;

[0036] The second acquisition module is configured to acquire a first optical flow vector field based on the first image set, and to acquire a second optical flow vector field based on the second image set;

[0037] The third acquisition module is configured to acquire the vector numerical difference between the first optical flow vector field and the second optical flow vector field;

[0038] The determination module is configured to determine whether the driver is engaging in dangerous driving behavior based on the difference in the vector values.

[0039] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0040] A memory on which computer programs are stored;

[0041] A processor is configured to execute the computer program in the memory to implement the steps of the behavior recognition method provided in the first aspect of this disclosure.

[0042] According to a fourth aspect of the present disclosure, a vehicle is provided, the vehicle including: electronic equipment provided in the third aspect of the present disclosure.

[0043] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0044] In the above technical solution, a first image set and a second image set of the driver are acquired; a first optical flow vector field is obtained based on the first image set, and a second optical flow vector field is obtained based on the second image set; the vector numerical difference between the first and second optical flow vector fields is obtained; and the presence of dangerous driving behavior by the driver is determined based on the vector numerical difference. In this solution, by acquiring images of the driver's actions, representing the acquired image set as a vector field, and dividing the movement of different parts of the driver into zones to calculate the vector numerical difference, the analysis of the vector numerical difference can determine whether the driver is engaging in dangerous driving behavior. This provides a method for identifying dangerous driving behavior by analyzing changes in the optical flow vector field based on images, enabling real-time monitoring of dangerous driving behavior and timely prevention of safety hazards.

[0045] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart illustrating a behavior recognition method according to an exemplary embodiment.

[0048] Figure 2 This is a flowchart illustrating another behavior recognition method according to an exemplary embodiment.

[0049] Figure 3 This is a flowchart illustrating another behavior recognition method according to an exemplary embodiment.

[0050] Figure 4 This is a flowchart illustrating yet another behavior recognition method according to an exemplary embodiment.

[0051] Figure 5 This is a flowchart illustrating yet another behavior recognition method according to an exemplary embodiment.

[0052] Figure 6 This is a block diagram illustrating a behavior recognition device according to an exemplary embodiment.

[0053] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0055] It is understood that the terms "first," "second," etc., used in this disclosure are used to describe various types of information, but such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another and do not indicate a particular order or degree of importance.

[0056] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0057] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0058] Figure 1 This is a flowchart illustrating a behavior recognition method according to an exemplary embodiment, such as... Figure 1 As shown, the method includes the following steps:

[0059] In step S11, the first image set and the second image set of the driver are obtained.

[0060] For example, a camera can be installed in the driver's cab of a vehicle to capture images of the driver's driving actions. The images can be captured at fixed time intervals to obtain an image set, wherein each image set includes at least two frames, and the time interval between each frame is controlled within a certain time range to ensure the continuity of the images. Each image set is used to calculate an optical flow vector field once, and the time interval between each collected image set is controlled within a certain time range to ensure the continuity between two adjacent image sets.

[0061] Optionally, in one implementation, step S11 may include:

[0062] The system acquires at least two frames of images of the driver within a first set time period as the first image set, and acquires at least two frames of images of the driver within a second set time period as the second image set; wherein the first set time period and the second set time period are continuous.

[0063] For example, the duration of the first set time and the second set time can be equal. Taking the general frame rate of video shooting as an example, it is usually 24 frames per second. The time interval between image acquisition is 1 / 24 second. The duration of the first set time and the second set time can be 1 / 24 second. The at least two frames can be two or more frames. In this embodiment, two frames are taken as an example. Both the first image set and the second image set can include two consecutive adjacent frames. The first set time and the second set time are continuous. That is, it can be understood that the four frames, namely the two frames in the first image set and the two frames in the second image set, can be four consecutively acquired frames.

[0064] In step S12, a first optical flow vector field is obtained based on the first image set, and a second optical flow vector field is obtained based on the second image set.

[0065] For example, an optical flow vector field can be calculated based on at least two images in each image set. Therefore, the first optical flow vector field can be obtained by analyzing and calculating the first image set, and the second optical flow vector field can be obtained by analyzing and calculating the second image set.

[0066] Optionally, step S12 may include: determining the first optical flow vector field of the first image set and the second image set using the Horn-Schunck algorithm, based on the first image set and the second image set.

[0067] For example, the first image set can contain two frames of images, and the second image set can also contain two frames of images. The acquisition method of the two frames of images in each image set can refer to step S11 above. Using the Horn-Schunck algorithm, the first optical flow vector field can be obtained by calculating the two frames of images in the first image set, and the second optical flow vector field can be obtained by calculating the two frames of images in the second image set.

[0068] It should be noted that optical flow is the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object. The optical flow method utilizes the temporal changes of pixels in an image sequence (which can be understood as an image sequence composed of multiple frames; in the embodiments of this disclosure, an image set can be understood as an image sequence) and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames. The optical flow vector field, also called the optical flow field, is a two-dimensional vector field that reflects the trend of grayscale changes at each point in the image. It can be viewed as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information it contains is the instantaneous velocity vector information of each pixel.

[0069] The Horn-Schunck algorithm is introduced below. For example, two frames from the first image set and two frames from the second image set can be considered as an image sequence. For any image in the sequence, the brightness of any pixel (x, y) at time t can be expressed as: I(x, y, t), where (x, y) represents the coordinates in the image. Pixel motion is then represented as: dx, dy, dt. The same brightness before and after a pixel change is represented as:

[0070] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0071] Taylor expansion:

[0072]

[0073] Where ∈ is the minimum value, combining the above equations, we can obtain:

[0074]

[0075] Divide by dt:

[0076]

[0077] in Let be the pixel velocity, which is also known as optical flow. Then: I x u+ y v+It =0.

[0078] To solve the aforementioned optical flow problem, a new smoothing constraint needs to be introduced. It should be noted that different optical flow methods introduce different constraints. The Horn-Schunck algorithm estimates motion in the image by introducing a global smoothing constraint. Horn and Schunk assume that the velocity of a pixel in the image is similar to or the same as the velocity of its neighboring pixels, and that the velocity change at every point in the optical flow field is smooth and not abrupt. The constraint introduced by the Horn-Schunck algorithm can be expressed as:

[0079]

[0080] for and After approximation, it can be expressed as:

[0081]

[0082] The two sets of constraints obtained above are:

[0083] ξ b =I x u+I y v+I t

[0084]

[0085] Combining the two sets of constraints mentioned above, we can express it as follows:

[0086]

[0087] Minimize ξ 2 Since its corresponding partial derivative is 0, we can obtain:

[0088]

[0089]

[0090] After simplification, we can obtain

[0091]

[0092]

[0093] have to

[0094]

[0095]

[0096] The above formulas can be used to obtain u(x,y) and v(x,y), which in turn yields the desired optical flow vector field E(u,v).

[0097] By calculating the first image set and the second image set using the above method, the first optical flow vector field and the second optical flow vector field can be obtained.

[0098] In step S13, the vector numerical difference between the first optical flow vector field and the second optical flow vector field is obtained.

[0099] For example, since the optical flow vector field can characterize the instantaneous velocity field generated by the movement of grayscale pixels on the image plane, the difference between two consecutively acquired optical flow vector fields can reflect the movement of each pixel in the image, thereby allowing analysis of whether the driver's actions during driving have undergone significant local changes. For instance, if the driver's hands simultaneously leave the steering wheel, or if the driver's body deviates significantly from safe driving actions, the first and second optical flow vector fields of the pixels corresponding to the positions of the hands or the driver's body in the image will show a certain difference. It should be noted that in this embodiment, the first and second optical flow vector fields refer to the first and second optical flow vector fields of a certain pixel in the image. The first and second optical flow vector fields can be calculated for all pixels in the entire acquired image, or they can be calculated for each pixel in a local region of the image.

[0100] Based on the above principle, the difference between the vector values ​​in the second optical flow vector field and the first optical flow vector field can be calculated to obtain the corresponding vector value difference. Optionally, this step S13 includes:

[0101] Based on the vector values ​​at the same position in the first optical flow vector field and the second optical flow vector field, the difference in vector values ​​at the same position is obtained to obtain the difference in vector values ​​at each of the multiple positions.

[0102] For example, for a certain pixel, if the vector value u1 is taken in its first optical flow vector field E1(u1,v1) and the vector value u2 is taken in its second optical flow vector field E2(u2,v2), then the vector value difference between the vector values ​​u2 and u1 can be obtained, which is expressed as U0 = u2 - u1.

[0103] In step S14, the presence of dangerous driving behavior by the driver is determined based on the difference in vector values.

[0104] For example, in one implementation, to reduce the amount of data processed by the image, the camera can capture images of specific areas of the driver, such as the driver's hands, head, and body. Key points can be set for different key locations in the captured images; for example, 1-2 key points can be set for the hands, 1-2 key points for the head, and 1-3 key points for the body. This allows for the calculation of the optical flow vector field of the pixels corresponding to the key points in the image, thereby obtaining the vector value difference of the optical flow vector field of each key point of the driver. If the vector value difference of a certain key point is greater than a preset threshold, it indicates that the movement range of the part to which the key point belongs is too large. For example, if the vector value difference of a certain key point on the hand exceeds the preset threshold, it indicates that the range of hand movement is large, and dangerous driving behavior can be identified.

[0105] Additionally, it should be noted that in this embodiment, taking two frames of images in each set as an example in step S11, the above scheme can be understood as generating optical flow vector fields for pixels of key parts (such as hands, head, and body) in each of the two acquired images. Thus, by acquiring images of the driver in real time, multiple optical flow vector fields for that pixel can be obtained. The first image set and the second image set can be any two adjacent image sets acquired. Therefore, the first optical flow vector field and the second optical flow vector field are any two adjacent optical flow vector fields. By acquiring the optical flow vector fields in real time, real-time detection of the driver's driving behavior can be achieved.

[0106] Optionally, when a driver engages in dangerous driving behavior, the data of such behavior can be transmitted to the vehicle's alarm system to remind the driver to drive properly via in-vehicle voice alarm. Alternatively, the data can be simultaneously uploaded to a relevant monitoring platform, where relevant monitoring personnel can record the data and follow up on the dangerous driving behavior.

[0107] In the above scheme, by acquiring images of the driver's actions, the acquired image set is represented in the form of a vector field, and the movement of different parts of the driver is divided into partitions to calculate the vector numerical difference. By analyzing the vector numerical difference, it is possible to determine whether the driver is engaging in dangerous driving behavior. This provides a method for identifying whether a driver is engaging in dangerous driving behavior based on image analysis of changes in optical flow vector field. It can monitor whether a driver is engaging in dangerous driving behavior in real time and avoid potential safety hazards in a timely manner.

[0108] Figure 2 This is a flowchart illustrating another behavior recognition method according to an exemplary embodiment, such as... Figure 2 As shown, step S11 may include the following steps:

[0109] In step S111, two consecutive frames of the driver's image are acquired as the first image set.

[0110] In step S112, after acquiring the first image set, two consecutive frames of the driver's image are acquired as the second image set, and the two frames in the second image set are consecutive to the two frames in the first image set.

[0111] For example, by using a camera installed in the vehicle's cab to acquire images of the driver's driving actions, two consecutive frames can be used as the first image set. After acquiring the first image set, the camera continues to acquire the next two consecutive frames of the driver, and these two frames are used as the second image set, thus obtaining four consecutive images. The method for acquiring the two consecutive frames in the first and second image sets has been described above and will not be repeated here. Optionally, the four images mentioned above do not have to be consecutive frames, but the time interval between each frame is short, which can still ensure the continuity of the images. For example, one frame can be acquired every 1-2 frames, and two frames can be acquired as one image set. After obtaining the first and second image sets, the first optical flow vector field and the second optical flow vector field mentioned above are calculated for the first and second image sets respectively. The specific method can be referred to step S12 above and will not be repeated here.

[0112] Figure 3 This is a flowchart illustrating another behavior recognition method according to an exemplary embodiment, such as... Figure 3 As shown, step S14 may include the following steps:

[0113] In step S141, the absolute value of the vector numerical difference is compared with a preset threshold.

[0114] In step S142, if the difference in the vector values ​​is greater than the preset threshold, it is determined that the driver has engaged in dangerous driving behavior.

[0115] In step S143, if the difference in the vector values ​​is less than the preset threshold, it is determined that the driver does not engage in dangerous driving behavior.

[0116] For example, the vector difference U0 (the calculation method for U0 can be referred to step S13 above, and will not be repeated) obtained by subtracting the vector values ​​in two continuous optical flow vector fields is compared with a preset threshold u'. When the vector difference is greater than the preset threshold, it is determined that the driver is engaging in dangerous driving behavior. For example, if the preset threshold u' is set to 30cm, when |U0|≥30cm, it can be determined that the driver's driving actions exceed the safe driving action standards, that is, the driver is engaging in dangerous driving behavior. When |U1|<30cm, it can be determined that the driver's driving actions meet the safe driving action standards, that is, the driver is not engaging in dangerous driving behavior.

[0117] Optionally, different thresholds can be set for different body parts, such as different thresholds for vector numerical differences for the head, hands, and body.

[0118] Figure 4 This is a flowchart illustrating yet another behavior recognition method according to an exemplary embodiment, such as... Figure 4 As shown, step S14 may include the following steps:

[0119] In step S144, the absolute value of the vector numerical difference is compared with a preset threshold.

[0120] In step S145, if the absolute value of the vector numerical difference is greater than or equal to the preset threshold, the steps of acquiring the first image set and the second image set of the driver to acquiring the vector numerical difference between the first optical flow vector field and the second optical flow vector field are repeated to monitor the vector numerical difference.

[0121] It is understandable that if the absolute value of the vector numerical difference is detected to be greater than or equal to the preset threshold, the above steps S11 to S13 can be executed again to re-judge the vector numerical difference and avoid misjudgment. The method for re-judging can be referred to step S146 below.

[0122] In step S146, if the absolute value of the vector numerical difference is greater than or equal to the preset threshold, and the duration of the vector numerical difference being greater than or equal to the preset threshold is greater than a set duration, it is determined that the driver has engaged in dangerous driving behavior.

[0123] In step S147, if the absolute value of the vector numerical difference is greater than or equal to the preset threshold within the set time period, and the direction of motion represented by the vector numerical difference is opposite to the direction of motion represented by the previously acquired vector numerical difference, it is determined that the driver does not engage in dangerous driving behavior.

[0124] For example, when the absolute value of the previous vector numerical difference |U0| is greater than or equal to a preset threshold u', steps S11 to S13 are repeated to obtain a new absolute value of the vector numerical difference |U0'|. If |U0'| is greater than or equal to the preset threshold u', and the duration of the difference is greater than or equal to the preset threshold u' is longer than a set duration, it can be determined that the driver's driving behavior is dangerous. However, if |U0'| is greater than or equal to the preset threshold u', and the direction of motion represented by U0' is opposite to the direction of motion represented by U0, it is determined that the driver does not engage in dangerous driving behavior.

[0125] For example, with a preset threshold u' = 30cm and a set duration t = 5s, when |U0| ≥ 30cm, the absolute value of the vector difference |U0'| is obtained again. If |U0'| is still greater than or equal to 30cm, it is further determined whether the duration t for |U0'| to be greater than or equal to 30cm is greater than or equal to 5s. If t ≥ 5s, it can be determined that the driver has engaged in dangerous driving behavior. When |U0| ≥ 30cm, t < 5s, and the direction of motion represented by U0' is opposite to the direction of motion represented by U0 (for example, it can be represented by U0' and U0 having positive and negative signs respectively), it indicates that the driver has corrected the previous behavior, and therefore it can be determined that the driver is in a safe driving state.

[0126] Figure 5 This is a flowchart illustrating yet another behavior recognition method according to an exemplary embodiment, such as... Figure 5 As shown, prior to step S11, the method further includes the following steps:

[0127] In step S15, the vehicle speed is obtained.

[0128] In step S16, if the vehicle speed is greater than the set speed, it is determined that the vehicle is in a driving state.

[0129] In step S17, in response to the driving state, the step of acquiring the first image set and the second image set of the driver is performed.

[0130] For example, before the camera in the vehicle's cab captures images of the driver's driving actions, the vehicle can obtain its speed to determine whether it is in motion. When the vehicle's speed is greater than zero, the vehicle is currently in motion. In response to the vehicle's motion status, the camera in the vehicle's cab begins to capture images of the driver's driving actions and analyzes these images according to steps S11-S14 described above. If the vehicle's speed is 0, the method shown in steps S11-S14 can be omitted.

[0131] The above scheme involves capturing images of the driver's actions, representing the captured image set as a vector field, dividing the movement of different parts of the driver's body into zones, calculating the vector numerical difference, and comparing the magnitude and direction of the vector numerical difference with a preset threshold to further determine whether the driver is engaging in dangerous driving behavior.

[0132] Figure 6 This is a block diagram illustrating a behavior recognition device according to an exemplary embodiment, such as... Figure 6 As shown, the behavior recognition device includes a first acquisition module 601, a second acquisition module 602, a third acquisition module 603, and a determination module 604.

[0133] The first acquisition module 601 is used to acquire a first image set and a second image set of the driver;

[0134] The second acquisition module 602 is used to acquire a first optical flow vector field based on the first image set, and to acquire a second optical flow vector field based on the second image set;

[0135] The third acquisition module 603 is used to acquire the vector numerical difference between the first optical flow vector field and the second optical flow vector field;

[0136] The determination module 604 is used to determine whether the driver has engaged in dangerous driving behavior based on the difference in vector values.

[0137] Optionally, the first acquisition module 601 can be used for:

[0138] The system acquires at least two frames of images of the driver within a first set time period as the first image set, and acquires at least two frames of images of the driver within a second set time period as the second image set; wherein the first set time period and the second set time period are continuous.

[0139] Optionally, the first acquisition module 601 can be used for:

[0140] Two consecutive frames of images of the driver are acquired as the first image set;

[0141] After acquiring the first image set, two consecutive frames of the driver's image are acquired as the second image set, and the two frames in the second image set are consecutive to the two frames in the first image set.

[0142] Optionally, the second acquisition module 602 can be used for:

[0143] Based on the first image set and the second image set, the first optical flow vector field of the first image set and the second optical flow vector field of the second image set are determined using the Horn-Schunck algorithm.

[0144] Optionally, the third acquisition module 603 can be used for:

[0145] Based on the vector values ​​at the same position in the first optical flow vector field and the second optical flow vector field, the difference in vector values ​​at the same position is obtained to obtain the difference in vector values ​​at each of the multiple positions.

[0146] Alternatively, in one implementation, the determining module 604 can be used to:

[0147] The absolute value of the difference between the vector values ​​is compared with a preset threshold.

[0148] If the difference in the vector values ​​is greater than the preset threshold, it is determined that the driver has engaged in dangerous driving behavior.

[0149] If the difference in the vector values ​​is less than the preset threshold, it is determined that the driver is not engaging in dangerous driving behavior.

[0150] Alternatively, in another implementation, the determining module 604 can be used to:

[0151] The absolute value of the difference between the vector values ​​is compared with a preset threshold.

[0152] If the absolute value of the vector numerical difference is greater than or equal to the preset threshold, the steps of acquiring the first image set and the second image set of the driver to acquiring the vector numerical difference between the first optical flow vector field and the second optical flow vector field are repeated to monitor the vector numerical difference.

[0153] If the absolute value of the vector numerical difference is greater than or equal to the preset threshold, and the duration of the vector numerical difference being greater than or equal to the preset threshold is greater than the set duration, it is determined that the driver has engaged in dangerous driving behavior.

[0154] If the absolute value of the vector numerical difference is greater than or equal to the preset threshold within the set time period, and the direction of motion represented by the vector numerical difference is opposite to the direction of motion represented by the previously acquired vector numerical difference, it is determined that the driver does not engage in dangerous driving behavior.

[0155] Optionally, the behavior recognition device may further include:

[0156] An alarm output device is used to transmit the data of dangerous driving behavior to the vehicle alarm system and / or output voice alarm information when the driver engages in dangerous driving behavior.

[0157] Optionally, the behavior recognition device may further include:

[0158] The fourth acquisition module is used to acquire vehicle speed;

[0159] The status recognition module is used to determine that the vehicle is in motion when the vehicle speed is greater than a set speed.

[0160] An execution module is used to perform the step of acquiring the first image set and the second image set of the driver in response to the driving state.

[0161] In the above scheme, by acquiring images of the driver's actions, the acquired image set is represented in the form of a vector field, and the movement of different parts of the driver is divided into partitions to calculate the vector numerical difference. By analyzing the vector numerical difference, it is possible to determine whether the driver is engaging in dangerous driving behavior. This provides a method for identifying whether a driver is engaging in dangerous driving behavior based on image analysis of changes in optical flow vector field. It can monitor whether a driver is engaging in dangerous driving behavior in real time and avoid potential safety hazards in a timely manner.

[0162] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0163] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. For example... Figure 7 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0164] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the aforementioned behavior recognition method. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0165] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the behavior recognition method described above.

[0166] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the behavior recognition method described above. For example, the computer-readable storage medium may be the memory 702 including program instructions, which may be executed by the processor 701 of the electronic device 700 to complete the behavior recognition method described above.

[0167] This disclosure also provides a vehicle, the vehicle including... Figure 6 The behavior recognition device 600 shown, or the vehicle including Figure 7 The electronic device 700 shown, and the specific operation of the behavior recognition device 600 or electronic device 700 in the vehicle have been described in detail in the embodiments of the behavior recognition method described above, and will not be elaborated here.

[0168] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described behavior recognition method when executed by the programmable device.

[0169] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0170] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0171] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A behavior recognition method, characterized in that, include: Obtain the first and second image sets of the driver; A first optical flow vector field is obtained based on the first image set, and a second optical flow vector field is obtained based on the second image set; Obtain the vector numerical difference between the first optical flow vector field and the second optical flow vector field; The driver's dangerous driving behavior is determined based on the difference in the vector values. The acquisition of the first image set and the second image set of the driver includes: The driver is acquired at least two frames within a first set time period as the first image set, and at least two frames within a second set time period as the second image set; wherein the first set time and the second set time period are consecutive.

2. The method according to claim 1, characterized in that, Acquiring at least two frames of images of the driver within a first predetermined time period as the first image set, and acquiring at least two frames of images of the driver within a second predetermined time period as the second image set, includes: Two consecutive frames of images of the driver are acquired as the first image set; After obtaining the first image set, two consecutive frames of the driver's image are obtained as the second image set, wherein the two frames in the second image set are consecutive to the two frames in the first image set.

3. The method according to claim 1, characterized in that, The first optical flow vector field includes optical flow vector fields at multiple locations, and the second optical flow vector field includes optical flow vector fields at multiple locations, wherein the multiple locations are multiple locations in any image of the first image set and the second image set. The step of obtaining the vector numerical difference between the first optical flow vector field and the second optical flow vector field includes: Based on the vector values ​​at the same positions in the first optical flow vector field and the second optical flow vector field, the vector value difference at the same positions is obtained to obtain the vector value difference at each of the plurality of positions.

4. The method according to claim 1, characterized in that, Determining whether the driver has engaged in dangerous driving behavior based on the vector numerical difference includes: The absolute value of the vector numerical difference is compared with a preset threshold. If the difference in the vector values ​​is greater than the preset threshold, it is determined that the driver has engaged in dangerous driving behavior. If the difference in the vector values ​​is less than the preset threshold, it is determined that the driver is not engaging in dangerous driving behavior.

5. The method according to claim 1, characterized in that, Determining whether the driver has engaged in dangerous driving behavior based on the vector numerical difference includes: The absolute value of the vector numerical difference is compared with a preset threshold. If the absolute value of the vector numerical difference is greater than or equal to the preset threshold, the steps of acquiring the first image set and the second image set of the driver to acquiring the vector numerical difference between the first optical flow vector field and the second optical flow vector field are repeated to monitor the vector numerical difference. If the absolute value of the vector numerical difference is greater than or equal to the preset threshold, and the duration of the vector numerical difference being greater than or equal to the preset threshold is greater than a set duration, it is determined that the driver has engaged in dangerous driving behavior. If the absolute value of the vector numerical difference is greater than or equal to the preset threshold within the set time period, and the direction of motion represented by the vector numerical difference is opposite to the direction of motion represented by the previously acquired vector numerical difference, it is determined that the driver does not engage in dangerous driving behavior.

6. The method according to claim 1, characterized in that, Before acquiring the first and second image sets of the driver, the method further includes: Get vehicle speed; If the vehicle speed is greater than a set speed, it is determined that the vehicle is in motion. In response to the driving state, the step of acquiring the first image set and the second image set of the driver is performed.

7. A behavior recognition device, characterized in that, include: The first acquisition module is configured to acquire a first image set and a second image set of the driver; The second acquisition module is configured to acquire a first optical flow vector field based on the first image set, and to acquire a second optical flow vector field based on the second image set; The third acquisition module is configured to acquire the vector numerical difference between the first optical flow vector field and the second optical flow vector field; The determination module is configured to determine whether the driver is engaging in dangerous driving behavior based on the vector numerical difference. The acquisition of the first image set and the second image set of the driver includes: The driver is acquired at least two frames within a first set time period as the first image set, and at least two frames within a second set time period as the second image set; wherein the first set time and the second set time period are consecutive.

8. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-6.

9. A vehicle, characterized in that, The vehicle includes the electronic device as described in claim 8.

Images