Method, device, electronic device and storage medium for determining object state

Abstract

The present disclosure provides a method for determining the state of an object, which relates to the field of artificial intelligence, and in particular to the field of computer vision and intelligent transportation technology. The specific implementation scheme is: determining an image set from a video frame sequence for a target object, the image set being provided with a time feature based on the production time of the video frame sequence; determining the behavior feature for the image set according to the behavior information of the target object in each image in the image set; and determining the driving state of the target object according to the behavior feature and the time feature of the image set. The present disclosure also provides a device, an electronic device, and a storage medium for determining the state of an object.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to computer vision and intelligent transportation technology. More specifically, the present disclosure provides a method, device, electronic device and storage medium for determining the state of an object. Background Art

[0002] Fatigue driving is one of the main causes of traffic accidents, causing huge losses to people's lives and property. Timely and accurate warning of fatigue driving behavior is of great significance to maintaining road traffic safety. Summary of the invention

[0003] The present disclosure provides a method, apparatus, device and storage medium for determining a state of an object.

[0004] According to a first aspect, a method for determining an object state is provided, the method comprising: determining an image set from a video frame sequence for a target object, the image set being provided with a time feature based on a production time of the video frame sequence; determining a behavior feature for the image set based on behavior information of the target object in each image in the image set; and determining a driving state of the target object based on the behavior feature and the time feature of the image set.

[0005] According to a second aspect, a device for determining a state of an object is provided, the device comprising: a time feature determination module for determining an image set from a video frame sequence for a target object, the image set being provided with a time feature based on a production time of the video frame sequence; a behavior feature determination module for determining a behavior feature for the image set based on behavior information of the target object in each image in the image set; and a driving state determination module for determining the driving state of the target object based on the behavior feature and the time feature of the image set.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and apparatus for determining a state of an object can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of a method for determining a state of an object according to an embodiment of the present disclosure;

[0013] Figure 3 is a flow chart of a method for determining behavioral features for an image set according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a method for determining eye closing behavior characteristics of an image set according to an embodiment of the present disclosure;

[0015] Figure 5A is a statistical graph of a first sliding window for yawning behavior according to an embodiment of the present disclosure;

[0016] Figure 5B is a statistical graph of a second sliding window for head-down behavior according to an embodiment of the present disclosure;

[0017] Figure 5C is a statistical graph of a third sliding window for behaviors of not looking ahead according to an embodiment of the present disclosure;

[0018] Figure 6 is a time distribution diagram of driver fatigue according to an embodiment of the present disclosure;

[0019] Figure 7 is a training principle diagram of a machine learning model according to an embodiment of the present disclosure;

[0020] Figure 8 is an overall flow chart of a method for determining an object state according to an embodiment of the present disclosure;

[0021] Fig. 9 is a block diagram of an apparatus for determining a state of an object according to an embodiment of the present disclosure;

[0022] Fig.10 is a block diagram of an electronic device for a method of determining a state of an object according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Currently, methods for determining whether a target object (eg, a driver) is in a state of fatigue driving include three categories based on the way information is obtained.

[0025] The first type is to monitor fatigue driving behavior based on vehicle driving signals, and analyze fatigue driving status by obtaining various sensors on the vehicle, such as vehicle driving trajectory, steering wheel steering mode, braking behavior analysis, etc. This type of method can obtain driving data more conveniently and does not affect the driver's driving, but its analysis is easily affected by the driver's driving habits, driving skills, and road conditions, and has a low accuracy rate.

[0026] The second type is to monitor fatigue driving behavior based on the driver's physiological signals. Common physiological signals include electroencephalogram, electrocardiogram and electrooculogram. This type of method can accurately predict whether the driver is in a fatigue state through physiological signals, but its signal acquisition depends on various signal measurement tools worn by the driver. The tool cost is high, and wearing these tools will greatly reduce driving comfort and easily affect driving behavior.

[0027] The third category is to extract and analyze the driver's facial and head features based on computer vision technology to monitor fatigue driving behavior. For example, based on eye movement data, the driver's eye features are extracted, and the fatigue state is judged by judging whether the eyes are closed. For example, through methods such as Adaboost (Adaptive Boosting) algorithm, Multi-Task Convolutional Neural Network (MTCNN), Multi-mode Fusion Recurrent Neural Network (MFRNN), etc., after obtaining the features of key parts of the face, the eye features and mouth features are fused, and combined with PERCLOS (Percentage of Eyelid Closure over the Pupil over Time, a physical quantity that measures fatigue / drowsiness, which refers to the percentage of time in which the degree of eye closure exceeds a certain ratio (such as 70% or 80%) per unit time to the total time) method to comprehensively judge fatigue behavior. Compared with monitoring methods based on vehicle signals and driver physiological signals, this type of method is more robust, low-cost, and non-contact and more driver-friendly. However, this type of method focuses on how to extract facial information and often uses a single facial feature to analyze fatigue driving behavior, which has certain shortcomings in universality and accuracy.

[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] Figure 1 is a schematic diagram of an exemplary system architecture to which the method and apparatus for determining the state of an object can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] The terminal device 101 can interact with the server 103 through the network 102 to receive or send messages, etc. The terminal device 101 can be any electronic device with an image acquisition function, including but not limited to a vehicle-mounted camera, etc.

[0033] The method for determining the state of an object provided in the embodiment of the present disclosure can generally be executed by the server 103. Accordingly, the apparatus for determining the state of an object provided in the embodiment of the present disclosure can generally be arranged in the server 103. The method for determining the state of an object provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103. Accordingly, the apparatus for determining the state of an object provided in the embodiment of the present disclosure can also be arranged in a server or a server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103.

[0034] Figure 2 is a flowchart of a method for determining a state of an object according to an embodiment of the present disclosure.

[0035] like Figure 2 As shown, the method 200 for determining the state of an object may include operations S210 to S230.

[0036] In operation S210, an image set is determined from a video frame sequence for a target object.

[0037] For example, the target object may be the driver of the vehicle. During the driving process of the vehicle, a video of the driver may be taken using an image acquisition device such as a camera in the vehicle, and the video may be transmitted to a cloud server. The execution subject of the embodiment of the present disclosure may be a cloud server.

[0038] The cloud server processes the video into a video frame sequence according to a preset frame rate (eg, 20 frames / s). The number of video frames transmitted per second (Frames Per Second, FPS) in the video frame sequence is, for example, 20. Each video frame in the video frame sequence can be used as an image.

[0039] It is also possible to obtain multiple video frames from the video frame sequence according to a preset sampling rate to form an image set. Processing the image set can improve processing efficiency compared to processing the entire video.

[0040] For example, multiple video frames can be extracted from a video frame sequence at a sampling rate of 10 frames / s, and the video frames extracted every 10 seconds are taken as an image set. The video frames extracted for the next 10 seconds are also taken as an image set. Thus, multiple image sets can be obtained.

[0041] The production time of the video frame sequence may be, for example, the production time of the video for the driver, or the time period during which the image acquisition device acquires video of the target object, such as “XX hours XX minutes XX seconds to YY hours YY minutes XX seconds”.

[0042] The time when the driver drives the vehicle is also one of the factors that affect driving fatigue. For example, the driver may be most likely to become fatigued between 0:00 and 3:00 in the morning, followed by 4:00 to 7:00 in the morning, and then 20:00 to 23:00 in the evening, and so on.

[0043] Therefore, a time feature may be set for an image set based on the production time of the video frame sequence. For example, for the first image set, the second image set, and the third image set obtained, the production time of the first image set is from 0:00 to 3:00 in the morning, the production time of the second image set is from 4:00 to 7:00 in the morning, and the production time of the third image set is from 20:00 to 23:00 in the evening. A first time feature (e.g., 1.0000) may be set for the first image set, a second time feature (e.g., 0.8333) may be set for the second image set, and a third time feature (e.g., 0.6667) may be set for the third image set.

[0044] Since the influence of the driver's driving time on fatigue driving is taken into consideration, a time feature is set for the image set, thereby improving the accuracy of identifying the target object in a fatigue driving state.

[0045] In operation S220 , a behavior feature for the image set is determined according to the behavior information of the target object in each image in the image set.

[0046] For example, when a driver is fatigued, he or she is usually accompanied by behaviors such as closing eyes, yawning, lowering head, not looking forward, etc. Therefore, the behavior information of the target object may include closing eyes, yawning, lowering head, and not looking forward. Behavioral features can be determined based on these behavior information as an influencing factor for determining whether the target object is in a fatigue driving state.

[0047] For example, the target object in the image is scored for the degree of eye closure, the degree of yawning, the degree of head lowering, and the degree of not looking forward, and the eye closure behavior characteristics, yawning behavior characteristics, head lowering behavior characteristics, and not looking forward behavior characteristics are determined based on the eye closure score, yawning score, head lowering score, and not looking forward score.

[0048] In operation S230 , the driving state of the target object is determined according to the behavior characteristics and the time characteristics of the image set.

[0049] For example, the time features, eye closing behavior features, yawning behavior features, head lowering behavior features, and not looking forward behavior features for an image collection may be fused together, and whether the target object is in a fatigue driving state may be determined based on the fused features.

[0050] For example, the fused feature vector is input into a machine learning model (such as GBDT (Gradient Boosting Decision Tree)), and the driving state of the target object can be output through the decision and classification of the machine learning model. For example, output 1 indicates that the target object is in a fatigue driving state, and output -1 indicates that the target object is not in a fatigue driving state.

[0051] The embodiment of the present disclosure integrates the behavioral characteristics of the target object and the temporal characteristics of the video of the target object to determine whether the target object is in a fatigue driving state, which can improve the accuracy of fatigue driving recognition.

[0052] According to an embodiment of the present disclosure, operation S210 may include: determining a sampling interval according to the number of video frames transmitted per second in the video frame sequence and a preset sampling rate; and acquiring multiple video frames from the video frame sequence as an image set according to the sampling interval.

[0053] For example, the sampling interval can be obtained according to the following formula 1.

[0054]

[0055] Wherein, fps represents the number of video frames transmitted per second in the video frame sequence (eg, 20 frames / s), eps represents a preset sampling rate (eg, 10 frames / s), and gap represents a sampling interval.

[0056] For example, the video frame sequence can be recorded as FL = {f1, f2, f3, ..., f n}, where n is an integer greater than 3, f1, f2, f3, ..., f n is the video frame (image) arranged in the production time sequence in the video frame sequence FL. After determining the sampling interval gap, an indication set SL = {s1, s2, s3, ..., s n} is used to indicate whether the video frame in FL needs to be sampled. The corresponding relationship between FL and SL can be expressed by the following formula 2.

[0057]

[0058] Where i represents the number of video frames in the video frame sequence. i modulo gap = 0 means that i is divisible by gap, and i modulo gap ≠ 0 means that i is not divisible by gap. iThe video frames with a value of 1 can form an image set, which can be recorded as IL = {i1, i2, i3, ..., i m}, where m is an integer greater than 3 and not greater than n, i1, i2, i3, …, i m The images in the image set IL are arranged in order of production time.

[0059] Figure 3 The present invention is a flowchart of a method for determining a behavior feature for an image set according to an embodiment of the present disclosure.

[0060] like Figure 3 As shown, the method may include operations S321 to S324.

[0061] In operation S321 , for each image in the image set, a behavior evaluation value of a target object in the image is determined as the behavior evaluation value of the image.

[0062] For example, for the image set IL = {i1, i2, i3, ..., i m}, determine the behavior evaluation value of the target object in the image as the behavior evaluation value of the image.

[0063] For example, the behavior information of the target object in the image includes closing eyes, yawning, lowering the head, and not looking forward. The behavior evaluation value of the target object in the image may include the eye closure degree score, the yawning degree score, the head lowering degree score, and the not looking forward degree score. Accordingly, the behavior evaluation value of the image may include the eye closure degree score, the yawning degree score, the head lowering degree score, and the not looking forward degree score.

[0064] In operation S322, a plurality of image subsets are sequentially determined from the image set using a sliding window.

[0065] For example, for eye closing behavior, yawning behavior, head lowering behavior and not looking forward behavior, the length of the sliding window corresponding to each may be determined respectively. The length of the sliding window may refer to the number of images that may be included in the sliding window, for example.

[0066] For example, if the length of the sliding window for the eye-closing behavior is 20, the sliding window for the eye-closing behavior may include 20 images, and the image subset for the eye-closing behavior determined using the sliding window also includes 20 images.

[0067] In operation S323, for each image subset, a behavior evaluation value of the image subset is determined according to the behavior evaluation values ​​of the respective images in the image subset.

[0068] For example, for each image subset in the multiple image subsets of eye-closing behavior, an average of the eye-closing degree scores of the images in the image subset may be calculated as the eye-closing degree score of the image subset.

[0069] Similarly, a yawning degree score for each image subset in multiple image subsets for yawning behavior, a head-down degree score for each image subset in multiple image subsets for head-down behavior, and a not-looking-forward degree score for each image subset in multiple image subsets for not-looking-forward behavior can be calculated.

[0070] In operation S324, a behavior feature of the image set is determined according to the behavior evaluation values ​​of each of the plurality of image subsets.

[0071] For example, in a plurality of image subsets for eye-closing behavior, the maximum value of the eye-closing degree score is 0.8, and 0.8 may be used as the eye-closing behavior feature of the image set IL.

[0072] Similarly, the yawning behavior characteristics, the head-down behavior characteristics, and the behavior characteristics of not looking forward of the image set IL may also be determined.

[0073] According to an embodiment of the present disclosure, before executing operation S322, at least one of the following operations may also be performed: determining the length of the sliding window for closing eyes behavior; determining the length of the sliding window for yawning behavior; determining the length of the sliding window for head lowering behavior; determining the length of the sliding window for not looking forward behavior.

[0074] For example, for eye-closing behavior, the length of the sliding window for the eye-closing behavior may be determined specifically according to the duration corresponding to the image set.

[0075] For example, according to the PERCLOS fatigue metric proposed by the Carnegie Mellon Institute, when the PERCLOS calculation is greater than 0.2, the fatigue correlation is the highest. Therefore, the length of the sliding window for eye closing behavior can be determined according to the following formula 3.

[0076] window=VideoTime*0.2 (Formula 3)

[0077] Among them, window represents the length of the sliding window for eye closing behavior, and VideoTime represents the duration corresponding to the image set.

[0078] For example, the images in the image set are arranged in the order of production time, and the duration corresponding to the image set can be determined based on the time difference between the last image and the first image in the image set. Or in the sampling step, the video frames extracted every 10 seconds are taken as an image set, and the duration corresponding to the image set is 10 seconds.

[0079] For example, the duration corresponding to the image set is 10s, then window=10*0.2=2s. According to the sampling rate eps=10 frames / s, the window contains 20 images, so the length of the sliding window for eye closing behavior can be 20.

[0080] Figure 4 is a schematic diagram of a method for determining eye closing behavior characteristics of an image set according to an embodiment of the present disclosure.

[0081] like Figure 4 As shown, the image set 400 is, for example, a set of video frames extracted within 10 seconds, and the video frame numbers are 1, 2, ..., 30, ... The length of the sliding window for eye closing behavior can be, for example, 20. Using the sliding window, multiple image subsets can be determined from the image set 400 in sequence, for example, including a first image subset 401, a second image subset 402, a third image subset 403, ..., wherein the first image subset 401 includes the 1st to 20th images, the second image subset 402 includes the 2nd to 21st images, and the third image subset 403 includes the 3rd to 22nd images.

[0082] For example, the eye closure degree score of each image in the first image subset 401 may be calculated. i The average value average1 of (i=1-20) is used to calculate the eye closure degree score of each image in the second image subset 402. i The average value average2 of (i=2-21), the eye closure degree score score of each image in the third image subset 403 i The average values ​​average3, . . . , and so on of (i=3 to 22) are determined, and the maximum value is determined from average1, average2, average3, . . . as the eye closing behavior feature of the image set 400 .

[0083] The length of the sliding window for yawning behavior, the length of the sliding window for lowering the head behavior, and the length of the sliding window for not looking forward behavior can be determined by statistically analyzing multiple image sets. Figures 5A-5C Provide explanation.

[0084] Figure 5Ais a statistical graph of a first sliding window for yawning behavior according to an embodiment of the present disclosure.

[0085] like Figure 5A As shown, the statistical graph 501 of the first sliding window includes multiple first sliding windows of different lengths. For example, the lengths of the multiple first sliding windows can be each integer from 10 to 29, and correspondingly, the durations of the multiple first sliding windows are duration = 1.0, duration = 1.1, ..., duration = 2.9.

[0086] For each first sliding window, multiple image subsets are determined in sequence from the image set to be counted using the first sliding window, and for each image subset, the yawning degree score of each image in the image subset is determined, and the average yawning degree of each image in the image subset is calculated as the yawning degree score of the image subset. The yawning degree score of each image subset is used as the horizontal coordinate (for example, the yawning degree score can be equal to 0.2, 0.4, 0.6, etc.), and the accuracy, precision, and recall corresponding to the yawning degree score of each image subset are calculated, and the accuracy, precision, and recall are used as the vertical coordinates to draw a statistical graph of the first sliding window for yawning behavior.

[0087] like Figure 5A As shown, the statistical graphs of the first sliding windows with duration = 1.0, duration = 1.1, ..., duration = 2.9 are obtained respectively. In the statistical graphs of the above multiple first sliding windows, when the horizontal axis (yawning degree score of the image subset) is the same, the accuracy, precision and recall rate of the statistical graph of the first sliding window with duration = 2.0 have stabilized and reached a relatively high level, which can be understood as the largest area under the curve. Therefore, the first sliding window with duration = 2.0 (i.e., a length of 20) can be used as a sliding window for yawning behavior.

[0088] After determining that the length of the sliding window for yawning behavior is 20, for the target image set, a sliding window with a length of 20 is used to determine multiple image subsets from the target image set in turn, determine the yawning degree score of each image subset, and take the maximum value of the yawning score as the yawning behavior feature of the target image set.

[0089] Figure 5B It is a statistical chart of the second sliding window for the head-down behavior according to an embodiment of the present disclosure.

[0090] like Figure 5B As shown, the second sliding window statistics graph 502 includes multiple second sliding windows of different lengths. For example, the lengths of the multiple second sliding windows may be integers from 10 to 29, and correspondingly, the durations of the multiple second sliding windows are duration = 1.0, duration = 1.1, ..., duration = 2.9.

[0091] For each second sliding window, use the second sliding window to determine multiple image subsets in sequence from the image set to be counted, for each image subset, determine the head-down degree score of each image in the image subset, and calculate the average head-down degree of each image in the image subset as the head-down degree score of the image subset. Take the head-down degree score of each image subset as the horizontal coordinate (for example, the head-down degree score can be equal to 0.2, 0.4, 0.6, etc.), and calculate the accuracy, precision and recall rate corresponding to the head-down degree score of each image subset, and use the accuracy, precision and recall rate as the vertical coordinate to draw a statistical graph of the second sliding window for the head-down behavior.

[0092] like Figure 5B As shown, the statistical graphs of the second sliding windows with duration = 1.0, duration = 1.1, ..., duration = 2.9 are obtained respectively. In the statistical graphs of the above multiple second sliding windows, when the horizontal coordinates (the head-down degree scores of the image subsets) are the same, the accuracy, precision and recall rate of the statistical graph of the second sliding window with duration = 1.5 have all stabilized and reached a relatively high level, which can be understood as the largest area under the curve. Therefore, the second sliding window with duration = 1.5 (i.e., a length of 15) can be used as a sliding window for head-down behavior.

[0093] After determining that the length of the sliding window for the head-down behavior is 15, for the target image set, use a sliding window with a length of 15 to determine multiple image subsets from the target image set in turn, determine the head-down degree score of each image subset, and take the maximum value of the head-down degree score as the head-down behavior feature of the target image set.

[0094] Figure 5C 4 is a statistical chart of the third sliding window for the behavior of not looking forward according to an embodiment of the present disclosure.

[0095] like Figure 5CAs shown, the statistical graph 503 of the third sliding window includes multiple third sliding windows of different lengths. For example, the lengths of the multiple third sliding windows can be each integer from 10 to 29, and correspondingly, the durations of the multiple third sliding windows are duration = 1.0, duration = 1.1, ..., duration = 2.9.

[0096] For each third sliding window, the third sliding window is used to sequentially determine multiple image subsets from the image set to be counted, and for each image subset, the non-looking forward degree score of each image in the image subset is determined, and the average non-looking forward degree of each image in the image subset is calculated as the non-looking forward degree score of the image subset. The non-looking forward degree score of each image subset is used as the horizontal coordinate (for example, the non-looking forward degree score can be equal to 0.2, 0.4, 0.6, etc.), and the accuracy, precision and recall corresponding to the non-looking forward degree score of each image subset are calculated, and the accuracy, precision and recall are used as the vertical coordinates to draw a statistical graph of the third sliding window for the non-looking forward behavior.

[0097] like Figure 5C As shown, statistical graphs of the third sliding windows with duration = 1.0, duration = 1.1, ..., duration = 2.9 are obtained respectively. Among the statistical graphs of the above multiple third sliding windows, when the horizontal axis (the degree of not looking forward of the image subset) is the same, the accuracy, precision and recall of the statistical graph of the third sliding window with duration = 1.6 have stabilized and reached a relatively high level, which can be understood as the area under the curve is the largest. Therefore, the third sliding window with duration = 1.6 (i.e., a length of 16) can be used as a sliding window for the behavior of not looking forward.

[0098] After determining that the length of the sliding window for the not looking forward behavior is 16, for the target image set, use a sliding window with a length of 16 to determine multiple image subsets from the target image set in turn, determine the not looking forward degree score of each image subset, and take the maximum value of the not looking forward degree score as the not looking forward behavior feature of the target image set.

[0099] The disclosed embodiment determines the length of the sliding window through a statistical graph. Since the accuracy, precision and recall are measured in the statistical process, the accuracy, precision and recall of the recognition of fatigue driving status can be improved.

[0100] According to an embodiment of the present disclosure, the time the driver drives the vehicle is also one of the factors that affect fatigue driving. The time feature can be set for the image set for the driver according to the time the driver drives the vehicle. Figure 6 Provide explanation.

[0101] Figure 6 It is a time distribution diagram of driver fatigue according to an embodiment of the present disclosure.

[0102] like Figure 6 As shown, the first time period 601 when the driver is most likely to get tired can be from 0:00 to 3:00 in the morning, accounting for about 22%; the second time period 602 when the driver is most likely to get tired can be from 4:00 to 7:00 in the morning, accounting for about 20%; the third time period 603 when the driver is most likely to get tired can be from 20:00 to 23:00 in the evening, accounting for about 18%; the fourth time period 604 when the driver is most likely to get tired can be from 8:00 to 11:00 in the morning, accounting for about 17%; the fifth time period 605 when the driver is most likely to get tired can be from 16:00 to 19:00 in the afternoon, accounting for about 14%; the sixth time period 606 when the driver is most likely to get tired (the time period when the driver is least likely to get tired) can be from 12:00 to 15:00 in the afternoon, accounting for about 9%. The above data can come from the production data of fatigue driving of operating vehicles (such as buses, coaches, etc.), and the acquisition of the data has been authorized and agreed.

[0103] According to the above-mentioned ease of fatigue, for each image in the image set, the image whose production time belongs to the first time period can be assigned a time feature value of 1.0000; the image whose production time belongs to the second time period can be assigned a time feature value of 0.8333; the image whose production time belongs to the third time period can be assigned a time feature value of 0.6667; the image whose production time belongs to the fourth time period can be assigned a time feature value of 0.5000; the image whose production time belongs to the fifth time period can be assigned a time feature value of 0.3333; and the image whose production time belongs to the sixth time period can be assigned a time feature value of 0.1667.

[0104] For an image set, the maximum value of the time feature in the image set can be taken as the time feature of the image set.

[0105] It should be noted that the above method of assigning time feature values ​​to image sets based on the production data of fatigue driving of commercial vehicles and the size of the feature values ​​are only examples, and the present disclosure does not impose any restrictions on this. For example, in practical applications, time feature values ​​can also be assigned to image sets based on actual fatigue driving production data of non-commercial vehicles (such as cars, etc.).

[0106] According to an embodiment of the present disclosure, the behavior characteristics and time characteristics may be input into a machine learning model to obtain the driving state of the target object.

[0107] For example, for the image set IL = {i1, i2, i3, ..., i m}, the eye closure degree score set of the image in IL can be recorded as IC = {ic1, ic2, ic3, ..., ic m}, the yawning degree score set can be recorded as Yawn = {yawn1, yawn2, yawn3, ..., yawn m}, the head-lowering degree score set can be recorded as HeadLow = {hl1, hl2, hl3, ..., hl m}, the score set of not looking forward can be recorded as NotFacingFront = {nff1, nff2, nff3, ..., nff m}, the time feature value set can be recorded as Time = {t1, t2, t3, ..., t m}.

[0108] After determining the eye closing behavior features, yawning behavior features, head lowering behavior features, not looking forward behavior features and time features of the image set according to the above set, the determined behavior features and time features are combined together to construct a fatigue driving feature vector for the user (driver). For example, each image set corresponds to a user, and for the i-th user, the fatigue driving feature vector of the user can be recorded as x i =(t i , ic i ,yawn i ,hl i , nff i ).

[0109] Among them, x i represents the i-th user, t i represents the time characteristics of the i-th user, ic i represents the eye-closing behavior characteristics of the i-th user, yawn i represents the yawning behavior characteristics of the i-th user, hl i represents the head-down behavior characteristics of the i-th user, nff i Represents the behavior characteristics of the i-th user who does not look forward.

[0110] For example, the machine learning model may be a GBDT model, and the samples used to train the model may include fatigue driving feature vectors of M image sets, which can be recorded as For example, M is an integer greater than 100, where X represents a sample set, each image set corresponds to a user, and x i represents the fatigue driving feature vector of the i-th user, y i represents the label of the i-th user, y i The value of is 1 or -1. For example,i =1 indicates that the ith user is in a fatigue driving state, and yi=-1 indicates that the ith user is not in a fatigue driving state.

[0111] Figure 7 It is a training principle diagram of a machine learning model according to an embodiment of the present disclosure.

[0112] like Figure 7 The machine learning model shown is, for example, a GBDT model. The model may include multiple weak classifiers, for example, including weak classifiers 710, 720, and 730. Samples are input to each weak classifier for prediction, and respective predicted values ​​are output, and residuals representing the difference between the predicted values ​​and the sample labels are generated. The residuals generated by each weak classifier are input to the next weak classifier, and each weak classifier can be regarded as a new prediction model formed based on the residual of the previous weak classifier. Multiple weak classifiers are iterated serially to form a strong classifier, and the predicted results generated by all weak classifiers are finally accumulated to obtain final predicted results.

[0113] For example, the sample is input to the weak classifier 710, the weak classifier 710 classifies the sample into fatigue driving states, outputs a first prediction result 711, and generates a first residual 712, which represents the difference between the first prediction result 711 and the label of the sample.

[0114] The first residual 712 is input to the weak classifier 720 , the weak classifier 720 is updated based on the first residual 712 , the sample is input to the weak classifier 720 , and the weak classifier 720 outputs a second prediction result 721 and a second residual 722 .

[0115] The second residual 722 is input to the weak classifier 730 , the weak classifier 730 is updated based on the second residual 722 , the sample is input to the weak classifier 730 , and the weak classifier 730 outputs a third prediction result 731 .

[0116] The first prediction result 711, the second prediction result 721 and the third prediction result 731 are accumulated to obtain the final prediction result of the sample.

[0117] In the above training process, the loss function can be expressed by the following formula 4.

[0118] L = log (1 + exp (-2yF (x))) (Formula 4)

[0119] Among them, L represents the loss function, F(x) represents the prediction function of the weak classifier, x represents the input sample, and y represents the label of the input sample.

[0120] For a GBDT model including K trees (K is, for example, an integer greater than 2, and each tree represents a weak classifier), the process of predicting the fatigue driving state of an input sample can be expressed by the following pseudo code.

[0121]

[0122] The disclosed embodiment uses a machine learning model to identify fatigue driving status, which greatly improves the recognition efficiency compared to manual review.

[0123] Figure 8 It is an overall flow chart of a method for determining the state of an object according to an embodiment of the present disclosure.

[0124] like Figure 8 As shown, video 801 may be a video taken by an image acquisition device in a vehicle for a driver during the driving process of the vehicle. The image acquisition device may send video 801 to a cloud server, and the cloud server extracts frames of video 801 at a preset frame rate (e.g., 20 frames / s) to obtain a video frame sequence. For the video frame sequence, multiple video frames are extracted from the video frame sequence at a preset sampling rate (e.g., 10 frames / s) to form an image set 802.

[0125] The cloud server can transmit the image set 802 to the driver monitoring system (Driver Monitor System, DMS) 803. The driving monitoring system 803 can include a recognition model based on machine learning, which is used to identify the facial features, head features, etc. of the target object in the image for each image in the image set 802, and calculate the target object's eye closing degree score, yawning degree score, head lowering degree score, and degree of not looking forward degree score based on the identified facial features and head features, and return these scores to the cloud server.

[0126] The cloud server determines the behavior features 804 of the target object corresponding to the image set using a sliding window according to the eye closing degree score, yawning degree score, head lowering degree score, and not looking forward degree score of each image in the image set. The behavior features 804 may include eye closing behavior features, yawning behavior features, head lowering behavior features, and not looking forward behavior features. For example, each image set corresponds to a target object. For the i-th (i=1, 2, 3..., n) target object, the behavior feature set of the target object can be recorded as (ti, ici, yawn i , hli).

[0127] Since the target object’s driving time is also one of the factors affecting fatigue driving, we can set time features for the target object’s image set according to the target object’s driving time, and combine the time features with the behavior features to obtain the fatigue driving feature vector for the target object. For example, the fatigue driving feature vector of the i-th target object can be recorded as x i =(t i , ic i ,yawn i ,hl i , nff i ).

[0128] The fatigue driving feature vector x for the i-th target object is i =(t i , ic i ,yawn i ,hl i , nff i ) is input into a machine learning model (such as a GBDT model) to predict the fatigue driving state, and a prediction result is output 805. For example, output 1 indicates that the i-th target object is in a fatigue driving state, and output -1 indicates that the i-th target object is not in a fatigue driving state.

[0129] Fig. 9 is a block diagram of an apparatus for determining a state of an object according to an embodiment of the present disclosure.

[0130] like Fig. 9 As shown, the device 900 for determining the state of an object includes a time feature determination module 901 , a behavior feature determination module 902 and a driving state determination module 903 .

[0131] The time feature determination module 901 is used to determine an image set from a video frame sequence for a target object, and the image set is provided with a time feature based on the production time of the video frame sequence.

[0132] The behavior feature determination module 902 is used to determine the behavior feature for the image set according to the behavior information of the target object in each image in the image set.

[0133] The driving state determination module 903 is used to determine the driving state of the target object according to the behavior characteristics and time characteristics of the image set.

[0134] According to an embodiment of the present disclosure, the behavior feature determination module 902 includes a first evaluation unit, an image subset determination unit, a second evaluation unit, and a behavior feature determination unit.

[0135] The first evaluation unit is used to determine, for each image in the image set, a behavior evaluation value of a target object in the image as the behavior evaluation value of the image.

[0136] The image subset determination unit is used to sequentially determine a plurality of image subsets from the image set using a sliding window.

[0137] The second evaluation unit is used to determine, for each image subset, a behavior evaluation value of the image subset according to the behavior evaluation value of each image in the image subset.

[0138] The behavior feature determination unit is used to determine the behavior feature of the image set according to the behavior evaluation values ​​of each of the plurality of image subsets.

[0139] According to an embodiment of the present disclosure, the behavior includes at least one of closing eyes, yawning, lowering the head, and not looking forward; the behavior feature determination module 902 also includes a length determination unit.

[0140] The length determination unit is used to determine the length of the sliding window based on behavior, where the behavior includes at least one of closing eyes, yawning, lowering the head, and not looking forward.

[0141] According to an embodiment of the present disclosure, images in an image set are arranged in chronological order; and a length determination unit is used to determine the length of a sliding window for eye closing behavior according to a duration corresponding to the image set when the behavior is eye closing.

[0142] According to an embodiment of the present disclosure, a length determination unit is used to set a plurality of sliding windows of different lengths when the behavior is any one of yawning, lowering the head, and not looking forward; for each sliding window, determine a behavior evaluation value for a plurality of image subsets of the sliding window; and determine the precision and recall rate corresponding to the behavior evaluation value of each image subset; generate a statistical graph of the sliding window according to the behavior evaluation value, precision, and recall rate of each image subset; and determine a sliding window of target length from the plurality of sliding windows according to the respective statistical graphs of the plurality of sliding windows.

[0143] According to an embodiment of the present disclosure, the second evaluation unit is specifically configured to determine an average value of the behavior evaluation values ​​of the images in the image subset as the behavior evaluation value of the image subset.

[0144] The behavior feature determination unit is specifically configured to determine a maximum value of the behavior evaluation values ​​of the plurality of image subsets as the behavior feature of the image set.

[0145] The driving state determination module is specifically used to input the behavior characteristics and time characteristics into the machine learning model to obtain the driving state of the target object.

[0146] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0147] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0148] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0149] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0150] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a method for determining the state of an object. For example, in some embodiments, the method for determining the state of an object may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method for determining the state of an object described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the method for determining the state of the object in any other appropriate manner (eg, by means of firmware).

[0151] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0153] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0156] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0157] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a state of an object, comprising: determining an image set from a video frame sequence for a target object, the image set being provided with a time feature based on a production time of the video frame sequence; Determining a behavior feature for the image set according to behavior information of a target object in each image in the image set; as well as Determining the driving state of the target object according to the behavior characteristics of the image set and the time characteristics; Wherein, determining the behavior feature for the image set according to the behavior information of the target object in each image in the image set includes: For each image in the image set, determining a behavior evaluation value of a target object in the image as the behavior evaluation value of the image; sequentially determining a plurality of image subsets from the image set using a sliding window; For each image subset, determining a behavior evaluation value of the image subset according to the behavior evaluation values ​​of each image in the image subset; and The behavior feature of the image set is determined according to the behavior evaluation values ​​of each of the plurality of image subsets.

2. The method according to claim 1, wherein: The method further comprises: The length of the sliding window is determined based on a behavior, wherein the behavior includes at least one of closing eyes, yawning, lowering the head, and not looking forward.

3. The method according to claim 2, wherein: The images in the image set are arranged in chronological order; In a case where the behavior is closing eyes, determining the length of the sliding window based on the behavior includes: The length of the sliding window for the eye closing behavior is determined according to a time length corresponding to the image set.

4. The method according to claim 2, wherein: In a case where the behavior is any one of yawning, lowering the head, and not looking forward, the determining the length of the sliding window based on the behavior includes: Setting multiple sliding windows of different lengths; For each sliding window, determine a behavior evaluation value for the plurality of image subsets for the sliding window; and determine a precision and a recall rate corresponding to the behavior evaluation value for each image subset; and generate a statistical graph of the sliding window according to the behavior evaluation value, precision and recall rate for each image subset; and A sliding window of a target length is determined from the plurality of sliding windows according to respective statistical graphs of the plurality of sliding windows.

5. The method according to claim 1, wherein: The step of determining the behavior evaluation value of each image subset according to the behavior evaluation value of each image in the image subset includes: Determine an average value of the behavior evaluation values ​​of the images in the image subset as the behavior evaluation value of the image subset; Determining the behavior characteristics of the image set according to the behavior evaluation values ​​of each of the plurality of image subsets comprises: A maximum value of the behavior evaluation values ​​of the plurality of image subsets is determined as a behavior feature of the image set.

6. The method according to claim 1, wherein: The determining the driving state of the target object according to the behavior feature and the time feature of the image set includes: The behavior characteristics and the time characteristics are input into a machine learning model to obtain the driving state of the target object.

7. A device for determining a state of an object, comprising: A time feature determination module, for determining an image set from a video frame sequence for a target object, the image set being provided with a time feature based on a production time of the video frame sequence; A behavior feature determination module, configured to determine a behavior feature for the image set based on behavior information of a target object in each image in the image set; as well as A driving state determination module, used to determine the driving state of the target object according to the behavior characteristics of the image set and the time characteristics; Wherein, the behavior feature determination module includes: A first evaluation unit, configured to determine, for each image in the image set, a behavior evaluation value of a target object in the image as the behavior evaluation value of the image; An image subset determining unit, configured to sequentially determine a plurality of image subsets from the image set using a sliding window; a second evaluation unit, configured to determine, for each image subset, a behavior evaluation value of the image subset according to the behavior evaluation values ​​of the individual images in the image subset; and The behavior feature determination unit is used to determine the behavior feature of the image set according to the behavior evaluation values ​​of each of the multiple image subsets.

8. The device according to claim 7, wherein: The behavior feature determination module also includes: The length determination unit is used to determine the length of the sliding window based on behavior, wherein the behavior includes at least one of closing eyes, yawning, lowering the head, and not looking forward.

9. The device according to claim 8, wherein: The images in the image set are arranged in chronological order; the length determination unit is used to determine the length of the sliding window for the eye-closing behavior according to the duration corresponding to the image set when the behavior is eye-closing.

10. The device according to claim 8, wherein: The length determination unit is used to set a plurality of sliding windows of different lengths when the behavior is any one of yawning, lowering the head and not looking forward; and for each sliding window, determine a behavior evaluation value of the plurality of image subsets for the sliding window; And determine the precision and recall rate corresponding to the behavior evaluation value of each image subset; generate a statistical graph of the sliding window according to the behavior evaluation value, precision and recall rate of each image subset; and determine a sliding window of target length from the multiple sliding windows according to the respective statistical graphs of the multiple sliding windows.

11. The device according to claim 7, wherein: The second evaluation unit is specifically used to determine an average value of the behavior evaluation values ​​of the images in the image subset as the behavior evaluation value of the image subset; The behavior feature determination unit is specifically configured to determine a maximum value of the behavior evaluation values ​​of the plurality of image subsets as the behavior feature of the image set.

12. The device according to claim 7, wherein: The driving state determination module is specifically used to input the behavior characteristics and the time characteristics into a machine learning model to obtain the driving state of the target object.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images