A fatigue driving detection method based on driving video data

Abstract

The application discloses a kind of fatigue driving detection methods based on driving video data, specifically include: by analyzing the PSNR change between video frames and accurately detecting the driving state of driver driving using AI algorithm, further judge whether the driver is long time driving vehicle is in fatigue state, wherein, PSNR change is calculated by mean square error MSE, that is, for each pixel position of image, the square of the difference of pixel value is calculated, then average value is obtained;AI algorithm uses optical flow estimation to analyze the motion state between multiple image pixels.The application uses the existing driving video data in video driving recorder, avoids the high cost of detection equipment and the complex installation process.The present application provides an efficient, accurate, economical fatigue driving detection technology by fusing multidimensional data, using intelligent algorithm and optimizing system design, provides powerful guarantee means for timely and accurate evidence detection of vehicle accident.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fatigue driving detection based on driving video data, and in particular, a method and system for detecting fatigue driving by analyzing driving state using image indicators and AI algorithms based on video data stored in a video driving recorder. BACKGROUND

[0002] Fatigue driving is one of the main causes of traffic accidents. Existing fatigue driving detection techniques mainly rely on physiological indicators of the driver (such as eye and head movements) or vehicle driving data (such as speed and position). However, these methods have certain limitations in practical application, such as high cost of physiological indicator detection equipment and complexity and delay in collecting driving data through the vehicle CAN bus. Therefore, developing a fatigue driving detection technique based on driving video data is of great significance for improving our post-facto evidence of fatigue driving and reducing the difficulty of evidence collection in traffic accidents.

[0003] Existing technologies often rely on a single data source, such as physiological data or vehicle driving data, when detecting the fatigue state of a driver, which may be limited by the cost, complexity, and delay of detection equipment. SUMMARY

[0004] To address the above shortcomings, the present application proposes a fatigue driving detection technique based on driving video data, which can more accurately detect the driving state of the driver by analyzing the PSNR changes between video frames and using AI algorithms, and then determine whether the driver has been driving for a long time in a fatigue state. The specific steps are as follows:

[0005] Obtain all video files from the video driving recorder, and list sort the video files according to the video channel number and creation time;

[0006] Calculate the time difference between the update time and the creation time of the video file, and re-verify the file time according to the time difference;

[0007] Extract the initial video from the video file, record it as V0, and determine whether the channel of the initial video V0 is consistent with the channel of the current video Vi. If so, add the initial video V0 to the current timeline set T, and update the state of the current video Vi to the initial video V0;

[0008] Determine whether the update time of the current video Vi is greater than the update time of the initial video V0. If so, use the update time of the current video Vi as the update time of the initial video V0, and add the current video Vi to the set T;

[0009] Set the start time of the set T as the initial time, and set the end time of the set T as the end time; extract the optimal frame video point data from the video file for decoding and put it into the frame video point data set P;

[0010] Traverse the frame video point data set P, decode two images and calculate the peak signal-to-noise ratio PSNR between the two images, and use the peak signal-to-noise ratio PSNR to judge the driving motion state;

[0011] Use AI algorithm to analyze three consecutive frame video point data for secondary confirmation, judge whether the current state is the same as the state of the last video in the set T, if yes, update the end time;

[0012] Determine the motion state of the current video segment image and put it into the motion state set M, and judge whether the vehicle in the motion state set M has been driving continuously for more than 4 hours, if yes, there is fatigue driving.

[0013] Further, the judgment of whether the channel of the initial video V0 is consistent with the channel of the current video Vi, if not, a new timeline set T is created.

[0014] Determine whether the current processed video file belongs to the same timeline (i.e. the same channel), which is specifically represented as: check whether V0 and Vi come from the same channel. If the channels are consistent, they are considered to belong to the same timeline; otherwise, a new timeline needs to be created.

[0015] Create a new timeline set T, which is to create independent timelines for video files of different channels for separate processing; when it is found that Vi does not belong to the existing timeline, a new timeline set T is created to store the video files of this channel.

[0016] Further, the judgment of whether the update time of the current video Vi is greater than the update time of the initial video V0, if not, rejudge whether the channel of the initial video V0 is consistent with the channel of the current video Vi.

[0017] Ensure the correct time sequence and avoid adding video files in reverse order. If the update time of Vi is later than that of V0, continue processing; otherwise, return to the previous step to reevaluate the channel consistency.

[0018] Further, the calculation of the peak signal-to-noise ratio PSNR between the two images is specifically represented by the following formula:

[0019]

[0020] Where MAX1 represents the maximum value of the image signal, and MSE represents the mean square error.

[0021] The mean square error is used for the average value of the square difference between the pixel values of two images, and the specific formula is as follows:

[0022]

[0023] Wherein, I1 and I2 represent two images compared respectively, M represents the width of the image, N represents the height of the image, and (i, j) represents the pixel position of the image.

[0024] Further, the step of judging the driving state by using the peak signal-to-noise ratio (PSNR) includes judging the PSNR value, and if the PSNR value is lower than a preset low standard value, it indicates that there is a significant change between images, and the current state is driving, and if the PSNR is higher than a preset high standard value, it indicates that the change between images is small, and the current state is parking.

[0025] Based on the PSNR, it is judged whether the vehicle is driving or parking, a PSNR threshold value is defined in advance to distinguish the driving and parking states, if PSNR < threshold value: there is a significant change between images, and it is determined that the state is driving, and if PSNR >= threshold value: the change between images is small, and it is determined that the state is parking.

[0026] Further, the AI algorithm is used to analyze the continuous three frame video point data for secondary confirmation, and the stability of the current driving state is analyzed to improve the accuracy of the judgment, wherein the AI algorithm uses optical flow estimation to analyze the motion state between multiple image pixels.

[0027] Through the comprehensive analysis of the AI algorithm on multiple continuous frames, the accuracy of the state judgment can be improved.

[0028] Further, the state of the current state is compared with the state of the last video in the set T, if not, a new motion state record is created, and the peak signal-to-noise ratio (PSNR) is recalculated.

[0029] The state of the current judgment is compared with the state recorded last time to ensure that the continuous state record is accurate and correct, and the state information is updated in time.

[0030] According to the second aspect of the present application, a computer program product is provided, and one or more computer programs are stored on the computer program product, and the one or more computer programs are executed by a computer processor to implement the above method.

[0031] The one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0032] 1. No additional physiological detection device is needed, and the cost and complexity are reduced.

[0033] 2. Utilize driving video data, can accurately reflect the change of driving state, improve the reliability of detection.

[0034] 3. Combined with PSNR change and AI algorithm, improve the accuracy of detection.

[0035] 4. By analyzing the PSNR change between video frames, different driving states (such as driving, parking) can be distinguished, providing additional basis for judging the fatigue state of the driver. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0037] Figure 1 A flowchart of a fatigue driving detection method based on driving video data according to an embodiment of the present application is shown.

[0038] Figure 2 is a structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings.

[0040] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0041] Figure 1 A flowchart of a fatigue driving detection method based on driving video data according to an embodiment of the present application is shown, as shown in Figure 1

[0042] In this embodiment, the set is defined as follows:

[0043] The set V represents the set of current video file information obtained from the driving recorder, wherein V0 is the initial video in the set V, and the current video Vi traversed from the second video;

[0044] ​T represents a set of timelines filtered from the video set V obtained from the driving recorder;

[0045] P represents a set of valid frame image point data obtained from each video file in the timeline set T;

[0046] M represents a set of image motion states obtained from the point data set P.

[0047] S1, obtain all video files from the video driving recorder, put the video files into set V, and sort the video file list according to the video channel number and creation time;

[0048] Collect all available video files and ensure that they are sorted by channel number and creation time, which includes: traversing the storage location (such as SD card, hard disk, etc.) in the video driving recorder, extracting all video file paths, sorting according to the video channel number, and sorting according to the creation time under the same channel number.

[0049] S2, correct the file time according to the time difference between the file update time and the creation time;

[0050] Correcting the timestamp of the video file can ensure the accuracy of the time information of the video file. For each video Vi, the timestamp is calculated as Δt = t update -t create , where t update represents the update time, t create represents the creation time; if the time difference is significant or not as expected, adjust the timestamp of the file to reflect the actual recording time.

[0051] S3, determine whether the time channel of the initial video V0 is consistent with the current video Vi channel, if yes, go to step S5, if no, go to step S4;

[0052] Check if V0 and Vi come from the same channel, if the channels are consistent, consider them to belong to the same timeline; otherwise, a new timeline needs to be created.

[0053] S4, create a new timeline set T;

[0054] When Vi is found not to belong to the existing timeline, create a new timeline set T to store the video files of this channel.

[0055] S5, add the initial video V0 to the current timeline set T, and take it as the starting point of the timeline;

[0056] S6, update the status of the current video Vi to the initial video V0 for subsequent video file comparison and processing;

[0057] S7, judge whether the update time of the current video file Vi is greater than the update time of the initial video V0, ensure the correct time sequence, avoid adding video files in reverse order, if yes, go to step S8, if no, return to step S3;

[0058] S8, set the update time of Vi as the update time of V0, and add Vi to the set T;

[0059] Update the latest time point of the timeline, and add the current video file to the timeline.

[0060] S9, set the start time of the set T as the initial time, and set the end time of the set T as the end time; define the range of the timeline, and clearly define the time span of the video file.

[0061] S10, extract the best frame video point data from the video file for decoding and put it into the video point data set P;

[0062] According to certain rules (such as every fixed number of seconds, key scene changes, etc.), select the frame in the video. Decode the selected frame from the compressed format to the original image format. Store the decoded frame data in set P.

[0063] S11, traverse the video point data set P, decode two images and calculate the peak signal-to-noise ratio PSNR between the two images to quantify the image difference, wherein the calculation process of PSNR is as follows:

[0064]

[0065] Wherein, MAX1 represents the maximum value of the image signal, for 8-bit image, MAX1 = 255, MSE represents the mean square error;

[0066] The mean square error MSE is used to calculate the average value of the square difference between the pixel values of two images, and the specific formula is as follows:

[0067]

[0068] Wherein, I1 and I2 represent two images to be compared, M represents the width of the image, N represents the height of the image, and (i,j) represents the pixel position of the image.

[0069] S12, judge the PSNR value, if the PSNR value is lower than the preset low standard value, it means that there is a significant change between the images, and the current is in driving state, if the PSNR is higher than the preset high standard value, the image change is small, then the current is in parking state;

[0070] S13, use AI algorithm to analyze the secondary confirmation of the continuous three image point data, analyze the stability of the current driving state, in order to improve the accuracy of the judgment;

[0071] In this embodiment, the AI algorithm uses optical flow estimation to analyze the motion state between multiple image pixels, and the specific formula is expressed as: I(i,j,t) = I(i+Di,j+Dj,t+Dt), where I represents the image intensity, (i,j) represents the pixel coordinates, t represents the time, Di and Dj represent the pixel displacement, and Dt represents the time interval.

[0072] S14, judge whether the current state is the same as the last state in the list, if yes, update the end time, if not, create a new motion state record and enter step S11;

[0073] S15, determine the motion state of the current video segment image and put it into the motion state set M; store the result of each judgment to form a complete motion state sequence.

[0074] S16, judge whether the vehicle in the motion state set M has been driving continuously for more than 4 hours, if yes, there is fatigue driving.

[0075] If the continuous driving time exceeds 4 hours: mark as fatigue driving, otherwise, continue to monitor other state records; if fatigue driving is detected, trigger the corresponding early warning mechanism (such as issuing an alarm, notifying the driver to rest, etc.).

[0076] The present application provides a more intelligent and comprehensive video driving recorder data processing flow, which can more accurately judge the driving state of the vehicle by combining traditional image processing technology and advanced AI algorithm, and effectively identify potential safety hazards (fatigue driving). This multi-level analysis method not only improves the accuracy of state judgment, but also enhances the reliability and practicality of the system.

[0077] Among them, the existing driving video data in the video driving recorder is used, which avoids the high cost of detection equipment and the complex installation process. By fusing multi-dimensional data, using intelligent algorithms and optimizing system design, an efficient, accurate and economical fatigue driving detection technology is provided, which provides a powerful guarantee for timely and accurate evidence detection of vehicle accidents, and has significant innovation and application value for promoting road safety.

[0078] Reference is made below to Figure 2 which shows a structural schematic diagram of a computer system 200 of an electronic device suitable for implementing the embodiments of the present application. Figure 2 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0079] As Figure 2As shown, the computer system 200 includes a central processing unit (CPU) 201 which can perform various appropriate actions and processes in accordance with programs stored in a read only memory (ROM) 202 or programs loaded from a storage section 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0080] Connected to the I / O interface 205 are an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a display such as a liquid crystal display (LCD), and a speaker, etc.; the storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as required. A removable recording medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 210 as required, so that a computer program read therefrom is installed into the storage section 208 as required.

[0081] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 209, and / or installed from the detachable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above-described functions defined in the methods of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be either a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium can include a data signal that propagates in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take on many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable storage medium that can be used to carry or store program code for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted as program code signals using any suitable medium, including but not limited to wireless, wire line, optical fiber, RF, or any suitable combination of the above.

[0082] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0083] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0084] The modules involved in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware.

[0085] As another aspect, the application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire all video files from a video driving recorder, list sort the video files according to video channel number and creation time; calculate the time difference between the video file update time and the creation time, and recheck the file time according to the time difference; extract the initial video from the video file, record as V0, judge whether the channel of the initial video V0 is consistent with the channel of the current video Vi, if yes, add the initial video V0 to the current timeline set T, and update the state of the current video Vi to the initial video V0; judge whether the update time of the current video Vi is greater than the update time of the initial video V0, if yes, take the update time of the current video Vi as the update time of the initial video V0, and add the current video Vi to the set T; set the start time of the set T as the initial time, and set the end time of the set T as the end time; decode the best frame video point data from the video file, and put it into the frame video point data set P; traverse the frame video point data set P, decode two images and calculate the peak signal-to-noise ratio PSNR between the two images, and use the peak signal-to-noise ratio PSNR to judge the driving motion state; use AI algorithm to analyze the continuous three frame video point data for secondary confirmation, judge whether the current state is the same as the state of the last video in the set T, if yes, update the end time; determine the motion state of the current video segment image, put it into the motion state set M, and judge whether the vehicle in the motion state set M is continuously driven for more than 4 hours, if yes, there is fatigue driving.

[0086] The above description is merely the preferred embodiments of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A fatigue driving detection method based on driving video data, characterized by, The application relates to a method for judging driving fatigue of a vehicle. The method comprises the following steps: all video files are obtained from a video driving recorder, and the video files are listed and sorted according to video channel numbers and creation times; a time difference between a video file update time and a creation time is calculated, and the time of rechecking the file is determined according to the time difference; an initial video is extracted from the video file, recorded as V0, and it is judged whether the channel of the initial video V0 is consistent with the channel of a current video Vi; if yes, the initial video V0 is added to a current timeline set T, and the state of the current video Vi is updated to the initial video V0; it is judged whether the update time of the current video Vi is greater than the update time of the initial video V0; if yes, the update time of the current video Vi is taken as the update time of the initial video V0, and the current video Vi is added to the set T; the start time of the set T is set as an initial time, the end time of the set T is set as an end time, the best frame video point data are cut from the video file, decoded and put into a frame video point data set P; the frame video point data set P is traversed, two images are decoded, the peak signal-to-noise ratio (PSNR) between the two images is calculated, and the driving motion state is judged by using the PSNR; three continuous frame video point data are analyzed by using an AI algorithm for secondary confirmation, it is judged whether the current state is the same as the state of the last video in the set T, and if yes, the end time is updated; 2. The fatigue driving detection method according to claim 1, characterized by, the motion state of a current video segment image is determined and put into a motion state set M, and it is judged whether the vehicle in the motion state set M is continuously driven for more than 4 hours; if yes, fatigue driving exists.

3. The method of detecting driver fatigue according to claim 2, wherein if the channel of the initial video V0 is not consistent with the channel of the current video Vi, a new timeline set T is created.

4. The method of claim 1, wherein if the update time of the current video Vi is not greater than the update time of the initial video V0, it is rejudged whether the channel of the initial video V0 is consistent with the channel of the current video Vi. the peak signal-to-noise ratio (PSNR) between the two images is calculated, and the specific formula is as follows:

5. The method of detecting driver fatigue according to claim 4, wherein wherein MAX1 represents the maximum value of an image signal, and MSE represents a mean square error. the mean square error is used for the average value of the square difference between the pixel values of two images, and the specific formula is as follows:

6. The method of detecting driver fatigue according to claim 1, wherein wherein I1 and I2 respectively represent two compared images, M represents the width of the image, N represents the height of the image, and (i, j) represents the pixel position of the image.

7. The method of detecting driver fatigue according to claim 1, wherein the specific steps for judging the driving motion state by using the peak signal-to-noise ratio (PSNR) further comprise the following steps: the PSNR value is judged; if the PSNR value is lower than a preset low standard value, it is indicated that there is a significant change between the images, and the current state is driving; and if the PSNR is higher than a preset high standard value, it is indicated that the change between the images is small, and the current state is parking. the AI algorithm is used to analyze the stability of the current driving state by analyzing the motion state between multiple image pixels, so that the accuracy of judgment is improved.

8. The method of detecting driver fatigue according to claim 1, wherein The judgment whether the current state is same as the state of the last video in the set T, if not, a new motion state record is created and the peak signal-to-noise ratio (PSNR) is recalculated.

9. A computer program product, characterised in that, A computer program product comprising a computer readable medium having stored thereon computer program, the computer program comprising instructions executable by a processor to cause the processor to perform the method according to any one of claims 1-8.

10. A computing system, comprising: A computer program product comprising a computer readable medium having stored thereon computer program, the computer program comprising instructions executable by a processor to cause the processor to perform the method according to any one of claims 1-8. A computer program product comprising a computer readable medium having stored thereon computer program, the computer program comprising instructions executable by a processor to cause the processor to perform the method according to any one of claims 1-8.

Images