Fatigue driving detection method based on driving video data

Abstract

The invention discloses a fatigue driving detection method based on driving video data, and the method specifically comprises the steps: analyzing the PSNR change between video frames, accurately detecting the driving state of a driver through an AI algorithm, and then judging whether the driver is in a fatigue state after long-time driving of the vehicle, in other words, for each pixel position of the image, the square of the difference of pixel values is calculated, and then the average value is calculated; the AI algorithm employs optical flow estimation to analyze motion states between a plurality of image pixels. According to the invention, the existing driving video data in the video driving recorder is utilized, so that high cost and complex installation process of detection equipment are avoided. By fusing multi-dimensional data and adopting an intelligent algorithm and optimization system design, an efficient, accurate and economical fatigue driving detection technology is provided, and a powerful guarantee means is provided for timely and accurate evidence collection and detection of vehicle accidents.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving detection based on driving video data, and in particular, to a method and system for analyzing driving status using image indicators of video data stored in a video driving recorder and an AI algorithm to detect whether fatigue driving exists, and mainly to a fatigue driving detection method based on driving video data. Background Art

[0002] Fatigue driving is one of the main causes of traffic accidents. Existing fatigue driving detection technologies mainly rely on the driver's physiological indicators (such as eye and head movements) or the vehicle's driving data (such as speed and position). However, these methods have certain limitations in practical applications, such as the high cost of physiological indicator detection equipment, and the complexity and delay of collecting driving data through the vehicle CAN bus. Therefore, developing a fatigue driving detection technology based on driving video data is of great significance to improving our post-facto evidence of fatigue driving and reducing the difficulty of evidence collection in traffic accidents.

[0003] Existing technologies often rely on a single data source, such as physiological data or vehicle driving data, when detecting driver fatigue, which may be limited by the cost, complexity and latency of the detection equipment. Summary of the invention

[0004] In view of the above shortcomings, the present invention proposes a fatigue driving detection technology based on driving video data. By analyzing the PSNR changes between video frames and using AI algorithms, the driving state of the driver can be detected more accurately, and then it can be determined whether the driver is in a fatigue state after driving the vehicle for a long time. The specific steps are as follows:

[0005] Obtain all video files from the video driving recorder, and sort the video files into a list according to video channel number and creation time;

[0006] Calculate the time difference between the video file update time and creation time, and recheck the file time according to the time difference;

[0007] Extract the initial video from the video file and record it as V0, determine whether the channel of the initial video V0 is consistent with the channel of the current video Vi, and if so, add the initial video V0 to the current timeline set T, and update the status of the current video Vi to the initial video V0;

[0008] Determine whether the update time of the current video Vi is greater than the update time of the initial video V0. If so, use the update time of the current video Vi as the update time of the initial video V0, and add the current video Vi to the set T;

[0009] The start time of the set T is set as the initial time, and the end time of the set T is set as the end time; the best frame video point data is intercepted from the video file for decoding, and is put into the frame video point data set P;

[0010] Traversing the frame video point data set P, decoding two images and calculating a peak signal-to-noise ratio (PSNR) between the two images, and using the peak signal-to-noise ratio (PSNR) to determine the vehicle motion state;

[0011] Use AI algorithm to analyze three consecutive frames of video point data for secondary confirmation to determine whether the current state is the same as the state of the last video in the set T, and if so, update the end time;

[0012] The motion state of the current video clip image is determined and put into the motion state set M, and it is judged whether the vehicle in the motion state set M has been driven continuously for more than 4 hours. If so, fatigue driving exists.

[0013] Furthermore, it is determined whether the channel of the initial video V0 is consistent with the channel of the current video Vi. If not, a new timeline set T is created.

[0014] Determine whether the currently processed video file belongs to the same timeline (i.e. the same channel), specifically: check whether V0 and Vi are from the same channel. If the channels are consistent, they are considered to belong to the same timeline; otherwise, a new timeline needs to be created.

[0015] A new timeline set T is created, specifically, independent timelines are created for video files of different channels so as to be processed separately; when it is found that Vi does not belong to an existing timeline, a new timeline set T is created to store the video files of the channel.

[0016] Furthermore, it is determined whether the update time of the current video Vi is greater than the update time of the initial video V0. If not, it is re-determined whether the channel of the initial video V0 is consistent with the channel of the current video Vi.

[0017] Ensure that the time sequence is correct and avoid adding video files in reverse order. If the update time of Vi is later than V0, continue processing; otherwise, return to the previous step to re-evaluate channel consistency.

[0018] Furthermore, the peak signal-to-noise ratio PSNR between the two images is calculated, and the specific formula is as follows:

[0019]

[0020] Wherein, MAX1 represents the maximum value of the image signal, and MSE represents the mean square error.

[0021] The mean square error is used to calculate the average value of the square difference between the pixel values ​​of two images. The specific formula is as follows:

[0022]

[0023] Among them, I1 and I2 represent two compared images, M represents the width of the image, N represents the height of the image, and (i, j) represents the pixel position of the image.

[0024] Furthermore, the specific steps of using the peak signal-to-noise ratio PSNR to determine the driving motion state also include: determining the PSNR value. If the PSNR value is lower than a preset low standard value, it indicates that there are significant changes between images and the vehicle is currently in a driving state. If the PSNR value is higher than a preset high standard value, it indicates that there are small changes between images and the vehicle is currently in a parking state.

[0025] Based on PSNR, it is determined whether the vehicle is driving or parked. A PSNR threshold is pre-defined to distinguish between driving and parking states. If PSNR < threshold: there is a significant change between images, it is determined to be in a driving state. If PSNR ≥ threshold: the image changes are small, it is determined to be in a parking state.

[0026] Furthermore, the AI ​​algorithm is used to analyze three consecutive frames of video point data for secondary confirmation, and analyze the stability of the current driving state to improve the accuracy of the judgment, wherein the AI ​​algorithm uses optical flow estimation to analyze the motion state between multiple image pixels.

[0027] Comprehensive analysis of multiple consecutive frames through AI algorithms can improve the accuracy of status judgment.

[0028] Furthermore, it is determined whether the current state is the same as the state of the last video in the set T. If not, a new motion state record is created and the peak signal-to-noise ratio PSNR is recalculated.

[0029] Compare the currently determined state with the most recently recorded state to ensure that continuous state records are accurate and that state information is updated in a timely manner.

[0030] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0031] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0032] 1. No additional physiological testing equipment is required, reducing cost and complexity.

[0033] 2. The use of driving video data can accurately reflect changes in driving status and improve the reliability of detection.

[0034] 3. Combining PSNR changes and AI algorithms to improve detection accuracy.

[0035] 4. By analyzing the PSNR changes between video frames, different driving states (such as driving and parking) can be distinguished, providing additional basis for judging the driver's fatigue state. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0037] Figure 1 A schematic flow chart of a method for detecting fatigue driving based on driving video data according to an embodiment of the present invention is shown.

[0038] Figure 2 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] Figure 1 FIG. 4 is a flow chart showing a method for detecting fatigue driving based on driving video data according to an embodiment of the present invention. Figure 1 As shown:

[0042] In this embodiment, the set is defined as follows:

[0043] The set V represents the set of current video file information obtained from the driving recorder, where V0 is the initial video in the set V, and the current video Vi traversed from the second video;

[0044] The set T represents the set of timelines that meet the conditions selected from the video set V obtained from the driving recorder;

[0045] The set P represents the set of valid frame image point data obtained from each video file of the timeline set T;

[0046] The set M represents the set of image motion states obtained from the point data set P.

[0047] S1. Obtain all video files from the video driving recorder, put the video files into a set V, and sort the video file list according to the video channel number and creation time;

[0048] Collect all available video files and ensure that they are arranged in order by channel number and creation time, including: traversing the storage locations in the video driving recorder (such as SD card, hard disk, etc.), extracting all video file paths, sorting them according to the video channel number, and then sorting them according to the creation time under the same channel number.

[0049] S2, re-correcting the file time according to the time difference between the file update time and the creation time;

[0050] Correcting the timestamp of a video file can ensure that the time information of the video file is accurate. For each video Vi, the timestamp is calculated as Δt = t update -t create , where t update represents the update time, t create Indicates creation time; if the time difference is significant or unexpected, adjust the file's timestamp to reflect the actual recording time.

[0051] S3, determine whether the time channel of the initial video V0 is consistent with the time channel of the current video Vi, if so, go to step S5, if not, go to step S4;

[0052] Check whether V0 and Vi are from the same channel. If the channels are consistent, they are considered to belong to the same timeline; otherwise, a new timeline needs to be created.

[0053] S4. Create a new timeline set T;

[0054] When it is found that Vi does not belong to the existing timeline, a new timeline set T is created to store the video files of this channel.

[0055] S5, adding the initial video V0 to the current timeline set T, and using it as the starting point of the timeline;

[0056] S6, updating the status of the current video Vi to the initial video V0 for comparison and processing of subsequent video files;

[0057] S7, determine whether the update time of the current video file Vi is greater than the update time of the initial video V0, ensure the correct time sequence, and avoid adding video files in reverse order. If yes, go to step S8, if not, return to step S3;

[0058] S8. Use the update time of Vi as the update time of V0 and add Vi to the set T;

[0059] Update the latest time point of the timeline and add the current video file to the timeline.

[0060] S9. Set the start time of set T as the initial time, and set the end time of set T as the end time; define the range of the timeline and clarify the time span of the video file.

[0061] S10, extracting the best frame video point data from the video file, decoding it, and putting it into the video point data set P;

[0062] Select frames in the video according to specific rules (such as every fixed number of seconds, key scene changes, etc.). Decode the selected frames from the compressed format to the original image format. Store the decoded frame data in the set P.

[0063] S11, traverse the video point data set P, decode the two images, and calculate the peak signal-to-noise ratio PSNR between the two images to quantify the image difference, wherein the calculation process of PSNR is as follows:

[0064]

[0065] Wherein, MAX1 represents the maximum value of the image signal. For an 8-bit image, MAX1=255, and MSE represents the mean square error.

[0066] The mean square error MSE is used to calculate the average value of the square difference between the pixel values ​​of two images. The specific formula is as follows:

[0067]

[0068] Among them, I1 and I2 represent two compared images, M represents the width of the image, N represents the height of the image, and (i, j) represents the pixel position of the image.

[0069] S12, determining the PSNR value. If the PSNR value is lower than the preset low standard value, it indicates that there is a significant change between the images and the vehicle is currently in a driving state. If the PSNR value is higher than the preset high standard value, the image change is small and the vehicle is currently in a parking state.

[0070] S13, using AI algorithm to analyze three consecutive image point data for secondary confirmation, analyzing the stability of the current driving state to improve the accuracy of judgment;

[0071] In this embodiment, the AI ​​algorithm uses optical flow estimation to analyze the motion state between multiple image pixels. The specific formula is: I(i,j,t)=I(i+Δi,j+Δj,t+Δt), where I represents image intensity, (i,j) represents pixel coordinates, t represents time, Δi and Δj represent pixel displacement, and Δt represents time interval.

[0072] S14, determine whether the current state is the same as the last state in the list, if so, update the end time, if not, create a new motion state record and go to step S11;

[0073] S15, determining the motion state of the current video clip image, and putting it into a motion state set M; storing the result of each determination to form a complete motion state sequence.

[0074] S16. Determine whether the vehicle in the motion state set M has been driven continuously for more than 4 hours. If so, fatigue driving occurs.

[0075] If the continuous driving time exceeds 4 hours: mark it as fatigue driving, otherwise, continue to monitor other status records; if fatigue driving is detected, trigger the corresponding early warning mechanism (such as issuing an alarm, notifying the driver to take a rest, etc.).

[0076] The present invention provides a more intelligent and comprehensive video driving recorder data processing flow. By combining traditional image processing technology and advanced AI algorithms, it can more accurately judge the driving status of the vehicle and effectively identify potential safety hazards (fatigue driving). This multi-level analysis method not only improves the accuracy of status judgment, but also enhances the reliability and practicality of the system.

[0077] Among them, the use of existing driving video data in the video driving recorder avoids the high cost and complicated installation process of the detection equipment. By integrating multi-dimensional data, using intelligent algorithms and optimizing system design, an efficient, accurate and economical fatigue driving detection technology is provided, which provides a powerful guarantee for timely and accurate evidence collection and detection of vehicle accidents, and has significant innovation and application value in promoting road safety.

[0078] Reference below Figure 2 , which shows a schematic diagram of the structure of a computer system 200 suitable for implementing an electronic device of an embodiment of the present application. Figure 2 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 2As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0080] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0081] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0082] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0085] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: obtains all video files from the video driving recorder, and lists and sorts the video files according to the video channel number and the creation time; calculates the time difference between the video file update time and the creation time, and rechecks the file time according to the time difference; extracts the initial video from the video file, records it as V0, and determines whether the channel of the initial video V0 is consistent with the channel of the current video Vi. If so, the initial video V0 is added to the current timeline set T, and the status of the current video Vi is updated to the initial video V0; determines whether the update time of the current video Vi is greater than the update time of the initial video V0. If so, the update time of the current video Vi is used as the update time of the initial video V0. A new time is set, and the current video Vi is added to the set T; the start time of the set T is set as the initial time, and the end time of the set T is set as the end time; the best frame video point data is intercepted from the video file for decoding, and is put into the frame video point data set P; the frame video point data set P is traversed, the two images are decoded and the peak signal-to-noise ratio PSNR between the two images is calculated, and the driving motion state is determined by the peak signal-to-noise ratio PSNR; the AI ​​algorithm is used to analyze three consecutive frame video point data for secondary confirmation to determine whether the current state is the same as the state of the last video in the set T, and if so, the end time is updated; the motion state of the current video clip image is determined, and the motion state is put into the motion state set M, and it is determined whether the vehicle in the motion state set M has been driven continuously for more than 4 hours, and if so, fatigue driving exists.

[0086] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A fatigue driving detection method based on driving video data, characterized in that: include: Obtain all video files from the video driving recorder, and sort the video files into a list according to video channel number and creation time; Calculate the time difference between the video file update time and creation time, and recheck the file time according to the time difference; Extract the initial video from the video file and record it as V0, determine whether the channel of the initial video V0 is consistent with the channel of the current video Vi, and if so, add the initial video V0 to the current timeline set T, and update the status of the current video Vi to the initial video V0; Determine whether the update time of the current video Vi is greater than the update time of the initial video V0. If so, use the update time of the current video Vi as the update time of the initial video V0, and add the current video Vi to the set T; The start time of the set T is set as the initial time, and the end time of the set T is set as the end time; the best frame video point data is intercepted from the video file for decoding, and is put into the frame video point data set P; Traversing the frame video point data set P, decoding two images and calculating a peak signal-to-noise ratio (PSNR) between the two images, and using the peak signal-to-noise ratio (PSNR) to determine the vehicle motion state; Use AI algorithm to analyze three consecutive frames of video point data for secondary confirmation to determine whether the current state is the same as the state of the last video in the set T, and if so, update the end time; The motion state of the current video clip image is determined and put into the motion state set M, and it is judged whether the vehicle in the motion state set M has been driven continuously for more than 4 hours. If so, fatigue driving exists.

2. The fatigue driving detection method according to claim 1, characterized in that: The process determines whether the channel of the initial video V0 is consistent with the channel of the current video Vi. If not, a new timeline set T is created.

3. The fatigue driving detection method according to claim 2, characterized in that: The determination is made whether the update time of the current video Vi is greater than the update time of the initial video V0. If not, it is determined again whether the channel of the initial video V0 is consistent with the channel of the current video Vi.

4. The fatigue driving detection method according to claim 1, characterized in that: The peak signal-to-noise ratio PSNR between the two images is calculated, and the specific formula is as follows: Wherein, MAX1 represents the maximum value of the image signal, and MSE represents the mean square error.

5. The fatigue driving detection method according to claim 4, characterized in that: The mean square error is used to calculate the average value of the square difference between the pixel values ​​of two images. The specific formula is as follows: Among them, I1 and I2 represent two compared images, M represents the width of the image, N represents the height of the image, and (i, j) represents the pixel position of the image.

6. The fatigue driving detection method according to claim 1, characterized in that: The specific steps of using the peak signal-to-noise ratio PSNR to determine the driving motion state also include: determining the PSNR value. If the PSNR value is lower than a preset low standard value, it indicates that there are significant changes between images and the vehicle is currently in a driving state. If the PSNR value is higher than a preset high standard value, it indicates that there are small changes between images and the vehicle is currently in a parking state.

7. The fatigue driving detection method according to claim 1, characterized in that: The AI ​​algorithm is used to analyze three consecutive frames of video point data for secondary confirmation, and analyze the stability of the current driving state to improve the accuracy of the judgment. The AI ​​algorithm uses optical flow estimation to analyze the motion state between multiple image pixels.

8. The fatigue driving detection method according to claim 1, characterized in that: The determination is as to whether the current state is the same as the state of the last video in the set T. If not, a new motion state record is created and the peak signal-to-noise ratio PSNR is recalculated.

9. A computer program product, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A computing system, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 8.

Images