A method and apparatus for analyzing locomotive operation video

Abstract

The present application relates to a kind of locomotive operation video analysis method and device, belong to video intelligent identification technical field.The present application extracts corresponding video data according to the time period of the action of operation standard requirement appears, and the video data extracted is divided into multiple rounds to judge, every round only judges the frame data of every second, if corresponding action is judged in a certain round, the extracted other frame data will not be judged, greatly save the judgment length.In addition, for the analysis of illegal action, interval analysis method is used, and video data is extracted and analyzed according to the set time interval, further improve the analysis efficiency.

Description

Technical Field

[0001] This invention relates to a method and apparatus for analyzing locomotive operation videos, belonging to the field of video intelligent recognition technology. Background Technology

[0002] With the increasing number of digital equipment on trains, the data sources for locomotive crew operation data analysis have become more complex and massive. Relying on traditional manual retrieval and analysis is severely inadequate to meet safety management needs, preventing locomotive safety equipment from reaching its full potential. With advancements in science and technology, artificial intelligence and big data technologies are becoming increasingly mature, and train data has entered a stage of intelligent and automated analysis. Video data is a crucial component of locomotive crew operation data, and video analysis primarily addresses two types of needs: one is identifying inappropriate actions, such as inattentiveness, using mobile phones, or idle chatter in certain safety positions; the other is identifying the absence of required actions, such as whether operational standards and procedures are followed in inspection positions. Currently, video data analysis mainly utilizes various image analysis software based on deep learning, analyzing the obtained video data frame by frame. However, locomotive operation video data is generally very large, and meeting video analysis requirements requires significant time and hardware costs. For example, analyzing 5TB of video data would require approximately 10 devices simultaneously for 24 hours, resulting in very low analysis efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for analyzing locomotive operation videos, so as to solve the problem of low analysis efficiency caused by the large amount of video data in the current locomotive operation video analysis process.

[0004] To address the aforementioned technical problems, this invention provides a method for analyzing locomotive operation videos. This video analysis method includes the following steps:

[0005] 1) Acquire the locomotive operation video data to be analyzed and perform time synchronization on it;

[0006] 2) Obtain the time period in which the action required by the work standard occurs, find the video data in the corresponding time period from the video data that has undergone time synchronization, and cache it;

[0007] 3) Starting from the beginning of the time period, select N frames of data from each second of the data frame according to the jump analysis mechanism for the first round of judgment. Determine whether there is a corresponding action in each selected frame of data. If there is, it is considered that there is a corresponding action in the time period, and no further judgment is made on other frames of data in the time period. If there is no action, select N frames from the next second of the data frame for judgment, until a corresponding action is determined or the end of the time period is reached, and the first round of judgment is completed.

[0008] 4) If the corresponding action is not determined after the first round, the second round of judgment is performed in the same way as in step 3). In the second round of judgment, N frames are selected from the remaining MN frames of data per second, where M is the video frame rate.

[0009] 5) Repeat step 4) until a corresponding action is determined or the end time of the last round is reached.

[0010] This invention extracts corresponding video data based on the time period of the action required by the work standard, and divides the extracted video data into multiple rounds for judgment. Each round only judges a few frames of data per second. If a corresponding action is judged in a certain round, the other extracted frames will not be judged, which greatly saves the judgment time. At the same time, considering that the duration of the action required by the work standard is relatively short, judging only a few frames of video per second in each round further improves the analysis efficiency.

[0011] Furthermore, N is 2.

[0012] Furthermore, in each round of judgment, two frames of data with an interval of M / 2 per second are selected.

[0013] This invention only judges two frames of data per second in each round, and the interval between the two frames is half the video frame rate. If a corresponding action exists, it can be detected earlier, further improving the analysis efficiency.

[0014] Furthermore, the method also includes the step of deleting the data frames that have already been judged from the cache during each round of judgment.

[0015] After each judgment, the present invention deletes the video frame used for the judgment from the cache, reducing the amount of data in the cache and making subsequent judgments more convenient and faster.

[0016] Furthermore, the method also includes judging the video data according to a set interval duration to determine whether there are any actions that should not occur.

[0017] Violations typically last for a certain period of time. Therefore, this invention uses interval analysis to judge and analyze violations that should not occur, thereby improving the efficiency of violation analysis.

[0018] Furthermore, the set interval duration is 3 seconds.

[0019] Furthermore, when it is determined that there is an action that should not occur, video frames at preset times before and after the action occur are obtained, and it is determined whether the duration of the action is greater than a preset duration threshold. If it is greater, an alarm is triggered.

[0020] When an unacceptable action is detected, this invention acquires video data before and after the action and analyzes it second by second. If the analysis continues for a set duration, the action is considered to be illegal and unacceptable, thus eliminating some unnecessary interference and further improving the accuracy of detecting illegal actions.

[0021] Furthermore, if no corresponding action is detected within the specified time period after all rounds of judgment have been completed, an alarm will be triggered.

[0022] This invention provides an alarm reminder when no corresponding action is detected within a specific time period, ensuring that management personnel can detect it in a timely manner.

[0023] Furthermore, a deep learning algorithm model is used to determine whether an action has occurred from the video frames.

[0024] This invention uses a deep learning algorithm model to analyze and judge video frame data, which can quickly and accurately analyze video data.

[0025] The present invention also provides an analysis apparatus for locomotive operation videos, the apparatus comprising a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the locomotive operation video analysis method of the present invention by executing the executable instructions.

[0026] This invention extracts corresponding video data based on the time period of the action required by the work standard, and divides the extracted video data into multiple rounds for judgment. Each round only judges a few frames of data per second. If a corresponding action is judged in a certain round, the other extracted frames will not be judged, which greatly saves the judgment time. At the same time, considering that the duration of the action required by the work standard is relatively short, judging only a few frames of video per second in each round further improves the analysis efficiency. Detailed Implementation

[0027] The specific embodiments of the present invention will be further described below.

[0028] Examples of methods for analyzing locomotive operation videos

[0029] This invention extracts corresponding video data based on the time period of the action required by the work standard, and divides the extracted video data into multiple rounds for judgment. Each round only judges a few frames of data per second. If a corresponding action is judged in a certain round, the other extracted frames will not be judged. In addition, for the analysis of violations, an interval analysis method is used, extracting and analyzing data from the video data at set time intervals. The above process is described in detail below.

[0030] The analysis of locomotive operation videos mainly falls into two categories. One is to identify actions that shouldn't occur, such as inattentiveness, using a mobile phone, or idle chatter in certain safety positions. The other is to identify the absence of required actions, such as whether inspection personnel are following operational standards or procedures in certain inspection positions. These two categories will be explained in detail below.

[0031] 1. Identify and analyze the actions required by the work standards.

[0032] Locomotive operation standards require drivers to follow the standards during operation. For example, when arriving at a designated area, the driver must stand up and look out. Therefore, the purpose of this process is to quickly and accurately identify whether the driver has performed the actions required by the operation standards within a specified time period from a large amount of video data. The specific identification and analysis steps are as follows.

[0033] 1. Acquire the locomotive operation video data to be analyzed and perform time synchronization on it.

[0034] During operation, the driver will record the entire operation process with video and audio. The time in the original video data obtained by this invention is determined by the corresponding recording device. However, the time on the recording device may not be the standard time. The time period in which the operation is required to occur is divided according to the standard time. Therefore, it is necessary to perform time synchronization on the acquired locomotive operation video data to convert it to the standard time (such as Beijing time).

[0035] 2. Obtain the time period in which the action required by the work standard occurs, find the video data in the corresponding time period from the video data that has undergone time synchronization, and cache it.

[0036] The task requires obtaining the time period in which the action occurs. Based on this time period, the video data that has undergone time synchronization is searched to find the video data for the corresponding time period. To facilitate subsequent identification and analysis, the video data for the corresponding time period is cached.

[0037] 3. The cached video data is divided into multiple rounds for judgment. Each round only judges a few frames of data per second. If a corresponding action is judged in a certain round, the other extracted frames will no longer be judged.

[0038] Since the work standards require actions to be relatively fast, such as hand gestures and eye movements, the duration may be less than half a second. Therefore, it is necessary to ensure that the video recognition and analysis are not missed. At the same time, in order to improve the analysis efficiency, this invention adopts a skip analysis mechanism to divide the extracted video data into multiple rounds for analysis.

[0039] The first round: Starting from the beginning of the current time period, N frames of data are selected from each second's data frame according to the jump analysis mechanism for the first round of judgment. It is determined whether there is a corresponding action in each selected frame. If there is, it is considered that there is a corresponding action in that time period, and no further judgment is made on other frames of data in that time period. If there is no action, N frames are selected from the next second's data frame for judgment, until a corresponding action is determined or the end of the time period is reached. For example, in this embodiment, N is 2, and the video frame rate M is 24. At the beginning of the first round, two frames are randomly selected from the 24 frames of the first second for judgment. If a corresponding action is determined in these two frames, the round ends; otherwise, two frames are randomly selected from the 24 frames of the second second for judgment. If a corresponding action is determined in these two frames, the round ends; otherwise, the judgment continues until a corresponding action is determined or the last second of the time period is reached.

[0040] Second round: If the corresponding action is not identified after the first round, the second round of judgment is performed. Similar to the first round, it also starts from the first second of the time period. However, N frames are selected from the remaining MN frames of the first round for the second round of judgment, until the corresponding action is identified or the end of the time period is reached.

[0041] Round i: If the corresponding action is not determined after the (i-1)th round, the i-th round of judgment is performed. Similar to the previous round, it starts from the first second of the time period. However, N frames are selected from the remaining M-(i-1)N frames of data in the first round for the i-th round of judgment, until the corresponding action is determined or the end time of the time period is reached.

[0042] If the corresponding action is still not identified in the last round, it is considered that the driver did not perform the corresponding action as required by the operating standards during that time period, and an alarm will be triggered to remind the management personnel.

[0043] In the above process, N can take the value of 2, 3, 4, etc. The N frames selected per second are the data frames at intervals. For example, when N is 2, the two frames selected per second are two frames of data with an interval of M / 2. In the first round, the first frame and the M / 2+1th frame are selected. As other implementation methods, different interval methods can also be used.

[0044] Each round of judgment can employ a deep learning model, such as a neural network model. Training data is constructed using image data of various actions required by the standard and normal image data. This training data is then used to train the neural network model, which can then be used to identify each extracted frame of data.

[0045] In this embodiment, assuming a video frame rate of 24 frames per second, the first round of analysis can analyze frames 1 and 13, while the remaining frames are cached in memory. If the required content is found during the first round of analysis, the entire time period is skipped, as the job requirements are met. However, if no corresponding action or element is found by the end of the time period, a second round of analysis is required on the cached video frames. This analysis also employs skip-based analysis. For example, if frame numbers 2 to 12 and 14 to 24 are in memory after the first round of analysis, then frames 2 and 14 will be analyzed in the second round, and these frames will be removed from the buffer. Once the requirements are met during the analysis, the analysis of that time period is skipped, and the buffer is cleared. If the requirements are still not met, a third round of analysis is performed on the data in the buffer, starting with frames 3 and 15. This process is repeated several times until all video frames in the buffer have been analyzed. If the requirements are still not met after this process, an analysis alarm is immediately generated.

[0046] Second, identify and analyze actions that are not allowed to occur according to the work standards.

[0047] The operating standards require that actions that cannot occur include driver inattention, intermittent lookout, using a mobile phone, improper posture, and prolonged head-down time. These actions generally last for a certain duration, such as more than 3 seconds. Therefore, this invention uses interval analysis to identify these prohibited actions.

[0048] The specific process is as follows: First, video data is extracted at set time intervals, for example, one frame is extracted every 3 seconds. The extracted frames are analyzed to determine if any prohibited actions are present. This determination can also be made using a deep learning model, such as a neural network model. Training data is constructed using images of various prohibited actions and normal images. This training data is then used to train the neural network model, which can then identify each extracted frame. When a frame is found to contain a prohibited action, such as playing on a mobile phone, the video data before and after a set time is retrieved based on the time of that frame. The duration of the action is then checked to see if it exceeds a set duration threshold. If it does, an alarm is triggered. For example, using data from 2 seconds before and 2 seconds after the specified time, the video data is analyzed second by second, extracting frames every second for a total of 4 frames. The analysis checks if the prohibited action is also present in these 4 frames, thus detecting the duration of the action. If the duration exceeds 3 seconds, an alarm is triggered; if the duration is short, the action is considered acceptable, and no alarm is triggered.

[0049] Through the above process, this invention can meet the analysis needs of locomotive operation videos by analyzing a small number of video frames, greatly improving analysis efficiency. For 5TB of video data, existing analysis methods require approximately 10 devices simultaneously for 24 hours to complete the analysis. However, with this invention, because many invalid video frames are filtered out, the analysis can be completed with only 1 to 2 devices. Therefore, this invention also reduces costs while ensuring analysis effectiveness.

[0050] Example of a device for analyzing locomotive operation videos

[0051] The locomotive operation video analysis device of the present invention includes a processor and a memory. The processor executes a computer program stored in the memory to implement the method described in the above-described method embodiments. That is, the method in the above method embodiments should be understood as a flow of the locomotive operation video analysis method that can be implemented by computer program instructions. These computer program instructions can be provided to the processor, causing the processor to execute these instructions to produce the functions specified for implementing the above-described method flow.

[0052] In this embodiment, the processor refers to a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA); the memory refers to a physical device used to store information, which typically involves digitizing the information and then storing it using media that utilizes electrical, magnetic, or optical methods. Examples include: various types of memory that store information using electrical energy, such as RAM and ROM; various types of memory that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and various types of memory that store information using optical methods, such as CDs or DVDs. Of course, there are other types of memory, such as quantum memories and graphene memories.

[0053] The device consisting of the aforementioned memory, processor, and computer program is implemented in a computer by the processor executing the corresponding program instructions. The processor can run various operating systems, such as Windows, Linux, Android, and iOS.

[0054] As an alternative implementation, the device may also include a display for showing the classification results for staff reference.

[0055] The processor in this embodiment can be a Linux server. The processor executes corresponding program instructions as Python scripts. The Python scripts are executed daily at set times to write the classification results into the database and display them on the energy management system interface.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Many changes and modifications can be made without departing from the scope of the invention. Therefore, the above detailed description is intended to be illustrative rather than restrictive, and it should be understood that the following claims (including all equivalents) are intended to define the spirit and scope of the invention. These embodiments should be understood as illustrative only and not intended to limit the scope of protection of the invention. After reading the description of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent changes and modifications also fall within the scope defined by the claims of this invention.

Claims

1. A method for analyzing locomotive operation videos, characterized in that, This video analysis method includes the following steps: 1) Acquire the locomotive operation video data to be analyzed and perform time synchronization on it; 2) Obtain the time period in which the action required by the work standard occurs, find the video data in the corresponding time period from the video data that has undergone time synchronization, and cache it; 3) Starting from the beginning of the time period, select N frames of data from each second of the data frame according to the jump analysis mechanism for the first round of judgment. Determine whether there is a corresponding action in each selected frame of data. If there is, it is considered that there is a corresponding action in the time period, and no further judgment is made on other frames of data in the time period. If there is no action, select N frames from the next second of the data frame for judgment, until a corresponding action is determined or the end of the time period is reached, and the first round of judgment is completed. 4) If the corresponding action is not determined after the first round, the second round of judgment is performed in the same way as in step 3). In the second round of judgment, N frames are selected from the remaining MN frames of data per second, where M is the video frame rate. 5) Repeat step 4) until a corresponding action is determined or the end time of the last round is reached.

2. The method for analyzing locomotive operation videos according to claim 1, characterized in that, The value of N is 2.

3. The method for analyzing locomotive operation videos according to claim 2, characterized in that, In each round of judgment, two frames of data with an interval of M / 2 per second are selected.

4. The method for analyzing locomotive operation videos according to claim 1, characterized in that, The method also includes a step of deleting the data frames that have already been judged from the cache during each round of judgment.

5. The method for analyzing locomotive operation videos according to claim 1, characterized in that, The method also includes judging the video data according to a set interval to determine whether there are any actions that should not occur.

6. The method for analyzing locomotive operation videos according to claim 5, characterized in that, The set interval duration is 3 seconds.

7. The method for analyzing locomotive operation videos according to claim 5, characterized in that, When an action that should not occur is detected, video frames are acquired before and after the action occurs within a set time frame. The duration of the action is then checked to see if it exceeds a set time threshold. If it does, an alarm is triggered.

8. The method for analyzing locomotive operation videos according to claim 1, characterized in that, If no corresponding action is detected within the specified time period after all rounds of judgment have been completed, an alarm will be triggered.

9. The method for analyzing locomotive operation videos according to any one of claims 1-8, characterized in that, The deep learning algorithm model is used to determine whether an action has occurred in a video frame.

10. An analysis device for locomotive operation videos, characterized in that, The device includes a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method for analyzing locomotive operation video according to any one of claims 1-9 by executing the executable instructions.