Video online identification method and device for locomotive

Abstract

The invention discloses a video online identification method and device for a locomotive, and relates to the technical field of railway vehicles, and the method comprises the steps: receiving a steward monitoring video stream sent by a front camera and a pantograph monitoring video stream sent by a pantograph camera; based on the steward monitoring video stream, determining whether the steward has a view deviation abnormal condition and a discontinuous lookout abnormal condition, and determining whether the front camera has a shielding abnormal condition; based on the pantograph monitoring video stream, determining whether the pantograph has a structure abnormal condition, a spark abnormal condition and a foreign matter invasion abnormal condition; in response to determining that at least one of the visual field deviation abnormal condition, the intermittent outlook abnormal condition and the shielding abnormal condition exists, carrying out online real-time reminding on the steward; and in response to determining that at least one of the structure abnormal condition, the spark abnormal condition and the foreign matter invasion abnormal condition exists, determining a fault sub-video from the pantograph monitoring video stream, and sending the fault sub-video to the client.

Description

Technical Field

[0001] The present invention relates to the technical field of rail vehicle safety monitoring, and in particular to a video online recognition method and device for a locomotive. Background Art

[0002] With the rapid development of railways, people are paying more and more attention to train operation safety. On the one hand, the railway transportation widely implements the working mode of long routes and single driver on duty, which makes the locomotive crew's operating section long and the work pressure high. Especially when on duty at night, the probability of locomotive crew getting fatigue at work increases, which has become a key risk source for train operation safety. The mental state of the crew is directly related to driving safety; on the other hand, the pantograph is a device for locomotive vehicles to collect current from the contact network. The state of the pantograph directly affects the safety of the train, and its failure may even cause transportation interruption.

[0003] At present, all locomotives on the railway are equipped with 6A system video monitoring subsystems to monitor the driver's cab, pantographs and road conditions ahead. At present, data analysis of 6A system videos is mostly done manually, which not only requires a large number of video data analysts, but also fails to provide real-time online reminders to crew members; in addition, it is also unable to effectively identify problems caused by blurred images of pantograph monitoring equipment.

[0004] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section. Summary of the invention

[0005] In order to solve at least one technical problem in the background technology part, the present application provides a method and device for online video recognition of locomotives, which performs online real-time monitoring and early warning of crew status and pantograph status, effectively improving the safety of locomotive operation.

[0006] In a first aspect, the present application provides a method for online video recognition of a locomotive, comprising:

[0007] Receive the crew monitoring video stream sent by the front camera and the pantograph monitoring video stream sent by the pantograph camera;

[0008] Based on the crew monitoring video stream, determining whether the crew has abnormal field of vision deviation and abnormal intermittent observation, and determining whether the front camera has abnormal occlusion;

[0009] Based on the pantograph monitoring video stream, determining whether the pantograph has structural abnormalities, spark abnormalities, and foreign object intrusion abnormalities;

[0010] In response to determining that at least one of the abnormal situation of field of view deviation, the abnormal situation of intermittent observation and the abnormal situation of obstruction exists, providing an online real-time reminder to the crew;

[0011] In response to determining the existence of at least one of the structural abnormality, the spark abnormality and the foreign object intrusion abnormality, a fault sub-video is determined from the pantograph monitoring video stream and the fault sub-video is sent to the client.

[0012] In some optional aspects of this embodiment, the step of determining whether the flight attendant has an abnormal field of vision deviation based on the flight attendant monitoring video stream includes:

[0013] Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0014] Determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the front face of the crew member cannot be detected within a first preset time period;

[0015] In response to the determination being yes, it is determined that the flight attendant has the visual field deviation abnormality.

[0016] In some optional aspects of this embodiment, the step of determining whether the crew has an abnormal situation of intermittent lookout based on the crew monitoring video stream includes:

[0017] Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0018] Determining whether the crew monitoring video stream continuously contains closed-eye crew monitoring frame images within a second preset time period;

[0019] In response to determining yes, it is determined that the crew member has the intermittent lookout abnormality.

[0020] In some optional aspects of this embodiment, the step of determining whether the front camera is blocked abnormally based on the crew monitoring video includes:

[0021] Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0022] determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the complete face of the crew member cannot be detected within a third preset time period;

[0023] In response to the determination being yes, it is determined that the front camera has the occlusion abnormality.

[0024] In some optional aspects of this embodiment, the step of determining whether the pantograph has a structural abnormality based on the pantograph monitoring video stream includes:

[0025] Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0026] Performing pantograph posture recognition on the pantograph monitoring frame image to determine a plurality of key points in the pantograph, wherein the types of the key points include two ends of the carbon slide plate, left and right bow angles of the pantograph, a bow head bracket and a balance bar;

[0027] Based on each of the key points, determine whether the pantograph has any structural abnormality, wherein the structural abnormality includes carbon slide plate abnormality and bow angle abnormality.

[0028] In some optional aspects of this embodiment, the step of determining whether the pantograph has an abnormal spark condition based on the pantograph monitoring video stream includes:

[0029] Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0030] Based on a pre-trained spark recognition model, determining whether sparks exist in the pantograph monitoring frame image;

[0031] In response to determining that the spark exists, determining whether an area of ​​the spark is greater than or equal to a preset area;

[0032] In response to determining yes, determining that an abnormal spark is present in the pantograph;

[0033] Determining whether the pantograph monitoring video stream continuously contains a first preset number of frames of pantograph monitoring frame images including the abnormal spark within a fourth preset time period;

[0034] In response to determining yes, it is determined that the pantograph has the spark abnormality condition.

[0035] In some optional aspects of this embodiment, the step of determining whether there is an abnormal situation of foreign object intrusion in the pantograph based on the pantograph monitoring video stream includes:

[0036] Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0037] Based on a pre-trained foreign object recognition model, determining whether there is a foreign object in the pantograph monitoring frame image;

[0038] In response to the determination being yes, determining whether the pantograph monitoring video stream continuously contains at least a second preset number of frames including the pantograph monitoring frame image of the foreign object within a fifth preset time period;

[0039] In response to determining yes, it is determined that the pantograph has the foreign object intrusion abnormality.

[0040] In a second aspect, the present application further provides a video online recognition device for a locomotive, comprising:

[0041] A video stream receiving module is configured to receive a crew monitoring video stream sent by a front camera and a pantograph monitoring video stream sent by a pantograph camera;

[0042] The crew and front camera abnormality monitoring module is configured to determine whether the crew has abnormal vision deviation and abnormal intermittent observation based on the crew monitoring video stream, and determine whether the front camera has abnormal occlusion;

[0043] A pantograph abnormality monitoring module is configured to determine whether the pantograph has structural abnormality, spark abnormality and foreign object intrusion abnormality based on the pantograph monitoring video stream;

[0044] A crew reminder module, configured to, in response to determining that at least one of the abnormal situation of field of view deviation, the abnormal situation of intermittent observation and the abnormal situation of occlusion exists, give an online real-time reminder to the crew;

[0045] The fault sub-video pushing module is configured to determine the fault sub-video from the pantograph monitoring video stream in response to determining the existence of at least one of the structural abnormality, spark abnormality and foreign object intrusion abnormality, and send the fault sub-video to the client.

[0046] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned online video recognition method for locomotives when executing the computer program.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for online video recognition of locomotives is implemented.

[0048] In a fifth aspect, an embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program, and the computer program, when executed by a processor, implements the above-mentioned method for online video recognition of locomotives.

[0049] An embodiment of the present invention provides a method and device for online video recognition of a locomotive. The method and device determine whether the crew has abnormal field of view deviation and intermittent observation based on the crew monitoring video stream, and whether the front camera has abnormal occlusion; determine whether the pantograph has structural abnormalities, spark abnormalities, and foreign object intrusion abnormalities based on the pantograph monitoring video stream; in response to determining the existence of at least one of the abnormal field of view deviation, intermittent observation abnormality, and occlusion abnormality, the crew is reminded online in real time; in response to determining the existence of at least one of the abnormal structure, spark abnormality, and foreign object intrusion abnormality, a fault sub-video is determined from the pantograph monitoring video stream, and the fault sub-video is sent to a client, so that the crew status and pantograph status can be monitored, analyzed, and warned online in real time, effectively improving the safety of locomotive operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0051] Figure 1 This is one of the flow charts of a method for online video recognition of a locomotive in an embodiment of the present invention;

[0052] Figure 2 This is a second flow chart of a method for online video recognition of a locomotive in an embodiment of the present invention;

[0053] Figure 3 The third flowchart of a method for online video recognition of a locomotive according to an embodiment of the present invention;

[0054] Figure 4 This is a fourth flow chart of a method for online video recognition of a locomotive according to an embodiment of the present invention;

[0055] Figure 5 Schematic diagram of the fifth process of a method for online video recognition of a locomotive in an embodiment of the present invention;

[0056] Figure 6 6 is a flow chart of a method for online video recognition of a locomotive according to an embodiment of the present invention;

[0057] Figure 7 FIG7 is a flow chart of a method for online video recognition of a locomotive according to an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the structure of a 6A video surveillance system in an embodiment of the present invention;

[0059] Fig. 9 is a schematic diagram of key points detected on a pantograph in an embodiment of the present invention;

[0060] Fig.10 Schematic diagram of the structure of a video online recognition device for a locomotive in an embodiment of the present invention;

[0061] Fig.11 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0063] It should be noted that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0064] For example, provide users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, they will enter the expert decision-making process.

[0065] With the rapid development of railways, train operation safety has attracted more and more attention. The mental state of the crew is directly related to driving safety. During the operation of the train, the poor mental state of the locomotive crew or intermittent lookout is a hidden danger to the safety of train operation. At present, the railway transportation widely implements the long route and single driver duty mode, which leads to the long operation section and high work pressure of the locomotive crew. Especially when on duty at night, the probability of fatigue of the locomotive crew at work increases, which has become a key risk to the safety of train operation. The mental state of the source crew is directly related to the safety of driving; on the other hand, the pantograph is a device for locomotive vehicles to collect current from the contact network. The state of the pantograph directly affects the safety of the train, and its failure may even cause transportation interruption.

[0066] At present, all locomotives on the railway are equipped with 6A system video monitoring subsystems to monitor the driver's cab, pantographs and road conditions ahead. At present, data analysis of 6A system videos is mostly done by manual browsing, which not only requires a large number of video data analysts, but also fails to provide real-time online reminders to crew members; in addition, it is also unable to effectively identify problems caused by blurred pantograph monitoring equipment, and there are still deficiencies in the detection and classification of abnormal problems such as foreign body intrusion.

[0067] Based on this, in order to solve at least one of the above problems, the embodiment of the present application provides a video online recognition method for locomotives, such as Figure 1 As shown, the method includes:

[0068] Step 10: Receive the crew monitoring video stream sent by the front camera and the pantograph monitoring video stream sent by the pantograph camera.

[0069] In the present application, the crew is generally monitored by a front camera. The crew of this embodiment is generally the driver in the cab, and the crew can also be monitored by a camera in the driver's cab; the pantograph is generally monitored by a pantograph camera.

[0070] Specifically, the execution entity of the present application receives the crew monitoring video stream sent by the front camera and the pantograph monitoring video stream sent by the pantograph camera.

[0071] Step 20: Based on the crew monitoring video stream, determine whether the crew has abnormal field of view deviation and abnormal intermittent observation, and determine whether the front camera has abnormal occlusion.

[0072] In the present application, based on the crew monitoring video stream, it is possible to determine whether the crew has abnormal field of view deviation and abnormal intermittent observation, and determine whether the front camera has abnormal occlusion.

[0073] In some optional embodiments of this embodiment, Figure 2 As shown, the step of determining whether the flight attendant has an abnormal field of vision deviation based on the flight attendant monitoring video stream includes:

[0074] Step 201: Process the crew monitoring video stream frame by frame to obtain multiple crew monitoring frame images;

[0075] Step 202: determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the front face of the crew member cannot be detected within a first preset time period;

[0076] Step 203: In response to the determination being yes, determining that the flight attendant has an abnormal visual field deviation.

[0077] In the present application, the crew monitoring video stream is first processed frame by frame to obtain multiple crew monitoring frame images; secondly, it is determined whether the crew monitoring video stream contains crew monitoring frame images in which the front face of the crew member cannot be detected within a first preset time period; finally, if it is determined that such images exist, it is determined that the crew member has the abnormal field of view deviation.

[0078] In some optional embodiments of this embodiment, Figure 3 As shown, based on the crew monitoring video stream, the step of determining whether the crew has an abnormal situation of intermittent lookout includes:

[0079] Step 204: Process the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0080] Step 205: determining whether the crew monitoring video stream contains continuous closed-eye crew monitoring frame images within a second preset time period;

[0081] Step 206: In response to the determination being yes, determining that the crew member has the abnormal intermittent lookout condition.

[0082] In the present application, the crew monitoring video stream is first processed frame by frame to obtain multiple crew monitoring frame images; secondly, it is determined whether the crew monitoring video stream contains continuous crew monitoring frame images with closed eyes within a second preset time period; finally, if it is determined that there is such a situation, it is determined that the crew has an abnormal intermittent lookout situation.

[0083] In some optional embodiments of this embodiment, Figure 4 As shown, the step of determining whether the front camera is blocked abnormally based on the crew monitoring video includes:

[0084] Step 207: Process the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0085] Step 208: determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the complete face of the crew member cannot be detected within a third preset time period;

[0086] Step 209: In response to the determination being yes, determining that the front camera has the occlusion anomaly.

[0087] In the present application, the crew monitoring video stream is first processed frame by frame to obtain multiple crew monitoring frame images; secondly, it is determined whether the crew monitoring video stream continuously contains crew monitoring frame images in which the complete face of the crew member cannot be detected within a third preset time period; finally, if it is determined that such images exist, the front camera is abnormally blocked.

[0088] Specifically, in this application, the high-definition video online intelligent recognition system of the locomotive on-board safety protection system is taken as an example to illustrate this application. It is generally composed of a pantograph high-definition camera (i.e., pantograph camera), a driver's cab camera (i.e., driver's cab camera), a panoramic camera, a front camera, a video storage server, an image intelligent analysis board, a monitoring screen (i.e., a display terminal), a network switch, and a network cable.

[0089] The video storage server obtains monitoring videos such as the pantograph camera, the cab camera, and the front camera through the ONVIF and RTSP protocols to realize real-time preview and video storage. The display screen realizes real-time preview and video playback of monitoring videos such as the pantograph high-definition camera, the cab camera, and the front camera through the RTSP protocol. The image intelligent analysis board has the functions of online recognition of crew status and intelligent recognition of pantograph status, and is configured to execute the online video recognition method for locomotives of this application.

[0090] It should be noted that the crew status online recognition function includes three functional modules: the intermittent lookout online reminder module, the field of view deviation online reminder module and the front camera occlusion online reminder module. Among them, by pulling the real-time video stream for algorithm analysis, the crew's working status is monitored in real time, and if an abnormal state occurs, an early warning is issued in real time. The intermittent lookout online reminder module and the field of view deviation online reminder module use the same YOLOv5 model structure as the pantograph anomaly detection, and process the real-time video stream obtained from the camera frame by frame, including image scaling and standardization, to meet the input requirements of the YOLOv5 model; the pre-processed images are sent to the trained YOLOv5 model for analysis.

[0091] Intermittent lookout online reminder module: By detecting the flight attendant's facial state, it is determined whether the flight attendant is in an intermittent lookout state. If the flight attendant is detected to have closed eyes and this state lasts for a certain period of time (i.e., the second preset time period), the system will issue the first, second, and third level online reminders in sequence; during the reminder process, if the flight attendant resumes looking straight ahead, the system will automatically cancel the reminder.

[0092] Vision deviation online reminder module: When the system cannot detect the front face of the flight attendant but detects the side face, it is judged as a vision deviation state. If this state lasts for more than 60 seconds (i.e. the first preset time period), the system will issue a vision deviation reminder; if the flight attendant's face re-enters the camera's field of view, the reminder will be automatically lifted.

[0093] Front camera occlusion online reminder module: Use deep neural networks for image classification to determine whether the camera's field of view is blocked. Among them, the deep neural network extracts multi-level features of the image through multiple residual blocks, so as to effectively identify whether the front camera is blocked. For example, first, the crew monitoring video stream is processed frame by frame to obtain multiple crew monitoring frame images; secondly, it is determined whether the crew monitoring video stream has continuous crew monitoring frame images in which the complete face of the crew member cannot be detected within a third preset time period; finally, if it is determined that it exists, the front camera is abnormally blocked; once the system detects that the camera's field of view is blocked, the module will immediately issue a reminder to deal with the occlusion problem in a timely manner to ensure the normal operation of the camera and the continuity of monitoring.

[0094] It should be understood that the embodiment of the present application uses high-definition monitoring equipment, which can also distinguish warnings for different problems and provide a basis for train maintenance based on graded quantitative evaluation of their severity and warning prompts.

[0095] At this point, in response to the lookout status and operating standard problems that may arise when the driver is on duty, this embodiment realizes functions such as online real-time monitoring, voice warning, vehicle-ground interaction, data analysis, and equipment status tracking through real-time identification of the driver's status. It reverses the locomotive system's control of driving safety from post-analysis to online real-time monitoring, warning, and hierarchical real-time intervention and control of on-site operations by the ground duty station, and effectively moves the safety risk management checkpoint forward.

[0096] Step 30: Based on the pantograph monitoring video stream, determine whether the pantograph has structural abnormalities, spark abnormalities, and foreign object intrusion abnormalities.

[0097] It should be noted that the pantograph is a device for locomotives to collect current from the contact network. The condition of the pantograph directly affects the safe and reliable operation of the train, and its failure may even cause transportation interruption. Through online real-time monitoring and analysis of the pantograph by high-definition video monitoring equipment, it is possible to promptly and accurately find the area where the pantograph network has problems, understand the abnormality or damage of the pantograph network, which will be conducive to timely maintenance and repair of the pantograph network.

[0098] When the locomotive is running at high speed, the pantograph working environment is complex, and a series of problems may occur during this period, causing damage to the locomotive pantograph, which may seriously affect the safe and smooth operation of the locomotive, such as abnormal pantograph structure, abnormal sparks and foreign object intrusion.

[0099] Specifically, when the locomotive is running at high speed, the pantograph is subjected to complex forces, and the special working environment makes the pantograph components prone to various problems. The different states or defects of the pantograph can reflect the corresponding pantograph-net faults. When the pantograph carbon slide plate is deformed, the bow angle is broken or falls off, it can be classified as a pantograph structural abnormality. There must always be a certain contact pressure between the pantograph and the contact network to ensure that the locomotive is in good current receiving condition. When the contact pressure is too small or even zero, the pantograph slide bar will detach from the contact network and go offline. Although medium and small offline will not affect the operation of the locomotive, the sparks or arcs generated at the moment of offline will increase the electrical wear of the contact wire and the pantograph slide bar. Since the pantograph works in an outdoor environment, when the locomotive is running at high speed, in addition to the faults caused by the pantograph itself, it may also be invaded by foreign objects caused by bad weather, human factors, etc., resulting in foreign objects hanging on the pantograph or near the pantograph, thereby affecting the normal operation of the pantograph.

[0100] In this implementation, the status of the pantograph is monitored in real time by a high-definition pantograph video surveillance device installed on the roof. The image intelligent analysis board (i.e., the executor of the online video recognition method for locomotives in this application) analyzes the video data, identifies whether the pantograph status is abnormal, and classifies and reports the problem.

[0101] In some optional embodiments of this embodiment, Figure 5 As shown, the step of determining whether the pantograph has a structural abnormality based on the pantograph monitoring video stream includes:

[0102] Step 301: Process the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0103] Step 302: performing pantograph posture recognition on the pantograph monitoring frame image to determine a plurality of key points in the pantograph;

[0104] Step 303: Based on each of the key points, determine whether the pantograph has any structural abnormality, wherein the structural abnormality includes carbon slide plate abnormality and bow angle abnormality.

[0105] In the present application, the pantograph monitoring video stream is first processed frame by frame to obtain multiple pantograph monitoring frame images; secondly, the pantograph monitoring frame images are subjected to pantograph posture recognition to determine multiple key points in the pantograph, wherein the types of key points include the two ends of the carbon slide board, the left and right bow angles of the pantograph, the bow head bracket and the balance bar; finally, based on each key point, it is determined whether there is any structural abnormality in the pantograph, wherein the structural abnormality includes carbon slide board abnormality and bow angle abnormality.

[0106] In some optional embodiments of this embodiment, Figure 6As shown, the step of determining whether the pantograph has an abnormal spark condition based on the pantograph monitoring video stream includes:

[0107] Step 304: Process the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0108] Step 305: Determine whether there is a spark in the pantograph monitoring frame image based on a pre-trained spark recognition model;

[0109] Step 306: In response to determining that the spark exists, determining whether the area of ​​the spark is greater than or equal to a preset area;

[0110] Step 307, in response to the determination being yes, determining that the pantograph has abnormal sparks;

[0111] Step 308: determine whether the pantograph monitoring video stream contains a first preset number of consecutive pantograph monitoring frame images including the abnormal spark within a fourth preset time period;

[0112] Step 309: In response to the determination being yes, determine whether the pantograph has the abnormal spark condition.

[0113] In some optional embodiments of this embodiment, Figure 7 As shown, based on the pantograph monitoring video stream, the step of determining whether there is an abnormal situation of foreign object intrusion in the pantograph includes:

[0114] Step 310: Process the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0115] Step 311: Determine whether there is a foreign object in the pantograph monitoring frame image based on a pre-trained foreign object recognition model;

[0116] Step 312: In response to the determination being yes, determining whether the pantograph monitoring video stream continuously contains at least a second preset number of frames of pantograph monitoring frame images of the foreign object within a fifth preset time period;

[0117] Step 313: In response to the determination being yes, determining that the pantograph has the foreign object intrusion abnormality.

[0118] The identification of structural abnormalities, spark abnormalities, and foreign matter intrusion abnormalities in steps 301 to 312 is still described using the aforementioned system. Specifically:

[0119] In a specific example, if Figure 8As shown, the image intelligent analysis board is installed in the cabinet of the 6A host. The video streams of the front camera and the pantograph camera can be directly pulled from the 6A video system to analyze the mental state of the crew and the working status of the pantograph in real time, and the analysis results are stored and marked.

[0120] like Figure 8 As shown in the figure, the intelligent recognition system of 6A high-definition video surveillance is mainly composed of multiple high-definition cameras (such as road condition cameras, pantograph cameras, driver's cab cameras, machine room cameras, front cameras and coupler cameras, etc.), image intelligent analysis boards (installed in the cabinet of the 6A host), monitoring screens (such as I-end display terminal, II-end display terminal), etc. It should be noted that each locomotive is equipped with at least one set of image intelligent analysis boards, one set of monitoring screens, one set of pantograph high-definition cameras and one set of front cameras.

[0121] In the abnormal identification of the pantograph, the working process of the image intelligent analysis board is as follows:

[0122] (1) Based on the intelligent analysis pipeline framework, multiple pantograph videos and driver's cab videos can be analyzed in real time, and the videos one minute before and after the fault can be stored in real time;

[0123] (2) Based on the optimized live555 streaming technology, the fault video is pushed to the RTSP server. The client can pull the video stream on the RTSP server for real-time preview according to the search conditions;

[0124] (3) Pull the real-time video stream for algorithm analysis, monitor the pantograph tilt, structural anomalies, spark anomalies, and foreign object intrusion in real time, send the detected pantograph fault message to the client, and view the warning picture on the monitoring screen accordingly, and pull the real-time video of the one minute and one minute before and after the fault.

[0125] In this implementation, the image intelligent analysis board uses a variety of artificial intelligence technical means to perform fault analysis on the acquired pantograph high-definition monitoring video, mainly including the pantograph posture recognition algorithm and the pantograph target detection based on the YOLOV5 algorithm, among which:

[0126] 1. Pantograph posture recognition algorithm:

[0127] In order to accurately detect the pantograph posture, the present invention first defines the location of the key points based on the structural characteristics of the pantograph. Fig. 9As shown, these key points include the two ends of the carbon slide plate 11 (such as key point M and key point N), the left and right bow angles of the pantograph, the bow head bracket 12 (such as key point P, key point D, key point E, key point F, key point G and key point H) and the balance bar, etc. The selection of these positions not only considers the overall shape of the pantograph, but also pays attention to the possible changes in these positions under different working conditions, so it has important detection significance.

[0128] Based on these key points, the data set required for model training was constructed. The specific construction process includes the following steps: First, pantograph image data covering various working conditions were collected. These images can reflect the posture changes of the pantograph in different operating environments and states; then, the key point positions in these images were accurately annotated through manual annotation methods; finally, all the annotated data were integrated to form a high-quality data set for model training.

[0129] After the data set is constructed, the pantograph posture recognition algorithm is trained to detect whether the pantograph is tilted. The pantograph posture recognition algorithm designed by the present invention consists of four main modules, namely, data preprocessing module, feature extraction module, key point detection module and post-processing module. These four modules are responsible for different tasks respectively, and work together as a whole to ensure that the final output detection results are highly accurate and reliable.

[0130] First, the data preprocessing module is responsible for a series of standardization processes on the collected pantograph monitoring frame images. The purpose of this step is to unify the input format of the image and eliminate the differences caused by factors such as image source, shooting angle or ambient lighting. The specific processing process includes adjusting the image size to meet the input requirements of the model; normalizing the image pixel values ​​to ensure that different images can be calculated in a consistent manner during the training process; and appropriately adjusting the color and brightness of the image to enhance the diversity of the training data, thereby improving the generalization ability of the key point detection model. Through these preprocessing steps, the image quality has been significantly improved, and it also provides more favorable conditions for subsequent feature extraction and posture detection.

[0131] Next, the feature extraction module uses the target detection algorithm to identify and extract the relevant areas of the pantograph from the preprocessed image. The main task of this module is to identify the location of the pantograph by scanning the pantograph area that may be contained in the image and separate the area from the background. This step ensures that the subsequent posture analysis is only performed on the pantograph itself without being interfered by background information or other irrelevant parts. The result of feature extraction directly affects the accuracy of subsequent key point detection, so the execution quality of this step is crucial to the entire process.

[0132] Subsequently, the keypoint detection module is one of the core components of the pantograph posture recognition algorithm, which undertakes the task of detailed analysis of the pantograph posture. At this stage, the pantograph image segmented by the object detection algorithm is input to the spatial transformer network (STN) to standardize the input pantograph posture, making the subsequent pantograph keypoint detection process more robust. The single person posture estimation (SPPE) is responsible for detecting the pantograph keypoints from the image output by the STN and optimizing the prediction accuracy of the pantograph keypoints through the mean square error loss function as described below:

[0133]

[0134] in, is the predicted heat map, is the real heat map, and N is the number of key points. Subsequently, the reverse spatial transformer network (SDTN) further optimizes and refines the key point positions, and introduces an offset loss function to calculate the mean square error between the predicted offset value and the true offset value, as described below:

[0135]

[0136] in, is the key point offset predicted by SDTN, is the real offset. SDTN corrects errors by performing local spatial transformation on the key point positions, thereby improving the accuracy of posture estimation. The pantograph key point detection module designed by the present invention can reduce background interference in the input image and ultimately accurately identify the posture changes of the pantograph. This module must not only consider the posture of the pantograph under normal operating conditions, but also be able to identify various abnormal posture changes caused by external factors. Through a series of optimizations and adjustments, the model can maintain high-precision detection capabilities in a variety of complex environments, thereby ensuring that the positioning of key point positions is accurate.

[0137] Finally, the post-processing module uses the pose non-maximum suppression (Pose NMS) technology to further refine and confirm the results of the previous detection steps. Pose NMS eliminates redundant and inaccurate detection information through confidence screening and distance calculation, ensuring that the final output results are highly reliable. The post-processing module removes noise data that may affect the final analysis results through a series of screening and optimization algorithms, thereby providing a more accurate basis for monitoring the pantograph status. Through this process, the pantograph posture recognition algorithm can output the most reliable posture detection results, providing a solid foundation for subsequent analysis and judgment.

[0138] Combining the collaborative work of the above four modules, the trained pantograph posture recognition algorithm can perform posture analysis on the image data in the real-time video stream. After booting up, the pantograph image detected and segmented by the first 100 frames of images collected by the camera is used to calculate the average value of the pantograph posture, and it is recorded as the initial posture of the pantograph. Subsequently, the overall and local posture changes of the pantograph are calculated frame by frame, and compared with the initial posture of the pantograph to analyze the rotation of the pantograph in the up and down and left and right directions. At the same time, the angle change of the bow angle is also calculated to determine whether the bow angle is broken or detached. The detected tilt angle is compared with the set threshold. If the current tilt angle exceeds the threshold, it is determined that the pantograph has a tilt fault.

[0139] 2. Pantograph target detection based on YOLOV5 algorithm:

[0140] The present invention develops a target detection subsystem for pantographs based on the YOLOv5 model. The subsystem includes a preprocessing module, a model network module, a pantograph spark anomaly detection module, a pantograph carbon slide anomaly detection module, a pantograph bow angle anomaly detection module, and a pantograph foreign object intrusion detection module. These modules work together to detect various abnormal conditions of the pantograph in real time, ensuring the efficiency and reliability of the system. Specifically:

[0141] a. Preprocessing module

[0142] Before the image is input into the YOLOv5 model, the preprocessing module first scales the image to a uniform size and performs pixel normalization. This step eliminates the scale differences between different images and helps improve the processing efficiency of the model. To enhance the generalization ability of the model, the preprocessing module also increases the diversity of training data through a series of operations (such as cropping, rotation, color adjustment, etc.). In order to overcome the problem of insufficient negative samples in the training data, the system uses the Generative Adversarial Network (GAN) for data enhancement to generate more negative samples, thereby improving the performance of the model in anomaly detection tasks.

[0143] b. Model network module

[0144] The model network module of YOLOv5 consists of three parts: backbone network, neck network and head network, each of which plays a key role in the overall detection process.

[0145] The backbone network is responsible for extracting different levels of features from the pantograph image. Through a series of image processing steps, from low-level detail features to high-level abstract information, the backbone network gradually extracts and integrates these features to provide a basis for subsequent detection.

[0146] The neck network plays an important role in feature fusion. It combines multi-scale features to enable the model to effectively detect pantographs under different lighting and background conditions. This module not only helps the model to utilize high-resolution information, but also improves the detection accuracy of small targets (such as missing carbon slide corners) by enhancing feature fusion capabilities.

[0147] The head network is responsible for generating the final detection results. It generates outputs suitable for targets of different sizes, such as pantographs, sparks, and foreign object intrusions, through multiple prediction layers. The output of each prediction layer includes the target's bounding box, confidence, and category probability, ensuring that the model can accurately identify and locate various abnormal situations.

[0148] The loss function of the YOLOv5 model used in the present invention is composed of the bounding box regression loss (L GIoU ), target confidence loss (L obj ) and classification loss (L cls ) together as follows:

[0149] LOSS=λ box ×L GIoU +λ obj ×L obj +λ cls ×L cls

[0150] Among them, λ box , obj , cls is the weight factor that controls the contribution of each loss, the bounding box regression loss L GIoU The function definition is as follows:

[0151]

[0152] Among them, A and B are the predicted border and the real border respectively, and C represents the minimum enclosing rectangular area of ​​A and B. The target confidence loss is used to measure whether the cell contains the target, and the formula is expressed as:

[0153]

[0154] Among them, S 2 Indicates the number of cells, Indicates whether cell i contains the target, if it contains, it is 1, otherwise it is 0. i is the target confidence of the prediction of the i-th cell. BCE(p,q)=-[q log(p)+(1-q)log(1-p)] is the binary cross entropy loss. The classification loss measures the gap between the predicted category and the actual category, expressed as:

[0155]

[0156] Where C is the total number of categories, p i (c) represents the probability that the i-th cell predicts category c, t i (c) is the one-hot encoding of the category label.

[0157] c. Pantograph spark abnormality detection module

[0158] The present invention sets the category of sunlight to distinguish the working conditions where sunlight exists in the background, so as to avoid misjudging sunlight as large sparks under sunlight working conditions. The pantograph image is captured frame by frame from the video stream in real time, and the image is input into the model reasoning program. If a spark category is detected in the model reasoning output, and the area of ​​the spark occupies 1 / 20 of the image area, it is judged that a spark appears in the pantograph. When sparks are judged to appear for 25 consecutive frames (i.e., the first preset number of frames) within 1 second (i.e., the fourth preset time period) in the real-time video stream data, the pantograph spark anomaly detection module triggers an early warning, and the module will immediately record and stream large spark fault pictures and videos.

[0159] d. Pantograph carbon slide abnormality detection module

[0160] The carbon slide anomaly detection module is designed to monitor the status of the pantograph carbon slide in real time. The pantograph image is captured frame by frame from the video stream in real time and the image is input into the model inference program. If the model inference output detects the presence of carbon slide anomaly categories (such as carbon slide bending), the pantograph carbon slide is judged to be abnormal, and the module will immediately record and stream the carbon slide anomaly fault pictures and videos.

[0161] e. Pantograph Angle Abnormal Detection Module

[0162] The pantograph angle abnormality detection module is used to identify structural damage to the pantograph angle, such as breakage or detachment. The pantograph image is captured frame by frame from the video stream in real time and the image is input into the model inference program. If the pantograph angle abnormality category (such as pantograph angle breakage) is detected in the model inference output, the pantograph angle is judged to be abnormal, and the module will immediately record and stream the pantograph angle abnormality fault picture and video.

[0163] f. Pantograph foreign body intrusion detection module

[0164] The foreign object intrusion detection module is responsible for monitoring the safety of the environment around the pantograph, especially identifying foreign objects that may affect the normal operation of the pantograph. The pantograph image is captured frame by frame from the video stream in real time, and the image is input into the model inference program. The foreign object category is detected in the model inference output, and the pantograph is judged to be invaded by foreign objects. When at least 100 frames (i.e., the second preset number of frames) of images in the real-time video stream data within 5 seconds (i.e., the fifth preset time period) detect foreign object intrusion, the pantograph foreign object intrusion detection module triggers an early warning, and the module will immediately record and push foreign object intrusion fault pictures and videos.

[0165] It should be noted that the pantograph posture recognition algorithm based on the pantograph and the pantograph target detection based on the YOLOV5 algorithm of this embodiment jointly realize the identification of pantograph structural abnormalities, spark abnormalities and foreign object intrusion of this application.

[0166] In this application, pantograph structural abnormalities, spark abnormalities, and foreign object intrusion are problems that locomotives may encounter during high-speed travel. By using high-definition video surveillance equipment to monitor the pantograph online in real time, the high-quality video data collected can effectively improve the problem detection rate during algorithm analysis and avoid the risk of missed problems. The image intelligent analysis board pulls the video stream for real-time algorithm analysis, which can promptly and accurately detect the area where the pantograph has problems and classify the causes of the problems. The staff can promptly determine whether the pantograph is abnormal or damaged and accurately locate the problem. This will be conducive to the timely maintenance and inspection of the pantograph network and improve work efficiency.

[0167] Step 40: In response to determining that at least one of the abnormal situation of field of view deviation, the abnormal situation of intermittent observation and the abnormal situation of obstruction exists, an online real-time reminder is given to the crew.

[0168] In the present application, if it is determined that at least one of the abnormal situation of field of vision deviation, the abnormal situation of intermittent observation and the abnormal situation of obstruction exists, the crew is reminded online in real time.

[0169] Step 50: In response to determining that at least one of the structural abnormality, the spark abnormality and the foreign object intrusion abnormality exists, determine a fault sub-video from the pantograph monitoring video stream, and send the fault sub-video to the client.

[0170] In the present application, if it is determined that at least one of the structural abnormality, spark abnormality and foreign object intrusion abnormality exists, a fault sub-video is determined from the pantograph monitoring video stream and the fault sub-video is sent to the client, where the client is, for example, a display terminal.

[0171] In this embodiment, the image intelligent analysis board performs algorithm analysis by pulling the real-time video stream of the pantograph, and detects the pantograph structure abnormalities, spark abnormalities, foreign object intrusion and other points in real time. The detected pantograph fault forms an early warning message and sends it to the client for classified early warning. The client can also view the early warning picture and pull the video recording of one minute and one minute before and after the fault. In a specific example:

[0172] (1) When the vehicle speed is higher than 40 km / h and the pantograph high-definition camera is at the preset position, when the spark area of ​​25 consecutive frames within 1 second is larger than 1 / 20 of the entire screen area, the large spark fault video is pushed and the staff can view the specific pictures and video information on the intelligent analysis interface of the monitoring screen.

[0173] (2) When the vehicle speed is higher than 40 km / h and the pantograph high-definition camera is at the preset point, when the carbon slide plate is deformed, the bow angle is broken or falls off, the push-flow pantograph structure is abnormal and the staff can view the specific pictures and video information on the intelligent analysis interface of the monitoring screen.

[0174] (3) When the vehicle speed is higher than 40 km / h and the pantograph high-definition camera is at the preset point, when there is a foreign object larger than 10*10 cm in the pantograph area and 100 frames appear within 5 seconds, the foreign object intrusion video will be streamed and the staff can view the specific pictures and video information on the intelligent analysis interface of the monitoring screen.

[0175] In this application, by conducting real-time online intelligent diagnosis of video images and issuing early warning reminders on the crew status and abnormal pantograph status, the mental state of the locomotive crew and the abnormal status of the pantograph can be monitored in real time during train operation, and when the locomotive crew members doze off or fall asleep, or when the pantograph shows abnormalities, prompt warning signals to wake up the locomotive crew members are issued in a timely and graded manner; transforming safety risk management from post-analysis of results to online real-time reminder prevention is of great practical significance to railway transportation safety management.

[0176] The present invention also provides a video online recognition device for locomotives, as described in the following embodiments. Since the principle of the device to solve the problem is similar to that of the video online recognition method for locomotives, the implementation of the device can refer to the implementation of the video online recognition method for locomotives, and the repeated parts will not be repeated.

[0177] like Fig.10 As shown, the video online recognition device for locomotives includes:

[0178] The video stream receiving module 601 is configured to receive the crew monitoring video stream sent by the front camera and the pantograph monitoring video stream sent by the pantograph camera;

[0179] The crew and front camera abnormality monitoring module 602 is configured to determine whether the crew has abnormal vision deviation and abnormal intermittent observation based on the crew monitoring video stream, and determine whether the front camera has abnormal occlusion;

[0180] The pantograph abnormality monitoring module 603 is configured to determine whether the pantograph has structural abnormality, spark abnormality and foreign object intrusion abnormality based on the pantograph monitoring video stream;

[0181] The crew reminder module 604 is configured to, in response to determining that at least one of the abnormal situation of field of view deviation, the abnormal situation of intermittent observation and the abnormal situation of occlusion exists, give the crew an online real-time reminder;

[0182] The fault sub-video push module 605 is configured to determine the fault sub-video from the pantograph monitoring video stream in response to determining the existence of at least one of the structural abnormality, spark abnormality and foreign object intrusion abnormality, and send the fault sub-video to the client.

[0183] In some optional embodiments of the present embodiment, the flight attendant and front camera abnormality monitoring module includes a field of view deviation abnormality monitoring unit, an intermittent observation abnormality monitoring unit and an occlusion abnormality monitoring unit.

[0184] In some optional aspects of this embodiment, the visual field deviation abnormality monitoring unit is configured as follows:

[0185] Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0186] Determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the front face of the crew member cannot be detected within a first preset time period;

[0187] In response to the determination being yes, it is determined that the flight attendant has the visual field deviation abnormality.

[0188] In some optional aspects of this embodiment, the intermittent observation abnormal situation monitoring unit is configured as follows:

[0189] Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0190] Determining whether the crew monitoring video stream continuously contains closed-eye crew monitoring frame images within a second preset time period;

[0191] In response to determining yes, it is determined that the crew member has the intermittent lookout abnormality.

[0192] In some optional aspects of this embodiment, the shielding abnormality monitoring unit is configured as follows:

[0193] Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images;

[0194] determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the complete face of the crew member cannot be detected within a third preset time period;

[0195] In response to the determination being yes, it is determined that the front camera has the occlusion abnormality.

[0196] In some optional aspects of this embodiment, the pantograph abnormality monitoring module includes a structure abnormality monitoring unit, a spark abnormality monitoring unit and a foreign object intrusion abnormality monitoring unit.

[0197] In some optional aspects of this embodiment, the structural abnormality monitoring unit is configured as follows:

[0198] Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0199] Performing pantograph posture recognition on the pantograph monitoring frame image to determine a plurality of key points in the pantograph, wherein the types of the key points include two ends of the carbon slide plate, left and right bow angles of the pantograph, a bow head bracket and a balance bar;

[0200] Based on each of the key points, determine whether the pantograph has any structural abnormality, wherein the structural abnormality includes carbon slide plate abnormality and bow angle abnormality.

[0201] In some optional aspects of this embodiment, the spark abnormality monitoring unit is configured as follows:

[0202] Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0203] Based on a pre-trained spark recognition model, determining whether sparks exist in the pantograph monitoring frame image;

[0204] In response to determining that the spark exists, determining whether an area of ​​the spark is greater than or equal to a preset area;

[0205] In response to determining yes, determining that an abnormal spark is present in the pantograph;

[0206] Determining whether the pantograph monitoring video stream continuously contains a first preset number of frames of pantograph monitoring frame images including the abnormal spark within a fourth preset time period;

[0207] In response to determining yes, it is determined that the pantograph has the spark abnormality condition.

[0208] In some optional aspects of this embodiment, the foreign body intrusion abnormal situation monitoring unit is configured as follows:

[0209] Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images;

[0210] Based on a pre-trained foreign object recognition model, determining whether there is a foreign object in the pantograph monitoring frame image;

[0211] In response to the determination being yes, determining whether the pantograph monitoring video stream continuously contains at least a second preset number of frames including the pantograph monitoring frame image of the foreign object within a fifth preset time period;

[0212] In response to determining yes, it is determined that the pantograph has the foreign object intrusion abnormality.

[0213] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0214] An electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of a method for online video recognition of locomotives in the aforementioned embodiment.

[0215] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the steps of a method for online video recognition of a locomotive in the aforementioned embodiment.

[0216] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of a method for online video recognition of a locomotive in the aforementioned embodiment.

[0217] Fig.11 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0218] like Fig.11 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0219] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0220] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as a video online recognition method for locomotives.

[0221] For example, in some embodiments, a method for online video identification of a locomotive may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method for online video identification of a locomotive described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform a method for online video identification of a locomotive in any other suitable manner (e.g., by means of firmware).

[0222] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0223] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0224] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0225] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0226] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0227] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0228] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0229] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for online video recognition of locomotives, characterized in that: include: Receive the crew monitoring video stream sent by the front camera and the pantograph monitoring video stream sent by the pantograph camera; Based on the crew monitoring video stream, determining whether the crew has abnormal field of vision deviation and abnormal intermittent observation, and determining whether the front camera has abnormal occlusion; Based on the pantograph monitoring video stream, determining whether the pantograph has structural abnormalities, spark abnormalities, and foreign object intrusion abnormalities; In response to determining that at least one of the abnormal situation of field of view deviation, the abnormal situation of intermittent observation and the abnormal situation of obstruction exists, providing an online real-time reminder to the crew; In response to determining the existence of at least one of the structural abnormality, the spark abnormality and the foreign object intrusion abnormality, a fault sub-video is determined from the pantograph monitoring video stream and the fault sub-video is sent to the client.

2. The online identification method according to claim 1, characterized in that: The step of determining whether the flight attendant has an abnormal field of vision deviation based on the flight attendant monitoring video stream includes: Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images; Determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the front face of the crew member cannot be detected within a first preset time period; In response to the determination being yes, it is determined that the flight attendant has the visual field deviation abnormality.

3. The online identification method according to claim 1, characterized in that: The step of determining whether the crew has an abnormal situation of intermittent lookout based on the crew monitoring video stream includes: Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images; Determining whether the crew monitoring video stream continuously contains closed-eye crew monitoring frame images within a second preset time period; In response to determining yes, it is determined that the crew member has the intermittent lookout abnormality.

4. The online identification method according to claim 1, characterized in that: The step of determining whether the front camera is blocked abnormally based on the crew monitoring video comprises: Processing the crew monitoring video stream frame by frame to obtain a plurality of crew monitoring frame images; determining whether the crew monitoring video stream continuously contains crew monitoring frame images in which the complete face of the crew member cannot be detected within a third preset time period; In response to the determination being yes, it is determined that the front camera has the occlusion abnormality.

5. The online identification method according to claim 1, characterized in that: The step of determining whether the pantograph has a structural abnormality based on the pantograph monitoring video stream comprises: Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images; Performing pantograph posture recognition on the pantograph monitoring frame image to determine a plurality of key points in the pantograph, wherein the types of the key points include two ends of the carbon slide plate, left and right bow angles of the pantograph, a bow head bracket and a balance bar; Based on each of the key points, determine whether the pantograph has any structural abnormality, wherein the structural abnormality includes carbon slide plate abnormality and bow angle abnormality.

6. The online identification method according to claim 1, characterized in that: The step of determining whether the pantograph has an abnormal spark condition based on the pantograph monitoring video stream comprises: Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images; Based on a pre-trained spark recognition model, determining whether sparks exist in the pantograph monitoring frame image; In response to determining that the spark exists, determining whether an area of ​​the spark is greater than or equal to a preset area; In response to determining yes, determining that an abnormal spark is present in the pantograph; Determining whether the pantograph monitoring video stream continuously contains a first preset number of frames of pantograph monitoring frame images including the abnormal spark within a fourth preset time period; In response to determining yes, it is determined that the pantograph has the spark abnormality condition.

7. The online identification method according to claim 1, characterized in that: The step of determining whether there is an abnormal situation of foreign object intrusion in the pantograph based on the pantograph monitoring video stream comprises: Processing the pantograph monitoring video stream frame by frame to obtain a plurality of pantograph monitoring frame images; Based on a pre-trained foreign object recognition model, determining whether there is a foreign object in the pantograph monitoring frame image; In response to the determination being yes, determining whether the pantograph monitoring video stream continuously contains at least a second preset number of frames including the pantograph monitoring frame image of the foreign object within a fifth preset time period; In response to determining yes, it is determined that the pantograph has the foreign object intrusion abnormality.

8. A video online recognition device for locomotives, characterized in that: include: A video stream receiving module is configured to receive a crew monitoring video stream sent by a front camera and a pantograph monitoring video stream sent by a pantograph camera; The crew and front camera abnormality monitoring module is configured to determine whether the crew has abnormal vision deviation and abnormal intermittent observation based on the crew monitoring video stream, and determine whether the front camera has abnormal occlusion; A pantograph abnormality monitoring module is configured to determine whether the pantograph has structural abnormality, spark abnormality and foreign object intrusion abnormality based on the pantograph monitoring video stream; A crew reminder module, configured to, in response to determining that at least one of the visual field deviation abnormality, the intermittent observation abnormality and the occlusion abnormality exists, give the crew an online real-time reminder; The fault sub-video pushing module is configured to determine the fault sub-video from the pantograph monitoring video stream in response to determining the existence of at least one of the structural abnormality, spark abnormality and foreign object intrusion abnormality, and send the fault sub-video to the client.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for online video recognition of a locomotive as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for online video recognition of a locomotive according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method for online video recognition of a locomotive according to any one of claims 1 to 7 is implemented.

Images