Foreign matter judgment method and device for pantograph image and video

Abstract

The invention discloses a foreign matter judgment method and device for a pantograph image and a pantograph video. The foreign matter judgment method comprises the following steps that the pantograph image is acquired; positioning a pantograph region from the pantograph image; positioning semantic contours of a plurality of preset parts from the pantograph region; and judging whether foreign matters are attached to the pantograph in the pantograph image or not based on the semantic contours of a plurality of preset components. The foreign matter judgment method can detect whether the foreign matter is attached to the pantograph or not.

Description

Technical Field

[0001] The present invention relates to the field of rail transit technology, and in particular to a foreign body judgment method and device for pantograph images and videos. Background Art

[0002] All rail transit trains obtain electrical energy from the outside through the sliding contact between the train pantograph and the contact network. The contact relationship between the pantograph and the contact network is crucial to the safety and normal operation of the train. Due to the special contact relationship and structural characteristics between the pantograph and the contact network, it is not convenient to install test equipment on the pantograph or the contact network of the operating train. Moreover, once foreign objects are attached to the pantograph, it will affect the normal operation or safety of the operating train. In the existing technology, staff are required to use special attachment equipment to climb on the roof of the train for inspection, or the train needs to change the pantograph to maintain operation, which directly affects the punctual operation of the train.

[0003] Therefore, designing a device that can detect whether there is foreign matter attached to the pantograph becomes an urgent problem to be solved. Summary of the invention

[0004] In view of this, the main purpose of the present invention is to provide a foreign body judgment method and device for pantograph images and videos.

[0005] To achieve the above-mentioned purpose, the technical solution of the present invention is implemented as follows: a method for judging foreign objects in a pantograph image, comprising the following steps: acquiring a pantograph image; locating a pantograph area from the pantograph image; locating semantic contours of several preset components from the pantograph area; and judging whether a foreign object is attached to the pantograph in the pantograph image based on the semantic contours of the several preset components.

[0006] As an improvement of the embodiment of the present invention, the “locating the pantograph area from the pantograph image” specifically includes: locating the pantograph area from the pantograph image based on the YOLOV algorithm.

[0007] As an improvement of an embodiment of the present invention, the "locating semantic contours of several preset components from the pantograph area" specifically includes: locating semantic contours of several preset components from the pantograph area based on the DeepLab segmentation network.

[0008] As an improvement of an embodiment of the present invention, the "locating the semantic contours of several preset components from the pantograph area" specifically includes: locating the semantic contours of the carbon slide board, the semantic contours of the ram's horns and the semantic contours of the cross bar from the pantograph area.

[0009] As an improvement of an embodiment of the present invention, the “determining whether the pantograph in the pantograph image has foreign matter attached thereto based on the semantic contours of several preset components” specifically includes: performing contour search on the semantic contour of the carbon slide board to obtain several first contours; when the areas of at least two of the first contours exceed a first preset threshold, the carbon slide board in the pantograph image has foreign matter attached thereto; otherwise, the carbon slide board in the pantograph image has no foreign matter attached thereto; performing contour search on the semantic contour of the crossbar to obtain several second contours; when the areas of at least two of the second contours exceed a first preset threshold, the carbon slide board in the pantograph image has foreign matter attached thereto; otherwise, the carbon slide board in the pantograph image has no foreign matter attached thereto; performing contour search on the semantic contour of the crossbar to obtain several second contours; When the product exceeds the second preset threshold, there is foreign matter attached to the crossbar in the pantograph image, otherwise, there is no foreign matter attached to the crossbar in the pantograph image; the length and height of the semantic segmentation contour of the goat's horns are calculated, and when the length is < the third preset threshold, or the height is < the fourth preset threshold, there is foreign matter attached to the goat's horns in the pantograph image, otherwise, there is no foreign matter attached to the goat's horns in the pantograph image; when there is foreign matter attached to the carbon slide board in the pantograph image, or the crossbar or the horns, there is foreign matter attached to the pantograph in the pantograph image.

[0010] An embodiment of the present invention also provides a device for determining whether foreign objects are attached to a pantograph, comprising the following modules: an image acquisition module for acquiring a pantograph image; an area positioning module for locating a pantograph area from the pantograph image; a contour acquisition module for locating semantic contours of several preset components from the pantograph area; and a first processing module for determining whether foreign objects are attached to the pantograph in the pantograph image based on the semantic contours of several preset components.

[0011] As an improvement of the embodiment of the present invention, the region positioning module is further used to: locate the pantograph region from the pantograph image based on a YOLOV algorithm.

[0012] An embodiment of the present invention also provides a foreign object judgment method for a pantograph video, comprising the following steps: photographing a pantograph to be processed and obtaining a pantograph video, wherein the pantograph video is composed of Num1 pantograph images arranged in chronological order; based on the above-mentioned foreign object judgment method, the Num1 pantograph images are all judged and processed, and it is determined that foreign objects are attached to the pantographs in the Num2 pantograph images; wherein Num1 and Num2 are both natural numbers, and Num1≥Num2; when Num2 / Num1≥a preset threshold, it is determined that foreign objects are attached to the pantograph to be processed, wherein the preset threshold>0.

[0013] As an improvement to the embodiment of the present invention, the preset threshold value=0.6.

[0014] An embodiment of the present invention also provides a foreign object judgment device for pantograph video, comprising the following modules: a video acquisition module, used to shoot the pantograph to be processed and obtain the pantograph video, wherein the pantograph video is composed of Num1 pantograph images arranged in chronological order; a second processing module, used to judge and process the Num1 pantograph images based on the above-mentioned foreign object judgment method, and determine that the pantographs in the Num2 pantograph images have foreign objects attached; wherein Num1 and Num2 are both natural numbers, and Num1≥Num2; a judgment module, used to determine that the pantograph to be processed has foreign objects attached when Num2 / Num1≥preset threshold, wherein the preset threshold>0.

[0015] The foreign object judgment method and device for pantograph images and videos provided by the embodiments of the present invention have the following advantages: The embodiments of the present invention disclose a foreign object judgment method and device for pantograph images and videos, and the foreign object judgment method comprises the following steps: obtaining a pantograph image; locating a pantograph area from the pantograph image; locating the semantic contours of several preset components from the pantograph area; and judging whether a foreign object is attached to the pantograph in the pantograph image based on the semantic contours of several preset components. The foreign object judgment method can detect whether a foreign object is attached to the pantograph. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic flow chart of a method for determining foreign matter in a pantograph image in an embodiment;

[0017] Figure 2 and 3 is an example diagram of a pantograph image in an embodiment;

[0018] Figure 4 is a structural diagram of a pantograph in an embodiment;

[0019] Figure 5 and Figure 6 It is a schematic diagram of the foreign matter judgment method of the pantograph image in the embodiment. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below in conjunction with the embodiments shown in the accompanying drawings. However, the embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on the embodiments are all within the protection scope of the present invention.

[0021] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0022] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0023] Embodiment 1 of the present invention provides a foreign object judgment method for a pantograph image. Here, an execution device may be provided in a train or a railway management center, and the foreign object judgment method is executed by the execution device. The execution device may be an industrial computer, a PC (Personal Computer) machine or an MCU (Microcontroller Unit), etc.

[0024] like Figure 1 As shown, the following steps are included:

[0025] Step 101: Obtain the pantograph image; here, a camera may be provided on the top of the train, the camera is aimed at the pantograph, and the pantograph image including the pantograph is captured in real time. In practice, the camera will obtain a video stream including the pantograph, and the pantograph image may be a video image frame in the video stream. In the programming process, the inventors designed it as follows: first obtain the RTSP (Real Time Streaming Protocol) address of the camera, and use the videocapture function of opencv to read the video stream in the RTSP address. The function can automatically convert the video stream into an image matrix in RGB (R is Red, G is Green, and B is Blue) format, and thus obtain the pantograph image. Figure 2 A pantograph image is shown.

[0026] Step 102: locating a pantograph area from the pantograph image; Figure 3 The figure shows the final execution effect.

[0027] Step 103: locate the semantic outlines of several preset components from the pantograph area; here, Figure 5 A pantograph semantic outline diagram is shown.

[0028] Step 104: Based on the semantic contours of several preset components, determine whether the pantograph in the pantograph image has foreign matter attached thereto.

[0029] In the foreign object judgment method of the embodiment of the present invention, there is no need to collect a large amount of foreign object data. It also has a good detection effect when facing some foreign objects of unknown shapes and types. It is more robust and can significantly reduce the missed reporting of foreign objects, greatly ensuring the safe operation of the pantograph and safeguarding the normal operation of the EMU.

[0030] In this embodiment, the “locating the pantograph area from the pantograph image” specifically includes: locating the pantograph area from the pantograph image based on the YOLOV algorithm.

[0031] Here, the YOLOV algorithm is a detection algorithm. The YOLO algorithm redefines object detection as a regression problem. It applies a single convolutional neural network (CNN) to the entire image, divides the image into grids, and predicts the class probability and bounding box of each grid. When in use, the convolutional neural network needs to be trained. During training, the collected pantograph image is first annotated with data, and the rectangular box of the pantograph area is annotated. The convolutional neural network is trained based on the annotated image data combined with the annotation information. After that, the pantograph image is input into the convolutional neural network, and the coordinate information of the pantograph position is output, so as to locate the pantograph area.

[0032] In this embodiment, the "locating the semantic contours of several preset components from the pantograph area" specifically includes: locating the semantic contours of several preset components from the pantograph area based on the DeepLab segmentation network. Here, the DeepLab segmentation network is a method that combines a deep convolutional neural network and a probabilistic graph model. The versions of the DeepLab segmentation network can be: DeepLab V1, DeepLab V2, DeepLab V3, DeepLab V3+, etc.

[0033] When encoding, the DeepLab segmentation network first enters the image into a deep convolutional neural network, and extracts a low-level feature for subsequent decoding. The deep convolutional neural network includes four convolutional layers and an image pooling layer. The four convolutional layers are 1*1 convolutional layer, 3*3 convolutional layer with expansion rate of 6, 3*3 convolutional layer with expansion rate of 12, and 3*3 convolutional layer with expansion rate of 18. The features extracted by these four convolutional layers plus the features after the pooling layer are then combined to obtain five parts, which are called ASPP (Atrous Spatial Pyramid Pooling), which are deep features. A feature map is obtained by passing through a 1*1 convolutional layer. This 1*1 convolutional layer is for dimensionality reduction, so the encoding is completed. In the decoding part, the feature map obtained by encoding through a deep convolutional neural network and dimensionality reduction is up-sampled, and the feature map obtained by the previous encoding is concatenated with the feature map obtained by up-sampling to obtain a new feature map. Then, it passes through a 3*3 convolution layer and is up-sampled to obtain the predicted effect.

[0034] During training, it is necessary to first obtain several images containing pantographs, mark the above-mentioned several preset components in these images, and then train the DeepLab segmentation network.

[0035] In this embodiment, the "locating the semantic outlines of several preset components from the pantograph area" specifically includes: locating the semantic outlines of the carbon slide plate, the semantic outlines of the ram's horns and the semantic outlines of the crossbar from the pantograph area. Here, Figure 4 The carbon slide plate 1, the horn 2 and the cross bar 3 are shown. In the long-term work of the inventor, it is found that the carbon slide plate 1, the horn 2 and the cross bar 3 are most likely to be attached by foreign objects. Once attached, they will have a great impact on the normal operation of the train.

[0036] In this embodiment, “determining whether a foreign object is attached to the pantograph in the pantograph image based on the semantic contours of the plurality of preset components” specifically includes:

[0037] Performing contour search on the semantic contour of the carbon slide plate to obtain a plurality of first contours; when the areas of at least two of the first contours exceed a first preset threshold, foreign matter is attached to the carbon slide plate in the pantograph image; otherwise, no foreign matter is attached to the carbon slide plate in the pantograph image;

[0038] Performing contour search on the semantic contour of the crossbar to obtain a plurality of second contours; when the areas of at least two of the second contours exceed a second preset threshold, foreign matter is attached to the crossbar in the pantograph image; otherwise, foreign matter is not attached to the crossbar in the pantograph image;

[0039] Calculating the length and height of the semantic segmentation contour of the goat's horns, when the length is less than the third preset threshold, or the height is less than the fourth preset threshold, there are foreign objects attached to the goat's horns in the pantograph image; otherwise, there are no foreign objects attached to the goat's horns in the pantograph image;

[0040] When foreign matter is attached to the carbon sliding plate in the pantograph image, or foreign matter is attached to the crossbar, or foreign matter is attached to the horn, then foreign matter is attached to the pantograph in the pantograph image.

[0041] Here, when foreign matter is attached to the pantograph carbon slide area, the DeepLab segmentation network fails to learn the characteristics of the foreign matter, resulting in the failure to segment the area, which leads to the interruption of the contour of the pantograph area. First, the appearance contour of the carbon slide is obtained (such as Figure 6 Then, the findcontours function in opencv is used to search the segmented carbon slide semantic contour. When two or more contours with an area exceeding a certain threshold are found, it is considered that there is a foreign object in the current pantograph carbon slide area. The same method can be applied to the crossbar area.

[0042] Embodiment 2 of the present invention provides a device for determining whether foreign matter is attached to a pantograph, comprising the following modules: an image acquisition module for acquiring a pantograph image; an area positioning module for locating a pantograph area from the pantograph image; a contour acquisition module for locating semantic contours of several preset components from the pantograph area; and a first processing module for determining whether foreign matter is attached to the pantograph in the pantograph image based on the semantic contours of several preset components.

[0043] In this embodiment, the area positioning module is further used to: locate the pantograph area from the pantograph image based on the YOLOV algorithm.

[0044] Embodiment 3 of the present invention provides a foreign body judgment method for pantograph video, comprising the following steps:

[0045] Step 1: photograph the pantograph to be processed and obtain a pantograph video, wherein the pantograph video consists of Num1 pantograph images arranged in chronological order;

[0046] Step 2: Based on the foreign object judgment method in the first embodiment, Num1 pantograph images are judged and processed, and it is determined that foreign objects are attached to the pantographs in Num2 pantograph images; wherein Num1 and Num2 are both natural numbers, and Num1≥Num2;

[0047] Step 3: When Num2 / Num1≥preset threshold, it is determined that there is foreign matter attached to the pantograph to be processed, wherein the preset threshold>0. In practice, errors may occur, therefore, when Num2 / Num1≥preset threshold, it can be considered that there is foreign matter attached to the pantograph to be processed.

[0048] In this embodiment, the preset threshold value = 0.6.

[0049] Embodiment 4 of the present invention provides a foreign object judgment device for pantograph video, comprising the following modules: a video acquisition module, used to shoot the pantograph to be processed and obtain the pantograph video, wherein the pantograph video is composed of Num1 pantograph images arranged in chronological order; a second processing module, used to judge and process the Num1 pantograph images based on the foreign object judgment method in Embodiment 1, and determine that the pantographs in Num2 pantograph images have foreign objects attached; wherein Num1 and Num2 are both natural numbers, and Num1≥Num2; a judgment module, used to determine that the pantograph to be processed has foreign objects attached when Num2 / Num1≥preset threshold, wherein the preset threshold>0.

[0050] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0051] The present invention may be a system, a method and / or a computer program product. The computer program product may include a readable storage medium carrying computer readable program instructions for causing a processor to implement various aspects of the present invention.

[0052] The readable storage medium can be a tangible device that holds and stores the instructions used by the instruction execution device. The readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0053] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A foreign body judgment method for pantograph images, characterized in that: The following steps are involved: Obtain pantograph images; Locating a pantograph area from the pantograph image; Locating semantic outlines of a plurality of preset components from the pantograph area; Based on the semantic contours of several preset components, it is determined whether the pantograph in the pantograph image has foreign matter attached thereto.

2. The foreign body identification method according to claim 1, characterized in that: The “locating the pantograph area from the pantograph image” specifically includes: Based on the YOLOV algorithm, the pantograph area is located from the pantograph image.

3. The foreign body identification method according to claim 1, characterized in that: The “locating the semantic outlines of a plurality of preset components from the pantograph area” specifically includes: Based on the DeepLab segmentation network, the semantic contours of several preset components are located from the pantograph area.

4. The foreign body identification method according to claim 1, characterized in that: The “locating the semantic outlines of a plurality of preset components from the pantograph area” specifically includes: The semantic outline of the carbon slide plate, the semantic outline of the ram's horn and the semantic outline of the crossbar are located from the pantograph area.

5. The foreign body identification method according to claim 4, characterized in that: The “determining whether a foreign object is attached to the pantograph in the pantograph image based on the semantic profiles of a plurality of preset components” specifically includes: Performing contour search on the semantic contour of the carbon slide plate to obtain a plurality of first contours; when the areas of at least two of the first contours exceed a first preset threshold, foreign matter is attached to the carbon slide plate in the pantograph image; otherwise, no foreign matter is attached to the carbon slide plate in the pantograph image; Performing contour search on the semantic contour of the crossbar to obtain a plurality of second contours; when the areas of at least two of the second contours exceed a second preset threshold, foreign matter is attached to the crossbar in the pantograph image; otherwise, foreign matter is not attached to the crossbar in the pantograph image; Calculating the length and height of the semantic segmentation contour of the goat's horns, when the length is less than the third preset threshold, or the height is less than the fourth preset threshold, there are foreign objects attached to the goat's horns in the pantograph image; otherwise, there are no foreign objects attached to the goat's horns in the pantograph image; When foreign matter is attached to the carbon sliding plate in the pantograph image, or foreign matter is attached to the crossbar, or foreign matter is attached to the horn, then foreign matter is attached to the pantograph in the pantograph image.

6. A device for judging whether foreign matter is attached to a pantograph, characterized in that: Includes the following modules: An image acquisition module, used for acquiring a pantograph image; A region positioning module, used for locating the pantograph region from the pantograph image; A contour acquisition module, used to locate the semantic contours of several preset components from the pantograph area; The first processing module is used to determine whether there is foreign matter attached to the pantograph in the pantograph image based on the semantic contours of a plurality of preset components.

7. The judgment device according to claim 6, characterized in that: The area positioning module is also used to locate the pantograph area from the pantograph image based on the YOLOV algorithm.

8. A foreign body judgment method for pantograph video, characterized in that: The following steps are involved: Shooting the pantograph to be processed and obtaining a pantograph video, wherein the pantograph video is composed of Num1 pantograph images arranged in chronological order; Based on the foreign object judgment method according to any one of claims 1 to 5, Num1 pantograph images are all judged and processed, and it is determined that foreign objects are attached to the pantographs in Num2 pantograph images; wherein Num1 and Num2 are both natural numbers, and Num1≥Num2; When Num2 / Num1≥preset threshold, it is determined that foreign matter is attached to the pantograph to be processed, wherein the preset threshold>0.

9. The foreign body identification method according to claim 8, characterized in that: Default threshold = 0.

6.

10. A foreign body judgment device for pantograph video, characterized in that: Includes the following modules: A video acquisition module, used for shooting the pantograph to be processed and obtaining a pantograph video, wherein the pantograph video is composed of Num1 pantograph images arranged in chronological order; A second processing module, configured to perform judgment processing on Num1 pantograph images based on the foreign object judgment method according to any one of claims 1 to 5, and determine that foreign objects are attached to the pantographs in Num2 pantograph images; wherein Num1 and Num2 are both natural numbers, and Num1≥Num2; The judgment module is used to determine that there is foreign matter attached to the pantograph to be processed when Num2 / Num1≥preset threshold, wherein the preset threshold>0.

Images