A locomotive roof abnormality detection method and system based on change recognition and anti-shake

Abstract

The present invention discloses a locomotive roof anomaly detection method and system based on change recognition and anti-shake, belonging to the technical field of roof anomaly detection. Based on the method framework of traditional change detection, aiming at the effective utilization characteristics of multi-phase images, the idea of ​​twin branch network is introduced, and the change detection method of image contrast and the change analysis method of video time series multi-frame detection are embedded. The advantages of static analysis method and dynamic analysis method are combined in algorithm form, and the weighted distinction and adversarial enhancement attention mechanism are combined in the quality of image feature extraction, forming a structural paradigm and detection method framework of change detection network that can achieve high recognition rate under the condition of low real-time requirements. A camera layout that can completely cover the roof of the locomotive is designed to solve the problem of misidentification caused by jitter caused by low-cost camera equipment and convenient installation methods, and finally realize a roof anomaly detection system with a detection accuracy of more than 95%.

Description

Technical Field

[0001] The present invention relates to the technical field of roof anomaly detection, and in particular to a locomotive roof anomaly detection method and system based on change recognition and anti-shake. Background Art

[0002] At present, although my country's national railway standard maintenance yards are equipped with electric locomotive roof operation safety monitoring systems, they do not yet have the functions of automatic detection, identification, calibration, and warning of foreign objects on the roof. This may bring some safety hazards, such as tools and objects left behind by maintenance personnel after completing roof operations, and other foreign objects that may exist on the locomotive roof, which will seriously affect the safe operation of the train.

[0003] Physical signs and objects in nature are easily affected by shooting angles, light, deformation of attached objects, etc., thus affecting the detection accuracy. With the rapid development of artificial intelligence in recent years, methods based on deep learning can achieve accuracy and robustness in recognition effects that are beyond the reach of traditional methods. In addition, change detection is a major topic in the field of intelligent transportation. The current recognition methods and work done are far from being automatically applicable to different vertical scenarios of change detection. For example, how to align, splice, compare, and segment data collected by different types of shooting equipment, how to map, match, and superimpose target data at different phases, how to intelligently understand target data at different angles, how to make refined use of them, and how to cross-cover at low cost. These problems all require a lot of work to be studied and solved, so as to achieve the purpose of intelligent operation and maintenance of intelligent detection and intelligent integrated management.

[0004] Traditional change detection methods are derived from the field of remote sensing images. They are mainly based on comparison and classification methods. They use independent pixels or multi-pixel area blocks as detection units, perform pixel-by-pixel traversal analysis, and finally form a difference detection map. In recent years, with the support of artificial intelligence methods, change detection methods have also been greatly improved. The most obvious improvement is that the detection unit has changed from pixels to objects, which has brought two urgent problems to be solved: object recognition and object classification. The change detection method framework based on artificial intelligence methods solves the object recognition problem through a convolutional neural network that can effectively extract image features, and solves the object classification problem through a detection segmentation network that can input differential images. In the system of foreign object detection on the roof, since the detected object is unknown and unpredictable, it is different from the application in the field of remote sensing. It has the characteristics of salt and pepper effect and rich information caused by ground object pixels. The classification target of the change detection network for foreign object recognition only needs to be classified into two categories in the set area. The identification target needs to identify the location of the foreign object. In the change detection method paradigm, the changed location is the foreign object. The traditional encoding and decoding network structure cannot be adaptively sensitive to the features of the change class. D-SNet and D-WNet, which are based on the U-net concept, also use the full convolutional neural network as the main idea for feature fusion. Although the end-to-end network structure can ensure effective detection in one-stage training, the indiscriminate use of change features does not achieve the purpose of effective use of information, which means that the network structure used for change detection has room for further improvement in its adaptive sensitivity to change features. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a locomotive roof anomaly detection method and system based on change recognition and anti-shake.

[0006] The objective of the present invention is achieved through the following technical solutions: The first aspect of the present invention provides: a locomotive roof abnormality detection method based on change recognition and anti-shake, characterized in that: it includes the following steps:

[0007] S1: Camera layout and image calibration stage: Install the camera on the roof so that the camera image can cover the entire roof area, and then adjust the image to improve the image clarity and reduce the edge imaging distortion of the target area;

[0008] S2: Pre-configuration stage, configure the area to be detected and the detection sensitivity; including: demarcate the area to be detected; set a first preset time to limit the time when the expected foreign object appears, and exclude the foreign objects whose duration ts is less than or equal to the first preset time; set the first sensitivity to limit the expected confidence of the detected foreign objects, and exclude the foreign objects whose algorithm sensitivity α is less than or equal to the first sensitivity; set the video detection period tcd to limit the detection time period of the presence of foreign objects, and exclude the foreign objects detected outside the video detection period tcd; set a reference image for each camera and shoot it in real time, and detect all areas that have changed compared with the reference image in the change detection network;

[0009] S3: Foreign object detection stage. After the pre-configuration is completed, the deep learning image processing method is used to determine whether there are foreign objects on the roof after the inspection; if there are foreign objects, an alarm is triggered; the deep learning image processing method includes the following steps: video detection is performed, and the video detection takes n frames as the images to be tested, and the change detection algorithm of the dual-phase image is used as a detection unit to perform change detection within the video detection period tcd; the dual-phase image is the current frame image and the reference image, where n=tcd*25, that is, there are n frames in the detection pool, and continuous change detection of frame extraction is performed in the detection pool with a step size of 5, and finally all actual foreign objects detected in the video detection period tcd are output, the algorithm sensitivity α of the actual foreign objects is greater than the first sensitivity, the foreign object duration ts is greater than the first preset value, and the frame number i <ts*25。

[0010] Preferably, the change detection network is an end-to-end codec network, which uses a weight-shared twin encoder as a feature extraction module, uses a feature fusion module DEFM to implement a difference-enhanced attention mechanism, and uses a long short-term memory decoder LSTM; when the change detection algorithm of a dual-phase image is used to perform change detection within a video detection period tcd, the dual-phase images Image0 and Image1 are first input into the twin encoder to generate dual-phase feature maps Feature0 and Feature1, and the dual-phase feature maps are input into the feature fusion module DEFM to obtain a feature-enhanced difference map, and finally the long short-term memory decoder LSTM is used to decode and output a change detection map containing change confidence.

[0011] Preferably, the twin encoder has a total of 5 layers, the number and scale of convolutional units in the 5 different encoding paths are different, and the output channels after encoding are 64, 128, 256, 512, and 1024, respectively. The dual-phase feature map obtained by the shallow encoding path contains the detail information of the dual-phase image, and the dual-phase feature map obtained by the deep encoding path contains the global information of the dual-phase image; the shallow encoding path includes at least the first layer, and the non-shallow encoding path is the deep encoding path.

[0012] Preferably, in the S1: camera layout and picture adjustment stage, the camera is installed above the roof for double oblique side shooting. While completely covering the roof, redundant shooting layout is performed on the pantograph areas at the front and rear ends of the roof. The focal length is adjusted according to the established working distance of the camera so that the actual target size of the picture in each camera is within 1 / 4 of the length of the entire vehicle area; then the camera angle is adjusted according to the focal length of the camera, and a multi-screen redundant layout is performed for the roof edge area perpendicular to the single-sided camera line, and the picture position is expanded for the roof edge area parallel to the single-sided camera line.

[0013] Preferably, in the S2: pre-configuration stage, the area to be detected is configured by setting a single or multiple areas to be detected; the detection sensitivity is configured by configuring multiple levels of sensitivity and selecting the highest detection sensitivity.

[0014] Preferably, in the S3: foreign object detection stage, before video detection, the foreign objects left behind by the climbing operation are simulated on-site, and the simulated image data is transmitted to the algorithm server. Then, a mathematical model is used in combination with a deep learning image processing method to analyze the underlying image features and feature context information, and foreign objects are identified and calibrated to provide an analysis conclusion.

[0015] Preferably, foreign objects are calibrated into three categories: the first foreign object is slender in shape and has weak texture features, the second foreign object is evenly distributed in shape vertically and horizontally, and the third foreign object is large in size and has obvious texture features; after the video detection is completed, the calibrated foreign objects are judged using a deep learning image processing method and the name of the foreign object is output; for uncalibrated foreign objects, a deep learning image processing method is used for comparison, and the image to be tested is subjected to difference detection to identify the foreign objects.

[0016] Preferably, the first foreign objects include small hexagonal wrenches, screwdrivers, and small animal bodies; the second foreign objects include large screwdrivers, open-end wrenches and work gloves, pliers, tape measures, and electrical tape; and the third foreign objects include rags, plastic bags, buckets, and irregular objects.

[0017] Preferably, there are 8 cameras installed on two parallel sides obliquely above the roof, with 4 cameras installed on each side.

[0018] The second aspect of the present invention provides: a locomotive roof anomaly detection system based on change recognition and anti-shake, which is used for any of the above-mentioned locomotive roof anomaly detection methods based on change recognition and anti-shake, including: a camera, the shooting picture of the camera can include the complete roof; an image adjustment module, which can improve the image clarity and reduce the edge imaging distortion of the target object; a pre-configuration module, which can configure the area to be detected and the detection sensitivity; a foreign object detection module, which can use a deep learning image processing method to determine whether there is a foreign object on the roof after maintenance; if a foreign object is present, an alarm is triggered.

[0019] The beneficial effects of the present invention are:

[0020] 1) In the verification of the actual project of the National Railway Maintenance Company, the automatic identification, analysis and early warning of foreign objects such as common maintenance tools were achieved, with a detection coverage rate of ≥99% and an identification rate of ≥95%.

[0021] 2) It eliminates the potential safety hazards caused by foreign objects left on the roof of the national railway electric locomotive and reduces the safety issues of human operations.

[0022] 3) It realizes the integrated intelligent management of the roof operation safety monitoring system, saving about 10 minutes of manual verification time per time.

[0023] 4) A change detection method based on image contrast and multi-frame temporal analysis of video is designed, and a memory network is cascaded for decoding and multi-level thresholds are adaptively used to prevent false positives caused by light time-displacement, thereby enhancing the robustness of the system method.

[0024] 5) By using image comparison combined with video frame sequence analysis methods, the business process is improved without single image comparison, making full use of the spatial and temporal information of the video frame to improve the system detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart for foreign body detection;

[0026] Figure 2 This is the structure diagram of the change detection network;

[0027] Figure 3 This is a diagram of the multi-frame detection network architecture. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] The present invention mainly performs three parts of work under the large framework of change detection network, realizes a detection method of feature post-fusion dual-branch network of differential array images, that is, in order to better and effectively utilize differential images, improves a sequence-sequence change detection network; introduces weighted differentiation and adversarial enhancement attention mechanism, embeds attention feature fusion module in the change detection network to enhance the feature extraction and post-fusion ability of the change detection network, and obtains the location of the change area through high-level semantic interpretation of the image through effective segmentation; designs a change detection method of image contrast set video multi-frame time series analysis, and cascades memory network for decoding and multi-level thresholds to adaptively prevent false alarms caused by lightweight time-displacement, thereby enhancing the robustness of the system method.

[0030] The present invention arranges multiple industrial cameras on the roof, and the field of view can completely cover the roof of the electric locomotive. By comparing the images before and after the personnel climb the roof, the functions of automatic identification, calibration, and alarm of various objects and foreign objects left on the roof can be realized. The cameras on both sides are respectively installed at the left and right angles of 45° above the locomotive, and are used to collect images of components within the fixed field of view of the roof; the cameras have the functions of adaptive focusing and automatic light source compensation; different numbers of cameras can be configured according to needs to achieve the field of view covering the entire roof.

[0031] The roof abnormality automatic detection system is mainly composed of the application end, the processing end, and the image acquisition equipment. The application end is used for visual management and operation of business flows. The processing end uses a server that integrates data storage and algorithm deployment. The image acquisition uses an industrial camera to obtain high-quality images. It belongs to the category of 2D array visual perception tasks. The technical implementation route revolves around the perception task, starting with the imaging of the locomotive roof, and then performing configuration and change detection in sequence, and finally realizing the alarm linkage of foreign object detection, and strengthening the intelligent perception function of the scene. The specific process is as follows Figure 1 shown.

[0032] See also Figure 1-Figure 3 , the first aspect of the present invention provides: a locomotive roof anomaly detection method based on change recognition and anti-shake, comprising the following steps: S1: camera layout and image adjustment stage, a camera is installed above the roof so that the camera image can include the complete roof area, and then the image clarity is improved and the edge imaging distortion of the target area is reduced through image adjustment;

[0033] S2: Pre-configuration stage, configure the area to be detected and the detection sensitivity; including: demarcate the area to be detected; set a first preset time to limit the time when the expected foreign object appears, and exclude the foreign objects whose duration ts is less than or equal to the first preset time; set the first sensitivity to limit the expected confidence of the detected foreign objects, and exclude the foreign objects whose algorithm sensitivity α is less than or equal to the first sensitivity; set the video detection period tcd to limit the detection time period of the presence of foreign objects, and exclude the foreign objects detected outside the video detection period tcd; set a reference image for each camera (the reference image can be set and covered multiple times) and shoot it in real time, and detect all areas that have changed compared with the reference image in the change detection network;

[0034] S3: Foreign object detection stage. After the pre-configuration is completed, the deep learning image processing method is used to determine whether there are foreign objects on the roof after the inspection; if there are foreign objects, an alarm is triggered; the deep learning image processing method includes the following steps: video detection is performed, and the video detection takes n frames as the images to be tested, and the change detection algorithm of the dual-phase image is used as a detection unit to perform change detection within the video detection period tcd; the dual-phase image is the current frame image and the reference image, where n=tcd*25, that is, there are n frames in the detection pool, and continuous change detection of frame extraction is performed in the detection pool with a step size of 5, and finally all actual foreign objects detected in the video detection period tcd are output, the algorithm sensitivity α of the actual foreign objects is greater than the first sensitivity, the foreign object duration ts is greater than the first preset value, and the frame number i <ts*25。

[0035] Delimiting the area to be detected is to calibrate the area to be detected in the field of view of the camera equipment. The 8-camera layout described in the present invention covers different fields of view respectively, that is, it is necessary to delimit the area within the field of view of the 8 cameras. The foreign object duration ts limits the time when the expected foreign object appears. The foreign object detected within the first preset time is not considered to be the actual foreign object. The foreign object duration ts is in seconds and the setting range is 1~300. The algorithm sensitivity limits the expected confidence of the foreign object detection. The foreign object detected within the first sensitivity is not considered to be the actual foreign object. The algorithm sensitivity α has no actual unit and is a normalized probability index with a setting range of 0.1~1. The video detection period limits the detection time period of the presence of foreign objects. Foreign objects detected outside this time period are not considered to be the actual foreign objects detected this time. The tcd is in seconds and the setting range is 1~300.

[0036] In some embodiments, the change detection network is an end-to-end codec network, which uses a weight-shared twin encoder as a feature extraction module, uses a feature fusion module DEFM to implement a difference-enhanced attention mechanism, and uses a long short-term memory decoder LSTM; when the change detection algorithm of a dual-phase image is used to perform change detection within a video detection period tcd, the dual-phase images Image0 and Image1 are first input into the twin encoder to generate dual-phase feature maps Feature0 and Feature1, and the dual-phase feature maps are input into the feature fusion module DEFM to obtain a feature-enhanced difference map, and finally the long short-term memory decoder LSTM is used to decode and output a change detection map containing change confidence.

[0037] The present invention uses an end-to-end codec network, in which the encoder and decoder serve as the network backbone and are responsible for the key image feature extraction and image feature fusion tasks. Within the framework of the traditional codec network structure, a weight-sharing encoder is added, namely, as a feature extraction module; a difference-enhanced attention mechanism is added, namely, as a feature fusion module (DEFM); and a long short-term memory (LSTM) decoder is added. First, the dual-phase images Image0 and Image1 are input into the twin encoder to generate dual-phase feature maps Feature0 and Feature1. The dual-phase feature maps are input into the difference enhancement fusion module to obtain feature-enhanced difference maps. Finally, the perception layer is used for decoding to output a change detection map containing change confidence, thereby realizing a lightweight network structure for change detection, such as Figure 2 .

[0038] In some embodiments, the twin encoder has a total of 5 layers, and the number and scale of convolutional units in the 5 different encoding paths are different. The output channels after encoding are 64, 128, 256, 512, and 1024, respectively. The dual-phase feature map obtained by the shallow encoding path contains the detail information of the dual-phase image, and the dual-phase feature map obtained by the deep encoding path contains the global information of the dual-phase image; the shallow encoding path includes at least the first layer, and the non-shallow encoding path is the deep encoding path.

[0039] The main contents of change detection include three parts: obtaining difference maps after preprocessing the original data, generating difference feature maps after encoding the difference maps, and generating change detection maps by fusing and decoding the difference feature maps. The quality of the preprocessed images is guaranteed by the built-in illumination compensation and denoising algorithms of the camera equipment, which is determined by the configuration business process (such as Figure 1The configuration steps of the present invention are used for subsequent triggering, thereby obtaining the dual-phase image data information of different channel fields of the target area including the coordinates to be detected, the expected confidence (sensitivity) of the change area and the expected duration, and sending them to the network encoder to extract features. The encoder described in the present invention is a twin differential structure, which can learn the image features of different phases through the weight sharing of the network structure, and can perform differential operations on the difference features. The encoder has a total of 5 layers, and the number and scale of convolution units in 5 different encoding paths are different. The output channels after encoding are 64, 128, 256, 512, and 1024, respectively. The dual-phase feature map obtained by the shallow encoding path mainly contains the detail information of the dual-phase image, and the dual-phase feature map obtained by the deep path has the global information of the dual-phase image. This encoding module deepens the feature fusion of the dual-phase image through the fusion of shallow and deep information, and enhances the network's ability to extract difference information. The design of 5 layers is a network hierarchical structure determined by balancing network performance and resource consumption after a large number of tests.

[0040] The key task of change detection is to distinguish the areas of change and non-change in the image. The feature fusion used in the present invention uses DEFM, i.e., feature difference enhancement module, and introduces the attention mechanism to enhance the difference features. Under the guidance of network learning, this method can enhance the bi-phase difference features between changes and non-changes, and output the detection result map of whether the entire model has a change area after decoding. The above is an analysis of the change detection steps for a single frame. The encoding-decoding framework for multiple frames is as follows: Figure 3 As shown in the figure, the difference is that multiple frames have an additional memory function recognition at the decoding end. An LSTM network with memory function is used at the decoding end, which contains features of 256 dimensions. This can prevent misrecognition of changes caused by slight time-displacement of the image and realize the encoding and decoding processing of multi-frame sequence-sequence information.

[0041] In some embodiments, the S1: camera layout and image calibration stage, the camera is installed on the roof for double oblique shooting. While completely covering the roof, the pantograph area at the front and rear ends of the roof is redundantly shot and arranged, and the focal length is adjusted according to the established working distance of the camera, and the established working distance is determined by the installation position. After the position is determined, the actual area size corresponding to the imaging screen of each camera is adjusted to be stable at 1 / 4 of the length of the entire vehicle area; finally, the shooting angle is adjusted according to the established focal length of the camera. Multi-image redundant layout is performed on the edge area of ​​the roof perpendicular to the single-sided camera connection line, with the aim of performing detection backup on the area with more serious edge imaging distortion; the image position of the edge area of ​​the roof parallel to the single-sided camera connection line is expanded, with the aim of avoiding placing the area to be detected at the edge so that the detection effect is affected by imaging distortion. In the above, a high-definition and fully covered roof picture is obtained through camera layout and image calibration, while reducing the edge imaging distortion of the target area. Double oblique side shooting on the roof helps to avoid obstruction of areas where high inherent objects on the roof are located, thereby preventing foreign objects in the obscured areas from being missed. Redundant camera layout is performed for areas where the probability of foreign objects appearing and the impact is greater (such as the pantograph areas at both ends of the vehicle body).

[0042] In some embodiments, the S2: pre-configuration stage, the configuration of the area to be detected can set a single or multiple areas to be detected in a single picture. The area to be detected is an irregular quadrilateral determined by four corner points, and the single detection area is required to be larger than 30*30 pixels; the detection sensitivity configuration, the present invention can configure multiple levels of detection sensitivity, and high sensitivity corresponds to high detection rate. The present invention selects the highest detection sensitivity based on the analysis of the service characteristics implemented.

[0043] In some embodiments, the S3: foreign body detection stage, first set a background image of the roof without any residue as the reference image, obtain the current image as the image to be detected when the foreign body needs to be detected, and transmit the two image data to the algorithm server at the same time, and report the recognition results in a system linkage. The deployed detection algorithm analyzes the data features of the reference image and the image to be tested through a deep learning image processing method, selects the feature differences, performs feature fusion, and then sends them to the difference detection network, converts the difference detection problem into an image segmentation problem, outputs the changed area in the image, that is, identifies the foreign body and calibrates the position, and finally links the safety interlock system to report the recognition results.

[0044] In some embodiments, foreign bodies are classified into three categories according to their size: the first foreign body is slender and has weak texture features, the second foreign body is evenly distributed in the vertical and horizontal directions, and the third foreign body is large in size and has obvious texture features; for various foreign bodies on the roof that are large in size or have normalized aspect ratios and obvious texture features, directly using the trained difference detection network can achieve a recognition rate of more than 95% to achieve the purpose of identifying the second foreign body and the third foreign body. For various foreign bodies on the roof that are small in size or have non-normalized aspect ratios and weak texture features, it is necessary to collect image pairs and develop specific tasks. By collecting training sets of image pairs, the detection performance of the difference detection network is improved to achieve the purpose of identifying the first foreign body. As mentioned above, the development process of the foreign body detection algorithm in this paper is divided into two stages. In the first stage, a large-scale foreign body data training is realized, which can meet 85%~90% of the detection scenes; in the second stage, specific development is carried out for individual data sets and tasks, and supervised training is performed using image pairing training sets to improve detection performance, which is used for scene detection with a large probability left after the first stage, and finally the detection rate of the entire system can reach more than 95%.

[0045] In some embodiments, the first foreign matter includes a small hexagonal wrench, a screwdriver, and a small animal body; the second foreign matter includes a large screwdriver, an open-end wrench, work gloves, pliers, a tape measure, and electrical tape; and the third foreign matter includes a rag, a plastic bag, a bucket, and an irregular object. The main types of foreign matter detected by the present invention are divided into three parts according to their size: the first part is a small hexagonal wrench, a screwdriver, a small animal body, etc., which are slender in shape and have weak texture features; the second part is a large screwdriver, an open-end wrench, work gloves, pliers, a tape measure, electrical tape, etc., and the shapes of the foreign matter are evenly distributed horizontally and vertically; the third part is an irregular object such as a rag, a plastic bag, and a bucket, which are large in size and have obvious texture features.

[0046] In some embodiments, there are 8 cameras installed at parallel positions on both sides of the upper part of the roof for oblique overhead shooting. 4 cameras are allocated on each side of the upper part of the roof to ensure that the field of view of the 8 cameras can fully cover the entire roof area of ​​the vehicle after splicing.

[0047] The second aspect of the present invention provides: a locomotive roof anomaly detection system based on change recognition and anti-shake, which is used to implement any of the above-mentioned locomotive roof anomaly detection methods based on change recognition and anti-shake, including: a camera, the shooting picture of the camera can include the complete roof; an image adjustment module, which can improve the image clarity and reduce the edge imaging distortion of the target object; a pre-configuration module, which can configure the area to be detected and the detection sensitivity; a foreign object detection module, which can use a deep learning image processing method to determine whether there is a foreign object on the roof after maintenance; if a foreign object is present, an alarm is triggered and the location of the foreign object is identified.

[0048] The encoder is used as the backbone of the change detection network to extract the difference information of the dual-phase images. The number and scale of the convolution units used in this paper are different, which can increase the receptive field of feature extraction and realize the fusion of multi-scale features. The improved network structure fully transforms the information of each scale to realize the fusion of multi-scale features, thereby replacing the single convolution stacking structure of the traditional network. The improved image quality is sufficient and accurate, and the quality of image feature extraction plays a decisive role in the final segmentation of the model, so that the detection of adaptive change areas can be achieved.

[0049] After the encoder extracts the image features at different times, it performs multi-level and multi-scale fusion. When performing feature fusion, the expressiveness of high-level features will affect the quality of the fused features, so it is necessary to introduce an attention mechanism. The fused features are further enhanced by the attention module. Without introducing redundant parameters, the purpose of enhancing the information weight of the changed area can be achieved, and a multi-scale and multi-level difference detection method for change detection is realized. This method can increase the weight of the target feature in the receptive field, so that the network can better focus on important areas and important features of the image, and can selectively focus on the features of the changed pixels from a large number of features, extract more representative information, and improve detection accuracy.

[0050] Due to the constraint of low cost, device jitter is a key problem that needs to be solved. The present invention has made technical breakthroughs in algorithm form and processing flow respectively. It uses image comparison combined with video frame sequence analysis method to improve the business process without single image comparison, and makes full use of the spatial and temporal information of the video frame to improve the system detection accuracy.

[0051] The structure of the sequence-sequence deep learning network was improved, and the effective fusion and utilization of differential features were strengthened on the basis of single image comparison. A lightweight dual-phase input sequence-sequence change detection network was realized to enhance the weight of the information in the change area, and a multi-scale and multi-level difference detection method for change detection was realized to improve the system recognition rate. In multi-frame detection, a long short-term memory network was added to memorize and decode the detection image after feature fusion, avoiding the misidentification caused by the lightweight time-displacement of the image in the detection principle, and improving the robustness of the system.

[0052] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A locomotive roof anomaly detection method based on change recognition and anti-shake, characterized in that: The following steps are involved: S1: Camera layout and image calibration stage: Install the camera on the roof so that the camera image can cover the entire roof area, and then adjust the image to improve the image clarity and reduce the edge imaging distortion of the target area; S2: Pre-configuration stage, configure the area to be detected and the detection sensitivity; including: demarcate the area to be detected; set a first preset time to limit the time when the expected foreign object appears, and exclude the foreign objects whose duration ts is less than or equal to the first preset time; set the first sensitivity to limit the expected confidence of the detected foreign objects, and exclude the foreign objects whose algorithm sensitivity α is less than or equal to the first sensitivity; set the video detection period tcd to limit the detection time period of the presence of foreign objects, and exclude the foreign objects detected outside the video detection period tcd; set a reference image for each camera and shoot it in real time, and detect all areas that have changed compared with the reference image in the change detection network; S3: Foreign object detection stage. After the pre-configuration is completed, a deep learning image processing method is used to determine whether there are foreign objects on the roof after inspection. If there are foreign objects, an alarm is triggered. The deep learning image processing method includes the following steps: video detection is performed, and the video detection takes n frames as the images to be tested, and the change detection algorithm of the dual-phase image is used as a detection unit to perform change detection within the video detection period tcd; the dual-phase image is the current frame image and the reference image, where n=tcd*25, that is, there are n frames in the detection pool, and continuous change detection of frame extraction is performed in the detection pool with a step size of 5. Finally, all actual foreign objects detected in the video detection period tcd are output, and the algorithm sensitivity α of the actual foreign objects is greater than the first sensitivity, the foreign object duration ts is greater than the first preset value, and the frame number i <ts*25; The change detection network is an end-to-end codec network, which uses a weight-shared twin encoder as a feature extraction module, uses a feature fusion module DEFM to implement a difference-enhanced attention mechanism, and uses a long short-term memory decoder LSTM; when a change detection algorithm for a dual-phase image is used to perform change detection within a video detection period tcd, the dual-phase images Image0 and Image1 are first input into the twin encoder to generate dual-phase feature maps Feature0 and Feature1, and the dual-phase feature maps are input into the feature fusion module DEFM to obtain a feature-enhanced difference map, and finally the long short-term memory decoder LSTM is used to decode and output a change detection map containing change confidence; In the S1: camera layout and image adjustment stage, the camera is installed on the roof for double oblique shooting. While completely covering the roof, redundant shooting layout is performed on the pantograph areas at the front and rear ends of the roof. The focal length is adjusted according to the established working distance of the camera so that the actual target size of the image in each camera is within 1 / 4 of the length of the entire vehicle area; then the camera angle is adjusted according to the focal length of the camera, and a multi-image redundant layout is performed on the roof edge area perpendicular to the single-side camera connection line, and the image position is expanded for the roof edge area parallel to the single-side camera connection line; The S2: pre-configuration stage, the to-be-detected area configuration, setting a single or multiple to-be-detected areas; the detection sensitivity configuration, configuring multiple levels of sensitivity, selecting the highest detection sensitivity.

2. The locomotive roof abnormality detection method based on change recognition and anti-shake according to claim 1, characterized in that: The twin encoder has a total of 5 layers. The number and scale of convolutional units in the 5 different encoding paths are different. The output channels after encoding are 64, 128, 256, 512, and 1024, respectively. The dual-phase feature map obtained by the shallow encoding path contains the detail information of the dual-phase image, and the dual-phase feature map obtained by the deep encoding path contains the global information of the dual-phase image; the shallow encoding path includes at least the first layer, and the non-shallow encoding path is the deep encoding path.

3. The locomotive roof abnormality detection method based on change recognition and anti-shake according to claim 1, characterized in that: In the S3: foreign object detection stage, before video detection, the foreign objects left behind by the climbing operation are simulated on site, and the simulated image data is transmitted to the algorithm server. Then, a mathematical model is used in combination with a deep learning image processing method to analyze the underlying image features and feature context information, and the foreign objects are identified and calibrated to provide an analysis conclusion.

4. The locomotive roof abnormality detection method based on change recognition and anti-shake according to claim 3 is characterized in that: Foreign objects are calibrated into three categories: the first type is slender in shape with weak texture features, the second type is evenly distributed in vertical and horizontal directions, and the third type is large in size with obvious texture features. After the video detection is completed, the calibrated foreign objects are judged using deep learning image processing methods and the names of the foreign objects are output. For uncalibrated foreign objects, deep learning image processing methods are used for comparison, and the images to be tested are subjected to difference detection to identify the foreign objects.

5. The locomotive roof abnormality detection method based on change recognition and anti-shake according to claim 4 is characterized in that: The first foreign objects include small hexagonal wrenches, screwdrivers, and small animal bodies; the second foreign objects include large screwdrivers, open-end wrenches and work gloves, pliers, tape measures, and electrical tape; and the third foreign objects include rags, plastic bags, buckets, and irregular objects.

6. The locomotive roof abnormality detection method based on change recognition and anti-shake according to any one of claims 1 to 5, characterized in that: There are 8 cameras installed on two parallel sides above the roof, with 4 cameras installed on each side.

7. A locomotive roof anomaly detection system based on change recognition and anti-shake, characterized in that: A locomotive roof anomaly detection method based on change recognition and anti-shake as described in any one of claims 1 to 6 includes: a camera, the shooting picture of the camera can include the complete roof; an image adjustment module, which can improve the image clarity and reduce the edge imaging distortion of the target object; a pre-configuration module, which can configure the area to be detected and the detection sensitivity; a foreign object detection module, which can use a deep learning image processing method to determine whether there is a foreign object on the roof after maintenance; if a foreign object is present, an alarm is triggered.

Images