Video image content analysis method for railway line dangerous situation

Abstract

The invention relates to the field of image recognition, and provides a railway line dangerous situation-oriented video image content analysis method, which comprises the following steps of: receiving an alarm message through a cloud server, and obtaining an alarm image according to the alarm message; detecting the alarm image to obtain a cloud detection result; comparing the cloud detection result with the edge detection result, and determining a comparison result; the side end detection result and the alarm image are obtained by detecting a video stream of the railway line through side end equipment, and the alarm image is an image with an abnormal target; and summarizing a plurality of comparison results for weighting processing to obtain a final alarm result. According to the invention, the defect of low accuracy of railway line dangerous situation detection in the prior art is overcome, and efficient and accurate railway line dangerous situation video image content analysis is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a video image content analysis method for railway line danger situations. Background Art

[0002] With the rapid development of the global railway transportation network, especially the rapid expansion of China's high-speed railway network, the railway operating mileage has continued to increase, and the train operation density has increased significantly. High-speed rail has become an important part of modern transportation with its high speed and convenience. However, the safety issues of high-speed rail have also become increasingly prominent. High-speed rail lines pass through complex geological environments and changeable meteorological conditions, facing a variety of potential dangers such as rock collapse, mudslides, slope collapse, and people / large animals invading railway limits, all of which pose a serious threat to the safe operation of high-speed rail. Therefore, how to effectively monitor and prevent these dangers and ensure the safety and reliability of high-speed rail operation has become an urgent problem that railway management departments and technical R&D personnel need to solve.

[0003] For monitoring railway line danger, existing technologies mainly include video-based detection methods, radar intelligent recognition technology, and fiber optic sensor recognition technology. Although the above technologies have played a certain role in railway danger monitoring, single algorithm detection is prone to inaccurate detection results. Summary of the invention

[0004] The present invention provides a method for analyzing the content of railway line dangerous conditions video images, which solves the defect of low accuracy in detecting railway line dangerous conditions in the prior art and realizes efficient and accurate railway line dangerous conditions video image content analysis.

[0005] The present invention provides a method for analyzing video image content for railway line danger, comprising the following steps: Receiving an alarm message through a cloud server, and acquiring an alarm image according to the alarm message; Detecting the alarm image to obtain a cloud detection result; The cloud detection result is compared with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line by the edge device, and the alarm image is an image with abnormal targets; Multiple comparison results are summarized and weighted to obtain the final alarm result.

[0006] According to a video image content analysis method for railway line danger provided by the present invention, the video stream of the railway line is detected by an edge device, specifically comprising: performing a differential operation on the current frame image in the video stream and the latest background image through an established background difference model to obtain a differential image; comparing the pixel value of each pixel point in the differential image with a preset threshold, binarizing the differential image according to the comparison result, and performing morphological processing on the binarized differential image to obtain a first target detection result; performing target detection on the frame image in the video stream through a trained convolutional neural network target detection model to obtain a second target detection result; the trained convolutional neural network target detection model is obtained by training with a training set, and the training set is a railway danger sample data set marked with a foreign object category; the first target detection result and the second target detection result are merged to determine the edge detection result, and an alarm image with an abnormal target is determined according to the edge detection result.

[0007] According to a video image content analysis method for railway line danger provided by the present invention, the alarm image is detected to obtain a cloud detection result, which specifically includes: preprocessing the alarm image; dividing the preprocessed image into a plurality of independent blocks, and determining the corresponding pixel matrix according to each independent block; performing multi-stage processing on each pixel matrix based on the SwinTransformer network to obtain feature maps of different stages; fusing the feature maps of different stages and inputting them into a detection head for target detection, and outputting a category probability map and a bounding box offset; generating a predicted bounding box according to the category probability map and the bounding box offset; and obtaining the cloud detection result after processing the predicted bounding box through non-maximum suppression.

[0008] According to a video image content analysis method for railway line danger situations provided by the present invention, the cloud detection result is compared with the edge detection result to determine the comparison result, which specifically includes: comparing the cloud detection result with the edge detection result based on a trained contrast learning model to determine the comparison result; the trained contrast learning model is trained according to a training set; the training set is positive samples with the same category label and negative samples with different category labels selected from a railway line danger sample library.

[0009] According to a video image content analysis method for railway line crises provided by the present invention, a convolutional neural network target detection model is trained according to a training set, specifically comprising: obtaining a railway crisis sample data set containing various types of crisis targets; annotating each image in the railway crisis sample data set to obtain a training set; inputting the training set into a convolutional neural network target detection model for training to obtain a predicted target label; calculating a loss function according to the predicted target label and the true target label, and updating the model weight parameters according to the back propagation of the loss function; continuously iterating and updating the model weight parameters, and when the number of training times reaches a stopping condition, outputting the trained convolutional neural network target detection model.

[0010] According to a video image content analysis method for railway line danger provided by the present invention, a contrast learning model is trained according to a training set, specifically comprising: selecting images from a constructed railway line danger sample library, using images with the same category label as positive sample pairs, and images with different category labels as negative sample pairs, wherein the positive sample pairs and the negative sample pairs constitute positive and negative sample pairs; performing data enhancement on the positive and negative sample pairs; encoding the positive and negative sample pairs after data enhancement, and extracting deep features of the input images; performing dimensionality reduction and filtering on the extracted deep feature dimensions, and outputting a multidimensional feature vector; calculating a contrast loss function according to the multidimensional feature vector, and updating the parameters of the contrast learning model according to the calculation results; and continuing to iteratively update the contrast learning model until an iteration stop condition is met.

[0011] According to a video image content analysis method for railway line danger situations provided by the present invention, an alarm message is received through a cloud server, and an alarm image is obtained according to the alarm message, which specifically includes: a cloud server message queue receives alarm information from an edge device, and the alarm information includes a uniform resource locator of the alarm image; and a corresponding alarm image is downloaded according to the uniform resource locator of the alarm image.

[0012] The present invention also provides a video image content analysis device for railway line danger, comprising the following modules: An image acquisition module is used to receive an alarm message through a cloud server and acquire an alarm image according to the alarm message; A cloud detection module, used to detect the alarm image and obtain a cloud detection result; The cloud-edge comparison module is used to compare the cloud detection result with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line by the edge device, and the alarm image is an image with abnormal targets; The alarm result determination module summarizes multiple comparison results and performs weighted processing to obtain the final alarm result.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video image content analysis method for railway line danger as described in any one of the above methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the video image content analysis method for railway line danger as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned video image content analysis methods for railway line danger situations.

[0016] The present invention provides a video image content analysis method for railway line dangers, including the following beneficial effects: The present invention combines the dual detection mechanism of the cloud and the edge. By detecting the alarm image in the cloud and comparing it with the edge detection result, the probability of false alarms and missed alarms can be significantly reduced, thereby improving the high accuracy of danger detection. By distributing the video stream to the edge device and transmitting only the key video images and information to the cloud, the bandwidth pressure on the central cloud is reduced. At the same time, the high computing power resources of the cloud are used in conjunction with the high-complexity deep learning network to enhance the detection accuracy. By combining the advantages of multiple models in the cloud and combining the detection of multiple models, the risk of overfitting can be reduced, the generalization ability can be improved, the model bias can be reduced, the prediction stability can be improved, the model interpretability can be enhanced, and the adaptation to different data distributions can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is the overall structure diagram of the cloud-edge collaboration provided by the present invention.

[0019] Figure 2 It is one of the flow charts of railway line emergency video image content analysis based on cloud-edge integration provided by the present invention.

[0020] Figure 3 This is a structural diagram of the edge fusion perception algorithm provided by the present invention.

[0021] Figure 4 It is a flow chart of the background difference algorithm provided by the present invention.

[0022] Figure 5 This is a schematic diagram of edge alarm message transmission provided by the present invention.

[0023] Figure 6 This is the second flowchart of the video image content analysis method for railway line danger provided by the present invention.

[0024] Figure 7 This is a cloud detection flow chart provided by the present invention.

[0025] Figure 8 This is a diagram of the multi-way algorithm voting mechanism provided by the present invention.

[0026] Fig. 9 It is a structural schematic diagram of a video image content analysis device for railway line danger provided by the present invention.

[0027] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] As the world's longest operating mileage, China's high-speed railway network has the highest operating speed, the highest train density, complex geological conditions, and diverse meteorological conditions. With the rapid expansion of the high-speed railway network, the continuous expansion of transportation scale, and the iterative upgrade of technical equipment, the situation facing high-speed rail safety work is becoming increasingly severe and complex. Landslides, mudslides, slope collapses, and the intrusion of people / large animals into railway limits pose a serious threat to high-speed rail driving safety. In order to improve the ability to respond to risks, it is necessary to strengthen high-speed rail safety protection measures in many aspects and establish a high-speed rail safety protection system that combines technical defense, physical defense, and human defense. Manual video inspections have problems such as high labor intensity, long intervals, and uncontrollable concentration. It is urgent to study cost-effective and applicable technical equipment to assist manual video inspections, improve work efficiency, and enhance the ability to discover and deal with dangerous situations.

[0030] In recent years, the combination and application of image processing technology and computer technology has been widely studied, and researchers have integrated intelligent detection technology into the existing railway video comprehensive monitoring system. At present, there are two main types of railway emergency detection measures: contact detection and non-contact detection. Common technologies include intelligent video analysis, laser radar, vibration fiber and other categories.

[0031] Railway monitoring scenarios are complex and have unique perception requirements. Using monitoring resources at high-risk locations such as tunnel entrances, through intelligent analysis, we can automatically identify dangerous line conditions that intrude on railway limits, such as rock collapse, debris flow, slope collapse, and intrusion of people / large animals, and trigger alarms. There are the following difficulties: First, in the outdoor environment, the image information is unstable due to seasonal and day-night light and shadow changes, weather conditions such as rain and snow, as well as surrounding interference (car lights, insects, birds, tree branch shadows, etc.), image jitter, and image blur; second, there is insufficient sample data for normal and abnormal behaviors / events of various dangerous situations, and effective sample data suitable for railway line intrusion target detection is scarce; third, the equipment needs to be economical to use, stable in operation, and meet real-time monitoring requirements. Existing monitoring methods have the following shortcomings in the face of unique railway line danger monitoring scenarios.

[0032] 1) Video-based detection methods: Traditional video monitoring methods are difficult to deal with noise interference, are greatly affected by picture fluctuations, and are prone to false alarms.

[0033] 2) The economic cost of radar intelligent identification technology is high, and the maintenance cycle and cost are long. At the same time, the attenuation of millimeter-wave radar is large in high-humidity environments such as rain, fog, and snow. The influence of high-power devices and insertion loss reduces the detection distance of millimeter-wave radar. It has poor penetration ability in bush scenes and is easily affected by vegetation. LiDAR is also greatly affected by weather and atmosphere when working. In severe weather such as heavy rain, thick smoke, and dense fog, the attenuation increases sharply and the propagation distance is greatly affected.

[0034] 3) Existing fiber-optic distributed vibration monitoring systems have introduced deep learning classification tasks to effectively identify vibration events, thereby reducing the false alarm rate. However, the cost of fiber-optic sensors is relatively high, and the fiber-optic network may involve multiple sensors and connection points. The fault diagnosis and maintenance of the system are relatively complex. Fiber-optic sensors are also very sensitive to environmental conditions. In extreme environmental conditions such as railway lines, their service life will be seriously affected.

[0035] The present invention adopts a cloud-edge combined method to monitor railway line danger conditions, which can solve the problems of low accuracy caused by edge computing resource limitations and the inability to detect in real time due to limited network bandwidth.

[0036] From the perspective of computing resource limitations, high-precision target detection systems usually rely on powerful computing resources. Edge devices usually have limited computing power and it is difficult to use complex deep learning models. This leads to the limitation of computing resources becoming a major technical challenge for real-time target detection on edge devices. The cloud-edge combined target detection method effectively utilizes the computing power of edge devices by transferring part of the computing tasks from the cloud to edge devices. The edge device completes preliminary detection using a low-computing-power-requirement algorithm, and then transmits the processed data to the cloud with high-computing-power resources for detailed analysis and identification using a high-precision complex detection network.

[0037] From the perspective of network bandwidth limitations, traditional target detection systems usually need to transmit a large amount of image or video data to the cloud for processing. However, the network bandwidth is limited, especially in mobile network environments, the speed and stability of data transmission cannot be guaranteed. This leads to the limitation of network bandwidth becoming a major technical challenge in real-time target detection applications. The cloud-edge combined target detection method performs pre-detection on edge devices, which can greatly reduce the amount of data that needs to be transmitted to the cloud. The edge device only uploads alarm pictures and alarm information, reducing the scale of data transmission, thereby reducing the demand for network bandwidth.

[0038] Combine the following Figure 1-Figure 10 Embodiments of the present invention are described in detail.

[0039] The cloud-edge-based railway line emergency video image content analysis provided by the present invention deploys unmanned intelligent monitoring equipment in key areas to autonomously conduct intelligent monitoring and detect abnormal intrusion behaviors. Figure 1 The figure shows the overall structure of cloud-edge collaboration. The railway line emergency video image content analysis based on cloud-edge integration includes two parts: the cloud and the edge. The edge uses an intrusion target recognition algorithm that integrates background difference and deep learning, and achieves preliminary detection through two algorithms: background difference and convolutional neural network (CNN network). The cloud includes the Swin Transformer target detection method, contrast learning method, and cloud voting mechanism to complete the secondary review of the edge alarm image.

[0040] According to the requirements of mature technology, simple methods, reliable equipment, reasonable logic, stable system, and easy maintenance, combined with the actual conditions of railway line video surveillance cameras, network transmission bandwidth, and natural environment of monitoring points, through front-end equipment acquisition, edge computing analysis, and cloud-edge collaborative detection technology research, a railway line hazard monitoring intelligent improvement plan based on video image content analysis is formed. The collected video stream is distributed to the edge device to achieve rapid detection of the hazard. Only key video images and information are transmitted to the cloud, which can reduce the bandwidth pressure on the central cloud. The cloud-based railway line hazard identification algorithm filters out false alarms. The cloud-edge collaborative detection mechanism fully integrates the detection capabilities of both ends to achieve rapid identification of railway line intrusion targets, improving the real-time performance of intelligent monitoring and the overall detection effect.

[0041] The overall process of railway line emergency video image content analysis based on cloud-edge integration is as follows: Figure 2 As shown. The goal of edge reasoning is to ensure latency and throughput, and to achieve rapid detection and alarm of various targets in multi-channel concurrent detection. The edge detection algorithm uses an intelligent algorithm box to quickly detect anomalies, generate preliminary detection results, and then send the alarm image to the cloud for secondary review. The transmission method uses multiple frames before and after the alarm image is packaged, and the alarm information uses message queues and other related technologies to transmit data. The cloud algorithm uses a server to perform secondary identification of the warning image sent by the edge in the cloud to obtain the final result, and then the platform user manually reviews the alarm. During the user review process, the false alarm image record is saved, and the false alarm data is fed back to the cloud. When the false alarm data reaches a certain scale, the feedback learning mechanism is started, and the false alarm data is used again for cloud model training to reduce the false alarm rate, and the cloud algorithm update is completed at the same time.

[0042] The above scheme has the following advantages: 1) Railway line hazard monitoring method based on video image content analysis. Camera technology is very mature and widely used in various scenarios, such as security monitoring, traffic management, industrial automation, etc. It is less affected by weather and environment. Rain, fog and dust have little impact on its stable operation. Maintenance and operation are relatively simple, no special skills or equipment are required, and the operating cost is low and stable.

[0043] 2) The cloud-edge combined monitoring method can achieve real-time and high accuracy of intelligent monitoring. Video streams are distributed to edge devices, and only key video images and information are transmitted to the cloud, reducing the bandwidth pressure on the central cloud. At the same time, the high computing power resources of the cloud are used in conjunction with highly complex deep learning networks to enhance detection accuracy.

[0044] 3) The core idea of ​​the background difference algorithm used at the edge is to compare the current frame with the background difference model to detect the foreground object and reduce the impact of insufficient sample data. By learning the background difference model, the algorithm can accurately distinguish the foreground object from the background when it appears. Even if the sample data is insufficient, the algorithm can still identify and separate the foreground object through the established background difference model.

[0045] 4) The core of the Transformer-based object detection algorithm used in the cloud is the self-attention mechanism, which can capture the long-distance dependencies between different areas in the image. In object detection, this means that the model can better understand the overall structure and contextual information of the target object. This global modeling capability helps improve the model's ability to handle complex scenes and occlusions, thereby improving detection accuracy.

[0046] 5) The contrastive learning method used in the cloud learns feature representation by comparing the similarities between image pairs, and then distinguishes between positive sample pairs (similar images) and negative sample pairs (dissimilar images), making the model more robust to image transformation and perturbation. This means that even if the detection samples are disturbed by lighting, occlusion, etc., the model can still accurately determine whether they are the same sample.

[0047] 6) The cloud-based voting mechanism combines the advantages of multiple models and the predictions of multiple models to reduce the risk of overfitting, improve generalization ability, reduce model bias, improve prediction stability, enhance model interpretability, and adapt to different data distributions.

[0048] Before using each model in the embodiment of the present invention, it is necessary to build a data sample library. A large-scale, high-quality annotated sample library is one of the necessary conditions for the success of the railway line danger target detection algorithm. Although there are some public sample sets about intrusion, most of the images in the sample sets do not contain targets, and the railway operating environment is very special. The public sample set is not suitable for direct use as target detection algorithm training data. It is necessary to build a training sample library for railway scenes to improve the algorithm recognition effect.

[0049] The sample library shall collect samples with a camera resolution of no less than 1080P, a minimum illumination of no less than 0.0005Lux, and a fill light distance of no less than 200m. The railway site collection and simulated intrusion collection methods shall be used. The real samples shall be collected based on the time points of the actual foreign object intrusion events that appear in the safety supervision system and the engineering patrol system. Video image samples shall be collected through the comprehensive video monitoring system. The simulated samples shall be collected on the East Suburbs Loop Line or the main railway line through props such as simulated stones of various colors and sizes, reflective clothing, etc., to simulate various intrusion events such as falling rocks at the tunnel entrance, personnel, mudslides, branches, etc.

[0050] In one embodiment of the present invention, under low illumination conditions (at night or on cloudy days), the brightness of the image decreases, the contrast decreases, the details are blurred, and the image noise increases, resulting in unclear edges of the target and reduced contrast between the target and the background, making it difficult to detect the target and difficult to identify the target, increasing the false detection rate and missed detection rate. In order to solve the above problems, image enhancement can be performed. Retinex is a classic algorithm for image enhancement, which aims to solve the problem of uneven brightness of images under different lighting conditions. The core idea of ​​the algorithm is to restore the brightness and contrast of the image by decomposing and reconstructing the local and global information of the image. At present, Retinex has been widely used in the field of image enhancement, especially in image enhancement under low light conditions. It can effectively improve the brightness, contrast and details of the image, and can maintain the true color of the object.

[0051] In one embodiment of the present invention, the railway line environment is complex and changeable, and is easily affected by interference factors such as rain, snow, wind, fog, haze and other weather factors, resulting in a decrease in the accuracy of algorithm recognition. Therefore, it is necessary to study solutions for the impact of severe weather scenes on image quality and target detection. There have been many public studies on single algorithms for rain removal, fog removal, snow removal, etc., but a single algorithm can only remove a single severe weather and cannot be applied to a real environment. The present invention adopts an all-in-one rain removal, fog removal, and snow removal algorithm based on knowledge distillation. By constructing a multi-teachers and single-student learning network, multiple teacher networks are responsible for denoising tasks of different severe weather conditions, and then the learned knowledge is transferred to the student network, so that the student network can have performance comparable to that of the teacher network while ensuring that the model size is small enough.

[0052] In one embodiment of the present invention, the edge adopts an intrusion target recognition algorithm that integrates background difference and deep learning (i.e., an edge fusion perception algorithm). The process of the edge fusion perception algorithm is as follows: Figure 3 As shown in the figure, an intrusive target recognition algorithm integrating background difference and deep learning is constructed to enhance the detection rate of untrained samples and realize reliable intelligent perception of all categories of foreign objects in railway scenarios.

[0053] For fixed-category foreign objects, a deep learning algorithm is trained with a large number of samples, and a multi-frame confirmation method is adopted to reduce false alarms, thereby realizing classified detection and alarm of foreign objects such as falling rocks, personnel, and trains. For other foreign objects not included in the training samples, positioning detection and alarm are realized based on background difference. This fusion algorithm has high recognition accuracy in a variety of environments, can more accurately extract data features, realize full-category foreign object detection in railways, and thus improve the recognition accuracy of the model.

[0054] According to a video image content analysis method for railway line danger provided by the present invention, the video stream of the railway line is detected by an edge device, specifically including: performing a differential operation on the current frame image in the video stream and the latest background image through an established background differential model to obtain a differential image; comparing the pixel value of each pixel point in the differential image with a preset threshold, binarizing the differential image according to the comparison result, and performing morphological processing on the binarized differential image to obtain a first target detection result; performing target detection on the frame image in the video stream through a trained CNN network target detection model to obtain a second target detection result; the trained CNN network target detection model is obtained by training with a training set, and the training set is a railway danger sample data set marked with foreign body categories; the first target detection result and the second target detection result are merged to determine the edge detection result, and an alarm image with abnormal targets is determined according to the edge detection result.

[0055] Specifically, the background difference algorithm used in edge detection is one of the most commonly used moving target detection algorithms in complex scenes. The background difference algorithm process is as follows: Figure 4 As shown in the figure, firstly, the current frame image is obtained from the video stream collected by the front-end image acquisition device (such as a high-definition camera, etc.), and the background image is obtained and updated by the background modeling method. Then, the current frame image and the latest background image are used for differential operation. The pixel value of each point in the differential image is compared with the threshold value obtained in advance based on experience or experiments, and then the differential image is binarized. Due to the complex background of the railway line environment, the binary image after background difference is easily interfered by external noise, that is, part of the background image is regarded as a foreground target. By using the mathematical morphology method to process the binary image, not only can the small range of noise be removed, but also the internal holes in the foreground target can be filled, and the external interference can be eliminated to obtain a more ideal moving target image.

[0056] When modeling, a mixed Gaussian background modeling method can be used. The monitoring scene is extremely complex. The background in the video sequence will be constantly affected by external factors such as weather, light and shadow, and change over time. Therefore, it is necessary to establish a suitable background difference model to timely reflect the changes in the background. In the railway emergency monitoring scenario, the use of mixed Gaussian background modeling difference method can achieve better target detection effect. The mixed Gaussian model uses 3-5 Gaussian models to describe the distribution of each pixel in the video image, compares the matching degree of each pixel in the current image with the pixel in the mixed Gaussian model, and determines whether the point is a background point. For multiple video frames in the video sequence, the weighted average of three single Gaussian models is used for background modeling. Since the background will change during the movement of the moving target, when the mixed Gaussian model is used for background modeling, it is necessary to continuously update the established background, mainly to update the mean, covariance and weight in the mixed Gaussian model, so as to establish a real-time scene that can reflect the background of the current frame image and improve the detection accuracy.

[0057] According to a video image content analysis method for railway line danger provided by the present invention, a CNN network target detection model is trained according to a training set, specifically including: obtaining a railway danger sample data set containing various types of danger targets; annotating each image in the railway danger sample data set to obtain a training set; inputting the training set into the CNN network target detection model for training to obtain a predicted target label; calculating a loss function according to the predicted target label and the true target label, and updating the model weight parameters according to the back propagation of the loss function; continuously iterating and updating the model weight parameters, and outputting the trained CNN network target detection model when the number of training times reaches the stopping condition.

[0058] Specifically, the deep learning target detection algorithm used in edge detection adopts the CNN network target detection model. The CNN network algorithm is composed of an input end, a backbone network, a feature fusion layer and three decoupling heads. The input end is responsible for scaling the collected image to a size of 640x640. Then the backbone network extracts the target features in the image. The backbone network includes a convolution module, a C2f structure and an SPPF module. The backbone network adds more jump connections and segmentation operations to enrich the gradient flow. The backbone network outputs three feature maps with different resolutions. The feature fusion network uses a feature pyramid network and a path aggregation structure to enhance the fusion of high-level semantic features and low-level detail features. The output end uses a decoupling head structure to separate the classification and detection heads. After being processed by the decoupling head structure, three category probability scores and four regression parameters corresponding to each feature point can be output. The category probability score is used to determine the target type included in each feature point. The four regression parameters corresponding to each feature point are decoded and adjusted to form a prediction box. The first two parameters represent the distance to the upper left corner of the prediction box, and the last two parameters represent the distance to the lower right corner of the prediction box. For all the prediction results of the feature points, the prediction boxes with scores greater than the confidence threshold are filtered out, and the final target is filtered out by combining the non-maximum suppression algorithm.

[0059] During the CNN network training process, images and sample labels in the sample library are loaded in batches. Data enhancement methods such as mosaic and image mixing are used to help the model learn more generalized feature representations. The pre-trained model is loaded to accelerate convergence. The CNN network predicts the target and the real target label through the back propagation of the loss function to update the weight parameters. The loss function consists of regression loss and classification loss. The regression loss is the judgment of the four regression parameters of the target feature point bounding box, and its loss function is CIOU Loss+DFL. CIOU Loss introduces an additional term on the basis of IOU Loss to measure the distance between the center point of the predicted rectangle and the center point of the labeled rectangle, as well as a penalty term for adjusting the difference in aspect ratio between the prediction and the annotation. The classification loss is the probability that the feature point contains the type of the target, and its loss function is Varifocal Loss. After 300 epochs (iterations) of the CNN network training, a good convergence effect is achieved, which can effectively identify the types and locations of most targets in railway line dangers.

[0060] In one embodiment of the present invention, the edge fusion perception algorithm is deployed on the edge algorithm box. The edge algorithm box can meet the communication requirements of receiving task instructions and uploading alarm information. The algorithm program is encapsulated by combining the communication interface to construct the overall functional structure.

[0061] The edge alarm process is as follows Figure 5As shown. When a foreign object is found during the analysis of the camera video stream, the edge computing device will upload the alarm image to the Minio server and obtain the returned uniform resource locator, i.e., the URL address. The URL address and the alarm information are constructed together into a json-formatted alarm information, which is sent to the cloud through Rabbit mq (message queue). The json-formatted alarm information includes: edge algorithm box device ID, camera ID, monitoring zone, alarm image URL, alarm target category, alarm target confidence, alarm target coordinate box, and alarm time.

[0062] Figure 6 FIG. 1 is one of the flow charts of the method for analyzing the video image content of railway line danger provided by the present invention, such as Figure 6 As shown, the method comprises the following steps: S610: Receive an alarm message through a cloud server, and obtain an alarm image according to the alarm message.

[0063] According to a video image content analysis method for railway line danger situations provided by the present invention, an alarm message is received through a cloud server, and an alarm image is obtained according to the alarm message, which specifically includes: the cloud server message queue receives the alarm information from the edge device, the alarm information includes the uniform resource locator of the alarm image; and the corresponding alarm image is downloaded according to the uniform resource locator of the alarm image.

[0064] Specifically, the cloud server first receives alarm information from edge devices through a message queue (such as RabbitMQ), which includes the URL address of the alarm image, device ID, camera ID, monitoring zone, alarm time, etc. According to the received URL address, the cloud server downloads the corresponding alarm image from the storage server (Minio).

[0065] Cloud testing process Figure 7 As shown in the figure. The cloud uses deep learning-based target detection and contrastive learning algorithms to detect foreign object intrusion in typical railway line scenarios. The Transformer-based target detection model is used to extract the deep features of dangerous targets in the edge alarm image and preliminarily identify dangerous objects in the image. Combined with the contrastive learning model, the identified objects are compared with the edge alarm results to further verify the accuracy of the alarm information.

[0066] S620: Detect the alarm image and obtain the cloud detection result.

[0067] According to a video image content analysis method for railway line danger provided by the present invention, an alarm image is detected to obtain a cloud detection result, which specifically includes: preprocessing the alarm image; dividing the preprocessed image into a plurality of independent blocks, and determining the corresponding pixel matrix according to each independent block; performing multi-stage processing on each pixel matrix based on a Swin Transformer network to obtain feature maps of different stages; fusing the feature maps of different stages and inputting them into a detection head for target detection, and outputting a category probability map and a bounding box offset; generating a predicted bounding box according to the category probability map and the bounding box offset; and obtaining a cloud detection result after processing the predicted bounding box by non-maximum suppression.

[0068] Specifically, the cloud target detection method uses the Swin Transformer network. When the collected monitoring images are preprocessed (such as resizing, normalization, etc.) and then input into the network, the block partitioning module divides the image into several independent blocks. Each small block area consists of 4×4×3 pixels. The pixel matrix is ​​divided, and the pixel values ​​in all channels of each small block area are converted into a one-dimensional vector with a vector length of 48. Then all vectors are combined according to the original area to obtain a pixel matrix. The matrix goes through four stages. In the first stage, the block needs to be linearly encoded and then passed to the Swin Transformer module. The encoding method only changes the number of channels of the feature without reducing the resolution. Each of the remaining stages includes a block merging module and a Swin Transformer module. Block merging can reduce the resolution of the feature map. After each stage, the feature map selects 1 element every 2 in the row and column directions and splices them together to form a complete tensor. At this time, the number of matrix channels is reduced by 4 times, and then a fully connected layer is used to adjust the number of channels to twice the original, thereby increasing the image resolution. The feature maps at different stages are fused and sent to the detection head composed of a fully convolutional network for target detection. The category probability map and bounding box offset are output, and a predicted bounding box is generated based on the category probability map and the bounding box offset. The detected target is then obtained through post-processing such as non-maximum suppression.

[0069] S630: Compare the cloud detection result with the edge detection result to determine the comparison result.

[0070] The edge detection results and alarm images are obtained by detecting the video stream of the railway line through the edge device. The alarm image is an image with abnormal targets.

[0071] According to a video image content analysis method for railway line danger situations provided by the present invention, the cloud detection results are compared with the edge detection results to determine the comparison results, which specifically includes: comparing the cloud detection results with the edge detection results based on a trained contrast learning model to determine the comparison results; the trained contrast learning model is trained according to a training set; the training set is positive samples with the same category label and negative samples with different category labels selected from a railway line danger sample library.

[0072] According to a video image content analysis method for railway line danger provided by the present invention, a contrast learning model is trained according to a training set, specifically comprising: selecting images from a constructed railway line danger sample library, taking images with the same category label as positive sample pairs, and images with different category labels as negative sample pairs, wherein the positive sample pairs and the negative sample pairs constitute positive and negative sample pairs; performing data enhancement on the positive and negative sample pairs; encoding the positive and negative sample pairs after data enhancement, and extracting deep features of the input images; performing dimensionality reduction and filtering on the extracted deep feature dimensions, and outputting a multidimensional feature vector; calculating a contrast loss function according to the multidimensional feature vector, and updating the parameters of the contrast learning model according to the calculation results; and continuing to iteratively update the contrast learning model until an iteration stop condition is met.

[0073] Specifically, the contrastive learning method adopts a method based on deep convolutional networks. The contrastive learning framework has three modules: encoder, classifier, and mapper. The encoder processes the image to extract deep features, and the classifier further abstracts high-level features. At the same time, the mapper and contrastive loss function are used to further restrict the representation learned by the encoder. The goal is to reduce the distance of the same category and expand the distance of different categories. The contrastive learning training process is as follows: 1. Construct positive and negative samples. Select targets with the same category label as positive samples and targets with different category labels as negative samples from the constructed railway line crisis sample library.

[0074] 2. Six image transformation methods, including left-right flipping, up-down flipping, contrast transformation, Gaussian noise addition, brightness transformation, and Gaussian noise plus brightness transformation, are used to perform data enhancement on the samples.

[0075] 3. The positive and negative samples pass through the ResNet network in the encoder, which contains 13 convolutional layers and ReLU activation function layers, as well as 5 maximum pooling layers.

[0076] 4. The extracted sample features are passed through the four-layer fully connected neural network in the classifier to reduce the feature dimension learned by the encoder.

[0077] 5. The sample features are passed through 2 layers of fully connected neural networks and 1 ReLU activation layer in the mapper to further filter the features extracted by the encoder and extract more features related to the contrastive learning task. The mapper finally outputs a 128-dimensional feature vector.

[0078] 6. The 128-dimensional feature vector is passed through a joint loss function consisting of two loss functions, namely the negative logarithm function of the contrast loss function and the cross entropy classification loss, to update the parameters of the entire model.

[0079] After training, the preset goal of contrastive learning can be achieved, which is to increase the similarity between the features of positive samples and reduce the similarity between the features of negative samples.

[0080] S640, summarizing multiple comparison results and performing weighted processing to obtain a final alarm result.

[0081] In one embodiment of the present invention, a voting mechanism is set up in the cloud, and the multi-way algorithm voting mechanism is as follows: Figure 8 As shown in the figure, the camera collects the scene image to form a video stream, the edge device pulls the video stream, and the edge uses an intelligent algorithm box (including background difference algorithm for detecting moving targets, CNN network target detection algorithm for detecting foreign objects and their categories, and integrating the detection results of the two algorithms) to identify dangerous situations, quickly detect abnormalities, and send the detection results, alarm images, foreign object locations, categories and other alarm information to the cloud through message queues. After receiving the alarm image, the cloud uses the cloud algorithm (including the SwinTransformer method to detect dangerous targets in the alarm image, and compares the detection results with the edge detection results through the comparative learning algorithm) for secondary review, outputs the comparison results, removes false alarms, and retains the real dangerous targets. Finally, the voting weights are set by combining the performance of multiple cloud algorithms and the scenarios they are good at, the types of foreign objects they are good at identifying, and the anti-interference ability, and the comparison results of multiple algorithms are summarized. The final alarm result is obtained by combining the advantages of multiple algorithms according to the voting strategy. The multi-channel algorithm voting mechanism can adopt the following common voting strategies: 1. Voting: Each algorithm model votes on the result, and finally selects the result with the most votes as the final decision result. This voting strategy is suitable for binary classification problems or multi-classification problems.

[0082] 2. Weighted voting: Assign weights to each algorithm model, and perform weighted summation or weighted average of the results according to their importance to obtain the final decision result. This voting strategy is suitable for situations where there are different levels of trust in different algorithm models.

[0083] 3. Confidence voting: Each algorithm model gives a confidence or credibility assessment for the result, and the results are weighed and selected according to the confidence level. This voting strategy is suitable for situations where the confidence assessment of the algorithm model is reliable.

[0084] 4. Decision tree voting: According to the pre-defined decision tree or rules, the results of each algorithm model are judged and selected. This voting strategy is suitable for situations where different decisions need to be made according to different situations.

[0085] By combining the results of multiple algorithms, the accuracy, robustness and reliability of the system can be improved and the limitations and uncertainties of a single algorithm can be reduced.

[0086] The method of the present invention can monitor and analyze railway lines in real time to detect potential dangerous situations. It can take advantage of edge computing and cloud computing, use lightweight machine learning models on edge devices to perform preliminary analysis of images, screen out areas where problems may exist, and then transmit the processed data to the cloud for further analysis using more complex deep learning models. This solves the technical problems of computing resource limitations and network bandwidth limitations in traditional railway line monitoring systems on the one hand; on the other hand, it achieves high-precision detection effects and reduces false alarm rates.

[0087] The present invention adopts a cloud-edge combination method to monitor dangerous conditions on railway lines, and has the following advantages: first, it is economical, only using cameras as front-end acquisition devices, and runs smoothly; second, the use of traditional background difference methods for preliminary detection can avoid the omission of unlearned categories by deep learning methods, and has strong scene applicability; third, the cloud-edge combination method, the edge device pre-detects the monitoring area, and the cloud reviews the edge alarm image, which can not only reduce bandwidth pressure to achieve real-time monitoring, but also use the high computing power resources of the cloud for secondary detection, remove false alarms caused by interference such as light and shadow changes, weather changes, etc., and improve the accuracy rate.

[0088] The video image content analysis device for railway line danger situations provided by the present invention is described below. The video image content analysis device for railway line danger situations described below and the video image content analysis method for railway line danger situations described above can be referenced to each other.

[0089] like Fig. 9 The present invention shows a video image content analysis device for railway line danger, comprising: The image acquisition module 910 is used to receive the alarm message through the cloud server and acquire the alarm image according to the alarm message; A cloud detection module 920 is used to detect the alarm image and obtain a cloud detection result; The cloud-edge comparison module 930 is used to compare the cloud detection result with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line by the edge device, and the alarm image is an image with abnormal targets; The alarm result determination module 940 aggregates multiple comparison results and performs weighted processing to obtain a final alarm result.

[0090] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a video image content analysis method for railway line danger, the method comprising: receiving an alarm message through a cloud server, obtaining an alarm image according to the alarm message; detecting the alarm image to obtain a cloud detection result; comparing the cloud detection result with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line through the edge device, and the alarm image is an image with an abnormal target; and summarizing multiple comparison results for weighted processing to obtain a final alarm result.

[0091] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video image content analysis method for railway line crises provided by the above methods, and the method includes: receiving an alarm message through a cloud server, and obtaining an alarm image according to the alarm message; detecting the alarm image to obtain a cloud detection result; comparing the cloud detection result with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line through the edge device, and the alarm image is an image with abnormal targets; summarizing multiple comparison results for weighted processing to obtain the final alarm result.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the video image content analysis method for railway line crises provided by the above-mentioned methods, the method comprising: receiving an alarm message through a cloud server, and obtaining an alarm image according to the alarm message; detecting the alarm image to obtain a cloud detection result; comparing the cloud detection result with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line by an edge device, and the alarm image is an image with abnormal targets; and summarizing multiple comparison results for weighted processing to obtain a final alarm result.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0095] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video image content analysis method for railway line danger, characterized in that: include: Receiving an alarm message through a cloud server, and acquiring an alarm image according to the alarm message; Detecting the alarm image to obtain a cloud detection result; The cloud detection result is compared with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line by the edge device, and the alarm image is an image with abnormal targets; Multiple comparison results are summarized and weighted to obtain the final alarm result.

2. The method for analyzing video image content for railway line danger according to claim 1, characterized in that: The video stream of the railway line is detected through the edge device, including: Performing a difference operation on the current frame image in the video stream and the latest background image by using the established background difference model to obtain a difference image; Comparing the pixel value of each pixel in the differential image with a preset threshold, binarizing the differential image according to the comparison result, and performing morphological processing on the binarized differential image to obtain a first target detection result; Performing target detection on the frame image in the video stream by using a trained convolutional neural network target detection model to obtain a second target detection result; the trained convolutional neural network target detection model is obtained by training a training set, and the training set is a railway danger sample data set marked with a foreign object category; The first target detection result and the second target detection result are fused to determine an edge detection result, and an alarm image with an abnormal target is determined according to the edge detection result.

3. The method for analyzing video image content for railway line danger according to claim 1, characterized in that: The detecting of the alarm image to obtain a cloud detection result specifically includes: Preprocessing the alarm image; The preprocessed image is divided into a number of independent blocks, and the corresponding pixel matrix is ​​determined according to each independent block; Based on the Swin Transformer network, each pixel matrix is ​​processed in multiple stages to obtain feature maps at different stages; The feature maps at different stages are fused and input into the detection head for target detection, and the category probability map and bounding box offset are output; Generate a predicted bounding box according to the category probability map and the bounding box offset; The predicted bounding box is processed by non-maximum suppression to obtain a cloud detection result.

4. The method for analyzing video image content for railway line danger according to claim 1, characterized in that: The comparing the cloud detection result with the edge detection result to determine the comparison result specifically includes: Comparing the cloud detection result with the edge detection result based on the trained contrastive learning model to determine a comparison result; The trained contrastive learning model is obtained by training based on a training set; the training set is positive samples with the same category label and negative samples with different category labels selected from a railway line crisis sample library.

5. The method for analyzing video image content for railway line danger according to claim 2, characterized in that: The convolutional neural network target detection model is trained according to the training set, including: Obtain a railway emergency sample data set containing various emergency targets; Annotating each image in the railway emergency sample data set to obtain a training set; Inputting the training set into a convolutional neural network target detection model for training to obtain a predicted target signature; Calculate the loss function based on the predicted target label and the true target label, and update the model weight parameters based on the back propagation of the loss function; The model weight parameters are continuously updated iteratively, and when the number of training times reaches the stopping condition, the trained convolutional neural network target detection model is output.

6. The method for analyzing video image content for railway line danger according to claim 4, characterized in that: The training and comparison learning model is trained based on the training set, including: Select images from the constructed railway line danger sample library, use images with the same category label as positive sample pairs, and use images with different category labels as negative sample pairs, the positive sample pairs and the negative sample pairs constitute positive and negative sample pairs; Performing data enhancement on the positive and negative sample pairs; Encode the positive and negative sample pairs after data enhancement to extract the deep features of the input image; Perform dimensionality reduction and filtering on the extracted deep feature dimensions and output a multi-dimensional feature vector; Calculating a contrast loss function according to the multidimensional feature vector, and updating parameters of the contrast learning model according to the calculation result; Continue to iteratively update the contrastive learning model until the iteration stop condition is met.

7. The method for analyzing video image content for railway line danger according to claim 1, characterized in that: The step of receiving an alarm message through a cloud server and obtaining an alarm image according to the alarm message specifically includes: The cloud server message queue receives the alarm information from the edge device, wherein the alarm information includes the uniform resource locator of the alarm image; The corresponding alarm image is downloaded according to the uniform resource locator of the alarm image.

8. A video image content analysis device for railway line danger, characterized in that: include: An image acquisition module is used to receive an alarm message through a cloud server and acquire an alarm image according to the alarm message; A cloud detection module, used to detect the alarm image and obtain a cloud detection result; The cloud-edge comparison module is used to compare the cloud detection result with the edge detection result to determine the comparison result; the edge detection result and the alarm image are obtained by detecting the video stream of the railway line by the edge device, and the alarm image is an image with abnormal targets; The alarm result determination module summarizes multiple comparison results and performs weighted processing to obtain the final alarm result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video image content analysis method for railway line danger situations as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video image content analysis method for railway line danger situations as described in any one of claims 1 to 7 is implemented.

Images