Railway dangerous tree detection terminal, method, device and equipment

Abstract

The invention relates to the technical field of computer vision, in particular to a railway dangerous tree detection terminal, method, device and equipment. The method comprises the following steps: acquiring a video of a railway operation environment shot when a train runs, if determining that a tree group exists in the video, acquiring a video segment in which the tree group exists, determining a first optical flow value corresponding to a tree between every two adjacent image frames in the video segment, and determining a second optical flow value corresponding to a tree between every two adjacent image frames due to the fact that an object closer to the horizontal position of an optical center point of image acquisition equipment is, the optical flow value is larger; and the size of the optical flow value is related to the vehicle speed, so that different optical flow threshold values are stored for different vehicle speeds, when the first optical flow value corresponding to the tree between two adjacent image frames of the tree group is greater than the optical flow threshold value, the existence of the dangerous tree is determined, and the dangerous tree detection can be realized as long as the video for recording the railway operation environment can be obtained. And the dangerous tree detection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a railway dangerous tree detection terminal, method, device and equipment. Background Art

[0002] Dangerous trees refer to trees planted or growing naturally along the railway line, which pose a potential safety hazard to the overhead line, return line, self-closing line, through line, power supply line, low-voltage power line, and railway driving safety and equipment safety. Since trees are in a state of continuous growth, if dangerous railway trees are not discovered in time, they will seriously threaten the safe operation of trains.

[0003] In the related art, it is generally determined whether there are dangerous trees in the railway area by on-duty personnel patrolling the roads. However, this method of detecting dangerous trees requires a lot of manpower and is extremely inefficient.

[0004] Therefore, how to efficiently detect dangerous trees on railways has become an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a railway dangerous tree detection terminal, method, device and equipment to solve the problem of low efficiency of dangerous tree detection in the prior art.

[0006] The present application provides a railway dangerous tree detection terminal, the terminal comprising a display and a processor:

[0007] The processor is configured to:

[0008] Obtain videos of the railway operating environment taken while the train is running;

[0009] If it is determined that there is a group of trees in the video, then obtaining a video segment containing the group of trees; determining a first optical flow value corresponding to the trees between every two adjacent image frames in the video segment based on an optical flow estimation algorithm;

[0010] Obtaining an optical flow threshold stored for the vehicle speed corresponding to the video segment, and if any first optical flow value is greater than the optical flow threshold, determining that there is a dangerous tree in the video segment;

[0011] The display is configured to display the detection result of whether there are dangerous trees in the video.

[0012] The above technical solution has the following advantages or beneficial effects: a video of the railway operation environment taken when the train is running is obtained. If it is determined that there is a tree group in the video, a video segment with the tree group is obtained, and the first optical flow value corresponding to the trees between each two adjacent image frames in the video segment is determined based on the optical flow estimation algorithm, and the optical flow threshold value saved for the vehicle speed corresponding to the video segment is obtained. If any first optical flow value is greater than the optical flow threshold value, it is determined that there is a dangerous tree in the video segment. Since the closer the object is to the horizontal position of the optical center point of the image acquisition device, the larger the optical flow value, and the size of the optical flow value is related to the vehicle speed, different optical flow thresholds are saved for different vehicle speeds. When the first optical flow value corresponding to the trees between two adjacent image frames with a tree group is greater than the optical flow threshold value, it is determined that there is a dangerous tree. As long as the video recording the railway operation environment can be obtained, dangerous tree detection can be achieved, thereby improving the efficiency of dangerous tree detection.

[0013] Furthermore, the processor is also configured to:

[0014] If a tree group exists in the image frame corresponding to the first optical flow value greater than the optical flow threshold, the subsequent step of determining whether a dangerous tree exists in the video segment is continued.

[0015] The above technical solution has the following advantages or beneficial effects: after determining that any first optical flow value is greater than the optical flow threshold, it is further determined whether there is a tree group in the corresponding image frame. If so, it is determined that there are dangerous trees in the video segment, thereby improving the accuracy of dangerous tree detection.

[0016] Furthermore, the processor is also configured to:

[0017] If there is no tree group in the image frame corresponding to the first optical flow value greater than the optical flow threshold, it is determined that there is no dangerous tree in the video segment.

[0018] The above technical solution has the following advantages or beneficial effects: after determining that any first optical flow value is greater than the optical flow threshold, it is further determined whether there is a tree group in the corresponding image frame. If not, it is determined that there are no dangerous trees in the video segment, thereby further improving the accuracy of dangerous tree detection.

[0019] Furthermore, the processor is also configured to:

[0020] Determine a target image frame in which a utility pole exists in the video;

[0021] For each target image frame, determine and save the second optical flow value of the electric pole between the target image frame and the next target image frame;

[0022] The processor is specifically configured to:

[0023] A second optical flow value corresponding to the target image frame included in the video segment is determined as the optical flow threshold.

[0024] The above technical solution has the following advantages or beneficial effects: the optical flow threshold is determined according to the optical flow values ​​of the electric poles in different videos, the optical flow threshold is dynamically determined, and the accuracy of dangerous tree detection is improved.

[0025] Further, if the video segment includes a plurality of target image frames, the processor is specifically configured to:

[0026] For each first optical flow value, determine first position information of an image frame corresponding to the first optical flow value in the video, and second position information of each target image frame in the video;

[0027] Determine, according to the first position information and each of the second position information, a target image frame closest to the image frame corresponding to the first optical flow value, and determine the second optical flow value corresponding to the closest target image frame as the optical flow threshold corresponding to the first optical flow value;

[0028] The processor is specifically configured to:

[0029] If any first optical flow value is greater than an optical flow threshold corresponding to the first optical flow value, it is determined that a dangerous tree exists in the video segment.

[0030] The above technical solution has the following advantages or beneficial effects: when comparing each first optical flow value, the target image frame closest to the image frame corresponding to the first optical flow value is determined, and the second optical flow value corresponding to the closest target image frame is determined as the optical flow threshold corresponding to the first optical flow value, which is equivalent to using the optical flow values ​​at adjacent moments for comparison, thereby improving the accuracy of dangerous tree detection.

[0031] Furthermore, the processor is further configured to:

[0032] Determining a target area in the target image frame according to a size of the target image frame and a pre-saved size ratio;

[0033] For each target image frame, if the electric pole in the target image frame is not located in the target area, the target image frame is deleted.

[0034] The above technical solution has the following advantages or beneficial effects: through the pre-saved size ratio, the target area in the target image frame is determined, and the target image frames where the electric poles do not exist in the target area are deleted, thereby retaining high-quality target image frames, thereby improving the accuracy of dangerous tree detection.

[0035] Furthermore, the processor is further configured to:

[0036] If it is determined that there is no group of trees in the video, it is determined that there is no dangerous tree in the video.

[0037] The present application provides a railway dangerous tree detection method, the method comprising:

[0038] Obtain videos of the railway operating environment taken while the train is running;

[0039] If it is determined that there is a group of trees in the video, then obtaining a video segment containing the group of trees; determining a first optical flow value corresponding to the trees between every two adjacent image frames in the video segment based on an optical flow estimation algorithm;

[0040] An optical flow threshold stored for the vehicle speed corresponding to the video segment is obtained, and if any first optical flow value is greater than the optical flow threshold, it is determined that a dangerous tree exists in the video segment.

[0041] The present application provides a railway dangerous tree detection device, the device comprising:

[0042] An acquisition module is used to acquire a video of the railway operation environment shot when the train is running;

[0043] A determination module, configured to obtain a video segment containing the tree group if it is determined that the video contains the tree group; and determine a first optical flow value corresponding to the tree between each two adjacent image frames in the video segment based on an optical flow estimation algorithm;

[0044] The detection module is used to obtain an optical flow threshold stored for the vehicle speed corresponding to the video segment, and if any first optical flow value is greater than the optical flow threshold, it is determined that there is a dangerous tree in the video segment.

[0045] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned railway dangerous tree detection methods when executing a computer program stored in a memory.

[0046] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned methods for detecting dangerous railway trees. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1A schematic diagram of the structure of a railway dangerous tree detection terminal provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a railway passenger-addition scenario provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the principle of determining an optical flow value provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a railway dangerous tree detection process provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a railway dangerous tree detection process provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a railway dangerous tree detection device provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0056] The embodiments of the present application provide a railway dangerous tree detection terminal, method, device and equipment, the terminal including a display and a processor: the processor is configured to: obtain a video of the railway operation environment taken when a train is running, if it is determined that there is a tree group in the video, then obtain a video segment with the tree group, and determine the first optical flow value corresponding to the trees between each two adjacent image frames in the video segment based on an optical flow estimation algorithm, obtain the optical flow threshold saved for the vehicle speed corresponding to the video segment, if any first optical flow value is greater than the optical flow threshold, it is determined that there is a dangerous tree in the video segment; the display is configured to: display the detection result of whether there is a dangerous tree in the video.

[0057] The railway dangerous tree detection terminal 10 provided in the present application can be used for dangerous tree detection in railway passenger scenarios. Figure 1A schematic diagram of the structure of a railway dangerous tree detection terminal provided in an embodiment of the present application, the video monitoring terminal 10 includes a display 101 and a processor 102, wherein the processor 102 is configured to: obtain a video of the railway operating environment taken when a train is running, if it is determined that there is a tree group in the video, then obtain a video segment with the tree group, and determine the first optical flow value corresponding to the trees between each two adjacent image frames in the video segment based on an optical flow estimation algorithm, obtain the optical flow threshold saved for the vehicle speed corresponding to the video segment, if any first optical flow value is greater than the optical flow threshold, it is determined that there is a dangerous tree in the video segment; the display 101 is configured to: display the detection result of whether there is a dangerous tree in the video.

[0058] Figure 2 A schematic diagram of a railway passenger-adding scenario provided in an embodiment of the present application, the railway passenger-adding scenario is as follows Figure 2 As shown, the video stream collected in the driving cabin of a train on the rails in the forward direction is generally used to check whether there are any objects or events ahead that may hinder the train's progress. Figure 2 The branches and leaves of the tree marked in the middle rectangular box have extended into the vicinity of the railway track area and may cause scratches to the trains running on the track. The tree is considered a dangerous tree.

[0059] In order to detect whether there are dangerous trees near the rail area, in the embodiment of the present application, the processor 102 of the railway dangerous tree detection terminal 10 can obtain a video of the railway operating environment taken when the train is running. The video can be collected by an image acquisition device installed on the front of the train. The image acquisition device can be any device with image acquisition function. Those skilled in the art can configure it as needed, as long as the image of the railway operating environment can be clearly captured. The video can be a real-time file or an offline file.

[0060] After obtaining the video, it can be determined whether there is a group of trees in the video. If so, it means that the corresponding area may have the risk of dangerous trees. The video segment with the group of trees can be captured in the video. The video segment can be one or more. Each video segment is then tested to determine whether there are dangerous trees. When determining whether there is a group of trees in the video, it can be determined based on a pre-trained classification model.

[0061] Specifically, the video frames of the railway passenger scene can be classified and annotated in advance, and the annotations can be divided into lush tree scenes and other scenes. The annotated images are used as training sets to train a classification model, which can be a deep learning network based on the deep residual network 18 layers (Residual Network-18, ResNet-18). The trained classification model is then used to classify the scene of the video. When the classification result is a lush tree scene, it can be considered that there are tree groups in the video.

[0062] Since the closer the object is to the horizontal position of the optical center point of the image acquisition device, the larger the optical flow value, therefore, in the embodiment of the present application, an optical flow estimation algorithm can be used to estimate the optical flow of the image frame with the tree group and the next frame of the image, and the optical flow value at the reference position is compared with the optical flow threshold. If it is greater than the optical flow threshold, the frame is an alarm candidate frame. Specifically, in the embodiment of the present application, after obtaining the video segment with the tree group, the first optical flow value corresponding to the tree between each two adjacent image frames in the video segment can be determined based on the optical flow estimation algorithm. In other words, whether the corresponding tree is too close to the train is determined based on the size of each first optical flow value. Among them, the optical flow estimation algorithm can be a deep learning algorithm or other optical flow estimation algorithm, and technicians in this field can configure it as needed.

[0063] Since the change of train speed will affect the optical flow value of the object in the image, in the embodiment of the present application, the optical flow threshold can be saved in advance for different speeds. The optical flow threshold can be determined in advance based on the buildings that often appear near the rails, such as electric poles, transformers, etc. For example, the electric poles that supply power to the train next to the rails can be used as a reference, and trees that are closer to the rails than the electric poles can be judged as dangerous trees.

[0064] After determining each first optical flow value, the optical flow threshold value saved for the vehicle speed corresponding to the video segment can be obtained, and each first optical flow value can be judged whether it is greater than the optical flow threshold value. If any first optical flow value is greater than the optical flow threshold value, it means that the tree in the image frame corresponding to the first optical flow value is close to the train, and the tree can be considered to be a dangerous tree. In other words, there are dangerous trees in the video segment. After all video segments have been detected, the processor 102 can control the display 101 to display the detection results of whether there are dangerous trees in the video. Exemplarily, the video information of each video segment and the corresponding detection results can be listed in sequence.

[0065] In the embodiment of the present application, a video of the railway operating environment taken when the train is running is obtained. If it is determined that there is a tree group in the video, a video segment with the tree group is obtained, and the first optical flow value corresponding to the trees between each two adjacent image frames in the video segment is determined based on the optical flow estimation algorithm, and the optical flow threshold stored for the vehicle speed corresponding to the video segment is obtained. If any first optical flow value is greater than the optical flow threshold, it is determined that there is a dangerous tree in the video segment. Since the closer the object is to the horizontal position of the optical center point of the image acquisition device, the larger the optical flow value, and the size of the optical flow value is related to the vehicle speed, different optical flow thresholds are stored for different vehicle speeds. When the first optical flow value corresponding to the trees between two adjacent image frames with a tree group is greater than the optical flow threshold, it is determined that there is a dangerous tree. As long as the video recording the railway operating environment can be obtained, dangerous tree detection can be achieved, thereby improving the efficiency of dangerous tree detection.

[0066] For ease of understanding, the principle of dangerous tree detection based on optical flow estimation is explained below. Figure 3 A schematic diagram of the principle of determining an optical flow value provided in an embodiment of the present application. Figure 3 This is a simple schematic diagram of the imaging of the passenger scene from a bird's-eye view. Point O in the figure is the optical center of the passenger scene acquisition device, and the lowest horizontal line is the imaging surface. We transform the ground reference system into the acquisition device reference system, and the forward movement of the train is transformed into the movement of the front scene toward the train. Among them, A1 and B1 are two objects in front of the train at a certain moment, and their position relationship is shown in the figure; A2 and B2 are two objects in front of the train at another moment, and their position relationship is shown in the figure, that is, the horizontal distance between B1 and point O is smaller than that between A1. They are all imaged at point A'1 (B'1) on the imaging surface, and the imaging surface can be understood as the position in the video image. After the train moves forward for a short period of time, the two objects are located at A2 and B2 respectively (A1A2=B1B2). At this time, they are imaged at points A'2 and B'2 on the imaging surface respectively, and it is easy to prove that B'1B'2>A'1A'2. If the short period of time is the time interval between two frames of the video, A'1A'2 is the optical flow of object A1 at A'1, and B'1B'2 is the optical flow of object B1 at B'1. Therefore, for objects imaged at the same point on the imaging surface, the closer the horizontal position of the object is to the optical center point, the greater the optical flow value. In the dangerous tree detection scheme of the embodiment of the present application, A'1(B'1) can be regarded as a reference position, A1 can be regarded as a reference object, a telephone pole, B1 can be regarded as a dangerous tree, and A'1A'2 can be regarded as a reference optical flow threshold.

[0067] In order to further improve the accuracy of railway dangerous tree detection, based on the above embodiment, in the embodiment of the present application, the processor 102 is further configured to:

[0068] If a tree group exists in the image frame corresponding to the first optical flow value greater than the optical flow threshold, the subsequent step of determining whether a dangerous tree exists in the video segment is continued.

[0069] Since there are usually warning signs on both sides of the railroad tracks, and the warning signs are relatively close to the trains. In order to avoid the optical flow estimation algorithm mistakenly taking the optical flow value of the warning sign as the first optical flow value of the tree when determining the first optical flow value, and making an erroneous detection. In an embodiment of the present application, after determining that any first optical flow value is greater than the optical flow threshold, before determining that there are dangerous trees in the video segment, an image frame corresponding to the first optical flow value greater than the optical flow threshold can be obtained, and a judgment based on a pre-trained classification model can be made to determine whether there is a group of trees in the image frame. If there is, the subsequent steps of determining whether there are dangerous trees in the video segment can be continued. That is, when there is indeed a group of trees in the image frame, it is determined that there are dangerous trees in the video segment.

[0070] Specifically, the area near the reference position can be cut out from the railway passenger scene video frame in advance and the area can be classified and labeled, and the labels are divided into tree groups and other objects. The labeled images are used as training sets to train a classification model, such as a deep learning network based on resnet18. The trained classification model is then used to classify the corresponding image frame. If it is classified as a tree group, it is confirmed that a dangerous tree is detected in the image frame.

[0071] In an embodiment of the present application, after determining that any first optical flow value is greater than the optical flow threshold, it is further determined whether there is a tree group in the corresponding image frame. If so, it is determined that there are dangerous trees in the video segment, thereby improving the accuracy of dangerous tree detection.

[0072] In order to further improve the accuracy of railway dangerous tree detection, based on the above embodiments, in the embodiment of the present application, the processor 102 is further configured to:

[0073] If there is no tree group in the image frame corresponding to the first optical flow value greater than the optical flow threshold, it is determined that there is no dangerous tree in the video segment.

[0074] If there is no tree group in the image frame corresponding to the first optical flow value greater than the optical flow threshold, it means that there is indeed no dangerous tree in the corresponding video segment, and the optical flow estimation algorithm may have made a misjudgment when determining the first optical flow value.

[0075] In an embodiment of the present application, after determining that any first optical flow value is greater than the optical flow threshold, it is further determined whether there is a tree group in the corresponding image frame. If not, it is determined that there are no dangerous trees in the video segment, thereby further improving the accuracy of dangerous tree detection.

[0076] In order to further improve the accuracy of railway dangerous tree detection, based on the above embodiments, in the embodiment of the present application, the processor 102 is further configured to:

[0077] Determine a target image frame in which a utility pole exists in the video;

[0078] For each target image frame, determine and save the second optical flow value of the electric pole between the target image frame and the next target image frame;

[0079] The processor 102 is specifically configured to:

[0080] A second optical flow value corresponding to the target image frame included in the video segment is determined as the optical flow threshold.

[0081] Since the construction standards of different railway sections are different, the distances between the electric poles and the rails around different railway sections are also different, so the corresponding optical flow values ​​are also different. In addition, the imaging effects of images taken by different image acquisition devices for the same object are also different, so the optical flow values ​​determined based on the data of different imaging effects will also be different. Therefore, in order to further improve the accuracy of dangerous tree detection, in the embodiment of the present application, the corresponding optical flow threshold can be calculated for each video.

[0082] In an embodiment of the present application, multiple utility poles can be detected in a video to obtain multiple reference positions and reference thresholds to adapt to changes in the speed of the train. After the video is acquired, it is possible to determine the target image frame in which the utility pole exists in the video. Exemplarily, the power supply utility poles next to the railroad tracks can be annotated in advance in the video frame of the railway passenger scene, and a target detection model such as YOLOv9 can be trained using the annotated images as a training set. The trained target detection model is then used to detect the video and determine the target image frame in which the utility pole exists.

[0083] After determining each target image frame in which a utility pole exists, optical flow estimation can be performed on the target image frame in which the utility pole is detected and the next frame of the image to obtain the optical flow value at the utility pole area. The position of the utility pole on the image is used as a reference position, and the optical flow value of the utility pole area can be subsequently set as an optical flow threshold. Specifically, in an embodiment of the present application, for each target image frame, a second optical flow value of the utility pole between the target image frame and the next target image frame can be determined and saved. The second optical flow value can be used as an optical flow threshold for subsequent dangerous tree detection.

[0084] When obtaining the optical flow threshold value saved for the speed corresponding to the video segment, the target image frame included in the video segment can be determined, that is, which target image frame containing the electric pole is present in the video segment. Since the speed of the train will not change significantly under normal circumstances, the speed when the target image frame is photographed will not differ greatly from the speed when the image frame of the corresponding tree is photographed. Therefore, in the embodiment of the present application, the second optical flow value corresponding to the target image frame included in the video segment can be determined as the optical flow threshold value.

[0085] In the embodiment of the present application, the optical flow threshold is determined according to the optical flow values ​​of the electric poles in different videos, thereby realizing the dynamic determination of the optical flow threshold and improving the accuracy of dangerous tree detection.

[0086] In order to further improve the accuracy of dangerous tree detection, based on the above embodiments, in an embodiment of the present application, if the video segment includes multiple target image frames, the processor 102 is specifically configured as follows:

[0087] For each first optical flow value, determine first position information of an image frame corresponding to the first optical flow value in the video, and second position information of each target image frame in the video;

[0088] Determine, according to the first position information and each of the second position information, a target image frame closest to the image frame corresponding to the first optical flow value, and determine the second optical flow value corresponding to the closest target image frame as the optical flow threshold corresponding to the first optical flow value;

[0089] The processor is specifically configured to:

[0090] If any first optical flow value is greater than an optical flow threshold corresponding to the first optical flow value, it is determined that a dangerous tree exists in the video segment.

[0091] Since the electric poles on both sides of the railroad tracks appear frequently, the video segment may include multiple target image frames, and multiple optical flow thresholds can be determined based on the multiple target image frames. In order to further improve the accuracy of dangerous tree detection, in the embodiment of the present application, when judging each first optical flow value, the second optical flow value corresponding to the target image frame closest to the image frame corresponding to the first optical flow value can be obtained as the optical flow threshold.

[0092] In the embodiment of the present application, for each first optical flow value, the first position information of the image frame corresponding to the first optical flow value in the video and the second position information of each target image frame in the video can be determined. The first position information and the second position information can be any data that can describe the position of the corresponding image frame in the video, for example, the frame number.

[0093] After determining the first position information and each second position information, the target image frame closest to the image frame corresponding to the first optical flow value can be determined based on the first position information and each second position information, and the second optical flow value corresponding to the closest target image frame can be determined as the optical flow threshold corresponding to the first optical flow value.

[0094] When each first optical flow value is subsequently judged, the judgment may be made according to the optical flow threshold corresponding to each first optical flow value. If any first optical flow value is greater than the optical flow threshold corresponding to the first optical flow value, it may be determined that there is a dangerous tree in the video segment.

[0095] In an embodiment of the present application, when comparing each first optical flow value, the target image frame closest to the image frame corresponding to the first optical flow value is determined, and the second optical flow value corresponding to the closest target image frame is determined as the optical flow threshold corresponding to the first optical flow value, which is equivalent to using the optical flow values ​​at nearby moments for comparison, thereby improving the accuracy of dangerous tree detection.

[0096] In order to further improve the accuracy of dangerous tree detection, based on the above embodiments, in the embodiment of the present application, the processor 102 is further configured to:

[0097] Determining a target area in the target image frame according to a size of the target image frame and a pre-saved size ratio;

[0098] For each target image frame, if the electric pole in the target image frame is not located in the target area, the target image frame is deleted.

[0099] When the electric pole is too close to the center of the image, the optical flow value is too small, and the impact of noise / error is too great; when it is too close to the edge of the image, the object may exceed the imaging range in the next frame and the optical flow value cannot be estimated, and the distortion of the edge of the image will also cause errors. Therefore, in the embodiment of the present application, the size ratio can be saved in advance, and after determining the target image frame with the electric pole, the current target image frame can be screened based on the size ratio to obtain a high-quality target image frame.

[0100] In an embodiment of the present application, the target area in the target image frame can be determined according to the size of the target image frame and the pre-saved size ratio. For example, the size ratio can be 1 / 4 and 3 / 4. The area around 1 / 4 of the horizontal axis length of the target image frame and the area around 3 / 4 can be determined as the target area.

[0101] After the target area is determined, for each target image frame, it can be determined whether the electric pole in the target image frame is located in the target area. If not, it means that the imaging effect of the electric pole in the target image frame is not ideal, and the target image frame can be deleted.

[0102] In an embodiment of the present application, the target area in the target image frame is determined by pre-saving the size ratio, and the target image frames where the utility poles do not exist in the target area are deleted, thereby retaining high-quality target image frames and improving the accuracy of dangerous tree detection.

[0103] In order to further improve the accuracy of dangerous tree detection, based on the above embodiments, in the embodiment of the present application, the processor 102 is further configured to:

[0104] If it is determined that there is no group of trees in the video, it is determined that there is no dangerous tree in the video.

[0105] If there is no group of trees in the video, there will be no dangerous trees. Therefore, after obtaining the video of the railway operating environment shot when the train is running, if it is determined that there is no group of trees in the video, it can be determined that there are no dangerous trees in the video and the detection of the video can be ended.

[0106] The following describes a specific embodiment of the process of detecting dangerous trees by a railway dangerous tree detection terminal. Figure 4 A schematic diagram of a railway dangerous tree detection process provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, first read the video stream of the railway operation environment taken when the train is running. Detect the electric poles at the appropriate position in the video stream, estimate the optical flow of the detected electric poles, and determine a reference position and optical flow threshold according to the position of the electric poles and the optical flow value. Then classify the video stream, and estimate the optical flow of the image frames classified as lush tree scenes. After determining each optical flow value, compare the optical flow value with the optical flow threshold. If the optical flow value at the reference position of any image frame is greater than the optical flow threshold, the image frame is used as an alarm candidate; finally, the target classification is performed on the area near the reference position of the alarm candidate frame. If it is classified as a lush tree area, it is confirmed that there are dangerous trees in the frame, otherwise, it is determined that there are no dangerous trees. If there is no lush tree scene in the video, no dangerous tree detection is performed.

[0107] In an embodiment of the present application, firstly, telephone pole detection is performed on the input video stream, and optical flow estimation is performed on the telephone poles in the previous and next frames of the video. A reference position and optical flow threshold are determined based on the telephone pole position and the optical flow value; then, scene classification is performed on the video frames, and optical flow estimation is performed on the frames classified as lush tree scenes. If the optical flow value at the reference position is greater than the reference threshold, the frame is used as an alarm candidate frame; finally, target classification is performed on the area near the reference position of the alarm candidate frame. If it is classified as a lush tree area, it is confirmed that there is a dangerous tree in the frame. The greatest advantage of the dangerous tree detection scheme provided in the embodiment of the present application is that the relative distance is detected by the motion information between video frames, which provides a solution to the dangerous tree detection task in railway passenger scenes.

[0108] On the basis of the above embodiments, an embodiment of the present application further provides a railway dangerous tree detection method, which is consistent with the methods in the above embodiments and will not be described in detail here. Figure 5 A schematic diagram of a railway dangerous tree detection process provided in an embodiment of the present application, the process includes:

[0109] S501: Acquire a video of the railway operating environment shot when the train is running.

[0110] S502: If it is determined that there is a group of trees in the video, obtain a video segment containing the group of trees; and determine a first optical flow value corresponding to the trees between every two adjacent image frames in the video segment based on an optical flow estimation algorithm.

[0111] S503: Obtain an optical flow threshold stored for the vehicle speed corresponding to the video segment, and if any first optical flow value is greater than the optical flow threshold, determine that there is a dangerous tree in the video segment.

[0112] In a possible implementation manner, after any first optical flow value is greater than the optical flow threshold and before determining that a dangerous tree exists in the video segment, the method further includes:

[0113] If a tree group exists in the image frame corresponding to the first optical flow value greater than the optical flow threshold, the subsequent step of determining whether a dangerous tree exists in the video segment is continued.

[0114] In a possible implementation, the method further includes:

[0115] If there is no tree group in the image frame corresponding to the first optical flow value greater than the optical flow threshold, it is determined that there is no dangerous tree in the video segment.

[0116] In a possible implementation manner, before obtaining the optical flow threshold stored for the vehicle speed corresponding to the video segment, the method further includes:

[0117] Determine a target image frame in which a utility pole exists in the video;

[0118] For each target image frame, determine and save the second optical flow value of the electric pole between the target image frame and the next target image frame;

[0119] The obtaining of the optical flow threshold value stored for the vehicle speed corresponding to the video segment includes:

[0120] A second optical flow value corresponding to the target image frame included in the video segment is determined as the optical flow threshold.

[0121] In a possible implementation manner, if the video segment includes a plurality of target image frames, obtaining the optical flow threshold stored for the vehicle speed corresponding to the video segment includes:

[0122] For each first optical flow value, determine first position information of an image frame corresponding to the first optical flow value in the video, and second position information of each target image frame in the video;

[0123] Determine, according to the first position information and each of the second position information, a target image frame closest to the image frame corresponding to the first optical flow value, and determine the second optical flow value corresponding to the closest target image frame as the optical flow threshold corresponding to the first optical flow value;

[0124] If any first optical flow value is greater than the optical flow threshold, determining that there is a dangerous tree in the video segment includes:

[0125] If any first optical flow value is greater than an optical flow threshold corresponding to the first optical flow value, it is determined that a dangerous tree exists in the video segment.

[0126] In a possible implementation manner, after determining that there is a target image frame with a utility pole in the video, and before determining, for each target image frame, a second optical flow value of the utility pole between the target image frame and a next target image frame and saving the value, the method further includes:

[0127] Determining a target area in the target image frame according to a size of the target image frame and a pre-saved size ratio;

[0128] For each target image frame, if the electric pole in the target image frame is not located in the target area, the target image frame is deleted.

[0129] In a possible implementation, the method further includes:

[0130] If it is determined that there is no group of trees in the video, it is determined that there is no dangerous tree in the video.

[0131] Based on the above embodiments, Figure 6 A schematic diagram of the structure of a railway dangerous tree detection device provided in an embodiment of the present application, the device comprises:

[0132] An acquisition module 601 is used to acquire a video of the railway operation environment shot when the train is running;

[0133] The determination module 602 is configured to obtain a video segment containing the tree group if it is determined that the video contains the tree group; and determine a first optical flow value corresponding to the tree between each two adjacent image frames in the video segment based on an optical flow estimation algorithm;

[0134] The detection module 603 is used to obtain an optical flow threshold stored for the vehicle speed corresponding to the video segment, and if any first optical flow value is greater than the optical flow threshold, it is determined that there is a dangerous tree in the video segment.

[0135] In a possible implementation manner, the detection module 603 is further configured to continue to execute a subsequent step of determining whether there are dangerous trees in the video segment if there are tree groups in the image frame corresponding to the first optical flow value greater than the optical flow threshold.

[0136] In a possible implementation manner, the detection module 603 is further configured to determine that there are no dangerous trees in the video segment if there are no tree groups in the image frame corresponding to the first optical flow value greater than the optical flow threshold.

[0137] In a possible implementation manner, the determination module 602 is further configured to determine a target image frame in which a utility pole exists in the video; for each target image frame, determine and save a second optical flow value of the utility pole between the target image frame and the next target image frame;

[0138] The detection module 603 is specifically configured to determine the second optical flow value corresponding to the target image frame included in the video segment as the optical flow threshold.

[0139] In a possible implementation, if the video segment includes multiple target image frames, the detection module 603 is specifically configured to determine, for each first optical flow value, first position information of the image frame corresponding to the first optical flow value in the video, and second position information of each target image frame in the video;

[0140] According to the first position information and each of the second position information, determine the target image frame closest to the image frame corresponding to the first optical flow value, and determine the second optical flow value corresponding to the closest target image frame as the optical flow threshold corresponding to the first optical flow value; if any first optical flow value is greater than the optical flow threshold corresponding to the first optical flow value, it is determined that there is a dangerous tree in the video segment.

[0141] In a possible implementation, the determination module 602 is further used to determine the target area in the target image frame based on the size of the target image frame and a pre-saved size ratio; for each target image frame, if the electric pole existing in the target image frame is not located in the target area, the target image frame is deleted.

[0142] In a possible implementation, the detection module 603 is further configured to determine that there are no dangerous trees in the video if it is determined that there are no tree groups in the video.

[0143] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, it includes: a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704;

[0144] The memory 703 stores a computer program. When the program is executed by the processor 701, the processor 701 executes the steps of the railway dangerous tree detection method described in the above embodiments.

[0145] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 702 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0146] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0147] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based image generation method described in the above embodiments.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A railway dangerous tree detection terminal, characterized in that: The terminal includes a display and a processor: The processor is configured to: Obtain videos of the railway operating environment taken while the train is running; If it is determined that a group of trees exists in the video, obtaining a video segment containing the group of trees; Determine a first optical flow value corresponding to a tree between every two adjacent image frames in the video segment based on an optical flow estimation algorithm; Obtaining an optical flow threshold stored for the vehicle speed corresponding to the video segment, and if any first optical flow value is greater than the optical flow threshold, determining that there is a dangerous tree in the video segment; The display is configured to display the detection result of whether there are dangerous trees in the video.

2. The terminal according to claim 1, characterized in that The processor is further configured to: If a tree group exists in the image frame corresponding to the first optical flow value greater than the optical flow threshold, the subsequent step of determining whether a dangerous tree exists in the video segment is continued.

3. The terminal according to claim 2, characterized in that: The processor is further configured to: If there is no tree group in the image frame corresponding to the first optical flow value greater than the optical flow threshold, it is determined that there is no dangerous tree in the video segment.

4. The terminal according to claim 1, characterized in that: The processor is further configured to: Determine a target image frame in which a utility pole exists in the video; For each target image frame, determine and save the second optical flow value of the electric pole between the target image frame and the next target image frame; The processor is specifically configured to: A second optical flow value corresponding to the target image frame included in the video segment is determined as the optical flow threshold.

5. The terminal according to claim 4, characterized in that: If the video segment includes a plurality of target image frames, the processor is specifically configured to: For each first optical flow value, determine first position information of an image frame corresponding to the first optical flow value in the video, and second position information of each target image frame in the video; Determine, according to the first position information and each of the second position information, a target image frame closest to the image frame corresponding to the first optical flow value, and determine the second optical flow value corresponding to the closest target image frame as the optical flow threshold corresponding to the first optical flow value; The processor is specifically configured to: If any first optical flow value is greater than an optical flow threshold corresponding to the first optical flow value, it is determined that a dangerous tree exists in the video segment.

6. The terminal according to claim 4, characterized in that: The processor is further configured to: Determining a target area in the target image frame according to a size of the target image frame and a pre-saved size ratio; For each target image frame, if the electric pole in the target image frame is not located in the target area, the target image frame is deleted.

7. The terminal according to claim 1, characterized in that: The processor is further configured to: If it is determined that there is no group of trees in the video, it is determined that there is no dangerous tree in the video.

8. A method for detecting dangerous trees on railways, characterized in that: The method comprises: Obtain videos of the railway operating environment taken while the train is running; If it is determined that there is a group of trees in the video, then obtaining a video segment containing the group of trees; determining a first optical flow value corresponding to the trees between every two adjacent image frames in the video segment based on an optical flow estimation algorithm; An optical flow threshold stored for the vehicle speed corresponding to the video segment is obtained, and if any first optical flow value is greater than the optical flow threshold, it is determined that a dangerous tree exists in the video segment.

9. A railway dangerous tree detection device, characterized in that: The device comprises: An acquisition module is used to acquire a video of the railway operation environment shot when the train is running; A determination module, configured to obtain a video segment containing the tree group if it is determined that the video contains the tree group; and determine a first optical flow value corresponding to the tree between each two adjacent image frames in the video segment based on an optical flow estimation algorithm; The detection module is used to obtain an optical flow threshold stored for the vehicle speed corresponding to the video segment, and if any first optical flow value is greater than the optical flow threshold, it is determined that there is a dangerous tree in the video segment.

10. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the railway dangerous tree detection method as claimed in claim 8 when executing the computer program stored in the memory.

Images