Abnormal object detection alert system, method and electronic device

Abstract

An abnormal object detection and warning system, applicable to unmanned vehicles performing track inspection tasks, includes: a storage module, a detection module, and a notification module. An abnormal object detection method includes: storing image data captured by the unmanned vehicle; detecting the image data to identify image data with abnormal characteristics; and transmitting the image data with abnormal characteristics to a user's mobile device. An electronic device has system software including the abnormal object detection and warning system, which can be updated online.

Description

Technical Field

[0001] This invention relates to an abnormal object detection warning system and method, and particularly to an abnormal object detection warning system and method that uses neural networks to detect curved rails and foreign objects on railways. Prior Technology

[0002] Many modern cities now incorporate railway systems into their public transportation systems, such as subways, metro systems, or light rail. Some of these railways are built above ground, some underground or in tunnels, and some in the air. Because railway transportation offers advantages in terms of speed and capacity for passenger transport, the safety of the railway system, especially the safety of the tracks, is paramount. It is known that subway operators send staff to inspect the tunnels late at night after operations close each day for any abnormalities. Furthermore, post-disaster inspections are necessary after natural disasters (such as earthquakes and typhoons) to assess tunnel damage and determine if operations can be resumed, all of which require additional manpower.

[0003] Traditional tunnel inspections require numerous personnel to walk and inspect each section of the tunnel, mostly at night, resulting in high overtime costs. Using unmanned vehicles (such as drones or drones) for inspections would significantly reduce costs. However, in dimly lit tunnels, ensuring the drones can safely navigate the environment and capture identifiable photographs presents a major challenge. Therefore, we must overcome these technical hurdles to ensure the drones can accurately identify any anomalies within the tunnel and fly safely and stably within the confined space.

[0004] Unmanned vehicles equipped with cameras are used to inspect underground subway tunnels. The captured images, combined with artificial intelligence technology, automatically detect abnormal conditions within the tunnels. This reduces the need for extensive daily manpower for inspections and speeds up the resumption of subway train services after disasters. The inspections cover whether any objects are left behind or have fallen onto the tunnels and tracks (e.g., fallen or retained maintenance tools, fallen lights, tunnel rings, and concrete blocks), and whether there are any abnormal bends in the tracks that could affect train safety. Summary of the Invention

[0005] This disclosure provides an abnormal object detection and warning system, applicable to unmanned vehicles performing track inspection tasks. The system includes a storage module, a detection module, and a notification module. The storage module is configured to store image data captured by the unmanned vehicle. The detection module is configured to detect the image data and identify image data with an abnormal feature. The notification module is configured to transmit a warning signal for the image data with the abnormal feature. For example, the warning signal can be transmitted to a user's (e.g., a worker's) mobile device. Additionally, the notification module can also transmit the image data with the abnormal feature to the user.

[0006] This disclosure provides an embodiment of an abnormal object detection method, comprising: a storage module configured to store image data captured by an unmanned vehicle; detecting the image data to identify image data with abnormal features from the image data; and transmitting the image data with abnormal features to a user's mobile device.

[0007] This disclosure provides an embodiment of an electronic device, including a storage unit, a memory unit, and a network unit; the storage unit is used to store image data captured by an unmanned vehicle; the memory unit is used to store system software for execution by the electronic device; and the network unit is used to transmit image data with unusual characteristics, wherein the electronic device updates the system software online through the network unit.

[0008] This disclosure provides a system software comprising a detection module configured to detect image data to identify image data with anomalous features; the detection module includes a neural network module configured to receive image data, process the image data through the neural network to obtain a prediction output; and the detection module generates labels on the image data with anomalous features based on the prediction output. Simple Explanation of the Diagram

[0009] Figure 1 illustrates a schematic diagram of a first embodiment of an abnormal object detection warning system according to an embodiment of the present disclosure. Figure 2 illustrates a block diagram of a second embodiment of an abnormal object detection warning system according to an embodiment of the present disclosure. Figure 3 illustrates a block diagram of a third embodiment of an abnormal object detection and warning system according to an embodiment of the present disclosure. Figure 4 illustrates a flowchart of an abnormal object detection method according to an embodiment of the present disclosure. Figure 5 illustrates a block diagram of a first embodiment of a detection module of an abnormal object detection warning system according to an embodiment of the present disclosure. Figure 6 illustrates a flowchart of a first embodiment of an abnormal object detection method according to an embodiment of the present disclosure. Figure 7 is a block diagram illustrating the neural network architecture of an object detection module according to an embodiment of the present disclosure. Figure 8 illustrates a schematic diagram of a rail boundary after processing by a rail boundary processing module according to an embodiment of this disclosure. Figure 9 illustrates a schematic diagram of the slope of a segment after processing by a linear regression processing module according to an embodiment of this disclosure. Figure 10 illustrates a block diagram of a second embodiment of the detection module of an abnormal object detection warning system according to an embodiment of the present disclosure. Figure 11 illustrates a flowchart of a second embodiment of an abnormal object detection method according to an embodiment of the present disclosure. Figure 12 illustrates a schematic diagram of a region of interest extraction module according to an embodiment of the present disclosure. Figure 13 illustrates a schematic diagram of an anomaly detection module according to an embodiment of the present disclosure. Figure 14 illustrates a schematic diagram of anomaly features detected by an anomaly detection module according to an embodiment of the present disclosure. Figure 15 illustrates a schematic diagram of the markings of an anomaly feature according to an embodiment of the present disclosure. Implementation

[0010] Please refer to Figure 1, which illustrates a schematic diagram of a first embodiment of an abnormal object detection and warning system 100 according to an embodiment of this disclosure. As shown in Figure 1, the unmanned vehicle (UAV) carries a camera and a lighting device to perform railway inspection tasks. It automatically takes pictures along the railway tracks according to the inspection schedule. After the inspection, it automatically flies back to the base station to recharge, and simultaneously uploads the captured photos 101 to the server 102. In dimly lit subway tunnels, the UAV can use the lighting device to provide a light source to identify the environment and fly safely, and to capture identifiable photos 101. The server 102 may include a remote server or a cloud server connected via a network. The UAV can upload the captured photos 101 to the server 102. The server 102 receives the photos 101 via a communication port (e.g., serial port, parallel port, USB port, or network port) in a wired or wireless manner and stores the photos 101 to a storage unit (e.g., a hard drive) via a storage command. At this time, the abnormal object detection warning system 100 automatically activates the AI ​​image recognition service 103 to identify the abnormal features in the photo 101. The AI ​​image recognition service 103 then transmits the photo 101 containing the abnormal features back, marking the location of the abnormal features on the photo 101. The AI ​​image recognition service 103 may include an artificial intelligence module with a neural network. The AI ​​image recognition service 103 can use the central processing unit or graphics processing unit of the server 102 to detect the abnormal features in the photo 101. Abnormal features may include curved rails and foreign objects on the railway. After receiving the photo 101 containing the abnormal features transmitted back by the AI ​​image recognition service 103, the abnormal object detection warning system 100 will automatically transmit a warning signal for the photo 101 containing the abnormal features through a communication system or network system, or transmit the photo 101 containing the abnormal features to the mobile device equipped by the staff member 104, notifying the staff member 104 to go to the site to check and troubleshoot the problem.

[0011] Please refer to Figure 2, which illustrates a block diagram of a second embodiment of the abnormal object detection warning system 200 according to an embodiment of this disclosure. As shown in Figure 2, the abnormal object detection warning system 200 can be applied to image data 210 captured by an unmanned vehicle performing a track inspection task. The abnormal object detection warning system 200 includes an electronic device 220, which can be one of the following: an embedded system, a smart mobile device, an unmanned vehicle, a personal computer, a server, or a cloud server. The electronic device 220 can receive the image data 210 (e.g., photos or videos) and store the image data 210 in a storage unit 201 (e.g., a hard drive). The electronic device 220 of the abnormal object detection warning system 200 includes system software 230. The system software 230 is stored in a memory unit 202 (e.g., flash memory). Optionally, the system software 230 can also be stored in the storage unit 201. The central processing unit 203 (e.g., a central processing unit) can execute system software 230 to perform image recognition on image data 210 to detect abnormal features in the image data 210. Furthermore, to improve the accuracy of the abnormal object detection warning system 200 in identifying abnormal features, the electronic device 220 can connect to the network 240 via the network unit 204 to update the system software 230 online. Through the online update mechanism, the abnormal object detection warning system 200 can obtain the latest version of the system software 230, thereby improving the accuracy of image recognition.

[0012] In one embodiment, system software 230 may include a detection module 302, which may include a neural network module 304. The neural network module 304 is configured to receive image data 210 and process the image data 210 through a neural network to obtain a prediction output. The detection module 302 generates markers on the image data 210 with abnormal features.

[0013] Please refer to Figure 3, which illustrates a block diagram of a third embodiment of the abnormal object detection and warning system 300 according to an embodiment of this disclosure. As shown in Figure 3, the abnormal object detection and warning system 300 can be applied to unmanned vehicles performing track inspection tasks. The abnormal object detection and warning system 300 includes: a storage module 301, a detection module 302, and a notification module 303. The storage module 301 is configured to store image data 210 captured by the unmanned vehicle. The detection module 302 is configured to detect the image data 210 to identify image data 210 with abnormal features from the image data 210. The detection module 302 includes a neural network module 304, which is configured to receive the image data 210, process the image data 210 through the neural network to obtain a prediction output; the detection module 302 generates a marker on the image data 210 with abnormal features based on the prediction output. Notification module 303 is configured to transmit a warning signal for image data 210 with abnormal characteristics.

[0014] Please refer to Figure 4, which illustrates a flowchart of an abnormal object detection method according to an embodiment of this disclosure. As shown in Figure 4, the abnormal object detection method includes steps S401 to S403. In step S401, the storage module 301 stores image data 210 captured by the unmanned vehicle. In step S402, the detection module 302 detects the image data 210 to identify image data 210 with abnormal features from the image data 210. In step S403, the notification module 303 transmits the image data 210 with abnormal features to the user's mobile device.

[0015] In one embodiment, taking an unmanned vehicle (e.g., an unmanned aerial vehicle, a drone, etc.) as an example, the unmanned vehicle can store the image data 210 captured by the unmanned vehicle in flash memory or a memory card through the storage module 301. The unmanned vehicle can perform real-time image recognition on the image data 210 through the detection module 302 to identify abnormal features (e.g., curved rails and foreign objects on the railway). In one embodiment, the unmanned vehicle can perform local computing, edge computing, or cloud computing on the image data 210 to achieve the purpose of real-time identification of abnormal features. In one embodiment, multiple unmanned vehicles can simultaneously perform inspection tasks on multiple tracks, and work collaboratively through edge computing to share data and computing tasks from multiple unmanned vehicles, thereby accelerating the efficiency of real-time image recognition. In one embodiment, when the abnormal object detection warning system 300 detects abnormal features in the image data 210, the unmanned vehicle can immediately notify or warn the user through the notification module 303, including transmitting a warning signal of the image data 210 with abnormal features, or transmitting the image data 210 with abnormal features to the user's mobile device.

[0016] Please refer to Figure 5, which illustrates a block diagram of a first embodiment of the detection module 302 of an abnormal object detection and warning system 300 according to an embodiment of the present disclosure. As shown in Figure 5, the detection module 302 includes an object detection module 501, an image segmentation module 502, a rail boundary processing module 503, a linear regression processing module 504, and a rail anomaly judgment module 505. The object detection module 501 is configured to detect the rail region in the image data 210 to obtain a rail region image. The image segmentation module 502 is configured to remove non-rail background images from the rail region image to obtain a rail boundary image. The rail boundary processing module 503 is configured to process the rail boundary image to obtain the rail boundary and segment the rail boundary into a plurality of segments. The linear regression processing module 504 is configured to perform linear regression on each of the plurality of segments to obtain the segment slope of each segment and calculate the average slope of the plurality of segments. The rail anomaly detection module 505 is configured to calculate a judgment value based on the section slope and the average slope, and then determine whether the rail is bent based on the judgment value.

[0017] In one embodiment, the detection module 302 includes a neural network module 304, which in turn includes an object detection module 501. The object detection module 501 can be a neural network model for object detection, such as YOLO, R-CNN, etc. A preferred embodiment of the object detection module 501 is trained using YOLO V4, but it is not limited to this model. The trained object detection module 501 is specifically used to capture the region of the railway track in the image data 210 to obtain an image of the railway track region. Since the railway track only occupies a portion of the image data 210, to simplify image processing and reduce the influence of other objects besides the railway track, we only extract the image of the region containing the railway track in the image data 210 (box 1201 as shown in Figure 12). In one embodiment, an image of the middle section of the railway track can be further extracted from the image containing the railway track region (box 1202 as shown in Figure 12). This middle section image clearly includes the railway track and the areas on both sides of the track. In addition, cropping only the middle section of the railway track can reduce the image size, significantly reducing the resolution compared to the original image and greatly reducing the computational load in subsequent steps.

[0018] In one embodiment, the detection module 302 includes a neural network module 304, which in turn includes an image segmentation module 502. The image segmentation module 502 can be a neural network model for image segmentation, such as UNET, MASK R-CNN, etc. A preferred embodiment of the image segmentation module 502 is to use UNET, coupled with EfficientnetB3 as the backbone network. The trained image segmentation module 502 is specifically designed to remove background images other than railway tracks. Since the railway track area image obtained from the object detection module 501 still contains some unwanted background images, the image segmentation module 502 can be used to identify locations containing only railway tracks and remove non-railway background images from the railway track area image to obtain the railway track boundary image (as shown in Figure 8).

[0019] In one embodiment, the rail boundary processing module 503 can process the rail boundary image to obtain the rail boundary. Since the rail boundary image obtained by the image segmentation module 502 has the rail positions on both the left and right sides, the rail boundary processing module 503 can process the rail boundary image to obtain the rail boundaries on both sides (as shown in the selected areas 801 and 802 in Figure 8). If the rail has an abnormal curvature, the rail boundary will also have an abnormal curvature. Therefore, we can determine whether the rail is abnormal based on whether the rail boundary is curved. In addition, the rail boundary processing module 503 will also divide the rail boundaries on both sides into multiple segments. In principle, the more segments there are, the higher the accuracy of determining whether the rail is curved. Usually, the number of segments can be evaluated based on the image size or the computing power of the hardware. One preferred embodiment is to divide the rail boundaries on both sides into 5 segments each.

[0020] In one embodiment, the linear regression processing module 504 can perform linear regression on each of the plurality of segments to obtain the slope of each segment and calculate the average slope of the plurality of segments. That is, the linear regression processing module 504 uses linear regression to calculate the slope of each segment of the left and right rail boundaries. Linear regression finds a linear equation from all points on the rail boundary of each segment, and then sequentially calculates the slope of each linear regression (as shown in Figure 9). Then, the average slope of the plurality of segments of the left and right rail boundaries is calculated separately.

[0021] In one embodiment, the rail anomaly detection module 505 can calculate a detection value based on the slope of a section and the average slope, and determine whether the rail is bent based on the detection value. That is, the rail anomaly detection module 505 includes a formula for calculating the detection value and a threshold value. When the detection value of a section exceeds the threshold value, the image data 210 is determined to have the abnormal characteristic of rail bending; otherwise, it is not. The threshold value is used to adjust the strictness of the detection, and can be changed to adapt to different situations. One preferred embodiment of the threshold value is 0.3. The formula for the detection value is as follows: Where, det_value represents the judgment value; slop[i] represents the slope of the i-th segment; and m represents the average slope.

[0022] Please refer to Figure 6, which illustrates a flowchart of a first embodiment of an abnormal object detection method according to an embodiment of this disclosure. As shown in Figure 6, the abnormal object detection method includes steps S601 to S605. In step S601, the object detection module 501 detects the railway track region in the image data 210 to obtain a railway track region image. In step S602, the image segmentation module 502 removes the non-railway background image from the railway track region image to obtain a railway track boundary image. In step S603, the railway track boundary processing module 503 processes the railway track boundary image to obtain the railway track boundary and divides the railway track boundary into a plurality of segments. In step S604, the linear regression processing module 504 performs linear regression on each of the plurality of segments to obtain the segment slope of each segment and calculates the average slope of the plurality of segments. In step S605, the rail anomaly judgment module 505 calculates the judgment value based on the section slope and the average slope, and judges whether the rail is bent based on the judgment value.

[0023] Please refer to Figure 7, which is a block diagram illustrating the neural network architecture of an object detection module 501 according to an embodiment of the present disclosure. As shown in Figure 7, the neural network architecture of the object detection module 501 includes: an image data input 701, a backbone network layer 702, a neck layer 703, a detection head layer 704, and a prediction output 705. The image data input 701 is configured to provide image data 210 to the backbone network layer 702. The backbone network layer 702 is configured to extract multiple scale features from the image data 210. The neck layer 703 is configured to fuse multiple scale features to improve the ability to detect objects of various sizes. The detection head layer 704 is configured to predict the bounding box of the object and the object category based on the fused multiple scale features. The prediction output 705 is configured to obtain the prediction result of the object classification.

[0024] In one embodiment, a preferred embodiment of the object detection module 501 is YOLO V4 (You Only Look Once-version 4). YOLO V4 is a real-time object detection technology that simultaneously locates and classifies objects in a single network architecture. Due to its high accuracy, speed, and ease of training, it has a wide range of applications, including: defect detection in manufacturing, monitoring systems for access control and intelligent transportation, and object recognition in unmanned vehicles.

[0025] Please refer to Figure 8, which illustrates a schematic diagram of the rail boundary after processing by the rail boundary processing module 503 according to an embodiment of this disclosure. The left side of Figure 8 shows a schematic diagram of the left and right rail boundaries (circled areas 801 and 802 as shown in Figure 8) after processing by the rail boundary processing module 503. The right side of Figure 8 shows a schematic diagram of the rail boundary processing module 503 dividing the left and right rail boundaries into a plurality of segments (e.g., 5 segments).

[0026] Please refer to Figure 9, which illustrates a schematic diagram of the segment slope after processing by the linear regression processing module 504 according to an embodiment of this disclosure. Figure 9 shows a schematic diagram of the segment slope of each segment after processing by the linear regression processing module 504. In one embodiment, if the judgment value of each segment of the left rail boundary does not exceed the threshold value, the left rail is determined to be normal; if the judgment values ​​of the three segments below the right rail boundary exceed the threshold value, the right rail is determined to be abnormally curved.

[0027] Please refer to Figure 10, which illustrates a block diagram of a second embodiment of the detection module 302 of an abnormal object detection warning system 300 according to an embodiment of the present disclosure. As shown in Figure 10, the detection module 302 includes an object detection module 501, a region of interest extraction module 1001, and an anomaly detection module 1002. The object detection module 501 is configured to detect the railway track area in the image data 210 to obtain a railway track area image. The region of interest extraction module 1001 is configured to extract an image of the middle section of the railway track from the railway track area image. The anomaly detection module 1002 is configured to detect abnormal features in the image of the middle section of the railway track, wherein the abnormal features include foreign objects on the railway.

[0028] Please refer to Figures 7 and 10 simultaneously. The object detection module 501 can be a neural network model for object detection. After training, the object detection module 501 is specifically used to capture the area of ​​the railway track in the image data 210 to obtain the railway track area image. In one embodiment, the backbone network layer 702 of the object detection module 501 is a Cross Stage Partial Darknet (CSPDarknet), the neck layer 703 is a combination of Spatial Pyramid Pooling (SPP) and Path Aggregation Network (PAN), and the detection head layer 704 is a YOLO Detection Head (YOLO-Head).

[0029] In one embodiment, the Region of Interest (ROI) extraction module 1001 can extract the middle section image of the railway track (box 1202 as shown in Figure 12) from the railway track area image (box 1202 as shown in Figure 12). Because unmanned vehicles (e.g., drones) experience spatial displacement during flight, the railway position in each image data 210 cannot be fixed in the same area. If only the railway track and the areas on both sides of the track can be cropped, the scenario for subsequent foreign object detection will be simpler and more stable. The railway track and the areas on both sides of the track are the Region of Interest (ROI) of the ROI extraction module 1001. The ROI extraction module 1001 can extract the middle section image of the railway track (box 1202 as shown in Figure 12) from the railway track area image (box 1202 as shown in Figure 12). In one embodiment, the image size extracted by the region of interest extraction module 1001 can be adjusted according to the quality of the image data 210 captured by the unmanned vehicle. The region of interest (ROI) is defined as one that clearly includes the railway track and the areas on both sides of the track. One preferred embodiment is that the image size of the middle section of the railway track is approximately half the size of the image of the entire railway track area.

[0030] Anomaly detection is an unsupervised technique suitable for situations with little or no anomaly data, where normal and anomaly data are similar, or where there is no anomaly data at all. Therefore, it is highly suitable for applications such as automated optical inspection (AOI) in factories. For unsupervised anomaly detection, what constitutes an anomaly depends on what constitutes normal data. Anything not present in the training data is considered an anomaly, allowing anomaly detection to detect any type of anomaly, regardless of whether it has been seen before. While anomaly detection is unaffected by anomaly data, it is highly sensitive to background changes. Any change in the background can cause model instability and be considered an anomaly feature. Therefore, fixing the input within a stable range before anomaly detection can significantly improve its effectiveness. In one embodiment of this invention, the object detection module 501 and the region of interest extraction module 1001 can extract the region of interest (ROI) from the image data 210.

[0031] In one embodiment, the detection module 302 includes a neural network module 304, which in turn includes an anomaly detection module 1002. The anomaly detection module 1002 can be a neural network model for anomaly detection. A preferred embodiment uses Patch Distribution Modeling (PaDiM) and a ResNet (Residual Network) as the backbone network (e.g., ResNet18) to achieve lightweight and fast prediction. Furthermore, to improve prediction accuracy, a larger backbone network can be used for retraining. The anomaly detection module 1002 takes the image of the middle section of the railway track as input and can detect anomalous objects in the middle section image (as shown in the circled area 1401 in Figure 14).

[0032] Please refer to Figure 11, which illustrates a flowchart of a second embodiment of the abnormal object detection method according to an embodiment of the present disclosure. As shown in Figure 11, the abnormal object detection method includes steps S1101 to S1103. In step S1101, the object detection module 501 detects the railway track region in the image data 210 to obtain a railway track region image. In step S1102, the region of interest extraction module 1001 extracts the railway track intermediate section image from the railway track region image. In step S1103, the anomaly detection module 1002 detects abnormal features in the railway track intermediate section image, wherein the anomaly detection module 1002 performs local distribution modeling (PaDiM), and the abnormal features include foreign objects on the railway.

[0033] Please refer to Figure 12, which illustrates a schematic diagram of a region of interest extraction module 1001 according to an embodiment of this disclosure. Figure 12 shows a schematic diagram (box 1202 in Figure 12) of the extracted image of the middle section of the railway track after processing by the region of interest extraction module 1001. The image of the middle section of the railway track clearly includes the railway track and the areas on both sides of the railway track. In addition, the size of the image of the middle section of the railway track is approximately half that of the image of the railway track area (box 1201 in Figure 12).

[0034] Please refer to Figure 13, which illustrates a schematic diagram of an anomaly detection module 1002 according to an embodiment of this disclosure. As shown in Figure 13, PaDiM uses a Convolutional Neural Network (CNN) to extract features from image data. These features are divided into several small patches. The features of these patches are used to build a Gaussian distribution model. The principle of PaDiM anomaly detection is to determine whether two images belong to the same category by comparing each pixel. As long as only normal images are used during training, the model can regard images that do not have the features of normal images as anomalies. First, the input image is divided into multiple patches with a specific height (H) and width (W). Then, features are extracted through a pre-trained CNN, and features at different levels are combined. Finally, these combined features are used to calculate the Gaussian distribution parameters of each patch. Inference is achieved by calculating the Mahalanobis distance between each pixel in the image to be detected and its corresponding Gaussian distribution. The higher the anomaly score, the further the pixel is from the Gaussian distribution trained from normal images, and the more likely it is to be an anomaly. The Gaussian distribution is typically described by two parameters: the covariance matrix (Σ i,j) and the mean vector (μ i,j). These two parameters together define the shape and position of the Gaussian distribution. The mean vector (μ i,j) represents the center position of the data, describing the average value; the covariance matrix (Σ i,j) describes the dispersion of the data and the correlation between different dimensions. The covariance matrix is ​​symmetric; the diagonal elements represent the variation in each dimension, while the off-diagonal elements represent the covariance between different dimensions. In one embodiment, a threshold value can be set; anything exceeding this threshold is considered an anomaly. Using this method, any potentially abnormal locations can be detected. In one embodiment, the threshold value can be set or adjusted according to the actual application scenario to reduce misjudgment or improve the accuracy rate. One preferred embodiment of the threshold value is 0.5.

[0035] Please refer to Figure 14, which illustrates a schematic diagram of abnormal features detected by an anomaly detection module 1002 according to an embodiment of the present disclosure. Figure 14 shows the abnormal features detected by the anomaly detection module 1002 after processing the intermediate segment image, and the circled area 1401 shown in Figure 14 indicates the location of the abnormal object.

[0036] Please refer to Figure 15, which illustrates a schematic diagram of marking anomaly features according to an embodiment of this disclosure. Figure 15 shows that the detection module 302 generates markings on image data 210 with anomaly features based on the prediction output of the anomaly detection module 1002 (as shown in Figure 15). For example, the anomaly detection module 1002 infers that the image data 210 can obtain an anomaly score, and the detection module 302 can add markings to areas with high anomaly scores. The markings can be colors, text, or symbols. In one embodiment, different colors or different shades of the same color can be used to mark the image data 210 according to the anomaly score. The darker the color shade, the more likely it is to be an anomalous object. However, the image marked with color shades looks different from the actual scene. Therefore, the detection module 302 can use a visualization method of adding colored graphic frames (e.g., rectangular frames or circular frames) to the anomalous objects in the original image to select the anomalous objects as much as possible for easy viewing by the user.

[0037] 100: Abnormal Object Detection Warning System 101: Photo 102: Server 103: AI Image Recognition Service 104: Staff 200: Abnormal Object Detection Warning System 201: Storage Unit 202: Memory Unit 203: Central Processing Unit 204: Network Unit 210: Video footage 220: Electronic devices 230: System Software 240: Network 300: Abnormal Object Detection Warning System 301: Storage Module 302: Detection Module 303: Notification Module 304: Neural Network Module 501: Object Detection Module 502: Image Segmentation Module 503: Rail Boundary Processing Module 504: Linear Regression Processing Module 505: Rail Anomaly Detection Module 701: Image Data Input 702: Backbone Network Layer 703: Neck layer 704: Detecting the head layer 705: Predicted Output 801, 802: Selected Area 1001: Region of Interest Extraction Module 1002: Anomaly Detection Module 1201, 1202: Boxes 1401: Selected Area S401~S403, S601~S605, S1101~S1103: Steps

Claims

1. An abnormal object detection and warning system, applicable to image data captured by an unmanned vehicle performing a track inspection task, comprising: a detection module configured to detect the image data and process the image data through a neural network module to obtain a predictive output, wherein the neural network module includes an object detection module and an image segmentation module, the object detection module being configured to detect a railway track region in the image data to obtain a railway track region image, and the image segmentation module being configured to remove non-railway background images from the railway track region image to obtain a railway track boundary image, wherein... The detection module includes a rail boundary processing module, a linear regression processing module, and a rail anomaly detection module. The rail boundary processing module is configured to process the rail boundary image to obtain a rail boundary and divide the rail boundary into a plurality of segments. The linear regression processing module is configured to perform linear regression on each of the plurality of segments to obtain the slope of the plurality of segments and calculate an average slope of the plurality of segments. The rail anomaly detection module is configured to calculate a judgment value based on the segment slope and the average slope, and determine whether the rail has an abnormal feature of rail curvature based on the judgment value. The detection module is further configured to generate a mark on the image data with the abnormal feature based on the prediction output to identify the image data with the abnormal feature from the image data. A notification module is configured to transmit a warning signal with the abnormal feature.

2. The abnormal object detection and warning system as described in claim 1, wherein the object detection module comprises: a backbone network layer configured to extract multiple scale features from the image data; a neck layer connected to the backbone network layer and configured to fuse the multiple scale features to improve the ability to detect objects of various sizes; and a detection head layer connected to the neck layer and configured to predict the bounding box of the object and the category of the object based on the fused multiple scale features; wherein the backbone network layer is a Cross-Stage Partial Darknet (CSPDarknet), the neck layer is a combination of Spatial Pyramid Pooling (SPP) and Path Aggregation Network (PAN), and the detection head layer is a YOLO Detection Head (YOLO-Head).

3. The abnormal object detection and warning system as described in claim 2, wherein the detection module includes a region of interest extraction module configured to extract an image of a middle section of the railway track from the image of the railway track area.

4. The abnormal object detection and warning system as described in claim 3, wherein the neural network module includes an abnormality detection module configured to detect the abnormal feature in the image of the middle section of the railway track, wherein the abnormal feature includes a foreign object on the railway.

5. The abnormal object detection and warning system as described in claim 1, further comprising a storage module configured to store the image data captured by the unmanned vehicle.

6. An abnormal object detection method, comprising: (a) storing image data captured by an unmanned vehicle in a storage module; (b) detecting the image data in a detection module and processing the image data through a neural network module to obtain a prediction output; (b1) detecting a railway track region in the image data in an object detection module to obtain a railway track region image; (b2) removing non-railway background images from the railway track region image in an image segmentation module to obtain a railway track boundary image; (b3) processing the railway track boundary image in a railway track boundary processing module to obtain a railway track boundary and segmenting the railway track boundary into a plurality of segments; (b4) performing linear regression on each of the plurality of segments in a linear regression processing module to obtain the slope of the plurality of segments and calculating an average slope of the plurality of segments. (b5) A rail anomaly detection module calculates a detection value based on the slope of the section and the average slope, and determines whether the rail has an abnormal characteristic of rail curvature based on the detection value. The detection module generates a marker on the image data with the anomalous feature based on the prediction output, so as to identify the image data with the anomalous feature from the image data; and (c) the image data with the anomalous feature is transmitted to the user's mobile device by a notification module.

7. The abnormal object detection method as described in claim 6, wherein the neural network architecture of the object detection module includes a backbone network layer, a neck layer, and a detection head layer, wherein the neck layer is connected to the backbone network layer, and the detection head layer is connected to the neck layer, wherein step (d) includes: (d1) extracting multiple scale features from the image data by the backbone network layer; (d2) fusing the multiple scale features by the neck layer to improve the ability to detect objects of various sizes; and (d3) predicting the bounding box of the object and predicting the category of the object by the detection head layer based on the fused multiple scale features; wherein the backbone network layer is a Cross-Stage Partial Darknet (CSPDarknet), the neck layer is a combination of Spatial Pyramid Pooling (SPP) and Path Aggregation Network (PAN), and the detection head layer is a YOLO Detection Head (YOLO-Head).

8. The abnormal object detection method as described in claim 7, wherein the detection module includes a region of interest extraction module, and step (b) includes: (b6) extracting a middle section image of the railway track from the railway track area image by the region of interest extraction module.

9. The abnormal object detection method as described in claim 8, wherein the neural network module includes an anomaly detection module, and step (b) includes: (b7) detecting the abnormal feature of the intermediate section image of the railway track by the anomaly detection module, wherein the anomaly detection module is a local distribution modeling (PaDiM), and the abnormal feature includes a foreign object on the railway.

10. An electronic device comprising: a storage unit for storing image data captured by an unmanned vehicle; and a memory unit for storing system software for execution by the electronic device, the system software comprising: A detection module is configured to detect image data and process the image data through a neural network module to obtain a prediction output. The neural network module includes an object detection module and an image segmentation module. The object detection module is configured to detect a railway track region in the image data to obtain a railway track region image. The image segmentation module is configured to remove non-railway background images from the railway track region image to obtain a railway track boundary image. The detection module includes a railway track boundary processing module, a linear regression processing module, and a railway track anomaly detection module. The railway track boundary processing module is configured to process the railway track boundary image to obtain a railway track boundary and then... The railway track boundary is divided into multiple segments. A linear regression processing module is configured to perform linear regression on each of the multiple segments to obtain the slopes of the multiple segments and calculate an average slope of the multiple segments. A railway track anomaly detection module is configured to calculate a judgment value based on the segment slope and the average slope, and to determine whether the railway track has an abnormal feature of track curvature based on the judgment value. The detection module is further configured to generate a mark on the image data with the abnormal feature based on the prediction output to identify the image data with the abnormal feature from the image data. A network unit is used to transmit the image data with the abnormal feature. The electronic device updates the system software online through the network unit.

Images