Railway area work machine abnormality detection method and device based on image data

Abstract

The application discloses a kind of based on image data's track area abnormal detection method and device of machine tool, it is related to image recognition technical field.The method includes: using image recognition algorithm, introduce multi-objective optimization algorithm, build machine tool target detection model, and connect to candidate region segmentation engine and space-time correlation filter engine;Video stream of track area is collected, and pretreatment is carried out, and the image data of continuous frame is obtained;The image data of continuous frame is input to candidate region segmentation engine, and the ROI region of each frame of image data is obtained;ROI region is input to machine tool target detection model, and the detection frame of machine tool target is obtained;The detection frame of machine tool target is input to space-time correlation filter engine, identifies stationary effective target, and carries out abnormal detection to stationary effective target.The application solves the problems of low detection accuracy, difficult model deployment and poor anti-interference ability in the prior art.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and apparatus for detecting abnormalities in track-bound machinery based on image data. Background Technology

[0002] The track area of ​​railways and urban rail transit is a critical area for train operation. During track construction, maintenance, and other operations, workers need to use various tools and equipment (such as wrenches, crowbars, rail clamps, etc.). If these tools and equipment are not removed from the track area after the work is completed, the leftover tools and equipment may cause serious safety accidents such as train derailment. Therefore, it is of great significance to realize the automatic detection and alarm of tools and equipment left in the track area.

[0003] Traditional inspection methods rely primarily on manual inspection, which is not only inefficient but also prone to missed detections in inclement weather or at night. While computer vision-based automated inspection technology has improved efficiency to some extent, it still faces many challenges: 1) Low detection accuracy: The track area has a complex environment with complex backgrounds such as rails, fasteners, and ballast, and the lighting conditions change drastically (such as shadows in the tunnel, low light at night, and strong light reflection), making it difficult to extract the features of tools and equipment.

[0004] 2) Difficulty in model deployment: Existing target detection models typically have fixed network structures, making it difficult to strike a balance between detection accuracy and inference speed. If the model is too complex, it cannot meet real-time requirements on edge computing devices; if the model is too simple, the false negative rate is high in complex backgrounds.

[0005] 3) Poor anti-interference ability: Detection based solely on a single frame image is easily affected by interference factors such as flying birds, changes in light and shadow, and swaying weeds, leading to frequent false alarms and a high false alarm rate.

[0006] Therefore, there is an urgent need for a method for detecting abnormalities in track-mounted machinery that can adapt to complex lighting and backgrounds, balance accuracy and speed, and effectively eliminate dynamic interference. Summary of the Invention

[0007] This invention provides a method and apparatus for detecting anomalies in track-mounted machinery based on image data. This invention solves the problems of low detection accuracy, difficult model deployment, and poor anti-interference capability in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide a method for detecting anomalies in track area machinery based on image data, the method comprising: Using image recognition algorithms, a multi-objective optimization algorithm is introduced to construct a tool and equipment target detection model, and the tool and equipment anomaly detection model is connected to the candidate region segmentation engine and the spatiotemporal correlation filtering engine. The video stream of the track area is acquired and preprocessed to obtain image data of consecutive frames; Image data from consecutive frames is input into a candidate region segmentation engine to perform background difference-guided candidate region segmentation, thereby obtaining the ROI region of each frame of image data. The ROI regions of consecutive frame image data are input into the tool and equipment target detection model to perform tool and equipment target detection, and the detection box of the tool and equipment target in each frame of image data is obtained. The detection bounding boxes of the machine targets in the continuous frame image data are input into the spatiotemporal correlation filtering engine for spatiotemporal correlation filtering, to identify stationary valid targets, and to perform anomaly detection on the stationary valid targets.

[0009] The technical solution provided in this application has at least the following beneficial effects: By incorporating semantic enhancement, detail compensation, and attention weighting modules into the feature fusion neck network of the tool and equipment target detection model, and combining these with image preprocessing techniques, the edge texture and semantic features of tools and equipment against complex backgrounds such as railway tracks and fasteners can be effectively enhanced, while suppressing background noise. This allows the model to accurately extract tool and equipment features even under extreme lighting conditions such as tunnel shadows, low nighttime illumination, and strong light reflection, significantly reducing the false negative rate caused by environmental factors. A multi-objective alligator optimization algorithm is used to construct a multi-objective optimization function, simultaneously optimizing for both detection accuracy loss and model complexity (number of parameters, floating-point operations). Weighting is dynamically set based on the actual server operating status, enabling the solution to effectively detect edge-related features. When computing resources are limited, the system automatically prioritizes lightweight models to ensure real-time performance, while when resources are plentiful, it prioritizes high-precision models to ensure detection effectiveness. This achieves the best balance between accuracy and speed under different hardware environments. A spatiotemporal correlation filtering engine is introduced, combining Kalman filter prediction and Hungarian matching algorithms for target tracking. A counter is used to count the occurrence and position change rate of the target in consecutive frames. Only when the target exists continuously for multiple consecutive frames and its position is basically stationary is it determined to be a valid stationary target. This mechanism effectively filters out transient dynamic interference such as birds flying by, weeds swaying, and instantaneous changes in light and shadow, solving the problem of high false alarm rates and frequent false alarms that are easily caused by relying solely on single-frame image detection.

[0010] In one alternative implementation, an image recognition algorithm is used, a multi-objective optimization algorithm is introduced, a tool and equipment target detection model is constructed, and the tool and equipment anomaly detection model is connected to a candidate region segmentation engine and a spatiotemporal correlation filtering engine, including: Using image recognition algorithms, an initial tool and equipment target detection model is constructed. The tool and equipment target detection model includes a backbone network, a feature fusion neck network, and a detection head. The feature fusion neck network is equipped with a semantic enhancement module, a detail compensation module, a feature fusion module, and an attention weighting module. Construct a multi-objective optimization function for a multi-objective optimization algorithm, which includes a detection accuracy loss function and a model complexity function; Based on a multi-objective optimization function, a multi-objective optimization algorithm is used to optimize the hyperparameters of the initial tool and equipment target detection model, resulting in the final tool and equipment target detection model. The input of the final tool and equipment anomaly detection model is connected to the candidate region segmentation engine, and the output of the final tool and equipment anomaly detection model is connected to the spatiotemporal correlation filtering engine.

[0011] In one alternative implementation, the multi-objective optimization algorithm is the multi-objective alligator optimization algorithm, and the formula for the multi-objective optimization function of the multi-objective alligator optimization algorithm is:

[0012] In the formula, For hyperparameters The corresponding optimized target value of the tool and equipment target detection model on the validation set; For hyperparameters The detection accuracy loss value of the corresponding tool and equipment target detection model on the validation set; For hyperparameters The model complexity of the corresponding tool and equipment target detection model;

[0013] In the formula, The bounding box regression loss value; This is the classification loss value; Weights are assigned to account for the loss of detection accuracy.

[0014] In the formula, This represents the total number of model parameters. The hardware must withstand an upper limit threshold for model parameters; This represents the number of floating-point operations. The upper limit threshold for the hardware to withstand floating-point operations; These are the model complexity weights.

[0015] In one alternative implementation, based on a multi-objective optimization function, a multi-objective optimization algorithm is used to optimize the hyperparameters of the initial tool and equipment target detection model to obtain the final tool and equipment target detection model, including: Several historical images of consecutive frames in the track area are collected, preprocessed and labeled to obtain several sample images, and these sample images are divided into training set and validation set. The hyperparameters of the tool and equipment target detection model are encoded into individual vectors. A chaotic sequence is generated using a Logistic mapping, and the chaotic sequence is mapped to the solution space of the individual vectors to obtain the initial crocodile population. Based on the candidate hyperparameters corresponding to each initial crocodile individual in the initial crocodile population, the corresponding candidate tool and machinery target detection model is trained using the training set, and the corresponding fitness value is calculated using a multi-objective optimization function based on the validation set. Based on fitness values, the initial crocodile population was divided into an initial raiding population and an initial lurking population in a 3:7 ratio according to ascending power sort. Based on the fitness value, the initial raiding population is subjected to fast non-dominated sorting to obtain different front surface levels. Several initial raiding individuals in the first front surface level whose crowding distance is greater than the crowding distance threshold are stored as non-dominated solutions in the external archive set. By introducing the Levy flight mechanism and the gray wolf cooperative concept, the initial lurking population is updated in position to obtain the updated first crocodile population; By introducing a convergence factor, the initial attacking population is updated in position to obtain an updated second crocodile population. The updated first crocodile population and the updated second crocodile population are integrated to obtain an updated crocodile population. A multi-objective optimization function is used to calculate the fitness value of each updated crocodile individual in the updated crocodile population. Based on the fitness value, update the frontier surface hierarchy and external archive set using the updated crocodile individuals; Until the number of iterations reaches the iteration threshold, all updated crocodile individuals in the external archive set are output as the Pareto non-dominated optimal solution set; Based on the operating status data of the server where the machine tool anomaly detection model is located, optimize the weighting weights, and based on the optimized weighting weights, optimize the fitness value of each solution in the Pareto non-dominated optimal solution set to obtain the weighted fitness value. The individual vectors of the solutions corresponding to the optimal weighted fitness values ​​are decoded to obtain the optimal hyperparameters. Based on the optimal hyperparameters, the hyperparameters of the initial tool and equipment target detection model are optimized to obtain the optimized tool and equipment target detection model. Based on several sample images, the optimized tool and equipment target detection model is fine-tuned and trained to obtain the final tool and equipment target detection model.

[0016] In one alternative implementation, the candidate region segmentation engine includes a background modeling module, a difference calculation module, a threshold segmentation module, and a ROI extraction module. The spatiotemporal correlation filtering engine includes a target tracking module, a spatiotemporal correlation module, and an interference target filtering module.

[0017] In one optional implementation, a video stream of the track area is acquired and preprocessed to obtain image data of consecutive frames, including: High-definition industrial cameras deployed in the track area are used to capture video streams of the track area, and the video streams are frame-by-frame extracted to obtain raw images of consecutive frames. The keyframes of the original images of consecutive frames are extracted using the inter-frame difference method based on the hash algorithm, and the original keyframe images of the consecutive frames are obtained. The original image of each keyframe is preprocessed to obtain image data of consecutive frames.

[0018] In one optional implementation, image data from consecutive frames is input into a candidate region segmentation engine to perform background difference-guided candidate region segmentation, obtaining the ROI region of each frame's image data, including: The background modeling module of the candidate region segmentation engine constructs a background model of the track area based on several historical images of consecutive frames during the no-work period. The image data of the current frame is input into the candidate region segmentation engine, and the difference calculation module of the candidate region segmentation engine is used to calculate the difference image between the image data of the current frame and the background model. Using the threshold segmentation module of the candidate region segmentation engine, Otsu adaptive threshold segmentation is performed on the difference image of the current frame to obtain a binarized mask. The ROI extraction module of the candidate region segmentation engine is used to perform morphological processing on the binarized mask and extract the minimum bounding rectangle of all connected components as the ROI region of the image data of the current frame. By traversing all frames of image data, background difference-guided candidate region segmentation is performed to obtain the ROI region of each frame of image data.

[0019] In one optional implementation, the ROI regions of consecutive frame image data are input into a tool and equipment target detection model to perform tool and equipment target detection, obtaining the detection bounding box of the tool and equipment target in each frame of image data, including: Input the ROI region of the current frame's image data into the tool target detection model; Using the backbone network of the tool and equipment target detection model, low-level image features, mid-level image features, and high-level image features of the current frame image data are extracted. The semantic enhancement module of the neck network, which is used to fuse the features of the tool and equipment target detection model, is used to semantically enhance the mid-level and high-level features to obtain semantic features. The feature fusion neck network detail compensation module of the tool and equipment target detection model is used to perform spatial detail compensation on semantic features, low-level features and high-level features to obtain compensated image features. The feature fusion module of the neck network of the tool and equipment target detection model is used to fuse semantic features and compensated depth features to obtain fused image features. The attention weighting module of the neck network is used to fuse the features of the tool and equipment target detection model to generate an attention weight map of the fused image features, and then weighted with the fused image features to obtain weighted image features. Using the detection head of the tool and equipment target detection model, based on weighted image features, tool and equipment target detection is performed to obtain the coordinates, class probability, and confidence of the detection box of the tool and equipment target in the current frame image data; Traverse the ROI regions of all frames of image data, perform tool and equipment target detection, and obtain the detection box of the tool and equipment target in each frame of image data.

[0020] In one optional implementation, the detection bounding boxes of the machine target in the image data of consecutive frames are input into a spatiotemporal correlation filtering engine for spatiotemporal correlation filtering to identify stationary valid targets and perform anomaly detection on the stationary valid targets, including: Input the detection bounding box of the tool target in the image data of the current frame into the spatiotemporal correlation filtering engine; The Kalman filter of the target tracking module of the spatiotemporal correlation filtering engine is used to track the detection box of the tool target in the current frame and predict the predicted position of the tool target in the next frame. The spatiotemporal correlation module of the spatiotemporal correlation filtering engine performs Hungarian matching on the coordinates of the detection box of the tool target in the current frame and the coordinates of the predicted position in the next frame, and assigns a unique target ID to the same tool target that is successfully matched. The detection boxes of the machine targets in the image data of consecutive frames are traversed, and spatiotemporal correlation filtering is performed to obtain the target IDs of several machine targets. The counter of the interference target filtering module of the spatiotemporal correlation filtering engine is used to count the consecutive frames of the same target ID to obtain the corresponding number of consecutive frames. If the number of consecutive frames with a target ID is greater than the number threshold, and the rate of change of the target position of the detection box of the target ID is less than the rate of change threshold, then the tool target corresponding to the target ID is regarded as a stationary valid target and proceeds to the next step; otherwise, the detection box of the tool target corresponding to the target ID is canceled. Map the coordinates of stationary valid targets to the physical coordinate system to obtain physical coordinates, and detect whether the physical coordinates intrude into the electronic fence of the orbital area. If so, a tooling anomaly occurs, and a tooling anomaly detection report is generated based on the physical coordinates of the stationary valid target, the corresponding category probability, and the confidence level.

[0021] Secondly, embodiments of the present invention provide a track area tool anomaly detection device based on image data, used to implement a track area tool anomaly detection method, the device comprising: The model building unit is used to construct a tool and equipment target detection model by using image recognition algorithms and introducing multi-objective optimization algorithms, and connects the tool and equipment anomaly detection model to the candidate region segmentation engine and the spatiotemporal correlation filtering engine. The preprocessing unit is used to acquire the video stream of the track area and preprocess the video stream to obtain image data of consecutive frames; The candidate region segmentation unit is used to input image data from consecutive frames into the candidate region segmentation engine to perform background difference-guided candidate region segmentation and obtain the ROI region of each frame of image data. The target detection unit is used to input the ROI region of the image data of consecutive frames into the tool target detection model to perform tool target detection and obtain the detection box of the tool target in each frame of image data; The anomaly detection unit is used to input the detection box of the tool target in the image data of consecutive frames into the spatiotemporal correlation filtering engine, perform spatiotemporal correlation filtering, identify stationary valid targets, and perform anomaly detection on the stationary valid targets.

[0022] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0023] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a method for detecting abnormalities in track area tools based on image data, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the functional units of a track area tool anomaly detection device based on image data provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0026] The present invention will be further described below with reference to the accompanying drawings.

[0027] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0028] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0029] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0030] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for a track area tool anomaly detection device based on image data.

[0031] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the track area tool anomaly detection device based on image data stored in the memory 1005 through the processor 1001, and executes the track area tool anomaly detection method based on image data provided in the embodiment of the present invention.

[0032] Reference Figure 2 The present invention provides a method for detecting anomalies in track area machinery based on image data, the method comprising: S201: Using image recognition algorithms, introducing multi-objective optimization algorithms, constructing a tool and equipment target detection model, and connecting the tool and equipment anomaly detection model to the candidate region segmentation engine and the spatiotemporal correlation filtering engine; S202: Acquire video stream from the track area and preprocess the video stream to obtain image data of consecutive frames; S203: Input the image data of consecutive frames into the candidate region segmentation engine to perform background difference-guided candidate region segmentation to obtain the region of interest (ROI) of each frame of image data. S204: Input the ROI region of the image data of consecutive frames into the tool and equipment target detection model to perform tool and equipment target detection and obtain the detection box of the tool and equipment target in each frame of image data; S205: Input the detection bounding box of the machine target in the image data of consecutive frames into the spatiotemporal correlation filtering engine to perform spatiotemporal correlation filtering, identify stationary valid targets, and perform anomaly detection on stationary valid targets.

[0033] The technical solution provided in this application has at least the following beneficial effects: By incorporating semantic enhancement, detail compensation, and attention weighting modules into the feature fusion neck network of the tool and equipment target detection model, and combining these with image preprocessing techniques, the edge texture and semantic features of tools and equipment against complex backgrounds such as railway tracks and fasteners can be effectively enhanced, while suppressing background noise. This allows the model to accurately extract tool and equipment features even under extreme lighting conditions such as tunnel shadows, low nighttime illumination, and strong light reflection, significantly reducing the false negative rate caused by environmental factors. A multi-objective alligator optimization algorithm is used to construct a multi-objective optimization function, simultaneously optimizing for both detection accuracy loss and model complexity (number of parameters, floating-point operations). Weighting is dynamically set based on the actual server operating status, enabling the solution to effectively detect edge-related features. When computing resources are limited, the system automatically prioritizes lightweight models to ensure real-time performance, while when resources are plentiful, it prioritizes high-precision models to ensure detection effectiveness. This achieves the best balance between accuracy and speed under different hardware environments. A spatiotemporal correlation filtering engine is introduced, combining Kalman filter prediction and Hungarian matching algorithms for target tracking. A counter is used to count the occurrence and position change rate of the target in consecutive frames. Only when the target exists continuously for multiple consecutive frames and its position is basically stationary is it determined to be a valid stationary target. This mechanism effectively filters out transient dynamic interference such as birds flying by, weeds swaying, and instantaneous changes in light and shadow, solving the problem of high false alarm rates and frequent false alarms that are easily caused by relying solely on single-frame image detection.

[0034] In one alternative implementation, an image recognition algorithm is used, a multi-objective optimization algorithm is introduced, a tool and equipment target detection model is constructed, and the tool and equipment anomaly detection model is connected to a candidate region segmentation engine and a spatiotemporal correlation filtering engine, including: S2011: Using image recognition algorithms, an initial tool and equipment target detection model is constructed. The tool and equipment target detection model includes a backbone network, a feature fusion neck network, and a detection head. The feature fusion neck network is equipped with a semantic enhancement module, a detail compensation module, a feature fusion module, and an attention weighting module. In this embodiment, the backbone network is constructed as follows: a deep convolutional neural network (such as CrossStage Partial Darknet (CSPDarknet) or MobileNet) is selected as the feature extraction backbone; this network is responsible for extracting low-level (edges, textures), mid-level (shapes, local structures) and high-level (semantics, object categories) image features from the input image. Feature fusion neck network design: A neck network is connected after the backbone network, and this network specifically includes the following four sub-modules: Semantic enhancement module: Employs dilated convolution or global attention mechanisms to expand the receptive field and enhance the semantic understanding of tool categories by mid-to-high-level features; Detail compensation module: Introduces skip connections or feature pyramid networks (FPN) to fuse low-level high-resolution features in the backbone network with high-level features to compensate for spatial detail information lost during downsampling (such as fine tool outlines). Feature fusion module: Using concatenation or element-wise addition operations, semantically enhanced features and detail-compensated features are deeply integrated along the channel dimension. Attention weighting module: Introducing a squeeze-and-excitation block (SE-Block) or a convolutional block attention module (CBAM) to automatically learn the importance weight of each channel in the fused feature map, weight the key feature channels, and suppress background noise channels; Detection head construction: Construct a detection head containing convolutional layers and fully connected layers to output the final prediction results, including the coordinates of the target bounding box, the class probability and its confidence score; S2012: Construct the multi-objective optimization function of the multi-objective optimization algorithm, which includes the detection accuracy loss function and the model complexity; S2013: Based on a multi-objective optimization function, a multi-objective optimization algorithm is used to optimize the hyperparameters of the initial tool and equipment target detection model to obtain the final tool and equipment target detection model; S2014: Connect the input of the final tool and equipment anomaly detection model to the candidate region segmentation engine, and connect the output of the final tool and equipment anomaly detection model to the spatiotemporal correlation filtering engine.

[0035] In one alternative implementation, the multi-objective optimization algorithm is the multi-objective alligator optimization algorithm, and the formula for the multi-objective optimization function of the multi-objective alligator optimization algorithm is:

[0036] In the formula, For hyperparameters The corresponding optimized target value of the tool and equipment target detection model on the validation set; For hyperparameters The detection accuracy loss value of the corresponding tool and equipment target detection model on the validation set; For hyperparameters The model complexity of the corresponding tool and equipment target detection model;

[0037] In the formula, The bounding box regression loss value is the Complete Intersection over Union (CIoULoss) loss, which is used to accurately regress the bounding box. The classification loss value is represented by the binary cross-entropy loss (BCE Loss), which measures the classification accuracy. Weights are used to detect the loss of accuracy, and are used to balance regression and classification tasks;

[0038] In the formula, This represents the total number of model parameters. The hardware must withstand an upper limit threshold for model parameters; This represents the number of floating-point operations. The upper limit threshold for the hardware to withstand floating-point operations; Weights for model complexity; A multi-objective alligator optimization algorithm is used to optimize the model hyperparameters, taking into account both the loss of detection accuracy and the model complexity.

[0039] In one alternative implementation, based on a multi-objective optimization function, a multi-objective optimization algorithm is used to optimize the hyperparameters of the initial tool and equipment target detection model to obtain the final tool and equipment target detection model, including: S20131: Collect several historical images of consecutive frames in the track area, preprocess and add labels to the historical images to obtain several sample images, and divide the sample images into training set and validation set; In this embodiment, the historical images cover raw image data from different time periods (morning light, direct sunlight, night tunnel, dusk) and different weather conditions (sunny, rainy, snowy); Preprocessing includes: Histogram equalization: To address uneven lighting conditions (such as backlighting or nighttime), contrast-limited adaptive histogram equalization is applied to the brightness channel (V channel or grayscale image) of the image data to enhance the local contrast of the image and make the texture of objects in dark areas clearer. Denoising: Gaussian blur or bilateral filtering is applied to the image data to remove high-frequency noise generated by high-definition industrial cameras while preserving target edge information; Distortion correction: If the high-definition industrial camera is a wide-angle or fisheye lens, distortion correction of the image data is required based on the preset camera intrinsic parameter matrix and distortion coefficients to restore the true geometric shape of the target. Size normalization and padding: Proportional scaling: Calculate the scaling ratio of the length and width of the image data to the preset model input size, keep the aspect ratio unchanged, and scale the long side of the image data to match the original size; Edge padding: Calculate the pixel difference of the image data in another dimension, and use a fixed color value (such as gray (114) or black (0)) to pad the right or bottom edge of the image to generate an image with a strictly square size; Data normalization: Numerical normalization: Normalizing the pixel values ​​of a square image from 0... The integer range of 255 is linearly mapped to 0.0. The floating-point range is 1.0; Standardization: Subtract the mean of the ImageNet dataset and divide by the standard deviation to obtain the preprocessed historical image data; Labeling: Use the LabelImg tool to add rectangular boxes to the tools (such as wrenches, crowbars, rail clamps, etc.) in the preprocessed historical image data, and generate corresponding XML or TXT tag files; S20132: Encode the hyperparameters of the tool and equipment target detection model into individual vectors, use Logistic mapping to generate chaotic sequences, and map the chaotic sequences to the solution space of the individual vectors to obtain the initial crocodile population; The formula is:

[0040] In the formula, For the first n+ 1. n One chaotic variable; The stability coefficient is typically 4. This sequence is ergodic and random, ensuring that the initial population is uniformly distributed in the solution space, avoiding getting trapped in local optima, which is superior to traditional random initialization. n Indicator of chaotic variables;

[0041] In the formula, The first in the initial crocodile population i The initial individual crocodile; For the first i One chaotic variable; These are the upper and lower bounds of the search space; For individual indicators; Introducing a Logistic chaotic mapping to initialize the population improves the algorithm's global search capability and convergence speed, and avoids getting trapped in local optima; S20133: Based on the candidate hyperparameters corresponding to each initial crocodile individual in the initial crocodile population, train the corresponding candidate tool and machinery target detection model using the training set, and calculate the corresponding fitness value using a multi-objective optimization function based on the validation set. S20134: Based on fitness values, the initial crocodile population is divided into an initial raiding population and an initial lurking population in a 3:7 ratio according to ascending power sort. S20135: Based on the fitness value, perform a fast non-dominated sort on the initial attack population to obtain different front surface levels, and store several initial attack individuals in the first front surface level whose crowding distance is greater than the crowding distance threshold as non-dominated solutions in the external archive set. S20136: Introducing the Levy flight mechanism and the gray wolf cooperative concept, the initial lurking population is updated to obtain the updated first crocodile population; In the external archive set, based on fitness values, the top three solutions are identified and defined as Alpha individuals, Beta individuals, and Delta individuals in the Gray Wolf Collaboration Idea, respectively. The formula is:

[0042] In the formula, Number of iterations t+ 1 of i The first updated crocodile individual; Number of iterations t The i A new set of latent individuals, the initial latent individuals in the first iteration; for Levy Distribute random numbers; b for Levy Step length, and b ∈[1,2]; Number of iterations t Alpha, Beta, and Delta individuals; The cooperation coefficient; A random number between (0, 1); For flight coefficient; t The iteration count is used as an indicator; the Levy flight mechanism and gray wolf cooperation concept are introduced into the latent population update, and the search is guided by Alpha, Beta and Delta individuals, which enhances the global exploration capability; S20137: Introduce a convergence factor to update the position of the initial raiding population and obtain the updated second crocodile population; The formula is:

[0043] In the formula, Number of iterations t+ 1 of i A newer second crocodile individual; A random number between (0, 1); Number of iterations t The convergence factor improves the attack step size;

[0044] In the formula, These are the maximum and minimum values ​​of the convergence factor; This is the threshold for the number of iterations; , To adjust the parameters; The hyperbolic tangent function is used; a convergence factor based on the hyperbolic tangent function is introduced in the raid population update, which enhances the local exploitation capability; S20138: Integrate the updated first crocodile population and the updated second crocodile population to obtain an updated crocodile population. Use a multi-objective optimization function to calculate the fitness value of each updated crocodile individual in the updated crocodile population. S20139: Update the frontier hierarchy and external archive set using the updated crocodile individuals based on fitness values; S201310: Until the number of iterations reaches the iteration threshold, output all updated crocodile individuals in the external archive as the Pareto non-dominated optimal solution set; S201311: Based on the operating status data of the server where the machine tool anomaly detection model is located, set the optimized weighting weight, and based on the optimized weighting weight, optimize the fitness value of each solution in the Pareto non-dominated optimal solution set to obtain the weighted fitness value. In this embodiment, specific weighting weights are set according to the actual operating status of the server (such as GPU load and power consumption limits). For example, if the GPU load is much greater than the normal range, the optimization weighting weight is set to... At this point, the algorithm will strictly penalize solutions with high complexity and prioritize solutions with fewer model parameters and faster inference speed (low floating-point operation) to meet real-time requirements; If the GPU load is significantly lower than the normal GPU load range, then set the optimization weighting to [value]. At this point, the algorithm will tend to choose the solution with a large number of model parameters, complex calculations, but extremely high accuracy; If the GPU load is close to or within the normal range, then set the optimization weighting to [value]. At this point, the algorithm seeks the optimal balance between accuracy and speed; The formula is:

[0045] In the formula, For the first i Group hyperparameters The corresponding weighted fitness value; For the first i Group hyperparameters The corresponding detection accuracy loss value; For the first i Group hyperparameters The corresponding model complexity; To optimize the weighting, the weighting is dynamically set according to the actual running status of the server, and the model configuration most suitable for the current hardware environment is adaptively selected from the Pareto non-dominated optimal solution set, thus achieving the best match between detection performance and computing resources. S201312: Decode the individual vector of the solution corresponding to the optimal weighted fitness value to obtain the optimal hyperparameters, and optimize the hyperparameters of the initial tool and equipment target detection model based on the optimal hyperparameters to obtain the optimized tool and equipment target detection model. S201313: Based on several sample images, fine-tune and train the optimized tool and equipment target detection model to obtain the final tool and equipment target detection model.

[0046] In one alternative implementation, the candidate region segmentation engine includes a background modeling module, a difference calculation module, a threshold segmentation module, and a ROI extraction module. The spatiotemporal correlation filtering engine includes a target tracking module, a spatiotemporal correlation module, and an interference target filtering module.

[0047] In one optional implementation, a video stream of the track area is acquired and preprocessed to obtain image data of consecutive frames, including: S2021: Use high-definition industrial cameras deployed in the track area to capture video streams of the track area, and extract frames from the video stream to obtain raw images of continuous frames; In this embodiment, multiple high-definition industrial cameras are deployed to cover the blind spots of the track area. The video stream is pulled in real time through the RTSP or ONVIF protocol, and the original image sequence is captured at a frame rate of 25 FPS. S2022: Using the inter-frame difference method based on hash algorithm, keyframes are extracted from the original images of consecutive frames to obtain the original keyframe images of consecutive frames. In this embodiment, the perceptual hash value of two adjacent original images is calculated, and the Hamming distance of the perceptual hash value is calculated. If the distance is greater than the set change threshold, the next frame is determined to be a key frame; otherwise, it is considered a redundant frame and discarded, and the key frame original images of consecutive frames are output. S2023: Preprocess the original image of each keyframe to obtain image data of consecutive frames; In this embodiment, denoising, distortion correction, and normalization operations are performed on keyframes to generate image data of continuous frames that meet the model input requirements.

[0048] In one optional implementation, image data from consecutive frames is input into a candidate region segmentation engine to perform background difference-guided candidate region segmentation, obtaining the ROI region of each frame's image data, including: S2031: Background modeling module using candidate region segmentation engine, constructs background model of track area based on several historical images of consecutive frames during no-operation periods; In this embodiment, 100-200 frames of images are collected during periods when no trains pass by and no personnel are working. A static background model of the track area is constructed using a Gaussian mixture model or the mean method. S2032: Input the image data of the current frame into the candidate region segmentation engine, and use the difference calculation module of the candidate region segmentation engine to perform pixel-level subtraction between the image data of the current frame and the background model, and calculate the difference image between the image data of the current frame and the background model. S2033: Using the threshold segmentation module of the candidate region segmentation engine, perform Otsu adaptive threshold segmentation on the difference image of the current frame to obtain a binarized mask. In this embodiment, the grayscale histogram of the difference image is calculated, the optimal segmentation threshold is automatically calculated using the Otsu method, the difference image is binarized, and a binarization mask (foreground is white, background is black) is generated. S2034: Using the ROI extraction module of the candidate region segmentation engine, morphological processing is performed on the binarized mask to extract the minimum bounding rectangle of all connected components as the ROI region of the image data of the current frame; In this embodiment, a morphological opening operation (erosion followed by dilation) is performed on the binarized mask to remove isolated noise points. A morphological closing operation (dilation followed by erosion) is performed to fill the holes inside the target. All connected components in the mask are found, the minimum bounding rectangle of each connected component is calculated, and the rectangular coordinate region is cropped from the current frame image as the ROI region. S2035: Traverse the image data of all frames, perform background difference-guided candidate region segmentation, and obtain the ROI region of the image data of each frame; By using a candidate region segmentation engine to guide background subtraction and extract ROI regions, the interference of background noise is reduced, thus reducing the computational load of subsequent detection.

[0049] In one optional implementation, the ROI regions of consecutive frame image data are input into a tool and equipment target detection model to perform tool and equipment target detection, obtaining the detection bounding box of the tool and equipment target in each frame of image data, including: S2041: Input the ROI region of the current frame's image data into the tool target detection model; S2042: Using the backbone network of the tool and equipment target detection model, extract the low-level image features, mid-level image features, and high-level image features of the current frame's image data; S2043: The semantic enhancement module of the neck network of the tool and equipment target detection model is used to perform semantic enhancement on the mid-level and high-level features. By adjusting the convolution kernel, the description of the overall shape of the tool and equipment is strengthened to obtain semantic features. S2044: The detail compensation module of the neck network of the feature fusion model of the tool and equipment target detection model is used to perform spatial detail compensation on semantic features, low-level features and high-level features to obtain compensated image features. In this embodiment, the detail compensation module receives the upsampled high-level features and the low-level features directly connected to the backbone network, and compensates the edge texture information of the low-level features back into the high-level semantic features through feature addition. S2045: The feature fusion module of the neck network of the tool and equipment target detection model is used to fuse semantic features and compensated depth features to obtain fused image features. S2046: Use the attention weighting module of the neck network of the tool and equipment target detection model to generate an attention weight map of fused image features, and perform weighted processing with the fused image features to obtain weighted image features; S2047: Using the detection head of the tool and equipment target detection model, based on weighted image features, tool and equipment target detection is performed to obtain the coordinates, class probability, and confidence of the detection box of the tool and equipment target in the current frame image data; In this embodiment, if no tool or equipment target is detected within the ROI area, it is determined that there is no anomaly and no further complex calculations are required. S2048: Traverse the ROI regions of all frames of image data, perform tool and equipment target detection, and obtain the detection box of the tool and equipment target in each frame of image data.

[0050] In one optional implementation, the detection bounding boxes of the machine target in the image data of consecutive frames are input into a spatiotemporal correlation filtering engine for spatiotemporal correlation filtering to identify stationary valid targets and perform anomaly detection on the stationary valid targets, including: S2051: Input the detection box of the tool target in the image data of the current frame into the spatiotemporal correlation filtering engine; S2052: The Kalman filter of the target tracking module of the spatiotemporal correlation filtering engine is used to track the detection box of the tool target in the current frame and predict the predicted position of the tool target in the next frame. S2053: The spatiotemporal correlation module using the spatiotemporal correlation filtering engine performs Hungarian matching on the coordinates of the detection box of the tool target in the current frame and the coordinates of the predicted position in the next frame, and assigns a unique target ID to the same tool target that is successfully matched. In this embodiment, the intersection-union distance matrix between the actual detection box and the Kalman predicted position in the current frame is calculated; the Hungarian algorithm is used to solve for the optimal matching relationship and associate the actual detection box with the existing trajectory. Assign and update a unique target ID to a successfully matched detection box; initialize unmatched detection boxes as new trajectories; mark unmatched old trajectories. S2054: Traverse the detection boxes of the machine targets in the image data of consecutive frames, perform spatiotemporal correlation filtering, obtain the target IDs assigned to several machine targets, and use the counter of the interference target filtering module of the spatiotemporal correlation filtering engine to count consecutive frames for the same target ID in consecutive frames to obtain the corresponding number of consecutive frames. S2055: If the number of consecutive frames with a target ID is greater than the number threshold, and the target position change rate of the detection box of the target ID is less than the change rate threshold, then the tool target corresponding to the target ID is regarded as a stationary valid target and proceeds to the next step; otherwise, it is judged as interference (such as flying birds, light and shadow flickering), and the detection box of the tool target corresponding to the target ID is canceled. By using a spatiotemporal correlation filtering engine, combined with Kalman filtering and Hungarian matching, and identifying stationary targets based on continuous frame counts and position change rates, dynamic interference such as flying birds and flickering light and shadows is effectively eliminated, significantly reducing the false alarm rate. S2056: Map the coordinates of a stationary valid target to the physical coordinate system to obtain the physical coordinates, and detect whether the physical coordinates intrude into the electronic fence of the orbital area. In this embodiment, based on the extrinsic parameter matrix (rotation matrix and translation vector) of the camera calibration, the image pixel coordinates are converted into world physical coordinates using a pinhole camera model; it is then checked whether the physical coordinates of each pixel of a stationary valid target are located within the preset orbital zone electronic fence polygon area (usually the ray method is used to determine whether a point is within the polygon). S2057: If so, a tooling anomaly occurs, and a tooling anomaly detection report is generated based on the physical coordinates of the stationary valid target, the corresponding category probability, and the confidence level.

[0051] This invention also provides an anomaly detection device for track area tools based on image data, referring to... Figure 3 The device may include the following units: The model building unit 301 is used to construct a tool target detection model by using an image recognition algorithm and introducing a multi-objective optimization algorithm, and connect the tool anomaly detection model to the candidate region segmentation engine and the spatiotemporal correlation filtering engine. The preprocessing unit 302 is used to acquire the video stream of the track area and preprocess the video stream to obtain image data of consecutive frames; The candidate region segmentation unit 303 is used to input image data of consecutive frames into the candidate region segmentation engine to perform background difference-guided candidate region segmentation and obtain the ROI region of each frame of image data. The target detection unit 304 is used to input the ROI region of the image data of consecutive frames into the tool target detection model to perform tool target detection and obtain the detection box of the tool target in each frame of image data; The anomaly detection unit 305 is used to input the detection box of the tool target in the image data of consecutive frames into the spatiotemporal correlation filtering engine, perform spatiotemporal correlation filtering, identify stationary valid targets, and perform anomaly detection on the stationary valid targets.

[0052] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing the program stored in the memory, implements the image data-based track area tool anomaly detection method of the present invention.

[0053] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EI) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0054] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0055] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image data-based track area tool anomaly detection method of the embodiments of the present invention.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0061] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting anomalies in track-bound machinery based on image data, characterized in that, The method includes: Using image recognition algorithms, a multi-objective optimization algorithm is introduced to construct a tool and equipment target detection model, and the tool and equipment anomaly detection model is connected to the candidate region segmentation engine and the spatiotemporal correlation filtering engine. The video stream of the track area is acquired and preprocessed to obtain image data of consecutive frames; Image data from consecutive frames is input into a candidate region segmentation engine to perform background difference-guided candidate region segmentation, thereby obtaining the ROI region of each frame of image data. The ROI regions of consecutive frame image data are input into the tool and equipment target detection model to perform tool and equipment target detection, and the detection box of the tool and equipment target in each frame of image data is obtained. The detection bounding boxes of the machine targets in the continuous frame image data are input into the spatiotemporal correlation filtering engine for spatiotemporal correlation filtering, to identify stationary valid targets, and to perform anomaly detection on the stationary valid targets.

2. The method for detecting abnormalities in track area tools based on image data according to claim 1, characterized in that, Using image recognition algorithms, a multi-objective optimization algorithm is introduced to construct a tool and equipment target detection model. This model is then connected to a candidate region segmentation engine and a spatiotemporal correlation filtering engine, including: An initial tool and equipment target detection model is constructed using an image recognition algorithm. The tool and equipment target detection model includes a backbone network, a feature fusion neck network, and a detection head. The feature fusion neck network is equipped with a semantic enhancement module, a detail compensation module, a feature fusion module, and an attention weighting module. A multi-objective optimization function for a multi-objective optimization algorithm is constructed, wherein the multi-objective optimization function includes a detection accuracy loss function and a model complexity; Based on a multi-objective optimization function, a multi-objective optimization algorithm is used to optimize the hyperparameters of the initial tool and equipment target detection model, resulting in the final tool and equipment target detection model. The input of the final tool and equipment anomaly detection model is connected to the candidate region segmentation engine, and the output of the final tool and equipment anomaly detection model is connected to the spatiotemporal correlation filtering engine.

3. The method for detecting abnormalities in track area tools based on image data according to claim 2, characterized in that, The multi-objective optimization algorithm is the multi-objective alligator optimization algorithm, and the formula for the multi-objective optimization function of the multi-objective alligator optimization algorithm is: In the formula, For hyperparameters The corresponding optimized target value of the tool and equipment target detection model on the validation set; For hyperparameters The detection accuracy loss value of the corresponding tool and equipment target detection model on the validation set; For hyperparameters The model complexity of the corresponding tool and equipment target detection model; In the formula, The bounding box regression loss value; This is the classification loss value; Weights are assigned to account for the loss of detection accuracy. In the formula, This represents the total number of model parameters. The hardware must withstand an upper limit threshold for model parameters; This represents the number of floating-point operations. The upper limit threshold for the hardware to withstand floating-point operations; These are the model complexity weights.

4. The method for detecting abnormalities in track area tools based on image data according to claim 3, characterized in that, Based on a multi-objective optimization function, a multi-objective optimization algorithm is used to optimize the hyperparameters of the initial tool and equipment target detection model, resulting in the final tool and equipment target detection model, including: Several historical images of consecutive frames in the track area are collected, preprocessed and labeled to obtain several sample images, and these sample images are divided into training set and validation set. The hyperparameters of the tool and equipment target detection model are encoded into individual vectors. A chaotic sequence is generated using a Logistic mapping, and the chaotic sequence is mapped to the solution space of the individual vectors to obtain the initial crocodile population. Based on the candidate hyperparameters corresponding to each initial crocodile individual in the initial crocodile population, the corresponding candidate tool and machinery target detection model is trained using the training set, and the corresponding fitness value is calculated using a multi-objective optimization function based on the validation set. Based on fitness values, the initial crocodile population was divided into an initial raiding population and an initial lurking population in a 3:7 ratio according to ascending power sort. Based on the fitness value, the initial raiding population is subjected to fast non-dominated sorting to obtain different front surface levels. Several initial raiding individuals in the first front surface level whose crowding distance is greater than the crowding distance threshold are stored as non-dominated solutions in the external archive set. By introducing the Levy flight mechanism and the gray wolf cooperative concept, the initial lurking population is updated in position to obtain the updated first crocodile population; By introducing a convergence factor, the initial attacking population is updated in position to obtain an updated second crocodile population. The updated first crocodile population and the updated second crocodile population are integrated to obtain an updated crocodile population. A multi-objective optimization function is used to calculate the fitness value of each updated crocodile individual in the updated crocodile population. Based on the fitness value, update the frontier surface hierarchy and external archive set using the updated crocodile individuals; Until the number of iterations reaches the iteration threshold, all updated crocodile individuals in the external archive set are output as the Pareto non-dominated optimal solution set; Based on the operating status data of the server where the machine tool anomaly detection model is located, optimize the weighting weights, and based on the optimized weighting weights, optimize the fitness value of each solution in the Pareto non-dominated optimal solution set to obtain the weighted fitness value. The individual vectors of the solutions corresponding to the optimal weighted fitness values ​​are decoded to obtain the optimal hyperparameters. Based on the optimal hyperparameters, the hyperparameters of the initial tool and equipment target detection model are optimized to obtain the optimized tool and equipment target detection model. Based on several sample images, the optimized tool and equipment target detection model is fine-tuned and trained to obtain the final tool and equipment target detection model.

5. The method for detecting abnormalities in track area tools based on image data according to claim 4, characterized in that, The candidate region segmentation engine includes a background modeling module, a difference calculation module, a threshold segmentation module, and a ROI extraction module; The spatiotemporal correlation filtering engine includes a target tracking module, a spatiotemporal correlation module, and an interference target filtering module.

6. The method for detecting abnormalities in track area tools based on image data according to claim 5, characterized in that, The video stream of the track area is acquired and preprocessed to obtain continuous frame image data, including: High-definition industrial cameras deployed in the track area are used to capture video streams of the track area, and the video streams are frame-by-frame extracted to obtain raw images of consecutive frames. The keyframes of the original images of consecutive frames are extracted using the inter-frame difference method based on the hash algorithm, and the original keyframe images of the consecutive frames are obtained. The original image of each keyframe is preprocessed to obtain image data of consecutive frames.

7. The method for detecting abnormalities in track area tools based on image data according to claim 6, characterized in that, Image data from consecutive frames is input into a candidate region segmentation engine for background subtraction-guided candidate region segmentation, yielding the Region of Interest (ROI) for each frame, including: The background modeling module of the candidate region segmentation engine constructs a background model of the track area based on several historical images of consecutive frames during the no-work period. The image data of the current frame is input into the candidate region segmentation engine, and the difference calculation module of the candidate region segmentation engine is used to calculate the difference image between the image data of the current frame and the background model. Using the threshold segmentation module of the candidate region segmentation engine, Otsu adaptive threshold segmentation is performed on the difference image of the current frame to obtain a binarized mask. The ROI extraction module of the candidate region segmentation engine is used to perform morphological processing on the binarized mask and extract the minimum bounding rectangle of all connected components as the ROI region of the image data of the current frame. By traversing all frames of image data, background difference-guided candidate region segmentation is performed to obtain the ROI region of each frame of image data.

8. The method for detecting abnormalities in track area tools based on image data according to claim 7, characterized in that, The ROI regions of consecutive frame image data are input into the tool and equipment target detection model to perform tool and equipment target detection, resulting in the detection bounding box of the tool and equipment target in each frame of image data, including: Input the ROI region of the current frame's image data into the tool target detection model; Using the backbone network of the tool and equipment target detection model, low-level image features, mid-level image features, and high-level image features of the current frame image data are extracted. The semantic enhancement module of the neck network, which is used to fuse the features of the tool and equipment target detection model, is used to semantically enhance the mid-level and high-level features to obtain semantic features. The feature fusion neck network detail compensation module of the tool and equipment target detection model is used to perform spatial detail compensation on semantic features, low-level features and high-level features to obtain compensated image features. The feature fusion module of the neck network of the tool and equipment target detection model is used to fuse semantic features and compensated depth features to obtain fused image features. The attention weighting module of the neck network is used to fuse the features of the tool and equipment target detection model to generate an attention weight map of the fused image features, and then weighted with the fused image features to obtain weighted image features. Using the detection head of the tool and equipment target detection model, based on weighted image features, tool and equipment target detection is performed to obtain the coordinates, class probability, and confidence of the detection box of the tool and equipment target in the current frame image data; Traverse the ROI regions of all frames of image data, perform tool and equipment target detection, and obtain the detection box of the tool and equipment target in each frame of image data.

9. The method for detecting abnormalities in track area tools based on image data according to claim 8, characterized in that, The detection bounding boxes of the machine targets in the continuous frame image data are input into the spatiotemporal correlation filtering engine for spatiotemporal correlation filtering to identify stationary valid targets and perform anomaly detection on the stationary valid targets, including: Input the detection bounding box of the tool target in the image data of the current frame into the spatiotemporal correlation filtering engine; The Kalman filter of the target tracking module of the spatiotemporal correlation filtering engine is used to track the detection box of the tool target in the current frame and predict the predicted position of the tool target in the next frame. The spatiotemporal correlation module of the spatiotemporal correlation filtering engine performs Hungarian matching on the coordinates of the detection box of the tool target in the current frame and the coordinates of the predicted position in the next frame, and assigns a unique target ID to the same tool target that is successfully matched. The detection boxes of the machine targets in the image data of consecutive frames are traversed, and spatiotemporal correlation filtering is performed to obtain the target IDs of several machine targets. The counter of the interference target filtering module of the spatiotemporal correlation filtering engine is used to count the consecutive frames of the same target ID to obtain the corresponding number of consecutive frames. If the number of consecutive frames with a target ID is greater than the number threshold, and the rate of change of the target position of the detection box of the target ID is less than the rate of change threshold, then the tool target corresponding to the target ID is regarded as a stationary valid target and proceeds to the next step; otherwise, the detection box of the tool target corresponding to the target ID is canceled. Map the coordinates of stationary valid targets to the physical coordinate system to obtain physical coordinates, and detect whether the physical coordinates intrude into the electronic fence of the orbital area. If so, a tooling anomaly occurs, and a tooling anomaly detection report is generated based on the physical coordinates of the stationary valid target, the corresponding category probability, and the confidence level.

10. A track area tool anomaly detection device based on image data, used to implement the track area tool anomaly detection method as described in any one of claims 1-9, characterized in that, The device includes: The model building unit is used to construct a tool and equipment target detection model by using image recognition algorithms and introducing multi-objective optimization algorithms, and connects the tool and equipment anomaly detection model to the candidate region segmentation engine and the spatiotemporal correlation filtering engine. The preprocessing unit is used to acquire the video stream of the track area and preprocess the video stream to obtain image data of consecutive frames; The candidate region segmentation unit is used to input image data from consecutive frames into the candidate region segmentation engine to perform background difference-guided candidate region segmentation and obtain the ROI region of each frame of image data. The target detection unit is used to input the ROI region of the image data of consecutive frames into the tool target detection model to perform tool target detection and obtain the detection box of the tool target in each frame of image data; The anomaly detection unit is used to input the detection box of the tool target in the image data of consecutive frames into the spatiotemporal correlation filtering engine, perform spatiotemporal correlation filtering, identify stationary valid targets, and perform anomaly detection on the stationary valid targets.

Images