Container door anomaly detection model construction method and anomaly detection method and device

Abstract

This application provides a method for constructing a container door anomaly detection model and an anomaly detection method and device, which are applied in the field of port automated visual monitoring technology. After preprocessing real anomaly samples, virtual anomaly samples are generated using a Wasserstein generative adversarial network to expand the training dataset, effectively solving the problem of model training difficulties caused by the scarcity of anomaly samples. A deep detection network with a joint attention enhancement mechanism is constructed, and a loss function based on Wasserstein distance is used as the regression loss during the training process, thereby obtaining a detection model that can accurately detect container door anomalies, significantly improving the detection accuracy of the model in complex backgrounds.

Description

Technical Field

[0001] This application relates to the field of port automation visual monitoring technology, specifically to a method for constructing a container door anomaly detection model and an anomaly detection method and device. Background Technology

[0002] With the growth of global shipping trade, Automated Container Terminals (ACTs) have become a mainstream trend in port development. In automated operations, quay cranes and rail-mounted gantry cranes frequently need to grab and release containers. However, the unauthorized opening of container doors is a type of safety accident that, while infrequent, has extremely serious consequences. If a container door accidentally opens during handling, it can easily lead to cargo spillage, equipment collisions, and even personal injury, severely impacting terminal operational efficiency and safety.

[0003] Currently, monitoring the status of container doors mainly relies on manual inspections or traditional visual inspection technologies. However, existing solutions have the following significant pain points and limitations in practical applications: First, the scarcity of abnormal accident samples makes it difficult to train deep learning models: Existing mainstream object detection algorithms (such as the YOLO series and R-CNN series) are all data-driven algorithms, and their performance is highly dependent on massive and evenly distributed training data. However, in actual port operations, abnormal opening of container doors is a typical "low-probability event" or "long-tailed event," with very few actual accident samples that are difficult to collect. Due to the lack of sufficient positive samples (abnormal samples) for training, existing models cannot learn the diverse features of container door opening, resulting in a high false negative rate in practical applications or a large number of false alarms due to overfitting, making it difficult to meet the reliability requirements of industrial-grade safety monitoring.

[0004] Secondly, the complex and ever-changing outdoor working environment severely interferes with image quality: Port operations are typically conducted in an open-air environment around the clock, with lighting conditions fluctuating drastically depending on the day, night, and weather. Strong daylight can cause specular reflections (highlights) on container surfaces, while artificial lighting at night can create shadows. This coexistence of strong light and shadow damages the original features of the image. Furthermore, in adverse weather conditions such as rain, fog, and haze, images are often accompanied by significant noise and a marked decrease in clarity, resulting in severe loss of detail and rendering traditional image processing-based detection algorithms ineffective.

[0005] Third, minute door opening features are difficult to extract accurately in complex backgrounds: The initial characteristics of abnormally opened container doors typically manifest as tiny gaps or subtle angular deflections. Compared to the massive container body and complex background (such as other containers in the yard or ground markings), these abnormal features are typical "tiny targets." Traditional convolutional neural networks (CNNs) are prone to losing such subtle spatial information during the downsampling process. Furthermore, existing loss functions based on intersection-over-union (IoU) fail to provide effective gradient feedback when the predicted bounding box does not overlap with the ground truth bounding box (common in small target detection), leading to difficulty in model convergence and insufficient localization accuracy to meet industrial-grade safety requirements.

[0006] Therefore, a new solution for detecting abnormalities in container doors is needed. Summary of the Invention

[0007] In view of this, the embodiments of this specification provide a method for constructing a container door anomaly detection model and an anomaly detection method and device, which can achieve accurate and reliable detection of the container door status in industrial scenarios where anomaly samples are scarce, complex environmental interference and small targets are difficult to capture accurately.

[0008] The embodiments in this specification provide the following technical solutions: This specification provides an embodiment of a method for constructing a container door anomaly detection model, including: Obtain real abnormal samples and preprocess the real abnormal samples; Based on the Wasserstein generative adversarial network, a virtual sample generation model is constructed. The generator is trained using the preprocessed real anomaly samples to generate virtual anomaly samples. The virtual anomaly samples are then mixed with the real anomaly samples to construct a training dataset. Construct a deep detection network that incorporates a joint attention enhancement mechanism; The deep detection network is trained using the training dataset, and a loss function based on Wasserstein distance is used as the regression loss during the training process to obtain a trained container door anomaly detection model.

[0009] This specification also provides an embodiment of a method for detecting abnormalities in container doors, including: Acquire images of the container doors when the spreader descends to the preset trigger height; The container door image is enhanced using a multi-scale wavelet decomposition algorithm to obtain an enhanced image. The enhanced image is input into the container door anomaly detection model constructed using the aforementioned container door anomaly detection model construction method for detection, and the door status detection result is obtained. When the door status detection result indicates that the door is open and the confidence level exceeds a preset threshold, it is determined that the door is abnormally open, and a stop signal is sent to the PLC within a preset response time to cut off the power source.

[0010] This specification also provides an embodiment of a container door anomaly detection device, comprising: The image acquisition module is used to acquire images of the container doors when the spreader descends to a preset trigger height; The image processing module is used to preprocess the container door image based on the multi-scale wavelet decomposition algorithm to obtain an enhanced image; The detection module is used to input the enhanced image into the container door anomaly detection model constructed using the aforementioned container door anomaly detection model construction method for detection, and obtain the door status detection result; The closed-loop real-time early warning module is used to determine that the door is abnormally opened when the door status detection result is that the door is open and the confidence level exceeds a preset threshold. It then sends a stop signal to the PLC within a preset response time to cut off the power source.

[0011] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least: 1. By acquiring real anomaly samples and preprocessing them, the impact of environmental interference on image quality is eliminated, enabling subsequent training to be based on high-quality samples. Virtual anomaly samples are generated based on the Wasserstein generative adversarial network to expand the training dataset, effectively solving the problem of model training difficulties caused by the scarcity of anomaly samples. 2. By constructing a deep detection network that includes a joint attention enhancement mechanism, we achieved precise focusing and extraction of tiny door opening features, significantly improving the model's detection accuracy in complex backgrounds; 3. By using a loss function based on Wasserstein distance as the regression loss during training, the gradient vanishing problem in small target detection is solved, effectively improving the model's convergence performance and localization accuracy. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for constructing an anomaly detection model for container doors according to this application; Figure 2This is a multidimensional distribution feature matrix of target bounding boxes in the training dataset of this application; Figure 3 This is a diagram of the overall architecture of the detection network using the joint attention mechanism in this application; Figure 4 This is a schematic diagram illustrating the detection effect of abnormal opening of the cabinet door in this application; Figure 5 This is a schematic diagram of the receptive field distance measurement principle based on Gaussian distribution modeling in this application; Figure 6 This is a graph showing the F1 score performance under different confidence thresholds in this application; Figure 7 This is a precision-recall curve of the detection model in this application. Detailed Implementation

[0014] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0015] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0017] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0018] Additionally, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that practice can be carried out without these specific details.

[0019] Regarding existing container door anomaly detection solutions, such as the prior art submitted in the background section, through problem analysis and exploration of solutions, it was found that: On the one hand, there is a fundamental defect of "scarcity of samples" at the source of data. Existing mainstream detection algorithms rely heavily on massive training data, but abnormal opening of the cabinet door is a typical low-probability event. There are very few real accident samples and it is difficult to collect them systematically. This makes it impossible for the model to learn the diverse features of the cabinet door opening. The training process is prone to overfitting, and the false negative rate remains high in practical applications, which cannot meet the requirements of industrial-grade reliability. Secondly, the image quality is fatally affected by "environmental interference". In the all-weather open-air operation environment of the port, the drastic changes in lighting and the alternation of rain and fog mean that strong light reflection, shadow dark areas and noise interference seriously damage the original texture features of the image. Traditional detection algorithms are basically ineffective under such complex weather conditions and cannot guarantee the clarity and stability of the input image. In three aspects, the features of tiny door openings are difficult to extract accurately in complex backgrounds. The initial features of abnormal door openings are often manifested as tiny gaps or slight angular deflections. Compared to the huge container body and complex background, these are typical tiny targets. Traditional convolutional neural networks are prone to losing such detailed information during downsampling. Furthermore, the gradient of the loss function based on intersection-over-union (IoU) is zero when the predicted box does not overlap with the real box, causing the model to fail to converge effectively. The positioning accuracy is far from meeting the industrial standards for safety warnings.

[0020] In view of this, the embodiments of this specification propose a scheme for constructing a container door anomaly detection model based on virtual sample augmentation and multi-dimensional attention feature fusion. After preprocessing real anomaly samples, virtual anomaly samples are generated using a Wasserstein generative adversarial network to expand the training dataset, effectively solving the problem of model training difficulties caused by the scarcity of anomaly samples. A deep detection network with a joint attention enhancement mechanism is constructed, and a loss function based on Wasserstein distance is used as the regression loss during the training process, thereby obtaining a detection model that can accurately detect container door anomalies, significantly improving the detection accuracy of the model in complex backgrounds.

[0021] The technical solutions provided by the various embodiments of this application are described below with reference to the accompanying drawings.

[0022] like Figure 1 As shown in the embodiments of this specification, a method for constructing a container door anomaly detection model is provided, including: Step S100: Obtain real abnormal samples and preprocess the real abnormal samples.

[0023] During implementation, real accident images of abnormally opened container doors were collected from the port operation site. Considering that images in the open-air environment of the port are easily affected by factors such as rain, fog, and uneven lighting, which affect the quality of the samples, the collected real abnormal samples were preprocessed to obtain clear, stable, and high-quality real abnormal samples, laying a reliable data foundation for subsequent virtual sample generation and model training.

[0024] Step S200: Based on the Wasserstein generative adversarial network, construct a virtual sample generation model, use the preprocessed real anomaly samples to train the generator, generate virtual anomaly samples, and mix the virtual anomaly samples with the real anomaly samples to construct a training dataset.

[0025] The feature distribution of the dataset after WGAN virtual sample augmentation is as follows: Figure 2 As shown, this figure uses a matrix format to display the bounding boxes in the image. Coordinates (location distribution) and Correlation between (scale distributions).

[0026] Among them: The bar chart in the upper left corner shows the number of instances in different categories (Container Overall CNTR, DoorOpen, DoorClose), reflecting the sample size of each category after data augmentation; Diagonal chart: A histogram showing the univariate distribution of each attribute; The scatter plot region shows the location preference of the target boxes in the image space (e.g., concentrated in the central area of ​​the image) and their aspect ratio characteristics. This distribution information verifies the consistency of statistical regularities between the virtual samples generated by the Generative Adversarial Network (WGAN) and the real samples, and also provides data support for the initial size design of the subsequent anchor boxes.

[0027] In implementation, the WGAN generative network is constructed: Build generator and discriminator We introduce Wasserstein distance as a loss metric to replace the JS divergence in traditional Generative Adversarial Networks (GANs). Its objective function is formalized as: ; in, This represents the distribution of real outlier samples. To generate the sample distribution.

[0028] The network is trained using a small number of real anomaly samples. When the network converges, the generator is used... Output virtual anomaly sample image Discriminator right The discriminator performs a discrimination process, and samples that pass the discriminator threshold are marked as valid virtual samples.

[0029] The effective virtual samples are mixed with real samples to construct the expanded training dataset. This effectively solves the problem of unstable training in generative adversarial networks, and can create highly realistic images of abnormally opened doors out of thin air, fundamentally solving the risk of overfitting caused by the lack of negative samples in deep learning models.

[0030] Step S300: Construct a deep detection network that includes a joint attention enhancement mechanism.

[0031] As analyzed above, in the complex port operation environment, there is a lot of interfering information on the surface of the container (such as stains, rust, text, and changes in light and shadow), while the early features of abnormal opening of the container door (such as tiny gaps) are extremely subtle. When standard convolutional neural networks extract features layer by layer, they are prone to losing details in the process of capturing global semantic information, and cannot distinguish which areas are critical (the container door) and which are irrelevant (the background).

[0032] Therefore, this application improves the network structure when constructing the core detection model by introducing a joint attention enhancement mechanism. This mechanism dynamically filters features from two complementary dimensions: on the one hand, it identifies which types of features in the image are more important for judging the state of the door. For example, texture features such as the door's lock bar and gap edges are given higher weights, while irrelevant backgrounds such as the sky and ground are suppressed. On the other hand, it locates the specific positions of these important features in the image, focusing the network on the image coordinate region where the door is located and ignoring interference outside this region. This enables the precise capture of the door's minute opening features, significantly improving the detection robustness and accuracy in complex environments.

[0033] Step S400: Train the deep detection network using the training dataset. During the training process, use the loss function based on Wasserstein distance as the regression loss to obtain the trained container door anomaly detection model.

[0034] In implementation, Wasserstein distance is introduced as a regression loss, which models the predicted box and the real target box from simple geometric figures as probability distributions. Even if the two do not overlap in space, the distance can be quantified by calculating the difference between the distributions. This can provide the model with continuous and effective gradient feedback, accurately indicating the adjustment direction of the predicted box in terms of position and size, thereby guiding the model to converge quickly and significantly improving the localization accuracy and training efficiency of small abnormal features such as gaps in door frames.

[0035] For example, a scenario: a batch of real abnormal cabinet door images. A batch of virtual images generated by a generator Discriminator Output a scalar rating; the higher the rating, the more authentic the result.

[0036] Example: ; ; in, This represents the score assigned by the discriminator to a true anomalous image; This represents the average score given by the discriminator to a batch of real abnormal images; This represents the score of the virtual sample generated by the generator; This represents the average score of the virtual samples generated by the generator; The WGAN loss is calculated as follows: ; The optimization objective of this loss is to minimize this value, thereby widening the score gap between true and false samples.

[0037] Compared to traditional GANs (JS divergence): when the two distributions do not overlap, the JS divergence tends to be constant. If the gradient is 0, training will fail. However, WGAN can still calculate the linear distance and provide an effective gradient even when the distributions do not overlap.

[0038] In summary, by utilizing Wasserstein Generative Adversarial Network (WGAN) combined with oversampling techniques to generate high-quality virtual anomaly samples, the problem of model training difficulties caused by the scarcity of anomaly samples is effectively solved. In the image preprocessing stage, rain and fog noise is removed and image details are enhanced to adapt to the complex operating environment of ports. In terms of detection model construction, a deep neural network including a channel and spatial joint attention module (CBMA) is designed to dynamically lock the key areas of the container door. In addition, Wasserstein distance is introduced to replace the traditional IoU calculation of receptive field distance, which optimizes the matching accuracy of small targets and non-overlapping anchor boxes, thereby effectively overcoming the influence of small samples and severe weather, and achieving millisecond-level accurate early warning of the container door status during container handling operations.

[0039] In some implementations, constructing the training dataset further includes: The virtual anomaly samples and the real anomaly samples are augmented with data to expand the training dataset; wherein the data augmentation includes at least one of geometric transformation and color transformation.

[0040] During implementation, geometric transformations (rotations) are performed to expand the training dataset. Scaling (and color changes).

[0041] In conjunction with the above embodiments, for example: inputting a random noise vector It follows a standard normal distribution. .

[0042] Generator Networks The processing flow is as follows: Fully connected layer: Deconvolution layer 1: Deconvolutional layer 2: stepwise upsampling; Output layer: RGB image.

[0043] Specific characteristics are manifested as follows: When At that time, the generated image showed the box door slightly ajar. ; when At that time, the generated image showed the box door wide open. .

[0044] By adjusting the noise vector The model can generate virtual anomaly samples from various angles in batches.

[0045] To simulate the shaking of the lifting equipment during operation and different viewing angles, the image coordinates were adjusted. Perform an affine transformation.

[0046] For example: set the rotation angle The rotation matrix is: ; For feature points :

[0047] As a result, a new sample is generated, and the feature points are moved to... This forces the detection model to learn rotation-invariant features, preventing the model from simply memorizing pixel positions.

[0048] In some implementations, the preprocessing of the real abnormal samples includes: A multi-scale wavelet decomposition algorithm is used to filter out rain and fog noise in the image; And / or, use an illumination equalization algorithm to adjust the brightness distribution of the image.

[0049] During implementation, for the collected sequences To mitigate rain and fog interference, a multi-scale wavelet decomposition algorithm is employed, specifically decomposing the image into low-frequency sub-images. and multi-scale high-frequency subgraphs ( ). For high-frequency subgraphs Adaptive Bayesian wavelet threshold estimation is used to filter out high-frequency noise components. The image is then reconstructed using inverse wavelet transform to obtain the dehazed intermediate image. By decomposing the image into sub-images of different frequencies, the "noise" generated by rain and fog (usually located in the high-frequency band) can be accurately separated and removed, while retaining the outline information of the container (located in the low-frequency band), thus solving the technical problem of detection failure caused by weather in open-air port operations.

[0050] Hypothesis: Extract an image affected by rain and fog. Pixel Microblock Processing is performed (simplifying the multi-scale decomposition process). The input pixel matrix (grayscale values, including noise) is: Among them, there is a 200-satellite mutation noise in the lower right corner.

[0051] 1. Wavelet decomposition (taking Haar wavelet as an example): Low-frequency components (LL, approximation): ; High-frequency components (HH, detail / noise): Calculate diagonal differences. ; It should be noted that this is a simplified illustration only, and actually includes three directions: LH, HL, and HH.

[0052] 2. Adaptive Bayesian threshold estimation: Assuming the threshold is calculated based on the overall noise level of the image, Determining high-frequency coefficients It is believed that it contains valid edge information, but also noise.

[0053] Applying the soft thresholding function: ; By subtracting the threshold, small-amplitude pure noise is suppressed while the weakened edge strength is preserved.

[0054] 3. Image reconstruction: Using the corrected high-frequency components and low frequency components Perform an inverse transform to obtain the dehazed pixel block. The reconstructed pixel values ​​are smoother and have less noise compared to the original values. The impact of ) is significantly reduced, and image contrast is preserved.

[0055] In some implementations, for The image undergoes illumination equalization channel processing, utilizing histogram equalization techniques to adjust the image brightness distribution and output a high-quality preprocessed image. .

[0056] During implementation, for the dehazed image Perform histogram equalization.

[0057] Input: Suppose there is a dark area in the image ( The pixel value of the region is: ; Most pixels are concentrated – The low brightness range For reflective points.

[0058] Calculation process: Statistical histogram: pixel values Appear Second-rate, Appear Second-rate, Appear Second-rate.

[0059] Calculate the cumulative distribution function (CDF): ; Mapping to new pixel values, the formula is: ; Original value Original value .

[0060] The result is: .

[0061] That is: the originally dim area ( – It was stretched to the medium-high brightness range. – This allows the machine to recognize the texture details on the container surface even in shadows.

[0062] In some implementations, constructing a deep detection network that includes a joint attention enhancement mechanism includes: The training dataset is input into the backbone network for feature extraction to obtain a feature map; The feature map is subjected to spatial pyramid pooling processing, and multi-scale contextual information is fused to obtain a multi-scale feature map; Global max pooling and global average pooling are performed on the multi-scale feature maps respectively, and the pooling results are input into a shared multilayer perceptron for processing to generate channel weight vectors. Max pooling and average pooling are performed on the multi-scale feature map along the channel axis, and the pooling results are concatenated and input into the convolutional layer for processing to generate a spatial weight map. The channel weight vector and the spatial weight map are multiplied by the multi-scale feature map to obtain the enhanced feature map.

[0063] During implementation, backbone network extraction: Input the backbone network.

[0064] like Figure 3 As shown, the backbone network on the left consists of multiple convolutional modules (ConvModule) and C3k2 modules stacked alternately, and is connected to an SPPF (spatial pyramid pooling) module at the end to expand the receptive field and output multi-scale feature maps. .

[0065] To intuitively demonstrate the internal computational process of a neural network, this embodiment uses the industry-standard tensor dimension change process for deduction, clearly showing how data flows and is processed within the neural network.

[0066] Assume the image size input to the detection network is normalized to... That is, an image has a height of 640 pixels, a width of 640 pixels, and 3 color channels (RGB). After the image is input, it undergoes down-sampling through a series of convolutional modules (ConvModule) and C3k2 modules, gradually transforming the data into a high-dimensional feature map. Specifically: enter: Image Tensor (Batch Size = 1).

[0067] Intermediate layer processing (C3k2 module): After 5 downsampling steps of 2 (Stride = 32): :

[0068] :

[0069]

[0070] (Deep features): Size changes The number of channels has been expanded from the initial 3 layers (RGB) to layer.

[0071] At this point, the tensor shape is: That is, one image was abstracted into 1024 images. The feature map.

[0072] Although the feature maps have been extracted, their size is somewhat small, which may result in the loss of some detailed information. Therefore, SPPF (spatial pyramid pooling) is used to expand the receptive field without reducing the size. Specifically: enter: ( ).

[0073] Max Pooling: Using kernel size Perform three consecutive pooling operations: ; ; .

[0074] It should be noted that the output size remains unchanged after each pooling operation. constant.

[0075] To fully utilize the information from different receptive fields, the original features and the results of triple pooling are stacked together along the channel dimension.

[0076] Concat: Concatenates the original input with the result of the third pooling along the channel dimension. .

[0077] The number of channels becomes ; Too many channels increase the computational burden, so in practical applications, dimensionality reduction is usually performed, typically by... Convolution dimensionality reduction or .

[0078] Output: Assuming the dimensionality reduction is achieved, the output is the feature map. .

[0079] As can be seen from the above embodiments, although the feature map size is small (20×20), each pixel is fused with , , It provides contextual information at multiple scales. For large-scale targets like containers, this ensures that the model "sees" more comprehensively and doesn't just focus on local textures.

[0080] like Figure 3 As shown, a Joint Attention Enhancement Module (CBMA Module) is connected to the output of the backbone network. This module includes parallel channel attention and spatial attention branches. Specifically, channel attention involves performing global max pooling and average pooling on the feature map, processing it through a shared multilayer perceptron (MLP), and outputting a channel weight vector. This is used to weight and emphasize feature channels containing box door textures; Spatial attention: the feature map is max-pooled and average-pooled along the channel axis, processed by a convolutional layer, and output as a spatial weight map. Focusing on the physical coordinate region where the box door exists in the image. .

[0081] Final output enhancement features: .

[0082] In conjunction with the above embodiments, regarding the Backbone output ( The CBMA module needs to calculate "where is important (space)" and "what is important (channel)," specifically: The first step is to calculate channel attention: enter: ( (There are several channels), some of which recorded the blue sky, some recorded the texture of the gaps in the box door, and some recorded the edges of the gaps.

[0083] Perform global pooling of information within each channel: Max pooling: ; Average pooling: ; MLP processing: Let the weights of the two fully connected layers in the shared MLP be... ,but: ; Weight generation: Obtain the channel weight vector: ; Example value: .

[0084] Meaning: Second channel ( ) contains the texture of the door seams, with its weight greatly enhanced; the first channel ( The background blue sky is suppressed.

[0085] The second step is to calculate spatial attention: Input: Channel-weighted feature map .

[0086] Channel compression: Perform max pooling and average pooling along the channel axis to obtain two... The image: ; Convolution processing: use Convolution kernel fusion: ; Weight generation: Obtain spatial mask .

[0087] Example value: In coordinates Location (corresponding to the center of the box door in the image). ; In coordinates Location (corresponding to the ground) .

[0088] The third step is the final fusion: ; The output tensor is still However, the values ​​in the key areas are amplified, and the background noise is reduced to near zero.

[0089] The above embodiments demonstrate that Joint Attention Enhancement (CBMA) is essentially an "adaptive filter." In nighttime rain mode, raindrop noise is spatially randomly distributed and cannot form coherent features; therefore, in... In spatial weight calculation, a low value is assigned; while the gap in the door has a specific geometric structure and is assigned a high value, thereby achieving anti-interference detection.

[0090] It should be noted that the surface of containers is usually subject to dirt, text and other interference. The CBMA mechanism simulates human vision, allowing the model to "first look where there is something (spatial attention), then look at what it is (channel attention)," which effectively suppresses background interference and improves sensitivity to tiny gaps.

[0091] In some embodiments, constructing a deep detection network including a joint attention enhancement mechanism further includes: The enhanced feature map is upsampled and then concatenated and fused with the shallow feature map output by the backbone network to obtain a fused feature map. The fused feature map is input into the detection head for convolution processing to obtain the class probability and bounding box coordinates of the door state.

[0092] In conjunction with the above embodiments, such as Figure 3 As shown, the middle neck network adopts a feature pyramid fusion structure achieved by upsampling and concatenation, which integrates deep semantics and shallow detailed features; the right head contains multiple detection heads, which output class probabilities and bounding box coordinates respectively.

[0093] Multi-scale information is fused using a Feature Pyramid Network (FPN) structure, and the final detection box is output. Specifically: 1. Neck Feature Fusion (Upsample & Concat): Deep features: Enhanced feature maps from CBMA output ; Upsampling: Magnifying the size by 2 times yields: This aligns it with the size of the shallow feature map.

[0094] Shallow features: Feature maps from the intermediate layers of the backbone: ( ).

[0095] splicing operation: and By splicing along the channel dimension, we obtain ; The output tensor shape is: ; After further convolutional dimensionality reduction using the C3k2 module, we obtain... The fusion feature retains both deep semantics and shallow details.

[0096] 2. Head detection output (Detect): Input: fused Feature map; Convolutional prediction: via The convolutional layer is mapped to the final output channel.

[0097] Assuming we detect three categories (CNTR, DoorOpen, DoorClose), each grid has three pre-defined anchor boxes.

[0098] Therefore, the number of output channels is: ; The final output tensor is: .

[0099] 3. Decoding Example: To obtain grid points For example, in the corresponding 24-dimensional data vector, the data of the first anchor box is: ; Confidence level judgment: If ,and If so, the anchor frame is determined to correspond to the "DoorOpen" (i.e., the door is open) category.

[0100] Coordinate Regression: Calculate the actual coordinates using the anchor frame formula: ; The center point of the box door in the original drawing is calculated as follows: Width and height are .

[0101] Therefore, Although the feature map has strong semantics (it indicates that it is a box door), its location is coarse; Although the feature map has weak semantics, its edges are clear. The Concat operation combines the two, enabling the system to accurately determine "the box door is open" (semantic) and accurately outline "which door is open" (localization), which is crucial for the subsequent calculation of the Wasserstein distance loss.

[0102] like Figure 4 As shown, due to the containers being squeezed or stacked unevenly, the door of the pink container on the right popped open abnormally. This application successfully detected and located the anomaly, generated a green bounding box in the image, and labeled it with the category "DoorOpen". Despite the small opening angle of the door and the complex background, the system still made the correct judgment and did not generate a false alarm for the closed door on the left, verifying the robustness of the algorithm under complex working conditions.

[0103] In some implementations, the use of a loss function based on Wasserstein distance as the regression loss includes: The predicted bounding boxes and the ground truth bounding boxes are mapped to the probability distribution space, and the Wasserstein distance between the two probability distributions is used as the regression loss.

[0104] In implementation, the effective receptive fields of the predicted and ground truth bounding boxes are modeled as two-dimensional Gaussian distributions. , where the mean covariance matrix From width and height Decide.

[0105] Calculate the distribution of the prediction box Distribution of true bounding boxes The second-order Wasserstein distance between them: ; For the diagonal covariance matrix, the formula simplifies to: ; Wasserstein distance provides effective gradients even in non-overlapping conditions, significantly improving the localization accuracy of small targets.

[0106] Construct an overall loss function that includes the Wasserstein distance, and perform backpropagation to update the network weights. First, define the overall loss function. Classification loss Regression loss Weighted sum: ; The regression loss term directly uses the second-order Wasserstein distance calculated in step S4.2, i.e. During backpropagation, the loss function is calculated for the predicted bounding box parameters. The partial derivatives. With the central coordinates... For example, its gradient is calculated as follows: ; Subsequently, the network parameters are updated based on the calculated gradients using stochastic gradient descent (SGD) or the Adam optimizer. : ,in This is the learning rate. Compared to the traditional IoU loss function, this application solves the gradient vanishing problem. When the tiny container gap target does not overlap with the preset anchor frame at all (IoU=0), the gradient of the traditional algorithm is 0, causing the parameter to fail to update to 1; while in this invention, as can be seen from the gradient calculation formula, even if the predicted box is far from the true box, The gradient still exists and is proportional to the distance, which ensures that the detection network can obtain continuous and effective gradient guidance, driving the predicted box to converge quickly to the real small target, significantly improving the training efficiency and robustness of the model under extremely small sample conditions.

[0107] For example: Assume the image size is: Pixel, target detection: a tiny gap where the "box door is not closed properly".

[0108] The bounding boxes output by the detection network are converted into a two-dimensional Gaussian distribution to enable distance calculation at the probability distribution level.

[0109] Ground Truth (GT) data: A manually labeled, tiny gap, with its center coordinates... (Image center); Size (A narrow slit, 10 pixels wide and 50 pixels high).

[0110] Gaussian parameter transformation: Converting a rectangle to a Gaussian distribution. Mean Covariance matrix An approximation of the frame size (usually the square of half the width and half the height): ; Prediction (Pred) data: A relatively biased guess given by the model during the early stages of training, including the center coordinates. The predicted center coordinates were shifted 20 pixels to the right; size The predicted size is 6 pixels wider than the actual size.

[0111] Gaussian parameter transformation: obtaining the distribution mean .

[0112] covariance matrix : ; At this point, the right boundary of the true bounding box is... The left boundary of the prediction box is in The two have absolutely no spatial overlap (No Overlap). If a traditional IoU algorithm is used, The gradient vanishes, and Loss cannot calculate the gradient, but after converting to a Gaussian distribution, as shown in the example... Figure 5 As shown, the blue curve represents the two-dimensional Gaussian distribution modeled as the effective receptive field (ERF) of the ground truth bounding box. The red curve represents the two-dimensional Gaussian distribution modeled as the effective receptive field of the prediction box. Even if the two distributions (peaks) do not overlap in their main spatial regions, there is still a calculable mathematical distance between them. This distance, as a loss function, can continuously provide gradient feedback to the neural network, ensuring that the localization parameters of the tiny box door gap target can be continuously updated and converged.

[0113] Calculate the relationship between the two Gaussian distributions mentioned above. Distance is the core component of the loss function.

[0114] Based on the second-order Wasserstein distance between the predicted and true bounding box distributions, the Wasserstein distance formula simplifies to the diagonal covariance matrix as follows: ; Expand into coordinate and dimension format: ; Numerical substitution calculation: The center point portion: ; Scale section (using half width / half height): Difference in width: ; Difference in height: ; Total distance at scale: ; Total distance squared: ; Normalized loss output: This value is usually normalized to the image scale or anchor box scale to prevent numerical explosion.

[0115] Assuming simple normalization, the Loss value .

[0116] Even if the two boxes do not touch, the calculation result This clearly tells the model two things: the main error lies in the positional offset (which contributes to...). The minor error was that the width prediction was too large (contributing to...). This provides an extremely precise direction for subsequent optimizations.

[0117] With a clear direction of error, the model can begin to be adjusted. Taking the correction process of the center point coordinate x as an example, let's set up the numerical environment: True center coordinates: (Image center).

[0118] Predicted center coordinates: (Offset 20 pixels to the right).

[0119] State: Due to the extremely narrow gap, the predicted bounding box does not overlap with the ground truth bounding box (IoU = 0).

[0120] Calculate the regression loss according to the gradient calculation formula. Forecast Center The partial derivatives of .

[0121] Assuming regression weights : ; Substitute the coordinate values ​​into the gradient formula above: ; Positive gradient value The gradient value indicates the current predicted location. The value is too high; the network parameters should be adjusted in the negative direction.

[0122] During backpropagation, this gradient is passed to the network weights.

[0123] To visually demonstrate the effect, let's assume a learning rate. Its direct corrective effect on the output results is as follows: ; ; Technical effect comparison: This application (Wasserstein Loss): Error before iteration:

[0124] Error after iteration:

[0125] Conclusion: The error was reduced from 20 pixels to 19.6 pixels. Even without overlap, the model still obtained a clear update direction, and the error was successfully reduced.

[0126] Existing technology (IoU Loss): Since the predicted bounding box does not overlap with the ground truth bounding box, Intersection = 0, resulting in Let be a constant, and calculate its gradient: ; Parameter update results: .

[0127] Conclusion: Gradient vanishing means the model cannot perceive the direction of the real target, training stagnates, and small targets are missed.

[0128] This embodiment intuitively demonstrates that for targets such as small and easily predictable deviations, like gaps in container doors, the Wasserstein distance loss function can provide a continuous and robust gradient flow. This makes the model of this invention converge much faster than traditional models, and it can be effectively trained on generated virtual samples (which may have some distribution offset), ultimately achieving a recognition rate of over 98%.

[0129] Based on the same inventive concept, this application also provides a method for detecting abnormalities in container doors, including: Acquire images of the container doors when the spreader descends to the preset trigger height; The container door image is enhanced using a multi-scale wavelet decomposition algorithm to obtain an enhanced image. The enhanced image is input into the container door anomaly detection model constructed using the container door anomaly detection model construction method described in any of the above embodiments for detection, and the door status detection result is obtained. When the door status detection result indicates that the door is open and the confidence level exceeds a preset threshold, it is determined that the door is abnormally open, and a stop signal is sent to the PLC within a preset response time to cut off the power source.

[0130] In implementation, the trained container door anomaly detection model is deployed in the actual operating environment to achieve a safety closed loop from "visual perception" to "equipment control," specifically: Deploy vision sensors at key locations on quay cranes or rail-mounted gantry cranes: install spreader vision sensors on the spreader and frame vision sensors on the trolley frame.

[0131] Set the collection trigger height (For example, 12 meters), when the programmable logic controller (PLC) provides feedback on the hoist height... At that time, the real-time image acquisition sequence is started. .

[0132] Setting parameters: System preset trigger height threshold meters; sampling frequency fps.

[0133] Implementation process: PLC transmits spreader height data stream in real time. ; time PLC reading is ,determination The system is in standby mode.

[0134] time PLC reading is ,determination Once the trigger condition is met (Triggered), the system immediately sends the command Capture_Start() to the camera, and... Mark the time with a timestamp and begin capturing image sequences of the box-grabbing / placing process at that height level.

[0135] For the collected sequence To address rain and fog interference, a multi-scale wavelet decomposition algorithm was used to decompose the image into low-frequency sub-images. and multi-scale high-frequency subgraphs ( For high-frequency subgraphs Adaptive Bayesian wavelet threshold estimation is used to filter out high-frequency noise components, and the image is reconstructed through inverse wavelet transform to obtain the dehazed intermediate image. .

[0136] It can also be used for The image undergoes illumination equalization channel processing, utilizing histogram equalization techniques to adjust the image brightness distribution and output a high-quality preprocessed image. .

[0137] Next, the preprocessed high-quality image The abnormality detection model of the container door, trained in the aforementioned embodiment, is used for detection to obtain the detection results of the container door status.

[0138] If the model output category is "DoorOpen" and the confidence level is... If a preset threshold is reached, it is considered an abnormality, and a stop signal is sent to the PLC within a preset response time to forcibly cut off the power until manual reset, thereby meeting the stringent safety requirements of industrial sites.

[0139] In implementation, the following assumptions are made: PLC control cycle: ; Frame rate of the detection system: FPS (frames per second) ); Preset detection height: ; Model confidence threshold: (Based on the optimal value setting of F1-Score).

[0140] The model for the current time For a single frame of image captured, classification inference is performed. Assume the classification probability vector output by the detection head for a candidate bounding box in the image is: .

[0141] Based on the preset model confidence threshold =0.596, and examples of judgments for different scenarios are as follows: Scenario A (obvious anomaly): ; Judgment: Category is DoorOpen; Confidence check: (Threshold); Result: Determined as abnormal.

[0142] Scenario B (Ambiguous Situation / False Alarm Suppression): ; Judgment: Although the probability of DoorOpen is relatively high, .

[0143] Result: Determined as normal or ignored (the system remains silent to prevent frequent accidental shutdowns).

[0144] Scenario C (Multi-objective comprehensive decision): In real-world scenarios, an image may contain multiple containers (such as the current container, the container adjacent to the left, and the container adjacent to the right). A comprehensive judgment is then made based on all detected targets. Target 1 (Current Box): DoorClose() ) normal; Target 2 (left neighbor): CNTR ( ) normal; Target 3 (right adjacent): DoorOpen ( ) An exception was triggered. Final frame result: If any target is determined to be abnormal, the entire frame is marked as: ; The threshold of 0.596 was not chosen arbitrarily, but rather based on a quantitative analysis of model performance, such as... Figure 6 As shown, the horizontal axis Confidence represents the confidence threshold, and the vertical axis F1 represents the F1 score (harmonic mean of precision and recall). The curves in the figure show the performance of the three categories: container overall (CNTR), door open (DoorOpen), and door closed (DoorClose). The thick blue solid line represents the overall performance of all categories.

[0145] Data shows that when the confidence threshold is set to 0.596, the model's overall F1 score reaches 0.99. This means that this application has extremely high superiority in balancing "false alarms" and "false positives," perfectly balancing safety and operational efficiency, and meeting the stringent safety requirements of port automation operations.

[0146] Furthermore, the container door anomaly detection model of this application also demonstrates excellent performance on the precision-recall curve, such as... Figure 7 As shown, the horizontal axis represents recall, and the vertical axis represents precision. The data shows that the mean precision (AP) for the CNTR and DoorOpe classes reached 0.995, while the DoorClose class reached 0.986, indicating a high overall average precision (mAP@0.5) of 0.992. The curve maintained a high precision level until the recall approached 1.0, demonstrating that the model could accurately identify abnormally opened doors with almost no false positives. This demonstrates that dehazing preprocessing (filtering out rain and fog noise from the image) and the CBMA attention enhancement mechanism effectively overcame environmental interference, providing a solid guarantee for high-precision detection.

[0147] Once the system detects an anomaly, it must complete the closed-loop control from "visual detection" to "mechanical shutdown" in a very short time to ensure that the power source is cut off as soon as the anomaly occurs, thus avoiding safety accidents.

[0148] In some embodiments, the container door anomaly detection method further includes: Obtain real-time status information of the spreader, the status information including at least one of: spreader height, locking signal and operation command; Based on the status information, determine whether the preset alarm activation conditions are met, and trigger the warning only when the conditions are met.

[0149] During implementation, the system determines whether to enable the detection results based on the external status signals transmitted from the PLC.

[0150] During the operation phase, if in box-grabbing mode, then If an anomaly is detected, an alarm is triggered; if the container is in unloading mode, monitoring continues while the spreader is locked.

[0151] Specifically, the input state variables include: Lifting height : Floating-point number, unit is meters; Spreading device locking signal : Boolean value, 1 indicates locking with box, 0 indicates unlocking with empty lifting device; Work instructions Enum, which includes Pick and Drop.

[0152] Case 1: High-altitude operation (non-detection area); state: , .

[0153] logic: (Preset detection height).

[0154] Action: Even if the visual algorithm falsely detects that the cabinet door is open (possibly due to reflective interference), the system logic layer will forcibly disable the alarm.

[0155] Result: System in standby mode.

[0156] Case 2: Container descent phase (empty container detection); state: , (Empty lifting equipment) .

[0157] logic: and .

[0158] Strategy: Activate the spreader camera to detect the container below, and activate the trolley frame camera to detect adjacent bay positions.

[0159] Result: Full detection was activated.

[0160] Case 3: Box placement and locking stage (load detection); state: , (with box) .

[0161] logic: .

[0162] Strategy: The lifting device continuously monitors whether the bottom / side of the container being lifted has an open door; the trolley frame camera detects adjacent containers at the target's landing position.

[0163] Result: Full detection was activated.

[0164] Since pure vision algorithms are easily affected by environmental interference, this application introduces PLC state constraints to construct a "spatiotemporal electronic fence," which only activates the alarm within a specific process window, demonstrating system-level robustness.

[0165] In some implementations, the system in The PLC sends a stop signal to forcibly cut off the power until it is manually reset.

[0166] Specifically, the process moves from the "algorithm layer" to the "control layer" to verify whether the condition is less than... The indicators demonstrate that the system has the capability for practical application in industrial settings.

[0167] Timeline deduction: After the camera exposure ends, the image is transferred to the industrial computer's video memory. (Algorithm Processing): The GPU performs dehazing, WGAN feature comparison, CBMA attention calculation, and head regression; the software internally uses flags. ; (Signal Output): The industrial control computer uses an I / O card to change the hard-wired voltage level from high ( ) lower ( ), as a fault trigger signal; (PLC Scan): The PLC detects that the input terminal has gone low in the next scan cycle and executes the following logic: ; (Mechanical Response): The PLC outputs a stop command, the frequency converter cuts off the motor power supply, and the holding brake closes.

[0168] Total response time calculation: ; The actual response time (55 ms) is far below the required specification, which can quickly cut off the power source and effectively prevent accidents from occurring.

[0169] This embodiment, through close cooperation between hardware and software, ensures that the power source is cut off immediately upon the occurrence of an anomaly, thereby preventing physical collisions between the spreader and the abnormally opened container door and protecting the safety of port equipment.

[0170] In some embodiments, the container door anomaly detection method further includes: When an abnormal opening of the cabinet door is detected, the current image is saved and an alarm message is sent to the remote control center.

[0171] For example: during the placement of the box, at a distance from the ground At this point, an abnormal opening of the adjacent door on the left is detected. At this time, the industrial control computer interface displays a red alarm box (the color is only for example and can be adjusted according to the actual configuration) and automatically saves the current screenshot. After receiving the alarm signal, the PLC immediately locks the lifting mechanism of the hoist and prohibits further descent. The remote control center receives an alarm pop-up window and the operator needs to manually intervene to confirm.

[0172] This application is particularly applicable to the real-time intelligent identification, precise positioning, and closed-loop safety early warning of container door status during container handling operations of quay cranes, rail-mounted gantry cranes, and horizontal transport equipment in automated container terminals (ACTs), especially in complex all-weather weather conditions such as rain, fog, and nighttime, and in scenarios where there is a scarcity of samples of abnormal container door opening incidents.

[0173] Based on the same inventive concept, this application also provides a container door anomaly detection device, comprising: The image acquisition module is used to acquire images of the container doors when the spreader descends to a preset trigger height; The image processing module is used to preprocess the container door image based on the multi-scale wavelet decomposition algorithm to obtain an enhanced image; The detection module is used to input the enhanced image into the container door anomaly detection model constructed using the container door anomaly detection model construction method described in any of the above embodiments for detection, and to obtain the door status detection result; The closed-loop real-time early warning module is used to determine that the door is abnormally opened when the door status detection result is that the door is open and the confidence level exceeds a preset threshold. It then sends a stop signal to the PLC within a preset response time to cut off the power source.

[0174] In this specification, the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the descriptions of the embodiments described later are relatively simple, and relevant parts can be referred to the descriptions of the foregoing embodiments.

[0175] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a container door anomaly detection model, characterized in that, include: Obtain real abnormal samples and preprocess the real abnormal samples; Based on the Wasserstein generative adversarial network, a virtual sample generation model is constructed. The generator is trained using the preprocessed real anomaly samples to generate virtual anomaly samples. The virtual anomaly samples are then mixed with the real anomaly samples to construct a training dataset. Construct a deep detection network that incorporates a joint attention enhancement mechanism; The deep detection network is trained using the training dataset, and a loss function based on Wasserstein distance is used as the regression loss during the training process to obtain a trained container door anomaly detection model.

2. The method for constructing a container door anomaly detection model according to claim 1, characterized in that, The construction of the training dataset also includes: The virtual anomaly samples and the real anomaly samples are augmented with data to expand the training dataset; wherein the data augmentation includes at least one of geometric transformation and color transformation.

3. The method for constructing a container door anomaly detection model according to claim 1, characterized in that, The preprocessing of the real abnormal samples includes: A multi-scale wavelet decomposition algorithm is used to filter out rain and fog noise in the image; And / or, use an illumination equalization algorithm to adjust the brightness distribution of the image.

4. The method for constructing a container door anomaly detection model according to claim 1, characterized in that, The construction of the deep detection network including the joint attention enhancement mechanism includes: The training dataset is input into the backbone network for feature extraction to obtain a feature map; The feature map is subjected to spatial pyramid pooling processing, and multi-scale contextual information is fused to obtain a multi-scale feature map; Global max pooling and global average pooling are performed on the multi-scale feature maps respectively, and the pooling results are input into a shared multilayer perceptron for processing to generate channel weight vectors. Max pooling and average pooling are performed on the multi-scale feature map along the channel axis, and the pooling results are concatenated and input into the convolutional layer for processing to generate a spatial weight map. The channel weight vector and the spatial weight map are multiplied by the multi-scale feature map to obtain the enhanced feature map.

5. The method for constructing a container door anomaly detection model according to claim 4, characterized in that, The construction of the deep detection network including the joint attention enhancement mechanism also includes: The enhanced feature map is upsampled and then concatenated and fused with the shallow feature map output by the backbone network to obtain a fused feature map. The fused feature map is input into the detection head for convolution processing to obtain the class probability and bounding box coordinates of the door state.

6. The method for constructing a container door anomaly detection model according to claim 1, characterized in that, The use of a loss function based on Wasserstein distance as the regression loss includes: The predicted bounding boxes and the ground truth bounding boxes are mapped to the probability distribution space, and the Wasserstein distance between the two probability distributions is used as the regression loss.

7. A method for detecting abnormalities in container doors, characterized in that, include: Acquire images of the container doors when the spreader descends to the preset trigger height; The container door image is enhanced using a multi-scale wavelet decomposition algorithm to obtain an enhanced image. The enhanced image is input into the container door anomaly detection model constructed using the container door anomaly detection model construction method as described in any one of claims 1-6 for detection, and the door status detection result is obtained. When the door status detection result indicates that the door is open and the confidence level exceeds a preset threshold, it is determined that the door is abnormally open, and a stop signal is sent to the PLC within a preset response time to cut off the power source.

8. The container door anomaly detection method according to claim 7, characterized in that, The container door anomaly detection method also includes: Obtain real-time status information of the spreader, the status information including at least one of: spreader height, locking signal and operation command; Based on the status information, determine whether the preset alarm activation conditions are met, and trigger the warning only when the conditions are met.

9. The container door anomaly detection method according to claim 7, characterized in that, The container door anomaly detection method also includes: When an abnormal opening of the cabinet door is detected, the current image is saved and an alarm message is sent to the remote control center.

10. A container door anomaly detection device, characterized in that, include: The image acquisition module is used to acquire images of the container doors when the spreader descends to a preset trigger height; The image processing module is used to preprocess the container door image based on the multi-scale wavelet decomposition algorithm to obtain an enhanced image; The detection module is used to input the enhanced image into a container door anomaly detection model constructed using the container door anomaly detection model construction method as described in any one of claims 1-5 for detection, and to obtain the door status detection result; The closed-loop real-time early warning module is used to determine that the door is abnormally opened when the door status detection result is that the door is open and the confidence level exceeds a preset threshold. It then sends a stop signal to the PLC within a preset response time to cut off the power source.

Images