Bucket tooth abnormality detection method, bucket tooth abnormality detection device, electronic device, and product

Abstract

This application relates to the field of visual inspection technology for engineering machinery, and provides a method, device, electronic device, and product for detecting bucket teeth anomalies. The bucket teeth anomaly detection method includes: training a multi-scale memory-enhanced autoencoder network based on normal bucket teeth samples, enabling the network to learn multi-scale feature representations and reconstruction capabilities of normal bucket teeth; reconstructing an input bucket teeth image to be detected using the trained multi-scale memory-enhanced autoencoder network to obtain a reconstructed image; calculating the reconstruction error between the bucket teeth image to be detected and the reconstructed image; and based on the reconstruction error, determining whether the bucket teeth image to be detected exhibits anomalies, and locating the abnormal bucket teeth if anomalies are present. This application enables reliable detection of bucket teeth anomalies even in the absence of abnormal samples, and can detect unknown anomaly types that have never appeared in the training set.

Description

Technical Field

[0001] This application belongs to the field of visual inspection technology for engineering machinery, and in particular relates to a method, device, electronic equipment and product for detecting abnormal bucket teeth. Background Technology

[0002] During operation, the bucket teeth of excavators (such as electric shovels) may experience abnormalities such as wear, breakage, or detachment. Timely detection of these abnormalities is crucial for ensuring safe equipment operation and preventing foreign objects from entering the crusher and causing downtime.

[0003] Existing bucket tooth anomaly detection solutions primarily employ supervised deep learning methods, such as YOLO object detection models combined with segmentation networks, or Regional Proposal Networks (RPNs). These solutions rely heavily on a large number of manually labeled broken and missing tooth samples for model training. However, in real-world production environments, bucket tooth anomalies are low-probability events, making it extremely difficult to collect anomaly samples, exhibiting a typical long-tail distribution. A single excavator might experience a bucket tooth loss only once every few months of continuous operation. This results in a severe shortage of samples for supervised learning models across anomaly categories, leading to poor generalization ability and a high likelihood of missed detections.

[0004] Furthermore, existing solutions generally categorize bucket tooth anomalies into limited types such as detachment, breakage, and wear, failing to detect unknown anomaly types not defined in the training set, such as twisting deformation and abnormal tilting. When new failure modes emerge, it is necessary to collect samples, label data, and train models again, resulting in long deployment cycles and high costs. Summary of the Invention

[0005] This application provides a method, device, electronic equipment, and product for detecting abnormal bucket teeth, which can reliably detect abnormal bucket teeth even in the absence of abnormal samples, and can detect unknown abnormality types that have never appeared in the training set.

[0006] In a first aspect, embodiments of this application provide a method for detecting abnormal bucket teeth, including: The multi-scale memory-enhanced autoencoder network is trained based on normal bucket teeth samples, enabling the multi-scale memory-enhanced autoencoder network to learn the multi-scale feature representation and reconstruction capabilities of normal bucket teeth. The input image of the target beak is reconstructed by the trained multi-scale memory-enhanced autoencoder network to obtain the reconstructed image. Calculate the reconstruction error between the image of the bucket teeth to be detected and the reconstructed image; Based on the reconstruction error, it is determined whether there is any abnormality in the bucket teeth image to be detected, and the abnormal bucket teeth are located if such abnormality exists.

[0007] In this embodiment, a multi-scale memory-enhanced autoencoder network (MMECN) is trained based on normal beak tooth samples. This allows the MMECN to learn the multi-scale feature representations and reconstruction capabilities of normal beak teeth, enabling it to accurately reconstruct normal beak tooth images with minimal reconstruction error. However, for any abnormal beak teeth that deviate from the learned multi-scale feature representations of normal beak teeth (including unknown abnormality types not found in the training set), significant reconstruction errors will occur. Therefore, this scheme uses the MMECN to reconstruct the beak tooth image to be detected and calculates the reconstruction error between the image to be detected and the reconstructed image. Based on this reconstruction error, it can accurately determine whether there are beak tooth abnormalities in the image to be detected, even in the absence of abnormal beak tooth samples, and locate abnormal beak teeth when they are present. This achieves reliable detection of beak tooth abnormalities and can detect unknown abnormality types that have never appeared in the training set.

[0008] In some embodiments of the first aspect, the multi-scale memory-enhanced autoencoder network includes: an encoder, a memory module, and a decoder employing a multi-scale feature pyramid structure; the reconstructing of the input to-beam image by using the trained multi-scale memory-enhanced autoencoder network to obtain a reconstructed image includes: The encoder is used to extract multi-scale features from the image of the tusk to be detected, thereby obtaining the coding feature maps of each layer of the encoder; The memory module retrieves a memory-enhanced feature vector related to the one-dimensional feature vector of the last layer of encoded feature map. This one-dimensional feature vector is obtained by performing a global average pooling operation on the last layer of encoded feature map. The memory-enhanced feature vector is obtained by combining at least two memory vectors from the memory matrix, which includes the normal beak teeth. There are 3 memory vectors, each with the same dimension as the one-dimensional feature vector. It is an integer greater than 1; The decoder performs a spatial reshaping operation on the memory-enhanced feature vector to restore it as a spatial feature map. When the remaining feature map of the encoder is passed to the corresponding layer of the decoder through skip connections, the spatial feature map and the remaining feature map are processed by the decoder to obtain the reconstructed image, wherein the remaining feature map is the feature map of each layer except the last layer of coding feature map.

[0009] In some embodiments of the first aspect, retrieving the memory-enhanced feature vector associated with the one-dimensional feature vector of the last layer encoded feature map via the memory module includes: Through the memory module, using the first formula, the one-dimensional feature vector and the memory matrix are processed to obtain the... Attention weights for each memory vector; Calculate the memory module The attention weights of each memory vector are multiplied by the corresponding memory vector, and the products are summed to obtain the memory-enhanced feature vector. The first formula is:

[0010] in, For the first Attention weights for each memory vector. For the first A memory vector, For the first A memory vector, Indicates the temperature coefficient. It is the transpose of the one-dimensional eigenvector.

[0011] In some embodiments of the first aspect, training the multi-scale memory-enhanced autoencoder network based on normal tusk samples includes: Based on the normal tooth samples, reconstruction loss, memory sparsity constraint loss, and memory compactness loss are determined. The memory sparsity constraint loss is used to constrain the attention weight of a single memory vector to not exceed the maximum allowed weight, where the maximum allowed weight is a positive number less than 1. The memory compactness loss is used to promote the clustering of the memory vectors. The reconstruction loss, the memory sparsity constraint loss, and the memory compactness loss are fused together to obtain a composite loss. The multi-scale memory-reinforced autoencoder network is trained based on the composite loss.

[0012] In some embodiments of the first aspect, the reconstruction error includes: pixel-level reconstruction error, perceptual-level reconstruction error, and structural similarity error, wherein the perceptual-level reconstruction error is calculated using intermediate layer features of the VGG16 network; the step of determining whether the detected beak tooth image has beak tooth abnormalities based on the reconstruction error, and locating abnormal beak teeth if beak tooth abnormalities exist, includes: The perceptual-level reconstruction error is interpolated to the original resolution to obtain the interpolated perceptual-level reconstruction error, wherein the original resolution is the resolution of the image of the tusk to be detected. The pixel-level reconstruction error, the interpolated perceptual-level reconstruction error, and the structural similarity error are normalized respectively. The pixel-level reconstruction error after normalization, the perceptual reconstruction error after interpolation, and the structural similarity error are fused to obtain a pixel-level anomaly score map of the to-be-detected tusk image. Based on the pixel-level anomaly scoring map, it is determined whether the image of the bucket teeth to be detected has any bucket tooth anomalies, and if the bucket tooth anomalies are present, the abnormal bucket teeth are located.

[0013] In some embodiments of the first aspect, determining whether the buckling tooth image to be detected has any buckling tooth abnormality based on the pixel-level anomaly scoring map, and locating the abnormal buckling tooth if the buckling tooth abnormality exists, includes: The pixel-level anomaly scoring map is binarized using an anomaly threshold to obtain an anomaly mask. If there is a connected region in the abnormal mask with an area greater than or equal to the area of ​​the minimum broken tooth, then the image of the tooth to be detected is determined to have the abnormal tooth condition, and the connected region is identified as the abnormal region. Based on the centroid abscissa of the abnormal region, the width of the image of the bucket tooth to be detected, and the total number of bucket teeth in the image of the bucket tooth to be detected, the bucket tooth partition where the abnormal region is located is determined, and the bucket tooth corresponding to the bucket tooth partition where the abnormal region is located is determined as the abnormal bucket tooth.

[0014] In some embodiments of the first aspect, prior to binarizing the pixel-level anomaly scoring map using an anomaly threshold, the method further includes: Calculate the information entropy of the image of the bucket teeth to be detected; The anomaly threshold is determined based on the information entropy, the mean and standard deviation of the pixel-level anomaly scores of the normal beak tooth samples.

[0015] In some embodiments of the first aspect, prior to binarizing the pixel-level anomaly scoring map using an anomaly threshold, the method further includes: The 95th percentile value of the pixel-level anomaly scoring map is determined as the image-level comprehensive anomaly score of the bucktooth image to be detected. The step of binarizing the pixel-level anomaly scoring map using an anomaly threshold includes: If the image-level comprehensive anomaly score is greater than or equal to a preset score threshold, then the pixel-level anomaly score map is binarized using the anomaly threshold.

[0016] In some embodiments of the first aspect, after obtaining the pixel-level anomaly scoring map of the image of the to-beam, the method further includes: The pixel-level anomaly scoring map of the bucket tooth image to be detected is mapped to a Jet pseudo-color heat map; The Jet pseudo-color heatmap is then overlaid on the image of the bucket teeth to be detected and displayed.

[0017] In some embodiments of the first aspect, after locating the abnormal teeth, the method further includes: If in the most recent consecutive If the number of frames in the image of the bucket teeth to be detected that locate the abnormal bucket teeth is greater than the trigger threshold, an abnormal alarm will be output. It is an integer greater than 1.

[0018] Secondly, embodiments of this application provide a bucket tooth abnormality detection device, comprising: The network training module is used to train a multi-scale memory-enhanced autoencoder network based on normal bucket teeth samples, so that the multi-scale memory-enhanced autoencoder network learns the multi-scale feature representation and reconstruction capability of normal bucket teeth. The image reconstruction module is used to reconstruct the input image of the target tusk using the trained multi-scale memory-enhanced autoencoder network to obtain a reconstructed image. An error calculation module is used to calculate the reconstruction error between the image of the bucket tooth to be detected and the reconstructed image; An anomaly detection module is used to determine whether there are any abnormalities in the bucket teeth image to be detected based on the reconstruction error, and to locate the abnormal bucket teeth if such abnormalities are found.

[0019] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device performs the method as described in any one of the first aspects above.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computer, implements the method as described in any one of the first aspects above.

[0021] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the method as described in any one of the first aspects above to be performed.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic flowchart of a method for detecting abnormal bucket teeth provided in an embodiment of this application; Figure 2 This is another schematic flowchart of the bucket tooth anomaly detection method provided in the embodiments of this application; Figure 3 This is a structural example diagram of a multi-scale memory-enhanced autoencoder network according to an embodiment of this application; Figure 4 This is another flowchart illustrating the method for detecting abnormal bucket teeth provided in the embodiments of this application; Figure 5 This is another schematic flowchart of the bucket tooth anomaly detection method provided in the embodiments of this application; Figure 6 This is another schematic flowchart of the bucket tooth anomaly detection method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the bucket tooth abnormality detection device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0030] The bucket tooth anomaly detection method provided in this application embodiment can be applied to electronic devices such as servers, tablets, vehicle-mounted equipment integrated into excavators (e.g., vehicle-mounted edge computing units), desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application embodiment does not impose any restrictions on the specific type of electronic device.

[0031] To illustrate the technical solution of this application, specific embodiments are described below.

[0032] Please see Figure 1 , Figure 1 The flowchart illustrating the bucket tooth anomaly detection method provided in this application is shown as an example and not a limitation. The method includes the following steps: Step 101: Train the multi-scale memory-enhanced autoencoder network based on normal beak tooth samples, so that the multi-scale memory-enhanced autoencoder network can learn the multi-scale feature representation and reconstruction ability of normal beak teeth.

[0033] In this embodiment, by training a multi-scale memory-reinforced autoencoder network (MRAM) with a large number of normal beak tooth samples, the MRAM can learn the multi-scale feature representations and reconstruction capabilities of normal beak teeth. By enabling the MRAM to learn these multi-scale feature representations and reconstruction capabilities, the detection paradigm for beak tooth anomalies can be shifted from "identifying what is abnormal" to "learning what is normal." This constructs a single-class learning detection framework based on normal beak tooth samples, classifying beak teeth that deviate from the multi-scale feature representations of normal beak teeth as abnormal, thus achieving the deployment of beak tooth anomaly detection capabilities under zero-abnormal beak tooth sample conditions. Furthermore, the MRAM in this embodiment learns the multi-scale feature representations of normal beak tooth samples, which solves the problem that traditional autoencoders cannot simultaneously consider global structure and local details when learning single-scale features.

[0034] Step 102: The input image of the target tusk is reconstructed using a trained multi-scale memory-enhanced autoencoder network to obtain the reconstructed image.

[0035] The image to be detected can refer to an image containing the teeth of the bucket.

[0036] In some embodiments, the electronic device can receive bucket images acquired in real time by a charge-coupled device (CCD) industrial camera with a resolution of 1920×1080 pixels and a frame rate of 25fps. The electronic device can then perform image enhancement processing on the bucket image, using adaptive histogram equalization to improve contrast, and subsequently use a pre-trained bucket detection model to crop out a local image containing only the bucket teeth (i.e., the bucket tooth region), which is the bucket tooth image to be detected.

[0037] Optionally, before inputting the image of the bucket tooth to be detected into the multi-scale memory-enhanced autoencoder network, the image can be normalized to a preset size. This unifies the scaling of images of the bucket tooth from different buckets or taken at different distances and angles to a preset resolution, providing standardized input for the multi-scale memory-enhanced autoencoder network and ensuring its normal operation. Furthermore, scaling to a preset resolution avoids complex memory allocation or frame-by-frame adjustments required due to varying image sizes, significantly improving the reconstruction efficiency and stability of the multi-scale memory-enhanced autoencoder network. Optionally, the preset resolution can be set based on empirical values ​​or actual needs. For example, the preset resolution could be 256×256 pixels.

[0038] Step 103: Calculate the reconstruction error between the image of the bucket teeth to be detected and the reconstructed image.

[0039] Among them, the reconstruction error characterizes the degree of difference between the image of the bucket tooth to be detected and the reconstructed image.

[0040] Step 104: Based on the reconstruction error, determine whether there are any abnormalities in the bucket teeth image to be detected, and locate the abnormal bucket teeth if abnormalities are found.

[0041] Because the multi-scale memory-enhanced autoencoder network has learned the multi-scale feature representation and reconstruction capabilities of normal beak teeth, when an image of normal beak teeth is input, the network can accurately reconstruct it using this learned feature representation and reconstruction capability. The output reconstructed image is very close to the normal beak tooth image, with minimal difference, meaning the reconstruction error is small. However, when an image of abnormal beak teeth is input (e.g., broken, detached, twisted, tilted, or with foreign matter adhering, etc.), its abnormal features cannot be accurately represented by the multi-scale feature representation learned by the network, nor can they be accurately reconstructed by the network's learned reconstruction capability (i.e., the reconstruction capability of normal beak teeth). This leads to reconstruction failure, and the output reconstructed image will show distortion or errors in the abnormal region, showing a significant difference from the abnormal beak tooth image, meaning the reconstruction error is large. Therefore, based on the reconstruction error, beak tooth anomaly detection can be accurately performed on the image to be detected, solving the problem of traditional autoencoders easily overgeneralizing (i.e., reconstructing abnormal beak tooth samples with low error). A normal fighting tooth image can be defined as an image in which all fighting teeth are normal and without any abnormalities. An abnormal fighting tooth image can be defined as an image containing at least one abnormal fighting tooth.

[0042] In this embodiment, a multi-scale memory-enhanced autoencoder network (MMECN) is trained based on normal beak tooth samples. This allows the MMECN to learn the multi-scale feature representations and reconstruction capabilities of normal beak tooth samples, enabling it to accurately reconstruct normal beak tooth images with minimal reconstruction error. However, for any abnormal beak tooth that deviates from the learned multi-scale feature representations of normal beak teeth (including unknown abnormality types not found in the training set), significant reconstruction errors will occur. Therefore, this scheme uses the MMECN to reconstruct the beak tooth image to be detected and calculates the reconstruction error between the image to be detected and the reconstructed image. Based on this reconstruction error, it can accurately determine whether there are beak tooth abnormalities in the image to be detected, even in the absence of abnormal beak tooth samples, and locate abnormal beak teeth when they are present. This achieves reliable detection of beak tooth abnormalities and can detect unknown abnormality types that have never appeared in the training set.

[0043] In some embodiments of this application, the multi-scale memory-enhanced autoencoder network includes an encoder, a memory module, and a decoder employing a multi-scale feature pyramid structure. Based on this, the above-mentioned reconstruction of the input to-beam image by the trained multi-scale memory-enhanced autoencoder network to obtain a reconstructed image may include steps 201 to 203.

[0044] Step 201: Multi-scale feature extraction is performed on the image of the bucket teeth to be detected by the encoder to obtain the coding feature maps of each layer of the encoder.

[0045] In this embodiment, the encoder adopts a multi-scale feature pyramid structure to extract features at different resolution levels. It can simultaneously capture shallow high-resolution features that retain rich local details such as edges and textures, as well as global semantic information that encodes the overall shape of the bucket teeth and the relative positions and layouts of each part, thereby achieving a comprehensive and refined representation of the bucket teeth features.

[0046] In some embodiments, the encoder may include multiple layers of convolutional blocks and max-pooling operations connected between different convolutional blocks to construct a multi-scale feature pyramid structure. Optionally, the number of layers in the convolutional blocks in the encoder can be set according to empirical values ​​or actual needs.

[0047] Step 202: Retrieve the memory-enhanced feature vector associated with the one-dimensional feature vector of the last layer encoded feature map through the memory module.

[0048] The one-dimensional feature vector is obtained by performing global average pooling on the last layer of encoded feature map. The memory-enhanced feature vector is obtained by combining at least two memory vectors from the memory matrix, which includes the normal beak teeth. There are 3 memory vectors, each with the same dimension as the one-dimensional feature vector. It is an integer greater than 1. Each memory vector is used to cover the variations in the appearance of normal bucket teeth under different working conditions (e.g., lighting, soil conditions, wear level, viewing angle, minor posture differences, etc.). Optionally, the size of the memory matrix can be set according to empirical values ​​or actual needs. .For example, It can be set to 500 to balance expressive power and generalization ability.

[0049] The memory module maintains a learnable memory matrix, which contains... All memory vectors have the same dimension. Each memory vector can be understood as a set of normal pattern feature prototypes or feature dictionaries automatically learned from a large number of normal buck teeth samples. Each memory vector is not a manually defined category such as broken teeth or missing teeth, but rather a high-level semantic feature optimized during training to represent a certain common normal appearance state (e.g., overall shape outline, structural relationship between tooth root and tooth tip, relative layout within the cropped area (i.e., buck teeth image)). During inference, the electronic device matches the original features (i.e., encoded feature maps) extracted from the currently input buck teeth image to be detected with this set of prototypes, using a weighted combination of several prototypes to constrain and replace the original features, and then reconstructs the image through a decoder. When the input buck teeth image to be detected belongs to the normal pattern (i.e., the buck teeth image to be detected is a normal buck teeth image), it can be represented well by these prototypes, resulting in high-quality reconstruction. When the input buck teeth image to be detected exhibits abnormal structural changes, the matching with these prototypes deteriorates, and the resulting reconstruction becomes distorted.

[0050] In this embodiment, by introducing a memory module between the encoder and decoder, the expressive power of the latent space can be limited, preventing the multi-scale memory-enhanced autoencoder network from remembering the features of abnormal tooth samples.

[0051] In this embodiment, the last layer of encoded feature map output by the encoder is subjected to global average pooling to compress the spatial dimension, resulting in a one-dimensional feature vector with the same dimension as the memory vector. This provides a standardized input for the retrieval of the memory module, and the spatial dimension compression greatly reduces the amount of data, significantly reducing the computation and storage overhead of the memory module.

[0052] In this embodiment, the enhanced feature vector is obtained by combining at least two memory vectors in the memory matrix. This can prevent the memory module from degenerating into an invalid identity mapping, improve the fine representation of the diversity of normal teeth, enhance generalization, and enhance the mismatch effect on abnormal teeth, thereby improving detection sensitivity.

[0053] Step 203: The memory-enhanced feature vector is spatially reshaped using a decoder to restore it to a spatial feature map.

[0054] The spatial feature map mentioned above has the same size as the last layer of encoded feature map output by the encoder.

[0055] In this embodiment, the decoder adopts a structure symmetrical to the encoder. The memory-enhanced feature vector (e.g., dimension 512) is reshaped to restore a spatial feature map (e.g., size 512). This enables the spatial feature map to match the spatial dimension of the encoder's deep features.

[0056] Step 204: In the case that the remaining feature map of the encoder is passed to the corresponding layer of the decoder through skip connections, the spatial feature map and the remaining feature map are processed by the decoder to obtain the reconstructed image.

[0057] The remaining feature maps are the feature maps in each layer of coding feature maps, excluding the last layer of coding feature maps.

[0058] In this embodiment, skip connections are introduced to directly pass the shallow features of the encoder to the corresponding layer of the decoder, which can enhance the decoder's ability to recover details. In this way, the decoder can utilize the packet resolution detail information retained by the encoder to improve the reconstruction quality.

[0059] It should be understood that skip connections do not weaken the anomaly suppression effect of the memory module because skip connections only transmit shallow features (such as low-level visual information like edges and textures), which do not include high-level semantics such as tooth integrity. Abnormal and normal teeth differ little in edge texture; what truly distinguishes them are deep semantic features (such as the shape integrity and relative positional relationship of the teeth), and these deep features must be processed by the memory module. The memory module only stores the semantic features of normal teeth and cannot accurately match the deep features of abnormal teeth, leading to reconstruction distortion. Based on this, abnormal tooth detection can be achieved.

[0060] In some embodiments of this application, for the one-dimensional feature map of the last layer of encoded feature map output by the encoder, relevant memory vectors can be retrieved from the memory matrix through an attention mechanism (the larger the attention weight of the memory vector, the more relevant it is to the one-dimensional feature vector), and the memory-enhanced feature vector is obtained based on the combination of the retrieved relevant memory vectors. Specifically, the above-mentioned retrieval of the memory-enhanced feature vector related to the one-dimensional feature vector of the last layer of encoded feature map through the memory module includes: By using the memory module and the first formula, the one-dimensional eigenvector and the memory matrix are processed to obtain... Attention weights for each memory vector; Calculation via memory module The attention weights of each memory vector are multiplied by the corresponding memory vector, and the products are summed to obtain the memory-enhanced feature vector. The first formula is:

[0061] in, For the first Attention weights for each memory vector. For the first A memory vector, For the first A memory vector, Indicates the temperature coefficient. It is the transpose of a one-dimensional eigenvector.

[0062] The formula for calculating the feature vector after memory enhancement is as follows:

[0063] in, This is the feature vector after memory enhancement.

[0064] The attention weight of a memory vector represents the degree of similarity or matching between the one-dimensional feature vector output by the encoder and the memory vector. The attention weight is dimensionless and ranges from 0 to 1.

[0065] The temperature coefficient can be understood as the degree of softness or hardness in attention allocation, or a discrimination adjustment coefficient, used to control whether the attention weights are more evenly distributed or more concentrated when retrieving the memory vector. The temperature coefficient is dimensionless. Optionally, the temperature coefficient can be set based on empirical values ​​or actual needs. For example, a temperature coefficient of 0.1.

[0066] In this embodiment, the temperature system can effectively control the concentration of attention weight distribution. A smaller temperature coefficient makes the attention weight distribution sharper and relies on the most similar minority memory vectors, while a larger temperature coefficient makes the attention weight distribution smoother and references more memory vectors more evenly. This improves the flexibility of the memory module, enabling it to achieve a balance between the "accuracy" and "robustness" of retrieval to adapt to different application scenarios.

[0067] In this embodiment, by using the attention weights of each memory vector and weighting and fusing them based on these attention weights, the system can automatically focus on several normal pattern prototypes most relevant to the currently input image of the beak to be detected. This enables adaptive feature enhancement based on similarity, allowing the output enhanced features to more accurately reflect the unique context of the image of the beak to be detected. This improves the discriminative power between normal and abnormal beaks and enhances the memory model's ability to characterize the diversity of normal beaks (e.g., changes in beaks at different wear stages and under different lighting conditions). This, in turn, improves the inclusiveness (i.e., generalization ability) of subtle differences in normal beaks.

[0068] like Figure 3 The diagram shown is an example of the structure of a multi-scale memory-enhanced autoencoder network according to an embodiment of this application. The size of the target bucket tooth image input to the encoder is 256×256. The encoder includes four convolutional blocks: convolutional block 1, convolutional block 2, convolutional block 3, and convolutional block 4. Max pooling operations are set between adjacent convolutional blocks (not in...). Figure 3 (As shown). The decoder includes four layers of deconvolution blocks: deconvolution block 1, deconvolution block 2, deconvolution block 3, and deconvolution block 4.

[0069] by Figure 3 The data processing of the encoder, memory module, and decoder is introduced using the multi-size memory-enhanced autoencoder network and the convolutional block as an example of a residual block containing two 3×3 convolutional layers.

[0070] The encoder's first The formula for calculating layer features is as follows:

[0071] in, For the first Layer-encoded feature map, size is , and The height and width of the image of the tusk to be detected (in pixels, value 256). For the first The number of channels per layer is 64, 128, 256, and 512, respectively. For a residual block containing two 3×3 convolutional layers, This is a 2×2 max pooling operation.

[0072] Feature map output by encoder (Size 16×16×512) First, a global average pooling operation is performed to compress the spatial dimension, resulting in a one-dimensional feature vector. The calculation formula is as follows:

[0073] in, It is a one-dimensional feature vector. This is a global average pooling operation. and For feature map Height and width, and For traversing feature maps Index variables for spatial dimensions (i.e., height and width).

[0074] The memory module maintains a learnable memory matrix. ,Include There are memory vectors, each with a dimension of . ( For the one-dimensional feature vector output by the encoder. The most relevant memory vector is retrieved from the memory matrix through an attention mechanism (the larger the attention weight of the memory vector, the more relevant it is to the one-dimensional feature vector). (The more relevant), the retrieved memory vectors are then weighted and fused to obtain the memory-enhanced feature vector. (Dimensions are 512).

[0075] The decoder uses a spatial reshaping operation to transform the memory-enhanced feature vectors. Restored to spatial feature map (size is) ), to match the spatial dimension of deep features of the encoder.

[0076] Subsequently, the decoder's deconvolutional blocks gradually recover the image resolution through upsampling and convolution. Building upon this, the decoder introduces skip connections, and then... The output of the first layer of deconvolution block The formula for calculating the layer decoding feature map is as follows:

[0077] The formula for calculating the first layer decoded feature map output from the first deconvolution block is as follows:

[0078] in, For the first Layer decoding feature map, It is a 2×2 bilinear interpolation upsampling function. For splicing operations, The first layer of decoded feature map can be used to determine the reconstructed image from the first layer decoder.

[0079] In some embodiments of this application, such as Figure 4 As shown, training a multi-scale memory-enhanced autoencoder network based on normal tusk samples can include steps 401 to 403.

[0080] Step 401: Based on normal bucket tooth samples, determine the reconstruction loss, memory sparsity constraint loss, and memory compactness loss.

[0081] Among them, the memory sparsity constraint loss is used to constrain the attention weight of a single memory vector to not exceed the maximum allowed weight, which is a positive number less than 1, and the memory compactness loss is used to promote the clustering of memory vectors.

[0082] The reconstruction losses mentioned above include pixel loss, perceptual loss, and structural similarity loss. The calculation formulas for these three losses can be found in the calculation formulas for pixel-level reconstruction error, perceptual-level error, and structural similarity error below.

[0083] In this embodiment, the training process of the multi-scale memory-enhanced autoencoder network only requires normal bucket tooth samples, without the need to collect and label abnormal bucket tooth samples. This enables reliable bucket tooth anomaly detection even with only normal bucket tooth samples available, significantly shortening the deployment cycle and reducing deployment costs compared to traditional supervised methods. The training set can be expanded by randomly cropping, rotating, color-jittering, or applying Gaussian blur to the normal bucket tooth samples.

[0084] Step 402: The reconstruction loss, memory sparsity constraint loss, and memory compactness loss are fused together to obtain the composite loss.

[0085] The formula for calculating the composite loss is as follows:

[0086] in, For compound loss, To reconstruct the loss, For memory sparsity constraint loss, For memory compactness loss, , and These are the reconstruction loss weights, memory sparsity constraint loss weights, and memory compactness loss weights, respectively. Optionally, these can be set based on empirical values ​​or actual needs. , and .For example, , , .

[0087] The formula for calculating the sparse memory constraint loss is as follows:

[0088] in, For memory sparsity constraint loss, This represents the maximum allowed weight for a single memory vector. Optionally, the maximum allowed weight can be set based on empirical values ​​or actual needs. For example, the maximum allowed weight could be 0.5.

[0089] To ensure the validity of the interval constraint for memory compactness loss, the memory vector is L2 normalized before computation, mapping it onto the unit hypersphere:

[0090] After normalization, the formula for calculating memory compactness loss is as follows:

[0091] in, For memory compactness loss, and Let N be two distinct memory vectors among N normalized memory vectors. This is the interval threshold, which is dimensionless. Optionally, the interval threshold can be set based on empirical values ​​or actual needs. For example, the interval threshold can be set to 0.5.

[0092] It should be understood that the maximum Euclidean distance between the two normalized memory vectors is 2 (in completely opposite directions). This means that the angle between any two memory vectors is at least approximately 29°, ensuring that the memory vectors are evenly distributed in the feature space and that different normal patterns are stored.

[0093] In this embodiment, the reconstruction loss enables the multi-scale memory-enhanced autoencoder network to learn and correctly reconstruct normal tooth samples, allowing the memory vectors to carry meaningful normal pattern information. The memory sparsity constraint loss ensures that each retrieval by the memory module uses at least two combinations of memory vectors, thereby preventing the memory matrix from degenerating into an identity mapping. The memory compactness loss promotes sufficient spacing between different memory vectors, avoiding redundancy.

[0094] Step 403: Train a multi-scale memory-enhanced autoencoder network based on composite loss.

[0095] In this embodiment, by designing and fusing three types of losses—reconstruction loss, memory sparsity constraint loss, and memory compactness loss—a multi-scale memory-enhanced autoencoder network capable of deeply understanding "what is normal" can be trained using only normal beak tooth samples, thereby achieving reliable detection of various abnormal beak teeth.

[0096] In some embodiments, the Adam optimizer can be used during training with an initial learning rate of Cosine annealing scheduling is used. The batch size is 32, and training lasts for 200 epochs. The memory modules are frozen for the first 50 epochs to train the basic reconstruction capabilities of the encoder-decoder, after which the memory modules are unfrozen and trained jointly. Here, the training memory module can refer to the memory matrix within the training memory module. An epoch refers to the complete process of traversing the entire training set.

[0097] After training the multi-scale memory-enhanced autoencoder network, the electronic device can input the test image of the bucket tooth into the multi-scale memory-enhanced autoencoder network to reconstruct the test image of the bucket tooth through the multi-scale memory-enhanced autoencoder network, and detect bucket tooth anomalies based on the reconstruction error between the test image and the reconstructed image.

[0098] In some embodiments of this application, the aforementioned reconstruction errors include: pixel-level reconstruction error, perceptual-level reconstruction error, and structural similarity error. The perceptual-level reconstruction error is calculated using intermediate layer features of the VGG16 network, which can extract more robust semantic-level differences. Based on this, as... Figure 5 As shown, based on the reconstruction error, it is determined whether there is any abnormality in the bucket tooth image to be detected, and the abnormal bucket tooth is located if there is an abnormality. This can include steps 501 to 504.

[0099] Step 501: Interpolate the perceptual-level reconstruction error to the original resolution to obtain the interpolated perceptual-level reconstruction error.

[0100] The original resolution is the resolution of the image of the bucket teeth to be detected.

[0101] In this embodiment, an upsampling function is used to interpolate the perceptual-level reconstruction error to the original resolution, thus bringing the perceptual-level reconstruction error to the same spatial size as the image of the tusk to be detected. After interpolation, the perceptual reconstruction error at each pixel location in the perceptual-level reconstruction error represents the local semantic or structural differences reflected by the receptive field of the intermediate layer features of the VGG16 network near that pixel location. After interpolation, the pixel locations in the perceptual-level reconstruction error, the pixel locations in the image of the tusk to be detected, and the pixel locations in the reconstructed image are aligned one-to-one.

[0102] Step 502: Normalize the pixel-level reconstruction error, the interpolated perceptual-level reconstruction error, and the structural similarity error respectively.

[0103] The formula for calculating pixel-level reconstruction error is as follows:

[0104] in, pixel position Pixel-level reconstruction error at the location, The image of the bucket teeth to be detected at the pixel position Pixel intensity value at that location To reconstruct the image at pixel locations The pixel intensity value at that location. The pixel intensity value can be a grayscale value or the intensity value of each color channel.

[0105] The formula for calculating the sensor-level reconstruction error is as follows:

[0106] in, For the perception-level reconstruction error, The image of the bucket teeth to be detected in the VGG16 network is shown in the first... Feature maps before each pooling layer To reconstruct the image in the VGG16 network Feature maps before each pooling layer Assign weights to each layer. Optionally, the weights for each layer can be set based on empirical values ​​or actual needs. For example, , , , The values ​​are 1.0, 0.5, 0.25, and 0.125, respectively.

[0107] The formula for calculating structural similarity error is as follows:

[0108] in, pixel position Structural similarity error at the location, , These are the mean pixel intensity values ​​within the local window of the image of the to-beam and the reconstructed image, respectively. , These represent the variances of pixel intensity values ​​within the local window for the image of the to-beam and the reconstructed image, respectively. Let be the covariance of the pixel intensity values ​​of the target image and the reconstructed image within a local window. and To maintain stability and prevent the denominator from being zero, the aforementioned local window refers to a window defined by pixel position. A local window centered on the user. Optionally, this can be set based on empirical values ​​or actual needs. and .For example, , .

[0109] Step 503: The pixel-level reconstruction error after normalization, the perceptual reconstruction error after interpolation, and the structural similarity error are fused to obtain the pixel-level anomaly score map of the to-beam image.

[0110] The three types of errors described above pertain to each pixel position in the image of the bucket teeth to be detected. Therefore, to ensure consistency in the dimensions of each error, the three errors can be normalized by taking the maximum value of the error at each pixel position. After normalization, a weighted fusion is then performed to obtain a pixel-level anomaly score map of the image of the bucket teeth to be detected. The pixel-level anomaly score map includes pixel-level anomaly scores at each pixel position. Any pixel position in the pixel-level anomaly score map... The formula for calculating the pixel-level anomaly score is as follows:

[0111] in, pixel position Pixel-level anomaly scoring at the location pixel position Interpolation-level reconstruction error at the sensor level. This represents the maximum pixel-level reconstruction error at each pixel location. This represents the maximum structural similarity error at each pixel location. This represents the maximum value of the perceptual-level reconstruction error at each pixel location after interpolation. , and These are the fusion weights for pixel-level reconstruction error, structural similarity error, and interpolated perceptual-level reconstruction error, respectively. The fusion weights accurately reflect the contribution ratio of each error. Optionally, they can be set based on empirical values ​​or actual needs. , and .For example , , .

[0112] Step 504: Based on the pixel-level anomaly scoring map, determine whether there are any abnormalities in the bucket teeth image to be detected, and locate the abnormal bucket teeth if abnormalities are found.

[0113] In this embodiment, pixel-level reconstruction error, interpolation-based perceptual-level reconstruction error, and structural similarity error measure image differences from different dimensions and have different robustness to interferences such as illumination and noise. By fusing them, they can complement each other's advantages, thereby obtaining a more stable score that is more focused on detecting beak tooth anomalies, thus solving the problem that traditional mean squared error (MSE) is sensitive to illumination and noise.

[0114] In some embodiments of this application, such as Figure 6 As shown, based on the pixel-level anomaly scoring map, it is determined whether there is an abnormality in the bucket tooth image to be detected, and the abnormal bucket tooth is located when there is an abnormality. This can include steps 601 to 603.

[0115] Step 601: Using the anomaly threshold, the pixel-level anomaly scoring map is binarized to obtain an anomaly mask.

[0116] In this embodiment, the pixel-level anomaly score at each pixel location in the pixel-level anomaly score map can be compared with the anomaly threshold. The pixel intensity value at the pixel location where the pixel-level anomaly score is greater than or equal to the anomaly threshold is marked as 1 (i.e. marked as anomaly, usually displayed as white or foreground), and the pixel intensity value at the pixel location where the pixel-level anomaly score is less than the anomaly threshold is marked as 0 (i.e. marked as normal, usually displayed as black or background), thereby determining the spatial distribution of all suspected abnormal pixels and forming a binary anomaly mask.

[0117] In some embodiments, after obtaining the anomaly mask, morphological opening operations can be performed on the anomaly mask to remove noise, and closing operations can be performed to fill holes. Then, the 8-connected region algorithm is used to determine the connected regions, thereby realizing the connected region analysis of the pixel-level anomaly scoring map and providing a data foundation for subsequent anomaly region localization.

[0118] Step 602: If there is a connected region in the abnormal mask with an area greater than or equal to the area of ​​the minimum broken tooth, then it is determined that there is an abnormal situation in the tooth image to be detected, and the connected region is identified as an abnormal region.

[0119] In this embodiment, the minimum broken tooth area is the minimum area of ​​structural defects in the abnormal region that reaches the actual bucket tooth area. It does not require or depend on stable measurement of the normal bucket tooth area, and can be more stable and less prone to false alarms in complex mining environments.

[0120] Optionally, the minimum broken tooth area can be set based on empirical values ​​or actual needs. For example, the minimum broken tooth area can be 2% of the area of ​​the bucket tooth to be detected. When the resolution of the bucket tooth area to be detected is 256×256 pixels, the minimum broken tooth area is approximately 1300 pixels.

[0121] Step 603: Based on the centroid abscissa of the abnormal region, the width of the image of the bucket teeth to be detected, and the total number of bucket teeth in the image of the bucket teeth to be detected, determine the bucket tooth partition where the abnormal region is located, and determine the bucket teeth corresponding to the bucket tooth partition where the abnormal region is located as abnormal bucket teeth.

[0122] It should be noted that the number of abnormal regions may be one or multiple, depending on the anomaly detection of the bucket tooth image to be detected.

[0123] In this embodiment, the tooth partitions can be pre-defined (e.g., the image of the tooth to be detected can be divided horizontally into equal parts). One tooth corresponds to one tooth partition. The centroid of the abnormal region is determined to fall within that partition, thereby achieving spatial localization of the abnormal tooth in the image. One tooth partition corresponds to one tooth, and one tooth corresponds to one tooth number. Determining the tooth partition where the abnormal region is located can refer to determining the tooth number of that partition. The formula for calculating the tooth number of the partition where the abnormal region is located is as follows:

[0124] in, For the first The bucket tooth number of the bucket tooth zone where the abnormal area is located. The x-coordinate of the centroid of the abnormal region (in pixels). The width of the image of the bucket teeth to be detected. This represents the total number of teeth. This is for rounding down.

[0125] In some embodiments, after determining the tooth number of the tooth zone where the abnormal area is located, "the tooth number" can be output. The message "Bug teeth suspected of being abnormal" helps maintenance personnel quickly locate and inspect abnormal bucket teeth.

[0126] Considering the drastic changes in lighting and the frequent presence of soil in the mining environment, fixed thresholds are prone to false alarms. This solution designs a dynamic thresholding strategy based on image information entropy. Specifically, before binarizing the pixel-level anomaly scoring map using the anomaly threshold, the following steps are also included: Calculate the information entropy of the image of the bucket teeth to be detected; Anomaly thresholds are determined based on information entropy, the mean and standard deviation of pixel-level anomaly scores for normal beak samples.

[0127] Information entropy can be used to measure the complexity of a tortoise shell image to be detected, and it is positively correlated with the anomaly threshold. The formula for calculating information entropy is as follows:

[0128] in, Information entropy (unit: bits / pixel, value range: [0, 8]). The grayscale value is The probability of a pixel appearing in an image satisfies , It should be understood that when the image of the bucket tooth to be detected is covered by soil or interfered with by light, the texture complexity increases and the information entropy increases; when the image of the bucket tooth to be detected is relatively simple and clear, the information entropy is low.

[0129] In some embodiments, based on normal beak samples collected during the training phase, the distribution of pixel-level anomaly scores for the normal beak samples can be statistically analyzed, and their mean and standard deviation can be calculated and stored. During the inference phase, the anomaly threshold can be calculated using the following formula:

[0130] in, This is the abnormal threshold. and These represent the mean and standard deviation of pixel-level anomaly scores for normal buck teeth samples. The basic sensitivity coefficient (dimensionless). This is the entropy weight adjustment factor (dimensionless). The entropy weight adjustment factor is used to control the influence weight of environmental complexity on the anomaly threshold.

[0131] Optionally, the base sensitivity coefficient and entropy weight adjustment factor can be set based on empirical values ​​or actual needs. For example, the base sensitivity coefficient can be set to a range of 2.5 to 3.5, with a recommended value of 3.0, and the entropy weight adjustment factor can be set to a range of 0.1 to 0.3, with a recommended value of 0.2.

[0132] In this embodiment, when the information entropy increases due to complex environment (e.g., lots of mud, poor lighting), the abnormal threshold is automatically raised to reduce false alarms; when the environment is good, the abnormal threshold is kept at a low level to ensure the sensitivity of bucket tooth anomaly detection.

[0133] In this embodiment, by using a dynamic threshold strategy based on information entropy, the system can automatically sense environmental complexity (such as the degree of soil obstruction) and adjust the sensitivity of the bucket tooth anomaly detection, maintaining stable detection performance under different weather and operating conditions.

[0134] In some embodiments of this application, before binarizing the pixel-level anomaly scoring map using an anomaly threshold, the method further includes: The 95th percentile value of the pixel-level anomaly scoring map is determined as the image-level comprehensive anomaly score of the image to be detected. Using anomaly thresholds, the pixel-level anomaly scoring map is binarized, including: If the image-level comprehensive anomaly score is greater than or equal to the preset score threshold, then the pixel-level anomaly score map is binarized using the anomaly threshold.

[0135] In this embodiment, the 95th percentile value of the pixel-level anomaly score map is determined as the image-level comprehensive anomaly score of the bucket tooth image to be detected. This can focus on the overall level of the part of the error with higher error (i.e., the abnormal area), taking into account both stability and sensitivity, thereby improving the accuracy of bucket tooth anomaly detection under working conditions with many interferences such as on-site lighting fluctuations and splashing mud and water.

[0136] In this embodiment, if the overall image anomaly score is greater than or equal to a preset scoring threshold, it can be determined that the image to be detected contains abnormal teeth. Then, the pixel-level anomaly score map is used to spatially locate the abnormal teeth. If the overall image anomaly score is less than the preset scoring threshold, it can be determined that the image to be detected does not contain abnormal teeth, and there is no need to perform binarization processing on the pixel-level anomaly score map. This reduces the system's average processing latency and power consumption, improves overall throughput, and is more suitable for deployment on edge devices with limited computing resources.

[0137] In some embodiments of this application, after obtaining the pixel-level anomaly scoring map of the image of the to-beam teeth, the method further includes: The pixel-level anomaly scoring map of the bucket tooth image to be detected is mapped to a Jet pseudo-color heat map; The Jet pseudo-color heatmap is overlaid on the image of the bucket teeth to be detected and then displayed.

[0138] In this embodiment, the Jet pseudo-color heatmap is overlaid on the image of the bucket tooth to be inspected, which can intuitively present the spatial distribution of pixel-level anomaly scores, helping users to quickly locate the bucket tooth and specific location of the anomaly. This facilitates on-site personnel to review and investigate, and also helps developers quickly identify the source of false alarms (such as structural defects or interference from reflections, mud spots, etc.) during parameter tuning or acceptance. At the same time, it can also provide direct visual alarm evidence for the monitoring terminal.

[0139] To address the issue that anomaly detection results in a single frame of the bucket tooth image are easily affected by transient interference (such as splashing mud or water droplets), the electronic device employs a sliding window mechanism to determine whether to trigger an anomaly alarm. The anomaly detection result indicates whether an abnormal bucket tooth has been located in the bucket tooth image. Specifically, if in the most recent consecutive... If the number of frames in the image of the bucket teeth to be detected that locate abnormal bucket teeth exceeds the trigger threshold, an abnormal alarm will be output. It is an integer greater than 1.

[0140] The calculation formula for the decision signal of the abnormal alarm is as follows:

[0141] in, This is a decision signal for abnormal alarms. As the trigger threshold, For the first Anomaly detection results for the frame of the bucket tooth image to be detected. For indicator functions, The frame number is the currently being processed image of the bucket teeth to be detected (i.e., the current frame of the image of the bucket teeth to be detected). This indicates that the anomaly detection result is characterized in the first... When an abnormal tooth is located in the frame of the image to be detected, its value is 1. This value is used to characterize the anomaly detection result in the [frame name missing]. When no abnormal bucket tooth is located in the frame of the bucket tooth image to be detected, its value is 0. The above decision signal can be a binary variable; a decision signal value of 1 outputs an abnormal alarm, and a decision signal value of 0 does not output an abnormal alarm. The summation range of the above decision signal calculation formula is from the nth preceding frame of the bucket tooth image to be detected. From the first frame of the image to be detected to the current frame of the image to be detected, the total Anomaly detection results for the frame.

[0142] In this embodiment, the electronic device maintains a length of A sliding window that stores the most recent contiguous... Anomaly detection results for the frame of the bucket tooth image to be detected. Only when the sliding window exceeds [a certain threshold]... The anomaly detection result of the frame to be detected indicates that an abnormal bucket tooth is located before an anomaly alarm is output, which effectively improves the anti-interference capability of the system.

[0143] As an example rather than a limitation, the sliding window length is set to 5 and the trigger threshold is set to 3. Therefore, the above-mentioned abnormal alarm mechanism means that an alarm will only be triggered if at least 3 out of 5 consecutive frames of the bucket tooth images to be detected show abnormalities.

[0144] In some embodiments, the above-mentioned abnormal output alarm includes, but is not limited to, triggering an audible and visual alarm and pushing it to the monitoring terminal.

[0145] In this embodiment, when abnormal bucket teeth are present in the image of bucket teeth to be detected, the Jet pseudo-color heatmap is combined with connected component analysis to accurately locate the specific position of the abnormal bucket teeth with high positioning accuracy.

[0146] It should be noted that, in offline evaluation tests, the accuracy of the bucket tooth anomaly detection method provided in this application embodiment is significantly better than that of traditional supervised methods that require labeled data, and no abnormal samples are required to participate in model training.

[0147] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0148] Corresponding to the bucket tooth anomaly detection method described in the above embodiments, Figure 7 A schematic diagram of the structure of the bucket tooth abnormality detection device provided in the embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0149] Reference Figure 7 The bucket tooth anomaly detection device includes: The network training module 701 is used to train a multi-scale memory-enhanced autoencoder network based on normal beak tooth samples, so that the multi-scale memory-enhanced autoencoder network learns the multi-scale feature representation and reconstruction capability of normal beak teeth. The image reconstruction module 702 is used to reconstruct the input image of the target beak through the trained multi-scale memory-enhanced autoencoder network to obtain a reconstructed image; Error calculation module 703 is used to calculate the reconstruction error between the image of the bucket tooth to be detected and the reconstructed image; Anomaly detection module 704 is used to determine whether there is anomaly in the bucket tooth image to be detected based on the reconstruction error, and to locate the abnormal bucket tooth if the abnormality exists.

[0150] In some embodiments, the multi-scale memory-enhanced autoencoder network includes: an encoder, a memory module, and a decoder employing a multi-scale feature pyramid structure; the image reconstruction module 702 includes: The feature extraction unit is used to perform multi-scale feature extraction on the image of the tusk to be detected by the encoder to obtain the coding feature maps of each layer of the encoder; A vector retrieval unit is configured to retrieve, via the memory module, a memory-enhanced feature vector related to a one-dimensional feature vector of the last layer encoded feature map. The one-dimensional feature vector is obtained by performing a global average pooling operation on the last layer encoded feature map. The memory-enhanced feature vector is obtained by combining at least two memory vectors from a memory matrix, where the memory matrix includes the normal beak teeth. There are 3 memory vectors, each with the same dimension as the one-dimensional feature vector. It is an integer greater than 1; The spatial reshaping unit is used to perform a spatial reshaping operation on the memory-enhanced feature vector through the decoder to restore it as a spatial feature map. When the remaining feature map of the encoder is passed to the corresponding layer of the decoder through skip connections, the spatial feature map and the remaining feature map are processed by the decoder to obtain the reconstructed image, wherein the remaining feature map is the feature map of each layer except the last layer of coding feature map.

[0151] In some embodiments, the vector retrieval unit is specifically used for: Through the memory module, using the first formula, the one-dimensional feature vector and the memory matrix are processed to obtain the... Attention weights for each memory vector; Calculate the memory module The attention weights of each memory vector are multiplied by the corresponding memory vector, and the products are summed to obtain the memory-enhanced feature vector. The first formula is:

[0152] in, For the first Attention weights for each memory vector. For the first A memory vector, For the first A memory vector, Indicates the temperature coefficient. It is the transpose of the one-dimensional eigenvector.

[0153] In some embodiments, the network training module 701 is specifically used for: Based on normal tooth samples, reconstruction loss, memory sparsity constraint loss, and memory compactness loss are determined. The memory sparsity constraint loss is used to constrain the attention weight of a single memory vector to not exceed the maximum allowed weight, where the maximum allowed weight is a positive number less than 1. The memory compactness loss is used to promote the clustering of the memory vectors. The loss fusion module is used to fuse the reconstruction loss, the memory sparsity constraint loss, and the memory compactness loss to obtain a composite loss; A network training module is used to train the multi-scale memory-reinforced autoencoder network based on the composite loss.

[0154] In some embodiments, the reconstruction error includes: pixel-level reconstruction error, perceptual-level reconstruction error, and structural similarity error, wherein the perceptual-level reconstruction error is calculated using intermediate layer features of the VGG16 network; the anomaly detection module 704 includes: An error interpolation unit is used to interpolate the perception-level reconstruction error to the original resolution to obtain the interpolated perception-level reconstruction error, wherein the original resolution is the resolution of the image of the tusk to be detected. The normalization unit is used to normalize the pixel-level reconstruction error, the interpolated perceptual-level reconstruction error, and the structural similarity error, respectively. An error fusion unit is used to fuse the normalized pixel-level reconstruction error, the interpolated perceptual-level reconstruction error, and the structural similarity error to obtain a pixel-level anomaly score map of the target beak image. An anomaly detection unit is used to perform anomaly detection on the target bucket tooth image based on the pixel-level anomaly scoring map.

[0155] In some embodiments, the anomaly detection unit is specifically used for: The pixel-level anomaly scoring map is binarized using an anomaly threshold to obtain an anomaly mask. If there is a connected region in the abnormal mask with an area greater than or equal to the area of ​​the minimum broken tooth, then the image of the tooth to be detected is determined to have the abnormal tooth condition, and the connected region is identified as the abnormal region. Based on the centroid abscissa of the abnormal region, the width of the image of the bucket tooth to be detected, and the total number of bucket teeth in the image of the bucket tooth to be detected, the bucket tooth partition where the abnormal region is located is determined, and the bucket tooth corresponding to the bucket tooth partition where the abnormal region is located is determined as the abnormal bucket tooth.

[0156] In some embodiments, the bucket tooth anomaly detection device further includes: The information entropy calculation module is used to calculate the information entropy of the target bucket tooth image before binarizing the pixel-level anomaly scoring map using the anomaly threshold. The threshold determination module is used to determine the anomaly threshold based on the information entropy, the mean and standard deviation of the pixel-level anomaly scores of the normal beak tooth samples.

[0157] In some embodiments, the anomaly detection unit is specifically used for: The 95th percentile value of the pixel-level anomaly scoring map is determined as the image-level comprehensive anomaly score of the bucktooth image to be detected. If the image-level comprehensive anomaly score is greater than or equal to a preset score threshold, then the pixel-level anomaly score map is binarized using the anomaly threshold.

[0158] In some embodiments, the anomaly detection module 703 further includes: The image mapping unit is used to map the pixel-level anomaly score map of the image to be detected into a Jet pseudo-color heat map after obtaining the pixel-level anomaly score map of the image to be detected. An image processing unit is used to overlay the Jet pseudo-color heat map onto the image of the bucket teeth to be detected and then display it.

[0159] In some embodiments, the bucket tooth anomaly detection device further includes: An abnormal alarm module is used to detect anomalies in the most recent consecutive If the number of frames in the image of the bucket teeth to be detected that locate abnormal bucket teeth is greater than the trigger threshold, an abnormal alarm will be output. It is an integer greater than 1.

[0160] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0161] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 (Only one is shown in the diagram), memory 81, and computer program 82 stored in said memory 81 and executable on said at least one processor 80, wherein said processor 80 executes said computer program 82 to implement the steps in any of the above method embodiments.

[0162] The electronic device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0163] The processor 80 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0164] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may be an external storage device of the electronic device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 8. Furthermore, the memory 81 may include both internal and external storage units of the electronic device 8. The memory 81 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0167] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0170] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of detecting an abnormality of a bucket tooth, characterized by, include: The multi-scale memory-enhanced autoencoder network is trained based on normal bucket teeth samples, enabling the multi-scale memory-enhanced autoencoder network to learn the multi-scale feature representation and reconstruction capabilities of normal bucket teeth. The input image of the target beak is reconstructed by the trained multi-scale memory-enhanced autoencoder network to obtain the reconstructed image. Calculate the reconstruction error between the image of the bucket teeth to be detected and the reconstructed image; Based on the reconstruction error, it is determined whether there is any abnormality in the bucket teeth image to be detected, and the abnormal bucket teeth are located if such abnormality exists.

2. The bucket tooth abnormality detection method according to claim 1, characterized by, The multi-scale memory-enhanced autoencoder network includes: an encoder, a memory module, and a decoder employing a multi-scale feature pyramid structure; the reconstructing of the input target buckle image using the trained multi-scale memory-enhanced autoencoder network to obtain the reconstructed image includes: The encoder is used to extract multi-scale features from the image of the tusk to be detected, thereby obtaining the coding feature maps of each layer of the encoder; The memory module retrieves a memory-enhanced feature vector related to the one-dimensional feature vector of the last layer of encoded feature map. This one-dimensional feature vector is obtained by performing a global average pooling operation on the last layer of encoded feature map. The memory-enhanced feature vector is obtained by combining at least two memory vectors from the memory matrix, which includes the normal beak teeth. There are 3 memory vectors, each with the same dimension as the one-dimensional feature vector. It is an integer greater than 1; The decoder performs a spatial reshaping operation on the memory-enhanced feature vector to restore it as a spatial feature map. When the remaining feature map of the encoder is passed to the corresponding layer of the decoder through skip connections, the spatial feature map and the remaining feature map are processed by the decoder to obtain the reconstructed image, wherein the remaining feature map is the feature map of each layer except the last layer of coding feature map.

3. The bucket tooth abnormality detection method according to claim 2, characterized by, The step of retrieving the memory-enhanced feature vector related to the one-dimensional feature vector of the last layer of encoded feature map through the memory module includes: By the memory module, the one-dimensional feature vector and the memory matrix are processed by using a first formula to obtain the attention weight of the memory vector. ​ Calculate the memory module The attention weights of each memory vector are multiplied by the corresponding memory vector, and the products are summed to obtain the memory-enhanced feature vector. The first formula is: wherein, is the attention weight for the th memory vector, is the th memory vector, is the attention weight for the th memory vector, denotes the temperature coefficient, is the transpose of the one-dimensional feature vector.

4. The bucket tooth abnormality detection method according to claim 3, characterized by, The training of the multi-scale memory-enhanced autoencoder network based on normal tooth samples includes: Based on the normal tooth samples, reconstruction loss, memory sparsity constraint loss, and memory compactness loss are determined. The memory sparsity constraint loss is used to constrain the attention weight of a single memory vector to not exceed the maximum allowed weight, where the maximum allowed weight is a positive number less than 1. The memory compactness loss is used to promote the clustering of the memory vectors. The reconstruction loss, the memory sparsity constraint loss, and the memory compactness loss are fused together to obtain a composite loss. The multi-scale memory-reinforced autoencoder network is trained based on the composite loss.

5. The bucket tooth abnormality detection method according to any one of claims 1 to 4, characterized by, The reconstruction error includes: pixel-level reconstruction error, perceptual-level reconstruction error, and structural similarity error. The perceptual-level reconstruction error is calculated using intermediate layer features of the VGG16 network. The step of determining whether the target beak image has beak anomalies based on the reconstruction error, and locating the abnormal beaks if such anomalies exist, includes: The perceptual-level reconstruction error is interpolated to the original resolution to obtain the interpolated perceptual-level reconstruction error, wherein the original resolution is the resolution of the image of the tusk to be detected. The pixel-level reconstruction error, the interpolated perceptual-level reconstruction error, and the structural similarity error are normalized respectively. The pixel-level reconstruction error after normalization, the perceptual reconstruction error after interpolation, and the structural similarity error are fused to obtain a pixel-level anomaly score map of the detected tusk image. Based on the pixel-level anomaly scoring map, it is determined whether the image of the bucket teeth to be detected has any bucket tooth anomalies, and if the bucket tooth anomalies are present, the abnormal bucket teeth are located.

6. The bucket tooth abnormality detection method according to claim 5, characterized by The step of determining whether the to-beam tooth image has any abnormalities based on the pixel-level anomaly scoring map, and locating the abnormal teeth if such anomalies exist, includes: The pixel-level anomaly scoring map is binarized using an anomaly threshold to obtain an anomaly mask. If there is a connected region in the abnormal mask with an area greater than or equal to the area of ​​the minimum broken tooth, then the image of the tooth to be detected is determined to have the abnormal tooth condition, and the connected region is identified as the abnormal region. Based on the centroid abscissa of the abnormal region, the width of the image of the bucket tooth to be detected, and the total number of bucket teeth in the image of the bucket tooth to be detected, the bucket tooth partition where the abnormal region is located is determined, and the bucket tooth corresponding to the bucket tooth partition where the abnormal region is located is determined as the abnormal bucket tooth.

7. The bucket tooth anomaly detection method according to claim 6, characterized by, Before performing binarization processing on the pixel-level anomaly scoring map using the anomaly threshold, the method further includes: Calculate the information entropy of the image of the bucket teeth to be detected; The anomaly threshold is determined based on the information entropy, the mean and standard deviation of the pixel-level anomaly scores of the normal beak tooth samples.

8. The method for detecting abnormal bucket teeth according to claim 6, characterized in that, Before performing binarization processing on the pixel-level anomaly scoring map using the anomaly threshold, the method further includes: The 95th percentile value of the pixel-level anomaly scoring map is determined as the image-level comprehensive anomaly score of the bucktooth image to be detected. The step of binarizing the pixel-level anomaly scoring map using an anomaly threshold includes: If the image-level comprehensive anomaly score is greater than or equal to a preset score threshold, then the pixel-level anomaly score map is binarized using the anomaly threshold.

9. The method for detecting abnormal bucket teeth according to claim 5, characterized in that, After obtaining the pixel-level anomaly scoring map of the image of the to-beam teeth, the method further includes: The pixel-level anomaly scoring map is mapped to a Jet pseudo-color heatmap; The Jet pseudo-color heatmap is then overlaid on the image of the bucket teeth to be detected and displayed.

10. The method for detecting abnormal bucket teeth according to any one of claims 1 to 4, characterized in that, After locating the abnormal bucket tooth, the process also includes: If in the most recent consecutive If the number of frames in the image of the bucket teeth to be detected that locate the abnormal bucket teeth is greater than the trigger threshold, an abnormal alarm will be output. It is an integer greater than 1.

11. A device for detecting abnormal bucket teeth, characterized in that, include: The network training module is used to train the multi-scale memory-enhanced autoencoder network based on normal beak tooth samples, so that the multi-scale memory-enhanced autoencoder network learns the multi-scale feature representation and reconstruction capability of normal beak teeth. The image reconstruction module is used to reconstruct the input image of the target tusk using the trained multi-scale memory-enhanced autoencoder network to obtain a reconstructed image. An error calculation module is used to calculate the reconstruction error between the image of the bucket tooth to be detected and the reconstructed image; An anomaly detection module is used to determine whether there are any abnormalities in the bucket teeth image to be detected based on the reconstruction error, and to locate the abnormal bucket teeth if such abnormalities are found.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the bucket tooth abnormality detection method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, The method includes a computer program, which, when run, causes the method for detecting abnormal teeth as described in any one of claims 1 to 10 to be performed.

Images