Casting surface defect segmentation method and quality inspection system based on multi-modal attention

Abstract

The invention relates to the technical field of image quality inspection, in particular to a casting surface defect segmentation method and quality inspection system based on multi-modal attention, and the method comprises the steps: collecting a color image and a depth image for a to-be-detected surface, and then sending the color image and the depth image into a casting defect recognition model; the casting recognition model is of a codec structure, color image features and depth image features are extracted respectively, fused and then subjected to up-sampling, and then a segmentation result corresponding to casting surface defects is obtained; for a depth image, a plurality of gradient operators with different scales are adopted to perform enhancement processing in an image feature extraction process. Aiming at the defects of different scales such as cracks, pores and large-area recesses on the surface of the casting, a plurality of gradient operators with different scales are designed to enhance the part of the depth image, so that the defects with different sizes can be effectively sensed in the extraction process of the depth image; and the method has better robustness for noise, illumination variation and tiny irregularity of the surface of the casting.

Description

Technical Field

[0001] This invention relates to the field of image quality inspection technology, specifically to a method and system for segmenting surface defects in castings based on multimodal attention. Background Technology

[0002] Surface defects in castings typically refer to issues such as porosity, cracks, spalling, and sand inclusions that occur on the surface of metal castings during the casting, demolding, and cooling processes due to factors related to materials, processes, and equipment. These defects can lead to stress concentration areas on the casting surface, resulting in a significant decrease in mechanical properties. To address these issues, current processes usually require visual inspection after the casting has cooled to locate surface defects, which are then eliminated through grinding, welding, or other methods.

[0003] In the existing technology, there are already technical solutions based on image recognition technology to assist in finding surface defects in castings.

[0004] For example, patent document CN202110317123.8 discloses a method for identifying surface defects in castings based on an improved DeepLabv3+ network model, comprising the following steps: Step S1, collecting a dataset of casting images to obtain a training set and a test set; Step S2, constructing a network model and training and correcting the network model using the training and test sets to generate a defect detection network; Step S3, designing the loss function of the defect detection network; Step S4, the defect detection network identifies and outputs the casting defect detection results and displays the detection time. Using deep learning to identify surface defects in castings improves the accuracy and speed of defect identification, providing a new approach for industrial casting quality inspection.

[0005] For example, patent document CN202211003001.2 discloses a method for identifying defects in metal castings based on industrial inspection. The method includes: obtaining left-skewed and right-skewed thresholds based on the gray values ​​corresponding to the main peak and trough of the gray-level histogram of the surface image of the casting to be inspected; dividing the pixels in the surface image to obtain abrupt change regions; calculating the structural complexity of each abrupt change region based on its gray-level run-length matrix; designating abrupt change regions with structural complexity greater than the complexity threshold as target regions; calculating the reflective interference value of each target region based on its gray-level run-length matrix; obtaining the upper bound threshold of the defect corresponding to each target region based on the reflective interference value and the standard lower bound threshold of the defect; and obtaining the defect region of the surface image of the casting to be inspected based on the upper bound threshold and the standard lower bound threshold of the defect corresponding to each target region.

[0006] However, in actual implementation, the inventors found that the above technical solutions are usually designed for identifying fixed defect areas, and a single modality cannot simultaneously capture color abrupt changes and geometric deformation features; the high rate of missed detection for texture-similar defects and the limited accuracy of identification are also problems. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, a method for segmenting casting surface defects based on multimodal attention is provided.

[0008] On the other hand, a quality inspection system based on the casting surface defect segmentation method is also provided.

[0009] The specific technical solution is as follows:

[0010] A method for segmenting surface defects in castings based on multimodal attention, comprising:

[0011] Color and depth images are acquired for the surface to be inspected, and then fed into the casting defect recognition model for identification.

[0012] The casting recognition model is an encoder-decoder structure. The casting recognition model extracts the color image features of the color image and the depth image features of the depth image respectively and fuses them to obtain a fused image. Then, the fused image is upsampled to obtain the segmentation result corresponding to the surface defects of the casting.

[0013] For the depth image, the casting recognition model uses multiple gradient operators of different scales for enhancement processing during the image feature extraction process.

[0014] On the other hand, the casting recognition model includes an independent depth image coding branch;

[0015] The depth image coding branch extracts the depth image features sequentially through multiple second convolutional layers;

[0016] The gradient operator processes the depth image to obtain an enhanced depth image, which is then fed into the second convolutional layer of the corresponding level for fusion.

[0017] On the other hand, the depth image coding branch has four levels of the second convolutional layer, and the receptive field of the second convolutional layer gradually expands;

[0018] The gradient operator includes three gradient operators with progressively increasing scales;

[0019] The gradient operator enhances the depth image to obtain the depth feature image, and then sequentially inputs it into the second convolutional layer of the first three layers.

[0020] On the other hand, the gradient operator corresponds to the small-scale operator of the second convolutional layer in the first layer, the medium-scale operator of the second convolutional layer in the second layer, and the large-scale operator of the second convolutional layer in the third layer;

[0021] The small-scale operator is represented as:

[0022] The mesoscale operator is represented as:

[0023] The large-scale operator is represented as:

[0024] On the other hand, the casting recognition model includes an independent color image coding branch;

[0025] The color image coding branch extracts the color image features from the color image through multiple first convolutional layers in sequence, with the number of first convolutional layers and second convolutional layers being the same.

[0026] On the other hand, the casting recognition model includes an independent self-attention fusion module, which fuses the color image features and the depth image features to obtain the fused image;

[0027] The self-attention fusion module includes:

[0028] A first semantic extraction module receives the color image features and performs semantic extraction to obtain color semantic features;

[0029] The first residual module receives the color image features and performs residual calculations on the color image features to obtain color residual features;

[0030] The first merging module performs pixel-by-pixel multiplication of the color semantic features and the color residual features, and then adds them to the color image features to obtain color self-attention features;

[0031] The second semantic extraction module receives the depth image features and performs semantic extraction to obtain depth semantic features.

[0032] The second residual module receives the depth image features and performs residual calculations on the depth image features to obtain depth residual features;

[0033] The second merging module performs pixel-by-pixel multiplication of the deep semantic features and the deep residual features, and then adds them to the deep image features to obtain the deep self-attention features;

[0034] An attention fusion module, which is connected to the first merging module and the second merging module respectively;

[0035] The attention fusion module adds the color self-attention features and the depth self-attention features pixel by pixel to obtain the fused image.

[0036] On the other hand, the casting recognition model also includes an independent upsampling module, which performs upsampling processing on the fused image through multiple linear interpolation upsampling layers and then outputs the segmentation result.

[0037] On the other hand, the casting recognition model also includes multiple cross-attention modules;

[0038] The input of the cross-attention module is connected to the first convolutional layer and the second convolutional layer at the corresponding level, respectively;

[0039] The cross-attention module performs weighted fusion on the first convolutional image output by the first convolutional layer and the second convolutional image output by the second convolutional layer to form a weighted fusion feature, thereby enhancing shallow spatial location perception.

[0040] The output of the cross-attention module is connected to the linear interpolation upsampling layer at the corresponding level via a skip structure;

[0041] The cross-attention module inputs the weighted fusion features into the linear interpolation upsampling layer so that the linear interpolation upsampling layer obtains the original input for cascading during the upsampling process.

[0042] On the other hand, the cross-attention module includes:

[0043] A first input layer, wherein the first input layer receives the first convolutional image;

[0044] The second input layer receives the second convolutional image;

[0045] A first convolutional network, wherein the input of the first convolutional network is connected to the first input layer;

[0046] The first convolutional network extracts the first convolutional image to form a first feature map and outputs it.

[0047] The first cross-computation module has its input terminals connected to the first convolutional network and the second input layer, respectively.

[0048] The first cross-computation module performs element-wise multiplication of the second convolutional image and the first feature map to obtain the first cross-computation intermediate image;

[0049] The first cross-calculation module then performs element-wise addition between the first cross-calculation intermediate image and the second convolutional image to obtain a depth cross-calculation image;

[0050] The second convolutional network, the input of which is connected to the second input layer;

[0051] The second convolutional network extracts the second convolutional image to form a second feature map and outputs it.

[0052] The second cross-computation module has its input terminals connected to the second convolutional network and the first input layer, respectively.

[0053] The second cross-computation module performs element-wise multiplication of the first convolutional image and the second feature map to obtain the second cross-computation intermediate image;

[0054] The second cross-calculation module then performs element-wise addition between the second cross-calculation intermediate image and the first convolutional image to obtain a color cross-calculation image;

[0055] A cross-merging module, which is connected to the first cross-computing module and the second cross-computing module respectively;

[0056] The cross-merging module first stitches together the deep cross-computation image and the color cross-computation image, and then uses a convolution kernel to extract the weighted fusion feature.

[0057] On the other hand, there are three cross-attention modules, which are respectively connected to the first convolutional layer and the second convolutional layer of the first three layers.

[0058] On the other hand, it also includes a discriminator, the input of which is connected to the output of the casting recognition model, so as to form an adversarial generative network with the casting recognition model for training.

[0059] A quality inspection system is implemented based on the above-mentioned method for segmenting surface defects in castings.

[0060] The above technical solution has the following advantages or beneficial effects:

[0061] To address the problem that existing single-modal recognition methods cannot effectively capture complex defects on the surface of castings, this solution introduces a multimodal recognition method based on the fusion of color and depth images. By establishing a spatial and semantic association mechanism between color and depth images, it achieves effective recognition of complex defects.

[0062] Furthermore, for defects of varying scales such as uncertain cracks, pores, and large-area depressions on the surface of castings, this solution enhances the depth image by designing multiple gradient operators of different scales. This enables the depth image to effectively perceive defects of different sizes during the extraction process, and provides better robustness to noise, changes in illumination, and minor irregularities on the surface of the casting. Attached Figure Description

[0063] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0064] Figure 1 This is an overall schematic diagram of an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the cross-attention module in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the self-attention fusion module in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the discriminator in an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0070] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0071] This invention includes:

[0072] A method for segmenting surface defects in castings based on multimodal attention, such as Figure 1 As shown, it includes:

[0073] Color and depth images are acquired for the surface to be inspected, and then fed into the casting defect recognition model for identification.

[0074] The casting recognition model is an encoder-decoder structure. The encoder branch of the casting recognition model includes color image coding branch 1 and depth image coding branch 2.

[0075] The casting recognition model uses multiple gradient operators of different scales to enhance the depth image and then inputs them into the corresponding layers of the depth image coding branch 2 for fusion processing.

[0076] Specifically, in view of the problem that existing single-modal recognition methods cannot effectively capture complex defects on the surface of castings, this solution introduces a multimodal recognition method based on the fusion of color images and depth images. By establishing a spatial and semantic association mechanism between color images and depth images, it achieves effective recognition of complex defects.

[0077] Specifically, when defects appear on the surface of a casting, they may manifest as uncertain defects, such as small-scale, clearly defined pores, pinholes, and spalling, or larger-scale, poorly defined weld scars, ridges, large-area depressions, and gentle slopes. They may also include cracks of varying sizes. Current technologies that rely solely on single-modal recognition of color images may not effectively capture all defects, especially those with relatively blurred boundaries, leading to a high rate of missed detections.

[0078] To address the aforementioned issues, this embodiment proposes a multimodal casting defect identification model, which is primarily based on an encoder-decoder structure. This model includes a downsampling encoding path and an upsampling decoding path. The encoding path is further divided into two branches: a color image encoding branch and a depth image encoding branch.

[0079] During the process of acquiring raw data, a sensor capable of simultaneously acquiring color images (RGB color images) and depth images is required, such as a binocular vision sensor or other equivalent device. Registration is then performed using the corresponding intrinsic parameter matrix so that the input model image has paired color and depth images.

[0080] Color and depth images are fed into color image coding branches and depth image coding branches respectively. Feature extraction is performed through a series of convolution kernels to obtain color image features and depth image features respectively. Then, feature fusion is performed to establish the spatial-semantic relationship before being sent into the upsampling decoding path for segmentation and refinement of boundary regions, thereby achieving accurate labeling of various defects.

[0081] However, considering that in practical applications, it is impossible to control the types of defects that may appear in the casting itself, especially the depth of the defects, some defects, such as pinholes, have insignificant depth variations (micro-depressions ≤0.5mm), or insignificant depth gradient variations, such as large-area sloping depressions. This means that during the depth image extraction process, the model cannot fully perceive the characteristics of depth changes.

[0082] To address this issue, this embodiment incorporates the characteristics of the original model structure, namely, that the depth image is extracted through a series of convolutional kernels in the encoding branch, and the receptive field gradually expands during the extraction process, and designs an additional image feature enhancement method.

[0083] Specifically, for multiple convolutional layers in the depth image coding branch, the input process typically involves further extraction of image features extracted from the previous layer. However, this scheme employs a gradient operator of the appropriate scale to calculate the gradient of the original gradient image for each convolutional layer at the corresponding level. This highlights the depth features at the corresponding scale, which are then fed into the convolutional layer within the corresponding receptive field and added to the image features extracted from the previous layer. This provides additional depth features to the convolutional layer, enhancing the sensitivity of feature representation to depth changes at different scales. This allows different layers to capture more refined depth feature changes, thereby improving the representational ability of depth information.

[0084] In one embodiment, the casting recognition model includes a color image coding branch 1, a depth image coding branch 2, a self-attention fusion module 3, and an upsampling module 4;

[0085] The color image coding branch 1 extracts color image features from the color image through multiple first convolutional layers 11 in sequence;

[0086] The depth image coding branch 2 extracts depth image features by sequentially processing multiple second convolutional layers 21.

[0087] Furthermore, the gradient operator processes the depth image to obtain an enhanced depth image, which is then fed into the second convolutional layer 12 of the corresponding layer for fusion.

[0088] Self-attention fusion module 3 is connected to color image coding branch 1 and depth image coding branch 2 respectively;

[0089] The self-attention fusion module 3 performs self-attention fusion on color image features and depth image features to enhance the deep semantic association between color image features and depth image features, and outputs the fused image.

[0090] The upsampling module 4 upsamples the fused image through multiple linear interpolation upsampling layers 41 and then outputs the result.

[0091] Specifically, in order to address the problem that existing single-modal recognition methods cannot effectively capture complex defects on the surface of castings, this solution introduces a multimodal recognition method based on the fusion of color and depth images, and constructs a casting recognition model with the aforementioned structure.

[0092] The casting recognition model mainly consists of an encoding and decoding structure, including a color image encoding branch 1 and a depth image encoding branch 2.

[0093] Among them, the color image coding branch 1 extracts color images sequentially through multiple first convolutional layers 11, capturing high-dimensional features in the images, thereby obtaining color image features.

[0094] The color image features in this section mainly correspond to the texture and color features of the casting surface.

[0095] The depth image coding branch 2 extracts depth image features by sequentially processing the enhanced depth image fed in by multiple second convolutional layers 21 and gradient operators.

[0096] This part of the depth image features mainly corresponds to the geometric deformation features of the casting surface.

[0097] Subsequently, the image features of the two branches are fused by self-attention fusion module 3, thereby strengthening the deep semantic association between color and depth, making it easier to distinguish scars / oxidized skin with similar colors.

[0098] Finally, the fused image is upsampled by upsampling module 4 to restore image features. Upsampling module 4 mainly includes multiple linear interpolation upsampling layers 41 of corresponding layer number, usually four. In the linear interpolation upsampling layers, features are reconstructed through CARAFE dynamic kernel feature convolution to restore detail boundaries.

[0099] The above model structure achieves a better segmentation effect for various defects.

[0100] In one embodiment, the color image coding branch 1 and the depth image coding branch 2 each have four levels of first convolutional layer 11 and second convolutional layer 21, and the receptive fields of the first convolutional layer 11 and the second convolutional layer 21 gradually expand.

[0101] The gradient operator includes three gradient operators with progressively increasing scales;

[0102] The gradient operator enhances the depth image to obtain the depth feature image, and then inputs it into the second convolutional layer 21 of the first three layers in sequence.

[0103] Gradient operators include:

[0104] Small-scale operator 2A1:

[0105] Medium-scale operator 2A2:

[0106] Large-scale operator 2A3:

[0107] Specifically, after determining the above model framework, in order to achieve better image segmentation results, this embodiment determines that the number of encoding and decoding layers in the model framework is four to achieve better feature capture. In the color image encoding branch 1 and the depth image encoding branch 2, following the structure of a traditional convolutional network, the receptive fields of the first convolutional layer 11 and the second convolutional layer 21 are gradually expanded for extraction.

[0108] Existing surface defects in castings typically present the following problems:

[0109] 1. Poor edge sharpness: Depth variations between casting defects (such as cracks and porosity) and normal areas often occur within relatively small local areas (pixel-level or a few pixels-level), especially at edges. Traditional single-scale gradient operators or deep networks themselves may not be able to adequately capture these subtle, localized depth abrupt changes.

[0110] 2. Internal undulations also exist: The interior of casting defects (such as shrinkage cavities and slag inclusions) may contain slower, more extensive depth variations (undulations). These variations span larger areas and may be weaker. Single small-scale gradient operators are insensitive to this, while large-scale operators may smooth out important local details.

[0111] Therefore, to address the above issues, considering that convolutional networks require high-resolution details (small / medium scale) in shallow layers and more semantic and global information (large scale) in deeper layers, we chose to insert the corresponding scale gradient operator to calculate the deep gradient features on the second convolutional layer 21 of the first three layers to supplement additional gradient information, in conjunction with the network layers of the original model structure.

[0112] Specifically, the three operators mentioned above have the following characteristics:

[0113] 1. Small-scale operator (3x3): Possesses the smallest receptive field, highly sensitive to extremely small depth changes within the pixel neighborhood. It essentially computes the most local depth derivative (approximately the first or second derivative). Specifically designed to enhance and highlight dramatic but spatially small depth jumps between defects and background, and at the edges of fine cracks within defects. It accurately delineates the contours of these sharp edges, providing a high-resolution edge response. Applied in the first layer of feature encoding (near the input layer), it directly processes the most raw depth information, maximizing the preservation of these fine edge structures.

[0114] 2. Medium-scale operator (5x5): With a medium receptive field, it smooths and integrates depth gradient changes within a local area (a few pixels). It is robust to noise and can capture local variation patterns over a larger range than the small operator. Complementing the small operator, it connects edge segments that may be disturbed by noise or slightly broken by the small operator, providing more coherent edge information. While its edge localization accuracy is slightly lower than the small operator, its noise resistance is better. It begins to respond to medium-range, moderate-intensity depth undulations (such as shallow pits or small-scale material gaps) within defects. These undulations may not appear significant on a single pixel, but the cumulative changes over several pixels can be effectively detected by the medium-scale operator. In the second layer application, it processes features that have undergone preliminary abstraction but still possess finer-grained information.

[0115] 3. Large-scale operator (7x7): Possesses the largest receptive field, sensitive to slowly changing depth gradients or average depth differences over a larger area. It calculates depth change trends within a more macroscopic region. Its core function is to capture slowly changing, large-scale depth undulations (such as large depressions or gentle slopes) existing within defects (or within large-scale defects themselves). Small-scale operators are almost unresponsive to these global, weak gradient changes, and mesoscale operators also have weak responses. Large-scale operators, through integral effects, can significantly enhance the signals of these large-scale, weak undulations, revealing the overall morphology and inhomogeneity within defects. In third-layer applications, it processes features with stronger semantic information but lower spatial resolution, focusing on more macroscopic structures.

[0116] The gradient operators at the three scales mentioned above are used to calculate the depth gradient of the original depth image, which can effectively enhance the depth feature image of the defect at different scales. The corresponding depth feature image is then superimposed on the input of the second convolutional layer 21 at the corresponding level, which makes it easier for the model to capture the corresponding features.

[0117] Among them, the second convolutional layer 21 with higher resolution in the first two stages provides accurate edge localization at a small scale, connects edge segments at a medium scale and provides a certain degree of noise resistance. Together, they make the finally detected defect boundaries clearer and more complete.

[0118] For the second and third level second convolutional layers 21, medium-scale undulations are captured at the mesoscale and macro-scale undulations are captured at the macroscale. The combination of the two enables the network to effectively perceive the inhomogeneity and material changes (manifested as depth undulations) inside the defects. This is crucial for distinguishing different types of defects (such as pores vs. shrinkage) and understanding the severity of defects.

[0119] Finally, the depth feature map formed by multi-scale response fusion contains rich information from microscopic details to macroscopic structures, and has better robustness to noise, illumination changes (which have a relatively small but still exist in the depth map) and minor irregularities on the casting surface.

[0120] In one embodiment, the casting recognition model further includes multiple cross-attention modules 5;

[0121] The input of the cross-attention module 5 is connected to the first convolutional layer 11 and the second convolutional layer 21 at the corresponding level, respectively;

[0122] The cross-attention module 5 performs weighted fusion on the first convolutional image output by the first convolutional layer 11 and the second convolutional image output by the second convolutional layer 21 to form a weighted fusion feature, thereby enhancing shallow spatial location perception.

[0123] The output of the cross-attention module 5 is connected to the linear interpolation upsampling layer 41 at the corresponding level through a skip structure;

[0124] The cross-attention module 5 inputs weighted fusion features into the linear interpolation upsampling layer 41 so that the linear interpolation upsampling layer 41 obtains the original input for cascading during the upsampling process.

[0125] Specifically, to enable the upsampling module in the codec structure to achieve better feature aggregation, this scheme also introduces a skip structure to supplement the linear interpolation upsampling layer 41 at a specific level with additional original feature inputs for aggregation. Compared with the skip structure of the traditional single-modality codec, this skip structure uses a cross-attention mechanism to complement the depth and color features on the two input branches, thereby simultaneously paying attention to regions where color / texture changes and regions where depth values ​​change, which helps the linear interpolation upsampling layer 41 to more accurately determine the object boundary contour.

[0126] In one embodiment, such as Figure 2 As shown, the cross-attention module 5 includes:

[0127] First input layer 51, first input layer 51 receives first convolutional image;

[0128] The second input layer 52 receives the second convolutional image;

[0129] The first convolutional network 53 has its input connected to the first input layer 51.

[0130] The first convolutional network 53 extracts the first convolutional image to form a first feature map and outputs it;

[0131] The first cross-computation module 54 has its input terminals connected to the first convolutional network 53 and the second input layer 52, respectively.

[0132] The first cross-computation module 54 performs element-wise multiplication of the second convolutional image and the first feature map to obtain the first cross-computation intermediate image;

[0133] The first cross-calculation module 54 then performs element-wise addition between the first cross-calculation intermediate image and the second convolution image to obtain the depth cross-calculation image;

[0134] The second convolutional network 55 has its input connected to the second input layer 52.

[0135] The second convolutional network 55 extracts the second convolutional image to form a second feature map and outputs it;

[0136] The second cross-computation module 56 has its input terminals connected to the second convolutional network 55 and the first input layer 51, respectively.

[0137] The second cross-computation module 56 performs element-wise multiplication of the first convolutional image and the second feature map to obtain the second cross-computation intermediate image;

[0138] The second cross-calculation module 56 then performs element-wise addition between the second cross-calculation intermediate image and the first convolution image to obtain a color cross-calculation image;

[0139] Cross-merging module 57 is connected to the first cross-calculation module 54 and the second cross-calculation module 56 respectively.

[0140] The cross-merging module 57 first stitches together the deep cross-computation image and the color cross-computation image, and then uses a convolution kernel to extract the weighted fusion features.

[0141] There are three cross-attention modules 4, which are respectively connected to the first convolutional layer 11 and the second convolutional layer 21 of the first three layers.

[0142] In one embodiment, such as Figure 3 As shown, the self-attention fusion module 3 includes:

[0143] The first semantic extraction module 31 receives color image features and performs semantic extraction to obtain color semantic features;

[0144] The first residual module 32 receives color image features and performs residual calculations on the color image features to obtain color residual features;

[0145] The first merging module 33 performs pixel-by-pixel multiplication of the color semantic features and the color residual features, and then adds them to the color image features to obtain the color self-attention features;

[0146] The second semantic extraction module 34 receives depth image features and performs semantic extraction to obtain depth semantic features.

[0147] The second residual module 35 receives depth image features and performs residual calculations on the depth image features to obtain depth residual features;

[0148] The second merging module 36 performs pixel-wise multiplication of the deep semantic features and the deep residual features, and then adds them to the deep image features to obtain the deep self-attention features.

[0149] Attention fusion module 37 is connected to the first merging module 35 and the second merging module 36 respectively;

[0150] The attention fusion module 37 adds the color self-attention features and the depth self-attention features pixel by pixel to obtain the fused image.

[0151] In one embodiment, such as Figure 4 As shown, it also includes a discriminator 6, the input of which is connected to the output of the casting recognition model, so as to form an adversarial generative network with the casting recognition model for training;

[0152] Discriminator 6 is tightly coupled with the encoding / decoding structure to form an end-to-end adversarial training framework, and directly optimizes the model parameters through backpropagation during the training process.

[0153] Specifically, in practical applications, for a certain specific defect, the number of samples that can be collected may be small (3-5), which leads to a small number of samples and imbalance of samples among different classes during training.

[0154] To address the aforementioned issues, this solution primarily employs adversarial training. Discriminator 6 uses the output features of the recognition network as conditional input to synthesize features consistent with the true label distribution. This network is then connected to the output of the main segmentation network, forming a dynamic adversarial optimization mechanism. In defect detection tasks, the adversarial network can generate defect images with specific depth variations based on the output segmentation results, covering complex defect types such as weld points, weld scars, and ridge protrusions, thereby solving the problem of insufficient recognition accuracy in small sample scenarios.

[0155] Meanwhile, by tightly coupling the discriminator 6 with the encoding / decoding structure, an end-to-end adversarial training framework is formed, and good training efficiency is achieved by directly optimizing the model parameters through backpropagation during the training process.

[0156] The model itself can use the traditional cross-entropy loss function.

[0157] A quality inspection system is implemented based on the above-mentioned method for segmenting surface defects in castings.

[0158] Specifically, the aforementioned method for segmenting surface defects in castings is primarily implemented as a software example within computer equipment, such as the processing unit of a quality inspection device. This quality inspection equipment typically has an externally connected set of acquisition devices capable of simultaneously acquiring color and depth images. Depending on the requirements, it should also be equipped with multi-axis accelerometers or other equivalent sensors for recording pose information to mark the positions of the corresponding images.

[0159] By feeding color and depth images into the pre-trained model, the corresponding defect areas can be identified as quality inspection results.

[0160] Depending on the needs, the quality inspection system may also be configured on a quality inspection robot. In this case, the sensors are usually set at the front end of the robotic arm and automatically collect images according to a preset program.

[0161] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for segmenting surface defects in castings based on multimodal attention, characterized in that, include: Color and depth images are acquired for the surface to be inspected, and then fed into the casting defect recognition model for identification. The casting recognition model is an encoder-decoder structure. The casting recognition model extracts the color image features of the color image and the depth image features of the depth image respectively and fuses them to obtain a fused image. Then, the fused image is upsampled to obtain the segmentation result corresponding to the surface defects of the casting. For the depth image, the casting recognition model uses multiple gradient operators of different scales for enhancement processing during the image feature extraction process.

2. The method for segmenting surface defects of castings according to claim 1, characterized in that, The casting recognition model includes an independent depth image coding branch; The depth image coding branch extracts the depth image features sequentially through multiple second convolutional layers; The gradient operator processes the depth image to obtain an enhanced depth image, which is then fed into the second convolutional layer of the corresponding level for fusion.

3. The method for segmenting surface defects of castings according to claim 2, characterized in that, The depth image coding branch has four levels of the second convolutional layer, and the receptive field of the second convolutional layer gradually expands. The gradient operator includes three gradient operators with progressively increasing scales; the gradient operators enhance the depth image to obtain the depth feature image, and then input the first three layers of the second convolutional layer in sequence.

4. The method for segmenting surface defects of castings according to claim 3, characterized in that, The gradient operator corresponds to the small-scale operator of the second convolutional layer in the first layer, the medium-scale operator of the second convolutional layer in the second layer, and the large-scale operator of the second convolutional layer in the third layer; The small-scale operator is represented as: The mesoscale operator is represented as: The large-scale operator is represented as:

5. The method for segmenting surface defects of castings according to claim 2, characterized in that, The casting recognition model includes an independent color image coding branch; The color image coding branch extracts the color image features from the color image through multiple first convolutional layers in sequence, with the number of first convolutional layers and second convolutional layers being the same.

6. The method for segmenting surface defects in castings according to claim 1, characterized in that, The casting recognition model includes an independent self-attention fusion module, which fuses the color image features and the depth image features to obtain the fused image; The self-attention fusion module includes: A first semantic extraction module receives the color image features and performs semantic extraction to obtain color semantic features; The first residual module receives the color image features and performs residual calculations on the color image features to obtain color residual features; The first merging module performs pixel-by-pixel multiplication of the color semantic features and the color residual features, and then adds them to the color image features to obtain color self-attention features; The second semantic extraction module receives the depth image features and performs semantic extraction to obtain depth semantic features. The second residual module receives the depth image features and performs residual calculations on the depth image features to obtain depth residual features; The second merging module performs pixel-by-pixel multiplication of the deep semantic features and the deep residual features, and then adds them to the deep image features to obtain the deep self-attention features; An attention fusion module, which is connected to the first merging module and the second merging module respectively; The attention fusion module adds the color self-attention features and the depth self-attention features pixel by pixel to obtain the fused image.

7. The method for segmenting surface defects in castings according to claim 5, characterized in that, The casting recognition model also includes an independent upsampling module, which performs upsampling processing on the fused image through multiple linear interpolation upsampling layers and then outputs the segmentation result.

8. The method for segmenting surface defects of castings according to claim 7, characterized in that, The casting recognition model also includes multiple cross-attention modules; The input of the cross-attention module is connected to the first convolutional layer and the second convolutional layer at the corresponding level, respectively; The cross-attention module performs weighted fusion on the first convolutional image output by the first convolutional layer and the second convolutional image output by the second convolutional layer to form a weighted fusion feature, thereby enhancing shallow spatial location perception. The output of the cross-attention module is connected to the linear interpolation upsampling layer at the corresponding level via a skip structure; The cross-attention module inputs the weighted fusion features into the linear interpolation upsampling layer so that the linear interpolation upsampling layer obtains the original input for cascading during the upsampling process.

9. The method for segmenting surface defects of castings according to claim 8, characterized in that, The cross-attention module includes: A first input layer, wherein the first input layer receives the first convolutional image; The second input layer receives the second convolutional image; A first convolutional network, wherein the input of the first convolutional network is connected to the first input layer; The first convolutional network extracts the first convolutional image to form a first feature map and outputs it. The first cross-computation module has its input terminals connected to the first convolutional network and the second input layer, respectively. The first cross-computation module performs element-wise multiplication of the second convolutional image and the first feature map to obtain the first cross-computation intermediate image; The first cross-calculation module then performs element-wise addition between the first cross-calculation intermediate image and the second convolutional image to obtain a depth cross-calculation image; The second convolutional network, the input of which is connected to the second input layer; The second convolutional network extracts the second convolutional image to form a second feature map and outputs it. The second cross-computation module has its input terminals connected to the second convolutional network and the first input layer, respectively. The second cross-computation module performs element-wise multiplication of the first convolutional image and the second feature map to obtain the second cross-computation intermediate image; The second cross-calculation module then performs element-wise addition between the second cross-calculation intermediate image and the first convolutional image to obtain a color cross-calculation image; A cross-merging module, which is connected to the first cross-computing module and the second cross-computing module respectively; The cross-merging module first stitches together the deep cross-computation image and the color cross-computation image, and then uses a convolution kernel to extract the weighted fusion feature.

10. The method for segmenting surface defects of castings according to claim 8, characterized in that, The cross-self-attention module has three parts, which are respectively connected to the first convolutional layer and the second convolutional layer of the first three layers.

11. The method for segmenting surface defects in castings according to claim 1, characterized in that, It also includes a discriminator, the input of which is connected to the output of the casting recognition model, so as to form an adversarial generative network with the casting recognition model for training.

12. A quality inspection system, characterized in that, This method is implemented based on the casting surface defect segmentation method according to any one of claims 1-11.

Images