Method and apparatus for semantic segmentation model construction and weld image processing

Abstract

The application discloses a method and device for semantic segmentation model construction and welding image processing, and relates to the technical field of industrial vision. The method comprises: acquiring an initial semantic segmentation model architecture, including a backbone network, an ASPP module and a decoder which process input images in sequence; and adding modules in the initial semantic segmentation model architecture to form a final semantic segmentation model. Wherein, an anti-lighting calibration module is added in the backbone network and is configured to perform photometric shaping on early features of the input image; a first photometric alignment module is added between the backbone network and the ASPP module, and a second photometric alignment module is added at the output end of the decoder, both of which are configured to align original image features to photometric enhanced image features at different positions. The photometric shaping function of the anti-lighting calibration module and / or the photometric alignment function of any one of the two photometric alignment modules can pull the representation of the original image to the representation of the bright image and the output.

Description

Technical Field

[0001] This application relates to the field of industrial vision technology, specifically to a method and apparatus for semantic segmentation model construction and welding image processing. Background Technology

[0002] Industrial cameras, for example, when used in welding automation and quality inspection scenarios, often capture grayscale or visible light images in environments with low light, strong reflections (such as molten pools and metal mirrors), local shadows, and unstable exposure timing. This frequently results in problems such as underexposed images, local overexposure, and workpiece reflections. When using images captured by industrial cameras for semantic segmentation tasks (such as segmenting workpieces and welds), these imaging lighting conditions can easily degrade semantic segmentation results, affecting the stability and accuracy of semantic segmentation.

[0003] Traditional approaches attempt to alleviate such lighting problems through three paths: first, imaging and optical path modification, such as modifying structured light, polarization, and optical path geometry; second, image preprocessing enhancement, such as Retinex, Gamma, HDR compression, and low-light enhancement, to improve visibility; and third, robust design of the algorithm layer, such as illuminance compensation and illumination-invariant modeling.

[0004] Based on these three paths, many image semantic segmentation optimization schemes have been proposed in existing technologies. Adapting to the core ideas of the above three paths, these schemes can be summarized as illumination invariant or enhancement strategies based on "preprocessing / hardware / traditional models". However, such strategies have at least the following drawbacks when facing deep semantic segmentation: 1) Long inference links and high costs: Multiple networks / branches (such as "enhancement + segmentation" or dual segmentation network fusion) increase latency and deployment complexity on industrial lines, which is not conducive to edge / real-time scenarios.

[0005] 2) Decoupling of preprocessing and downstream learning: Relying solely on image enhancement to suppress glare results in strong coupling between enhancement parameters and scene, and lacks constraints on segmentation features, thus limiting stability.

[0006] 3) The lack of cross-illuminance consistency constraints results in a large prediction bias in the "bright / dark" domain.

[0007] Therefore, new image semantic segmentation schemes are needed to be applicable to complex lighting scenarios such as welding. Summary of the Invention

[0008] The purpose of this application is to provide a semantic segmentation model construction method, a welding image processing method, a welding image processing apparatus and medium, so as to at least partially solve the above-mentioned technical problems.

[0009] To achieve the above objectives, a first aspect of this application provides a method for constructing a semantic segmentation model, comprising: obtaining an initial semantic segmentation model architecture, the initial semantic segmentation model architecture including a backbone network for sequentially processing an input image, an ASPP module, and a decoder; and adding one or more of an anti-illumination calibration module, a first photometric alignment module, and a second photometric alignment module to the initial semantic segmentation model architecture to form a final semantic segmentation model. The anti-illumination calibration module is disposed in the backbone network and configured to perform photometric shaping on early features of the input image; the first photometric alignment module is disposed between the backbone network and the ASPP module, and the second photometric alignment module is disposed at the output of the decoder, both configured to align original image features into photometric-enhanced image features at different locations.

[0010] In this embodiment, the anti-lighting calibration module includes the following modules that sequentially process the early features: a channel allocation module configured to suppress brightness-sensitive channels and highlight structure-related channels by redistributing information from each channel; a channel-by-channel processing module configured to perform spot processing within a set local range for each channel; an automatic gain module configured to generate 0-1 coefficients for each channel based on overall image brightness statistics to achieve adaptive normalization for overall exposure drift; and a residual pass-through module configured to output the superimposed features of the early features and the reshaped features processed by the channel allocation module, the channel-by-channel processing module, and the automatic gain module.

[0011] In this embodiment, the first photometric alignment module is obtained based on the following brightness-darkness consistency training steps: acquiring a training set of paired images, wherein the paired images include the original image and the corresponding photometric enhancement image; performing data processing on the training set to obtain original image features and photometric enhancement image features suitable for the output position of the backbone network; and performing brightness-darkness consistency training on the original image features and the photometric enhancement image features based on the L1 norm loss function to obtain the first photometric alignment module, wherein the L1 norm loss function is configured to set a stopping gradient for the photometric enhancement image features.

[0012] In this embodiment of the application, for the training set, the second photometric alignment module is obtained based on the following brightness-darkness consistency training steps: data processing is performed on the training set to obtain original image features and photometric enhancement map features suitable for the output position of the decoder; and brightness-darkness consistency training is performed on the original image features and the photometric enhancement map features based on the KL divergence loss function to obtain the second photometric alignment module, wherein the KL divergence loss function is configured to set a stopping gradient for the photometric enhancement map features.

[0013] In this embodiment of the application, the semantic segmentation model construction method further includes: configuring a total loss function for the semantic segmentation model, wherein the total loss function is associated with a supervision loss value for photometric enhancement map features only, a loss value for the output position of the backbone network determined based on the L1 norm loss function, and a loss value for the output position of the decoder determined based on the KL divergence loss function.

[0014] In this embodiment of the application, the total loss function is described as:

[0015] in, This represents the loss value of the entire semantic segmentation model. This represents the loss value determined based on the L1 norm loss function for the output position of the backbone network. This represents the loss value determined based on the KL divergence loss function for the output position of the decoder. and They represent and The corresponding weighting coefficients, This represents the supervised loss value applied only to the features of the photometric enhancement map.

[0016] In this embodiment, the anti-light calibration module is located at the first layer output position of the backbone network, and the first photometric alignment module is located at the last layer output position of the backbone network.

[0017] A second aspect of this application provides a welding image processing method, comprising: acquiring a grayscale image of a welding scene; acquiring a training set of paired images, wherein the paired images include an original image of the welding scene and a corresponding photometric enhancement image; constructing a semantic segmentation model for the welding image based on the training set using any of the aforementioned semantic segmentation model construction methods; and processing the grayscale image of the welding scene through the semantic segmentation model to obtain a final welding segmentation result image.

[0018] A third aspect of this application provides a welding image processing apparatus, comprising: an image acquisition unit for acquiring a grayscale image of a welding scene; a data pair construction unit for acquiring a training set of paired images, wherein the paired images include an original image of the welding scene and a corresponding photometric enhancement image; a model training unit for constructing a semantic segmentation model for the welding image based on the training set using any of the aforementioned semantic segmentation model construction methods; and an image processing unit for processing the grayscale image of the welding scene through the semantic segmentation model to obtain a final welding segmentation result image.

[0019] A fourth aspect of this application provides a welding image processing apparatus, comprising: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement any of the aforementioned semantic segmentation model construction methods or any of the aforementioned welding image processing methods.

[0020] The fifth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to execute any of the semantic segmentation model construction methods or any of the welding image processing methods described above.

[0021] Through the above technical solution, the embodiments of this application utilize the photometric shaping function of the anti-illumination calibration module and / or the photometric alignment function of any one of the two photometric alignment modules to pull the representation of the original image toward the representation and output of the bright image, so that the same scene still maintains a stable segmentation result under different illuminations. Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a flowchart of a semantic segmentation model construction method according to Embodiment 1 of this application; Figure 2 This diagram schematically illustrates an initial semantic segmentation model architecture according to an embodiment of this application. Figure 3 This illustration schematically shows the structure of an improved semantic segmentation model based on the ResNet-DeepLabV3+ architecture according to an embodiment of this application; Figure 4 A schematic diagram of the structure of an anti-lighting calibration module according to an embodiment of this application is shown.

[0023] Figure 5 The illustration shows an example original image and a corresponding photometric enhancement image according to an embodiment of this application, wherein the left side represents the original image and the right side represents the photometric enhancement image. Figure 6 A schematic flowchart of the welding image processing method according to Embodiment 2 of this application is shown. Figures 7(a) and 7(b) schematically illustrate the changes of the feature difference index and probability difference index of the training set according to the embodiments of this application with the rounds; Figures 8(a) and 8(b) schematically illustrate the changes of the feature difference index and probability difference index of the verification set according to the embodiments of this application with each round; Figures 9(a) and 9(b) schematically illustrate the changes in logits KL loss and total loss of the training set with the number of rounds according to embodiments of this application; Figures 10(a)-10(c) schematically illustrate the reasoning effect of the model according to the embodiments of this application on unfamiliar images with uniform brightness, wherein Figure 10(a) is the original image, Figure 10(b) is the control group, and Figure 10(c) is the experimental group. Figures 11(a)-11(c) schematically illustrate the reasoning effect of the model according to the embodiments of this application on unfamiliar images with non-uniform brightness, wherein Figure 11(a) is the original image, Figure 11(b) is the control group, and Figure 11(c) is the experimental group. Figure 12 A schematic diagram of the welding image processing apparatus according to Embodiment 3 of this application is shown; and Figure 13 A schematic diagram of the structure of a welding image processing apparatus according to another embodiment of this application is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0025] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0026] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0027] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0028] The following describes some of the terms used in the embodiments of this application, so that those skilled in the art can more clearly understand the embodiments of this application.

[0029] 1. ResNet-DeepLabV3+: DeepLabV3+ is an advanced semantic segmentation model, while ResNet-DeepLabV3+ specifically refers to a version that uses ResNet (usually ResNet-50 or ResNet-101) as the encoder (backbone network + ASSP module). See reference... Figure 3 The backbone network, for example, includes five hierarchical feature extraction layers, denoted as layer0 / layer1 / layer2 / layer3 / layer4. Layer0 outputs early image features, and layer4 refers to the highest-level semantic features before ASPP. The ASPP module, or Atrous Spatial Pyramid Pooling module, is a common module in the DeepLab framework and a key component of the model encoder. It captures features of objects in the image at different scales / contextual ranges simultaneously through multiple parallel dilated convolutional layers with different dilation rates (e.g., dilation rates = 6, 12, 18) and global pooling. The decoder is used to restore spatial resolution and generate segmentation results similar in size to the input image. The output of the last convolutional layer of the decoder (before activation) is denoted as logits, which will be used to calculate supervision and KL consistency in this embodiment.

[0030] 2. Original Image and Photometric Enhancement Image: In the embodiments of this application, the original image and the photometric enhancement image are a pair of relative concepts. For example, each sample in the training set has two images with the same name: x (original image), which is photometrically processed to obtain a photometric enhancement image x′. The two images are geometrically aligned and only differ in photometric attributes.

[0031] 3. Training and Inference Phases: The training phase is when the model "learns" or "discovers" the mapping pattern between input data (such as images) and output labels (such as segmentation maps) from labeled data. Essentially, it is the process of finding the optimal model parameters (weights and biases). The inference phase refers to using a pre-trained model with fixed parameters to predict new and unseen input data and output the results.

[0032] 4. CE (Cross-Entropy) loss, BCE (Binary Cross-Entropy) loss, and Dice loss: all are loss metric parameters.

[0033] 5. KL (Kullback-Leibler) divergence: A pixel-wise Bernoulli distribution KL used to measure, predict, or label the consistency between two outputs.

[0034] 6. Channel Allocation: This refers to the redistribution of information among the channels. For example, using a 1x1 convolution (or fully connected layer) to increase the number of channels in the input feature map by a certain factor (e.g., 4 times or 6 times), this factor can be understood as the "channel allocation" or "expansion factor". In the embodiments of this application, based on the channel allocation, channels sensitive to brightness and channels highlighting structure-related information are suppressed.

[0035] 7. Mild reconfiguration: Reconfiguration refers to the redistribution of information as described above. Mild reconfiguration, for example, refers to a basic, non-intelligent redistribution and expansion of channel information through a simple, position-independent linear transformation of the channel allocation.

[0036] 8. Stop Gradient (sg): This is a technique for controlling the gradient flow during neural network training. It keeps the input value unchanged during forward propagation, but ignores the gradient of the operation during backward propagation, which is equivalent to setting the gradient to zero, thereby freezing the relevant network parameters and preventing them from being updated.

[0037] Example 1 Figure 1 The illustration shows a flowchart of a semantic segmentation model construction method according to Embodiment 1 of this application. Figure 1 As shown in the figure, this application provides a semantic segmentation model construction method, which may include the following steps S100-S200.

[0038] Step S100: Obtain the initial semantic segmentation model architecture.

[0039] The initial semantic segmentation model architecture is as follows: Figure 2 As shown, it includes a backbone network that processes the input image sequentially, an ASPP module, and a decoder, which can employ methods such as...Figure 3 The architecture shown is "backbone network + ASPP module + decoder". However, it should be noted that, in addition to ResNet-DeepLabV3+, other conventional semantic segmentation models with "encoder + decoder" architectures can also be used as the initial model architecture for the embodiments of this application, such as the PSPNet model, the Swin-DeepLab model, the BiSeNet V2 and the HRNetV2 + OCR model, etc.

[0040] Step S200: Add one or more of the following modules to the initial semantic segmentation model architecture to form the final semantic segmentation model: an anti-illuminance calibration module set in the backbone network; a first photometric alignment module set between the backbone network and the ASPP module; and a second photometric alignment module set at the output of the decoder.

[0041] The Illumination Consistency Adapter (ICA) module, for example, is located at the output of the first layer of the backbone network. It is configured to perform photometric shaping on early features of the input image. This photometric shaping is a lightweight adaptive shaping method, which suppresses "illumination-induced artifacts," including overall brightness / darkness shifts and local highlight / shadow patches, while ensuring approximate identity under "abnormal lighting" through residual design. The Illumination Consistency Adapter module will be described in detail below with examples, and will not be repeated here.

[0042] The first photometric alignment module is positioned, for example, at the output of the last layer of the backbone network, while the second photometric alignment module is positioned at the output of the decoder. Thus, both are configured to align the original image features to photometric enhancement image features at different locations. The first and second photometric alignment modules are obtained during model training based on brightness consistency training on paired images (original image and photometric enhancement image): the former compresses feature drift between the "bright / dark regions" at the high semantic layer of the backbone network (before the ASPP module), ensuring stable input for subsequent classification heads under both illumination conditions (the illumination corresponding to the original image and the photometric enhancement image, respectively); the latter, at the output level, further constrains the category distribution of the original image branch, thereby eliminating the "bright / dark region" bias in the prediction probability. The compression of feature drift between the "bright / dark regions" and the elimination of bias can both be understood as falling under the category of photometric alignment.

[0043] In a preferred embodiment, the first photometric alignment module is obtained based on the following brightness-darkness consistency training steps: acquiring a training set of paired images, wherein the paired images include the original image and the corresponding photometric enhancement image; performing data processing on the training set to obtain original image features and photometric enhancement image features suitable for the output position of the backbone network; and performing brightness-darkness consistency training on the original image features and the photometric enhancement image features based on the L1 norm loss function to obtain the first photometric alignment module.

[0044] In a more preferred embodiment, for the training set, the second photometric alignment module is obtained based on the following brightness-darkness consistency training steps: data processing is performed on the training set to obtain original image features and photometric enhancement map features suitable for the output position of the decoder; and brightness-darkness consistency training is performed on the original image features and the photometric enhancement map features based on the KL divergence loss function to obtain the second photometric alignment module.

[0045] The following text will provide a detailed introduction to the specific training process and application of the two photometric alignment modules with examples, which will not be repeated here.

[0046] Thus, through steps S100-S200, the following was obtained: Figure 2 The semantic segmentation model shown utilizes the photometric shaping function of the anti-illumination calibration module and / or the photometric alignment function of any one of the two photometric alignment modules to pull the representation of the original image towards the representation and output of the bright image, so that the same scene still maintains stable segmentation results under different illuminations.

[0047] The following is through Figure 3 The following examples will be used to further illustrate the specific implementation process of the semantic segmentation model construction method in the embodiments of this application. Figure 3 This is a schematic diagram of the structure of an improved semantic segmentation model based on the ResNet-DeepLabV3+ architecture, constructed using the semantic segmentation model construction method of the embodiments of this application. The newly added modules are shown by grayscale filling.

[0048] Regarding this example, the core model construction idea includes: introducing a front-end ICA module between layer 0 and layer 1 to perform lightweight adaptive shaping on early features entering the backbone network; during training, using prepared paired images (original image x and brightness-enhanced image x′) to perform brightness / darkness consistency training: applying feature consistency (L1 norm loss) in layer 4 (before ASPP), applying unidirectional KL consistency (stop gradient / Teacher for x′) in logits, and combining it with conventional supervised loss (BCE+Dice); during inference, only a single input is used, ICA is always present, and there is zero consistency inference overhead.

[0049] Based on this model construction approach, the following sections will be divided into two parts: front-end anti-lighting calibration and bright / dark consistency training.

[0050] I. Front-end anti-light calibration.

[0051] The placement of the front-end anti-glare calibration module (ICA), which is a key module, is between the layer 0 (stem) output of the backbone network and the layer 1 input, with the input / output size and number of channels remaining unchanged (e.g., 64×H / 4×W / 4).

[0052] The functional goal of the illumination resistance calibration module (ICA) is to suppress "illumination-induced pseudo-differences" before entering the backbone network, including overall brightness / darkness drift and local highlight / shadow patches, while ensuring approximate immutability in "abnormal illumination" through residual design.

[0053] Based on the above placement location and functional objectives, such as Figure 4 As shown, the anti-lighting calibration module may include the following modules that sequentially process early features of the input image: a channel allocation module, configured to suppress brightness-sensitive channels and highlight structure-related channels by redistributing information from each channel; a channel-by-channel processing module, configured to perform spot processing within a set local range for each channel; an automatic gain module, configured to generate 0-1 coefficients for each channel based on the overall image brightness statistics to achieve adaptive normalization for overall exposure drift; and a residual pass-through module, configured to output the superimposed features of the early features and the reshaping features processed by the channel allocation module, the channel-by-channel processing module, and the automatic gain module.

[0054] More specifically, the four modules mentioned above are configured as follows: (a) Channel matching module (1×1 convolution + normalization + activation): performs a slight rematch of the response of each channel, first suppressing the channels that are sensitive to brightness, and then highlighting the structure-related channels.

[0055] (b) Channel-by-channel processing module (3×3 convolution (depthwise separable convolution) + normalization + activation): "squeeze" the highlight / shadow spots in each channel locally, while trying not to change the boundary geometry.

[0056] (c) Automatic gain control (SE gate): Generates coefficients of 0–1 for each channel based on the overall image brightness statistics, and adaptively normalizes the overall exposure drift.

[0057] (d) Residual pass-through module: Output = Input + (the above-shaped features × channel coefficients), ensuring stable convergence and low side effects.

[0058] The total number of parameters in the anti-light calibration module is less than 10,000 (taking C=64 as an example), and its impact on inference latency is negligible. In addition, unlike the common practice in existing technologies of "replacing IN / BN in multiple places in the network", this anti-light calibration module keeps the input / output space size and channels unchanged. Only one change is needed to maintain the statistical stability of the entire network, thereby avoiding the complexity and parameter tuning risks of large-scale structural modifications.

[0059] II. Light / Dark Consistency Training.

[0060] This part mainly involves the training period. The core idea is to directly apply supervision only to the illumination-enhanced image, and use ICA and consistency constraints to pull the representation of the original image towards the representation and output of the bright image, so that the same scene can still maintain stable segmentation results under different illuminations.

[0061] Based on this idea, the specific implementation process of consistency constraints includes three parts: data organization, data processing, and loss design, which are described in detail below.

[0062] 1. Data organization.

[0063] That is, obtain a training set of paired images. Each sample in the training set has two images with the same name: the original image x, such as... Figure 5 The left side is shown; and the photometrically enhanced image x′ is obtained by photometric processing of the original image x, as shown. Figure 5 The right side of the image is shown. The two images are geometrically aligned, differing only in their photometric properties.

[0064] 2. Data processing.

[0065] That is, the training set is processed to obtain the original image features and photometric enhancement image features suitable for the first photometric alignment module or the second photometric alignment module.

[0066] The main data processing includes forward pass and alignment, that is: concatenating x and x′ into the same batch in the batch dimension, and passing them forward in the same network (sharing weights and BN statistics) to obtain the two-way layer4 features and logits.

[0067] 3. Loss design.

[0068] For the entire segmentation model, a total loss function is configured, wherein the total loss function is associated with the supervision loss value for photometric enhancement map features only, the loss value for the output position of the backbone network determined based on the L1 norm loss function, and the loss value for the output position of the decoder determined based on the KL divergence loss function.

[0069] In the example, the total loss function is described as follows: (1) in, This represents the loss value of the entire semantic segmentation model. This represents the loss value determined based on the L1 norm loss function for the output position of the backbone network. This represents the loss value determined based on the KL divergence loss function for the output position of the decoder. and They represent and The corresponding weighting coefficients, This represents the supervised loss value applied only to the features of the photometric enhancement map.

[0070] It is easy to see that the loss function shown in the above formula (1) is a three-component synthesis, namely the supervised loss. Feature consistency loss Logits consistency loss The three are described in detail below.

[0071] (1) Supervision loss is essentially the loss of the original classification and segmentation tasks.

[0072] In the example, supervised loss It is determined according to the following formula: (2) In the formula, BCE represents the binary cross-entropy, CE represents the cross-entropy, and Dice represents the Dice loss. This represents the set of image pixel locations in the training set. u represents pixel coordinates, C represents the total number of categories, c represents a specific category, x′ represents the photometric enhancement map branch, and y represents the preset label. This represents the predicted probability of the light-intensity map branch x′ with respect to pixel u in category c. This indicates the hyperparameters that are set.

[0073] As can be seen from the above formula (2), the supervision term is the photometric enhancement map. That is, the supervision loss (BCE+Dice or CE+Dice) is calculated only on the photometric enhancement map x′ after illumination enhancement based on the label y, forming the dominant segmentation learning signal, while no direct supervision is applied to the original map x.

[0074] (2) Feature consistency loss .

[0075] This is a new loss calculation relative to existing technologies, and as mentioned above, it is calculated based on the L1 norm loss function, which is configured to have a stopping gradient for the photometric enhancement map features.

[0076] In the example, the L1 norm loss function is described as follows: (3) In the formula, This represents the loss value corresponding to the output position of the backbone network. This represents the set of image pixel locations in the training set. u Represents pixel coordinates, sg(·) indicates stopping the gradient for the corresponding image branch, x represents the original image branch, and x′ represents the photometric enhancement image branch. Let represent the feature tensor of the backbone network at the highest semantic layer, and ‖·‖1 represent the L1 norm.

[0077] As can be seen from formula (3), this example aligns the representations of paired samples {x, x′} in the high semantic layer of the segmentation network (such as layer 4 before ASPP), where x′ is obtained from x through only photometric transformation (without changing geometry). Intuitively, when applied to image processing in welding scenarios, the semantic structure of the weld and the workpiece should remain consistent before and after brightness enhancement, with the difference mainly coming from the contrast changes caused by photometric noise and reflections / shadows. Therefore, the original image x is taken as the "student" and the bright image x′ as the "teacher", and the gradient is stopped on the teacher branch to avoid the teacher being pulled by the noise of the student. The purpose of doing this is to directly compress the feature drift between the "bright domain / dark domain" in the high semantic space, so that the subsequent classification head can obtain stable input under both illuminations.

[0078] Combination Figure 3 The feature consistency loss at layer 4 constrains the pixel-wise L1 difference between the feature maps of x and x′ at layer 4 to make it as small as possible, thereby directly writing "lighting insensitivity" into the semantic feature layer.

[0079] Using this formula (3), the deviation between students and teachers in semantic features can be measured pixel by pixel, and the L1 norm is more robust to local anomalies (such as highlights).

[0080] (3) Logits consistency loss .

[0081] This involves a novel loss calculation compared to existing technologies, and as mentioned above, it is calculated based on the KL divergence loss function, which is configured to have a stopping gradient for the photometric enhancement map features. In the example, the KL divergence loss function is described as follows: (4) In the formula, This represents the loss value corresponding to the output position of the decoder. This represents the set of image pixel locations in the training set.u Represents pixel coordinates, C Indicates the total number of categories. c This indicates a specific category, and sg(·) indicates stopping the gradient for the corresponding image branch. x Indicates the branch of the original graph. x ′ indicates the photometric enhancement plot branch. Let x' represent the predicted probability of the light-intensity map branch x′ with respect to pixel u in category c. Formula (4) applies to each pixel... u Each category c Minimizing the relative entropy between teachers and students is equivalent to having students "imitate" the bright image under the original lighting for a more stable output. This is different from aligning only at the feature layer. compared to, It directly affects the final decision distribution, and the superposition of the two can simultaneously converge the representation and prediction.

[0082] According to formula (4), at the output level, this application treats the bright map branch as a more stable source of soft labels and constrains the category distribution of the original map branch through a one-way KL (teacher → student) to eliminate the "bright / dark domain" bias in the prediction probability. Unlike common symmetric consistency or mean teacher, the teacher in this example is only given by the bright map branch and the gradient stops, which avoids introducing an additional momentum network and ensures that the final inference is still a single path.

[0083] In addition, the consistency loss in the output stage (one-way KL) is achieved by applying KL divergence to the output of the original graph x using the output distribution of the photometric enhancement graph x′ as the teacher, which requires the two to be consistent at the probability level.

[0084] Regarding the consistency loss mentioned above, it can be seen that its training process relies solely on consistency without additional supervision. Therefore, this example completely constrains the alignment of x′ by determining the newly added consistency loss, thereby achieving the learning of the original image x.

[0085] Furthermore, returning to formula (1) above, this example, after completing the loss design, further assigns appropriate weights to the two consistency losses, i.e., determines... and For example, appropriate preset weights are assigned to the two consistency losses, and a "warm-up" operation is performed in the early epochs at the beginning of training, linearly increasing their weights from small values ​​to the target values ​​to smoothly guide feature alignment without prematurely "suppressing" representation learning. Thus, this example applies L1 feature consistency to layer 4 (the highest layer semantic features before ASPP), applies a one-way KL algorithm to x (student) relative to x′ (stopping gradient teacher) at logits, and uses a weight warm-up strategy to achieve constant scheduling (e.g., 0.05 → 0.25).

[0086] After the training period, this example yields a semantic segmentation model that can be directly used for inference. Combined with... Figure 3 It can be seen that during the inference period, the following characteristics are present: only a single input x is used; consistency constraints are no longer involved in training; ICA is resident and does not increase inference branches or significant latency. That is, through the semantic segmentation model in this example, it can be seen that the inference process can be directly applied to images without photometric processing.

[0087] In summary, through the above examples, the semantic segmentation model construction method of Embodiment 1 of this application has at least the following advantages: 1. Minimal structural changes, seamless deployment.

[0088] A lightweight ICA is inserted only between layer 0 and layer 1, with a parameter count of less than 10,000, making it compatible with existing DeepLabV3+ engineering systems. The inference structure remains unchanged, the ICA module is always resident, and there is only a single input path (the original image without increased brightness), with no second input path or teacher branch. Overall, compared to existing technologies such as "replacing the entire network layer / pre-enhancing the model / hardware modification," the engineering risk and cost are significantly lower.

[0089] 2. Incorporate "immutability" into the network, rather than simply "seen / enhanced".

[0090] Compared to existing techniques that only augment or enhance the foreground, the above example explicitly forces consistency between the bright and dark paths on layer 4 and logits. Furthermore, compared to training schemes that remain at the preprocessing / thresholding or device level, it forms an end-to-end consistent training loop from the semantic representation layer to the final output, reducing shortcuts such as "bright is foreground," and its robustness is measurable and verifiable.

[0091] Example 2 Figure 6 The schematic diagram illustrates a flow chart of a welding image processing method according to Embodiment 2 of this application, wherein the welding images include, but are not limited to, images of the workpiece and / or weld. Figure 6 As shown in Embodiment 2 of this application, a welding image processing method is provided, which may include the following steps S610-S640.

[0092] Step S610: Obtain the grayscale image of the welding scene.

[0093] For example Figure 5 The left side shows a grayscale image taken by a welding camera.

[0094] Step S620: Obtain a training set of paired images, wherein the paired images include the original image of the welding scene and the corresponding photometric enhancement image.

[0095] The original image and the photometric enhancement image can also be referenced. Figure 5 .

[0096] Step S630: Based on the training set, construct a semantic segmentation model for welding images using the semantic segmentation model construction method described in Embodiment 1 above.

[0097] That is, based on paired images and the model construction method of Example 1, the segmentation network is segmented and trained to consistency, resulting in a semantic segmentation model specifically for processing welding images.

[0098] Step S640: Process the grayscale image of the welding scene using the semantic segmentation model to obtain the final welding segmentation result image.

[0099] As is known, in Embodiment 1 above, without changing the backbone network and decoding structure of the semantic segmentation model, by adding front-end ICA and using ICA and consistency constraints during training, the representation of the original image can be shifted towards a bright image representation and output. Therefore, in Embodiment 2 of this application, through the above steps S610-S640, using this semantic segmentation model, the segmentation effect of the workpiece and weld seam in the welding image under different lighting conditions can be guaranteed, so that the quality of the output image will not be significantly affected by the lighting.

[0100] The following example demonstrates the final segmentation effect of the welding image processing method in Embodiment 2 of this application. In this example, a binary classification semantic segmentation model is tested, with the goal of segmenting a welded part from a grayscale image of a welding camera.

[0101] In this example, the data size is constant, which means that regardless of whether ICA and the consistency module are enabled (for ease of description, the first brightness alignment module and the second brightness alignment module are collectively referred to as the consistency module), the photometric enhancement image x′ and the original image x are always input into the network in pairs; when the module is disabled, the weights corresponding to the consistency module are simply set to zero, and ICA is degenerated into an identity mapping, so the number of training samples is exactly the same as the labeled number.

[0102] In addition, to facilitate verification of whether the brightness / darkness consistency holds, the training process records the following statistics in each iteration cycle: (1) Probability gap index (denoted as gapprob For two images of the same scene (enhanced image x′ and original image x), segmentation probability maps are obtained respectively. and Calculate the absolute difference pixel by pixel and then average it. The smaller the value, the more consistent the bright / dark output. In this example, the instantaneous values ​​(training set) and the average value of the validation set are recorded for each training batch, and the corresponding curves are named gap / trainprob and gap / valprob, which will be shown in Figure 7(a) and Figure 7(b), respectively.

[0103] (2) Feature difference index (feat-gap, denoted as gap) feat Extract the highest-level semantic features of the backbone. , Calculate the L1 distance element by element on the two-dimensional feature map and average it, i.e. This metric measures whether the ICA and the consistency module have eliminated illumination differences at the semantic representation layer. The corresponding instantaneous value and validation set curves are named gap / trainfeat and gap / valfeat, respectively, and will be shown in Figure 8(a) and Figure 8(b).

[0104] (3) Consistency loss item.

[0105] layer 4 L1 loss: i.e. As a feature consistency constraint for backpropagation, the curve is named loss / trainfeat (not shown in the figure). Logits KL loss: based on the probability distribution of the photointensity enhancement map Calculate the probability of the original graph for the teacher. The KL divergence, denoted as the loss / trainlogit curve, is shown in Figure 9(a). The two loss terms and the supervision term are weighted... They jointly participate in backpropagation. The curve loss / train represents the total loss after weighting the supervision and consistency terms, and will be shown in Figure 9(b).

[0106] The above statistics can be used to quantify whether the improved module (i.e., ICA + consistency module) involved in this example is effective: when ICA and consistency module are turned off, prob-gap, feat-gap and corresponding loss remain at high levels; after the module is turned on (even if the original graph has no direct supervision), these indicators drop rapidly and stabilize at low values, while the supervision loss continues to converge normally.

[0107] Further, control and experimental groups were set up. The control group was configured with ICA and the consistency module disabled, and the network only supervised the photometric enhancement map. The experimental group was configured with ICA and the consistency module enabled, and the teacher from KL was set as the highlight branch.

[0108] Figures 7(a)-7(b), 8(a)-8(b), and 9(a)-9(b) show the segmentation results, displayed using the visualization tool TensorBoard. The control group results are shown in gray, and the experimental group results are shown in green. Specifically, Figures 7(a) and 7(b) show the feature difference index gap of the training set. feat Sum of probability difference index gapprob The curves corresponding to the changes with epochs are denoted as gap / trainfeat and gap / trainprob, respectively; Figures 8(a) and 8(b) show the feature difference index gap of the validation set, respectively. feat Sum of probability difference index gapprob The curves corresponding to the changes with epoch are denoted as gap / valfeat and gap / valprob, respectively. Figures 9(a) and 9(b) show the changes of logits KL loss and total loss of the training set with epoch, respectively, and the curves corresponding to loss / trainlogit and loss / train are denoted as loss / trainlogit and loss / train, respectively.

[0109] According to TensorBoard evidence, when ICA and the consistency module are disabled (only bright image supervision is retained), gap / trainprob, gap / valprob, gap / trainfeat, and gap / valfeat remain at high levels, while loss / trainfeat and loss / trainlogit hardly decrease, indicating that the representation and output gap between the bright and dark paths cannot converge. When ICA and the consistency module are enabled (the original image still has no direct supervision), the above six curves decrease rapidly with training and stabilize at low values, indicating that the original bright image has the ability to handle brightness differences by aligning the bright image output, and can segment the dark image based on the segmentation ability learned under the bright image. The loss / train under the two configurations maintains a similar decreasing trajectory, proving that the improved module does not disrupt the convergence of the main supervision branch.

[0110] Furthermore, in the inference stage, Figures 10(a)-10(c) show the inference effect of the model on unfamiliar images with uniform brightness. Figure 10(a) is the original image, Figure 10(b) is the control group, and Figure 10(c) is the experimental group. It can be seen that the inference effect of the control group and the experimental group based on the image with adjusted brightness is basically the same.

[0111] Furthermore, Figures 11(a)-11(c) show the inference performance of the model on unfamiliar images with non-uniform brightness. Figure 11(a) is the original image, Figure 11(b) is the control group, and Figure 11(c) is the experimental group. It can be seen that the control group still outputs almost the entire background under the baseline processing, while the experimental group with the improvement module enabled can maintain a segmentation quality comparable to the bright image in Figure 10(c).

[0112] This example demonstrates that, without altering the backbone structure and inference morphology, the "ICA + Consistency Module" significantly reduces the differences in representation and output between bright and dark branches, making the inference quality of the unenhanced image close to that of the enhanced image, and suppressing the impact of strong reflections, local shadows, and overall exposure drift on segmentation stability and accuracy in welding scenarios.

[0113] The above experimental process and logs (TensorBoard, inference results) of this example provide quantitative and visual evidence that proves that the semantic segmentation module constructed in Embodiment 1 of this application has no adverse effect on the supervision loss of bright images when used for welding image processing in Embodiment 2 of this application, and does not increase the branching and significant computational overhead during inference.

[0114] Example 3 Figure 12 The schematic diagram illustrates the structure of a welding image processing apparatus according to Embodiment 3 of this application, which is based on the same inventive concept as the welding image processing method of Embodiment 2.

[0115] like Figure 12 As shown, the welding image processing device includes: an image acquisition unit for acquiring a grayscale image of a welding scene; a data pair construction unit for acquiring a training set of paired images, wherein the paired images include the original image of the welding scene and the corresponding photometric enhancement image; a model training unit for constructing a semantic segmentation model for welding images based on the training set using the semantic segmentation model construction method described in Embodiment 1 above; and an image processing unit for processing the grayscale image of the welding scene through the semantic segmentation model to obtain the final welding segmentation result image.

[0116] For example, the image acquisition unit is an industrial camera used to acquire grayscale images of the welding scene. The data pairing construction unit reads images with the same name, x (original image) and x′ (increased brightness image), from two folders to construct paired images and form a training set. The model training unit includes a segmentation network processing unit and a consistency training unit. The segmentation network processing unit contains the backbone network (resnet50), ASPP, decoder, and ICA front-end module mentioned in Embodiment 1 above. The consistency training unit is used to perform layer 4 L1 and logits unidirectional KL during training. The image processing unit is mainly used for result output and evaluation, such as outputting weld / workpiece segmentation results and quality indicators.

[0117] For more details on the implementation and effects of this third embodiment, please refer to the second embodiment described above, which will not be repeated here.

[0118] Figure 13 A schematic diagram of a welding image processing apparatus according to another embodiment of this application is shown. Figure 13 As shown, the welding image processing apparatus includes: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the semantic segmentation model construction method according to Embodiment 1 or the welding image processing method according to Embodiment 2. The welding image processing apparatus is, for example, a conventional controller and can be integrated into an industrial camera.

[0119] This application also provides a machine-readable storage medium storing instructions that cause a machine to execute the semantic segmentation model construction method described in Embodiment 1 or the welding image processing method described in Embodiment 2.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0126] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0128] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for constructing a semantic segmentation model, characterized in that, The semantic segmentation model construction method includes: Obtain the initial semantic segmentation model architecture, which includes a backbone network, an ASPP module, and a decoder that sequentially process the input image; and Add one or more of the following modules to the initial semantic segmentation model architecture to form the final semantic segmentation model: An anti-light calibration module located in the backbone network is configured to perform photometric shaping on early features of the input image. A first photometric alignment module located between the backbone network and the ASPP module, and a second photometric alignment module located at the output of the decoder, are configured to align the original image features into photometric enhancement image features at different locations.

2. The semantic segmentation model construction method according to claim 1, characterized in that, The anti-light calibration module includes the following modules that process the early features sequentially: The channel allocation module is configured to suppress brightness-sensitive channels and highlight structure-related channels by redistributing information from each channel. The channel-by-channel processing module is configured to perform spot processing within a set local area for each channel; The automatic gain module is configured to generate 0-1 coefficients for each channel based on the overall image brightness statistics in order to achieve adaptive normalization for overall exposure drift. as well as The residual pass-through module is configured to output the superimposed feature of the early feature and the shaped feature processed by the channel matching module, the channel-by-channel processing module and the automatic gain module.

3. The semantic segmentation model construction method according to claim 1, characterized in that, The first photometric alignment module is obtained based on the following brightness-darkness consistency training steps: Obtain a training set of paired images, wherein the paired images include the original image and the corresponding photometric enhancement image; The training set is processed to obtain the original image features and photometric enhancement image features suitable for the output position of the backbone network; as well as The first luminance alignment module is obtained by training the original image features and the luminance enhancement image features on the brightness-darkness consistency based on the L1 norm loss function, wherein the L1 norm loss function is configured to set a stopping gradient for the luminance enhancement image features.

4. The semantic segmentation model construction method according to claim 3, characterized in that, For the training set, the second photometric alignment module is obtained based on the following brightness-darkness consistency training steps: The training set is processed to obtain original image features and photometric enhancement image features suitable for the output position of the decoder; as well as The second luminance alignment module is obtained by training the original image features and the luminance enhancement image features on brightness and darkness consistency based on the KL divergence loss function, wherein the KL divergence loss function is configured to set a stopping gradient for the luminance enhancement image features.

5. The semantic segmentation model construction method according to claim 4, characterized in that, The semantic segmentation model construction method also includes: For the semantic segmentation model, a total loss function is configured, wherein the total loss function is associated with the supervision loss value for photometric enhancement map features only, the loss value for the output position of the backbone network determined based on the L1 norm loss function, and the loss value for the output position of the decoder determined based on the KL divergence loss function.

6. The semantic segmentation model construction method according to claim 5, characterized in that, The total loss function is described as follows: in, This represents the loss value of the entire semantic segmentation model. This represents the loss value determined based on the L1 norm loss function for the output position of the backbone network. This represents the loss value determined based on the KL divergence loss function for the output position of the decoder. and They represent and The corresponding weighting coefficients, This represents the supervised loss value applied only to the features of the photometric enhancement map.

7. The semantic segmentation model construction method according to any one of claims 1 to 6, characterized in that, The anti-light calibration module is located at the first layer output position of the backbone network, and the first photometric alignment module is located at the last layer output position of the backbone network.

8. A welding image processing method, characterized in that, include: Obtain a grayscale image of the welding scene; Obtain a training set of paired images, wherein the paired images include the original image of the welding scene and the corresponding photometric enhancement image; Based on the training set, a semantic segmentation model for welding images is constructed using the semantic segmentation model construction method according to any one of claims 1 to 7; and The welding scene grayscale image is processed by the semantic segmentation model to obtain the final welding segmentation result image.

9. A welding image processing apparatus, characterized in that, include: Image acquisition unit, used to acquire grayscale images of the welding scene; A data pairing construction unit is used to acquire a training set of paired images, wherein the paired images include the original image of the welding scene and the corresponding photometric enhancement image; A model training unit is configured to construct a semantic segmentation model for welding images based on the training set using the semantic segmentation model construction method according to any one of claims 1 to 7; and The image processing unit is used to process the grayscale image of the welding scene through the semantic segmentation model to obtain the final welding segmentation result image.

10. A welding image processing apparatus, characterized in that, include: The memory is configured to store instructions; as well as The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the semantic segmentation model construction method according to any one of claims 1 to 7 or the welding image processing method according to claim 8.

11. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the semantic segmentation model construction method according to any one of claims 1 to 7 or the welding image processing method according to claim 8.

Images