Molten pool image processing method, device and equipment and storage medium

Abstract

The invention provides a molten pool image processing method and device, equipment and a storage medium, and relates to the technical field of image analysis, and the method comprises the steps: firstly carrying out the morphological analysis of each molten pool image of a molten pool image pair, positioning and screening a molten pool region in the molten pool image, thereby remarkably reducing the interference of splashing noisy points in a non-molten pool region; differentiated optimization processing is carried out on the obtained first molten pool image and the second molten pool image after screening processing; and finally, carrying out pixel matching processing on the optimized first molten pool image and the screened second molten pool image to obtain a target disparity map. According to the method, key features in the molten pool image are fully highlighted, image details are enhanced on the premise of no distortion, the molten pool details included in the molten pool image can be effectively and fully extracted, and the accuracy of molten pool information such as the contour and depth of the molten pool in the obtained disparity map is improved.

Description

Technical Field

[0001] The present invention relates to the field of image analysis technology, and in particular to a molten pool image processing method, device, equipment and storage medium. Background Art

[0002] As a temporary state of liquid metal, the surface morphology of the molten pool contains rich information and can often reflect factors related to welding quality such as penetration. Therefore, the restoration of the 3D morphology of the molten pool has far-reaching significance. At present, cameras are usually used to capture the molten pool image pairs during the welding process, and then the molten pool contour and depth are calculated through stereo matching algorithms as an important reference for judging the molten pool penetration state, thereby achieving effective control of welding quality.

[0003] The existing stereo matching methods lack effective processing of melt pool images and cannot effectively and fully extract the melt pool details included in the melt pool images, resulting in low accuracy of melt pool information such as melt pool contour and depth in the obtained disparity map. Summary of the invention

[0004] Embodiments of the present invention provide a melt pool image processing method, device, equipment and storage medium to solve the problem of low accuracy of melt pool information such as melt pool contour and depth in a disparity map.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing a molten pool image, comprising:

[0006] Acquire a molten pool image pair; wherein the molten pool image pair includes a first molten pool image and a second molten pool image, and the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles;

[0007] Based on the gradient information of each pixel in the molten pool image in a preset direction, the molten pool areas in the first molten pool image and the second molten pool image are respectively located, and other shapes except the molten pool area in the molten pool image are eliminated to obtain the first molten pool image after screening and the second molten pool image after screening;

[0008] Optimizing the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image;

[0009] The optimized first molten pool image and the second molten pool image after the screening process are subjected to pixel matching processing to obtain a target disparity map; wherein the target disparity map includes contour information and depth information of the molten pool area.

[0010] In a possible implementation, based on the gradient information of each pixel in the molten pool image in a preset direction, the molten pool area in the first molten pool image is located, and other shapes except the molten pool area in the molten pool image are eliminated, including:

[0011] According to the gradients of the first molten pool image in multiple preset directions, a contour image of the first molten pool image in the corresponding direction is constructed to obtain multiple contour images; wherein the contour image is used to characterize the edge features of the first molten pool image in the corresponding direction;

[0012] Based on the size of the gradient and / or the direction of the gradient, a plurality of contour images are fused to obtain a comprehensive contour image corresponding to the first molten pool image; wherein the comprehensive contour image is used to characterize edge features of the first molten pool image in a plurality of preset directions;

[0013] The integrated contour image is subjected to connected region segmentation processing to obtain a plurality of connected regions, and each connected region is screened based on preset screening rules, and other shapes except the molten pool region in each connected region are eliminated to determine the molten pool region in the first molten pool image.

[0014] In a possible implementation, the performing connected region segmentation processing on the comprehensive contour image to obtain a plurality of connected regions includes:

[0015] Based on the similarity of edge gradients between adjacent pixel points, the comprehensive contour image is divided to obtain a plurality of partitions;

[0016] Based on a preset connectivity rule, each partition is merged to obtain a plurality of connected regions; wherein each connected region includes a plurality of adjacent pixels whose edge gradient similarity is greater than a preset threshold.

[0017] In a possible implementation manner, after determining the molten pool area in the first molten pool image, the method further includes:

[0018] Taking the inter-class variance between the molten pool area and the background area in the first molten pool image as the objective function, optimizing the segmentation threshold of the first molten pool image is calculated to determine the optimal segmentation threshold of the first molten pool image; wherein the optimal segmentation threshold is the grayscale threshold of the first molten pool image when the inter-class variance reaches the maximum value;

[0019] Based on the optimal segmentation threshold, a transition region between the molten pool region and the background region is generated.

[0020] In a possible implementation, the step of optimizing the grayscale distribution of the first molten pool image after the screening process to obtain the optimized first molten pool image includes:

[0021] Partitioning the first molten pool image after the screening process;

[0022] The pixel grayscale value of each pixel in each partition is remapped to obtain an optimized first molten pool image.

[0023] In a possible implementation, pixel matching is performed on the optimized first molten pool image and the second molten pool image after the screening process to obtain a target disparity map, including:

[0024] Performing feature extraction processing on the optimized first molten pool image and the second molten pool image after the screening processing respectively to obtain corresponding first feature maps and second feature maps;

[0025] Based on the first feature map and the second feature map, calculating the matching cost information of each pixel between the optimized first molten pool image and the second molten pool image after the screening process, and obtaining an initial disparity result based on the matching cost information;

[0026] The initial disparity result is optimized and improved to obtain a target disparity map.

[0027] In a possible implementation, the optimizing and improving the initial disparity result to obtain a target disparity map includes:

[0028] Based on a bidirectional matching criterion, incorrectly matched pixel points in the initial disparity result are screened to obtain an improved disparity map;

[0029] Based on the matching cost information corresponding to each pixel point, each pixel point in the improved disparity map is interpolated and optimized to obtain the target disparity map; wherein the matching cost information includes the matching cost of the corresponding pixel point and the local gradient information between the pixel point and other pixel points.

[0030] In a second aspect, an embodiment of the present invention provides a molten pool image processing device, comprising:

[0031] An acquisition unit, configured to acquire a pair of molten pool images; wherein the pair of molten pool images comprises a first molten pool image and a second molten pool image, wherein the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles;

[0032] A first processing unit is used to locate the melt pool areas in the first melt pool image and the second melt pool image respectively based on the gradient information of each pixel in the melt pool image in a preset direction, and to remove other shapes except the melt pool area in the melt pool image, so as to obtain the first melt pool image after screening processing and the second melt pool image after screening processing;

[0033] A second processing unit is used to optimize the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image;

[0034] The third processing unit is used to perform pixel matching processing on the optimized first molten pool image and the second molten pool image after the screening processing to obtain a target disparity map; wherein the target disparity map includes contour information and depth information of the molten pool area.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any possible implementation of the first aspect.

[0037] The embodiment of the present invention provides a method, device, equipment and storage medium for processing a molten pool image. First, a morphological analysis is performed on each molten pool image of a molten pool image pair, and the molten pool area in the molten pool image is located and screened, thereby significantly reducing the interference of splash noise in the non-molten pool area; then, the first molten pool image and the second molten pool image obtained after the screening process are differentially optimized: the grayscale distribution of the first molten pool image after the screening process is optimized to achieve the purpose of improving the local contrast of the image, revealing more image details, and enhancing the clarity of the edge, while the grayscale distribution of the second molten pool image after the screening process is not processed, which helps to maintain the true appearance of the molten pool in the image and reduce the risk of artifacts and image distortion; finally, the optimized first molten pool image and the second molten pool image after the screening process are pixel matched to obtain a target disparity map. The method fully highlights the key features in the molten pool image, enhances the image details without distortion, helps to effectively and fully extract the molten pool details included in the molten pool image, and improves the accuracy of the molten pool information such as the molten pool contour and depth in the obtained disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of the implementation of the molten pool image processing method provided by an embodiment of the present invention;

[0040] Figure 2 is a flowchart of another method for processing a molten pool image provided by an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of a molten pool area positioning effect provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the effects before and after optimization provided by an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of the effect before and after comprehensive optimization provided by an embodiment of the present invention;

[0044] Figure 6 is a schematic structural diagram of a molten pool image processing device provided by an embodiment of the present invention;

[0045] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention. SUMMARY OF THE INVENTION

[0048] With the improvement of automation level, welding robots gradually replace manual welding, and the requirements for welding quality monitoring are more stringent. Welding quality control is mainly achieved by controlling the size of the weld during the weld formation process, and the welding depth or penetration in the weld size is the most important factor related to the strength and related performance of the welded product. During the welding process, the molten pool is a temporary state of liquid metal, and its surface morphology contains rich information, which can often reflect factors related to welding quality such as penetration. Therefore, the restoration of the three-dimensional morphology of the molten pool has far-reaching significance for the realization of remote control automatic welding. Experienced welders ensure the quality by observing the three-dimensional shape of the molten pool to judge the penetration state. In order to simulate this ability, binocular stereo vision is introduced into the welding process. By using an industrial camera to capture the molten pool image pair during the welding process, the molten pool contour and depth are calculated by the stereo matching algorithm as an important reference for judging the molten pool penetration state, thereby achieving effective control of welding quality.

[0049] After research, the inventors found that the existing image stereo matching method lacks effective processing of the melt pool image and cannot effectively and fully extract the melt pool details included in the melt pool image, resulting in low accuracy of melt pool information such as the melt pool contour and depth in the obtained disparity map.

[0050] In order to improve the accuracy of the disparity map, in the implementation of the present application, a morphological analysis is first performed on each melt pool image of the melt pool image pair, and the melt pool area in the melt pool image is located and screened, thereby significantly reducing the interference of splash noise in the non-melt pool area; then, the first melt pool image and the second melt pool image obtained after the screening process are differentially optimized; finally, the optimized first melt pool image and the second melt pool image after the screening process are pixel matched to obtain the target disparity map. This method fully highlights the key features in the melt pool image, enhances the image details without distortion, helps to effectively and fully extract the melt pool details included in the melt pool image, and improves the accuracy of the melt pool information such as the melt pool contour and depth in the obtained disparity map.

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0052] Figure 1 The implementation flow chart of the molten pool image processing method provided by the embodiment of the present invention is detailed as follows:

[0053] Step 101 , obtaining a molten pool image pair; wherein the molten pool image pair includes a first molten pool image and a second molten pool image, and the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles.

[0054] Exemplarily, this embodiment acquires a pair of molten pool images, including two molten pool images, a first molten pool image and a second molten pool image, wherein the first molten pool image and the second molten pool image are molten pool images observed from different viewing angles.

[0055] In a feasible implementation manner, this embodiment obtains a pair of molten pool images through a pre-set shooting device. For example, the industrial camera is symmetrically set on the left and right sides of the molten pool. The first molten pool image is the left molten pool image, and the second molten pool image is the right molten pool image.

[0056] Step 102, based on the gradient information of each pixel in the melt pool image in a preset direction, the melt pool areas in the first melt pool image and the second melt pool image are located respectively, and other shapes except the melt pool area in the melt pool image are eliminated to obtain the first melt pool image after screening and the second melt pool image after screening.

[0057] Exemplarily, this embodiment extracts and identifies the contours of the first molten pool image and the second molten pool image based on the gradient information of each pixel in the molten pool image in a preset direction, realizes the positioning of the molten pool area in the image, and eliminates other shapes in the molten pool image except the molten pool area, such as shapes and areas that do not conform to the shape or size of the molten pool, such as spatter points, to obtain the first molten pool image after screening and the second molten pool image after screening. Both the first molten pool image after screening and the second molten pool image after screening only include the molten pool area.

[0058] Step 103, optimizing the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image.

[0059] Exemplarily, this embodiment performs differentiated image optimization processing on the first molten pool image and the second molten pool image after the screening process. For example, this embodiment selects a pair of molten pool images after positioning and screening processes to optimize the grayscale distribution, and does not perform additional processing on the other molten pool image after the screening process.

[0060] Step 104 , performing pixel matching processing on the optimized first molten pool image and the second molten pool image after screening processing to obtain a target disparity map; wherein the target disparity map includes contour information and depth information of the molten pool area.

[0061] Exemplarily, this embodiment performs feature extraction, matching cost calculation, disparity calculation, etc. based on the optimized first melt pool image and the second melt pool image after screening processing, and performs pixel matching processing on the optimized first melt pool image and the second melt pool image after screening processing to obtain an initial disparity result. On this basis, this embodiment can also optimize and improve the initial disparity result to obtain a target disparity map.

[0062] In one example, this embodiment uses geometric relationships to ensure the consistency of disparity matching between left and right images, thereby eliminating mismatching points; based on Gaussian weighted sub-pixel refinement technology, more refined interpolation calculations are performed between integer disparity values ​​to capture subtle depth changes, making the depth transition of the disparity map smoother, and obtaining a target disparity map.

[0063] In summary, the embodiment of the present invention provides a melt pool image processing method, device, equipment and storage medium. First, a morphological analysis is performed on each melt pool image of a melt pool image pair, and the melt pool area in the melt pool image is located and screened, thereby significantly reducing the interference of splash noise in the non-melt pool area; then, the first melt pool image and the second melt pool image obtained after the screening process are differentially optimized: the grayscale distribution of the first melt pool image after the screening process is optimized to achieve the purpose of improving the local contrast of the image, revealing more image details, and enhancing the clarity of the edge. At the same time, the grayscale distribution of the second melt pool image after the screening process is not processed, which helps to maintain the true appearance of the melt pool in the image and reduce the risk of artifacts and image distortion; finally, the optimized first melt pool image and the second melt pool image after the screening process are pixel matched to obtain a target disparity map. The method fully highlights the key features in the melt pool image, enhances the image details without distortion, helps to effectively and fully extract the melt pool details included in the melt pool image, and improves the accuracy of the melt pool information such as the melt pool contour and depth in the obtained disparity map.

[0064] In order to better highlight the key features and enhance the details in the molten pool image, this embodiment extracts the corresponding directional contour images based on the gradient information of each pixel in multiple preset directions. These contour images show the edge features of the image at different angles. In order to obtain more comprehensive edge information, these multi-directional contour images are weighted fused; further, this embodiment realizes the positioning and denoising of the molten pool area based on the connected region analysis technology, and further optimizes the image features through differential optimization processing, thereby improving the edge clarity while maintaining the realism of the molten pool. In addition, by optimizing the initial disparity results, the consistency of the disparity matching of the left and right images is ensured, and the non-matching points are eliminated. Based on the sub-pixel refinement technology, more sophisticated interpolation calculations are performed to capture subtle depth changes, so that the depth transition of the target disparity map is smoother.

[0065] Figure 2 Another implementation flow chart of a melt pool image processing method provided by an embodiment of the present invention is described in detail as follows:

[0066] Step 201 , obtaining a molten pool image pair; wherein the molten pool image pair includes a first molten pool image and a second molten pool image, and the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles.

[0067] Exemplarily, this step refers to step 101 and will not be described in detail.

[0068] Step 202, based on the gradient information of each pixel in the melt pool image in a preset direction, the melt pool areas in the first melt pool image and the second melt pool image are located respectively, and other shapes except the melt pool area in the melt pool image are eliminated to obtain the first melt pool image after screening and the second melt pool image after screening.

[0069] Exemplarily, taking the first molten pool image after screening as an example, step 202 includes the following steps:

[0070] According to the gradient of the first molten pool image in multiple preset directions, a contour image of the first molten pool image in the corresponding direction is constructed to obtain multiple contour images; wherein the contour image is used to characterize the edge features of the first molten pool image in the corresponding direction.

[0071] Based on the size of the gradient and / or the direction of the gradient, multiple contour images are fused to obtain a comprehensive contour image corresponding to the first molten pool image; wherein the comprehensive contour image is used to characterize the edge features of the first molten pool image in multiple preset directions.

[0072] The integrated contour image is segmented into connected regions to obtain multiple connected regions, and each connected region is screened based on preset screening rules to eliminate shapes other than the molten pool region in each connected region, thereby determining the molten pool region in the first molten pool image.

[0073] Exemplarily, this embodiment calculates the gradient of each pixel in the molten pool image based on the Sobel operator. In order to fully extract edge information, not only the gradients in the horizontal direction (x-axis) and the vertical direction (y-axis) are calculated, but also can be extended to 16 different directions. The directions include from 0 degrees to 360 degrees, and the gradient is calculated for each pixel with a step size of 22.5 degrees. The gradient in each direction reflects the change in the grayscale value of the image at a specific angle, thereby capturing the subtle differences in the edges of objects at different angles.

[0074] After obtaining the gradient information of each pixel in different directions, this embodiment sets a threshold value, which is used to determine whether a pixel is an edge pixel. For example, if the gradient value of a pixel in a certain direction is greater than the threshold value, the pixel is considered to be an edge pixel in this direction; after determining whether each pixel is an edge pixel in each direction, the contour image of each direction is constructed: for each specific direction, all points determined to be edge pixels in this direction are combined together to form a contour image in this direction.

[0075] In order to obtain more comprehensive edge information, this embodiment performs weighted fusion on these multi-directional contour images. Specifically, weights are assigned to the contour information in each direction, and these weights can be dynamically adjusted based on gradient amplitude, gradient direction consistency, or other specific features. By fusing edges in different directions, the network obtains a comprehensive contour image, and extracts corresponding directional contour images based on the gradient information of each pixel in different directions. These contour images show the edge features of the image at different angles.

[0076] In one example, this embodiment divides the comprehensive contour image based on the similarity of edge gradients between adjacent pixel points to obtain multiple partitions; based on preset connectivity rules, the partitions are merged to obtain multiple connected areas; wherein each connected area includes multiple adjacent pixels and the similarity between the edge gradients is greater than a preset threshold.

[0077] In a feasible implementation manner, the method further includes: taking the inter-class variance between the melt pool area and the background area in the first melt pool image as the objective function, optimizing the segmentation threshold of the first melt pool image, and determining the optimal segmentation threshold of the first melt pool image; wherein the optimal segmentation threshold is the grayscale threshold of the first melt pool image when the inter-class variance reaches the maximum value; based on the optimal segmentation threshold, generating a transition area between the melt pool area and the background area.

[0078] In order to further reduce noise interference and enhance the focus on the molten pool area, this embodiment performs connected region segmentation on the first molten pool image after screening: first, the adjacent pixel points are divided into several regions according to the edge gradient similarity. Based on these divided regions, the image is decomposed into multiple independent connected regions using the connected region analysis technology. Each connected region is composed of adjacent pixels with similar features.

[0079] In one example, the core of connected region analysis is to identify and label these regions. By analyzing the shape, size, contour and other characteristics of the region, the regions that meet specific attributes are identified as potential molten pool regions. The molten pool usually has a relatively regular shape and a certain size distribution, so those regions that are approximately circular or elliptical and meet the molten pool size range can be locked as candidate molten pool regions. By combining the shape characteristics and edge information of the molten pool, the connected regions are screened based on preset screening rules. Through shape constraints and size restrictions, the regions that do not meet the shape or size of the molten pool are excluded, thereby significantly reducing the interference of spatter noise points in non-molten pool areas.

[0080] In one example, this embodiment divides the comprehensive contour image by calculating the similarity of edge gradients between adjacent pixels. Pixels with similarity higher than a preset threshold belong to the same manure, thereby obtaining multiple partitions. Based on the preset connectivity rules, each partition is merged to obtain multiple connected areas. Among them, the edge gradient can be calculated by the Sobel operator, and the similarity of the edge gradient can be measured by calculating the absolute value of the difference in the size of the edge gradient. The smaller the difference, the more similar the two pixels are in terms of the size of the edge gradient. A normalized method can also be used to divide the absolute value of the difference between the two by the maximum value between the two, and the resulting similarity value range is between 0 and 1, with 0 indicating complete similarity and 1 indicating the maximum difference.

[0081] In one example, Figure 3 A schematic diagram of the molten pool area positioning effect provided by an embodiment of the present invention. Figure 3 As shown, Figure 3 The image on the left is extracted through contour extraction to obtain the image in the middle, and finally the image on the right is obtained through screening and positioning. The molten pool image after screening removes the spatter noise that may cause interference.

[0082] In a feasible implementation, in order to ensure the accuracy of stereo matching, this embodiment adds a certain pixel width to the edge of the molten pool area as a protection zone, i.e., a transition zone. Based on the positioning result of the molten pool, this embodiment finds the optimal segmentation threshold by maximizing the inter-class variance between the foreground, i.e., the molten pool area, and the background, and generates a molten pool area protection layer.

[0083] In stereo matching of images, parallax mutations often occur at the edges, especially in high-temperature areas such as the molten pool. Due to the effects of thermal radiation and light reflection, the edges may appear blurred or unstable. The setting of the protection band provides a smooth transition area, reducing the impact of such mutations. In this way, the molten pool will not be misjudged as the background even near the edge. In this process, the molten pool area is defined as the valid area, while the area outside the molten pool is defined as the invalid area. In this way, not only the key information of the molten pool is retained, but also the surrounding elements that may cause interference are excluded, thereby improving the accuracy and reliability of stereo matching.

[0084] Step 203 , optimizing the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image.

[0085] In one example, step 203 includes:

[0086] The first molten pool image after screening is partitioned; the pixel gray value of each pixel point in each partition is remapped to obtain an optimized first molten pool image.

[0087] Exemplarily, this embodiment selects a pair of grayscale distributions of the molten pool image after the screening process to be optimized. Taking the first molten pool image after the screening process as an example, this embodiment first divides the first molten pool image after the screening process into multiple partitions, and performs histogram equalization on each partition independently, so as to optimize the grayscale distribution of the image and obtain the optimized first molten pool image.

[0088] In one example, this embodiment first performs nonlinear transformation on the pixel values ​​of each partition to expand the range of pixel values, thereby enhancing the contrast of the image. Then, adaptive pixel cropping is performed based on the pixel ratio of the histogram. In this process, n is defined as the total number of pixels in the image, and n' represents the total number of pixels after cropping. For the kth gray level in the image, the original number of pixels is n', and the number of pixels after cropping is n' k , L-1 is the gray level threshold of the image. In addition, an adaptive parameter α is introduced, which represents the ratio of cropped pixels to the total number of pixels. Through this parameter, we can calculate the cropping coefficient s as follows:

[0089]

[0090] The shear coefficient is calculated through the histogram, and the original histogram is sheared to obtain a new histogram. The grayscale cumulative distribution function is constructed according to the grayscale cumulative distribution function of the new grayscale histogram, and then a grayscale transformation table is generated to remap the grayscale value of the pixel to the image.

[0091] This method can improve the local contrast of the image in a targeted manner, reveal more image details, and enhance the clarity of edges. In turn, it can extract more discriminative and detailed features from the optimized image, thereby more accurately capturing the subtle changes and details in the image.

[0092] In one example, Figure 4 The following is a schematic diagram of the effects before and after optimization provided by an embodiment of the present invention. Figure 4 As shown, Figure 4 Right side compared to Figure 4 The local contrast on the left side has been greatly improved.

[0093] Step 204 , performing feature extraction processing on the optimized first molten pool image and the second molten pool image after screening processing, respectively, to obtain corresponding first feature maps and second feature maps.

[0094] Exemplarily, this embodiment performs feature extraction processing on the optimized first melt pool image and the second melt pool image after screening processing, respectively. The melt pool image of size H×W first passes through three convolutional layers, using 3x3 convolution kernels and step sizes of (2, 1, 1) to perform preliminary downsampling and feature extraction of the image. Then, this embodiment constructs four residual blocks through the make_layer method. These residual blocks use different numbers of channels (32, 64, 128, 128) and different numbers of residual layers (16, 16, 3, 3). They gradually extract higher-level features by increasing the number of channels and adjusting the step size of the convolution layer to generate feature maps (l1, l2, l3) of different resolutions. The feature maps l2, l3, and l4 are spliced ​​in the channel dimension through the tensor splicing (torch.cat) function to form a comprehensive feature representation. Finally, the unary feature maps of the first molten pool image and the second molten pool image are obtained from the feature extraction, denoted as fl and fr, with a size of Nc×H / 4×W / 4 (Nc=32), completing the preliminary feature extraction of the molten pool.

[0095] Step 205 , based on the first feature map and the second feature map, calculate the matching cost information of each pixel between the optimized first molten pool image and the second molten pool image after the screening process, and obtain the initial disparity result based on the matching cost information.

[0096] Exemplarily, this embodiment obtains feature maps of three different levels (l1, l2 and l3) from the feature extraction module, and for each pixel at each level, a hole convolution block (atrouspatch) with a predefined size and adaptive learning weight is used to calculate the matching cost. The grouping idea is adopted to divide the features into groups and calculate the correlation map group by group. Two 3D convolutions and one 3D hourglass network (hourglass) are used for regularization, and then another convolution layer is used to compress the number of channels to 1 to obtain the attention weight A. Among them, the introduction of attention weight A helps to eliminate redundant information in the first stage, thereby improving the accuracy of matching. Further, four batch normalizations (Batch Normalization) and three-dimensional convolution layers with activation functions (ReLU) are used to enhance and normalize the features, and then two stacked 3D hourglass networks are used to further process the data through repeated encoding and decoding processes to extract deeper features. The three cost volumes generated are converted into probability volumes through upsampling and normalized exponential (softmax) functions. Finally, the predicted value, i.e. the initial disparity value, is calculated by soft minimizing the index (soft argmin) function, the attention weight and the feature volume.

[0097] Step 206: Optimize and improve the initial disparity result to obtain a target disparity map.

[0098] In one example, step 206 includes:

[0099] Based on the bidirectional matching criterion, the incorrectly matched pixels in the initial disparity result are filtered out to obtain an improved disparity map.

[0100] Based on the matching cost information corresponding to each pixel point, each pixel point in the improved disparity map is interpolated and optimized to obtain a target disparity map; wherein the matching cost information includes the matching cost of the corresponding pixel point and the local gradient information between the pixel point and other pixel points.

[0101] Exemplarily, in order to obtain more accurate and stable disparity results, this embodiment adopts a multi-stage comprehensive optimization strategy to improve the initial disparity results in all aspects. First, a cost array is constructed for each pixel in the first molten pool image, which stores the matching cost or similarity score with each possible matching position in the second molten pool image. Using the cost array information of the first molten pool image, the possible matching position of each pixel in the second molten pool image and its corresponding matching cost are reconstructed according to the geometric relationship of the disparity calculation. Based on the bidirectional matching criterion, it is required that each pixel in the first molten pool image has only one unique matching point in the second molten pool image. Through this consistency check, the disparity map of the image is bidirectionally matched. Specifically, the pixel matching of the second molten pool image is first predicted by the disparity map of the first molten pool image, and then the disparity map of the second molten pool image is reversely verified with the disparity map of the first molten pool image. For those pixels with inconsistent disparity values ​​or large errors, they are regarded as wrong matches and are gradually eliminated in an iterative manner, thereby effectively removing noise and artifacts. Thereby improving the accuracy and reliability of stereo matching.

[0102] On the basis of improving the disparity map, the key step to further improve the accuracy is to optimize the interpolation at the sub-pixel level, as shown below:

[0103]

[0104] Among them, d sub represents the disparity value after sub-pixel interpolation optimization, d p Representing the disparity values ​​at the integer level, c0, c1, and c2 are elements in the matching cost array.

[0105] Among them, this embodiment first constructs a matching cost array for each pixel again, and the array records the matching costs between each possible matching position in the second molten pool image. It lays the foundation for sub-pixel-level refined interpolation. Specifically, the cost array not only contains the matching cost corresponding to the integer disparity value, but also considers the local gradient between pixels, thereby providing richer information for interpolation. In the cost processing process, a weighting function based on Gaussian distribution is used to perform weighted average processing on the matching cost of each pixel and its neighborhood. The introduction of Gaussian weights makes the points closer to the center pixel contribute more to the cost. This local weighting method can better retain the local spatial information of the pixel and reduce matching errors caused by texture repetition or illumination changes. In this weighted process, Gaussian weights are not only used to calculate the weighted average of the matching cost, but also to estimate the spatial change rate of the disparity value. By fitting the cost curve of each pixel and its surrounding pixels, the smooth change trend of the disparity value is obtained, which provides an accurate reference for subsequent interpolation calculations. Sub-pixel interpolation allows disparity values ​​to smoothly transition between integers, rather than being limited to integer levels, thereby achieving a more refined representation of depth changes. In order to ensure the smoothness and accuracy of the depth transition, not only the cost of a single pixel should be considered during the interpolation process, but also the cost distribution of the surrounding pixels. The introduction of this spatial contextual information enables the algorithm to more accurately capture subtle depth changes in the image, especially in areas with complex edge details and surface textures. Through sub-pixel interpolation, the disparity map can more realistically reflect the three-dimensional structure of the scene. In order to prevent the influence of noise on the interpolation results, the algorithm also combines a robust filtering method to suppress outliers in the cost distribution. In this way, even in areas with weak image texture or strong noise, the smooth change of the matching cost can be ensured to avoid sudden changes or discontinuities in the disparity value.

[0106] In one example, Figure 5 The following is a schematic diagram of the effects before and after comprehensive optimization provided by an embodiment of the present invention. Figure 5 As shown, compared with the disparity map before optimization on the left, the optimized disparity map on the right is smoother and more continuous in depth details, the depth transition is more natural, and the disparity matching is more accurate.

[0107] Through comprehensive optimization processing, the target disparity map generated in the end not only ensures the consistency and accuracy of disparity matching compared to the initial disparity result, but is also smoother and more continuous in depth details, and the depth transition is more natural. Especially for scenes with complex textures and subtle depth changes such as the molten pool surface, this optimization method can significantly reduce the error in disparity calculation, improve the continuity and smoothness of the disparity map, and provide a more reliable data basis for subsequent 3D reconstruction and analysis.

[0108] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0109] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0110] Figure 6 The schematic diagram of the structure of the molten pool image processing device provided by the embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0111] like Figure 6 As shown, the molten pool image processing device 600 includes:

[0112] The acquisition unit 601 is used to acquire a pair of molten pool images, wherein the pair of molten pool images includes a first molten pool image and a second molten pool image, and the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles.

[0113] The first processing unit 602 is used to locate the melt pool areas in the first melt pool image and the second melt pool image respectively based on the gradient information of each pixel in the melt pool image in a preset direction, and to eliminate other shapes except the melt pool area in the melt pool image, so as to obtain the first melt pool image after screening and the second melt pool image after screening.

[0114] The second processing unit 603 is used to optimize the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image.

[0115] The third processing unit 604 is used to perform pixel matching processing on the optimized first molten pool image and the second molten pool image after screening processing to obtain a target disparity map, wherein the target disparity map includes contour information and depth information of the molten pool area.

[0116] In a possible implementation, the first processing unit 602 is specifically configured to:

[0117] According to the gradient of the first molten pool image in multiple preset directions, a contour image of the first molten pool image in the corresponding direction is constructed to obtain multiple contour images, wherein the contour image is used to characterize the edge features of the first molten pool image in the corresponding direction.

[0118] Based on the magnitude and / or direction of the gradient, a plurality of contour images are fused to obtain a comprehensive contour image corresponding to the first molten pool image, wherein the comprehensive contour image is used to characterize edge features of the first molten pool image in a plurality of preset directions.

[0119] The integrated contour image is segmented into connected regions to obtain multiple connected regions, and each connected region is screened based on preset screening rules to eliminate shapes other than the molten pool region in each connected region, thereby determining the molten pool region in the first molten pool image.

[0120] In a possible implementation, the first processing unit 602 is further configured to:

[0121] Based on the similarity of edge gradients between adjacent pixels, the comprehensive contour image is divided to obtain multiple partitions.

[0122] Based on the preset connectivity rule, each partition is merged to obtain multiple connected regions, wherein each connected region includes multiple adjacent pixels whose edge gradient similarity is greater than a preset threshold.

[0123] In a possible implementation, after the first processing unit 602, the molten pool image processing device 600 further includes: a fourth processing unit, specifically configured to:

[0124] Taking the inter-class variance between the molten pool area and the background area in the first molten pool image as the objective function, the segmentation threshold of the first molten pool image is optimized and calculated to determine the optimal segmentation threshold of the first molten pool image. The optimal segmentation threshold is the grayscale threshold of the first molten pool image when the inter-class variance reaches the maximum value.

[0125] Based on the optimal segmentation threshold, a transition area between the molten pool area and the background area is generated.

[0126] In a possible implementation, the second processing unit 603 is specifically configured to:

[0127] The first molten pool image after the screening process is partitioned.

[0128] The pixel grayscale value of each pixel in each partition is remapped to obtain an optimized first molten pool image.

[0129] In a possible implementation, the third processing unit 604 is specifically configured to:

[0130] Feature extraction processing is performed on the optimized first molten pool image and the second molten pool image after screening processing to obtain corresponding first feature maps and second feature maps.

[0131] Based on the first feature map and the second feature map, the matching cost information of each pixel between the optimized first molten pool image and the second molten pool image after the screening process is calculated, and based on the matching cost information, an initial disparity result is obtained.

[0132] The initial disparity result is optimized and improved to obtain the target disparity map.

[0133] In a possible implementation, the third processing unit 604 is further configured to:

[0134] Based on the bidirectional matching criterion, the incorrectly matched pixels in the initial disparity result are filtered out to obtain an improved disparity map.

[0135] Based on the matching cost information corresponding to each pixel, each pixel in the improved disparity map is interpolated and optimized to obtain the target disparity map, wherein the matching cost information includes the matching cost of the corresponding pixel and the local gradient information between the pixel and other pixels.

[0136] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 70 and a memory 71. The memory 71 is used to store a computer program 72, and the processor 70 is used to call and run the computer program 72 stored in the memory 71 to perform the steps in the above-mentioned various embodiments of the molten pool image processing method, such as Figure 1 Alternatively, the processor 70 is used to call and run the computer program 72 stored in the memory 71 to implement the functions of each module / unit in the above-mentioned device embodiments, such as Figure 6 Functionality of modules / units 601 to 604 is shown.

[0138] Exemplarily, the computer program 72 may be divided into one or more modules / units, one or more modules / units are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7. For example, the computer program 72 may be divided into Figure 6 Modules / units 601 to 604 are shown.

[0139] The electronic device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0140] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0141] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 71 may also include both an internal storage unit of the electronic device 7 and an external storage device. The memory 71 is used to store computer programs and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0142] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0149] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for processing a molten pool image, characterized in that: The method comprises: Acquire a molten pool image pair; wherein the molten pool image pair includes a first molten pool image and a second molten pool image, and the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles; Based on the gradient information of each pixel in the molten pool image in a preset direction, the molten pool areas in the first molten pool image and the second molten pool image are respectively located, and other shapes except the molten pool area in the molten pool image are eliminated to obtain the first molten pool image after screening and the second molten pool image after screening; Optimizing the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image; The optimized first molten pool image and the second molten pool image after the screening process are subjected to pixel matching processing to obtain a target disparity map; wherein the target disparity map includes contour information and depth information of the molten pool area.

2. The method for processing a molten pool image according to claim 1, characterized in that: Based on the gradient information of each pixel in the molten pool image in a preset direction, the molten pool area in the first molten pool image is located, and other shapes except the molten pool area in the molten pool image are eliminated, including: According to the gradients of the first molten pool image in multiple preset directions, a contour image of the first molten pool image in the corresponding direction is constructed to obtain multiple contour images; wherein the contour image is used to characterize the edge features of the first molten pool image in the corresponding direction; Based on the size of the gradient and / or the direction of the gradient, a plurality of contour images are fused to obtain a comprehensive contour image corresponding to the first molten pool image; wherein the comprehensive contour image is used to characterize edge features of the first molten pool image in a plurality of preset directions; The integrated contour image is subjected to connected region segmentation processing to obtain a plurality of connected regions, and each connected region is screened based on preset screening rules, and other shapes except the molten pool region in each connected region are eliminated to determine the molten pool region in the first molten pool image.

3. The method for processing a molten pool image according to claim 2, characterized in that: The connected region segmentation process is performed on the comprehensive contour image to obtain a plurality of connected regions, including: Based on the similarity of edge gradients between adjacent pixel points, the comprehensive contour image is divided to obtain a plurality of partitions; Based on a preset connectivity rule, each partition is merged to obtain a plurality of connected regions; wherein each connected region includes a plurality of adjacent pixels whose edge gradient similarity is greater than a preset threshold.

4. The method for processing a molten pool image according to claim 2, characterized in that: After determining the molten pool area in the first molten pool image, the method further includes: Taking the inter-class variance between the molten pool area and the background area in the first molten pool image as the objective function, optimizing the segmentation threshold of the first molten pool image is calculated to determine the optimal segmentation threshold of the first molten pool image; wherein the optimal segmentation threshold is the grayscale threshold of the first molten pool image when the inter-class variance reaches the maximum value; Based on the optimal segmentation threshold, a transition region between the molten pool region and the background region is generated.

5. The method for processing a molten pool image according to any one of claims 1 to 4, characterized in that: The step of optimizing the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image comprises: Performing partition processing on the first molten pool image after the screening process; The pixel grayscale value of each pixel in each partition is remapped to obtain an optimized first molten pool image.

6. The method for processing a molten pool image according to any one of claims 1 to 4, characterized in that: Performing pixel matching processing on the optimized first molten pool image and the second molten pool image after the screening processing to obtain a target disparity map, including: Performing feature extraction processing on the optimized first molten pool image and the second molten pool image after the screening processing respectively to obtain corresponding first feature maps and second feature maps; Based on the first feature map and the second feature map, calculating the matching cost information of each pixel between the optimized first molten pool image and the second molten pool image after the screening process, and obtaining an initial disparity result based on the matching cost information; The initial disparity result is optimized and improved to obtain a target disparity map.

7. The method for processing a molten pool image according to claim 6, characterized in that: The step of optimizing and improving the initial disparity result to obtain a target disparity map includes: Based on a bidirectional matching criterion, incorrectly matched pixel points in the initial disparity result are screened to obtain an improved disparity map; Based on the matching cost information corresponding to each pixel point, each pixel point in the improved disparity map is interpolated and optimized to obtain the target disparity map; wherein the matching cost information includes the matching cost of the corresponding pixel point and the local gradient information between the pixel point and other pixel points.

8. A molten pool image processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a pair of molten pool images; wherein the pair of molten pool images comprises a first molten pool image and a second molten pool image, wherein the first molten pool image and the second molten pool image are molten pool images observed at different viewing angles; A first processing unit is used to locate the melt pool areas in the first melt pool image and the second melt pool image respectively based on the gradient information of each pixel in the melt pool image in a preset direction, and to remove other shapes except the melt pool area in the melt pool image, so as to obtain the first melt pool image after screening processing and the second melt pool image after screening processing; A second processing unit is used to optimize the grayscale distribution of the first molten pool image after the screening process to obtain an optimized first molten pool image; The third processing unit is used to perform pixel matching processing on the optimized first molten pool image and the second molten pool image after the screening processing to obtain a target disparity map; wherein the target disparity map includes contour information and depth information of the molten pool area.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the molten pool image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of molten pool image processing as described in any one of claims 1 to 7 are implemented.

Images