Three-dimensional reconstruction method for forging surface defects

Abstract

The invention relates to a forge piece surface defect three-dimensional reconstruction method, and belongs to the technical field of forge piece defect detection.The method comprises the steps that semantic segmentation is conducted on defects in an original binocular image of the forge piece surface, and a binocular image after defect segmentation is obtained; fusing the original binocular image and the binocular image after defect segmentation to obtain a fused binocular image; performing row alignment on each row of pixels of the fused binocular image, searching a matching point pair with the minimum matching cost in the fused binocular image subjected to row alignment, determining image parallax of the fused binocular image according to the matching point pair, and calculating image depth information according to the image parallax; and performing three-dimensional reconstruction on the forge piece and the forge piece surface defects according to the image depth information. According to the method, the reconstruction effect is good, the defect can be clearly recognized in the reconstruction image, the three-dimensional space position of the defect and the size of the defect relative to the forged piece can be obtained, and the good visualization effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of forging defect detection, and in particular to a three-dimensional reconstruction method for forging surface defects. Background Art

[0002] During the manufacturing process of steel forgings, fluctuating process conditions can easily lead to defects such as cracks, folds, and slag inclusions, seriously affecting forging quality. The national standard "GB / T 12361-2016 General Technical Requirements for Steel Die Forgings" stipulates that all specified inspection items must undergo 100% inspection using non-destructive testing methods.

[0003] Currently, surface defect detection in steel forging production sites primarily relies on a combination of fluorescent magnetic particle development and manual visual inspection. This method suffers from shortcomings such as low efficiency, poor working environment, and significant subjective influence on test results. With the rapid development of machine vision technology, intelligent defect detection technology is gaining increasing application in the industrial sector. In the field of large-scale and widespread steel forging defect detection, intelligent forging defect detection based on machine vision has achieved certain technological breakthroughs and is gradually being applied in production sites.

[0004] Among them, the patent with patent number 202111390169.9 discloses a method and system for identifying and reconstructing magnetic traces of steel die forgings. The method first determines the depth information of the steel die forging image, then segments the magnetic traces of the steel die forging image through a semantic segmentation model, and finally reconstructs the three-dimensional coordinates of the magnetic traces in the camera coordinate system based on the segmented magnetic trace image and the depth information of the steel die forging image. Although this method obtains the three-dimensional coordinates of the surface defects of the forgings, the defect reconstruction effect of this method is not good due to the small size of the surface defects of the forgings and the large error in the depth information obtained by stereo matching. Moreover, only reconstructing the defects cannot quantify the size of the defects through the reconstructed point cloud, which makes the visualization effect of the surface defects of the forgings poor. Summary of the Invention

[0005] In view of this, it is necessary to provide a three-dimensional reconstruction method for forging surface defects to solve the problem that the existing technology has poor effect on the three-dimensional reconstruction of forging surface defects.

[0006] In order to solve the above problems, the present invention provides a three-dimensional reconstruction method for forging surface defects, comprising: Perform semantic segmentation on the defects in the original binocular image of the forging surface to obtain a binocular image after defect segmentation; Fusing the original binocular image with the binocular image after defect segmentation to obtain a fused binocular image; Aligning each row of pixels of the fused binocular image, searching for a matching point pair with a minimum matching cost in the aligned fused binocular image, determining an image disparity of the fused binocular image based on the matching point pair, and calculating image depth information based on the image disparity; The forging and the surface defects of the forging are three-dimensionally reconstructed according to the image depth information.

[0007] In a possible implementation, the method further includes: The surface of the forging that has been magnetized and sprayed with magnetic suspension liquid is photographed under ultraviolet light to obtain an original binocular image of the forging surface.

[0008] In a possible implementation, the U 2 -Net semantic segmentation model performs semantic segmentation on defects in the original binocular image of the forging surface.

[0009] In a possible implementation, the calculation formula of the matching cost is as follows:

[0010]

[0011]

[0012] in, represents the aggregate matching cost on each path, Indicates the direction of the path r Current pixel p In Parallax d The cumulative matching cost of Indicates the current pixel p In Parallax d The initial matching cost, Indicates that the previous pixel is at the same disparity d The cumulative matching cost of Indicates the previous pixel in the disparity The cumulative matching cost of Represents the previous pixel in all possible disparities i The minimum cumulative matching cost in Represents the previous pixel in all possible disparities k The minimum cumulative matching cost in 、 Indicates the smoothing penalty when the disparity between the current pixel and the adjacent pixels is small or large. ; Indicates the current pixel p The pixel coordinates in one of the binocular fused views, Indicates the current pixel p The pixel coordinates in the other binocular fused view.

[0013] In a possible implementation, calculating image depth information according to the image disparity includes: performing sub-pixel level interpolation processing on the image disparity according to the disparity gradient of the image disparity; The image depth information is calculated based on the interpolated image disparity.

[0014] In a possible implementation, performing three-dimensional reconstruction of the forging and the surface defects of the forging according to the image depth information includes: Determining the three-dimensional coordinates of each pixel in the binocular fused image in a camera coordinate system according to the image depth information to generate a three-dimensional point cloud; The three-dimensional point cloud is segmented by a point cloud segmentation algorithm to obtain target point clouds corresponding to the forging and the surface defects of the forging.

[0015] In a possible implementation, the method further includes: In the three-dimensional point cloud, the defective portion is displayed in a color different from the color of the forging.

[0016] The beneficial effects of the present invention are: The present invention semantically segments defects in an original binocular image of a forging surface to obtain a defect-segmented binocular image. The original binocular image and the defect-segmented binocular image are then fused to obtain a fused binocular image, which can make the defects in the original binocular image more visible. Rows of pixels in the fused binocular image are then aligned, and matching point pairs with the minimum matching cost are searched in the aligned fused binocular image, preventing target (defect) loss during the matching process. The image disparity of the fused binocular image is then determined based on the matching point pairs, and image depth information is calculated based on the image disparity. Finally, a three-dimensional reconstruction of the forging and its surface defects is performed based on this image depth information. The reconstruction is effective, with defects clearly identifiable in the reconstructed image, and the three-dimensional spatial position of the defect and its size relative to the forging are obtained, providing excellent visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1A schematic flow chart of an embodiment of the method for three-dimensional reconstruction of forging surface defects provided by the present invention; Figure 2 A schematic diagram of the structure of the defect segmentation model provided by the present invention; Figure 3 A schematic diagram of stereo matching of fused binocular images provided by the present invention; Figure 4 This is a schematic diagram of the three-dimensional reconstruction of forging surface defects provided by the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "plurality" is two or more. The "first", "second", etc. involved in the embodiments of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence, nor are they used to indicate or imply their relative importance or implicitly indicate the number of technical features indicated. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more.

[0021] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] Reference Figure 1 , which shows a flow chart of an embodiment of a method for three-dimensional reconstruction of forging surface defects provided by the present invention, the method comprising: S101, performing semantic segmentation on defects in the original binocular image of the forging surface to obtain a binocular image after defect segmentation.

[0023] Two industrial cameras can be used and their relative positions fixed, and then the forgings that have been magnetized and sprayed with magnetic suspension can be photographed vertically under ultraviolet light to obtain the original binocular image of the forging surface.

[0024] Then, the defects in the original binocular image are semantically segmented by the semantic segmentation model, and the defective parts on the forging surface are identified to obtain the binocular image after defect segmentation.

[0025] S102 , fusing the original binocular image with the binocular image after defect segmentation to obtain a fused binocular image, and enhancing the defect portion in the original binocular image.

[0026] S103 , aligning the rows of pixels of the fused binocular image, searching for matching point pairs with minimum matching cost in the aligned fused binocular image, obtaining the image disparity of the fused binocular image based on the matching point pairs, and calculating the image depth information based on the image disparity.

[0027] In an ideal binocular system (with parallel camera optical axes and coplanar imaging planes), the epipolar lines of the left and right images are horizontal. This means that a row of pixels in the left image must appear in the same row in the right image (an epipolar constraint). Row alignment is required to satisfy this constraint, and can be achieved through stereo rectification.

[0028] Then, for each pixel A in the reference image (one of the images in the fused binocular image), the possible disparity can be calculated on the same row in the target image (the other image in the fused binocular image). d Slide the window and calculate the matching cost of all candidate position pixels B and pixels A. The two corresponding pixels with the minimum matching cost are the matching point pairs.

[0029] The disparity of each pixel can be calculated based on each matching point pair, and the depth value corresponding to each pixel, that is, the image depth information, can be calculated based on the disparity of each pixel.

[0030] S104: Perform three-dimensional reconstruction of the forging and its surface defects based on the image depth information.

[0031] Based on the image depth information, the three-dimensional coordinates of each pixel in the fused binocular image in the camera coordinate system can be further calculated.

[0032] The 3D reconstruction method for forging surface defects provided in this embodiment can be applied to a 3D reconstruction system for forging surface defects. The 3D reconstruction system for forging surface defects can be a software system running on a terminal device. The terminal device can be a tablet computer, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), mobile phone, or other terminal device. This embodiment does not impose any restrictions on the specific type of terminal device.

[0033] The present invention semantically segments defects in an original binocular image of a forging surface to obtain a binocular image after defect segmentation. The original binocular image is then fused with the defect-segmented binocular image to obtain a fused binocular image, which can make the defect in the original binocular image more visible. The rows of pixels in the fused binocular image are aligned, and matching point pairs with the minimum matching cost are found in the aligned fused binocular image, so that the target (defect) is not lost during the matching process. The image disparity of the fused binocular image is then determined based on the matching point pairs, and image depth information is calculated based on the image disparity, making the depth information about the defect more accurate. Finally, the forging and its surface defects are three-dimensionally reconstructed based on this image depth information. The reconstruction effect is good, and the defects can be clearly identified in the reconstructed image. The position of the defect on the forging and the size of the defect relative to the forging are obtained, resulting in excellent visualization.

[0034] In some embodiments of the present invention, S101 includes: 2 -Net semantic segmentation model performs semantic segmentation on defects in the original binocular image of the forging surface.

[0035] Reference Figure 2 , which shows a schematic diagram of the structure of the defect segmentation model provided by the present invention. The U²-Net network uses a nested U-Net infrastructure, which has high performance in both object detection and semantic segmentation tasks. Each U-Net infrastructure, for example, Figure 2 En_1 through De_1 in the image extract and restore multi-layer features through two stages of downsampling and upsampling. When the original image of the forging surface is input, each layer of the U²-Net directly generates and outputs defect segmentation results of the same size through De_1 to De_5 and En_6. These defect segmentation results are merged through the fusion layer to extract more spatial information and obtain the final defect segmentation image.

[0036] Before segmenting the original binocular image, the U²-Net model must be trained. The training dataset used in this invention can be sourced from a production line for magnetic particle inspection of steel forgings. The dataset contains sample images of large and small sizes, as well as those with magnetic trace defects in multiple locations. Image enhancement is performed on the collected sample images to expand the segmented dataset. The final dataset contains 5,000 defect images, divided into training, validation, and test sets in an 8:1:1 ratio: 4,000 images for training, 500 for validation, and 500 for testing. Model training begins with an epoch of 300, a batch size of 6, and an initial learning rate of 0.001.

[0037] In some embodiments of the present invention, the calculation formula of the matching cost is as follows:

[0038]

[0039]

[0040] in, represents the aggregate matching cost on each path, Indicates the direction of the path r Current pixel p In Parallax d The cumulative matching cost of Indicates the current pixel p In Parallax d The initial matching cost, Indicates that the previous pixel is at the same disparity d The cumulative matching cost of Indicates the previous pixel in the disparity The cumulative matching cost of Represents the previous pixel in all possible disparities i The minimum cumulative matching cost in Represents the previous pixel in all possible disparities k The minimum cumulative matching cost in 、 Indicates the smoothing penalty when the disparity between the current pixel and the adjacent pixels is small or large. ; Indicates the current pixel p The pixel coordinates in one of the binocular fused views, Indicates the current pixel p Pixel coordinates in another binocular fused view; The image disparity of the fused binocular image is determined based on the matching point pairs. After the image disparity is obtained through stereo matching, the depth value corresponding to each pixel can be calculated based on the principle of similar triangles ( ).in, Z is the distance from the point to the camera, f is the focal length of the camera, T is the baseline distance between the two cameras, d It's parallax.

[0041] If stereo matching is performed directly on the original binocular image, the target, i.e., the defect, may be lost during the matching process, resulting in the defect area being unclear in the 3D reconstructed image when the matching result is used for 3D reconstruction. Therefore, this embodiment first performs semantic segmentation on the defects in the original binocular image of the forging surface to obtain a binocular image after defect segmentation. The original binocular image and the binocular image after defect segmentation are then fused to obtain a fused binocular image, which can make the defect portion in the original binocular image more visible. The rows of pixels in the fused binocular image are aligned, and matching point pairs with the minimum matching cost are found in the aligned fused binocular image. The image disparity of the fused binocular image is determined based on the matching point pairs, and image depth information is calculated based on the image disparity. Finally, the forging and the surface defects of the forging are 3D reconstructed based on the image depth information. The reconstruction effect is good, the defects can be clearly identified in the reconstructed image, and the position of the defect on the forging and the size of the defect relative to the forging are obtained, with good visualization effect.

[0042] In some embodiments of the present invention, to calculate the disparity formed by the target point in the three-dimensional scene between the left and right views, the two corresponding image points of the target point in the left and right views must first be matched. Therefore, in order to reduce the matching search range, stereo calibration of the binocular camera is required before stereo matching, reducing the matching search in two dimensions to one dimension. In other words, the pixels in the original binocular image are aligned in columns. At the same time, the original binocular images can also be dedistorted using their respective camera distortion coefficients and stereo corrected using the relative posture of the cameras to obtain a horizontally consistent image. The original binocular images are then pixel-matched using the SGBM stereo matching algorithm.

[0043] Reference Figure 3 , shows a schematic diagram of stereo matching of fused binocular images provided by the present invention. When matching in the horizontal direction, the center pixel of the window in the reference image (one of the images in the fused binocular image) is used. As the starting point, in the target image (another image in the fused binocular image) The search range is limited to the maximum disparity set in advance. Calculate the gray value difference between the two matching windows and select the cost function The center pixel of the target image window with the minimum value As a reference pixel matching point.

[0044] In some embodiments of the present invention, the step of calculating image depth information based on image disparity may include: performing sub-pixel interpolation processing on the image disparity based on the disparity gradient of the image disparity; and calculating image depth information based on the interpolated image disparity. This step can further improve the accuracy of the disparity map, making the object surface disparity smoother, reaching the sub-pixel level.

[0045] In some embodiments of the present invention, the step of performing three-dimensional reconstruction of forgings and surface defects of forgings based on image depth information may include: determining the three-dimensional coordinates of each pixel point in the binocular fusion image in the camera coordinate system based on the image depth information to generate a three-dimensional point cloud; and segmenting the three-dimensional point cloud using a point cloud segmentation algorithm to obtain a target point cloud corresponding to the forgings and surface defects of the forgings.

[0046] After calculating the depth value of each pixel, the three-dimensional coordinates of each pixel in the camera coordinate system can be further calculated using the following formula:

[0047] in, 、 is the optical center coordinate of the camera, are the pixel coordinates of the reconstructed region. Represents the 3D coordinates of a point in the camera coordinate system.

[0048] After generating a three-dimensional point cloud based on the three-dimensional coordinates of each pixel point in the binocular fusion image in the camera coordinate system, the three-dimensional point cloud can be segmented using a point cloud segmentation algorithm to obtain target point clouds corresponding to forgings and forging surface defects.

[0049] In some embodiments of the present invention, since the fluorescent color of the forging surface is fixed and obviously different from the background color, the point cloud data can carry color information. When the point cloud is segmented, the point cloud can be segmented based on the fluorescent color of the point cloud and the depth information of the point cloud at the same time to quickly segment and obtain the target point cloud.

[0050] In some embodiments of the present invention, the method for three-dimensional reconstruction of forging surface defects further includes: displaying the defective portion in a color different from the color of the forging in the three-dimensional point cloud.

[0051] Figure 4This is a schematic diagram of the 3D reconstruction of forging surface defects provided by the present invention. The original image includes a left view and a right view. In the original image, the green fluorescent portion represents the forging, and the blue portion represents the background. After semantic segmentation of the original image, defect segmentation images corresponding to the left and right views are obtained. The red lines in the defect segmentation images represent defects on the forging surface. The original image and the defect segmentation images are then fused and stereo matching is performed to obtain a disparity map. Finally, based on the disparity map, the forging and its surface defects are 3D reconstructed to obtain a 3D point cloud. In this 3D point cloud, the forging appears green, and the defects on the forging appear blue.

Claims

1. A three-dimensional reconstruction method for forging surface defects, characterized in that: include: Perform semantic segmentation on the defects in the original binocular image of the forging surface to obtain a binocular image after defect segmentation; Fusing the original binocular image with the binocular image after defect segmentation to obtain a fused binocular image; Aligning each row of pixels of the fused binocular image, searching for a matching point pair with a minimum matching cost in the aligned fused binocular image, determining an image disparity of the fused binocular image based on the matching point pair, and calculating image depth information based on the disparity; The forging and the surface defects of the forging are three-dimensionally reconstructed according to the image depth information.

2. The three-dimensional reconstruction method of forging surface defects according to claim 1, characterized in that: The method further comprises: The surface of the forging that has been magnetized and sprayed with magnetic suspension liquid is photographed under ultraviolet light to obtain an original binocular image of the forging surface.

3. The three-dimensional reconstruction method of forging surface defects according to claim 1, characterized in that: Through U 2 -Net semantic segmentation model performs semantic segmentation on defects in the original binocular image of the forging surface.

4. The three-dimensional reconstruction method of forging surface defects according to claim 1, characterized in that: The calculation formula of the matching cost is as follows: in, represents the aggregate matching cost on each path, Indicates the direction of the path r Current pixel p In Parallax d The cumulative matching cost of Indicates the current pixel p In Parallax d The initial matching cost, Indicates that the previous pixel is at the same disparity d The cumulative matching cost of Indicates the previous pixel in the disparity The cumulative matching cost of Represents the previous pixel in all possible disparities i The minimum cumulative matching cost in Represents the previous pixel in all possible disparities k The minimum cumulative matching cost in 、 Indicates the smoothing penalty when the disparity between the current pixel and the adjacent pixels is small or large. ; Indicates the current pixel p The pixel coordinates in one of the binocular fused views, Indicates the current pixel p The pixel coordinates in the other binocular fused view.

5. The three-dimensional reconstruction method of forging surface defects according to claim 1, characterized in that: The calculating of image depth information according to the disparity includes: performing sub-pixel level interpolation processing on the image disparity according to the disparity gradient of the image disparity; The image depth information is calculated based on the interpolated image disparity.

6. The three-dimensional reconstruction method of forging surface defects according to claim 1, characterized in that: Performing three-dimensional reconstruction of the forging and the surface defects of the forging according to the image depth information includes: Determining the three-dimensional coordinates of each pixel in the binocular fused image in a camera coordinate system according to the image depth information to generate a three-dimensional point cloud; The three-dimensional point cloud is segmented by a point cloud segmentation algorithm to obtain target point clouds corresponding to the forging and the surface defects of the forging.

7. The three-dimensional reconstruction method of forging surface defects according to claim 6, characterized in that: The method further comprises: In the three-dimensional point cloud, the defective portion is displayed in a color different from the color of the forging.

Images