A method and system for identifying and reconstructing magnetic traces of steel die forgings

Through the series connection between the target recognition network and the target segmentation network and the multi-view three-dimensional depth recovery model, the automatic identification and three-dimensional reconstruction of magnetic marks of steel die forgings is realized, and the problems of manual operational error detection and three-dimensional reconstruction are solved in traditional magnetic powder detection. Improves detection efficiency and accuracy, and realizes automated marking and polishing.

CN114140407BActive Publication Date: 2024-11-08WUHAN UNIV OF TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202111390169.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-23
Publication Date
2024-11-08
Estimated Expiration
2041-11-23

AI Technical Summary

Technical Problem

Traditional magnetic powder detection methods rely on manual operations, which are prone to false detection and missed detection, and fail to realize the three-dimensional spatial position reconstruction of magnetic traces, resulting in the inability to realize automated detection.

Method used

The connection between the target recognition network and the target segmentation network is adopted, combined with the multi-view three-dimensional depth recovery model, and the robot drives the camera to obtain forging images from different angles, identify and reconstruct the three-dimensional information of magnetic traces, and realize the automatic identification of magnetic traces and spatial position reconstruction. .

Benefits of technology

It improves the accuracy and detection efficiency of magnetic mark recognition, realizes automatic mark marking and polishing, meets the needs of automated detection, and reduces the computational complexity and imaging equipment costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114140407B_ABST
    Figure CN114140407B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for identifying and reconstructing magnetic traces of steel die forgings. A robot drives a camera to obtain images of forgings at known different positions; an image of a forging with magnetic traces is identified through a target detection network, and any image is selected as a reference image to obtain the position of a magnetic trace identification frame in the image; the image of the forging with magnetic traces is used as the input of a multi-view stereo depth recovery model to obtain the depth information of the reference image; the position of the magnetic trace area in the reference image is calculated, and the position of the magnetic trace pixel is obtained through a target segmentation network, and is converted into the reference image through image coordinates; the 2D-3D position correspondence between the magnetic trace pixel and the depth map is used to reconstruct the three-dimensional coordinates of the magnetic trace in the camera coordinate system, and the three-dimensional coordinates are converted to the robot coordinate system in combination with the posture relationship between the robot and the camera. The present invention provides spatial position information of magnetic traces for further robot grinding, which is conducive to the realization of fully automated magnetic powder detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of nondestructive testing and relates to a method and system for identifying and reconstructing magnetic traces of steel die forgings. Background Art

[0002] Magnetic particle testing uses the difference in magnetic permeability between surface and near-surface defects of magnetized steel products and the magnetic permeability of steel to produce magnetic field distortion and form a leakage magnetic field, thereby attracting magnetic particles to form magnetic marks, which reveal the location and shape of defects under appropriate lighting conditions.

[0003] As an extremely important part of non-destructive testing services, magnetic particle testing technology is widely used in the surface defect detection of large quantities of steel forgings such as steering arms, steering knuckles, and axles due to its advantages of simple operation, low equipment cost, and high detection efficiency. However, traditional magnetic particle testing is heavily dependent on the working status and experience level of the operator. Working under ultraviolet light in a dark room for a long time is not only prone to fatigue and missed detection, but also may cause damage to human skin and eye lenses due to improper operation.

[0004] On the other hand, the national standard "GB / T 12361-2003 General Technical Conditions for Steel Die Forgings" clearly states that when there are folds and cracks on the non-machined surface of the forging, they should be polished and removed, and detailed provisions are made for the polishing requirements. For magnetic marks within the specified range, it is a common practice for companies to adopt identification-marking-polishing-re-identification to improve the yield of forgings. Among them, identification and marking are in-situ integrated operations. Only when the magnetic marks are successfully marked can the subsequent polishing operation be realized. Therefore, the key to realizing automated magnetic particle testing lies in the accurate identification of magnetic marks and the reconstruction of the spatial position of magnetic marks.

[0005] In the existing literature, the literature "Deng Xin. Research on fluorescent magnetic particle flaw detection system based on deep learning" and the literature "Lin Guo. Research and implementation of intelligent nondestructive testing technology based on fluorescent magnetic powder" all use image processing and other methods to identify the target magnetic traces. The problem with these methods is that they do not consider the three-dimensional reconstruction process after magnetic trace identification. In actual engineering applications, manual marking of magnetic trace positions is still required, and the automation of magnetic particle detection cannot be achieved. Taking into account the shortcomings of traditional magnetic particle detection methods and existing magnetic trace image recognition, a method for magnetic trace identification and reconstruction of steel die forgings is proposed, which effectively solves the problems of false detection, missed detection and reconstruction of magnetic trace spatial position in manual operation. Summary of the invention

[0006] The purpose of the present invention is to realize the automatic recognition and reconstruction process of magnetic traces of steel die forgings, overcome the problems of false detection, missed detection, and missing magnetic trace reconstruction process in the prior art, and propose an automatic magnetic trace recognition and reconstruction method and system for steel die forgings. By connecting the target recognition network and the target segmentation network in series, the upper-level recognition result is used as the lower-level segmentation object to narrow the search range and improve the recognition accuracy. In addition, combined with the multi-perspective stereo depth recovery model, the three-dimensional information reconstruction of the magnetic trace is realized, and the computational complexity in the reconstruction process is reduced.

[0007] The technical solution adopted by the method of the present invention is: a method and system for identifying and reconstructing magnetic traces of steel die forgings, comprising the following steps:

[0008] Step 1: The robot drives the camera to obtain the forging image I at different known positions of the robot img ;

[0009] Step 2: Identify the forging image I with magnetic traces through the target detection network img_crack , and from I img_crack Select any image as the reference image I ref , other images are source images I source , obtain the reference image I ref The position of the center magnetic mark identification frame;

[0010] Step 3: Based on the reference image I in step 2 ref and source image I source , through the multi-view stereo depth recovery network, obtain the reference image I ref Depth information;

[0011] Step 4: Based on the reference image I in step 2 ref The magnetic mark frame position is calculated and its magnetic mark area image I is calculated. ref_seg , the magnetic trace area image I ref_seg As the input image of the target segmentation network, the magnetic trace in the magnetic trace area image I is obtained ref_seg and calculate the pixel position of the magnetic trace in the reference image I ref The following location information;

[0012] Step 5: Based on the reference image I in step 3 ref Depth information and magnetic traces in step 4 in reference image I ref The position information under the image is reconstructed through the 2D-3D position correspondence of the image, and the three-dimensional information of the magnetic trace is converted into the robot coordinate system by combining the position relationship between the robot and the camera.

[0013] The technical solution adopted by the system of the present invention is: a magnetic trace identification and reconstruction system for steel die forgings, comprising the following modules:

[0014] Module 1: The robot drives the camera to obtain forging images at different known positions of the robot. img ;

[0015] Module 2: Used to identify forging images with magnetic traces through the target detection network I img_crack , and from I img_crack Select any image as the reference image I ref , other images are source images I source , obtain the reference image I ref The position of the center magnetic mark identification frame;

[0016] Module 3: Based on the reference image I in step 2 ref and source image I source , through the multi-view stereo depth recovery network, obtain the reference image I ref Depth information;

[0017] Module 4: for the reference image I in step 2 ref The magnetic mark frame position is calculated and its magnetic mark area image I is calculated. ref_seg , the magnetic trace area image I ref_seg As the input image of the target segmentation network, the magnetic trace in the magnetic trace area image I is obtained ref_seg and calculate the pixel position of the magnetic trace in the reference image I ref The following location information;

[0018] Module 5: for the reference image I in step 3 ref Depth information and magnetic traces in step 4 in reference image I ref The position information under the image is reconstructed through the 2D-3D position correspondence of the image, and the three-dimensional information of the magnetic trace is converted into the robot coordinate system by combining the position relationship between the robot and the camera.

[0019] The technical effects of the present invention include:

[0020] 1) Through the series connection of the target recognition network and the target segmentation network, the target can be quickly located, and the target segmentation search range can be narrowed, the segmentation accuracy can be increased, and the detection accuracy can be improved, which is suitable for the identification of micro-magnetic traces under complex backgrounds.

[0021] 2) The multi-view stereo depth model is used in combination with the known camera shooting positions to restore the three-dimensional structure of the scene, reduce the cost of imaging equipment, avoid the computational complexity of traditional three-dimensional reconstruction methods, and improve detection efficiency.

[0022] 3) Based on the goal of in-situ integrated identification and marking operations, the spatial position of the magnetic trace is reconstructed. The three-dimensional information meets the needs of automated marking and polishing and has practical engineering significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a method according to an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of obtaining a forging image according to an embodiment of the present invention;

[0025] Figure 3 The magnetic trace recognition model of the embodiment of the present invention;

[0026] Figure 4 The magnetic trace three-dimensional depth recovery model of the embodiment of the present invention;

[0027] Figure 5 The magnetic trace segmentation model of the embodiment of the present invention;

[0028] Figure 6 A process for reconstructing depth information of magnetic traces in a camera coordinate system according to an embodiment of the present invention;

[0029] Explanation of the numbers in the figure: 1-camera; 1.1-coordinate system of the camera when shooting at different positions; 1.2-forging image corresponding to the camera position; 2-robot tool; 3-robot body; 4-steel forging; 5-world coordinate system; 6-a group of forging images taken by the camera at different positions; 6.1-forging image; 6.1.1-magnetic trace; 7-reference image; 8-source image; 9-target frame; 10-magnetic trace area in the reference image; 11-the area in the source image at the same position as the reference image; 12-image group for depth recovery; 13-depth map; 14-source image target segmentation map; DETAILED DESCRIPTION

[0030] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0031] Please see Figure 1 The present invention provides a method for identifying and reconstructing magnetic traces of steel die forgings, comprising the following steps:

[0032] Step 1: The robot drives the camera to obtain the forging image I at different known positions of the robot img ;

[0033] like Figure 2 As shown, the forging image acquisition in this embodiment is achieved by fixing a camera at the end of a robot and taking images by moving the robot to multiple different positions.

[0034] Step 2: Identify the forging image I with magnetic traces through the target detection network ima_crack , and from I img_crackSelect any image as the reference image I ref , other images are source images I source , obtain the reference image I ref The position of the center magnetic mark identification frame;

[0035] like Figure 3 As shown, the target detection network of this embodiment is a regional convolutional neural network, and the image feature extraction is four residual blocks based on the self-attention mechanism. Each residual block contains a downsampling layer, four self-attention layers, a convolution layer and a series connection with the downsampling layer. The dimension of the downsampling layer is Where H, W are the forging image sizes, C is the number of features, and i is the i-th residual block.

[0036] Step 3: Based on the reference image I in step 2 ref and source image I source , through the multi-view stereo depth recovery network, obtain the reference image I ref Depth information;

[0037] like Figure 4 As shown, the feature extraction module in the stereo depth recovery model of this embodiment adopts an encoding-decoding structure, and a self-attention module is added in the middle of the structure; the source image I source Transform to the reference image I through a differentiable homography ref In the coordinate system:

[0038]

[0039] Among them, H i (d) represents the i-th source image I source With reference image I ref Isomorphic relations at depth d, i represents the number of source images, K1 and K i Both represent the internal parameters of the camera, (R i , t i ) and (R1, t1) represent the source image I source and reference image I ref The rotation matrix and translation vector relative to the world coordinate system; I represents the identity matrix; n represents the camera axis corresponding to the reference image;

[0040] Step 4: Based on the reference image I in step 2 ref The magnetic mark frame position is calculated and its magnetic mark area image I is calculated. ref_seg , the magnetic trace area image I ref_seg As the input image of the target segmentation network, the magnetic trace in the magnetic trace area image I is obtained ref_seg and calculate the pixel position of the magnetic trace in the reference image I ref The following location information;

[0041] The reference image I in this embodiment ref Medium magnetic trace area (x, y, h seg , w seg ) is calculated as:

[0042]

[0043] Among them, h seg , w seg Indicates the magnetic trace area in the reference image I ref The height and width of the image; (x, y) is the reference image I detected by the target detection network in step 2 ref The center pixel of the magnetic mark frame, (h norm , w norm ) represents the normalized length and width of its identification box; max(h norm , w norm ) means (h norm , w norm ), R represents the scaling factor, f(h norm , w norm )express:

[0044]

[0045] like Figure 5 As shown, the target segmentation network of this embodiment is based on the U-Net encoding-decoding structure, and adopts a model including a five-layer convolutional neural network to extract image features, and the number of kernels is 32, 64, 128, 256, and 512 from the first layer to the fifth layer, respectively.

[0046] The magnetic trace pixel (x i ,y i ) in the reference image I ref The location information (x crack ,y crack )The solution process is:

[0047]

[0048] Among them, (x i ,y i ) is the image of the magnetic trace in the magnetic trace area I ref_seg The pixel position in h seg Indicates that the magnetic trace area image is in the reference image I ref The height in (x, y) is the reference image I detected by the target detection network in step 2 ref The center pixel of the magnetic trace identification box.

[0049] Step 5: Based on the magnetic trace area image I in step 3ref_seg Depth information and magnetic trace area image I in step 4 ref_seg The pixel position information of the magnetic trace is used to reconstruct the three-dimensional information of the magnetic trace through the 2D-3D position correspondence of the image. Combined with the position relationship between the robot and the camera, the three-dimensional information is converted into the robot coordinate system.

[0050] like Figure 6 As shown, the three-dimensional coordinates in the magnetic trace camera coordinate system of this embodiment are obtained by ref_seg Depth information and magnetic trace area image I ref_seg The corresponding relationship of the pixel position information of the magnetic trace is calculated, and its spatial position in the camera coordinate system [X cam , Y cam , Z cam ]:

[0051]

[0052] Among them, (x crack ,y crack ) is the image of the magnetic trace in the magnetic trace area I ref_seg The pixel position in Z cam is the magnetic trace area image I calculated in step 3 ref_seg Depth information; K represents the intrinsic parameter of the camera; [X cam , Y cam , Z cam ] is converted to the spatial position [X] in the robot tool coordinate system robot , Y robot , Z robot ]for:

[0053]

[0054] Where T represents the position relationship between the camera and the robot tool coordinate system.

[0055] The present invention integrates magnetic trace identification and marking as an integrated process, overcoming the problems of false detection and missed detection caused by manual operation in traditional magnetic particle detection methods, as well as the single identification object in existing magnetic trace image recognition and the lack of magnetic trace spatial position reconstruction process. The invention realizes the rapid identification and spatial position reconstruction of magnetic traces of steel die forgings, provides magnetic trace spatial position information for further robot grinding, and is conducive to the realization of fully automated magnetic particle detection.

[0056] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A method for identifying and reconstructing magnetic traces of steel die forgings, characterized in that: The following steps are involved: Step 1: The robot drives the camera to obtain the forging image I at different known positions of the robot img ; Step 2: Identify the forging image I with magnetic traces through the target detection network img_crack , and from I img_crack Select any image as the reference image I ref , other images are source images I source , obtain the reference image I ref The position of the center magnetic mark identification frame; The target detection network is a regional convolutional neural network, and the image feature extraction is four residual blocks based on the self-attention mechanism. Each residual block contains a downsampling layer, four self-attention layers, a convolution layer and a series connection with the downsampling layer. The dimension of the downsampling layer is Where H, W are the forging image sizes, C is the number of features, and i is the i-th residual block; Step 3: Based on the reference image Ir in step 2 ef And source image person ource , through the multi-view stereo depth recovery network, obtain the reference image I ref Depth information; The depth restoration network is a multi-view stereo depth restoration network based on a cost volume, wherein the feature extraction module adopts an encoding-decoding structure, and a self-attention module is added in the middle of the structure; the source image I source Transform to the reference image I through a differentiable homography ref In the coordinate system: Among them, H i (d) represents the i-th source image I source With reference image I ref Isomorphic relations at depth d, i represents the number of source images, K1 and K i Both represent the internal parameters of the camera, (R i , t i ) and (R1, t1) represent the source image I source and reference image I ref The rotation matrix and translation vector relative to the world coordinate system; I represents the identity matrix; n represents the camera axis corresponding to the reference image; Step 4: Based on the reference image I in step 2 ref The position of the magnetic mark frame is calculated, and the magnetic mark area image I is calculated. rer_seg , the magnetic trace area image I ref_seg As the input image of the target segmentation network, the magnetic trace in the magnetic trace area image I is obtained rer_seg and calculate the pixel position of the magnetic trace in the reference image I ref The following location information; Step 5: Based on the reference image I in step 3 ref Depth information and magnetic traces in step 4 in reference image I ref The position information under the image is reconstructed through the 2D-3D position correspondence of the image, and the three-dimensional information of the magnetic trace is converted into the robot coordinate system by combining the position relationship between the robot and the camera.

2. The method for identifying and reconstructing magnetic traces of steel die forgings according to claim 1, characterized in that: In step 4, the reference image I ref Medium magnetic trace area (x, y, seg , w seg ) is calculated as: in, seg , w seg Indicates the magnetic trace area in the reference image I ref The height and width of the image; (x, y) is the reference image I detected by the target detection network in step 2 ref The center pixel of the magnetic mark frame, ( norm , w norm ) represents the normalized length and width of its identification box; max(h norm , w norm )express( norm , w norm ), R represents the scaling factor, f(h norm , w norm )express:

3. The method for identifying and reconstructing magnetic traces of steel die forgings according to claim 1, characterized in that: In step 4, the target segmentation network is based on the U-Net encoding-decoding structure, and a model including a five-layer convolutional neural network is used to extract image features, and the number of kernels is 32, 64, 128, 256, and 512 from the first layer to the fifth layer, respectively.

4. The method for identifying and reconstructing magnetic traces of steel die forgings according to claim 1, characterized in that: In step 4, the magnetic trace pixels (x i ,y i ) in the reference image I ref The location information (x crack ,y crack )The solution process is: Among them, (x i ,y i ) is the image of the magnetic trace in the magnetic trace area I ref_seg The pixel position in seg Indicates that the magnetic trace area image is in the reference image I ref The height in (x, y) is the reference image I detected by the target detection network in step 2 ref The center pixel of the magnetic trace identification box.

5. The method for identifying and reconstructing magnetic traces of steel die forgings according to any one of claims 1 to 4, characterized in that: In step 5, the spatial position of the magnetic trace in the camera coordinate system [X cam , Y cam , Z cam ]: Among them, (x crack ,y crack ) is the image of the magnetic trace in the magnetic trace area I ref_seg The pixel position in Z cam The magnetic trace area image I calculated in step 3 ref_seg Depth information; K represents the intrinsic parameter of the camera; [X cam , Y cam , Z cam ] is converted to the spatial position [X] in the robot tool coordinate system robot , Y robot , Z robot ]for: Where T represents the position relationship between the camera and the robot tool coordinate system.

6. A magnetic trace identification and reconstruction system for steel die forgings, used to implement the method described in any one of claims 1 to 5; characterized in that: Includes the following modules: Module 1: The robot drives the camera to obtain forging images at different known positions of the robot. img ; Module 2: Used to identify forging images with magnetic traces through the target detection network I img_crack , and from I img_crack Select any image as the reference image I ref , other images are source images I source , obtain the reference image I ref The position of the center magnetic mark identification frame; Module 3: Based on the reference image I in step 2 ref and source image I source , through the multi-view stereo depth recovery network, obtain the reference image I ref Depth information; Module 4: Based on the reference image I in step 2 ref The magnetic mark frame position is calculated and its magnetic mark area image I is calculated. rer_seg , the magnetic trace area image I ref_seg As the input image of the target segmentation network, the magnetic trace in the magnetic trace area image I is obtained rer_seg and calculate the pixel position of the magnetic trace in the reference image I ref The following location information; Module 5: Based on the reference image I in step 3 ref Depth information and magnetic traces in step 4 in reference image I ref The position information under the image is reconstructed through the 2D-3D position correspondence of the image, and the three-dimensional information of the magnetic trace is converted into the robot coordinate system by combining the position relationship between the robot and the camera.

Citation Information

Cited By

  • Three-dimensional reconstruction method for forging surface defects

    CN120599129A

  • Forge piece surface defect segmentation method based on weak supervision positioning guidance and related device

    CN122574510A