An automatic molten steel sampling device based on machine vision system

Abstract

The present invention relates to an automated molten steel sampling device based on a machine vision system, belonging to the field of industrial automated inspection technology. The system utilizes a robotic arm to grasp samples from a cooling pool and automatically feeds the samples to be inspected into a molten steel sample defect detection module based on fringe projection. The defect detection module obtains the 3D contour distribution of the molten steel sample through fringe projection profilometry, calculates the volume of the molten steel sample based on 3D data partition integration, and evaluates the effectiveness of the current sampling by comparing it with the volume of a standard sample. The module has a simple structure and uses multi-frame projection, multi-frame acquisition, and label recognition to "control" the synchronization of the system's projection and acquisition timing, without the addition of an additional hardware timing control unit. The dual low-frequency-guided high-frequency phase calculation method proposed in the present invention achieves higher accuracy in 3D reconstruction of molten steel samples. The robustness of the system was evaluated using multiple molten steel samples with varying degrees of defects, demonstrating the effectiveness of the proposed system.

Description

Technical Field

[0001] The invention belongs to the technical field of industrial automation detection, and in particular relates to an automatic molten steel sampling device based on a machine vision system. Background Art

[0002] As the final product of a steel plant, monitoring its quality is crucial. However, due to the long production cycle and complex process flow, factors influencing its quality are complex and varied, requiring precise control of the steel production process. Molten steel, a primary product of the steelmaking process, is crucial for quality control.

[0003] Currently, one approach to testing molten steel quality involves quantitatively analyzing high-temperature molten steel using high-precision instruments to infer the quality of the final steel produced. This approach not only increases hardware costs but also, due to the reliance on prior knowledge to determine the final quality of the steel produced, can introduce a degree of error. Direct, visual analysis of small samples of high-temperature molten steel not only saves costs but also avoids the limitations of indirect testing methods. However, due to the dark, high ambient temperature, and high workload of the molten steel production environment, manual sampling of high-temperature molten steel presents significant safety risks. The increasing maturity of computer vision theory has led to breakthroughs in modern and intelligent manufacturing within the industrial sector. A robotic positioning and grasping module designed by combining a robotic arm with automated components effectively avoids these accidents.

[0004] The key technology of the robotic arm positioning and grasping module is to use the robotic arm to achieve precise positioning and grasping of molten steel samples. At present, positioning technology based on image recognition has been widely used in industrial automation, biomedicine and other fields due to its advantages of non-contact detection. This technology has been studied mainly from two directions: traditional algorithms and machine learning algorithms. For example, Kneip et al. used a stereo vision system combined with a mathematical model to segment the target object and obtained an overall edge model; for example, Wan et al. used a machine learning algorithm that combined multiple machine learning classifiers and used a one-way variance analysis algorithm to identify and locate pineapples.

[0005] Defect detection in molten steel samples can be performed by performing 3D reconstruction of the sample and calculating the difference between the current sample volume and the reference sample volume to determine if the sample is qualified. Fringe projection profilometry, a commonly used non-contact 3D reconstruction method, offers advantages such as high precision, high efficiency, a wide measurement range, and stability. It is widely used in medical diagnosis, industrial product monitoring, and reverse engineering. For example, Liu et al. used Fourier transform profilometry to project blue fringes for real-time 3D reconstruction of high-reflectivity weld plates. However, the phase calculation method involves frequency-domain filtering, resulting in large errors in the reconstructed weld plate edge regions. Yang et al. used phase-shifted fringes to project phase-shifted fringes and employed phase-shifting technology to reconstruct complex surface objects. However, the phase calculation requires multiple deformed fringes, which reduces the real-time performance of the measurement system. To improve the real-time performance of the system, Karpinsky et al. increased the speed of fringe projection and acquisition by removing the projector's color filter and adding hardware for externally generating a trigger signal. This approach also precisely controlled the synchronization between projection and acquisition. However, the addition of an external trigger signal device resulted in high hardware costs for the measurement system. Whether the 3D reconstruction process is based on Fourier analysis or phase shifting technology, the object's abruptness and reflectivity will affect the accuracy of phase unwrapping. To suppress phase unwrapping errors caused by surface jumps, Li et al. projected multiple frequency stripes and used a low-frequency method to guide high-frequency phase unwrapping to perform 3D reconstruction of isolated abruptly changing objects. However, this method has the disadvantage that when the measured object has a large abruptness and the low-frequency phase unwrapping results are erroneous, the phase-guided unwrapping process will cause error transmission, resulting in high-frequency phase unwrapping errors. Tan et al. projected two sets of frequency stripes and used a dual-frequency heterodyne method to reconstruct metal objects in 3D. Because the frequencies of the two projected stripes are close and the combined frequency must be larger than the measurement amplitude, the stripe frequency selection is low, limiting the accuracy of 3D reconstruction. Summary of the Invention

[0006] To address the challenges of existing technologies and effectively reduce hardware costs while ensuring the accuracy of 3D reconstruction of molten steel samples, this invention designs an automated molten steel sampling device based on a machine vision system. This device automatically samples molten steel samples and uses a molten steel sample defect detection module to perform defect detection on the sampled molten steel, ensuring that the sample meets inspection requirements.

[0007] To solve the above technical problems, the present invention adopts the following technical solution: an automatic molten steel sampling device based on a machine vision system, comprising a first robotic arm, an automatic tray, a cooling water pool, a second robotic arm, and a molten steel sample defect detection module based on fringe projection, wherein the first robotic arm is located between the furnace and the automatic tray, the cooling water pool is located on one side of the automatic tray and, after the automatic tray is flipped, the molten steel sample can fall into the cooling water pool, and the second robotic arm is located between the cooling water pool and the molten steel sample defect detection module based on fringe projection;

[0008] An industrial spoon is fixed at the upper end of the first robotic arm, and multiple molten steel sample molds are set on the automatic tray. The first robotic arm controls the industrial spoon to take samples in the furnace and pour them into the molten steel sample molds on the automatic tray. The automatic tray is provided with a flipping device. After the automatic tray flips, the molten steel samples are poured into the cooling water pool. A manipulator is installed at the end of the second robotic arm. The second robotic arm controls the manipulator to grab the molten steel samples in the cooling water pool and send them to the molten steel sample defect detection module based on fringe projection.

[0009] Furthermore, the specific working process of this device is as follows:

[0010] Step 1) An industrial spoon is attached to the end of a robotic arm without a gripper. Using positioning technology, the end of the robotic arm is moved to the furnace mouth area, and its joints are programmed to remove a small amount of molten steel. Next, the end of the robotic arm is positioned at the center of the sample mold groove area, and the robotic arm joints are moved and programmed to pour the removed molten steel into the mold groove.

[0011] Step 2) Set the automatic tray to flip after a certain period of time to pour the molded sample into the cooling pool;

[0012] Step 3) The sample in the cooling pool is positioned using a robotic arm grasping module. After positioning using machine vision recognition technology, the robotic arm equipped with a manipulator is controlled to the positioned sample point and an action signal is given to remove the sample and place it on the subsequent inspection workbench operation;

[0013] Step 4) After placing the molten steel sample to be tested on the testing workbench, the molten steel sample defect detection module based on fringe projection is used to obtain the 3D surface distribution of the molten steel sample to be tested through fringe projection profilometry, and the volume is calculated to determine whether the sample is qualified.

[0014] Furthermore, the robotic arm positioning and grasping module described in step 3) includes a workbench, a robotic arm, and a CCD. The CCD is fixed to the end of the robotic arm. The CCD is used to capture an image of the molten steel sample on the workbench. The grasping position and grasping direction are acquired through computer processing and converted into corresponding posture signals of the robotic arm for grasping. The specific steps are as follows:

[0015] 3.1. Feature recognition of molten steel samples

[0016] Feature recognition of molten steel samples involves two steps: target sample pose acquisition and feature extraction. The target sample pose acquisition is performed using a camera fixed at the end of the robotic arm. Feature extraction is combined with image recognition technology to process the acquired image. The specific recognition technologies are as follows:

[0017] A. Positioning coordinates

[0018] The molten steel sample is divided into two areas: the upper area "cylinder" and the lower area "handle". The upper area of ​​the sample is first identified, and the identified upper area is used to assist in obtaining the lower area. The steps for identifying the upper area are as follows:

[0019] First, the collected target sample image is processed using the Canny operator to obtain the primary edge contour of the sample I e (x, y), in order to avoid unclosed cracks in the primary edge contour, which may cause errors in subsequent region selection, the obtained edge contour is expanded by the following formula:

[0020]

[0021] Among them, M se It is a 3×3 square structural element. In order to eliminate the background noise contour in the primary edge contour map, the 4-pixel connected area is first used to detect the edge contour map of the sample I e All closed contours in ′(x,y), denoted as D e t, fill the detected contour using the following formula,

[0022]

[0023] Then fill the sample outline I pad (x, y) is then tested for 8-pixel connected regions, and the contour filling result with the largest detection area is taken as the overall area of ​​the sample, denoted as I outline (x, y), the obtained sample area is fitted with a standard circle using the invariant kernel circle detection algorithm, and the center coordinates of the fitted circle are marked as (x cir ,y cir ), the radius is recorded as R cir , considering that the circular area obtained by fitting may not be able to completely cover I outline The upper half of the area represented by (x, y) is expanded by 10 pixels to the radius of the fitted circle, which is recorded as R c ' ir , the expanded result is used as the upper half area outline of the molten steel sample, and the following formula is used to fill it as the upper half area of ​​the sample;

[0024]

[0025] Finally, the upper half area of ​​the sample and the entire area of ​​the sample are eliminated using the following formula, and the remaining area is used as the "handle" area of ​​the sample;

[0026]

[0027] Considering the symmetry of the molten steel sample structure, finding the center of the region as the positioning point can better meet the system requirements. The center of mass is the abbreviation of the center of mass, which is better for finding the center point of irregular objects with uniform mass distribution. Therefore, the center of mass of the "handle" area is taken as the positioning point of the target sample. The center of mass formula is transformed, and the mass of the object point is replaced by the gray value of the image point. The positioning grasping point can be solved using the following formula;

[0028]

[0029] Where D represents the “handle” area I of the sample handle (x,y), x i 、y i Represents the coordinate value of the regional image point, m i Represents the grayscale value of the image point area;

[0030] B. Deflection direction

[0031] For two-dimensional images, the length of the "vector" directed line segment is used to define the relationship between two pixels, and the concept of "angle" in mathematics is combined as the posture of the sample at the current moment. For the molten steel sample that has been divided into two areas, the feature points of the corresponding areas are selected as the representation points, and the center of the fitting circle (x cir ,y cir ) as the representation point of the area, and the positioning grasping point (x m ,y m ) as the representation point of the area, and use the following formula to get the vector representation between the two representation points;

[0032]

[0033] To establish the transformation between vector and angle, another vector parallel to the image plane axis is introduced Use the following formula to convert it into angle expression:

[0034]

[0035] Where round{·} is the rounding function;

[0036] 3.2. Robotic Arm Positioning Based on Image Feature Points

[0037] After obtaining the positioning direction and deflection direction from the image captured by the camera fixed at the end of the robotic arm, the camera calibration and hand-eye calibration technology are used to convert them into positioning and grasping points in the coordinate system of the robotic arm base, and the end effector movement is directed to achieve object positioning and grasping;

[0038] A. Positioning grab point conversion

[0039]

[0040] Among them, z c is the positioning point p pix (x m ,y m ) in the camera coordinate system (X C ,Y C ,Z C ) under the value of Z, M i ' ns is the intrinsic parameter matrix obtained by camera calibration, is the hand-eye calibration matrix, Represents the mapping matrix between the terminal coordinates and the base coordinates at the i-th moment;

[0041] B. Deflection direction conversion

[0042] Since the increment of the rotation angle of the manipulator axis joint is linearly related to the increment of the rotation value, the initial position of the axis joint rotation is O If the reference position is the same and the rotation direction is the same, then read the initial rotation angle value θ start , combined with the following formula to obtain the axis deviation angle θ H

[0043] θ H =θ O +θ start (9).

[0044] Furthermore, the molten steel sample defect detection module based on fringe projection described in step 4) includes a detection workbench, a digital projector and a CCD. The molten steel sample to be detected is placed on the detection workbench. The digital projector is used to project multiple frames of fringes onto the detection workbench. The CCD is used to continuously collect multiple frames of deformed fringe images. The collected fringe sets are then sent to a computer for selection and time sequence restoration. After that, the effective fringe images are subjected to phase calculation, phase unwrapping, phase height mapping and other algorithms to reconstruct the 3D surface distribution of the sample. Finally, the sample volume is calculated based on the 3D surface distribution to determine whether the sample is qualified. The specific steps are as follows:

[0045] Step 4.1) After placing the molten steel sample on the inspection workbench, a flash consisting of multiple frames of fringe patterns with a certain phase shift is played in a loop using a digital projector at a frame rate of 20 fps. The CCD is controlled to continuously capture multiple frames of the deformed fringe pattern at a frame rate of 30 fps and a capture duration of 2 s. The captured fringe patterns are then stored in memory.

[0046] Step 4.2) The collected fringe set is sent to a computer for label recognition and sorting to achieve fringe selection and time sequence restoration. The effective fringe image is then subjected to phase calculation, phase unwrapping, and phase height mapping to reconstruct the 3D surface distribution of the sample;

[0047] Step 4.3) The effective bottom area of ​​the captured image of the molten steel sample to be tested is selected, and the volume of the sample is calculated in combination with the obtained 3D surface distribution to determine whether the sample is qualified.

[0048] Furthermore, in step 4.1), the numbered fringes captured by the CCD are sorted. The specific steps are as follows:

[0049] After capturing a labeled stripe I cap (x, y) uses the invariant kernel circle detection algorithm to fit the standard circle. (x, y) represents the pixel coordinates of the captured image, and the center coordinates of the fitted circle are marked as (x circle ,y circle ), the radius is recorded as R, draw the fitted circle on the image with the same size as the label stripe, replace the center coordinates and radius of formula (3) with the current recognition result, and fill the circle as the label template M circle (x,y);

[0050] In order to obtain the label embedded in the current label stripe, the label area in the label stripe is first retained using the following formula, and the retained label information is recorded as I label (x,y);

[0051]

[0052] Then the label information obtained by formula (10) is extracted using the following formula, and the extracted label is recorded as I l ' abel (x,y),

[0053]

[0054] Since the template matching method is used to realize the sorting of label stripes, it is required that the two label images to be matched have the same size and the same grayscale information representation method. Therefore, the extracted labels should be scaled to the designed label image size first. Considering that the designed label information is single, the nearest neighbor interpolation is used to realize the scaling of the labels. The following formula represents the mapping relationship between the label pixels before scaling and the label pixels after scaling:

[0055]

[0056] Among them, (X, Y) represents the image pixels after scaling, I grade (X, Y) represents the scaled image, w and h represent the size of the image before scaling, and W and H represent the size of the image after scaling;

[0057] Then, the scaled labels are binarized using the following formula to obtain the final processed label image I: num(X,Y)

[0058]

[0059] Among them, thresh is the set threshold, and the threshold selection is calculated using the algorithm of minimizing the intra-class variance.

[0060] Compare the numbered drawing to be processed with the numbered drawing of the design Use the following formula to perform template matching. The template with the smallest difference among all matching results is the best matching template, and the label sequence corresponding to the template is recorded as the phase shift sequence of the fringe;

[0061]

[0062] Finally, determine whether the recognized labels are repeated. If not, name the label stripe. Otherwise, do not name it. Repeat the above operation to identify the remaining label stripes until all images are recognized. If all designed labels are recognized once, proceed to the next step. Otherwise, it indicates that the image data acquisition is not completed.

[0063] Furthermore, the specific process of phase calculation, phase unwrapping, and phase height mapping in step 4.2) to reconstruct the 3D surface distribution of the sample is as follows:

[0064] Three frequency stripes are projected, and the phases of the two lower frequency stripes are calculated using the dual-frequency heterodyne method. After the low-frequency phase is obtained, it is used to guide the phase calculation of the high-frequency stripes. This method can effectively improve the accuracy of phase calculation. The phase shift step of this method is set to five steps, with frequencies f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, f12, f13, f14, f15, f16, f17, f18, f19, f20, f21, f22, f23, f24, f35, f19, f25, f26, f36, f1 1′ , f2, and the relationship between the three frequencies is expressed as:

[0065]

[0066] The transmittance expression of the 15 stripes generated by the computer is:

[0067]

[0068] Among them, (x P ,y P ) represents the pixel coordinates of the projected image, a and b are constants;

[0069] The 15 stripes are combined into a flash and captured by CCD. After label recognition, the effective stripe expression extracted is:

[0070]

[0071] Where A(x,y) represents the average intensity, B(x,y) represents the intensity modulation, and Φ(x,y) represents the fringe phase after being modulated by the object. The truncated phase is then:

[0072]

[0073] Among them, φ i (x,y) represents the truncated phase of different frequency fringes, then the frequencies are f1, f 1′ The truncated phases of f2 fringes are represented by φ1(x,y) and φ 1′ (x,y), φ2(x,y);

[0074] The continuous phase difference between the two 合 (x,y) can be obtained based on dual-frequency heterodyning:

[0075]

[0076] Among them, Φ1(x,y), Φ 1′ (x,y) are frequencies f1 and f 1′ The fringe phase after being modulated by the object, similarly, the frequency f 合 It can be obtained by the following formula

[0077]

[0078] For a point on the measurement surface, the fringe phase after being modulated by the object can be expressed as

[0079] Φ j (x,y)=2πK j (x,y), j=1,1′ (21)

[0080] Among them, K1(x,y), K 1′ (x,y) represent the fringe frequencies f1 and f 1′ The fringe level when , the fringe level has the following relationship

[0081]

[0082] Among them, k j (x, y) represents the phase order. When the relative positions of the camera, projector and the object to be measured are fixed, the same point on the object has the same position on the grating images of different periods, so we can get

[0083]

[0084] Substituting formula (21) into formula (23), we get

[0085]

[0086] Combining formula (19) and formula (24), we can get the expression of the fringe phase Φ1(x,y) after being modulated by the object:

[0087]

[0088] or

[0089]

[0090] Φ1(x,y)=2π·k1(x,y)+φ1(x,y) (27)

[0091] Where k1(x,y) is the phase order of the truncated phase at a certain point when the fringe frequency is f1;

[0092] After obtaining Φ1(x,y), the phase order k2(x,y) of the truncated phase of the frequency f2 fringe can be calculated based on the relationship between the high-frequency and low-frequency absolute phases and the frequency:

[0093]

[0094] Substituting k2(x,y) into the relationship between truncated phase and continuous phase can obtain the continuous phase of high-frequency fringes. The expanded continuous phase can only reflect the surface contour of a three-dimensional object. To obtain the actual height of the object, the specific mapping relationship between the two must be known, namely the phase-height mapping matrix. This matrix requires moving the reference surface back and forth m (m ≥ 3) times a specified distance, and projecting and collecting fringes after each movement. The established relationship is usually written as:

[0095]

[0096] Where Z(x,y) represents the actual object height to be solved, Φ(x,y) represents the unwrapped phase result obtained by solving the deformed fringes modulated by the object, and a(x,y), b(x,y), and c(x,y) are the system mapping parameters to be solved, which can be calculated using the least squares method.

[0097] Furthermore, in step 4.3), the effective bottom area of ​​the sample image captured by the module is selected, and the specific steps for calculating the sample volume based on the 3D surface distribution are as follows:

[0098] According to the 3D contour distribution of the molten steel sample, the height distribution Z(x w ,y w ), the volume of the final molten steel sample can be obtained by the following integral formula

[0099] V=∫∫Z(x w ,y w )dxw dy w (30)

[0100] In the actual volume calculation process, in order to save computing resources and remove the interference of background noise and inevitable shadows on volume calculation, the pixel coordinates (x, y) are converted into world coordinates (x w ,y w ), the effective area of ​​the bottom contour of the molten steel sample was extracted first, and the contour of the molten steel sample was divided into two parts for extraction;

[0101] A. Contour extraction of the upper half of the sample area

[0102] When using fringe stagger to extract the contour of the upper half of the sample area, the residual image is obtained by subtracting the sample image taken by the measurement system from the undeformed reference fringe image obtained when no sample is placed on the workbench to effectively obtain the stagger information. The residual image can be obtained by the following formula:

[0103]

[0104] Among them, I1′ n (x, y) represents the undeformed reference fringe pattern obtained when the workbench is not placed with a sample at the frequency f1, and the residual mean image I is obtained. ref (x, y), the Canny operator is used to process the primary upper half area edge contour I eg (x, y), in order to avoid the error of region selection caused by the unclosed edge contour, the M se Replace it with a planar disc-shaped structural element with a radius of 1 and perform expansion processing. The processing result is recorded as I ei (x,y);

[0105] Due to the uneven illumination distribution caused by the working environment, the expanded primary edge contour image I ei (x, y) will contain a few noise contours. In order to eliminate these invalid contours, we first use 4-pixel connected area detection I ei All closed contours in (x,y) are denoted as q=1,2,3,···, and replace D in formula (2) e t is replaced by D I q for the primary edge contour image I ei (x,y) is filled, and the filling result is recorded as I fill (x,y);

[0106] Then the filled image I fill (x, y) uses 8-pixel connected area detection and detects the contour filling block with the largest area as the upper half area I′ of the sample fill(x,y), since the processed contours are all expanded, I′ fill (x,y) is then corroded using the following formula to form the final upper half of the sample:

[0107]

[0108] Among them, M se1 It is a planar disk-shaped structural element with a radius of 1;

[0109] B. Contour extraction of the lower half of the sample area

[0110] The modulation degree of the stripes can be obtained by the following formula:

[0111]

[0112] The obtained modulation M(x,y) is binarized using formula (13) to obtain the primary sample overall area M including the shadow area bina (x,y);

[0113] Due to the difference in surface characteristics between the upper and lower regions of the sample, the grayscale information of the entire region of the primary sample is expressed as “contour” in the upper region and as “region” in the lower region. Therefore, the two regions can be separated by “eroding” the entire region distribution map of the primary sample. In order to achieve the erosion effect during image processing, the image should be expanded. Then, the obtained modulation extreme value segmentation map is “eroded” by formula (1). The M used is se is a planar disk-shaped structural element with a radius of 2, and the processing result is recorded as M bi (x,y);

[0114] At the same time, in order to eliminate the regional “corrosion” effect caused by the above formula, formula (32) is used to convert M bi (x,y) is "expanded", and the M used is se1 is a planar disk-shaped structural element with a radius of 2, and the result is recorded as M close (x,y);

[0115] The processed primary lower half area contour M close (x, y) uses the Canny operator to perform edge detection, and similarly performs a closing operation on the primary edge contour after detection to bridge the contour cracks. The closed contour map is recorded as M eg (x, y), and in order to eliminate the background noise contour, the 4-pixel connected detection M eg All contours in (x,y) are recorded as p=1,2,3,···, and replace the Replace with To Meg (x,y) is used to fill the contour, and the filling result is recorded as M fill (x,y);

[0116] Then the padded image M fill (x, y) performs an 8-neighborhood connected region search and takes the largest connected region as the lower half of the sample, denoted as M′ fill (x, y), considering that edge detection will expand the lower half area, the area is corroded using formula (32), and the corroded area is used as the final lower half area of ​​the sample I hand (x,y);

[0117] Finally, the two areas of the sample are combined using the following formula as the final bottom area of ​​the sample;

[0118]

[0119] Get the bottom area I of the sample mask (x, y), the camera system parameters are obtained using the camera calibration algorithm, and the image two-dimensional coordinates are converted into spatial coordinates through the following coordinate mapping relationship;

[0120]

[0121] Where [xy 1] T Represents the homogeneous representation of the pixel coordinates of the captured image, [x w y w 1] T Represents the homogeneous representation of the world coordinates obtained by mapping, M ext 、M ins Represent the external parameters and intrinsic parameters of the camera respectively. After obtaining the world coordinates of each pixel, calculate the area corresponding to a single pixel. Assume that the area of ​​a single pixel unit is the shaded area S;

[0122] When solving the unit area S, you need to first calculate the four vertices of the quadrilateral corresponding to the point The coordinate value of the coordinate value can generally be calculated using the bilinear interpolation method. Set the unit vertex at the center of the four neighboring pixels, then It can be expressed as

[0123]

[0124] Similarly, the coordinate values ​​of the other three vertices can be obtained. Then, the quadrilateral is approximated as a rectangle and the value calculated using the following formula is used as the actual physical area of ​​the pixel block. The area of ​​all unit blocks is calculated and the corresponding block volume is obtained. Finally, the value calculated using formula (30) is used as the final sample volume.

[0125]

[0126] The advantages of the present invention and its positive effects are.

[0127] 1. The robot arm positioning and grasping module of the present invention can meet the requirements of accurate positioning and grasping of molten steel samples. Combining the principles of image recognition and calibration, it implements regional feature extraction of molten steel samples, and uses the extracted results as the pre-information of the robot arm positioning and grasping module. It uses the robot arm calibration technology and the end axis mapping function to realize the conversion of image features to robot arm positioning and grasping posture. Finally, this part is embedded in the entire system to realize automatic sampling of molten steel.

[0128] 2. Rapid Acquisition of the Image Positioning Coordinates and the Sample's Pose: The captured image of the molten steel sample is processed using edge detection, dilation, and filling algorithms to obtain an edge map containing the sample's edge contours. The entire area of ​​the sample is determined by selecting the maximum neighborhood of the filled area. Based on the sample's geometric characteristics, it is divided into an upper "cylinder" region and a lower "handle" region. If the upper region's geometric characteristics are close to a circle, a standard circle is fitted to the entire region map using an invariant kernel circle detection algorithm. The resulting circular contour is then pixel-expanded and filled with the expanded circular contour as the upper region of the sample. The lower region is obtained by "subtracting" the sample's overall contour from the upper region. The center of mass of the resulting lower region is used as the image positioning coordinates of the sample. The center of the fitted circle and the positioning coordinates are first described as "vectors" to describe the relationship between the two points. A unit vector parallel to the image plane axis is introduced, and the angle between the two vectors is calculated as the image pose of the sample.

[0129] 3. Use software to achieve timing control of projected stripes: Phase-shifted stripes are embedded in labels, using a circular outer contour and an internal pattern of Arabic numerals (0-9). Binarization is used to represent the image's grayscale information. The labeled stripes are sorted and assembled into a flash image, which is continuously captured by a projector and acquired by a CCD. A single labeled stripe is first identified by contour recognition. A circle fitting algorithm is used to determine the labeled region of the stripe using the invariant kernel circle detection algorithm. Pixels within this region are extracted. The extracted labeled image is scaled to the designed label size using the nearest neighbor interpolation algorithm and binarized to unify the grayscale representation. Finally, template matching is used to compare the labeled image with the designed label pixel by pixel. The template with the smallest difference is defined as the best matching template, and the label sequence corresponding to this template is recorded as the phase-shifted sequence of the labeled stripe. Finally, the phase-shifted sequence is determined to determine if it repeats. If it does, it is discarded; otherwise, it is retained. This process is repeated until all labeled stripes are identified. The existing method removes the color filter device of the projector and adds an external hardware that can give a trigger signal to achieve timing control of the stripes, which increases the hardware cost and the complexity of the equipment.

[0130] 4. The phase calculation method of dual low-frequency guidance of high frequency proposed in the present invention realizes high-precision restoration of objects with abrupt surface changes and uneven reflectivity. Three fringe frequencies are designed, two of which are close and low-frequency, and the other is high-frequency. After the three frequency fringes are projected onto the object to be measured and captured by CCD, the two low-frequency fringes with closer frequencies are first phase-calculated to obtain the truncated phase, and the dual-frequency heterodyne method is used to obtain the object contour (continuous phase) with lower precision. Then, the phase of the higher-frequency fringes is calculated to obtain the high-frequency truncated phase. The low-precision continuous phase is used to guide the truncated phase of the high-frequency fringes to obtain the continuous phase of the high-frequency fringes, and this phase is used as the final contour of the sample. The existing method uses the low-frequency guidance of high frequency alone to obtain the object phase, but requires the phase calculation result of the low-frequency fringes to be accurate, which has limitations; and the dual-frequency heterodyne method is used alone to obtain the object phase, which is limited by the selection of the fringe frequency, limiting the measurement accuracy.

[0131] 5. Calculate the volume of the molten steel sample to evaluate the sampling effectiveness: The bottom area of ​​the 3D contour of the molten steel sample obtained by the test is selected and divided into the upper half area and the lower half area according to the surface properties of the molten steel sample. Since the sample image captured by the measurement system contains stripe information, the contour of the upper half area of ​​the molten steel sample is obtained by first subtracting the deformed stripes from the undeformed reference stripes obtained when no sample is placed on the workbench to obtain a residual image, and the obtained residual image is subjected to edge detection, expansion, and filling image processing algorithms to obtain an edge image containing the contour of the upper half area of ​​the sample, and the contour of the upper half area of ​​the molten steel sample is obtained by selecting the maximum neighborhood of the filled area; the contour of the lower half area of ​​the molten steel sample is obtained by first using the deformed stripes and the undeformed reference stripes obtained when no sample is placed on the workbench to obtain the modulation degree, and then the modulation degree is binarized to obtain the overall area of ​​the sample, and the obtained overall area of ​​the sample is closed to "cut off" the upper half area and the lower half area, and then the obtained primary lower half area of ​​the sample is subjected to edge detection, closing operation, and filling image processing algorithms to further obtain an edge image containing the contour of the lower half area of ​​the sample, and the contour of the lower half area of ​​the molten steel sample is determined by selecting the maximum neighborhood of the filled area. Finally, the upper and lower regions of the molten steel sample are spliced, and the cracks are bridged using the closing operation and filling method to obtain the final bottom area contour of the molten steel sample.

[0132] The volume of the molten steel sample is determined by using the principle of "differentiation first, integration later". The molten steel sample is divided into several "small cubes" and the sum of the volumes of these cubes is calculated as the volume of the molten steel sample. The effective bottom area of ​​the molten steel sample is the bottom area contour obtained above. The obtained bottom area is divided into pixel regions. The coordinate mapping relationship of the pixel region obtained by the camera calibration algorithm is converted into the actual area. The phase information of the molten steel sample is converted into the actual height using the phase height mapping method. The height corresponding to the pixel block and its actual area are calculated to obtain the volume of a single pixel. The volume of all pixel blocks in the effective area of ​​the sample is added together as the volume of the molten steel sample to be measured. The volume of the standard sample is measured using the same measurement method. The relative volume error between the volume of the standard molten steel sample and the volume of the molten steel sample to be measured is calculated to determine the effectiveness of the sampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] Figure 1 It is an automatic molten steel sampling device based on a machine vision system.

[0134] Figure 2 Schematic diagram of the robotic arm positioning and grasping module.

[0135] Figure 3 Schematic diagram of the defect detection module based on fringe projection.

[0136] Figure 4 Schematic diagram of the molten steel sample to be tested.

[0137] Figure 5 Schematic diagram of the image and robot arm axis deviation mapping structure.

[0138] Figure 6 Design a style diagram for the label.

[0139] Figure 7 Schematic diagram of phase calculation principle.

[0140] Figure 8 Schematic diagram of volume calculation of molten steel sample, where (a) is the differential of molten steel sample, (b) is the bottom area division of molten steel sample, and (c) is the sample image taken by the measurement system.

[0141] Figure 9 Schematic diagram of the coordinate relationship of the sample bottom area.

[0142] Figure 10 These are some checkerboard images collected by the robotic arm in different postures.

[0143] Figure 11 This is the feature extraction diagram of the tested sample.

[0144] Figure 12 Extract verification diagram for the tested sample features.

[0145] Figure 13 Fringe pattern for label design and partially embedded label.

[0146] Figure 14 The test sample and fringe pattern collected by the 3D inspection platform.

[0147] Figure 15 The height distribution results obtained by 5 phase calculation methods.

[0148] Figure 16 This is the height distribution effect at different viewing angles.

[0149] Figure 17 The collected 15mm standard block gauge and fringe pattern are shown.

[0150] Figure 18 The height distribution of 15mm gauge blocks obtained by five algorithms.

[0151] Figure 19 The height error results of the 15mm block gauge obtained by five algorithms are shown in the figure.

[0152] Figure 20 This is the process of extracting the bottom area of ​​the upper half of the sample being tested.

[0153] Figure 21 This is the process of extracting the bottom area of ​​the lower half of the sample being tested.

[0154] Figure 22This is the process of splicing the bottom area of ​​the sample.

[0155] Figure 23 Figure 2 shows the tested molten steel samples with different defect levels.

[0156] Figure 24 The phase calculation results of molten steel samples with different defect levels are shown in the figure below, and the effect of the sample height is shown. DETAILED DESCRIPTION

[0157] In order to make the objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below.

[0158] like Figure 1 As shown, an automatic molten steel sampling device based on a machine vision system includes a first robotic arm, an automatic tray, a cooling water pool, a second robotic arm, and a molten steel sample defect detection module based on fringe projection. The first robotic arm is located between the furnace and the automatic tray, the cooling water pool is located on one side of the automatic tray and after the automatic tray is flipped, the molten steel sample can fall into the cooling water pool, and the second robotic arm is located between the cooling water pool and the molten steel sample defect detection module based on fringe projection.

[0159] An industrial spoon is fixed at the upper end of the first robotic arm, and multiple molten steel sample molds are set on the automatic tray. The first robotic arm controls the industrial spoon to take samples in the furnace and pour them into the molten steel sample molds on the automatic tray. The automatic tray is provided with a flipping device. After the automatic tray flips, the molten steel samples are poured into the cooling water pool. A manipulator is installed at the end of the second robotic arm. The second robotic arm controls the manipulator to grab the molten steel samples in the cooling water pool and send them to the molten steel sample defect detection module based on fringe projection.

[0160] like Figure 3 As shown in the figure, the molten steel sample defect detection module based on fringe projection includes a detection workbench, a digital projector and a CCD. The molten steel sample to be detected is placed on the detection workbench. The digital projector is used to project multiple frames of fringes onto the detection workbench. The CCD is used to continuously acquire multiple frames of deformation fringe images. The CCD acquisition frame rate is 30fps, the acquisition duration is set to 2s, and the acquired fringes are saved in the memory.

[0161] An automatic molten steel sampling device based on a machine vision system has the following specific working process:

[0162] Step 1) An industrial spoon is attached to the end of a robotic arm without a gripper. Using positioning technology, the end of the robotic arm is moved to the furnace mouth area, and its joints are programmed to remove a small amount of molten steel. Next, the end of the robotic arm is positioned at the center of the sample mold groove area, and the robotic arm joints are moved and programmed to pour the removed molten steel into the mold groove.

[0163] Step 2) Set the automatic tray to flip after a certain period of time to pour the molded sample into the cooling pool;

[0164] Step 3) The sample in the cooling pool is positioned using a robotic arm grasping module. After positioning using machine vision recognition technology, the robotic arm equipped with a manipulator is controlled to the positioned sample point and an action signal is given to remove the sample and place it on the subsequent testing station;

[0165] Step 4) After placing the molten steel sample on the inspection workbench, the fringe projection-based molten steel sample defect detection module is used to obtain the 3D surface distribution of the molten steel sample through fringe projection profilometry, and the volume is calculated to determine whether the sample is qualified. The specific steps are as follows:

[0166] Step 4.1) After placing the molten steel sample on the inspection workbench, a flash consisting of multiple frames of fringe patterns with a certain phase shift is played in a loop using a digital projector at a frame rate of 20 fps. The CCD is controlled to continuously capture multiple frames of the deformed fringe pattern at a frame rate of 30 fps and a capture duration of 2 s. The captured fringe patterns are then stored in memory.

[0167] Step 4.2) The collected fringe set is sent to a computer for label recognition and sorting to achieve fringe selection and time sequence restoration. The effective fringe image is then subjected to phase calculation, phase unwrapping, and phase height mapping to reconstruct the 3D surface distribution of the sample;

[0168] Step 4.3) The effective bottom area of ​​the captured image of the molten steel sample to be tested is selected, and the volume of the sample is calculated in combination with the obtained 3D surface distribution to determine whether the sample is qualified.

[0169] Throughout the system's entire process, steps 1) and 3) require a robotic arm to perform positioning and grasping operations, tasks that require integration with image recognition positioning technology. For step 1), due to the large area of ​​the furnace mouth and the wide recognition range, positioning accuracy requirements are lower. However, for operations like pouring molten steel from the ladle into the mold and step 3), the workpieces have a smaller effective area, requiring precise positioning. Therefore, this paper focuses on combining image recognition technology with a robotic arm to accomplish positioning and grasping tasks.

[0170] The robot positioning and grasping implementation module based on image recognition technology is as follows: Figure 2 When the system is working, it first places the molten steel sample to be tested in the water pool and controls the camera to collect the posture map. The collected image is then sent to the computer for image recognition and other algorithms to extract the image features of the sample. Finally, the mapping function is used to calculate the positioning point of the robot arm and the deflection direction of the grasping operation to drive the robot arm to grasp.

[0171] The molten steel sample to be tested of the present invention is as follows Figure 4 As shown in the figure, feature recognition of molten steel samples involves two steps: acquiring the target sample's posture and extracting its features. The target sample's posture is acquired using a camera mounted at the end of the robotic arm; feature extraction is combined with image recognition technology to process the captured image. The specific recognition techniques are as follows:

[0172] A. Positioning coordinates

[0173] according to Figure 4 The target sample can be divided into two areas, the upper half area "cylinder" and the lower half area "handle". In order to avoid errors in subsequent sample detection caused by contact with the sample surface when the robot arm grasps the sample, the present invention selects a point in the lower half area with a smaller contact area as the robot arm positioning and grasping coordinate. Since the method of directly identifying the entire area of ​​the sample to obtain the lower half area has a high computational complexity, and it is noted that the geometric features of the upper half area are more significant than those of the lower half area, the upper half area of ​​the sample is first identified, and the identified upper half area is used to assist in the acquisition of the lower half area. The steps for identifying the upper half area are as follows:

[0174] First, the collected target sample image is processed using the Canny operator to obtain the primary edge contour of the sample I e (x, y), in order to avoid unclosed cracks in the obtained primary edge contour, which may cause errors in subsequent region selection, the obtained edge contour is expanded by the following formula:

[0175]

[0176] Among them, M se It is a 3×3 square structural element. In order to eliminate the background noise contour in the primary edge contour map, the 4-pixel connected area is first used to detect the edge contour map of the sample I e All closed contours in ′(x,y), denoted as D e t , fill the detected contour using the following formula,

[0177]

[0178] Then fill the sample outline I pad (x, y) is then tested for 8-pixel connected regions, and the contour filling result with the largest detection area is taken as the overall area of ​​the sample, denoted as I outline (x, y). The obtained sample area is fitted with a standard circle using the invariant kernel circle detection algorithm proposed by Atherton et al., and the center coordinates of the fitted circle are marked as (x cir ,y cir ), the radius is recorded as R cirConsidering that the circular area obtained by fitting may not be able to completely cover I outline The upper half of the area represented by (x, y) is expanded by 10 pixels to the radius of the fitted circle, which is recorded as R c ' ir The expanded result is used as the upper half area outline of the molten steel sample, and the following formula is used to fill it as the upper half area of ​​the sample.

[0179]

[0180] Finally, the upper half area of ​​the sample and the entire area of ​​the sample are eliminated using the following formula, and the remaining area is used as the "handle" area of ​​the sample.

[0181]

[0182] from Figure 4 Note that the structure of the molten steel sample is symmetrical. Finding the center of the region as the positioning point can better meet the system requirements. The center of mass is the abbreviation of the center of mass. It is more effective for finding the center point of irregular objects with uniform mass distribution. Therefore, the center of mass of the "handle" area is taken as the positioning point of the target sample. The center of mass formula is transformed, and the mass of the object point is replaced by the gray value of the image point. Then, the positioning grasping point can be solved using the following formula;

[0183]

[0184] Where D represents the “handle” area I of the sample handle (x,y), x i 、y i Represents the coordinate value of the regional image point, m i Represents the grayscale value of the image point area;

[0185] B. Deflection direction

[0186] For two-dimensional images, the length of the "vector" directed line segment is used to define the relationship between two pixels, and the concept of "angle" in mathematics is combined as the posture of the sample at the current moment. For the molten steel sample that has been divided into two areas, the feature points of the corresponding areas are selected as the representation points, and the center of the fitting circle (x cir ,y cir ) as the representation point of the area, and the positioning grasping point (x m ,y m ) as the representation point of the area, and use the following formula to get the vector representation between the two representation points;

[0187]

[0188] To establish the transformation between vector and angle, another vector parallel to the image plane axis is introduced Use the following formula to convert it into angle expression:

[0189]

[0190] Where round{·} is the rounding function;

[0191] Robotic arm positioning based on image feature points

[0192] After obtaining the positioning direction and deflection direction from the image taken by the camera fixed at the end of the robotic arm, the camera calibration and hand-eye calibration technology are used to convert them into positioning and grasping points in the robotic arm base coordinate system, and the end effector movement is directed to achieve object positioning and grasping.

[0193] A. Positioning grab point conversion

[0194]

[0195] Among them, z c is the positioning point p pix (x m ,y m ) in the camera coordinate system (X C ,Y C ,Z C ) under the value of Z, M′ ins is the intrinsic parameter matrix obtained by camera calibration, is the hand-eye calibration matrix, Represents the mapping matrix between the terminal coordinates and the base coordinates at the i-th moment;

[0196] B. Deflection direction conversion

[0197] Since the increment of the rotation angle of the manipulator axis joint is linearly related to the increment of the rotation value, the initial position of the axis joint rotation is O The reference position is the same and the rotation direction is consistent, such as Figure 5 As shown. Then read the initial rotation angle value θ start , combined with the following formula to obtain the axis deviation angle θ H

[0198] θ H =θ O +θ start (9)

[0199] The above describes the robot arm positioning and grasping module in detail. The following describes the molten steel sample defect detection module based on fringe projection of this system in detail. Figure 3When the system is working, the molten steel sample to be tested is first placed on the testing workbench. A projector is used to sequentially project multiple frames of fringe patterns with a certain phase shift onto the sample surface, and the CCD is controlled to continuously acquire multiple frames of deformed fringe patterns. The acquired fringe sets are then sent to the computer for selection and time sequence restoration. After that, the effective fringe images are subjected to phase calculation, phase unwrapping, phase height mapping and other algorithms to reconstruct the 3D surface distribution of the sample. Finally, the sample volume is calculated based on the 3D surface distribution to determine whether the sample is qualified.

[0200] Because the system uses phase-shifting technology to acquire the 3D profile of a molten steel sample, this method requires controlling the simultaneous projection and acquisition of multiple frames of stripes during measurement. Existing devices often use additional hardware control units to synchronize projection and acquisition, increasing the complexity and hardware cost of the measurement equipment. To address this issue, the present invention employs software-controlled timing. By creating a flash memory, the projector is controlled to sequentially project sinusoidal stripes with phase shifts. A CCD continuously captures multiple frames of deformed stripes. Image recognition methods are then used to select and sort valid stripes for phase calculation.

[0201] The flash projected by the projection system consists of multiple sorted phase-shifted stripes with a frame rate of 20fps, which are played in a loop. The CCD acquisition frame rate is 30fps, the acquisition duration is set to 2s, and the acquired stripes are saved to the memory. For the captured series of stripes, the software is used to perform a recognition algorithm with low computational complexity on the image to achieve timing control. Since the inherent characteristics of the black and white sinusoidal stripe image are not obvious, it is less feasible to directly perform image recognition tasks on the stripe image to distinguish stripes with different phase shifts. A simple and efficient way is to embed the stripe images with different phase shifts into artificially designed labels. The labels should have the advantages of low cross-correlation, simple design, easy recognition, high recognition accuracy and no impact on the accuracy of the measurement system. Then, the stripe images with different phase shifts are distinguished by identifying different labels to achieve the purpose of sorting.

[0202] According to the label design requirements, the designed label style is as follows Figure 6 It uses the Arabic numerals "0-9" pattern to determine the order of phase-shifted stripes, and uses the outer contour of a standard circle to quickly identify the labeled area. It also uses binary representation of image information to reduce computational complexity when performing label recognition.

[0203] The specific steps for sorting the labeled fringes captured by the CCD in step 4.1) are as follows:

[0204] After capturing a labeled stripe I cap(x, y) is used to fit a standard circle using the invariant kernel circle detection algorithm proposed by Atherton et al. (x, y) represents the pixel coordinates of the captured image, and the center coordinates of the fitted circle are marked as (x circle ,y circle ), the radius is recorded as R, draw the fitted circle on the image with the same size as the label stripe, replace the center coordinates and radius of formula (3) with the current recognition result, and fill the circle as the label template M circle (x,y);

[0205] In order to obtain the label embedded in the current label stripe, the label area in the label stripe is first retained using the following formula, and the retained label information is recorded as I label (x,y);

[0206]

[0207] Then the label information obtained by formula (10) is extracted using the following formula, and the extracted label is recorded as I′ label (x,y),

[0208]

[0209] Since the template matching method is used to realize the sorting of label stripes, it is required that the two label images to be matched have the same size and the same grayscale information representation method. Therefore, the extracted labels should be scaled to the designed label image size first. Considering that the designed label information is single, the nearest neighbor interpolation is used to realize the scaling of the labels. The following formula represents the mapping relationship between the label pixels before scaling and the label pixels after scaling:

[0210]

[0211] Among them, (X, Y) represents the image pixels after scaling, I grade (X, Y) represents the scaled image, w and h represent the size of the image before scaling, and W and H represent the size of the image after scaling;

[0212] Then, the scaled labels are binarized using the following formula to obtain the final processed label image I: num (X,Y)

[0213]

[0214] Among them, thresh is the set threshold, and the threshold selection is calculated using the minimization of intra-class variance algorithm proposed by Otsu et al.

[0215] Finally, the labeled graph to be processed is Figure 7 Design number Use the following formula to perform template matching. The template with the smallest difference among all matching results is the best matching template, and the label sequence corresponding to the template is recorded as the phase shift sequence of the fringe;

[0216]

[0217] Finally, determine whether the recognized labels are repeated. If not, name the label stripe. Otherwise, do not name it. Repeat the above operation to identify the remaining label stripes until all images are recognized. If all designed labels are recognized once, proceed to the next step. Otherwise, it indicates that the image data acquisition is not completed.

[0218] The specific process of phase calculation, phase unwrapping, and phase height mapping in step 4.2 to reconstruct the 3D surface distribution of the sample is as follows:

[0219] The molten steel sample has height jumps at the edge and the connection between the upper and lower areas, such as Figure 4 As shown. The truncated phases obtained from these steep change regions are prone to errors when unwrapped. The current conventional solution is to use a time phase unwrapping algorithm, typical algorithms include low-frequency guidance of high-frequency, dual-frequency heterodyne method, etc. The low-frequency guidance of high-frequency method requires that the phase calculation results of the low-frequency f1 stripes are accurate. The reason is that when an error occurs in the low-frequency phase calculation, its error will be transferred to the high-frequency f2 phase measurement result; the dual-frequency heterodyne method requires two frequencies (f1 and f2) to be accurate. 1′ ) must be close and their combined frequency value f 合 This method is limited in its measurement accuracy. To address these issues, the present invention designs a phase calculation method that uses dual low-frequency guidance to calculate high-frequency phases. The calculation process is as follows: Figure 7 shown.

[0220] Three frequency stripes are projected, and the phases of the two lower frequency stripes are calculated using the dual-frequency heterodyne method. After the low-frequency phase is obtained, it is used to guide the phase calculation of the high-frequency stripes. This method can effectively improve the accuracy of phase calculation. The phase shift step of this method is set to five steps, with frequencies f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, f12, f13, f14, f15, f16, f17, f18, f19, f20, f21, f22, f23, f24, f35, f19, f25, f26, f36, f1 1′ , f2, and the relationship between the three frequencies is expressed as:

[0221]

[0222] The transmittance expression of the 15 stripes generated by the computer is:

[0223]

[0224] Among them, (x P ,y P ) represents the pixel coordinates of the projected image, a and b are constants;

[0225] The 15 stripes are combined into a flash and captured by CCD. After label recognition, the effective stripe expression extracted is:

[0226]

[0227] Where A(x,y) represents the average intensity, B(x,y) represents the intensity modulation, and Φ(x,y) represents the fringe phase after being modulated by the object. The truncated phase is then:

[0228]

[0229] Among them, φ i (x,y) represents the truncated phase of different frequency fringes, then the frequencies are f1, f 1′ The truncated phases of f2 fringes are represented by φ1(x,y) and φ 1′ (x,y), φ2(x,y);

[0230] The continuous phase difference between the two 合 (x,y) can be obtained based on dual-frequency heterodyning:

[0231]

[0232] Among them, Φ1(x,y), Φ 1′ (x,y) are frequencies f1 and f 1′ The fringe phase after being modulated by the object, similarly, the frequency f 合 It can be obtained by the following formula

[0233]

[0234] For a point on the measurement surface, the fringe phase after being modulated by the object can be expressed as

[0235] Φ j (x,y)=2πK j (x,y), j=1,1′ (21)

[0236] Among them, K1(x,y), K 1′ (x,y) represent the fringe frequencies f1 and f 1′ The fringe level when , the fringe level has the following relationship

[0237]

[0238] Among them, k j (x, y) represents the phase order. When the relative positions of the camera, projector and the object to be measured are fixed, the same point on the object has the same position on the grating images of different periods, so we can get

[0239]

[0240] Substituting formula (21) into formula (23), we get

[0241]

[0242] Combining formula (19) and formula (24), we can get the expression of the fringe phase Φ1(x,y) after being modulated by the object:

[0243]

[0244] or

[0245]

[0246] Φ1(x,y)=2π·k1(x,y)+φ1(x,y) (27)

[0247] Where k1(x,y) is the phase order of the truncated phase at a certain point when the fringe frequency is f1;

[0248] After obtaining Φ1(x,y), the phase order k2(x,y) of the truncated phase of the frequency f2 fringe can be calculated based on the relationship between the high-frequency and low-frequency absolute phases and the frequency:

[0249]

[0250] Substituting k2(x,y) into the relationship between truncated phase and continuous phase can obtain the continuous phase of high-frequency fringes. The expanded continuous phase can only reflect the surface contour of a three-dimensional object. To obtain the actual height of the object, the specific mapping relationship between the two must be known, namely the phase-height mapping matrix. This matrix requires moving the reference surface back and forth m (m ≥ 3) times a specified distance, and projecting and collecting fringes after each movement. The established relationship is usually written as:

[0251]

[0252] Where Z(x,y) represents the actual object height to be solved, Φ(x,y) represents the unwrapped phase result obtained by solving the deformed fringes modulated by the object, and a(x,y), b(x,y), and c(x,y) are the system mapping parameters to be solved, which can be calculated using the least squares method.

[0253] Step 4.3) Calculate the sample volume based on the 3D surface distribution as follows:

[0254] In order to evaluate the defects of molten steel samples, this algorithm first obtains the 3D contour of the sample and then converts it into volume to determine the degree of defects in the molten steel sample. When calculating the volume based on the 3D contour of the irregular object, the idea of ​​"differentiation first, then integration" is adopted. The bottom area of ​​the molten steel sample (X, Y plane) is differentiated in pixels, and the molten steel sample can be divided into several "small cubes", such as Figure 8 (a) As shown. Through camera calibration, the pixel coordinates (x, y) of the CCD image are converted into world coordinates (x w ,y w ), the actual physical size of each pixel unit can be obtained; according to the 3D contour distribution of the molten steel sample, the height distribution Z(x w ,y w ), the volume of the final molten steel sample can be obtained by the following integral formula

[0255] V=∫∫Z(x w ,y w )dx w dy w (30)

[0256] In the actual volume calculation process, in order to save computing resources and remove the interference of background noise and inevitable shadows on volume calculation, the pixel coordinates (x, y) are converted into world coordinates (x w ,y w ), the effective area of ​​the bottom contour of the molten steel sample was extracted first.

[0257] The bottom contour extraction of the molten steel sample can be achieved through edge detection algorithms. Classic edge detection algorithms in image processing include the Canny operator and the Sobel operator. Since the Canny operator is combined with a double threshold to extract edges, it effectively avoids extracting false edges, and the extracted edges have good continuity. Therefore, the present invention uses the Canny operator to extract the effective area of ​​the sample. In order to obtain the bottom contour of the molten steel sample, according to the surface properties of the sample, it is divided into the upper half area and the lower half area and extracted separately. The bottom contour of the molten steel sample is divided as follows Figure 8 (b) shown.

[0258] The molten steel sample defect detection module based on fringe projection designed by the present invention is used, and the molten steel sample covered by the fringe is captured as follows: Figure 8As shown in (c), the upper half of the molten steel sample is shown in the white dotted frame area A, and the lower half is shown in the white solid frame area B. When extracting the contour of the upper half of the molten steel sample, the surface reflectivity of area A is close to that of the reference surface (area C), which makes the grayscale distribution of the stripes in the two areas similar. In addition, due to the influence of the shadow area (area D framed by the red dotted line), it is impossible to directly use the edge detection algorithm to extract the contour of the upper half of the sample. Considering Figure 8 The notable feature of (c) is that the fringes are deformed after being modulated by the object. This deformation causes the fringes to stagger at the boundary between region C and region A. Therefore, the present invention uses the staggered fringes to extract the contour of the upper half of the sample. When extracting the contour of the lower half of the molten steel sample, region B has a lower reflectivity and a significant grayscale jump at the boundary with region C. However, due to the influence of the periodic grayscale distribution of the fringes, direct contour extraction using edge detection algorithms is still impossible. Considering that calculating the fringes' modulation can directly reveal the object's grayscale distribution characteristics, when extracting the contour of the lower half of the sample, the fringes' modulation is first calculated, and then the contour is extracted from the modulation map.

[0259] A. Contour extraction of the upper half of the sample area

[0260] When using fringe stagger to extract the contour of the upper half of the sample area, the residual image is obtained by subtracting the sample image taken by the measurement system from the undeformed reference fringe image obtained when no sample is placed on the workbench to effectively obtain the stagger information. The residual image can be obtained by the following formula:

[0261]

[0262] Among them, I1′ n (x, y) represents the undeformed reference fringe pattern obtained when the workbench is not placed with a sample at the frequency f1, and the residual mean image I is obtained. ref (x, y), the Canny operator is used to process the primary upper half area edge contour I eg (x, y), in order to avoid the error of region selection caused by the unclosed edge contour, the M se Replace it with a planar disc-shaped structural element with a radius of 1 and perform expansion processing. The processing result is recorded as I ei (x,y);

[0263] Due to the uneven illumination distribution caused by the working environment, the expanded primary edge contour image I ei (x, y) will contain a few noise contours. In order to eliminate these invalid contours, we first use 4-pixel connected area detection I ei All closed contours in (x,y) are denoted as And the formula (2) Replace with For the primary edge contour image Iei (x,y) is filled, and the filling result is recorded as I fill (x,y);

[0264] Then the filled image I fill (x, y) uses 8-pixel connected area detection and detects the contour filling block with the largest area as the upper half area I′ of the sample fill (x,y), since the processed contours are all expanded, I′ fill (x,y) is then corroded using the following formula to form the final upper half of the sample:

[0265]

[0266] Among them, M se1 It is a planar disk-shaped structural element with a radius of 1.

[0267] B. Contour extraction of the lower half of the sample area

[0268] The modulation degree of the stripes can be obtained by the following formula:

[0269]

[0270] The obtained modulation index M(x,y) is binarized using formula (13) to obtain the entire area of ​​the primary sample including the shadow area.

[0271] Due to the difference in surface characteristics between the upper and lower regions of the sample, the grayscale information of the entire region of the primary sample is expressed as a "contour" in the upper region and as a "region" in the lower region. Therefore, the two regions can be separated by "eroding" the distribution map of the entire region of the primary sample. However, since the reflectivity of region B is lower than that of region C, the obtained primary sample overall map uses extremely low values ​​to describe the sample information. Therefore, in order to achieve the erosion effect during image processing, the image should be expanded. Then, the obtained modulation extreme value segmentation map is "eroded" using formula (1), and the M used is se is a planar disk-shaped structural element with a radius of 2, and the processing result is recorded as M bi (x,y);

[0272] At the same time, in order to eliminate the regional “corrosion” effect caused by the above formula, formula (32) is used to convert M bi (x,y) is "expanded", and the M used is se1 is a planar disk-shaped structural element with a radius of 2, and the result is recorded as M close (x,y)

[0273] The processed primary lower half area contour M close(x, y) uses the Canny operator to perform edge detection, and similarly performs a closing operation on the primary edge contour after detection to bridge the contour cracks. The closed contour map is recorded as M eg (x, y), and in order to eliminate the background noise contour, the 4-pixel connected detection M eg All contours in (x,y) are recorded as p=1,2,3,···and the formula (2) Replace with To M eg (x,y) is used to fill the contour, and the filling result is recorded as M fill (x,y);

[0274] Then the padded image M fill (x, y) performs an 8-neighborhood connected region search, and takes the largest connected region as the lower half of the sample, denoted as M f ' ill (x, y), considering that edge detection will expand the lower half area, the area is corroded using formula (32), and the corroded area is used as the final lower half area of ​​the sample I hand (x,y):

[0275] Finally, the two areas of the sample are combined using the following formula to form the final bottom area of ​​the sample.

[0276]

[0277] Get the bottom area I of the sample mask (x, y), the camera system parameters are obtained using the camera calibration algorithm, and the image two-dimensional coordinates are converted into spatial coordinates through the following coordinate mapping relationship;

[0278]

[0279] Where [xy 1] T Represents the homogeneous representation of the pixel coordinates of the captured image, [x w y w 1] T Represents the homogeneous representation of the world coordinates obtained by mapping, M ext 、M ins Represent the external parameters and internal parameters of the camera respectively. After obtaining the world coordinates of each pixel, calculate the area corresponding to a single pixel. Assuming that the area of ​​a single pixel unit is Figure 9 The shaded area S;

[0280] When solving the unit area S, you need to first calculate the four vertices of the quadrilateral corresponding to the point The coordinate value of the coordinate value can generally be calculated using the bilinear interpolation method. Set the unit vertex at the center of the four neighboring pixels, then It can be expressed as

[0281]

[0282] Similarly, the coordinate values ​​of the other three vertices can be obtained. Then, the quadrilateral is approximated as a rectangle and the value calculated using the following formula is used as the actual physical area of ​​the pixel block. The area of ​​all unit blocks is calculated and the corresponding block volume is obtained. Finally, the value calculated using formula (30) is used as the final sample volume:

[0283]

[0284] To verify the effectiveness of the robotic arm positioning and grasping module, an experimental system was constructed according to the system design requirements. In this system, the camera is fixed to the grasping tool at the end of the robotic arm. The axis plane of the robotic arm end is required to remain parallel to the workbench throughout the entire process. The coordinate axes of the robotic arm base and the camera coordinate axes remain parallel throughout the entire system operation.

[0285] Calculation and error analysis of hand-eye conversion matrix

[0286] Place a 230mm×130mm chessboard on the platform and adjust the position so that it is completely within the camera's field of view. Set the CCD resolution to 768×1024, capture the first image as a reference image, and record the corresponding end-of-arm posture on the teach pendant. Then use the teach pendant to move the robot arm, keeping the chessboard completely within the camera's field of view during the movement, capture the chessboard image and record the corresponding end-of-arm position posture at the current moment. Repeat this operation 15 times. Part of the chessboard image captured in this process is shown below. Figure 10 shown.

[0287] The captured chessboard image is sent to the computer for camera calibration to obtain its internal and external parameters, which are then converted into a homogeneous matrix representation; then the corresponding robot arm end posture is converted into a corresponding homogeneous matrix representation based on the Euler angle transformation, and the hand-eye conversion matrix is ​​calculated. The calculation results of this system are

[0288]

[0289] To quantitatively evaluate the accuracy of the hand-eye conversion matrix, the robotic arm is moved to a working posture and a camera is used to capture images. Pixels in the captured images are randomly selected. The world distances between known points on the checkerboard grid in the reference image are used as standard values ​​to calculate the world distances of the selected point groups after mapping. The relative distance error is also calculated, as shown in Table 1.

[0290] Table 1 Hand-eye matrix mapping distance deviation results

[0291]

[0292]

[0293] As can be seen from the table, the calibration distances calculated in the X and Y directions respectively have calibration accuracy errors that can be controlled within ±2mm, and for the actual distance difference of the selected points, the relative distance error after calibration can be maintained within 1%, which can meet the design requirements of this system.

[0294] Molten steel sample feature recognition

[0295] According to the feature recognition algorithm proposed in this paper, Figure 11 (a) The placed samples are subjected to feature extraction. The specific process is as follows Figure 11 As shown in (b)-(i).

[0296] In order to verify the effectiveness of feature extraction, the center points of the two areas are measured with a vernier caliper as standard points, and marks are manually affixed on the standard points. The molten steel sample is placed on the reference surface, the sample is placed at a given deflection position, and the position of the sample is fixed. The camera is used to take a picture as a standard image, as shown in the figure below. Figure 12 (a); After removing the label, take the picture again and extract the features of the image. The extraction results are as follows Figure 12 As shown in (b)-(c).

[0297] Change the sample placement and repeat the above operation. Take the center position of the label area in the captured standard image as the coordinate standard value. The deviation between it and the identified feature point is shown in Table 2.

[0298] Table 2 Feature recognition positioning coordinate deviation results

[0299]

[0300] As shown in the table above, the positioning coordinate error obtained using feature recognition can be controlled within ±10 pixels. For the "handle" region of the molten steel sample in the u direction, the effective grasping pixel range is ≥10 pixels. The corresponding deflection values ​​obtained using feature point conversion are within ±2 rad of the standard deflection angle, effectively meeting the positioning and grasping requirements. This also proves that the angle description based on the corresponding feature points in the two regions is effective.

[0301] Finally, to simulate the cooling process of the molten steel sample, the molten steel sample to be measured was placed in a semi-enclosed container and filled with water to cover the sample. The sample was randomly positioned and the positioning coordinates obtained by the feature recognition algorithm were compared with the center position of the marked area to describe the effectiveness of the positioning point. The feature points identified in the two areas and the center position of the marked area were converted into deflection values ​​and compared to describe the effectiveness of the posture deflection. The results are shown in Table 3.

[0302] Table 3 Deviation calculation results

[0303]

[0304]

[0305] It can be seen from the table that the calculated coordinate deviation and angle deviation can meet the positioning and grasping requirements, which proves that the feature extraction algorithm proposed in this chapter is effective in processing molten steel samples immersed in water.

[0306] To verify the effectiveness of the fringe projection-based defect detection module for molten steel samples, an experimental system was constructed according to the system design requirements. The projector and CCD were positioned at a specific angle, with the plane formed by their optical axes coplanar with the sample's axis of symmetry. The projector and camera were positioned while reducing background brightness, ensuring shadows appeared in the upper half of the sample. Finally, appropriate exposure parameters were set to meet the measurement system's requirements.

[0307] According to the label design and phase calculation requirements, the designed labels are as follows Figure 13 As shown in (a), some of the results after embedding into stripes are as follows Figure 13 As shown in (b)-(d).

[0308] The molten steel sample to be tested is placed on the testing platform. The resolutions of the DLP projector and are 768×1024 and 768×1024 respectively. After the designed stripe images are combined into a video frame sequence, they are projected onto the molten steel sample using the projector, and a series of images are captured using the CCD and stored in the computer. The captured molten steel sample is as follows: Figure 14 As shown in (a), the fringe image is Figure 14 As shown in (b)-(d), the reference fringe pattern is pre-photographed and saved in the memory after the system is built.

[0309] Verification of the accuracy of label recognition in timing control

[0310] In order to verify the accuracy of label recognition, the above-collected images are processed for recognition. The recognition process and processing diagram are shown in Table 4.

[0311] Table 4 Label recognition process

[0312]

[0313]

[0314] The recognition results obtained from Table 4 are accurate. To avoid the accidental nature of the experiment, the experiment was repeated 100 times, and the recognition accuracy reached 100%. This shows that the label recognition method proposed in the present invention is effective.

[0315] Verification of the validity of phase calculation

[0316] In order to verify the effectiveness of the phase calculation method adopted by the present invention, Figure 14 (a) The surface profile of the molten steel sample was measured.

[0317] The phase calculation method uses a fringe image with a frequency of f1 = 1 / 30, a five-step phase shift method to calculate the phase, and a diamond phase unwrapping algorithm to calculate the continuous phase; a fringe image with a frequency of f2 = 1 / 15, a five-step phase shift method to calculate the truncated phase, and a diamond phase unwrapping algorithm to calculate the continuous phase; the continuous phase obtained by the frequency f1 = 1 / 30 guides the truncated phase calculated by the fringe image with a frequency of f2 = 1 / 15; the fringe image with a frequency of f1′ = 1 / 31, the five-step phase shift method to calculate the truncated phase, and the dual-frequency heterodyne method is used to calculate the continuous phase in combination with the truncated phase calculated by the fringe image with a frequency of f1 = 1 / 30; and the phase calculation method proposed by the present invention is used to obtain the following results. Figure 15 In order to further highlight the effect distribution of various phase calculation methods, different result perspectives are reselected, such as Figure 16 shown.

[0318] from Figure 15 and Figure 16 It can be seen that the surface profile obtained by measuring the low-frequency fringes with a frequency of f1 will have the problem of phase unwrapping errors due to high reflectivity and shadow areas, and the error area will be further increased by measuring the fringes with a higher frequency of f2. Therefore, the measurement method using the low-frequency f1 to guide the high-frequency f2 will further transfer the error of f1. The dual-frequency heterodyne method and the dual-frequency heterodyne-guided high-frequency method are not affected by this problem and can achieve profile measurement. Figure 16 It is noted that for the upper half of the object being measured, the result (e2) measured using the phase calculation method proposed in the present invention is smoother and has a better effect than the result (d2) of the dual-frequency heterodyne measurement. The reason is that the dual-frequency heterodyne method requires that the two frequencies be selected to be low and close, which limits the accuracy.

[0319] In order to further evaluate the accuracy of the phase calculation method proposed in the present invention, the present invention uses a standard flat block gauge with a height of 15 mm as the object to be measured, such as Figure 17As shown in (a), the captured fringe pattern is as follows Figure 17 (b)-17(d) are shown. The above-mentioned method is used to measure again, and the effective area of ​​the measurement result is selected and the mean value of the area is calculated. The steepness of the five methods is verified by comparing with the mean value. The measurement results and error distribution are shown in Figure 18 、 19 The corresponding RMS error values ​​are shown in Table 5.

[0320] Table 5 Error value comparison

[0321] Phase Method RMSE <![CDATA[Five-step phase-shifting method (fringe frequency is f1)]]> 1.8600 <![CDATA[Five-step phase-shifting method (fringe frequency is f2)]]> 1.8453 <![CDATA[Low-frequency guiding high-frequency method (fringe frequency f1 guides fringe frequency f2)]]> 1.8780 <![CDATA[Dual-frequency heterodyne method (fringe frequency is f1, f 11 synthetic frequency)]]> 1.4521 <![CDATA[The algorithm of the present invention (the fringe frequency is f1, f 11 After synthesizing the frequency, it guides f2)]]> 1.4472

[0322] Table 5 further illustrates that the phase errors obtained using low-frequency and high-frequency fringes alone are large. Therefore, using the low-frequency method to guide the high-frequency phase calculation further propagates the error in the fringe frequency f1. The algorithm of the present invention captures more detail than the dual-frequency heterodyne algorithm, achieving better measurement results. In the algorithm of the present invention used in the experiment, the low-frequency f1 and the high-frequency f2 are in a two-harmonic relationship, and the measurement effect is further improved with increasing the harmonic coefficient.

[0323] Volume integration results

[0324] Before selecting the bottom area of ​​the sample height, the captured stripe image is cropped first, and the cropped result is calculated as the modulation image and then combined with the edge detection algorithm designed by the present invention to select the bottom area. The upper and lower areas are selected as follows: Figure 20 and Figure 21 shown.

[0325] The result of splicing the selected upper and lower areas has the phenomenon of unclosed connected areas and regional faults. The present invention performs a closing operation on the spliced ​​result and then fills it. The corrected result is used as the final bottom area of ​​the sample. The specific process is as follows Figure 22 shown.

[0326] After obtaining the bottom area of ​​the molten steel sample, the volume of the sample is calculated. The method proposed by the present invention is used to measure the volume of the standard molten steel sample multiple times and calculate the average value as the standard volume. If the volume value of the measured sample exceeds the relative volume error of the standard volume by 5%, the sample is judged to be unqualified. Molten steel samples with different defect levels are selected and the pass level of the samples is judged after the algorithm flow of the present invention. The measured molten steel samples with different defect levels are as follows: Figure 23 As shown in the figure, the measured surface distribution is as follows Figure 24 The test results are shown in Table 6.

[0327] Table 6 Different sample testing results

[0328] Molten steel sample to be tested <![CDATA[Volume / mm 3 > <![CDATA[Standard volume mm 3 > Relative volume error Qualification level 1 8929.91 8230.00 8.50％ Unqualified 2 8474.28 8230.00 2.96％ qualified 3 6366.20 8230.00 22.65％ Unqualified 4 8498.11 8230.00 3.26％ qualified 5 7952.25 8230.00 3.37％ qualified 6 8528.35 8230.00 3.63％ qualified 7 7927.71 8230.00 3.67％ qualified 8 6379.81 8230.00 22.48％ Unqualified 9 5965.74 8230.00 27.51％ Unqualified 10 7954.98 8230.00 3.34％ qualified

[0329] As can be seen from the table above, the accuracy rate of testing molten steel samples without defects or overflow is 100%, and the accuracy rate of testing samples with defects or overflow is also 100%, which can meet the testing requirements.

[0330] This invention proposes an automated molten steel sampling device based on a machine vision system, designed for process control in steelmaking and improving the final quality of steel. This system investigates robotic arm positioning and grasping of molten steel samples in a cooling pool, combining image recognition. The system analyzes the sample contour characteristics and extracts location and grasping features based on regions. A robotic arm calibration algorithm is then used to map the resulting image features from camera coordinates to robotic arm positioning and grasping coordinates. Hand-eye conversion matrix experiments verify that the calibration results meet grasping requirements, and image feature recognition experiments demonstrate the effectiveness and accuracy of the feature extraction algorithm. The system then conducts an acceptance assessment of samples placed on a testing platform. This assessment process first addresses the timing control challenges of fringe projection in the automated system, reducing design costs. Furthermore, a phase calculation method using dual low-frequency guidance and high-frequency guidance is employed to improve detection accuracy. Finally, the base area of ​​the sample is selected and its volume is calculated using the features of different regions of the sample to determine its acceptance. Label recognition experiments verify the effectiveness and accuracy of the recognition. Comparative phase calculation experiments demonstrate that the proposed method achieves improved accuracy while restoring the sample contour. Defect detection experiments have confirmed that the system designed in the present invention can achieve the detection effect and meet the detection requirements.

[0331] The embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. An automatic molten steel sampling device based on a machine vision system, characterized by: The system comprises a first robotic arm, an automatic tray, a cooling water pool, a second robotic arm, and a molten steel sample defect detection module based on fringe projection. The first robotic arm is located between the furnace and the automatic tray. The cooling water pool is located on one side of the automatic tray. After the automatic tray is flipped, the molten steel sample can fall into the cooling water pool. The second robotic arm is located between the cooling water pool and the molten steel sample defect detection module based on fringe projection. An industrial spoon is fixed at the end of the first robotic arm, and multiple molten steel sample molds are set on the automatic tray. The first robotic arm controls the industrial spoon to take samples from the furnace and pour them into the molten steel sample molds on the automatic tray. The automatic tray is equipped with a flipping device. After the automatic tray flips, the molten steel samples are poured into the cooling water pool. A manipulator is installed at the end of the second robotic arm. The second robotic arm controls the manipulator to grab the molten steel samples in the cooling water pool and send them to the molten steel sample defect detection module based on fringe projection. The specific working process of this device is as follows: Step 1) An industrial spoon is attached to the end of a robotic arm without a gripper. Using positioning technology, the end of the robotic arm is moved to the furnace mouth area, and its joints are programmed to remove a small amount of molten steel. Next, the end of the robotic arm is positioned at the center of the sample mold groove area, and the robotic arm joints are moved and programmed to pour the removed molten steel into the mold groove. Step 2) Set the automatic tray to flip after a certain period of time to pour the molded sample into the cooling pool; Step 3) The sample in the cooling pool is positioned using a robotic arm grasping module. After positioning using machine vision recognition technology, the robotic arm equipped with a manipulator is controlled to the positioned sample point and an action signal is given to remove the sample and place it on the subsequent inspection workbench operation; The robot arm positioning and grasping module includes a workbench, a robot arm and a CCD. The CCD is fixed to the end of the robot arm. The CCD is used to capture the image of the molten steel sample on the workbench, and the grasping position and grasping direction are obtained through computer processing, and converted into corresponding posture signals for the robot arm to grasp; First, the collected target sample image is processed using the Canny operator to obtain the primary edge contour of the sample. Then, the obtained edge contour is sequentially expanded and filled to obtain the upper half area of ​​the sample. Finally, the upper half area of ​​the sample is eliminated from the entire area of ​​the sample. The remaining area is taken as the lower half area of ​​the sample, and the centroid of the lower half area is used as the positioning point of the target sample. The deflection direction uses the length of the "vector" directed line segment to define the relationship between two pixel points, and combines the mathematical concept of "angle" as the posture of the sample at the current moment. The center of the fitted circle in the upper half of the area is selected as the representative point of the area, and the positioning grasping point in the lower half of the area is selected as the representative point of the area. The deflection direction is determined by these two representative points. After obtaining the positioning direction and deflection direction from the image captured by the camera fixed at the end of the robotic arm, the camera calibration and hand-eye calibration technology are used to convert them into positioning and grasping points in the coordinate system of the robotic arm base, and the end effector movement is directed to achieve object positioning and grasping; Step 4) After placing the molten steel sample on the inspection workbench, a fringe projection-based molten steel sample defect detection module is used to obtain the 3D surface distribution of the molten steel sample through fringe projection profilometry, and the volume is calculated to determine whether the sample is qualified; The fringe projection-based molten steel sample defect detection module includes a detection workbench, a digital projector, and a CCD. The molten steel sample to be detected is placed on the detection workbench. The digital projector is used to project multiple frames of fringes onto the detection workbench. The CCD is used to continuously collect multiple frames of deformed fringe images. The collected fringe sets are then sent to a computer for selection and time sequence restoration. The effective fringe images are then subjected to phase calculation, phase unwrapping, phase height mapping, and other algorithms to reconstruct the 3D surface distribution of the sample. Finally, the sample volume is calculated based on the 3D surface distribution to determine whether the sample is qualified. The specific steps are as follows: Step 4.1) After placing the molten steel sample on the inspection workbench, a flash consisting of multiple frames of fringe patterns with a certain phase shift is played in a loop using a digital projector at a frame rate of 20 fps. The CCD is controlled to continuously capture multiple frames of the deformed fringe pattern at a frame rate of 30 fps and a capture duration of 2 s. The captured fringe patterns are then stored in memory. Step 4.2) The collected fringe set is sent to a computer for label recognition and sorting to achieve fringe selection and time sequence restoration. The effective fringe image is then subjected to phase calculation, phase unwrapping, and phase height mapping to reconstruct the 3D surface distribution of the sample; Step 4.3) The effective bottom area of ​​the captured image of the molten steel sample to be tested is selected, and the volume of the sample is calculated in combination with the obtained 3D surface distribution to determine whether the sample is qualified.

2. The automatic molten steel sampling device based on a machine vision system according to claim 1, characterized in that: The specific steps of the robotic arm positioning and grasping module are as follows: 3.

1. Feature recognition of molten steel samples Feature recognition of molten steel samples involves two steps: target sample pose acquisition and feature extraction. The target sample pose acquisition is performed using a camera fixed at the end of the robotic arm. Feature extraction is combined with image recognition technology to process the acquired image. The specific recognition technologies are as follows: A. Positioning coordinates The molten steel sample is divided into two areas: the upper area "cylinder" and the lower area "handle". The upper area of ​​the sample is first identified, and the identified upper area is used to assist in obtaining the lower area. The steps for identifying the upper area are as follows: First, the collected target sample image is processed using the Canny operator to obtain the primary edge contour of the sample I e (x, y), in order to avoid unclosed cracks in the obtained primary edge contour, which may cause errors in subsequent region selection, the obtained edge contour is expanded by the following formula: Among them, M se It is a 3×3 square structural element. In order to eliminate the background noise contour in the primary edge contour map, the 4-pixel connected area is first used to detect the edge contour map of the sample I e All closed contours in ′(x,y) are denoted as Fill the detected contour using the following formula: Then fill the sample outline I pad (x, y) is then tested for 8-pixel connected regions, and the contour filling result with the largest detection area is taken as the overall area of ​​the sample, denoted as I outline (x, y), the obtained sample area is fitted with a standard circle using the invariant kernel circle detection algorithm, and the center coordinates of the fitted circle are marked as (x cir ,y cir ), the radius is recorded as R cir , considering that the circular area obtained by fitting may not be able to completely cover I outline The upper half of the area represented by (x, y) is expanded by 10 pixels to the radius of the fitted circle, which is recorded as R′ cir , the expanded result is used as the upper half area outline of the molten steel sample, and the following formula is used to fill it as the upper half area of ​​the sample; Finally, the upper half area of ​​the sample and the entire area of ​​the sample are eliminated using the following formula, and the remaining area is used as the "handle" area of ​​the sample; Considering the symmetry of the molten steel sample structure, finding the center of the region as the positioning point can better meet the system requirements. The center of mass is short for the center of mass, which is more effective for finding the center point of irregular objects with uniform mass distribution. Therefore, the center of mass of the "handle" area is used as the positioning point of the target sample. The center of mass formula is transformed, replacing the mass of the object point with the grayscale value of the image point. The positioning grasping point can be solved using the following formula: Where D represents the "handle" area of ​​the sample I handle (x,y), x i 、y i Represents the coordinate value of the regional image point, m i Represents the grayscale value of the image point area; B. Deflection direction For two-dimensional images, the length of the "vector" directed line segment is used to define the relationship between two pixels, and the concept of "angle" in mathematics is combined as the posture of the sample at the current moment. For the molten steel sample that has been divided into two areas, the feature points of the corresponding areas are selected as the representation points, and the center of the fitting circle (x cir ,y cir ) as the representation point of the area, and the positioning grasping point (x m ,y m ) as the representation point of the area, and use the following formula to get the vector representation between the two representation points; To establish the transformation between vector and angle, another vector parallel to the image plane axis is introduced Use the following formula to convert it into angle expression: Where round{·} is the rounding function; 3.

2. Robotic Arm Positioning Based on Image Feature Points After obtaining the positioning direction and deflection direction from the image captured by the camera fixed at the end of the robotic arm, the camera calibration and hand-eye calibration technology are used to convert them into positioning and grasping points in the coordinate system of the robotic arm base, and the end effector movement is directed to achieve object positioning and grasping; A. Positioning grab point conversion Among them, z c is the positioning point p pix (x m ,y m ) in the camera coordinate system (X C ,Y C ,Z C ) under the value of Z, M′ ins is the intrinsic parameter matrix obtained by camera calibration, is the hand-eye calibration matrix, Represents the mapping matrix between the terminal coordinates and the base coordinates at the i-th moment; B. Deflection direction conversion Since the increment of the rotation angle of the manipulator axis joint is linearly related to the increment of the rotation value, the initial position of the axis joint rotation is O If the reference position is the same and the rotation direction is the same, then read the initial rotation angle value θ start , combined with the following formula to obtain the axis deviation angle θ H i H =θ O +θ start (9)。 3. The automatic molten steel sampling device based on a machine vision system according to claim 2, characterized in that: Step 4.1) Use the CCD to capture the labeled fringes and perform sorting operations. The specific steps are as follows: After capturing a labeled stripe I cap (x, y) uses the invariant kernel circle detection algorithm to fit the standard circle. (x, y) represents the pixel coordinates of the captured image, and the center coordinates of the fitted circle are marked as (x circle ,y circle ), the radius is recorded as R, draw the fitted circle on the image with the same size as the label stripe, replace the center coordinates and radius of formula (3) with the current recognition result, and fill the circle as the label template M circle (x,y); In order to obtain the label embedded in the current label stripe, the label area in the label stripe is first retained using the following formula, and the retained label information is recorded as I label (x,y); Then the label information obtained by formula (10) is extracted using the following formula, and the extracted label is recorded as I′ label (x,y), Since the template matching method is used to realize the sorting of label stripes, it is required that the two label images to be matched have the same size and the same grayscale information representation method. Therefore, the extracted labels should be scaled to the designed label image size first. Considering that the designed label information is single, the nearest neighbor interpolation is used to realize the scaling of the labels. The following formula represents the mapping relationship between the label pixels before scaling and the label pixels after scaling: Among them, (X, Y) represents the image pixels after scaling, I grade (X, Y) represents the scaled image, w and h represent the size of the image before scaling, and W and H represent the size of the image after scaling; Then, the scaled labels are binarized using the following formula to obtain the final processed label image I: num (X,Y) Among them, thresh is the set threshold, and the threshold selection is calculated using the algorithm of minimizing the intra-class variance. Compare the numbered drawing to be processed with the numbered drawing of the design Use the following formula to perform template matching. The template with the smallest difference among all matching results is the best matching template, and the label sequence corresponding to the template is recorded as the phase shift sequence of the fringe; Finally, determine whether the recognized labels are repeated. If not, name the label stripe. Otherwise, do not name it. Repeat the above operation to identify the remaining label stripes until all images are recognized. If all designed labels are recognized once, proceed to the next step. Otherwise, it indicates that the image data acquisition is not completed.

4. The automatic molten steel sampling device based on a machine vision system according to claim 3, characterized in that: The specific process of phase calculation, phase unwrapping, and phase height mapping in step 4.2) to reconstruct the 3D surface distribution of the sample is as follows: Three frequency stripes are projected, and the phases of the two lower frequency stripes are calculated using the dual-frequency heterodyne method. After the low-frequency phase is obtained, it is used to guide the phase calculation of the high-frequency stripes. This method can effectively improve the accuracy of phase calculation. The phase shift step of this method is set to five steps, with frequencies f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, f12, f13, f14, f15, f16, f17, f18, f19, f20, f21, f22, f23, f24, f35, f19, f25, f26, f36, f1 1′ , f2, and the relationship between the three frequencies is expressed as: The transmittance expression of the 15 stripes generated by the computer is: Among them, (x P ,y P ) represents the pixel coordinates of the projected image, a and b are constants; The 15 stripes are combined into a flash and captured by CCD. After label recognition, the effective stripe expression extracted is: Where A(x,y) represents the average intensity, B(x,y) represents the intensity modulation, and Φ(x,y) represents the fringe phase after being modulated by the object. The truncated phase is then: Among them, φ i (x,y) represents the truncated phase of different frequency fringes, then the frequencies are f1, f 1′ The truncated phases of f2 fringes are represented by φ1(x,y) and φ 1′ (x,y), φ2(x,y); The continuous phase difference between the two 合 (x,y) can be obtained based on dual-frequency heterodyning: Among them, Φ1(x,y), Φ 1′ (x,y) are frequencies f1 and f 1′ The fringe phase after being modulated by the object, similarly, the frequency f 合 It can be obtained by the following formula For a point on the measurement surface, the fringe phase after being modulated by the object can be expressed as Φ j (x,y)=2πK j (x,y),j=1,1′ (21) Among them, K1(x,y), K 1′ (x,y) represent the fringe frequencies f1 and f 1′ The fringe level when , the fringe level has the following relationship Among them, k j (x, y) represents the phase order. When the relative positions of the camera, projector and the object to be measured are fixed, the same point on the object has the same position on the grating images of different periods, so we can get Substituting formula (21) into formula (23), we get Combining formula (19) and formula (24), we can get the expression of the fringe phase Φ1(x,y) after being modulated by the object: or Φ1(x,y)=2π·k1(x,y)+φ1(x,y) (27) Where k1(x,y) is the phase order of the truncated phase at a certain point when the fringe frequency is f1; After obtaining Φ1(x,y), the phase order k2(x,y) of the truncated phase of the frequency f2 fringe can be calculated based on the relationship between the high-frequency and low-frequency absolute phases and the frequency: Substituting k2(x,y) into the relationship between truncated phase and continuous phase can obtain the continuous phase of high-frequency fringes. The expanded continuous phase can only reflect the surface contour of a three-dimensional object. To obtain the actual height of the object, the specific mapping relationship between the two must be known, namely the phase-height mapping matrix. This matrix requires moving the reference surface back and forth a specified distance m times, where m ≥ 3, and projecting and collecting fringes after each movement. The established relationship is usually written as: Where Z(x,y) represents the actual object height to be solved, Φ(x,y) represents the unwrapped phase result obtained by solving the deformed fringes modulated by the object, and a(x,y), b(x,y), and c(x,y) are the system mapping parameters to be solved, which can be calculated using the least squares method.

5. The automatic molten steel sampling device based on a machine vision system according to claim 4, characterized in that: Step 4.3) Select the effective bottom area of ​​the sample image captured by the module and calculate the sample volume based on the 3D surface distribution. The specific steps are as follows: According to the 3D contour distribution of the molten steel sample, the height distribution Z(x w ,y w ), the volume of the final molten steel sample can be obtained by the following integral formula V=∫∫Z(x w ,the w )dx w of w (30) In the actual volume calculation process, in order to save computing resources and remove the interference of background noise and inevitable shadows on volume calculation, the pixel coordinates (x, y) are converted into world coordinates (x w ,y w ), the effective area of ​​the bottom contour of the molten steel sample was extracted first, and the contour of the molten steel sample was divided into two parts for extraction; A. Contour extraction of the upper half of the sample area When using fringe stagger to extract the contour of the upper half of the sample area, the residual image is obtained by subtracting the sample image taken by the measurement system from the undeformed reference fringe image obtained when no sample is placed on the workbench to effectively obtain the stagger information. The residual image can be obtained by the following formula: Among them, I1′ n (x, y) represents the undeformed reference fringe pattern obtained when the workbench is not placed with a sample at the frequency f1, and the residual mean image I is obtained. ref (x, y), the Canny operator is used to process the primary upper half area edge contour I eg (x, y), in order to avoid the error of region selection caused by the unclosed edge contour, the M se Replace it with a planar disc-shaped structural element with a radius of 1 and perform expansion processing. The processing result is recorded as I ei (x,y); Due to the uneven illumination distribution caused by the working environment, the expanded primary edge contour image I ei (x, y) will contain a few noise contours. In order to eliminate these invalid contours, we first use 4-pixel connected area detection I ei All closed contours in (x,y) are denoted as q=1,2,3,···, and replace the Replace with For the primary edge contour image I ei (x,y) is filled, and the filling result is recorded as I fill (x,y); Then the filled image I fill (x, y) uses 8-pixel connected area detection and detects the contour filling block with the largest area as the upper half area I′ of the sample fill (x,y), since the processed contours are all expanded, I′ fill (x,y) is then corroded using the following formula to form the final upper half of the sample: Among them, M se1 It is a planar disk-shaped structural element with a radius of 1; B. Contour extraction of the lower half of the sample area The modulation degree of the stripes can be obtained by the following formula: The obtained modulation M(x,y) is binarized using formula (13) to obtain the primary sample overall area M including the shadow area bina (x,y); Due to the difference in surface characteristics between the upper and lower regions of the sample, the grayscale information of the entire region of the primary sample is expressed as "contour" in the upper region and as "region" in the lower region. Therefore, the two regions can be separated by "eroding" the distribution map of the entire region of the primary sample. In order to achieve the erosion effect during image processing, the image should be expanded. Then, the obtained modulation extreme value segmentation map is "eroded" by formula (1). The M used is se is a planar disk-shaped structural element with a radius of 2, and the processing result is recorded as M bi (x,y); At the same time, in order to eliminate the regional "corrosion" effect caused by the above formula, formula (32) is used to convert M bi (x,y) is "expanded", and the M used is se1 is a planar disk-shaped structural element with a radius of 2, and the result is recorded as M close (x,y); The processed primary lower half area contour M close (x, y) uses the Canny operator to perform edge detection, and similarly performs a closing operation on the primary edge contour after detection to bridge the contour cracks. The closed contour map is recorded as M eg (x, y), and in order to eliminate the background noise contour, the 4-pixel connected detection M eg All contours in (x,y) are recorded as p=1,2,3,···, and replace the Replace with To M eg (x,y) is used to fill the contour, and the filling result is recorded as M fill (x,y); Then the padded image M fill (x, y) performs an 8-neighborhood connected region search and takes the largest connected region as the lower half of the sample, denoted as M′ fill (x, y), considering that edge detection will expand the lower half area, the area is corroded using formula (32), and the corroded area is used as the final lower half area of ​​the sample I hand (x,y); Finally, the two areas of the sample are combined using the following formula as the final bottom area of ​​the sample; Get the bottom area I of the sample mask (x, y), the camera system parameters are obtained using the camera calibration algorithm, and the image two-dimensional coordinates are converted into spatial coordinates through the following coordinate mapping relationship; Where [xy 1] T Represents the homogeneous representation of the pixel coordinates of the captured image, [x w y w 1] T Represents the homogeneous representation of the world coordinates obtained by mapping, M ext 、M ins Represent the external parameters and intrinsic parameters of the camera respectively. After obtaining the world coordinates of each pixel, calculate the area corresponding to a single pixel. Assume that the area of ​​a single pixel unit is the shaded area S; When solving the unit area S, you need to first calculate the four vertices of the quadrilateral corresponding to the point The coordinate value of the coordinate value can generally be calculated using the bilinear interpolation method. Set the unit vertex at the center of the four neighboring pixels, then It can be expressed as Similarly, the coordinate values ​​of the other three vertices can be obtained. Then, the quadrilateral is approximated as a rectangle and the value calculated using the following formula is used as the actual physical area of ​​the unit. The area of ​​all unit blocks is calculated, and the corresponding block volume is obtained. Finally, the value calculated using formula (30) is used as the final sample volume;

Images