Visual guidance intelligent casting method and system

Abstract

The invention provides a vision-guided intelligent casting method and system. The casting method comprises the following steps that firstly, color and depth information of carbon bowls and steel forks is obtained through an image obtaining module; 2, performing three-dimensional reconstruction on the production scene through a three-dimensional simulation module; and step 3, a casting machine is controlled to move and topple through a PLC control module to realize product casting, and the step 1 comprises the following detailed steps: S1, a color picture flow and a depth camera flow of a camera are obtained through api provided by ros1, pixel-level alignment of the color picture flow and the depth camera flow can be realized, and a depth value d can be found for each rgb point in an image space; s2, dense point cloud reconstruction: for each point p-(u, v) in an image space, a space point P-(x, y, z) in a camera coordinate system corresponds to the point p-(u, v); compared with a manual operation mode of a worker, the production efficiency is improved, potential safety hazards of a high-temperature environment are reduced, in addition, badness of products can be reduced, and casting of aluminum products is facilitated.

Description

Technical Field

[0001] The present invention relates to the fields of industrial vision, machine perception, and intelligent casting technology, and in particular to a vision-guided intelligent casting method and system. Background Art

[0002] Casting is a manufacturing process, commonly used for forming metal, in which molten metal is poured into a mold and then allowed to cool and solidify into the desired shape. Casting can be used to create a variety of items, from small jewelry to large mechanical parts.

[0003] Aluminum plants need to use a casting process to produce aluminum products during the production process. The requirement is to pour molten aluminum into the gap between the charcoal bowl and the steel fork. After the aluminum liquid cools, the cast aluminum product is obtained. In the existing technology, the casting of the aluminum product is done manually, that is, the worker manually controls the movement of the casting machine and casts the molten aluminum into the gap between the charcoal bowl and the steel fork through the pouring gate.

[0004] However, manual operation by workers will lead to low production efficiency and high-temperature environments pose safety hazards. In addition, due to the high temperature of the casting machine, it is difficult for humans to observe directly with their eyes. If less aluminum liquid is poured into certain areas during the casting process, it will cause depressions in these areas of the cast aluminum product, which will lead to defective products and is not conducive to the casting of aluminum products. Therefore, we propose a vision-guided intelligent casting method and system. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the existing technology and provide a visually guided intelligent casting method. The color and depth information of the charcoal bowl and steel fork are obtained through an image acquisition module, the production scene is reconstructed in three dimensions through a three-dimensional simulation module, and the movement and pouring of the casting machine are controlled by a PLC control module to realize the casting of the product. Compared with the manual operation method of workers, the method improves production efficiency and reduces the safety hazards in high-temperature environments. In addition, it also reduces product defects and is conducive to the casting of aluminum products.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A vision-guided intelligent casting method comprises the following steps:

[0008] Step 1: Obtain the color and depth information of the charcoal bowl and steel fork through the image acquisition module;

[0009] Step 2: Reconstruct the production scene in 3D through the 3D simulation module;

[0010] Step 3: Use the PLC control module to control the casting machine's movement and pouring to achieve product casting.

[0011] The detailed steps of step one are as follows:

[0012] S1. Obtain the camera's color image stream and depth camera stream through the API provided by rosl. The two can be aligned at the pixel level: for each RGB point in the image space, a depth value d can be found;

[0013] S2. Dense point cloud reconstruction: For each point p-(u, v) in the image space, there is a corresponding spatial point P-(x, y, z) in the camera coordinate system. Assuming that the scaling factor from the camera space scale to the real space scale is s, we have

[0014] Pz=d / s

[0015] s is calculated as follows: obtain the original point cloud and save it; use the pcl_viewer tool to measure the length s1 of the carbon block in the point cloud, and use a tape measure to measure the actual size s2 of the carbon block, then we have

[0016] s=s2 / s1

[0017] Px=(pu-cx)*Pz / fx

[0018] Py=(pv-cy)*Pz / fy

[0019] The camera intrinsic parameters cx, cy, fx, and fy can be calibrated using the checkerboard method or directly obtained using the command-line tool provided by RealSense.

[0020] S3. Point cloud stitching: The original point cloud consists of four pieces. We use a self-developed C++ tool (based on PCL, simple to implement, outputs four transformation matrices, details omitted) to visualize them, perform rotation and translation operations, and then transform them into the same coordinate system.

[0021] S4. Determination of the casting machine mouth: Since the relative position between the casting machine mouth and the camera is fixed, you can use a tape measure to roughly measure the horizontal distance between them. Then, in the program, fine-tune this offset according to the actual result to obtain the precise position (visual error < 5mm);

[0022] The detailed steps of step 2 are as follows:

[0023] S1. You can choose any annotation tool. Label-studio is used in this project.

[0024] S2. The project selects only the 150x150px pixel area in the center of the image as the ROI, marks the gaps, paints the gaps with a brush, and exports the mask image;

[0025] S3. Use the ROI image and mask image in S2 as UNET input and use the UNET official model as a checkpoint for transfer learning. The default data augmentation method is used to obtain a relatively robust gap extraction model.

[0026] S4. Use the model obtained in S3 to perform inference and obtain the gap mask.

[0027] Step 2 also includes the following steps:

[0028] S5. Perform morphological closing on the mask obtained in the previous step to improve the accuracy of contour extraction;

[0029] S6. Contour extraction and bounding box filtering are performed to remove the surrounding contours that deviate from the center position and obtain the contours in the center area of ​​the ROI image (there may be more than one contour);

[0030] S7. Consider all the contour points obtained in the previous step and use the fitEllipse() function of OpenCV to do ellipse fitting.

[0031] S8. Ellipse polar diameter search: In the unet output mask image space, for the lower half of the ellipse, starting from the center of the circle, uniformly sample every angle and traverse each pixel on the polar diameter; count the number of black pixels n on the polar diameter; if the current value n> the maximum value n_max, update the maximum value n_max to the current value n, and update the target point to the point on the polar diameter closest to the ellipse center. The final target point is the point to be found p(u,v).

[0032] A vision-guided intelligent casting system includes a casting machine, an image acquisition module, a three-dimensional simulation module, a PLC control module, and a carbon block. The casting machine is provided with a pouring gate, and the carbon block is provided with multiple carbon bowls and steel forks. The casting machine can reciprocate along a track, and the carbon block can move linearly along a plate chain. The image acquisition module is a set of four depth cameras installed on the casting machine. The plate chain transports the carbon block to the casting station. The casting machine pours molten metal through the pouring gate onto the carbon bowls and steel forks of the carbon block.

[0033] Initially, it is necessary to collect information on the charcoal bowl, steel fork, and qualified aluminum products through step one, and reconstruct a three-dimensional model of the charcoal bowl, steel fork, and qualified aluminum products through step two, name it a standard model, and store it. In the subsequent production process, the reconstructed three-dimensional model of the charcoal bowl, steel fork, and aluminum products in production is compared with the standard model for similarity. If the difference is large, the PLC control module controls the casting machine to move and dump the product to achieve re-casting until the cast product is qualified.

[0034] After casting, the casting mouth is driven away, and the image acquisition module obtains the color and depth information of the aluminum product, charcoal bowl, and steel fork. The post-3D simulation module reconstructs the production scene in 3D to obtain a 3D model. After similarity comparison with the standard model, it is found that no gap is detected on the left, indicating that it has been filled, while a gap is detected on the right, indicating that it is not filled. At this time, three target casting points are set; the casting machine is driven to move, and the casting machine gate is driven to move to the three target casting points for casting, and finally a qualified product is obtained.

[0035] The beneficial effects of the present invention are:

[0036] (1) The present invention obtains the color and depth information of the charcoal bowl and steel fork through the image acquisition module, reconstructs the production scene in three dimensions through the three-dimensional simulation module, and controls the movement and pouring of the casting machine through the PLC control module to achieve the casting of the product. Compared with the manual operation method of workers, the present invention improves production efficiency and reduces the safety hazards in high-temperature environments. In addition, it also reduces product defects and is conducive to the casting of aluminum products.

[0037] (2) The present invention collects information on the charcoal bowl, steel fork, and qualified aluminum products through step one, and reconstructs a three-dimensional model of the charcoal bowl, steel fork, and qualified aluminum products through step two, names it as a standard model, and stores the standard model. In the subsequent production process, the reconstructed three-dimensional model of the charcoal bowl, steel fork, and aluminum products in production is compared with the standard model for similarity. If the difference is relatively large, the PLC control module is used to control the casting machine to move and dump the product to achieve re-casting until the cast product is qualified.

[0038] (3) The present invention drives the casting mouth away after casting, and the image acquisition module obtains the color and depth information of the aluminum product, charcoal bowl, and steel fork. The post-3D simulation module reconstructs the production scene in 3D to obtain a 3D model. After similarity comparison with the standard model, it is found that no gap is detected on the left, indicating that it is filled, while a gap is detected on the right, indicating that it is not filled. At this time, three target casting points are set; the casting machine is driven to move, and the casting machine gate is driven to move to the three target casting points for casting, and finally a qualified product is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0040] Figure 2 This is a schematic diagram of the first thread framework of the present invention;

[0041] Figure 3 This is a schematic diagram of the second thread framework of the present invention;

[0042] Figure 4 This is a schematic diagram of the casting machine of the present invention;

[0043] Figure 5 This is a schematic diagram of the contour extraction area of ​​the present invention;

[0044] Figure 6 This is a schematic diagram of the contour extraction of the present invention;

[0045] Figure 7 This is a schematic diagram of the cross point position marked on the casting point A of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0048] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0049] Example 1: Figures 1-6 As shown, this embodiment provides a vision-guided intelligent casting method, comprising the following steps:

[0050] Step 1: Obtain the color and depth information of the charcoal bowl and steel fork through the image acquisition module;

[0051] Step 2: Reconstruct the production scene in 3D through the 3D simulation module;

[0052] Step 3: Use the PLC control module to control the casting machine's movement and pouring to achieve product casting.

[0053] The detailed steps of step one are as follows:

[0054] S1. Obtain the camera's color image stream and depth camera stream through the API provided by rosl. The two can be aligned at the pixel level: for each RGB point in the image space, a depth value d can be found;

[0055] S2. Dense point cloud reconstruction: For each point p-(u, v) in the image space, there is a corresponding spatial point P-(x, y, z) in the camera coordinate system. The scaling factor from the camera space scale to the real space scale is set to s, and we have

[0056] Pz=d / s

[0057] s is calculated as follows: obtain the original point cloud and save it; use the pcl_viewer tool to measure the length s1 of the carbon block in the point cloud, and use a tape measure to measure the actual size s2 of the carbon block, then we have

[0058] s=s2 / s1

[0059] Px=(pu-cx)*Pz / fx

[0060] Py=(pv-cy)*Pz / fy

[0061] The camera intrinsic parameters cx, cy, fx, and fy can be obtained by calibrating with the checkerboard method or directly using the command line tool provided by realsense.

[0062] S3. Point cloud stitching: The original point cloud consists of four pieces. We use a self-developed C++ tool (based on PCL, simple to implement, outputs four transformation matrices, details omitted) to visualize them, perform rotation and translation operations, and then transform them into the same coordinate system.

[0063] S4. Determine the casting machine mouth: Since the relative position between the casting machine mouth and the camera is fixed, you can use a tape measure to roughly measure the horizontal distance between them. Then, in the program, fine-tune this offset according to the actual result to obtain the precise position.

[0064] The detailed steps of step 2 are as follows:

[0065] S1. You can choose any annotation tool. Label-studio is used in this project.

[0066] S2. The project selects only the 150x150 px pixel area at the center of the image as the ROI, marks the gaps, paints the gaps with a brush, and exports the mask image;

[0067] S3. Use the ROI image and mask image in S2 as UNET input and use the UNET official model as a checkpoint for transfer learning. The default data augmentation method is used to obtain a relatively robust gap extraction model.

[0068] S4. Use the model obtained in S3 to perform inference and obtain the gap mask.

[0069] Step 2 also includes the following steps:

[0070] S5. Perform morphological closing on the mask obtained in the previous step to improve the accuracy of contour extraction, such as Figure 5 As shown;

[0071] S6, contour extraction, and bounding box filtering, remove the environmental contours that deviate from the center position, and obtain the contour in the center area of ​​the ROI image (there may be more than one), such as Figure 6 As shown;

[0072] S7. Consider all the contour points obtained in the previous step and use the fitEllipse() function of OpenCV to do ellipse fitting.

[0073] S8, ellipse polar diameter search: In the unet output mask image space, for the lower half of the ellipse, starting from the center of the circle, uniformly sample every angle and traverse each pixel on the polar diameter; count the number of black pixels n on the polar diameter; if the current value n> the maximum value n_max, then update the maximum value n_max to the current value n, and update the target point to the point on the polar diameter closest to the ellipse center. The final target point is the point to be found p(u,v), specifically Figure 7 The cross point position at point A is shown;

[0074] It should be noted that: initially, it is necessary to collect information on the charcoal bowl, steel fork, and qualified aluminum products through step one, and reconstruct the 3D model of the charcoal bowl, steel fork, and qualified aluminum products through step two, name it the "standard model", and store the "standard model". In the subsequent production process, the similarity calculation is performed by comparing the reconstructed 3D model of the charcoal bowl, steel fork, and aluminum products in production with the "standard model", and setting a rated value. If it is lower than the rated value (unqualified), the PLC control module controls the casting machine to move and dump the product to achieve recasting until the cast product is qualified. It should be noted that the specific similarity calculation uses: cosine similarity calculation, which calculates the cosine value of the angle between two vectors. If two vectors are exactly the same, their cosine similarity is 1; if they are completely different, it is 0.

[0075] Formula: similarity = cos(θ) = (A·B) / (||A||*||B||) where A and B are two vectors;

[0076] like Figure 4 As shown, after casting, the pouring nozzle is driven away, and the image acquisition module obtains the color and depth information of the aluminum product, charcoal bowl, and steel fork. The post-3D simulation module performs 3D reconstruction of the production scene to obtain a 3D model. After similarity comparison with the standard model, it is found that no gap is detected on the left, indicating that it is filled, while a gap is detected on the right, indicating that it is not filled. At this time, three target casting points are set; the pouring machine is driven to move, and the pouring machine gate is driven to move to the three target casting points for casting, and finally a qualified product is obtained; it should be noted that: Figure 2 and Figure 3 Put together to form a complete picture.

[0077] Example 2: Figures 1-6 As shown, this embodiment provides a vision-guided intelligent casting system, including a casting machine, an image acquisition module, a three-dimensional simulation module, a PLC control module and a carbon block. The casting machine is provided with a gate, and the carbon block is provided with multiple carbon bowls and steel forks. The casting machine can reciprocate along the track, and the carbon block can make a single-line linear motion along the plate chain. The image acquisition module is four sets of depth cameras arranged on the casting machine. The plate chain transports the carbon block to the casting station, and the casting machine casts the molten metal on the carbon bowl and steel fork of the carbon block through the gate.

[0078] like Figure 2 As shown, the flowchart of the main thread indicates: it starts to judge "should exit?", if it is "yes", it ends directly, if it is "no", it connects to the PLC, then initializes each thread and enters "Run main loop";

[0079] The flowchart of the PLC Read thread indicates: "Read PLC data at a certain frequency"; "db1001: Detect in place? Carbon block removed?"; "sleep, wait for the next cycle";

[0080] The flowchart of the PLC Write thread indicates: "Send data to the PLC at a certain frequency"; "db1000: If the position is detected, calculate the offset and write the 4 offsets calculated so far into the DB"; "sleep, wait for the next cycle";

[0081] The flowchart of the Stream Sync thread shows: "Get RGB frames from 4 cameras"; "Preprocess"; "Update 8 streams";

[0082] The flowchart of the PCL Viewer thread states: "Get and display point cloud related information";

[0083] The flowchart of the logic Proc thread shows: "Copy the inspection station image ROI data"; "UNet inference to obtain the gap mask"; "Morphological processing"; "Edge detection + contour extraction"; "Contour screening + ellipse fitting"; "Apply polar radius search in the lower semicircle area of ​​the target ellipse in the gap mask"; "Find the polar radius with the widest gap, and the endpoint of the polar radius close to the center of the ellipse is the target point"; "Back-project the target point into space, and transform it into a unified world coordinate system according to the transformation matrix obtained by offline external parameter calibration"; "Sleep to match the preset frequency"; "Polling the current mode at a certain frequency"; "Search for the target gap point in the image space ROI area.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vision-guided intelligent casting method, characterized in that: The steps include: Step 1: Obtain the color and depth information of the charcoal bowl and steel fork through the image acquisition module; Step 2: Reconstruct the production scene in three dimensions through a three-dimensional simulation module; Step 3: Control the movement and pouring of the casting machine through the PLC control module to achieve the casting of the product.

2. A visually guided intelligent casting method according to claim 1, characterized in that: The detailed steps of step one are as follows: S1. Get the camera's color image stream and depth camera stream through the API provided by ros1; S2. Dense point cloud reconstruction: For each point p-(u,v) in the image space, there is a corresponding spatial point P-(x,y,z) in the camera coordinate system. The scaling factor from the camera space scale to the real space scale is set to s, and we have Pz=d / s.

3. A visually guided intelligent casting method according to claim 2, characterized in that: s is calculated by: obtaining the original point cloud and saving it; using the pcl_viewer tool to measure the length s1 of the carbon block in the point cloud, and using a tape measure to measure the actual size s2 of the carbon block, we have s=s2 / s1 Px=(pu-cx)*Pz / fx Py=(pv-cy)*Pz / fy Among them, the camera intrinsic parameters cx, cy, fx, and fy can be calibrated by the checkerboard method, or they can be directly obtained using the command line tool provided by realsense.

4. A visually guided intelligent casting method according to claim 3, characterized in that: Step one also includes the following steps: S3, point cloud stitching: There are 4 original point clouds in total. Use the self-developed C++ tool to visualize them, rotate and translate them, and then transform them to the same coordinate system; S4. Determination of the casting machine mouth: Since the relative position between the casting machine mouth and the camera is fixed, use a tape measure to measure the lateral distance, and then fine-tune the offset in the program according to the actual results to obtain the precise position.

5. A visually guided intelligent casting method according to claim 4, characterized in that: The detailed steps of step 2 are as follows: S1, labeling tool uses label-studio; S2. The project only selects the pixel area of ​​150x150 px in the center of the image as ROI, marks the gaps, paints the gaps with a brush, and exports the mask image; S3, use the ROI image and mask image in S2 as unet input, use the unet official model as checkpoint for transfer learning, where the data enhancement uses the default processing method to obtain a relatively robust gap extraction model; S4. Use the model obtained in S3 to perform inference and obtain the gap mask.

6. A vision-guided intelligent casting method according to claim 5, characterized in that: Step 2 also includes the following steps: S5, perform morphological closing processing on the mask obtained in the previous step to improve the accuracy of contour extraction; S6, contour extraction, and boundingbox filtering, to remove the environmental contours that deviate from the center position, and obtain the contour in the center area of ​​the ROI image; S7. Considering all the contour points obtained in the previous step, use the fitEllipse() function of opencv to fit the ellipse. S8. Ellipse polar diameter search: In the unet output mask image space, for the lower half of the ellipse, starting from the center of the circle, uniformly sample at every angle and traverse each pixel on the polar diameter; count the number of black pixels n on the polar diameter; if the current value n> the maximum value n_max, update the maximum value n_max to the current value n, and update the target point to the point on this polar diameter closest to the center of the ellipse. The final target point is the point to be found p(u,v).

7. A vision-guided intelligent casting system, applied to a vision-guided intelligent casting method according to any one of claims 1 to 6, characterized in that: The invention comprises a casting machine, an image acquisition module, a three-dimensional simulation module, a PLC control module and a carbon block. The casting machine is provided with a pouring gate, and the carbon block is provided with a plurality of carbon bowls and steel forks.

8. A vision-guided intelligent casting system according to claim 7, characterized in that: The casting machine can reciprocate along the track, and the carbon block can make a single-line linear motion along the plate chain.

9. A vision-guided intelligent casting system according to claim 8, characterized in that: The image acquisition module is a plurality of groups of depth cameras arranged on the casting machine.

10. A vision-guided intelligent casting system according to claim 9, characterized in that: The plate chain transports the carbon block to the casting station, and the casting machine pours the molten metal onto the carbon bowl and steel fork of the carbon block through the pouring gate.

Images