A visual identification positioning method suitable for satellite honeycomb panel glue filling operation

Abstract

The application provides a visual identification positioning method suitable for satellite honeycomb panel glue filling operation. The method uses a clip supervised stable-diffusion image enhancement expansion data set method, can expand a small amount of glue filling data set image into a large amount of high-quality image data set, and improves the identification accuracy and success rate of the trained yolo model. The method uses a polygon fitting method, can fit the irregular honeycomb hole boundary into a polygon, and calculates the center position coordinates of the glue filling honeycomb hole, thereby providing guidance for the operation of the glue filling equipment.

Description

Technical Field

[0001] This invention relates to a visual recognition and positioning method suitable for filling adhesive into satellite honeycomb panels, belonging to the field of satellite composite component assembly auxiliary technology. Specifically, the vision system identifies and positions the center of the honeycomb cells in the composite component that need to be filled with solid expanding foam, thereby assisting in the precise filling of the expanding foam. Background Technology

[0002] With the increasing number of satellite launches in my country, improving the efficiency of satellite production is becoming increasingly urgent. As a crucial component of satellite production, satellite composite structural panels require the development of automated assembly technologies to enhance their production efficiency.

[0003] Currently, the assembly of satellite structural panels is mostly done manually. In the foam filling process for honeycomb panels, the raw material for the foam strips is in sheet form. These strips are cut into thin strips by machine and stored in a freezer to prevent them from deteriorating. Workers need to remove the strips from the freezer, separate any sticky parts, and quickly insert them into the honeycomb holes near the openings in the honeycomb panel. However, the design drawings do not specify which honeycomb holes the foam should fill, and currently, the location of the foam-filled honeycomb holes is mainly determined visually. In an automated assembly solution, a robot would deliver freshly cut foam strips to the honeycomb holes under temperature control. Therefore, a vision guidance system suitable for automated assembly methods needs to be designed to more precisely locate the positions where the foam strips need to be filled, allowing automated filling equipment to complete the filling operation. Summary of the Invention

[0004] This invention provides a visual inspection system and technology for locating the filling holes of honeycomb panels in composite components based on visual recognition. It provides the center position of the foam filling honeycomb holes for automated filling devices, thereby improving the automated assembly efficiency of composite components.

[0005] To achieve the functions required above, the present invention provides the following technical solution: a visual recognition-based glue-filled honeycomb hole positioning technology, comprising: An image enhancement and dataset augmentation method based on the clip model and the Stable Diffusion model can be used to augment the dataset of glue-filled honeycomb cells, which are difficult to collect. Image recognition methods based on the YOLO model can use the aforementioned honeycomb hole dataset to train the YOLO model to identify honeycomb holes that require glue filling. The polygon edge reconstruction method can calculate the precise center position of the glue-filled honeycomb holes and supply the glue-filling equipment for glue filling.

[0006] As a preferred solution for visual recognition and localization methods in satellite honeycomb panel filling operations, the image enhancement and dataset augmentation methods based on the clip model and the Stable Diffusion model include: The clip model training method based on few samples can establish a more accurate clip image-text alignment model for specific environments of glue-filled honeycomb holes; The Stable Diffusion model training method based on transfer learning can train an image generation model for specific environments with glue-filled honeycomb holes under small sample conditions. The feature alignment method based on ResNet-50 can align the pixel features of an image with the text features. It features adaptive enhancement methods and feature-sharing multi-task model training methods, enabling faster and more efficient training of clip-supervised models and Stable Diffusion models; An image enhancement method starting from the original image can enhance the original honeycomb hole image to expand the dataset by using a trained Stable Diffusion model specifically designed for the honeycomb hole environment and a trained Clip model specifically designed for the honeycomb hole environment.

[0007] As a preferred solution for the visual recognition and positioning method for filling adhesive in satellite cellular panels, the image enhancement and dataset expansion method based on the clip model and the Stable Diffusion model requires the dataset to include images of several shapes, lighting conditions, and location conditions of the cellular holes. During the clip model training process, corresponding prompt text needs to be written as training samples based on the features of each image in the dataset.

[0008] As a preferred method for visual recognition and localization in satellite honeycomb panel filling operations, the improved CLIP supervised loss function, based on a small sample size, incorporates feature enhancement weights in the following training method: in Let T represent the i-th enhanced image. i This represents a text description corresponding to the image. This is an attention mask used to highlight the honeycomb aperture area. ⊙ represents element-wise multiplication. The mask generation formula is: Where (x) c ,y c) represents the center coordinates of the honeycomb hole, and r is the mask radius. This mask can highlight the image features within a certain range of the honeycomb hole near the insert (that is, the approximate area filled with foam).

[0009] As a preferred method for visual recognition and localization in satellite honeycomb panel filling operations, the Stable Diffusion model training method based on transfer learning improves the diffusion process through the I2I (Image-to-Image) mode, using the original image I original Combined with text T to guide generation: in Let α represent the latent variable at step t. t Represents noise scheduling parameters, ∈ t I represents Gaussian noise. original T represents the original image;

[0010] As a preferred solution for visual recognition and positioning methods in satellite cellular panel filling operations, the ResNet-50-based feature alignment method employs feature extraction and cross-modal projection. It upsamples the 2048-dimensional feature map extracted by ResNet-50 to a resolution of 224×224 using bilinear interpolation, and then projects it onto the image embedding space of the CLIP model through a 1×1 convolutional layer. f shared =Conv 1×1 (Interp blinear (ResNet50(I))); Where ResNet50(I) represents the image features extracted by ResNet-50, and Interp blinear Represents bilinear interpolation, Conv 1×1 Adjust the convolution based on the number of channels;

[0011] As a preferred solution for the visual recognition and positioning method in satellite cellular panel filling operations, the multi-task model training method with adaptive enhancement and feature sharing specifically implements the multi-task joint loss function as follows: in λ is the mask mean, and λ2 is the implicit optimization parameter of Stable Diffusion. For the loss of clip and stable diffusion;

[0012] As a preferred embodiment of the visual recognition and positioning method for satellite honeycomb panel filling operations, the image enhancement method uses the following image quality control formula: Among them I original Represents the original image, I gen f represents the generated image. I This represents the image encoder of the CLIP model, where τ = 0.8 is the similarity threshold, and cosine similarity is used for similarity calculation.

[0013] As a preferred solution for the visual recognition and positioning method for the glue filling operation of satellite honeycomb panels, the image recognition method based on the YOLO model requires that the honeycomb holes around the insert be marked in two forms, complete honeycomb holes and incomplete honeycomb holes, since the shearing of the insertion hole is random. Among them, the complete honeycomb hole is the appropriate honeycomb hole that needs to be filled with glue.

[0014] As a preferred solution for the visual recognition and positioning method for the filling of satellite honeycomb panels, the image recognition method based on the YOLO model aims to identify the positions of complete honeycomb holes in two rings around the mounting hole of the insert, while eliminating interference from complete honeycomb holes at greater distances.

[0015] As a preferred embodiment of the visual recognition and positioning method for satellite honeycomb panel filling operations, the polygon edge reconstruction method includes: A preliminary polygon edge extraction method based on edge feature extraction and line fitting algorithm; A polygon contour reconstruction method based on clustering.

[0016] As a preferred embodiment of the visual recognition and positioning method for satellite honeycomb panel filling operations, the polygon edge reconstruction method requires cropping the honeycomb hole images detected by the YOLO model from the overall image. Let the input image be... Based on the predicted box coordinates (x) min ,y min ,x max ,y max Perform affine transformation: Where w is the affine transformation operator. For horizontal direction operation factor, W and H are the vertical operation factors, where W and H are the width and height of the original image, respectively.

[0017] As a preferred solution for the visual recognition and positioning method for filling adhesive in satellite cellular panels, the preliminary polygon edge extraction method based on edge feature extraction and line fitting algorithm requires converting the cropped image into a grayscale image and using the Canny edge detection algorithm to extract its edge features.

[0018] As a preferred solution for the visual recognition and positioning method in satellite honeycomb panel filling operations, the preliminary polygon edge extraction method based on edge feature extraction and line fitting algorithm uses the Hough line detection method to extract straight lines from the edge features. This algorithm extracts a set of candidate straight lines. The equation of each line is: l i :ρ i =xcosθ i +ysinθ i ; Where (ρ i ,θ i Let be the polar coordinate parameters, and the constraints for straight line fitting are: Where H crop W crop The height and width of the cropped image;

[0019] As a preferred solution for visual recognition and positioning methods in satellite honeycomb panel filling operations, the aforementioned Hough line detection scheme incorporates angular constraints. To improve the efficiency of hexagon detection;

[0020] As a preferred solution for the visual recognition and positioning method in satellite honeycomb panel filling operations, the polygon contour reconstruction method based on clustering is used to reconstruct the polygon contour based on the extracted candidate line set feature vector v. i =[ρ i ,θ i ] T They were divided into 6 classes using the k-means algorithm. The clustering principle of this algorithm is as follows: Where C k Let v represent the k-th cluster. i This represents the eigenvector of the i-th line. For the cluster center, select the straight line closest to the center in each class as the edge of the hexagon for that cell:

[0021] As a preferred method for visual recognition and positioning in satellite honeycomb panel filling operations, in the above clustering method, if empty clusters appear, the centers are reinitialized using the following formula: Where α is the learning rate. For clustering loss function with respect to μ k The gradient;

[0022] As a preferred solution for the visual recognition and positioning method in satellite honeycomb panel filling operations, the polygon contour reconstruction method based on clustering requires determining the coordinates of the intersection points of the polygons to find the center coordinates of the polygonal honeycomb cells, and then calculating the coordinates of adjacent straight lines. and Find the intersection point: Obtain the hexagonal point set The center of the polygon can be obtained by the following formula:

[0023] As a preferred method for visual recognition and positioning in satellite honeycomb panel filling operations, after obtaining the pixel coordinates of the center of the filled honeycomb cell using the above method, its position coordinates in the world coordinate system are calculated: in Let T∈SE(3) be the camera calibration matrix, and Z be the robot base coordinate system transformation matrix. c This represents the depth of the honeycomb pore surface.

[0024] This invention has the following advantages: By employing a clip-supervised image enhancement method, the visual recognition scheme can expand the image dataset even with a limited dataset, thus avoiding overfitting during subsequent image detection model training and improving the generalization of the recognition model; through transfer learning and adaptive enhancement methods, the image enhancement module improves the accuracy of image enhancement for the special scenario of filled honeycomb holes in composite components, based on the original model; by using the YOLO model to identify the integrity of the honeycomb holes around the embedded part, the filling quality of the foam can be guaranteed; by identifying image edges and performing edge reconstruction, the position center of irregular honeycomb holes can be obtained, which can adapt to the position center positioning requirements of honeycomb holes under different extrusion shapes, and has considerable versatility. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the specific implementation will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the visual recognition and positioning method for the satellite honeycomb panel adhesive filling operation of the present invention;

[0027] Figure 2Flowchart of the training visual model for an embodiment of the visual recognition and positioning method for satellite honeycomb panel glue filling operation of the present invention;

[0028] Figure 3 This is an example image of the initial dataset from an embodiment of the visual recognition and positioning method for satellite honeycomb panel adhesive filling operations of the present invention.

[0029] Figure 4 This is a schematic diagram of the working environment of an embodiment of the visual recognition and positioning method for satellite honeycomb panel glue filling operation of the present invention. Detailed Implementation

[0030] To more clearly explain the objectives, technical solutions, and advantages of this invention, the invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] Example

[0034] Reference Figure 1 This invention provides a visual recognition and positioning method for satellite honeycomb panel adhesive filling operations, including... Figure 1 The steps are shown.

[0035] S1: Training the visual recognition model, the process is as follows Figure 2 As shown;

[0036] S101: First, 50-100 images of the honeycomb holes in the composite component need to be taken. Since the foaming adhesive in this invention is mostly filled near the insert, and the insert is usually already installed during actual filling, images of the honeycomb panel with the insert initially placed in its mounting holes can be taken. Then, images of approximately 2 to 3 rings of honeycomb holes centered on the insert can be taken as a dataset. Example images are shown below. Figure 3 As shown;

[0037] S102: Write a corresponding prompt for each captured image, in a format roughly as "a photograph of an aluminum honeycomb panel with some kind of insert and whose honeycomb holes are roughly regular / irregular in shape under some kind of lighting". In particular, different images can have the same prompt.

[0038] S103: Input each image and its corresponding prompt into the aforementioned clip and StableDiffusion multi-task training model for training;

[0039] S104: After each training session, compare the total loss of this training session with the currently saved best loss. If the total loss of this training session is smaller, save the total loss of this training session as the best loss and save the pt file of this training session.

[0040] S105: Write a variety of prompts similar to those mentioned in S102, and input them along with the initial dataset into the trained Stable Diffusion model to generate new high-quality images to expand the dataset under the supervision of clip.

[0041] S106, S107: For the newly generated dataset, tools such as labelimg are used to label complete and incomplete honeycomb holes, and the dataset is used to train the YOLO model; in particular, image enhancement methods such as cropping, splicing, and rotating images in YOLO are still retained.

[0042] S2: As Figure 4 As shown, composite component panels are assembled on the assembly line and then proceed to the foam filling station after completing other processes. In the figure, 401 is the filling device, which includes a feeding hopper and a filling robotic arm. 402 is a global camera, fixed on a frame above the processing line. 403 is the composite component panel being conveyed on the slide block of the assembly line. In particular, the mounting holes of the composite component inserts are operated by punching and shearing honeycomb holes with a rotating method to remove debris, which will cause the honeycomb holes near the inserts to deform to a certain extent.

[0043] S3, S4: such as Figure 4 As shown, the images used for image recognition are obtained by a global camera positioned above the workstation. Since the image range captured by the global camera is large, the host computer needs to determine the approximate world coordinate position of each embedded part according to the content of the composite component design drawings, and cut the image near the embedded part according to the requirements similar to S101, and then detect the position of the glue-filled honeycomb hole through the visual recognition model.

[0044] S5: Using the aforementioned method of edge fitting and center position calculation for honeycomb cells, the world coordinates of the filled honeycomb cells are obtained to guide the filling equipment in the filling operation.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit it. Although the present invention has been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application should not depart from the spirit and scope of the technical solutions of this application.

Claims

1. A system for locating adhesive filling holes in a honeycomb panel of composite components based on visual recognition, characterized in that, include: An image enhancement and augmentation module based on the CLIP model and the Stable Diffusion model is used to generate an augmented glue-filled honeycomb pore dataset. An image recognition module based on the YOLO model is used to detect complete honeycomb cells that need to be filled with adhesive. The polygon edge reconstruction module is used to calculate the precise center position of the glue-filled honeycomb holes; The coordinate transformation module converts pixel coordinates into world coordinates in the robot's base coordinate system.

2. The system according to claim 1, characterized in that, The image enhancement and expansion module includes: CLIP models trained on few samples incorporate a spatial attention mask into their loss function: in Let T represent the i-th enhanced image. i This represents a text description corresponding to the image. This is an attention mask used to highlight the honeycomb aperture area; ⊙ represents element-wise multiplication. The mask generation formula is: Where (x) c ,y c Let ) represent the coordinates of the cell center and r represent the mask radius. Based on the Stable Diffusion model of I2I mode, the condition generation process is as follows: in Let α represent the latent variable at step t. t Represents noise scheduling parameters, ∈ t I represents Gaussian noise. original T represents the original image, and T represents the text description.

3. The system according to claim 2, characterized in that, The CLIP model shares a feature extraction network with the Stable Diffusion model, specifically including: The 7×7 feature map output by ResNet-50 is upsampled to 224×224 resolution using bilinear interpolation; Using a 1×1 convolutional layer, the number of channels is reduced from 2048 to 3. The projection formula is: f shared =Conv 1×1 (Interp. blinear (ResNet50(I))); Where ResNet50(I) represents the image features extracted by ResNet-50, and Interp blinear Represents bilinear interpolation, Conv 1×1 Adjust the convolution to the number of channels.

4. The system according to claim 2, characterized in that, The image enhancement quality control formula is as follows: Among them I original Represents the original image, I gen f represents the generated image. I This represents the image encoder of the CLIP model, with a similarity threshold τ = 0.

8.

5. The system according to claim 1, characterized in that, The training data labeling strategy for the YOLO model includes labeling complete honeycomb holes within two rings around the embedding as positive samples; Incomplete honeycomb cells and complete honeycomb cells outside the two rings are labeled as negative samples; The annotation file includes cell hole integrity labels and predicted bounding box coordinates (x, y, y). min ,y min ,x max ,y max ).

6. The system according to claim 1, characterized in that, The polygon edge reconstruction module includes: The constraints for Hough line detection are: The objective function for k-means clustering is: Where C k Let v represent the k-th cluster. i This represents the eigenvector of the i-th line. It is the cluster center.

7. The system according to claim 6, characterized in that, The clustering method includes a strategy for handling empty clusters: When an empty cluster occurs, the centers are reinitialized using the following formula: Where α is the learning rate. For clustering loss function, The loss function represents the loss function with respect to cluster centers μ. k The gradient.

8. The system according to claim 6, characterized in that, The polygon center calculation specifically involves: Formula for the intersection of adjacent lines: Where ρ k and θ k Let the polar coordinates of the k-th line be represented, and the geometric center coordinates be calculated as follows: in Here, the coordinates of the k-th vertex are represented.

Images