Honeycomb structure part core defect detection method and system based on robot vision

Abstract

The application provides a honeycomb structure part core defect detection method and system based on robot vision, and relates to the technical field of industrial detection. The method comprises the following steps: a multi-view image of a honeycomb structure part to be detected is shot by using a robot vision system which has been calibrated, and a pose parameter of an industrial robot end in the robot vision system when the image is shot is saved; an independent core area is divided in each image by using a deep learning-based image instance segmentation model; and a defect category label of each core area is generated by using a deep learning image classification model; based on the obtained pose parameter, each core area in all images is projected one by one, and repeated cores are determined and removed in the projection process, so that a three-dimensional visual core defect detection result of the whole part is obtained. The application realizes accurate identification, positioning and three-dimensional visual presentation of surface defects and bottom welding defects of the honeycomb structure.

Description

Technical Field

[0001] This invention relates to the field of industrial inspection technology, and in particular to a method and system for detecting lattice defects in honeycomb structure parts based on robot vision. Background Technology

[0002] Honeycomb structure components are widely used in high-performance equipment such as aerospace, rail transportation, and electronic packaging due to their advantages of light weight, high strength, and good energy absorption performance. These components are typically composed of a honeycomb structure and metal panels, resulting in a complex structure and requiring high machining precision. During the manufacturing process, various structural defects can easily occur, seriously affecting the performance and service reliability of the components.

[0003] Typical defects generated during the processing and manufacturing of honeycomb structure parts mainly fall into the following two categories: one is surface defects of the honeycomb structure, which mainly include problems such as tearing, burning, and crushing of the honeycomb core, usually caused by material defects, improper hot working, or external impact; the other is bottom welding defects, which refer to defects generated during the brazing process of the honeycomb structure to the metal base plate, such as incomplete welding and excessive brazing filler metal.

[0004] Traditional inspection methods often rely on manual visual inspection or fixed-angle photography combined with image recognition algorithms. However, manual inspection is inefficient and highly subjective, making it difficult to meet the demands of large-scale, high-consistency industrial inspection. Fixed-angle image inspection, on the other hand, is affected by issues such as viewpoint obstruction, reflections from complex structures, and the difficulty in identifying small defects, making it difficult to achieve comprehensive coverage of defects in complex three-dimensional structures.

[0005] With the development of deep learning technology, image segmentation and classification models based on Convolutional Neural Networks (CNNs) have significantly improved the performance of target recognition in two-dimensional images. However, existing research mainly focuses on detecting targets in single images, lacking the ability to acquire, process, and fuse detection results from multiple perspectives. This fails to meet the industrial demand for comprehensive and high-precision detection and localization of cellular structure defects. Therefore, there is an urgent need for a method that combines robot vision systems, deep learning image understanding capabilities, and spatial geometric information processing capabilities to achieve accurate identification and three-dimensional visualization of defects in cellular structure parts. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, this invention proposes a method and system for detecting core defects in honeycomb structure parts based on robot vision. This aims to solve problems such as insufficient field of view coverage, difficulty in detecting defects across the entire workpiece, and lack of defect localization and 3D visualization in existing honeycomb structure part core defect detection methods. It achieves accurate identification, localization, and 3D visualization of surface defects and bottom welding defects in honeycomb structures, improving the comprehensiveness, automation, and engineering applicability of the detection.

[0007] On the one hand, this invention proposes a method for detecting core defects in honeycomb structure parts based on robot vision, which includes the following process:

[0008] The robot vision system, which has been calibrated with hand and eye, is used to acquire multi-view images of the honeycomb structure parts to be inspected, and the pose parameters of the end effector of the industrial robot in the robot vision system are acquired simultaneously when each image is acquired.

[0009] The acquired images are processed using a pre-trained deep learning-based image instance segmentation model to identify and delineate independent core regions in each image.

[0010] A pre-trained deep learning image classification model is used to identify defects in each divided core region, and a defect category label is generated for each core region.

[0011] Based on the acquired pose parameters, the spatial position and normal direction of each core region in the base coordinate system are restored by projecting each core region in all images onto a unified base coordinate system.

[0012] During the process of projecting each core region one by one, the spatial position and normal direction of all core regions in the base coordinate system, as well as the defect category label of each core region, are fused to generate a three-dimensional visualization of the core defect detection results of the honeycomb structure part to be inspected.

[0013] Furthermore, the robot vision system includes: an industrial control computer, a vision controller, a work platform, a honeycomb structure part positioning fixture, an industrial robot, an area scan camera, a telecentric lens, a right-angle prism, and a coaxial light source;

[0014] The honeycomb structure part positioning fixture is fixedly installed on the working platform to fix the honeycomb structure part to be tested;

[0015] The area array camera, telecentric lens, right-angle prism, and coaxial light source are integrated into a visual sensor assembly via a mechanical structure for image acquisition of the honeycomb structure parts to be inspected. The area array camera is directly connected to the telecentric lens, and the right-angle prism is mounted at the front end of the telecentric lens. The right-angle prism reflects the horizontal light path of the area array camera into a vertical light path. The coaxial light source and the right-angle prism are fixedly mounted on the same support structure, with the coaxial light source positioned below the right-angle prism. The light emission direction of the coaxial light source is consistent with the direction of the imaging light path after being deflected by the right-angle prism.

[0016] The vision sensor assembly is installed at the end of the industrial robot;

[0017] The industrial robot is used to move the vision sensor components to a predetermined position according to the control instructions from the industrial control computer in order to complete the automatic image acquisition task.

[0018] The vision controller is fixedly mounted on the work platform and is used to receive vision parameter instructions from the industrial control computer and control the working status of the area scan camera, telecentric lens and coaxial light source in the vision sensor assembly.

[0019] Furthermore, the pre-trained deep learning-based image instance segmentation model is as follows:

[0020] A single-stage instance segmentation model is adopted as the deep learning-based image instance segmentation model.

[0021] The single-stage instance segmentation model is used to extract and fuse multi-scale features of the input image to generate a mask feature map of the input image; the mask feature map is gridded, and each grid cell is used to predict whether there is a core instance at the center of the grid cell; a dynamic convolution kernel is predicted for each grid cell, and the dynamic convolution kernel of each grid cell is convolved with the mask feature map to generate an instance-aware mask and a corresponding confidence score for each grid cell; the instance-aware masks corresponding to each grid cell are filtered to remove overlapping and instance-aware masks with confidence scores below a preset threshold, and the filtered instance-aware masks are defined as mutually independent core regions in the input image;

[0022] A core-grid instance segmentation dataset was constructed by collecting several images of honeycomb structure parts, and a deep learning-based image instance segmentation model was trained using the core-grid instance segmentation dataset to obtain the trained deep learning-based image instance segmentation model.

[0023] Furthermore, the pre-trained deep learning image classification model is as follows:

[0024] A convolutional neural network based on residual learning is used as a deep learning image classification model;

[0025] The residual learning-based convolutional neural network is used to perform multi-level feature extraction on the image of each core grid region and compress the extracted image features into a feature vector. By performing non-linear mapping on the feature vector, the probability distribution of each core grid region belonging to each defect category is calculated, and a defect category label for each core grid region is generated according to the defect category with the highest probability value.

[0026] A grid defect classification dataset is constructed by acquiring several grid region images with labeled defect categories.

[0027] Supervised training of a deep learning image classification model was performed using a lattice defect classification dataset, resulting in a well-trained deep learning image classification model.

[0028] Furthermore, the defect category label includes at least: qualified core cells and unqualified core cells.

[0029] Furthermore, based on the acquired pose parameters, the spatial position and normal direction of each core region in the base coordinate system are reconstructed by projecting each core region in all images onto a unified base coordinate system:

[0030] Based on the hand-eye calibration results of the robot vision system, the base coordinate system of the industrial robot is determined. The end-effector coordinate system of industrial robots Camera coordinate system of area scan camera Then, construct the homogeneous transformation matrix from the camera coordinate system to the base coordinate system. , is represented as:

[0031]

[0032] For any lattice region, based on the calibrated telecentric lens in the robot vision system, when the lattice region is clearly imaged in the image, the working distance of the telecentric lens is adjusted. The depth of the center point of the core grid region;

[0033] Obtain the pixel coordinates of the center point of the grid region in the image, and combine this with the depth of the center point of the grid region to calculate the coordinates of the center point of the grid region in the camera coordinate system. , is represented as:

[0034]

[0035] in The three-dimensional coordinates of the center point of the core region in the camera coordinate system; This is the intrinsic parameter matrix of the camera; The pixel coordinates of the center point of the grid region in the image;

[0036] Based on the homogeneous transformation matrix from the camera coordinate system to the base coordinate system The coordinates of the center point of the lattice region in the camera coordinate system Project the center point of the lattice region onto the base coordinate system to obtain the coordinates of the center point of the lattice region in the base coordinate system. , is represented as:

[0037]

[0038] In the base coordinate system, the geometry of the core region is defined as follows: With point A as the center, the radius of the circumcircle is . The regular hexagon, and based on the homogeneous transformation matrix from the camera coordinate system to the base coordinate system. Determine the normal direction of the core grid region.

[0039] Furthermore, the method for determining the clarity of the image is as follows:

[0040] For any image, let it be denoted as... ,in Used to locate any pixel in the image;

[0041] Laplacian filtering is applied to the image of the core region to obtain the Laplacian filtering result. ;

[0042] Statistical Laplace Filtering Results gradient variance And it serves as an evaluation metric for image sharpness;

[0043] For any given grid region, images of that region are acquired at different distances within a preset range using a robot vision system that has undergone hand-eye calibration, and an image sequence is constructed. The gradient variance in the image sequence is then analyzed. The largest image is the one with the clearest imaging.

[0044] Furthermore, the specific content of generating the three-dimensional visualization of the cellular structure part to be inspected by integrating the spatial position and normal direction of all cellular regions in the base coordinate system and the defect category label of each cellular region during the process of projecting each cellular region is as follows:

[0045] Initialize a core grid sequence to record and save the spatial position and normal direction of each core grid region in the base coordinate system, as well as the defect category label of each core grid region, in the order of projection.

[0046] During the process of projecting each core region one by one, for the currently projected core region, on the XOY plane of the base coordinate system, with the center point of the core region... With the center point as the boundary, construct a side with length as the boundary point. The rectangular region is used as the neighborhood of the core region. ;in The center point of this lattice region Coordinates in the base coordinate system;

[0047] Obtain the current lattice sequence in the neighborhood. For all the center points of the core grid regions within the given set, determine whether there exists a center point that is the same as the center point of the given core grid region. The distance between them is less than The center point of the core grid region; if it does not exist, the core grid region of the current projection is saved to the current core grid sequence; if it exists, it means that there is a core grid region in the current core grid sequence that is the same as the core grid region of the current projection, and the core grid region that is the same in the current core grid sequence is taken as the original core grid region.

[0048] Calculate the distance between the center point of the original grid region and the center point of the image containing the original grid region. Calculate the distance between the center point of the currently projected grid region and the center point of the image containing the currently projected grid region. ;

[0049] like If the original core area is retained, the currently projected core area is ignored, and the projection of the next core area begins; if If the current core region is used to replace the original core region in the current core region sequence, then the projection of the next core region will begin.

[0050] After projecting all lattice regions, the final lattice sequence is used as the 3D visualization lattice defect detection result of the honeycomb structure part to be inspected.

[0051] On the other hand, this invention proposes a cellular structure component core defect detection system based on robot vision, the system comprising:

[0052] The data acquisition module uses a robot vision system that has undergone hand-eye calibration to acquire multi-view images of the honeycomb structure parts to be inspected, and simultaneously acquires the pose parameters of the industrial robot end effector in the robot vision system when each image is acquired.

[0053] The image segmentation module employs a deep learning-based instance segmentation method / deep learning-based image instance segmentation model to identify and divide mutually independent core regions in each image.

[0054] The defect identification module uses a deep learning image classification model to identify defects in each core grid region and generate a defect category label for each core grid region.

[0055] The core-grid reconstruction module, based on the acquired pose parameters, reconstructs the spatial position and orientation of each core-grid region in the base coordinate system by projecting each core-grid region in all images onto a unified base coordinate system.

[0056] The data fusion module, during the process of projecting each core area one by one, generates a three-dimensional visualized core defect detection result for the honeycomb structure part to be inspected by fusing the spatial position and normal direction of all core areas in the base coordinate system and the defect category label of each core area.

[0057] Thirdly, the present invention proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for detecting lattice defects in honeycomb structure parts based on robot vision.

[0058] The beneficial effects of adopting the above technical solution are as follows:

[0059] 1. This invention utilizes the flexible movement and precise positioning capabilities of industrial robots to acquire images that comprehensively cover honeycomb structure parts from multiple perspectives, avoiding missed detections caused by insufficient field of view or obstruction.

[0060] 2. This invention can uniformly map the core grid defect detection results to a base coordinate system, realizing the location and three-dimensional visualization of defective core grids. This provides an effective reference for subsequent quality traceability, defect repair, and production optimization.

[0061] 3. In the process of integrating information from multiple perspectives, duplicate cores in spatial location are identified and removed to avoid misjudgment and duplicate statistics, thereby effectively improving the uniqueness and accuracy of the detection results.

[0062] 4. The detection process of this invention has a high degree of automation, which can complete the defect detection of all core cells of the entire part in one go. It is easy to operate, has high detection efficiency, and has a wide range of engineering application value. Attached Figure Description

[0063] Figure 1 This is a flowchart of the cellular structure component core defect detection method based on robot vision in this embodiment;

[0064] Figure 2 This is a data flow diagram of the cellular structure component core defect detection method based on robot vision in this embodiment;

[0065] Figure 3 This is a schematic diagram of the robot vision system in this embodiment;

[0066] Figure 4 This is an example diagram illustrating the effect of lattice instance segmentation and defect determination in this embodiment;

[0067] Figure 5 This is a flowchart illustrating the fusion of the spatial location and defect category labels of all core regions in this embodiment;

[0068] Figure 6 This is an example diagram showing the effect of 3D visualization of defect detection results for honeycomb structure parts in this embodiment;

[0069] Figure 7 This is a structural diagram of the cellular structure component lattice defect detection system based on robot vision in this embodiment;

[0070] In the diagram: 1-Working platform, 2-Positioning fixture for honeycomb structure parts, 3-Industrial robot, 4-Area array camera, 5-Telecentric lens, 6-Right angle prism, 7-Coaxial light source, 8-Honeycomb structure part to be tested. Detailed Implementation

[0071] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0072] Example 1:

[0073] This embodiment presents a method for detecting core defects in honeycomb structure parts based on robot vision, such as... Figure 1 and Figure 2 As shown, the method includes the following steps:

[0074] The robot vision system, which has been calibrated with hand-eye alignment, is used to acquire multi-view images of the honeycomb structure parts to be inspected, and the pose parameters of the industrial robot end effector in the robot vision system are acquired simultaneously when each image is acquired.

[0075] The robot vision system includes: an industrial control computer, a vision controller, a work platform 1, a honeycomb structure part positioning fixture 2, an industrial robot 3, an area array camera 4, a telecentric lens 5, a right-angle prism 6, and a coaxial light source 7.

[0076] The honeycomb structure part positioning fixture 2 is fixedly installed on the working platform 1 to fix the honeycomb structure part 8 to be inspected.

[0077] The area array camera 4, telecentric lens 5, right-angle prism 6, and coaxial light source 7 are integrated into a visual sensor assembly through a mechanical structure for image acquisition of the honeycomb structure part 8 to be inspected. The area array camera 4 is directly connected to the telecentric lens 5, and the right-angle prism 6 is installed at the front end of the telecentric lens 5. The right-angle prism 6 is used to reflect the horizontal light path of the area array camera 4 into a vertical light path. The coaxial light source 7 and the right-angle prism 6 are fixedly installed on the same bracket structure, and the coaxial light source 7 is installed below the right-angle prism 6. The light emission direction of the coaxial light source 7 is consistent with the imaging light path direction after being deflected by the right-angle prism 6.

[0078] The vision sensor assembly is fixedly installed at the end of the industrial robot 3.

[0079] The industrial robot 3 is used to move the vision sensor components to a predetermined position according to the control instructions from the industrial control computer in order to complete the automatic image acquisition task.

[0080] The vision controller is fixedly installed on the working platform and is used to receive vision parameter instructions from the industrial control computer and control the working status of the area array camera 4, telecentric lens 5 and coaxial light source 7 in the vision sensor assembly.

[0081] In this embodiment, as Figure 3 As shown, the hardware structure of the robot vision system mainly consists of a working platform 1, a honeycomb structure part positioning fixture 2, an industrial robot 3, an area array camera 4, a telecentric lens 5, a right-angle prism 6, and a coaxial light source 7. The robot vision system also includes an industrial control computer and a vision controller mounted on the control platform 1. The base coordinate system of the industrial robot 3 is represented as follows: The end-effector coordinate system of industrial robot 3 is represented as The camera coordinate system of area scan camera 4 is represented as The robot vision system has undergone hand-eye calibration, and the camera's intrinsic parameter matrix is... .

[0082] The acquired images are processed using a pre-trained deep learning-based image instance segmentation model to identify and segment independent core regions in each image.

[0083] The pre-trained deep learning-based image instance segmentation model is as follows:

[0084] A single-stage instance segmentation model is adopted as the deep learning-based image instance segmentation model. This model extracts and fuses multi-scale features of the input image to generate a mask feature map. The mask feature map is then gridded, and each grid cell is used to predict whether a core instance exists at its center. A dynamic convolution kernel is predicted for each grid cell, and the dynamic convolution kernel of each grid cell is convolved with the mask feature map to generate an instance-aware mask and a corresponding confidence score for each grid cell. The instance-aware masks corresponding to each grid cell are then filtered to remove overlapping masks and those with confidence scores below a preset threshold. The filtered instance-aware masks are defined as mutually independent core regions in the input image.

[0085] A core-grid instance segmentation dataset was constructed by collecting several images of honeycomb structure parts, and a deep learning-based image instance segmentation model was trained using the core-grid instance segmentation dataset to obtain the trained deep learning-based image instance segmentation model.

[0086] In this embodiment, Solo-v2 is used as a deep learning-based image instance segmentation model, trained using a core-grid instance segmentation dataset. During training, a joint loss function consisting of Focal Loss and Dice Loss is used for iterative optimization. By minimizing the joint loss function, the trained Solo-v2 is obtained. The trained Solo-v2 is then used to segment independent core regions from the input image, achieving excellent segmentation results that meet the needs of engineering applications.

[0087] A pre-trained deep learning image classification model is used to identify defects in each divided core region, and a defect category label is generated for each core region.

[0088] The defect category label includes at least: qualified core cells and unqualified core cells.

[0089] The pre-trained deep learning image classification model is:

[0090] A convolutional neural network based on residual learning is used as a deep learning image classification model.

[0091] The residual learning-based convolutional neural network is used to perform multi-level feature extraction on the image of each core grid region and compress the extracted image features into feature vectors. By performing nonlinear mapping on the feature vectors, the probability distribution of each core grid region belonging to each defect category is calculated, and a defect category label for each core grid region is generated according to the defect category with the highest probability value.

[0092] A grid defect classification dataset is constructed by acquiring several grid region images with labeled defect categories.

[0093] Supervised training of a deep learning image classification model was performed using a lattice defect classification dataset, resulting in a well-trained deep learning image classification model.

[0094] In this embodiment, ResNet18 is used as the deep learning image classification model. ResNet18 comprises, in series: an initial convolutional layer, pooling layers, several stacked residual blocks, and a classifier. The initial convolutional layer and pooling layers are used for preliminary feature extraction and data downsampling of the input image. The residual blocks are used to further extract features from the input using the convolutional layers, and then fuse the extracted features with the input features to achieve residual learning. Different groups of residual blocks have different numbers of output channels, and the number of output channels increases sequentially from input to output. The classifier compresses the input feature map into a feature vector using global average pooling, then maps the obtained feature vector to the defect category space using a fully connected layer, and outputs the probability of each category using a Softmax function. Finally, a defect category label is generated for each core region based on the probability values. Other convolutional neural networks based on residual learning can also be used for deep learning image classification models. This embodiment uses ResNet18, which has high accuracy. Examples of the performance in core instance segmentation and defect determination are shown below. Figure 4 As shown.

[0095] Based on the acquired pose parameters, the spatial position and orientation of each core region in the base coordinate system are restored by projecting each core region in all images onto a unified base coordinate system.

[0096] In this embodiment, by using an actual circumscribed circle with a radius of... The center point has pixel coordinates in the image. The core grid is projected onto the base coordinate system to restore the position and normal direction of the core grid represented by each core grid region in the base coordinate system.

[0097] Based on the acquired pose parameters, the spatial position and normal direction of each core region in the images are reconstructed by projecting each core region in the images onto a unified base coordinate system.

[0098] Based on the hand-eye calibration results of the robot vision system, the base coordinate system of the industrial robot is determined. The end-effector coordinate system of industrial robots Camera coordinate system of area scan camera Then, construct the homogeneous transformation matrix from the camera coordinate system to the base coordinate system. .

[0099] The homogeneous transformation matrix from the camera coordinate system to the base coordinate system Represented as:

[0100]

[0101] in This is the homogeneous transformation matrix from the end coordinate system to the base coordinate system, which is read through the communication interface of the industrial robot control system. The homogeneous transformation matrix from the camera coordinate system to the end coordinate system is obtained through hand-eye calibration.

[0102] For any lattice region, based on the calibrated telecentric lens in the robot vision system, when the lattice region is clearly imaged in the image, the working distance of the telecentric lens is adjusted. The depth is the center point of the lattice region.

[0103] The method for determining image clarity is as follows:

[0104] For any image, let it be denoted as... ,in Used to locate any pixel in the image.

[0105] Laplacian filtering is applied to the image of the core region to obtain the Laplacian filtering result. , is represented as:

[0106]

[0107] Statistical Laplace Filtering Results gradient variance It is used as an evaluation index for image sharpness; its calculation formula is:

[0108]

[0109] in The width of the original image being acquired; The height of the original image being acquired; and The unit is pixels; This represents the Laplacian filtering result corresponding to all grid regions in the acquired original image. The average value.

[0110] For any given grid region, images of that region are acquired at different distances within a preset range using a robot vision system that has undergone hand-eye calibration, and an image sequence is constructed. The gradient variance in the image sequence is then analyzed. The largest image is the one with the clearest imaging.

[0111] In this embodiment, since the robot vision system uses a telecentric lens, its effective depth of field is a fixed value after calibration. Therefore, when a cell region is clearly imaged in the image, the depth of the center point of that cell can be considered equal to the working distance of the telecentric lens at that time. .

[0112] Obtain the pixel coordinates of the center point of the grid region in the image, and combine this with the depth of the center point of the grid region to calculate the coordinates of the center point of the grid region in the camera coordinate system. .

[0113] The coordinates of the center point of the grid region in the camera coordinate system for:

[0114]

[0115] in The three-dimensional coordinates of the center point of the core region in the camera coordinate system; This is the intrinsic parameter matrix of the camera; The pixel coordinates of the center point of the grid region in the image.

[0116] Based on the homogeneous transformation matrix from the camera coordinate system to the base coordinate system The coordinates of the center point of the lattice region in the camera coordinate system Project the center point of the lattice region onto the base coordinate system to obtain the coordinates of the center point of the lattice region in the base coordinate system. .

[0117] The coordinates of the center point of the lattice region in the base coordinate system for:

[0118]

[0119] In the base coordinate system, the geometry of the core region is defined as follows: With point A as the center, the radius of the circumcircle is . The regular hexagon, and based on the homogeneous transformation matrix from the camera coordinate system to the base coordinate system. Determine the normal direction of the core grid region.

[0120] In this embodiment, the center point of the lattice region in the base coordinate system is used as the reference point. Centered on, through Determine the direction of the normal and draw the circumcircle with a radius of [missing information]. The goal is to reconstruct the position and orientation of each core region in three-dimensional space using a regular hexagon. This involves creating a 3D model of each core region in actual three-dimensional space, including the following elements: Center point: the coordinates of the core region's center point in the base coordinate system; Planar shape: a circle centered on this center point with fixed values... It is a regular hexagon with a circumcircle radius; normal direction: usually taken from the Z-axis direction vector in the homogeneous transformation matrix from the camera coordinate system to the base coordinate system.

[0121] During the process of projecting each core region one by one, the spatial position and normal direction of all core regions in the base coordinate system, as well as the defect category label of each core region, are fused to generate a three-dimensional visualization of the core defect detection results of the honeycomb structure part to be inspected.

[0122] In this embodiment, as Figure 5 As shown, a real circumcircle with radius is The coordinates of the center point in the base coordinate system are: The new core grid is integrated into the original core grid sequence in the base coordinate system to complete the identification and removal of duplicate core grids. That is, by integrating the multi-view image detection results core by core, a core grid sequence in the base coordinate system is formed. During the integration process, duplicate core grids in spatial location are identified and removed to avoid redundancy, and finally, a complete 3D visualization defect detection result for the honeycomb structure part is generated.

[0123] The specific content of generating the three-dimensional visualization of the core grid defect detection result of the honeycomb structure part to be inspected by integrating the spatial position and normal direction of all core grid regions in the base coordinate system and the defect category label of each core grid region during the process of projecting each core grid region is as follows:

[0124] Initialize a core grid sequence to record and save the spatial position and normal direction of each core grid region in the base coordinate system, as well as the defect category label of each core grid region, in the order of projection.

[0125] During the process of projecting each core region one by one, for the currently projected core region, on the XOY plane of the base coordinate system, with the center point of the core region... With the center point as the boundary, construct a side with length as the boundary point. The rectangular region is used as the neighborhood of the core region. ;in The center point of this lattice region Coordinates in the base coordinate system.

[0126] In this embodiment, neighborhood Represented as:

[0127]

[0128] in To define the parameters for the neighborhood size, satisfying .

[0129] Obtain the current lattice sequence in the neighborhood. For all the center points of the core grid regions within the given set, determine whether there exists a center point that is the same as the center point of the given core grid region. The distance between them is less than The center point of the core region. If it does not exist, the core region of the current projection is saved to the current core region sequence; if it exists, it means that there is a core region in the current core region sequence that is the same as the core region of the current projection, and the core region that is the same in the current core region sequence is taken as the original core region.

[0130] Calculate the distance between the center point of the original grid region and the center point of the image containing the original grid region. Calculate the distance between the center point of the currently projected grid region and the center point of the image containing the currently projected grid region. .

[0131] like If the original core area is retained, the currently projected core area is ignored, and the projection of the next core area begins; if If the current projection area replaces the original area in the current grid sequence, then the projection of the next area begins.

[0132] After projecting all lattice regions, the final lattice sequence is used as the 3D visualization lattice defect detection result of the honeycomb structure part to be inspected.

[0133] In this embodiment, the Euclidean distance from the center point of the currently projected grid region and the original overlapping grid region to the center of their respective image regions is used to retain grid regions with smaller distances and remove grid regions with larger distances; specifically, this includes:

[0134] The pixel coordinates of the currently projected lattice region in its own image are: The original core grid center point has pixel coordinates in its own image. The Euclidean distance from the center point of each grid region to the center of its respective image is calculated using the following formula. and :

[0135]

[0136] in This represents Euclidean distance. If... If the current projection's core region is closer to the center of the image it is in, it has higher detection and localization accuracy. Therefore, the original core region that overlaps with it is removed from the core region sequence, and the core region of the current projection is retained. Conversely, if the current projection's core region is not closer to the center of the image, the original core region that overlaps with it is removed, and the original core region that overlaps with it is retained.

[0137] In this embodiment, after the above process, a complete three-dimensional visualization defect detection result for the honeycomb structure part is obtained, such as... Figure 6 As shown.

[0138] Example 2:

[0139] This embodiment presents a cellular structure component core defect detection system based on robot vision, such as... Figure 7 As shown, the system includes:

[0140] The data acquisition module uses a robot vision system that has undergone hand-eye calibration to acquire multi-view images of the honeycomb structure parts to be inspected, and simultaneously acquires the pose parameters of the industrial robot end effector in the robot vision system when each image is acquired.

[0141] The image segmentation module uses a pre-trained deep learning-based image instance segmentation model to process the acquired images, identify and divide the independent core regions in each image.

[0142] The defect identification module uses a pre-trained deep learning image classification model to identify defects in each divided core area and generate a defect category label for each core area.

[0143] The core-grid reconstruction module, based on the acquired pose parameters, reconstructs the spatial position and orientation of each core-grid region in the base coordinate system by projecting each core-grid region in all images onto a unified base coordinate system.

[0144] The data fusion module, during the process of projecting each core area one by one, generates a three-dimensional visualized core defect detection result for the honeycomb structure part to be inspected by fusing the spatial position and normal direction of all core areas in the base coordinate system and the defect category label of each core area.

[0145] Example 3:

[0146] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors cause the one or more processors to execute the robot vision-based cellular structure component lattice defect detection method.

[0147] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the cellular structure component lattice defect detection method based on robot vision as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0148] The processor is used to execute all or part of the steps in the robot vision-based cellular structure component lattice defect detection method described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0149] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the robot vision-based cellular structure component lattice defect detection method described in the above embodiments.

[0150] Example 4:

[0151] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0152] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the robot vision-based cellular structure part core defect detection method described in the various embodiments of this application.

[0153] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory, random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned robot vision-based cellular structure component lattice defect detection method.

[0154] Example 5:

[0155] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for detecting lattice defects in honeycomb structure parts based on robot vision.

[0156] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0157] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0158] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A method for detecting core defects in honeycomb structure parts based on robot vision, characterized in that, This method includes the following steps: The robot vision system, which has been calibrated with hand and eye, is used to acquire multi-view images of the honeycomb structure parts to be inspected, and the pose parameters of the industrial robot end in the robot vision system are acquired simultaneously when each image is acquired. The acquired images are processed using a pre-trained deep learning-based image instance segmentation model to identify and delineate independent core regions in each image. A pre-trained deep learning image classification model is used to identify defects in each divided core region, and a defect category label is generated for each core region. Based on the acquired pose parameters, the spatial position and normal direction of each core region in the base coordinate system are restored by projecting each core region in all images onto a unified base coordinate system. During the process of projecting each core region one by one, the spatial position and normal direction of all core regions in the base coordinate system, as well as the defect category label of each core region, are fused to generate a three-dimensional visualization of the core defect detection results of the honeycomb structure part to be inspected.

2. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 1, characterized in that, The robot vision system includes: an industrial control computer, a vision controller, a work platform, a honeycomb structure part positioning fixture, an industrial robot, an area scan camera, a telecentric lens, a right-angle prism, and a coaxial light source. The honeycomb structure part positioning fixture is fixedly installed on the working platform to fix the honeycomb structure part to be tested; The area array camera, telecentric lens, right-angle prism, and coaxial light source are integrated into a visual sensor assembly via a mechanical structure for image acquisition of the honeycomb structure parts to be inspected. The area array camera is directly connected to the telecentric lens, and the right-angle prism is mounted at the front end of the telecentric lens. The right-angle prism reflects the horizontal light path of the area array camera into a vertical light path. The coaxial light source and the right-angle prism are fixedly mounted on the same support structure, with the coaxial light source positioned below the right-angle prism. The light emission direction of the coaxial light source is consistent with the direction of the imaging light path after being deflected by the right-angle prism. The vision sensor assembly is installed at the end of the industrial robot; The industrial robot is used to move the vision sensor components to a predetermined position according to the control instructions from the industrial control computer in order to complete the automatic image acquisition task. The vision controller is used to receive vision parameter instructions from an industrial control computer and control the working status of the area scan camera, telecentric lens and coaxial light source in the vision sensor assembly.

3. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 2, characterized in that, The pre-trained deep learning-based image instance segmentation model is as follows: A single-stage instance segmentation model is adopted as the deep learning-based image instance segmentation model. The single-stage instance segmentation model is used to extract and fuse multi-scale features of the input image to generate a mask feature map of the input image; the mask feature map is gridded, and each grid cell is used to predict whether there is a core instance at the center of the grid cell; a dynamic convolution kernel is predicted for each grid cell, and the dynamic convolution kernel of each grid cell is convolved with the mask feature map to generate an instance-aware mask and a corresponding confidence score for each grid cell; the instance-aware masks corresponding to each grid cell are filtered to remove overlapping and instance-aware masks with confidence scores below a preset threshold, and the filtered instance-aware masks are defined as mutually independent core regions in the input image; A core-grid instance segmentation dataset was constructed by collecting several images of honeycomb structure parts, and a deep learning-based image instance segmentation model was trained using the core-grid instance segmentation dataset to obtain the trained deep learning-based image instance segmentation model.

4. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 3, characterized in that, The pre-trained deep learning image classification model is: A convolutional neural network based on residual learning is used as a deep learning image classification model; The residual learning-based convolutional neural network is used to perform multi-level feature extraction on the image of each core grid region and compress the extracted image features into a feature vector. By performing non-linear mapping on the feature vector, the probability distribution of each core grid region belonging to each defect category is calculated, and a defect category label for each core grid region is generated according to the defect category with the highest probability value. A grid defect classification dataset is constructed by acquiring several grid region images with labeled defect categories. Supervised training of a deep learning image classification model was performed using a lattice defect classification dataset, resulting in a well-trained deep learning image classification model.

5. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 4, characterized in that, The defect category label includes at least: qualified core cells and unqualified core cells.

6. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 5, characterized in that, Based on the acquired pose parameters, the spatial position and normal direction of each core region in the images are reconstructed by projecting each core region in the images onto a unified base coordinate system. Based on the hand-eye calibration results of the robot vision system, the base coordinate system of the industrial robot is determined. The end-effector coordinate system of industrial robots Camera coordinate system of area scan camera Then, construct the homogeneous transformation matrix from the camera coordinate system to the base coordinate system. , is represented as: ； For any lattice region, based on the calibrated telecentric lens in the robot vision system, when the lattice region is clearly imaged in the image, the working distance of the telecentric lens is adjusted. The depth of the center point of the core grid region; Obtain the pixel coordinates of the center point of the grid region in the image, and combine this with the depth of the center point of the grid region to calculate the coordinates of the center point of the grid region in the camera coordinate system. , is represented as: ； in The three-dimensional coordinates of the center point of the core region in the camera coordinate system; This is the intrinsic parameter matrix of the camera; The pixel coordinates of the center point of the grid region in the image; Based on the homogeneous transformation matrix from the camera coordinate system to the base coordinate system The coordinates of the center point of the lattice region in the camera coordinate system Project the center point of the lattice region onto the base coordinate system to obtain the coordinates of the center point of the lattice region in the base coordinate system. , is represented as: ； In the base coordinate system, the geometry of the core region is defined as follows: With point A as the center, the radius of the circumcircle is . The regular hexagon, and based on the homogeneous transformation matrix from the camera coordinate system to the base coordinate system. Determine the normal direction of the core grid region.

7. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 6, characterized in that, The method for determining image clarity is as follows: For any image, let it be denoted as... ,in Used to locate any pixel in the image; Laplacian filtering is applied to the image of the core region to obtain the Laplacian filtering result. ; Statistical Laplace Filtering Results gradient variance And it serves as an evaluation metric for image sharpness; For any given grid region, images of that region are acquired at different distances within a preset range using a robot vision system that has undergone hand-eye calibration, and an image sequence is constructed. The gradient variance in the image sequence is then analyzed. The largest image is the one with the clearest imaging.

8. The method for detecting core defects in honeycomb structure parts based on robot vision according to claim 7, characterized in that, The specific content of generating the three-dimensional visualization of the core grid defect detection result of the honeycomb structure part to be inspected by integrating the spatial position and normal direction of all core grid regions in the base coordinate system and the defect category label of each core grid region during the process of projecting each core grid region is as follows: Initialize a core grid sequence to record and save the spatial position and normal direction of each core grid region in the base coordinate system, as well as the defect category label of each core grid region, in the order of projection. During the process of projecting each core region one by one, for the currently projected core region, on the XOY plane of the base coordinate system, with the center point of the core region... With the center point as the boundary, construct a side with length as the boundary point. The rectangular region is used as the neighborhood of the core region. ;in The center point of this lattice region Coordinates in the base coordinate system; Obtain the current lattice sequence in the neighborhood. For all the center points of the core grid regions within the given set, determine whether there exists a center point that is the same as the center point of the given core grid region. The distance between them is less than The center point of the core grid region; if it does not exist, the core grid region of the current projection is saved to the current core grid sequence; if it exists, it means that there is a core grid region in the current core grid sequence that is the same as the core grid region of the current projection, and the core grid region that is the same in the current core grid sequence is taken as the original core grid region. Calculate the distance between the center point of the original grid region and the center point of the image containing the original grid region. Calculate the distance between the center point of the currently projected grid region and the center point of the image containing the currently projected grid region. ; like If the original core area is retained, the currently projected core area is ignored, and the projection of the next core area begins; if If the current core region is used to replace the original core region in the current core region sequence, then the projection of the next core region will begin. After projecting all lattice regions, the final lattice sequence is used as the 3D visualization lattice defect detection result of the honeycomb structure part to be inspected.

9. A robot vision-based cellular structure component core lattice defect detection system, used to implement the robot vision-based cellular structure component core lattice defect detection method according to any one of claims 1-8, characterized in that, The system includes: The data acquisition module uses a robot vision system that has undergone hand-eye calibration to acquire multi-view images of the honeycomb structure parts to be inspected, and simultaneously acquires the pose parameters of the industrial robot end effector in the robot vision system when each image is acquired. The image segmentation module employs a deep learning-based instance segmentation method / deep learning-based image instance segmentation model to identify and divide mutually independent core regions in each image. The defect identification module uses a deep learning image classification model to identify defects in each core grid region and generate a defect category label for each core grid region. The core-grid reconstruction module, based on the acquired pose parameters, reconstructs the spatial position and orientation of each core-grid region in the base coordinate system by projecting each core-grid region in all images onto a unified base coordinate system. The data fusion module, during the process of projecting each core area one by one, generates a three-dimensional visualized core defect detection result for the honeycomb structure part to be inspected by fusing the spatial position and normal direction of all core areas in the base coordinate system and the defect category label of each core area.

10. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the method for detecting lattice defects in honeycomb structure parts based on robot vision as described in any one of claims 1-8.

Images