A somatic autonomous imaging method and system for defect detection of complex shaped industrial components

Abstract

The application discloses a kind of somatic autonomous imaging method and system for complex special-shaped industrial component defect detection, comprising: two-stage viewpoint planning based on digital twin model, generate front and back surface optimization observation strategy;Workpiece is placed in imaging table, and the actual pose is obtained by 3D camera and the online correction of theoretical viewpoint is carried out, and executable viewpoint queue is generated;Control detection mechanical arm to execute imaging according to queue, and real-time quality evaluation is carried out to image, and local replay or re-planning is triggered for abnormal viewpoint;Change workpiece posture by double mechanical arm cooperative turning over, repeat the step to complete front and back surface imaging;The workpiece that has been detected is transferred to unloading area and the next workpiece is grabbed, and the state is updated and according to model selection strategy, form continuous cycle;Multi-view image defect identification result is mapped to three-dimensional coordinate system output. Through offline planning, online correction and double mechanical arm turning over, the autonomous imaging of workpiece front and back surface and occluded area is realized, and the imaging coverage ability and stability are improved.

Description

Technical Field

[0001] This invention belongs to the field of defect detection technology, and in particular relates to an embodied autonomous imaging method and system for defect detection of complex irregular-shaped industrial components. Background Technology

[0002] Complex and irregularly shaped industrial components are widely found in the automotive, aerospace, rail transportation, energy equipment, engineering machinery, and high-end manufacturing industries. Their surface quality and structural integrity directly affect the performance, reliability, and service safety of equipment. Defect detection of complex and irregularly shaped industrial components aims to identify defects such as cracks, scratches, dents, burrs, missing materials, dirt, and coating abnormalities that occur during material preparation, casting, forging, welding, machining, surface treatment, and assembly. This enables timely assessment of product quality and process traceability. As industrial manufacturing moves towards higher speeds, automation, and flexibility, traditional manual inspection methods are gradually becoming insufficient to meet production demands in terms of efficiency, consistency, traceability, and comprehensive coverage of complex surfaces.

[0003] In recent years, automated inspection equipment based on 2D images or 3D point clouds has been gradually applied to industrial inspection scenarios, which can improve inspection efficiency and consistency to a certain extent. However, complex and irregularly shaped industrial components typically have characteristics such as multiple curved surfaces, numerous occlusions, strong reflectivity, unstable orientation, and discrete distribution of inspection areas. This leads to problems such as insufficient viewpoint coverage, difficulty in imaging the bottom or occluded surfaces, and difficulty in timely closed-loop correction after local imaging failures when using existing technologies to image the surface of complex and irregularly shaped components. As a result, some defects are missed, affecting the accuracy and reliability of the inspection results. Existing research has explored robot vision inspection path planning, viewpoint placement based on 3D models, multi-robotic arm collaborative inspection, and multi-view defect recognition, but an integrated autonomous imaging closed-loop solution for full-surface defect detection of complex and irregularly shaped industrial components has not yet been formed.

[0004] In related technologies, some solutions mainly rely on multiple robots to collaboratively complete the photography task, but they cannot capture images of key areas such as the bottom surface of the workpiece. Therefore, in the scenario of full-surface inspection of complex and irregularly shaped industrial components, there may still be imaging blind spots, making it difficult to meet the high-coverage autonomous imaging requirements of complex structural components.

[0005] Another approach can generate scanning paths or measurement strategies based on 3D models, providing a foundation for model-driven automatic detection. However, it focuses more on scanning measurement or geometric detection, paying insufficient attention to the imaging quality, angle coverage, and imaging consistency of defect detection. Therefore, it is difficult to directly adapt to the high-sensitivity imaging requirements in the defect detection of complex irregular components.

[0006] Some solutions involve the collaborative application of multiple robotic arms in surface defect detection tasks. However, these solutions primarily focus on the collaborative detection itself, lacking sufficient disclosure on how to form an observation strategy around the workpiece during the detection of complex, irregularly shaped industrial components, and how to perform local re-imaging and replanning when imaging quality is abnormal or the local environment changes. Furthermore, the sequential coupling between multiple robotic arms means that any arm's replanning results in global waiting, leading to unstable cycle times. Therefore, there is still room for improvement in scenarios involving autonomous imaging and high-coverage defect detection of the entire surface of complex, irregularly shaped industrial components.

[0007] In addition, some solutions have proposed a robot visual inspection path planning method based on visual observation results, indicating that existing technologies have recognized the feasibility of robots carrying visual sensors for automatic inspection and path generation; however, such methods focus more on generating inspection paths based on observation information and do not directly address the system design of complex irregular industrial components in terms of flipping, covering obstructed areas, and closed-loop detection.

[0008] Therefore, how to achieve high-coverage autonomous imaging, effectively avoid imaging blind spots, ensure consistent imaging quality, and have adaptive adjustment capabilities when imaging is abnormal or local environment changes are needed in the full-surface defect detection of complex and irregular industrial components, thereby improving the stability, integrity and traceability of the detection results, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] To address the above technical problems, this invention provides an embodied autonomous imaging method and system for defect detection of complex, irregularly shaped industrial components.

[0010] The technical solution adopted by this invention to solve its technical problem is: An embodied autonomous imaging method for defect detection of complex, irregularly shaped industrial components includes the following steps: S100: Based on the digital twin model of the workpiece to be inspected, a two-stage viewpoint planning is performed to generate optimized observation strategies for the front and back of the workpiece, respectively, as the theoretical viewpoint basis; S200: After placing the workpiece to be inspected on the imaging stage, the actual pose of the workpiece is obtained through a 3D camera, and the theoretical viewpoint in the optimized observation strategy is corrected online using the actual pose to generate an executable viewpoint queue. S300: Controls the robotic arm to perform imaging tasks sequentially according to the viewpoint queue, and performs real-time quality assessment on the acquired images. When the images do not meet the requirements, it triggers local replanning or local re-shooting for abnormal viewpoints to achieve closed-loop adjustment of the imaging process. S400: By using the coordinated operation of two robotic arms or the attitude adjustment mechanism, the placement posture of the workpiece in the imaging station is changed to expose the unimaged area under the original posture. The online correction and viewpoint queue generation in S200 and the imaging and closed-loop feedback in S300 are repeated to complete the imaging of the front and back of the workpiece. S500: After completing the imaging of the front and back sides, the inspected workpiece is transferred to the unloading area. At the same time, or when there is no spatial interference, the next workpiece to be inspected is grabbed and placed on the imaging stage. The current workpiece status and environmental status are updated. Based on the model of the next workpiece, the pre-generated observation strategy is called or the viewpoint planning process is re-executed to form a continuous cycle of autonomous imaging operation. S600: It performs defect identification on the acquired multi-view images, maps the identified two-dimensional defect results to a unified three-dimensional coordinate system, and outputs the three-dimensional spatial location information of the defect.

[0011] Preferably, S100 includes: S110: The central control and data processing module establishes a digital twin model based on the CAD model of the workpiece to be inspected, and preprocesses the model according to the geometric features of the workpiece surface to extract key detection areas such as edges and openings, forming a lightweight geometric expression for subsequent viewpoint planning. S120: The viewpoint planning is divided into a front inspection stage and a back inspection stage. The set of front inspection areas is determined based on the workpiece's upright position on the imaging stage. The set of areas to be inspected on the reverse side is determined based on the target posture after the workpiece is flipped over. ; S130: The surface to be inspected is divided into several imaging sub-regions. The effective imaging coverage is determined based on the field of view, focal length, working distance range, depth of field range, incident angle limit, and resolution requirements of the 2D vision module. For each imaging sub-region, candidate coverage is constructed using a frustum coverage analysis method, taking into account the local surface orientation and imaging constraints of that sub-region. The candidate pose of the 2D vision module relative to the CAD model is then solved in reverse. This forms a set of positive candidate viewpoints. and the set of opposing candidate viewpoints ; S140: Visibility analysis, collision constraint analysis, safety distance analysis, and imaging quality analysis are performed on candidate viewpoints to screen out effective observation viewpoints that meet coverage requirements, imaging requirements, and robotic arm execution requirements, and front-side optimized observation strategies are obtained accordingly. and the opposite optimization observation strategy Furthermore, simulation analysis was conducted on the imaging effect of the workpiece under the candidate viewpoints to evaluate the coverage and imaging adaptability. Based on the candidate viewpoint set, the observability of the workpiece surface was analyzed. Through comprehensive analysis, the observation strategy was optimized and the imaging blind zone was reduced, serving as the basis for the theoretical viewpoint.

[0012] Preferably, S200 includes: S210: The loading robot arm uses an end-effector depth camera to visually locate the workpiece to be inspected, obtain the workpiece gripping posture, grip the workpiece through the clamping execution unit, and transfer the workpiece to the imaging stage area to complete the placement according to the predetermined posture. S220: The 3D camera above the imaging stage scans the frontal pose of the workpiece to obtain its real-time point cloud. This point cloud is then denoised, filtered, and feature-enhanced to create a lightweight point cloud model. This model is then registered with the CAD model to obtain the pose of the CAD model in the 3D camera coordinate system. ; S230: External parameters of the 3D camera relative to the workstation coordinate system obtained from the initial calibration. Calculate the real-time pose of the CAD model of the workpiece in the workstation coordinate system when the workpiece is in a frontal orientation. : S240: Based on this real-time pose, the set of theoretical frontal viewpoints is... Online calibration is performed to determine the target camera pose in the actual workstation coordinate system. : in, The first in the set of positive theoretical viewpoints One perspective; S250: Hand-eye calibration relationship combined with the 2D vision module at the end of the robotic arm. The target camera pose is converted into the target pose in the coordinate system of the end effector of the robotic arm. : , S260: By utilizing the installation extrinsic parameters of the robot arm base relative to the workstation coordinate system, the target pose in the robot arm end-effector coordinate system is converted into the execution target in the robot arm base coordinate system. : , in = ; S270: The system performs collision detection and reachability verification for each viewpoint and generates a frontal viewpoint execution queue. The frontal viewpoint execution queue includes at least the viewpoint number, target pose, reachability flag, execution status, and retake flag, which are used as inputs for subsequent path planning and frontal imaging execution.

[0013] Preferably, S300 includes: S310: The central control and data processing module combines the kinematic constraints of the robotic arm, the environmental state, and the viewpoint sequence to plan the viewpoint execution path and generate an execution trajectory that meets the collision-free requirements. S320: The detection robot arm arrives at each target viewpoint sequentially along the planned trajectory, and uses the end-effector 2D vision module to acquire frontal images of the workpiece. After each viewpoint image acquisition is completed, the system updates the execution status of the corresponding viewpoint and stores the image data in association with the viewpoint pose information. S330: During the imaging process, the system simultaneously performs quality assessment on the acquired images. The quality assessment includes at least image sharpness, exposure status, integrity of key areas, coverage, reflectivity, or detection result feedback. S340: When it is determined that the image of a certain viewpoint does not meet the requirements, the viewpoint is marked as to be reshot, and local path replanning or local reshot is triggered only for that viewpoint; S350: When new obstacles or changes in local spatial state occur in the environment, perform local obstacle avoidance replanning for incomplete viewpoints based on the updated environment model.

[0014] Preferably, in S400, changing the placement posture of the workpiece at the imaging station through the collaborative operation of dual robotic arms or the posture adjustment mechanism specifically includes: after the front imaging is completed, the system autonomously initiates a physical exploration of the unimaged area, and the central control and data processing module schedules the loading robotic arm and the detection robotic arm to work together to complete the workpiece flipping operation, or changes the posture of the workpiece on the imaging stage through the posture adjustment mechanism to expose the reverse side, bottom side or obstructed area that is difficult to image in the original upright position; wherein, the flipping angle is 180 degrees or any preset angle.

[0015] Preferably, after the workpiece is flipped and repositioned, the real-time point cloud of the workpiece is acquired again using a 3D camera and registered with the CAD model to obtain the actual pose of the workpiece during the reverse side inspection stage. ; Based on the actual pose of the reverse side, the set of theoretical viewpoints of the reverse side is... Corrected to target imaging pose in the work station coordinate system And further convert it into the target pose in the coordinate system of the end effector of the robotic arm. ,in, The first in the set of opposing theoretical viewpoints One perspective; After performing collision detection and accessibility verification on each reverse viewpoint, the system generates a reverse viewpoint execution queue and performs imaging and closed-loop feedback in the same manner as the S300 until the reverse detection area reaches the preset coverage requirement.

[0016] Preferably, S500 includes: when it is determined that all surface imaging tasks of the current workpiece have been completed, the inspection robot arm moves the inspected workpiece to the unloading area, and at the same time, or under the condition that no spatial interference occurs, the loading robot arm grabs the next workpiece to be inspected to the imaging stage. The central control and data processing module uniformly schedules the action sequence, path interlock and station occupancy status of the two robot arms; for the same type of workpiece, the pre-generated observation strategy is directly called and enters the online calibration and imaging stage; for different types of workpieces, the observation strategy of the corresponding model is called or the viewpoint planning process is re-executed, and the current workpiece status, viewpoint status and environmental status are updated.

[0017] Preferably, S600 includes: S610: Based on the intrinsic parameters of the 2D vision module, the hand-eye calibration relationship, and the actual workpiece pose obtained through online registration, the two-dimensional defect results from each viewpoint are mapped to the workstation coordinate system or the workpiece model coordinate system. S620: For duplicate defect results from different perspectives, fuse and deduplicate based on spatial location and detection consistency; among which, defect identification adopts target detection, segmentation, anomaly detection or traditional image processing methods, and fusion method adopts spatial proximity, geometric consistency, projection consistency, confidence weighting or cluster analysis methods; S630: The final output is the inspection result corresponding to the unique identifier of the workpiece. The result includes at least the defect category, defect confidence level and three-dimensional location of the defect, and also outputs the corresponding evidence image and visualization annotation result.

[0018] Preferably, the viewpoint planning of S100 is achieved by using a method based on digital twin model for view frustum coverage analysis, regular grid division, surface normal optimization, heuristic search, or learning strategy; the path planning during the imaging process uses sampling planning, search planning, or optimization planning methods, and the trajectory is smoothed, time-parameterized, or locally corrected.

[0019] An embodied autonomous imaging system for defect detection of complex, irregularly shaped industrial components includes: The central control and data processing module is used to execute an embodied autonomous imaging method for defect detection of complex and irregularly shaped industrial components; The loading robotic arm is equipped with a depth camera and a fixture changing module at its end. The depth camera is used to identify the gripping posture of the workpiece to be inspected, and the loading robotic arm is used to grip and transfer the workpiece to be inspected, and work together to flip the workpiece. The inspection robot arm is equipped with a 2D vision module and a fixture changing module at its end. The 2D vision module includes an industrial camera and a supplementary light source. The inspection robot arm is used to acquire multi-view images of the workpiece surface according to the planned viewpoint and complete the unloading operation. The inspection station includes an imaging station, a loading fixture, an unloading fixture, and a 3D camera mounted on the imaging station. The imaging station is used to carry the workpiece, and the 3D camera is used to acquire the three-dimensional point cloud information of the workpiece at the imaging station to determine the actual position and orientation of the workpiece. The central control and data processing module is connected to the feeding robot arm, the inspection robot arm, the 3D camera and the 2D vision module, respectively, and is used to complete viewpoint planning, pose correction, trajectory planning, image quality assessment, defect recognition, multi-view fusion and system scheduling control.

[0020] This invention combines multi-robotic arm collaboration with model-driven viewpoint planning to achieve high-coverage autonomous imaging of the entire surface of complex, irregularly shaped industrial components, effectively eliminating imaging blind spots and significantly improving the completeness and reliability of defect detection. Furthermore, this invention evaluates image quality in real time during imaging. When imaging anomalies or local environmental changes are detected, it can trigger local reshoots or replanning without global downtime, thus significantly improving imaging success rate and consistency while maintaining inspection cycle time. In addition, this invention supports closed-loop adaptive optimization of imaging parameters, dynamically adjusting the shooting angle, distance, and sensor parameters according to the material, surface condition, and ambient lighting conditions of different workpieces, avoiding local overexposure, underexposure, or defocusing problems caused by fixed strategies in traditional methods. Ultimately, this invention achieves full automation from viewpoint generation and image acquisition to quality verification and closed-loop adjustment, not only reducing the need for manual intervention but also ensuring the traceability of the inspection process and the reproducibility of results, providing reliable technical support for online full inspection of complex, irregularly shaped components in high-cycle manufacturing scenarios such as automotive, aerospace, and rail transportation. Attached Figure Description

[0021] Figure 1 This is a flowchart of an embodied autonomous imaging method for defect detection of complex irregular-shaped industrial components according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a two-dimensional imaging simulation of a workpiece from a candidate viewpoint in one embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the analysis of observable and unobservable areas on the surface of a workpiece in one embodiment of the present invention; Figure 4 This is a schematic diagram of the workpiece point cloud before registration with the CAD model in one embodiment of the present invention; Figure 5 This is a schematic diagram of the workpiece point cloud and CAD model after registration in one embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0023] I. System Composition The autonomous imaging system of this invention includes a detection platform, a central control and data processing module, a loading robotic arm, a detection robotic arm, and a 3D camera. The central control and data processing module is located on the manual work platform and is communicatively connected to the loading robotic arm, the detection robotic arm, the 3D camera, and the 2D vision module. It is used to complete viewpoint planning, pose correction, trajectory planning, image quality assessment, defect identification, multi-view fusion, and system scheduling and control.

[0024] The inspection station includes a loading fixture, an unloading fixture, an imaging station, and a 3D camera. The imaging station holds the workpiece to be inspected and serves as the imaging station; the 3D camera, positioned above the imaging station, acquires the 3D point cloud information of the workpiece at the imaging station to determine its actual position and orientation. The loading fixture is used for temporary storage and positioning of the workpiece to be inspected; the unloading fixture is used for temporary storage and transfer of inspected workpieces. The loading robotic arm and the inspection robotic arm are located on one side of the inspection station, their spatial layout meeting the collaborative operation requirements of workpiece gripping, handover, flipping, imaging, and unloading, while reducing the risk of interference between the two robotic arms.

[0025] The loading robotic arm is used to perform the gripping, loading, and collaborative flipping operations of the workpieces to be inspected. Its end effector can be equipped with a depth camera and a fixture changing module. The depth camera is used for gripping pose recognition of the workpieces, and the fixture changing module enables rapid replacement of different end effectors. The inspection robotic arm is used to perform multi-view image acquisition, collaborative flipping, and unloading operations. Its end effector is equipped with a 2D vision module and a fixture changing module. The 2D vision module includes an industrial camera and a supplementary lighting source, used to acquire workpiece surface images at the planned viewpoint.

[0026] II. Coordinate System and Calibration Relationship For ease of description, a station coordinate system {W} is established on the inspection table; base coordinate systems {B1} and {B2} are established on the loading robot arm and inspection robot arm respectively; tool coordinate systems {E1} and {E2} are established at the tool center points at the ends of the two robots respectively; a coordinate system {Cd} is established at the depth camera at the end of the loading robot arm; a coordinate system {C2} is established at the 2D vision module at the end of the inspection robot arm; a coordinate system {C3} is established at the 3D camera above the imaging table; a model coordinate system {M} is established on the workpiece CAD model; and the actual pose of the workpiece on the station is denoted as {O}. Rigid body transformations between any two coordinate systems are represented by homogeneous transformation matrices, where... This represents the pose transformation from coordinate system {B} to coordinate system {A}.

[0027] During system initialization, intrinsic parameter calibration, hand-eye calibration, and installation extrinsic parameter calibration are completed. The calibration results include at least: the hand-eye transformation relationship between the loading robot and its end-effector depth camera; the hand-eye transformation relationship between the inspection robot and its end-effector 2D vision module; the extrinsic parameter relationship of the 3D camera relative to the workstation coordinate system; and the installation extrinsic parameter relationship of the two robot bases relative to the workstation coordinate system. Based on these calibration relationships, the mapping and fusion of point clouds, image results, CAD models, and defect location results can be achieved in a unified coordinate system.

[0028] III. Specific Implementation of an Embodied Autonomous Imaging Method for Defect Detection of Complex Irregularly Shaped Industrial Components (1) S100: Based on the digital twin model of the workpiece to be inspected, a two-stage viewpoint planning is carried out to generate optimized observation strategies for the front and back of the workpiece, respectively, as the theoretical viewpoint basis.

[0029] Specifically, S100 includes: S110: The central control and data processing module establishes a digital twin model based on the CAD model of the workpiece to be inspected, and preprocesses the model according to the geometric features of the workpiece surface to extract key detection areas such as edges and openings, forming a lightweight geometric expression for subsequent viewpoint planning. S120: The viewpoint planning is divided into a front inspection stage and a back inspection stage. The set of front inspection areas is determined based on the workpiece's upright position on the imaging stage. The set of areas to be inspected on the reverse side is determined based on the target posture after the workpiece is flipped over. For the two stages mentioned above, the system generates corresponding candidate observation viewpoints respectively.

[0030] S130: The surface to be inspected is divided into several imaging sub-regions. Candidate viewpoints are generated based on the effective imaging coverage of the workpiece surface by the 2D vision module under different poses. The effective imaging coverage is determined by the camera's field of view, focal length, working distance range, depth of field range, incident angle limit, and resolution requirements. It characterizes the spatial range in which the camera can effectively observe the surface to be inspected under the corresponding pose. For each imaging sub-region, candidate coverage is constructed using a frustum coverage analysis method, taking into account the local surface orientation and imaging constraints of that sub-region. The candidate pose of the 2D vision module relative to the CAD model is then solved in reverse. This forms a set of positive candidate viewpoints. and the set of opposing candidate viewpoints ; S140: After candidate viewpoints are generated, visibility analysis, collision constraint analysis, safety distance analysis, and imaging quality analysis are performed on them to screen out effective observation viewpoints that meet coverage requirements, imaging requirements, and robotic arm execution requirements, thereby obtaining front-side optimized observation strategies. and the opposite optimization observation strategy .like Figure 2 As shown, simulation analysis is performed on the workpiece imaging effect under candidate viewpoints to evaluate the coverage of the target area and the degree of imaging adaptation under the corresponding viewpoints. Figure 3 As shown, the system also performs observability analysis on the workpiece surface based on a set of candidate viewpoints, where green areas represent observable areas and red areas represent unobservable areas. By comprehensively analyzing the imaging results of different candidate viewpoints and the distribution of observable areas on the workpiece surface, the observation strategy can be further optimized, the imaging blind spot can be reduced, and the optimized observation strategy can serve as the theoretical viewpoint basis for subsequent online pose correction and imaging execution.

[0031] Furthermore, the S100's viewpoint planning is achieved using methods such as view frustum coverage analysis based on a digital twin model, regular mesh generation, surface normal optimization, heuristic search, or learning-based strategies; the path planning during the imaging process employs sampling planning, search planning, or optimization planning methods, and the trajectory is smoothed, time-parameterized, or locally corrected.

[0032] (2) S200: After placing the workpiece to be inspected on the imaging stage, the actual pose of the workpiece is obtained through the 3D camera, and the theoretical viewpoint in the optimized observation strategy is corrected online using the actual pose to generate an executable viewpoint queue.

[0033] S200 includes: S210: Loading and gripping device: The loading robot arm uses an end-effector depth camera to visually locate the workpiece to be inspected, obtain the workpiece gripping posture, grip the workpiece through the clamping execution unit, and transfer the workpiece to the imaging stage area of ​​the inspection table, and complete the placement according to the predetermined posture, thereby establishing the initial workpiece posture required for frontal inspection. S220: Frontal Online Perception Calibration and Viewpoint Queue Generation: A 3D camera above the imaging stage scans the frontal pose of the workpiece, acquiring a real-time point cloud of the workpiece in the real environment. This point cloud is then denoised, filtered, and feature-enhanced to form a standardized, lightweight point cloud model. This model is then registered with the CAD model to obtain the pose of the CAD model in the 3D camera coordinate system. And based on this, the theoretical observation viewpoint is corrected online. Figure 4 This shows the relative state between the workpiece point cloud and the CAD model before registration. Figure 5 The correspondence after registration is displayed.

[0034] S230: External parameters of the 3D camera relative to the workstation coordinate system obtained from initial calibration. Calculate the real-time pose of the CAD model of the workpiece in the workstation coordinate system when the workpiece is in a frontal orientation. : S240: Based on this real-time pose, the set of theoretical frontal viewpoints is... Online calibration is performed to determine the target camera pose in the actual workstation coordinate system. : in, The first in the set of positive theoretical viewpoints One perspective; S250: Hand-eye calibration relationship combined with the 2D vision module at the end of the robotic arm. The target camera pose is converted into the target pose in the coordinate system of the end effector of the robotic arm. : , S260: By utilizing the installation extrinsic parameters of the robot arm base relative to the workstation coordinate system, the target pose in the robot arm end-effector coordinate system is converted into the execution target in the robot arm base coordinate system. : , in = ; S270: The system performs collision detection and reachability verification for each viewpoint and generates a frontal viewpoint execution queue. The frontal viewpoint execution queue includes at least the viewpoint number, target pose, reachability flag, execution status, and retake flag, which are used as inputs for subsequent path planning and frontal imaging execution.

[0035] (3) S300: Control the detection robot arm to perform imaging tasks in sequence according to the viewpoint queue, and perform real-time quality assessment on the acquired images. When the image does not meet the requirements, trigger local replanning or local re-shooting for abnormal viewpoints to achieve closed-loop adjustment of the imaging process.

[0036] The S300 includes: S310: The central control and data processing module combines the kinematic constraints of the robotic arm, the environmental state, and the viewpoint sequence to plan the viewpoint execution path and generate an execution trajectory that meets the collision-free requirements. S320: The detection robot arm arrives at each target viewpoint sequentially along the planned trajectory, and uses the end-effector 2D vision module to acquire frontal images of the workpiece. After each viewpoint image acquisition is completed, the system updates the execution status of the corresponding viewpoint and stores the image data in association with the viewpoint pose information. S330: During the imaging process, the system simultaneously performs quality assessment on the acquired images. The quality assessment includes at least image sharpness, exposure status, integrity of key areas, coverage, reflectivity, or detection result feedback. S340: When it is determined that the image corresponding to a certain viewpoint does not meet the detection requirements, the viewpoint is marked as to be reshot and local reshot or local path replanning is triggered without affecting the execution results of the completed viewpoints; thereby realizing closed-loop correction of the frontal imaging process, improving imaging stability and the consistency of detection cycle. S350: When new obstacles or changes in local spatial state occur in the environment, local obstacle avoidance replanning is performed on incomplete viewpoints based on the updated environment model to improve the system's adaptability in dynamic environments.

[0037] (4) S400: By using the coordinated operation of two robotic arms or the attitude adjustment mechanism, the placement posture of the workpiece at the imaging station is changed to expose the unimaged area under the original posture. The online correction and viewpoint queue generation in S200 and the imaging and closed-loop feedback in S300 are repeated to complete the imaging of the front and back of the workpiece.

[0038] In the S400, changing the placement posture of the workpiece at the imaging station through the collaborative operation of dual robotic arms or the posture adjustment mechanism specifically includes: after the front viewpoint queue is completed, i.e., after the front imaging is completed, the system autonomously initiates a physical exploration of the unimaged area. The central control and data processing module schedules the loading robotic arm and the detection robotic arm to work together to complete the workpiece flipping operation, or changes the posture of the workpiece on the imaging stage through the posture adjustment mechanism to expose the reverse side, bottom side, or obstructed area that is difficult to image in the original upright position; wherein, the flipping angle is 180 degrees or any preset angle.

[0039] Specifically, the attitude adjustment mechanism can be a flipping mechanism, a rotary table, a positioner, or a flipping fixture at the imaging stage to achieve workpiece attitude switching; as long as the workpiece attitude can be changed and cooperated with subsequent reverse imaging or side imaging, it is an equivalent alternative of the present invention.

[0040] After the workpiece is flipped and repositioned, real-time point cloud data of the workpiece is acquired again using a 3D camera. The central control and data processing module performs online registration on the flipped point cloud data to obtain the actual pose of the workpiece under the reverse detection posture. ; Based on the actual pose of the reverse side, the set of theoretical viewpoints of the reverse side output by S100 is... Online correction is used to determine the target imaging pose in the workstation coordinate system. And further convert it into the target pose in the coordinate system of the end effector of the robotic arm. ,in, The first in the set of opposing theoretical viewpoints One perspective; After performing collision detection and reachability verification on each opposing viewpoint, the system generates an execution queue for the opposing viewpoints. The robotic arm then sequentially performs reverse imaging according to the reverse viewpoint execution queue, and performs image quality assessment, re-enhancing viewpoint marking, local re-enhancing, and local path replanning in the same manner as S300, until the system determines that the reverse detection area has reached the preset coverage requirements. This completes closed-loop imaging of the entire workpiece surface.

[0041] (5) S500: After completing the front and back imaging, the inspected workpiece is transferred to the unloading area. At the same time, or when there is no spatial interference, the next workpiece to be inspected is grabbed and placed on the imaging stage. The current workpiece status and environmental status are updated. The pre-generated observation strategy is called or the viewpoint planning process is re-executed according to the model of the next workpiece, forming a continuous cycle of autonomous imaging operation.

[0042] In a preferred embodiment, S500 includes: when it is determined that all surface imaging tasks of the current workpiece have been completed, the inspection robot arm moves the inspected workpiece to the unloading area, and at the same time, or under the condition that no spatial interference occurs, the loading robot arm grabs the next workpiece to be inspected to the imaging stage.

[0043] The aforementioned collaborative action of the two robotic arms is uniformly scheduled by the central control and data processing module to coordinate the timing of their actions, path interlocks, and workstation occupancy status within the shared space, thereby avoiding mechanical interference and improving the cycle time of continuous operation.

[0044] After a new workpiece enters the imaging stage, the system updates the current workpiece status, viewpoint status, and environmental status, and then proceeds to the next round of inspection. For workpieces of the same model, the system can directly call the pre-generated observation strategy and enter the online calibration and imaging stage. For workpieces of different models, the system calls the corresponding model's observation strategy or re-executes the viewpoint planning process, and updates the current workpiece status, viewpoint status, and environmental status. This forms a continuous, cyclical autonomous imaging operation process.

[0045] (6) S600: Perform defect identification on the acquired multi-view images, and map the identified two-dimensional defect results to a unified three-dimensional coordinate system to output the three-dimensional spatial location information of the defect.

[0046] Specifically, after acquiring multi-view images from both the front and back sides, the system performs defect identification on each viewpoint image, outputting defect category, defect location, and confidence level information. Defect identification can employ deep learning detection models, segmentation models, or other image processing methods capable of surface defect recognition.

[0047] In a preferred embodiment, S600 includes: S610: Based on the intrinsic parameters of the 2D vision module, the hand-eye calibration relationship, and the actual workpiece pose obtained through online registration, the two-dimensional defect results under each viewpoint are uniformly mapped to the workstation coordinate system or the workpiece model coordinate system to achieve three-dimensional positioning of defects. S620: For duplicate defect results from different perspectives, fuse and deduplicate them based on spatial location and detection consistency to form a unified defect list; among them, defect identification adopts target detection, segmentation, anomaly detection or traditional image processing methods, and fusion method adopts spatial proximity, geometric consistency, projection consistency, confidence weighting or cluster analysis methods. S630: The final output is the inspection result corresponding to the unique identifier of the workpiece. The result includes at least the defect category, defect confidence level and three-dimensional location of the defect. At the same time, the corresponding evidence image and visual annotation result are output to support quality traceability and subsequent process processing.

[0048] In one embodiment, an embodied autonomous imaging system for defect detection of complex, irregularly shaped industrial components is also provided, comprising: The central control and data processing module is used to execute an embodied autonomous imaging method for defect detection of complex and irregularly shaped industrial components; The loading robotic arm is equipped with a depth camera and a fixture changing module at its end. The depth camera is used to identify the gripping posture of the workpiece to be inspected, and the loading robotic arm is used to grip and transfer the workpiece to be inspected, and work together to flip the workpiece. The inspection robot arm is equipped with a 2D vision module and a fixture changing module at its end. The 2D vision module includes an industrial camera and a supplementary light source. The inspection robot arm is used to acquire multi-view images of the workpiece surface according to the planned viewpoint and complete the unloading operation. The inspection station includes an imaging station, a loading fixture, an unloading fixture, and a 3D camera mounted on the imaging station. The imaging station is used to carry the workpiece, and the 3D camera is used to acquire the three-dimensional point cloud information of the workpiece at the imaging station to determine the actual position and orientation of the workpiece. The central control and data processing module is connected to the feeding robot arm, the inspection robot arm, the 3D camera and the 2D vision module, respectively, and is used to complete viewpoint planning, pose correction, trajectory planning, image quality assessment, defect recognition, multi-view fusion and system scheduling control.

[0049] Specific limitations regarding the embodied autonomous imaging system for detecting defects in complex, irregularly shaped industrial components can be found in the above-described limitations regarding the embodied autonomous imaging method for detecting defects in complex, irregularly shaped industrial components, and will not be repeated here. The modules in the aforementioned embodied autonomous imaging system for detecting defects in complex, irregularly shaped industrial components can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0050] The aforementioned embodied autonomous imaging method and system for defect detection of complex, irregularly shaped industrial components has the following advantages over existing technologies: (1) It can improve the imaging coverage of complex irregular industrial components: By combining front imaging with reverse imaging after dual robotic arms flipping the surface, the bottom surface, reverse surface and obscured areas that are difficult to observe directly in the original upright state can be exposed and imaged, thereby reducing the imaging blind zone of complex irregular components in defect detection and improving the full surface detection coverage.

[0051] (2) It can improve the consistency between offline planning results and actual execution status: The present invention first performs digital twin viewpoint planning based on the workpiece CAD model, then obtains the real-time pose of the workpiece in the real work station through a 3D camera, and performs online correction of the theoretical viewpoint, thereby mapping the offline generated observation strategy to the execution pose that matches the actual workpiece posture, reducing the impact of incoming material deviation, placement deviation and posture change on imaging quality and execution accuracy.

[0052] (3) It can maintain high task stability in complex industrial sites: During the imaging process, the present invention performs quality assessment on the acquired images, and when it finds that the image is blurry, the exposure is abnormal, the area is missing or the key area is not covered enough, it only triggers local reshoot or local path replanning for the abnormal viewpoint, without repeating the entire viewpoint execution process. Therefore, it is beneficial to improve the closed-loop correction capability of the imaging task, maintain the stability of the detection cycle and improve the system operating efficiency.

[0053] (4) Precise 3D defect localization: This invention combines the 2D defect recognition results in multi-view images with camera calibration parameters, hand-eye calibration relationship and workpiece real-time pose and maps them to the workstation coordinate system or workpiece model coordinate system to achieve cross-view defect fusion deduplication and precise 3D spatial localization.

[0054] (5) Applicable to continuous inspection operations under flexible manufacturing conditions: The present invention uses a central control and data processing module to uniformly schedule the two robotic arms, and combines a pre-generated observation strategy and an online sensing calibration mechanism to adapt to the imaging inspection requirements of different types of irregularly shaped components, making it easy to achieve continuous autonomous operation in multi-variety, small-batch and flexible production line scenarios.

[0055] The foregoing provides a detailed description of the embodied autonomous imaging method and system for defect detection of complex, irregularly shaped industrial components provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention, and the descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. An embodied autonomous imaging method for defect detection of complex, irregularly shaped industrial components, characterized in that, Includes the following steps: S100: Based on the digital twin model of the workpiece to be inspected, a two-stage viewpoint planning is performed to generate optimized observation strategies for the front and back of the workpiece, respectively, as the theoretical viewpoint basis; S200: After placing the workpiece to be inspected on the imaging stage, the actual pose of the workpiece is obtained through a 3D camera, and the theoretical viewpoint in the optimized observation strategy is corrected online using the actual pose to generate an executable viewpoint queue. S300: Controls the robotic arm to perform imaging tasks sequentially according to the viewpoint queue, and performs real-time quality assessment on the acquired images. When the images do not meet the requirements, it triggers local replanning or local re-shooting for abnormal viewpoints to achieve closed-loop adjustment of the imaging process. S400: By using the coordinated operation of two robotic arms or the attitude adjustment mechanism, the placement posture of the workpiece in the imaging station is changed to expose the unimaged area under the original posture. The online correction and viewpoint queue generation in S200 and the imaging and closed-loop feedback in S300 are repeated to complete the imaging of the front and back of the workpiece. S500: After completing the imaging of the front and back sides, the inspected workpiece is transferred to the unloading area. At the same time, or when there is no spatial interference, the next workpiece to be inspected is grabbed and placed on the imaging stage. The current workpiece status and environmental status are updated. Based on the model of the next workpiece, the pre-generated observation strategy is called or the viewpoint planning process is re-executed to form a continuous cycle of autonomous imaging operation. S6 00: The acquired multi-view images are used for defect identification, and the identified two-dimensional defect results are mapped to a unified three-dimensional coordinate system to output the three-dimensional spatial location information of the defects.

2. The method according to claim 1, characterized in that, S100 includes: S110: The central control and data processing module establishes a digital twin model based on the CAD model of the workpiece to be inspected, and preprocesses the model according to the geometric features of the workpiece surface to extract key detection areas such as edges and openings, forming a lightweight geometric expression for subsequent viewpoint planning. S120: The viewpoint planning is divided into a front inspection stage and a back inspection stage. The set of front inspection areas is determined based on the workpiece's upright position on the imaging stage. The set of areas to be inspected on the reverse side is determined based on the target posture after the workpiece is flipped over. ; S130: The surface to be inspected is divided into several imaging sub-regions. The effective imaging coverage is determined based on the field of view, focal length, working distance range, depth of field range, incident angle limit, and resolution requirements of the 2D vision module. For each imaging sub-region, candidate coverage is constructed using a frustum coverage analysis method, taking into account the local surface orientation and imaging constraints of that sub-region. The candidate pose of the 2D vision module relative to the CAD model is then solved in reverse. This forms a set of positive candidate viewpoints. and the set of opposing candidate viewpoints ; S140: Visibility analysis, collision constraint analysis, safety distance analysis, and imaging quality analysis are performed on candidate viewpoints to screen out effective observation viewpoints that meet coverage requirements, imaging requirements, and robotic arm execution requirements, and front-side optimized observation strategies are obtained accordingly. and the opposite optimization observation strategy Furthermore, simulation analysis was conducted on the imaging effect of the workpiece under the candidate viewpoints to evaluate the coverage and imaging adaptability. Based on the candidate viewpoint set, the observability of the workpiece surface was analyzed. Through comprehensive analysis, the observation strategy was optimized and the imaging blind zone was reduced, serving as the basis for the theoretical viewpoint.

3. The method according to claim 2, characterized in that, S200 includes: S210: The loading robot arm uses an end-effector depth camera to visually locate the workpiece to be inspected, obtain the workpiece gripping posture, grip the workpiece through the clamping execution unit, and transfer the workpiece to the imaging stage area to complete the placement according to the predetermined posture. S220: The 3D camera above the imaging stage scans the frontal pose of the workpiece to obtain its real-time point cloud. This point cloud is then denoised, filtered, and feature-enhanced to create a lightweight point cloud model. This model is then registered with the CAD model to obtain the pose of the CAD model in the 3D camera coordinate system. ; S230: External parameters of the 3D camera relative to the workstation coordinate system obtained from the initial calibration. Calculate the real-time pose of the CAD model of the workpiece in the workstation coordinate system when the workpiece is in a frontal orientation. : S240: Based on real-time pose, the set of frontal theoretical viewpoints Online calibration is performed to determine the target camera pose in the actual workstation coordinate system. : in, The first in the set of positive theoretical viewpoints One perspective; S250: Hand-eye calibration relationship combined with the 2D vision module at the end of the robotic arm. The target camera pose is converted into the target pose in the coordinate system of the end effector of the robotic arm. : , S260: By utilizing the installation extrinsic parameters of the robot arm base relative to the workstation coordinate system, the target pose in the robot arm end-effector coordinate system is converted into the execution target in the robot arm base coordinate system. : , in = ; S270: The system performs collision detection and reachability verification for each viewpoint and generates a frontal viewpoint execution queue. The frontal viewpoint execution queue includes at least the viewpoint number, target pose, reachability flag, execution status, and retake flag, which are used as inputs for subsequent path planning and frontal imaging execution.

4. The method according to claim 3, characterized in that, The S300 includes: S310: The central control and data processing module combines the kinematic constraints of the robotic arm, the environmental state, and the viewpoint sequence to plan the viewpoint execution path and generate an execution trajectory that meets the collision-free requirements. S320: The detection robot arm arrives at each target viewpoint sequentially along the planned trajectory, and uses the end-effector 2D vision module to acquire frontal images of the workpiece. After each viewpoint image acquisition is completed, the system updates the execution status of the corresponding viewpoint and stores the image data and viewpoint pose information together. S330: During the imaging process, the system simultaneously performs quality assessment on the acquired images. The quality assessment includes at least image sharpness, exposure status, integrity of key areas, coverage, reflectivity, or detection result feedback. S340: When it is determined that the image of a certain viewpoint does not meet the requirements, the viewpoint is marked as to be reshot, and local path replanning or local reshot is triggered only for that viewpoint; S350: When new obstacles or changes in local spatial state occur in the environment, perform local obstacle avoidance replanning for incomplete viewpoints based on the updated environment model.

5. The method according to claim 4, characterized in that, In the S400, changing the placement posture of the workpiece at the imaging station through the coordinated operation of dual robotic arms or the posture adjustment mechanism specifically includes: after the front imaging is completed, the system autonomously initiates a physical exploration of the unimaged area. The central control and data processing module schedules the loading robotic arm and the detection robotic arm to work together to complete the workpiece flipping operation, or changes the posture of the workpiece on the imaging stage through the posture adjustment mechanism to expose the reverse side, bottom side, or obstructed area that is difficult to image in the original upright position; wherein, the flipping angle is 180 degrees or any preset angle.

6. The method according to claim 5, characterized in that, After the workpiece is flipped and repositioned, a real-time point cloud of the workpiece is acquired again using a 3D camera and registered with the CAD model to obtain the actual pose of the workpiece during the reverse side inspection stage. ; Based on the actual pose of the reverse side, the set of theoretical viewpoints of the reverse side is... Corrected to target imaging pose in the work station coordinate system And further convert it into the target pose in the coordinate system of the end effector of the robotic arm. ,in, The first in the set of opposing theoretical viewpoints One perspective; After performing collision detection and accessibility verification on each reverse viewpoint, the system generates a reverse viewpoint execution queue and performs imaging and closed-loop feedback in the same manner as the S300 until the reverse detection area reaches the preset coverage requirement.

7. The method according to claim 6, characterized in that, The S500 includes: after all surface imaging tasks of the current workpiece are completed, the inspection robot arm moves the inspected workpiece to the unloading area. At the same time, or under the condition that no spatial interference occurs, the loading robot arm grabs the next workpiece to be inspected and places it on the imaging stage. The central control and data processing module uniformly schedules the action sequence, path interlock, and station occupancy status of the two robot arms. For the same type of workpiece, the pre-generated observation strategy is directly called and the online calibration and imaging stage is entered. For different types of workpieces, the observation strategy of the corresponding model is called or the viewpoint planning process is re-executed, and the current workpiece status, viewpoint status, and environmental status are updated.

8. The method according to claim 7, characterized in that, The S600 includes: S610: Based on the intrinsic parameters of the 2D vision module, the hand-eye calibration relationship, and the actual workpiece pose obtained through online registration, the two-dimensional defect results from each viewpoint are mapped to the workstation coordinate system or the workpiece model coordinate system. S620: For duplicate defect results from different perspectives, fuse and deduplicate based on spatial location and detection consistency; among which, defect identification adopts target detection, segmentation, anomaly detection or traditional image processing methods, and fusion method adopts spatial proximity, geometric consistency, projection consistency, confidence weighting or cluster analysis methods; S630: The final output is the inspection result corresponding to the unique identifier of the workpiece. The result includes at least the defect category, defect confidence level and three-dimensional location of the defect, and also outputs the corresponding evidence image and visualization annotation result.

9. The method according to claim 8, characterized in that, The S100's viewpoint planning is achieved through methods such as view cone coverage analysis based on a digital twin model, regular mesh generation, surface normal optimization, heuristic search, or learning-based strategies. Path planning during the imaging process employs sampling planning, search planning, or optimization planning methods, and the trajectory is smoothed, time-parameterized, or locally corrected.

10. A embodied autonomous imaging system for defect detection of complex, irregularly shaped industrial components, characterized in that, include: A central control and data processing module, used to execute the method according to any one of claims 1 to 9; The loading robotic arm is equipped with a depth camera and a fixture changing module at its end. The depth camera is used to identify the gripping posture of the workpiece to be inspected, and the loading robotic arm is used to grip and transfer the workpiece to be inspected, and work together to flip the workpiece. The inspection robot arm is equipped with a 2D vision module and a fixture changing module at its end. The 2D vision module includes an industrial camera and a supplementary light source. The inspection robot arm is used to acquire multi-view images of the workpiece surface according to the planned viewpoint and complete the unloading operation. The inspection station includes an imaging station, a loading fixture, an unloading fixture, and a 3D camera mounted on the imaging station. The imaging station is used to carry the workpiece, and the 3D camera is used to acquire the three-dimensional point cloud information of the workpiece at the imaging station to determine the actual position and orientation of the workpiece. The central control and data processing module is connected to the feeding robot arm, the inspection robot arm, the 3D camera and the 2D vision module, respectively, and is used to complete viewpoint planning, pose correction, trajectory planning, image quality assessment, defect recognition, multi-view fusion and system scheduling control.

Images