A Real-Time Detection System and Method for Part Rework Status Based on Augmented Reality Projection

Abstract

This application relates to the technical field of automated manufacturing, and in particular to a real-time detection system and method for the rework status of parts based on augmented reality projection. The method includes: performing a three-dimensional real-time scan of the surface of the part to be inspected to generate high-density point cloud data; transmitting the point cloud data to an edge computing unit for preprocessing; acquiring a standard CAD model of the part to be inspected and registering it with the preprocessed point cloud data using an improved ICP algorithm; after registration, comparing and analyzing the standard CAD model and the point cloud data to identify defect areas and outputting a defect report; projecting the defect areas onto the actual part surface using color coding or annotation based on projection technology; acquiring repair progress information in real time, generating a repair progress based on the defect report, and outputting guidance and prompts. This application has the advantages of improving the accuracy of anomaly identification and rework efficiency for parts to be reworked.

Description

Technical Field

[0001] This application relates to the technical field of automated production manufacturing, and in particular to a real-time detection system and method for the rework status of parts based on augmented reality projection. Background Technology

[0002] In modern manufacturing, parts rework is a crucial link in ensuring product quality and production efficiency. With increasingly fierce market competition and ever-rising consumer demands for product quality, efficient and accurate parts rework has become a significant challenge for enterprises. Traditional rework methods often rely on manual inspection and repair, which is not only time-consuming and labor-intensive but also easily affected by operator subjectivity, leading to inconsistent inspection results and unstable rework quality. These problems not only increase manufacturing costs but also affect production efficiency and customer satisfaction. Therefore, introducing advanced technologies and methods to improve the current state of parts rework has become an inevitable trend in the development of modern manufacturing.

[0003] Chinese patent application number CN202311118046.9 discloses a part rework control method, a rework control system, and a storage medium. The method includes: when an abnormal part is transported to the rework station, obtaining pre-set identification information on the part through a scanning device; obtaining abnormal information associated with the identification information from a server as the current abnormal information of the part; when completing the rework process based on the current abnormal information, uploading the rework information of the part to the server; placing the reworked part into a first rack corresponding to the part; and upon receiving a first operation instruction for transporting the first rack, controlling a robot to transport the first rack containing the part to the production station or scrap area corresponding to the part based on the rework information. This improves the efficiency of rework and ensures that reworked parts are accurately transported to their intended destinations such as production stations or scrap areas.

[0004] In the aforementioned patent, the scanning device obtains abnormal information of parts through identification information for rework. It is necessary to set identification information on each part, which increases the amount of preparation work. It is mainly applicable to parts that have been identified as abnormal and need to be reworked. It may not be able to handle unidentified abnormalities or newly emerging problems in a timely manner.

[0005] Secondly, during the rework process, rework personnel need to obtain abnormal information and then find the corresponding location on the actual part. There may be situations where the abnormal information and the actual part cannot be matched, which reduces the efficiency of rework. Summary of the Invention

[0006] To improve the accuracy of abnormal identification and efficiency of rework parts, this application provides a real-time detection system and method for the rework status of parts based on augmented reality projection.

[0007] Firstly, this application provides a method for real-time detection of the rework status of parts based on augmented reality projection, employing the following technical solution:

[0008] A method for real-time detection of part rework status based on augmented reality projection, comprising:

[0009] The surface of the part to be inspected is scanned in three dimensions in real time to generate high-density point cloud data;

[0010] The point cloud data is transmitted to the edge computing unit for preprocessing.

[0011] Obtain the standard CAD model of the part to be inspected, and register it with the preprocessed point cloud data using an improved ICP algorithm;

[0012] After registration, the standard CAD model and point cloud data are compared and analyzed to identify defect areas and output a defect report;

[0013] Based on projection technology, defective areas are projected onto the surface of the actual part in the form of color coding or marking;

[0014] It can obtain real-time information on the progress of the repair, generate a repair schedule based on the defect report, and output guidance and prompts.

[0015] In one embodiment: the three-dimensional real-time scanning adopts a scanning frequency of 10Hz-60Hz, and dynamic compensation is performed based on a motion compensation algorithm during the scanning process.

[0016] In one embodiment: during the 3D scanning process, the focus is automatically adjusted based on the shape and size of the part to be inspected.

[0017] In one embodiment: the defect report includes defect location, defect type and severity, and the guidance prompts are output in a preset order; the preset order is generated based on the defect report, and the priority of generation is severity, defect type and defect location in that order.

[0018] In one embodiment: the step of projecting the defect area onto the surface of the actual part using projection technology in the form of color coding or marking specifically includes:

[0019] Based on the intensity of light interference, inertial measurement data and machine vision features, an initial pose transformation prediction is generated by fusing them together.

[0020] Based on the pose offset error and visual optical flow compensation, the refined pose is output after the initial pose transformation prediction is corrected.

[0021] Based on the refined pose, the defect area is projected onto the actual part surface.

[0022] In one embodiment, the step of outputting a refined pose after correcting the initial pose transformation prediction based on pose offset error and visual optical flow compensation specifically includes:

[0023] Calculate the expected shadow change based on the surface normal vector, direction vector, and predicted pose transformation of the part.

[0024] By comparing the expected shadow change with the actual shadow change captured by the 3D scan, the pose offset error is obtained.

[0025] Based on the dynamic allocation of shadow compensation weights and visual optical flow weights according to the intensity of light interference, the pose offset error and visual optical flow compensation are weighted and fused, and the refined pose is output after the initial pose transformation prediction is corrected.

[0026] In one embodiment: the step of fusing light interference intensity, inertial measurement data, and machine vision features to generate an initial pose transformation prediction specifically includes:

[0027] Based on the intensity of light interference, the weights of inertial measurement data and machine visual features are dynamically allocated;

[0028] The inertial measurement data and the machine's visual features are weighted and fused to generate an initial pose transformation prediction.

[0029] In one embodiment: the light interference intensity is obtained based on real-time acquired ambient light parameters, after monitoring changes in ambient light intensity gradient and direction vector.

[0030] In one embodiment: the step of projecting the defect area onto the surface of the actual part using projection technology in the form of color coding or marking further includes:

[0031] Based on the intensity of light interference, the projection brightness and contrast are dynamically adjusted during the projection process.

[0032] Secondly, this application provides a real-time detection system for the rework status of parts based on augmented reality projection, which adopts the following technical solution:

[0033] A real-time detection system for the rework status of parts based on augmented reality projection, comprising:

[0034] The 3D scanning module is used to perform real-time 3D scanning of the surface of the part to be inspected, generating high-density point cloud data;

[0035] The data processing module is used to transmit point cloud data to the edge computing unit for preprocessing; to acquire the standard CAD model of the part to be inspected, and to register it with the preprocessed point cloud data using an improved ICP algorithm; and after registration, to compare and analyze the standard CAD model and point cloud data to identify defect areas and output a defect report.

[0036] AR projection module is used to project defect areas onto the surface of actual parts in the form of color coding or marking based on projection technology;

[0037] The user interaction module is used to obtain real-time repair progress information, generate repair progress based on defect reports, and output guidance and prompts.

[0038] In summary, this application has the following beneficial effects:

[0039] 1. Through a high-frequency 3D scanning module, high-density point cloud data is generated in real time, supporting continuous scanning in dynamic environments. All abnormalities of the parts to be repaired are automatically obtained by comparing with standard CAD models.

[0040] 2. By using AR projection equipment, defective areas are projected onto the surface of actual parts in the form of color coding or marking, providing intuitive visual feedback and effectively improving rework efficiency;

[0041] 3. Track the position and orientation changes of the parts to be repaired in real time and dynamically adjust the projected content to ensure that the markings always accurately correspond to the actual defect locations, further ensuring repair efficiency. Attached Figure Description

[0042] Figure 1 This is a framework diagram of the real-time detection system for the rework status of parts based on augmented reality projection in this embodiment;

[0043] Figure 2 This is a flowchart of the real-time detection method for the rework status of parts based on augmented reality projection in this embodiment.

[0044] In the diagram, 10 is the 3D scanning module; 20 is the data processing module; 30 is the AR projection module; and 40 is the user interaction module. Detailed Implementation

[0045] The present application will be further described in detail below with reference to the accompanying drawings.

[0046] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0047] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0049] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0050] like Figure 1 As shown, a real-time detection system for the rework status of parts based on augmented reality projection combines a 3D scanning device with an AR projection device to achieve real-time detection and visual feedback of the rework status of parts.

[0051] The aforementioned detection system includes a 3D scanning module 10, a data processing module 20, an AR projection module 30, and a user interaction module 40.

[0052] The 3D scanning module 10 uses a high-precision structured light or laser scanning device to perform real-time 3D scanning of the surface of the part to be inspected, generating high-density point cloud data.

[0053] The data processing module 20 is used to transmit point cloud data to the edge computing unit for preprocessing; to acquire the standard CAD model of the part to be inspected, and to register it with the preprocessed point cloud data using an improved ICP algorithm; and after registration, to compare and analyze the standard CAD model and the point cloud data to identify defect areas and output a defect report.

[0054] AR projection module 30 is used to project defect areas onto the surface of actual parts in the form of color coding or marking based on projection technology.

[0055] The user interaction module 40 is used to obtain repair progress information in real time, generate repair progress based on defect reports, and output guidance and prompts. The user interaction module 40 can provide an intuitive operation interface and feedback mechanism, making it easy for maintenance personnel to view defect locations and repair suggestions. At the same time, maintenance personnel can input repair progress information into the system through touch screen, gestures, or voice input.

[0056] This module also supports multilingual interfaces and voice prompts, enabling the system to update the repair progress in real time and provide corresponding feedback on the AR interface, while triggering voice prompts to guide the next step.

[0057] like Figure 2 As shown, this embodiment also provides a real-time detection method for the rework status of parts based on augmented reality projection, aiming to achieve efficient and accurate detection and guidance of the rework status of parts. The method includes the following steps:

[0058] S100 performs a three-dimensional real-time scan of the surface of the part to be inspected, generating high-density point cloud data.

[0059] In this step, the 3D real-time scanning uses a scanning frequency of 10Hz-60Hz to generate high-density point cloud data. At the same time, in order to ensure the stability and reliability of data acquisition, dynamic compensation is performed based on a motion compensation algorithm during the scanning process.

[0060] By predicting the trajectory of an object using motion compensation algorithms such as optical flow or ICP, point cloud distortion caused by displacement during the scanning process can be corrected.

[0061] For example: ;

[0062] in, The original point cloud, The motion compensation amount is estimated by IMU or visual SLAM.

[0063] It can improve scanning robustness in dynamic scenes by fusing data from multiple sensors, such as combining LiDAR and depth cameras (like Intel RealSense) and using Kalman filtering.

[0064] In addition, to accommodate parts of various shapes and sizes, especially those with curved or uneven surfaces, the scanning device can automatically focus based on the shape and size of the part during the scanning process. This function ensures that the scanning device is always focused on the surface of the part, thereby obtaining clear and accurate scanning data.

[0065] S200: Transmit point cloud data to the edge computing unit for preprocessing.

[0066] After point cloud data is transmitted to the edge computing unit, it needs to undergo preprocessing to improve data quality and provide a reliable foundation for subsequent analysis. Preprocessing mainly includes two key steps: noise reduction and smoothing.

[0067] A highly efficient filtering algorithm is employed to reduce the impact of noise, effectively filtering out various types of noise introduced during the scanning process. Simultaneously, smoothing processing eliminates high-frequency noise interference in the point cloud, resulting in smoother and more continuous point cloud data.

[0068] The entire preprocessing process is designed to achieve millisecond-level response speeds to meet the needs of real-time detection. Furthermore, to further improve processing efficiency, it supports multi-task parallel processing, meaning that denoising and smoothing can be performed simultaneously, thereby significantly reducing the time required for data preprocessing.

[0069] S300: Obtain the standard CAD model of the part to be inspected, and register it with the preprocessed point cloud data using the improved ICP algorithm.

[0070] After point cloud data preprocessing, a standard CAD model of the part to be inspected is obtained. The standard CAD model contains precise design information about the part and serves as an important reference for subsequent comparative analysis. An improved ICP algorithm is used to register the preprocessed point cloud data with the standard CAD model.

[0071] The entire registration process includes several key steps such as feature matching and iterative optimization. Feature matching identifies the corresponding feature points in the point cloud data and the standard CAD model; then, through iterative optimization, the matching relationship is continuously adjusted to make the registration result more accurate.

[0072] After registration, the S400 model and point cloud data are compared and analyzed to identify defective areas and output a defect report.

[0073] After registration, a thorough comparative analysis is performed on the standard CAD model and point cloud data. Based on the differences between the two, defect areas on the part surface are accurately identified.

[0074] During this process, visual information about the defect area is generated based on the comparison results, helping operators to intuitively understand the location, shape, size, and other characteristics of the defect, and finally forming a detailed defect report.

[0075] The defect report includes information such as the defect location, defect type, and severity.

[0076] S500, based on projection technology, projects defective areas onto the surface of the actual part in the form of color coding or marking.

[0077] Based on projection technology, the identified defect areas are clearly projected onto the surface of the actual part in the form of color codes or markings. The color codes can be designed according to factors such as the type and severity of the defect. For example, red indicates a serious defect, and yellow indicates a general defect. This allows maintenance personnel to quickly and intuitively locate the defect and clarify its condition during actual operation, thereby improving the targeting and efficiency of the repair work.

[0078] S600: Real-time acquisition of repair progress information, generation of repair progress based on defect reports, and output of guidance and prompts.

[0079] The system outputs guidance and prompts in a preset order, generated based on defect reports. The priority order is severity, defect type, and defect location. Specifically, the system prioritizes outputting prompts based on defect severity, showing prompts for high-severity defects first to guide maintenance personnel in addressing critical issues. Next, it provides targeted repair guidance based on defect type. Finally, it combines defect location information to offer more specific and precise operational guidance to maintenance personnel.

[0080] This systematic approach to providing guidance and prompts enables repair personnel to conduct repairs more systematically and efficiently. The system also updates the repair progress in real-time on the AR interface and provides corresponding feedback, allowing repair personnel to stay informed about the overall progress. Furthermore, the system features intelligent voice prompts that trigger based on the repair progress, guiding repair personnel to the next step. During the repair process, catering to individual needs, repair personnel can access more detailed repair suggestions at any time or adjust the AR interface layout as needed for a more convenient and efficient repair experience.

[0081] During the parts repair process, maintenance personnel can perform repair operations based on the projected results. Simultaneously, they can input repair progress information into the system in real time through various interactive methods such as touchscreens, gestures, or voice. The system acquires this repair progress information in real time and generates an accurate repair schedule based on the defect report.

[0082] Furthermore, the system will update the repair progress in real time on the AR interface and provide corresponding feedback information, enabling maintenance personnel to understand the overall progress of the repair work at any time.

[0083] In addition, the system features intelligent voice prompts that can trigger voice prompts based on the repair progress, guiding repair personnel to the next step. During the repair process, considering the personalized needs of repair personnel, they can access more detailed repair suggestions at any time, or adjust the AR interface layout according to actual operational needs, for a more convenient and efficient repair experience.

[0084] In one embodiment, the step of projecting the defect area onto the surface of the actual part in the form of color coding or marking, based on projection technology, specifically includes:

[0085] Based on the intensity of light interference, inertial measurement data and machine vision features, an initial pose transformation prediction is generated by fusing them together.

[0086] Based on the pose offset error and visual optical flow compensation, the refined pose is output after the initial pose transformation prediction is corrected.

[0087] Based on the refined pose, the defect area is projected onto the actual part surface.

[0088] In this embodiment, the light interference intensity is based on real-time acquired ambient light parameters, which are acquired using multiple ring-shaped distributed spectral sensors, including ambient light intensity gradient changes and direction vectors, and then obtained by monitoring the ambient light intensity gradient changes and direction vectors.

[0089] Taking four spectral sensors as an example, the ambient light intensity values ​​are first collected synchronously. and direction angle .

[0090] Ambient light intensity gradient change: Calculate the absolute value of the rate of change of light intensity between adjacent sensors, and take the maximum value as the value. The calculation formula is as follows:

[0091] ;

[0092] Direction vector: D is obtained by averaging the sensor direction angles for light intensities exceeding the threshold L. e The calculation formula is as follows:

[0093] ;

[0094] Where n is the number of sensors whose light intensity exceeds the threshold L.

[0095] In addition, the steps for fusing and generating the initial pose transformation prediction based on light interference intensity, inertial measurement data, and machine vision features in this embodiment specifically include:

[0096] Based on the intensity of light interference, the weights of inertial measurement data and machine visual features are dynamically allocated;

[0097] The inertial measurement data and the machine's visual features are weighted and fused to generate an initial pose transformation prediction.

[0098] The dynamic weight fusion formula for the initial pose transformation prediction is:

[0099] ;

[0100] In the formula, T imu For inertial measurement data, T vis For the machine's visual characteristics.

[0101] Weight k is based on light interference intensity Adjustments are made based on preset thresholds. For example, when there is strong light interference (i.e., the light interference intensity is greater than 2000), the visual system may be "blinded". In this case, 70% of the IMU data is trusted, i.e., k=0.7. When the light is stable, 80% of the visual data is trusted, i.e., k=0.2.

[0102] In one embodiment, the step of outputting the refined pose after correcting the initial pose transformation prediction based on the pose offset error and visual optical flow compensation specifically includes:

[0103] Calculate the expected shadow change based on the surface normal vector, direction vector, and predicted pose transformation of the part.

[0104] By comparing the expected shadow change with the actual shadow change captured by the 3D scan, the pose offset error is obtained.

[0105] Based on the dynamic allocation of shadow compensation weights and visual optical flow weights according to the intensity of light interference, the pose offset error and visual optical flow compensation are weighted and fused, and the refined pose is output after the initial pose transformation prediction is corrected.

[0106] In this embodiment, the ambient light direction vector D is used. e Construct a model of surface shadow variation for the part:

[0107] ;

[0108] In the formula, N is the point cloud normal vector matrix, and ΔPpred To predict pose offset.

[0109] The predicted pose offset is obtained based on the difference between the predicted initial pose transformations at different times. Then, the actual shadow change ΔSreal is captured by 3D scanning, and the result is compared with the actual shadow change ΔS. real Compared with the predicted value By inversely calculating the pose error E caused by inconsistent shadows, p .

[0110] The formula for calculating pose error Ep is:

[0111] ;

[0112] The formula for outputting the refined pose is:

[0113] ;

[0114] In the formula, P fine To refine the pose, E f This is the visual optical flow compensation amount calculated based on the optical flow of image feature points.

[0115] Among them, W s +W f =1, weight W s The specific value is dynamically allocated based on the intensity of light interference. The dynamic allocation logic is the same as that of weight k, but the thresholds for adjustment are different.

[0116] In one embodiment, the step of projecting the defect area onto the surface of the actual part in the form of color coding or marking, based on projection technology, further includes:

[0117] Based on the intensity of light interference, the projection brightness and contrast are dynamically adjusted during the projection process.

[0118] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for real-time detection of part rework status based on augmented reality projection, characterized in that, include: The surface of the part to be inspected is scanned in three dimensions in real time to generate high-density point cloud data; The point cloud data is transmitted to the edge computing unit for preprocessing. Obtain the standard CAD model of the part to be inspected, and register it with the preprocessed point cloud data using an improved ICP algorithm; After registration, the standard CAD model and point cloud data are compared and analyzed to identify defect areas and output a defect report; Based on projection technology, defective areas are projected onto the surface of the actual part in the form of color coding or marking; It can obtain real-time information on the progress of the repair, generate a repair schedule based on the defect report, and output guidance and prompts. The projection-based projection steps specifically include: Based on the intensity of light interference, inertial measurement data, and machine vision features, an initial pose transformation prediction is generated by fusing them together. Specifically, the steps for generating the initial pose transformation prediction include: dynamically allocating weights between inertial measurement data and machine vision features based on the intensity of light interference; and weighted fusing of inertial measurement data and machine vision features to generate the initial pose transformation prediction. Based on the pose offset error and visual optical flow compensation, a refined pose is output after correcting the initial pose transformation prediction. Specifically, the correction steps include: calculating the expected shadow change based on the part surface normal vector, direction vector, and predicted pose transformation; comparing the expected shadow change with the actual shadow change captured by 3D scanning to obtain the pose offset error; dynamically allocating shadow compensation weights and visual optical flow weights based on light interference intensity, weighted fusing the pose offset error and visual optical flow compensation, and outputting the refined pose after correcting the initial pose transformation prediction. Based on the refined pose, the defect area is projected onto the actual part surface.

2. The method for real-time detection of part rework status based on augmented reality projection according to claim 1, characterized in that: The real-time 3D scanning uses a scanning frequency of 10Hz-60Hz and performs dynamic compensation based on a motion compensation algorithm during the scanning process.

3. The real-time detection method for the rework status of parts based on augmented reality projection according to claim 1, characterized in that: During 3D scanning, the system automatically focuses based on the shape and size of the part to be inspected.

4. The method for real-time detection of part rework status based on augmented reality projection according to claim 1, characterized in that: The defect report includes the defect location, defect type, and severity, and the guidance and prompt information is output based on a preset order; The preset order is generated based on defect reports, and the priority of generation is, in order, severity, defect type, and defect location.

5. The method for real-time detection of part rework status based on augmented reality projection according to claim 1, characterized in that: The light interference intensity is obtained based on real-time acquired ambient light parameters, after monitoring changes in ambient light intensity gradient and direction vector.

6. The method for real-time detection of part rework status based on augmented reality projection according to claim 1, characterized in that, The step of projecting the defect area onto the surface of the actual part in the form of color coding or marking based on projection technology further includes: dynamically adjusting the projection brightness and contrast during the projection process based on the intensity of light interference.

7. A real-time detection system for the rework status of parts based on augmented reality projection, applied to the real-time detection method for the rework status of parts based on augmented reality projection as described in any one of claims 1-6, characterized in that, include: The three-dimensional scanning module (10) is used to perform three-dimensional real-time scanning of the surface of the part to be inspected and generate high-density point cloud data. The data processing module (20) is used to transmit point cloud data to the edge computing unit for preprocessing; The standard CAD model of the part to be inspected is used to obtain the standard CAD model and register it with the preprocessed point cloud data using an improved ICP algorithm; after registration, the standard CAD model and point cloud data are compared and analyzed to identify defect areas and output a defect report. AR projection module (30) is used to project the defect area onto the surface of the actual part in the form of color coding or marking based on projection technology; User interaction module (40) is used to obtain repair progress information in real time, generate repair progress based on defect report, and output guidance prompt information.

Images