A fast matching positioning optical projection system and method

Abstract

The application relates to a rapid matching positioning optical projection system and method, wherein the optical projection system comprises a projector, a camera, an information editing module, a rapid positioning module and a projection operation module; the information editing module is used for compiling virtual process information required by each working procedure according to an assembly process flow, and generating a guide information set file; the rapid positioning module is used for determining two-dimensional coordinates of a preset target ball mark reference point in a camera image after movement according to projected workpiece images before and after movement of the optical projection system, and corresponding points of the target ball mark reference point in a projector image, and determining pose information of the projected workpiece relative to a projector space coordinate system in combination with measurement data of three-dimensional coordinates of the target ball obtained by a measuring device; the projection operation module is used for generating a working procedure guide image through a three-dimensional space projection algorithm according to the guide information set file and the pose information; and the projector is used for projecting the working procedure guide image on a surface of the projected workpiece.

Description

Technical Field

[0001] This application relates to the field of aircraft assembly technology, and in particular to an optical projection system and method for rapid matching and positioning. Background Technology

[0002] Manual assembly, a crucial part of aircraft assembly, remains largely traditional, significantly hindering the development of the aerospace manufacturing industry. Due to the inherent limitations of manual assembly and the poor intuitiveness of traditional paper-based assembly process documents, workers must repeatedly consult assembly process cards and mark workpiece surfaces, greatly disrupting the assembly rhythm and limiting efficiency. Furthermore, it easily leads to misassemblies and omissions, affecting assembly quality. Augmented reality-based assembly systems offer a crucial solution to the low efficiency and error-prone nature of manual operations. Depending on the projection device, they are generally divided into laser scanning and projection imaging types. Projection imaging devices typically use ordinary DLP, 3LCD, or LCD projectors, offering rich projection content and high projection frequency. They employ stereoscopic vision algorithms to accurately project virtual process information onto corresponding spatial locations. These devices typically use vision sensors to dynamically acquire the relative pose of workpieces through contour matching, feature matching, and point cloud matching. However, for larger workpieces or those with limited surface texture, positioning accuracy is poor, and they cannot incorporate measurement data from high-precision on-site measuring equipment.

[0003] Therefore, the inventors have provided an optical projection system and method for rapid matching and positioning. Summary of the Invention

[0004] (1) Technical problems to be solved

[0005] This application provides an optical projection system and method for rapid matching and positioning. The technical problem to be solved is that existing methods cannot achieve accurate positioning for workpieces with large size or incomplete surface texture, and cannot be combined with measurement data from high-precision measuring equipment commonly used in assembly sites, such as laser trackers and photogrammetry equipment, which affects the positioning accuracy.

[0006] (2) Technical solution

[0007] In a first aspect, this application provides an optical projection system for rapid matching and positioning, including a projector, a camera, an information editing module, a rapid positioning module, and a projection operation module;

[0008] The information editing module is used to arrange the virtual process information required for each process according to the assembly process flow and generate a guide information set file;

[0009] The rapid positioning module is used to determine the two-dimensional coordinates of the preset target ball marker reference point in the camera image after the movement of the optical projection system based on the images of the projected workpiece captured by the camera before and after the movement of the optical projection system. It also determines the corresponding point of the target ball marker reference point in the camera image in the projector image through the structured light method. Combined with the measurement data of the three-dimensional coordinates of the target ball obtained by the measuring equipment, the module determines the pose information of the projected workpiece relative to the spatial coordinate system of the projector.

[0010] The projection operation module is used to generate process guidance images based on the guidance information set file and pose information, using a three-dimensional spatial projection algorithm.

[0011] A projector is used to project process guidance images onto the surface of the workpiece being projected.

[0012] Furthermore, the optical projection system also includes a system calibration module; the system calibration module is used to calibrate the internal parameters of the projector.

[0013] Furthermore, the system calibration module is specifically used to: control the projector to project structured light stripes onto the checkerboard optical calibration board, and control the camera to capture images of the structured light stripes each time they are projected; identify the corner points of the optical calibration board in the camera images, decode the image coordinates of the corner points of the optical calibration board in the projector images based on the image sequence recorded by the camera, and calculate the projector intrinsic parameters through the calibration algorithm.

[0014] Furthermore, the fast localization module is specifically used to: identify and match geometric features in the images before and after the movement using a deep learning template matching algorithm, calculate the homography matrix between images based on the matching results, and then map the preset target sphere bounding box in the image after the movement.

[0015] Furthermore, the fast positioning module is also specifically used to: perform super-resolution processing on the image within the target ball's bounding box, then perform image processing, calculate the coordinates of the target ball's marker reference point based on the marker pattern and restore it to its original size, thereby calculating the two-dimensional coordinates of the target ball's marker reference point in the camera image.

[0016] Furthermore, the rapid positioning module is also specifically used to: establish the correspondence between the camera and projector two-dimensional image coordinate systems using structured light technology, and calculate the corresponding point of the target ball marker reference point in the camera image in the projector image based on the correspondence.

[0017] Furthermore, the rapid positioning module is also specifically used to: solve the pose information of the projected workpiece relative to the projector's spatial coordinate system using the PNP algorithm based on the two-dimensional coordinates of the target ball marker reference point in the projector image and the measurement data of the three-dimensional coordinates of the target ball marker reference point.

[0018] Furthermore, the projection operation module is specifically used to: discretize the virtual process information into a three-dimensional point set, and generate a process guidance image through a three-dimensional spatial projection algorithm based on the pose matrix of the projected workpiece relative to the projector's spatial coordinate system, the projector's internal parameters, and the three-dimensional point set.

[0019] Furthermore, the projection operation module is specifically used to enable interaction between the operator and the optical projection system.

[0020] Secondly, this application provides a fast matching and positioning optical projection method, implemented based on the fast matching and positioning optical projection system described above, comprising:

[0021] The information editing module arranges the virtual process information required for each process according to the assembly process flow and generates a guide information set file;

[0022] The rapid positioning module determines the two-dimensional coordinates of the preset target ball marker reference point in the camera image after the movement of the optical projection system based on the images of the projected workpiece captured by the camera before and after the movement of the optical projection system. It also determines the corresponding point of the target ball marker reference point in the camera image in the projector image through the structured light method. Combined with the measurement data of the target ball's three-dimensional coordinates obtained by the measuring equipment, the module determines the pose information of the projected workpiece relative to the projector's spatial coordinate system.

[0023] The projection operation module generates process guidance images based on the guidance information set file and pose information using a three-dimensional spatial projection algorithm;

[0024] The projector projects process guidance images onto the surface of the workpiece.

[0025] (3) Beneficial effects

[0026] The above-mentioned technical solution of this application has the following advantages:

[0027] The optical projection system for rapid matching and positioning provided in the first aspect of this application employs a deep learning template matching algorithm to identify and match geometric features in images before and after movement. This enables dynamic tracking of the target bounding box and identification of the target center point within the bounding box. This secondary positioning method achieves rapid positioning of workpieces with large dimensions or incomplete surface textures, satisfying both efficiency and positioning accuracy requirements. The system can incorporate measurement data from commonly used high-precision measuring equipment in assembly areas and convert it to the same coordinate system, allowing multiple systems to be used in conjunction to meet the projection-assisted assembly needs of workpieces of different sizes.

[0028] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 A schematic diagram of the optical projection system for rapid matching and positioning provided in this application;

[0031] Figure 2 This application provides a schematic diagram of the optical projection system in operation.

[0032] Figure 3 A flowchart of the optical projection method for rapid matching and positioning provided in this application.

[0033] Reference numerals: 1. Projector; 2. Camera; 3. GPU computing unit; 4. Movable tripod; 5. Aircraft panel; 6. Rib; 7. Tooling; 8. TB point; 9. Target ball. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0035] It should be understood that, when used in this application specification and appended claims, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, in the description of this application specification and appended claims, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0037] With the rapid development of automation, digitalization, and intelligent technologies, aircraft assembly efficiency and quality have been significantly improved. Augmented reality-based assembly systems are an important way to address the low efficiency and error-prone nature of manual operations. Depending on the projection device, they are generally divided into laser scanning and projection imaging types. Laser scanning devices use mechanical light scanning devices, offering high projection accuracy. However, when there is too much projected information, the human eye can observe the trajectory of the moving laser point, and the projection content is often monotonous, affecting the usability. Projection imaging devices have lower projection accuracy than laser scanning devices.

[0038] In practical applications, because the assembly site was not designed with the use of optical projection systems in mind, no space was reserved for their use. This leads to frequent changes in the position between the optical projection system and the projected workpiece, requiring repositioning of the workpiece and affecting the flexibility of equipment use and assembly efficiency. Furthermore, existing methods cannot accurately position large workpieces or those with intricate surface textures, and they cannot be combined with measurement data from high-precision measuring equipment commonly used in assembly sites, such as laser trackers and photogrammetry equipment, thus affecting positioning accuracy.

[0039] To address the aforementioned problems, this application provides an optical projection system and method for rapid matching and positioning, which can solve the problems mentioned above.

[0040] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0041] The optical projection system for rapid matching and positioning provided in this embodiment includes a projector, a camera, an information editing module, a rapid positioning module, and a projection operation module. The information editing module is used to arrange the virtual process information required for each process according to the assembly process flow and generate a guidance information set file. The rapid positioning module is used to determine the two-dimensional coordinates of the preset target ball marker reference point in the camera image after movement by using a deep learning template matching method based on the images of the projected workpiece before and after the optical projection system moves, and to determine the corresponding point of the target ball marker reference point in the camera image in the projector image by using a structured light method. Combined with the measurement data of the target ball's three-dimensional coordinates obtained by the measuring device, the pose information of the projected workpiece relative to the projector's spatial coordinate system is determined. The projection operation module is used to generate a process guidance image by using a three-dimensional spatial projection algorithm based on the guidance information set file and the pose information. The projector is used to project the process guidance image onto the surface of the projected workpiece.

[0042] In some embodiments, the optical projection system further includes a system calibration module; the system calibration module is used to calibrate the internal parameters of the projector.

[0043] In some embodiments, the system calibration module is specifically used to: control the projector to project structured light stripes onto the checkerboard optical calibration board, and control the camera to capture images of the structured light stripes each time they are projected; identify the corner points of the optical calibration board in the camera images, decode the image point coordinates of the corner points of the optical calibration board in the projector images based on the image sequence recorded by the camera, and calculate the projector intrinsic parameters through a calibration algorithm.

[0044] In applications, such as Figure 1 As shown, the optical projection system hardware mainly consists of a projector 1, a camera 2, a GPU computing unit 3, a connecting bracket, and a mobile tripod 4. The software mainly includes an information editing module, a quick positioning module, a projection operation module, and a system calibration module. The projector is used to project process guidance images onto the workpiece surface and to project structured light pattern sequences. Key parameters such as resolution and lumens are mainly determined by the distance between the system and the projected workpiece, the size of the projected workpiece, the ambient light intensity, and the required projection accuracy. The projector is not limited to using LCP, 3LCD, DLP, or other projection technologies. The projector's internal parameters, including {f}, need to be calibrated. p u p v p k1 p k2 p k3 p p1 p p2 p}, where f p Indicates focal length, u p v p Represents the principal point coordinates, {k1 p k2p k3 p} represents radial distortion, {p1 p p2 p The symbol} represents tangential distortion. The camera tracks and identifies the target center and uses the recorded structured light sequence to convert the target center coordinates from the camera image to the projector image. The key parameter, resolution, is primarily determined by the distance between the system and the projected workpiece, the size of the projected workpiece, and the visual positioning accuracy. Camera and lens parameters can be selected based on the actual assembly environment, positioning accuracy requirements, positioning distance requirements, and the size of the projected workpiece, without needing to calibrate the camera's internal parameters. The projector image covers the entire projected workpiece; multiple optical projection systems can also be used to project large workpieces together. The camera can use fixed-focus or zoom lenses, adjusting parameters such as exposure and gamma correction according to the actual ambient light intensity, and obtaining a clear image of the projected workpiece through zooming and focusing.

[0045] The system hardware is connected and fixed to a mobile tripod via a bracket. The GPU computing unit is used to control the projector and camera, deploy the optical projection system software, and enable the operation of various functional modules. The mobile tripod facilitates the movement of the optical projection system. The information editing module can extract virtual process information from the 3D digital models or 2D drawings (DWG, DXF, etc.) of the projected workpiece in common data formats such as STP and STL, and can arrange the virtual process information required for each process according to the assembly process flow. It can also add text, images, and other information to generate a guidance information set file. The rapid positioning module can quickly match and position the projected workpiece to determine its pose relative to the projector's spatial coordinate system; the projection operation module can generate guidance images for each process and control the projector to project the corresponding process content; the system calibration module can perform parameter calibration on the projector.

[0046] When using the system for the first time, only the internal parameters of the projector need to be calibrated. In the information editing module, the CAD model of the workpiece to be projected can be imported, assembly process information can be extracted according to the assembly process flow, and each process can be represented by arranging various information. The orientation of the optical projection system can be adjusted so that the projector image can cover the entire workpiece. Camera parameters can be adjusted, zoomed, and focused to ensure clear acquisition of the image of the projected workpiece. According to the laser tracker usage specifications, at least six target ball mounts should be fixed on the fixture within the projection range, and the laser tracker target ball should be placed there. The laser tracker should be used to measure the target ball, and the measurement results should be converted to the workpiece coordinate system. The three-dimensional coordinates P of the target ball should be recorded. iAfter the measurement is completed, the laser tracker is removed; the laser tracker target ball is replaced with a photogrammetric conversion target ball of the same size. The system uses commonly used measuring equipment in aircraft assembly sites, such as laser trackers and photogrammetric equipment, to measure the reference points on the workpiece or tooling, instead of directly using the theoretical values ​​of the reference point digital model, so that the three-dimensional coordinate data of the reference points are more consistent with reality.

[0047] The system only needs to calibrate the projector's internal parameters, eliminating the need to calibrate the camera-projector 3D spatial relationship and the camera's internal parameters, thus reducing system complexity. The system calibration module controls the projector to project structured light fringes onto a checkerboard optical calibration board, allowing for the use of various structured light forms such as Gray code grating fringes, sinusoidal grating fringes, and colored grating fringes. Simultaneously, it controls the camera to record images of each projected structured light fringe, identifying the corner points of the optical calibration board in the camera images. Then, based on the image sequence recorded by the camera, it decodes the coordinates of these corner points in the projector image. After recording the coordinates of the corner points on the checkerboard optical calibration board in more than 10 different poses in the projector image, calibration algorithms such as Zhang Zhengyou's checkerboard calibration method are used to calculate the projector's internal parameters.

[0048] In some embodiments, the fast localization module is specifically used to: identify and match geometric features in the images before and after the movement using a deep learning template matching algorithm, calculate the homography matrix between images based on the matching results, and then map a preset target sphere bounding box in the image after the movement.

[0049] In some embodiments, the fast positioning module is further specifically used to: perform super-resolution processing on the image within the target ball range frame, then perform image processing, calculate the coordinates of the target ball marker reference point according to the marker pattern and restore it to the original size, thereby calculating the two-dimensional coordinates of the target ball marker reference point in the camera image.

[0050] In some embodiments, the rapid positioning module is further specifically used to: establish a correspondence between the camera and projector two-dimensional image coordinate systems using structured light technology, and calculate the corresponding point of the target ball marker reference point in the camera image in the projector image based on the correspondence.

[0051] In some embodiments, the rapid positioning module is further specifically used to: solve the pose information of the projected workpiece relative to the projector spatial coordinate system using the PNP algorithm based on the two-dimensional coordinates of the target ball marker reference point in the projector image and the measurement data of the three-dimensional coordinates of the target ball marker reference point.

[0052] In the application, the optical projection system captures the current image, marks the target sphere's bounding box on the tooling in the rapid positioning module, and associates it with the target sphere's three-dimensional coordinates P. iWhen the relative position between the optical projection system and the projected workpiece changes, the optical projection system captures an image after the movement. The rapid positioning module automatically maps the target sphere bounding box into the image after the movement using a deep learning matching algorithm, and identifies the two-dimensional coordinates p of the target sphere marker reference point within the image bounding box. ci The optical projection system projects a sequence of structured light to calculate the target ball's marker reference point p. ci The corresponding point p in the projector image pi Combining the measurement results of the target sphere's three-dimensional coordinates using a laser tracker, P i The PNP algorithm can be used to directly and quickly solve the pose information of the projected workpiece relative to the spatial coordinate system of the projector.

[0053] Deep learning template matching algorithms identify and match geometric features in images before and after movement, and calculate the homography matrix between images based on the matching results. This allows for mapping the target sphere bounding box in the moved image, enabling rapid tracking of the target sphere bounding box. The target sphere marker reference point recognition method involves: first, performing super-resolution processing on the image within the target sphere bounding box; then performing image processing such as filtering, binarization, erosion, and dilation; calculating the coordinates of the marker reference points based on the marker pattern and restoring them to their original size; and finally calculating the coordinates p of the target sphere marker reference points in the camera image. ci By employing a secondary positioning method, rapid positioning of workpieces with large dimensions or inconspicuous surface textures can be achieved, satisfying both efficiency and positioning accuracy requirements.

[0054] Determine the corresponding point p of the target ball's reference point in the projector image. pi Structured light technology can be used to establish the correspondence between the camera and projector's two-dimensional image coordinate system. First, structured light stripes, without restriction to using Gray code grating stripes, sinusoidal grating stripes, or colored grating stripes, are projected onto the surface of the workpiece. Simultaneously, the camera records the image of each projected structured light stripe. The pixels within the projection area in the camera image are decoded, and the correspondence F between camera pixels and projector pixels within the projection area is established. Then:

[0055] (u ip v ip )=F(u jc v jc Equation (1)

[0056] Where (u jc v jc (u) represents the lower point in the camera pixel coordinate system within the projection area. ip v ip ) represents the point (u) in the camera pixel coordinate system. jc v jcThe corresponding point in the projector's pixel coordinate system. Based on the correspondence F, the reference point p of the target ball marker in the image can be calculated. ci The corresponding point p in the projector image pi Furthermore, reference points can be directly projected onto the actual target ball to check the target center recognition effect.

[0057] The pose determination method for the projected workpiece relative to the projector coordinate system, based on the coordinate system transformation relationship, the system includes the camera spatial coordinate system, the projector spatial coordinate system, and the projected workpiece spatial coordinate system. The pose transformation matrix B between the projected workpiece and the camera coordinate system can be determined using the visual positioning method, and the pose transformation matrix C between the camera and the projector coordinate system can be determined using the camera-projector three-dimensional spatial calibration. This can be done according to equation (2):

[0058] D = B·C (Equation 2)

[0059] The pose matrix D of the projected workpiece relative to the projector coordinate system is calculated, but this method introduces four parts: camera intrinsic parameter calculation error, projector intrinsic parameter calculation error, camera-projector 3D spatial calibration error, and visual positioning algorithm error. This application uses structured light technology to calculate the reference point p of the target ball marker in the camera image. ci The corresponding point p in the projector image pi Combined with the actual measurement results of the target sphere's three-dimensional coordinates P i The PNP algorithm is used to directly solve the pose of the projected workpiece relative to the spatial coordinate system of the projector. The error of this method consists of only two parts: the error of the projector intrinsic parameter calculation and the error of the visual positioning algorithm. It is better than the indirect calculation method through coordinate system transformation.

[0060] In some embodiments, the projection operation module is specifically used to: discretize virtual process information into a three-dimensional point set, and generate a process guidance image through a three-dimensional spatial projection algorithm based on the pose matrix of the projected workpiece relative to the spatial coordinate system of the projector, the internal parameters of the projector, and the three-dimensional point set.

[0061] In some embodiments, the projection operation module is also specifically used to enable interaction between the operator and the optical projection system.

[0062] In the application, the projection operation module generates a guide image based on a 3D spatial projection algorithm. Operators can interact with the optical projection system through the software interface of the projection operation module, or via gestures, voice, etc., to project the required process information and complete auxiliary assembly tasks. Simultaneously, operators can manually adjust the position of virtual process information in the guide image and add auxiliary process information such as text and images based on the actual on-site projection effect.

[0063] The three-dimensional spatial projection algorithm discretizes the virtual assembly process information into a three-dimensional point set. The system uses equation (3) to accurately project and superimpose the virtual assembly process information onto the surface of the real workpiece, thus achieving distortion-free projection in three-dimensional space.

[0064]

[0065] Among them, P w (X w ,Y w Z w This represents virtual assembly process information in the form of a three-dimensional point set. Let A represent the pose matrix of the workpiece to be projected relative to the projector coordinate system, A represent the projector intrinsic parameters, p(u,v) represent the image point coordinates in the projection pixel coordinate system, and s is the scale factor.

[0066] The optical projection system for rapid matching and positioning provided in this application uses a deep learning template matching algorithm to identify and match geometric features in images before and after movement, thereby achieving dynamic tracking of the target bounding box and identification of the target center point within the bounding box. This secondary positioning method enables rapid positioning of workpieces with large dimensions or incomplete surface textures, meeting both efficiency and positioning accuracy requirements. The system employs camera-projector two-dimensional image calibration, mapping the reference points identified in the camera image onto the projector image, directly calculating the pose parameters of the projected workpiece relative to the projector. This avoids errors introduced during coordinate system transformation, improving the system's workpiece positioning accuracy. The system can incorporate measurement data from commonly used high-precision measuring equipment in assembly areas and can convert it to the same coordinate system, enabling multiple systems to be used together to meet the projection-assisted assembly needs of workpieces of different sizes.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application.

[0068] This embodiment also provides a fast matching and positioning optical projection method, implemented based on the fast matching and positioning optical projection system described in the above embodiment, including:

[0069] The information editing module arranges the virtual process information required for each process according to the assembly process flow and generates a guide information set file;

[0070] The rapid positioning module determines the two-dimensional coordinates of the preset target ball marker reference point in the camera image after the movement of the optical projection system based on the images of the projected workpiece captured by the camera before and after the movement of the optical projection system. It also determines the corresponding point of the target ball marker reference point in the camera image in the projector image through the structured light method. Combined with the measurement data of the three-dimensional coordinates of the target ball obtained by the measuring device, the module determines the pose information of the projected workpiece relative to the spatial coordinate system of the projector.

[0071] The projection operation module generates process guidance images based on the guidance information set file and the pose information using a three-dimensional spatial projection algorithm;

[0072] The projector projects the process guidance image onto the surface of the workpiece.

[0073] In application, such as Figure 2 Taking the assembly of aircraft panel ribs as an example, this embodiment illustrates the optical projection method for rapid matching and positioning.

[0074] The assembly personnel use the operating system calibration module to calibrate the projector's internal parameters. In the optical projection system information editing module, the process engineers import the aircraft panel 5 digital model, extract the outline of the ribs 6 to be assembled, determine the rib outlines to be projected for each process according to the rib installation sequence specified in the assembly process flow, and save this as a guide information set file.

[0075] Assembly personnel fix the aircraft panel 5 onto the fixture 7 according to the positioning holes. Measurement personnel use a laser tracker to measure the spatial coordinates of multiple TB points 8 on the fixture, and then fit these coordinates with the corresponding theoretical coordinates to establish the transformation relationship between the laser tracker coordinate system and the aircraft panel coordinate system. Assembly personnel adjust the attitude of the optical projection system so that the projected image covers the entire aircraft panel 5. Assembly personnel adjust the camera parameters, zooming and focusing to ensure a clear image of the aircraft panel.

[0076] The surveyor fixed six target ball mounts on the fixture within the projection range of the optical projection system according to the laser tracker measurement and usage specifications, placed the laser tracker target ball, measured the target ball using the laser tracker, and converted it to the aircraft panel coordinate system, recording the three-dimensional coordinates P of the target ball. i After completing the measurement of 6 target balls, the laser tracker is removed. The assembly personnel replace the laser tracker target balls fixed on the 6 target ball mounts on the aircraft tooling with photogrammetry conversion target balls 9 of the same size, and at the same time, the reflective mark at the center of the target ball 9 is oriented towards the optical projection system.

[0077] The assembly personnel use the optical projection system software to capture the current image, mark the bounding box of the target ball 9 on the tooling 7 in the rapid positioning module, and associate the three-dimensional coordinates P of the target ball 9. i The assembly personnel move the optical projection system to the work site during the simulation operation, adjusting its attitude so that the projected image covers the entire aircraft panel 5. The assembly personnel then operate the optical projection system to capture images after the movement. The rapid positioning module automatically maps the target sphere's bounding box into the moved image using a matching algorithm.

[0078] The rapid positioning module automatically identifies the target ball marker reference point p within the range box. ci The rapid positioning module automatically projects a structured light sequence and calculates the target ball marking reference point p. ci The corresponding point p in the projector image pi The system projects the target ball's reference point, and the assembly personnel check the accuracy of the center projection of the target ball 9. The rapid positioning module, combined with the measurement data of the target ball 9 from the laser tracker, uses the PNP algorithm to solve the workpiece pose in the current state.

[0079] The projection operation module automatically generates guide images for each process. Assembly personnel operate the projection operation module to project the outline of the ribs to be assembled in each process, select the corresponding ribs, and assemble them according to the projected outline to complete the wall panel rib assembly task.

[0080] The optical projection method for rapid matching and positioning provided in this application embodiment has the following overall process: Figure 3 As shown. Compared to similar augmented reality-assisted assembly methods, this method uses projection to directly overlay virtual assembly process information onto the surface of the real workpiece. Compared to augmented reality methods such as head-mounted AR glasses and video, it does not require additional workload for operators and facilitates collaborative work among multiple people. This application adopts a fast matching and positioning method. First, a deep learning template matching algorithm is used to identify and match geometric features in the images before and after movement. Based on the homography relationship between images, the target bounding box on the workpiece or tooling is dynamically tracked. Then, an image processing algorithm is used to accurately identify the coordinates of the target ball marker reference point and determine the corresponding point of the target ball marker reference point in the projector image. Measurement data of the target ball from commonly used measuring equipment in aircraft assembly sites, such as laser trackers and photogrammetry equipment, are introduced. The PNP algorithm is used to directly calculate the pose of the projected workpiece relative to the projector coordinate system. Compared to contour matching and point cloud matching, this method has less computation and faster calculation speed. Compared to traditional feature matching methods, this method has higher matching and positioning accuracy. The optical projection system and method for rapid matching and positioning in this application can combine measurement data from commonly used measuring equipment in aircraft assembly sites, such as laser trackers and photogrammetric equipment. It has high positioning accuracy and projection accuracy of less than ±1mm, which meets the accuracy requirements of manual assembly in aircraft assembly tasks.

[0081] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0082] Those skilled in the art will recognize that the device and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An optical projection system for rapid matching and positioning, characterized in that, It includes a projector, camera, information editing module, quick positioning module, and projection operation module; The information editing module is used to arrange the virtual process information required for each process according to the assembly process flow and generate a guide information set file. The rapid positioning module is used to determine the two-dimensional coordinates of the preset target ball marker reference point in the camera image after the movement of the optical projection system based on the images of the projected workpiece captured by the camera before and after the movement of the optical projection system, and to determine the corresponding point of the target ball marker reference point in the camera image in the projector image by using the structured light method, and to determine the pose information of the projected workpiece relative to the projector spatial coordinate system by combining the measurement data of the target ball three-dimensional coordinates obtained by the measuring device. The projection operation module is used to generate process guidance images based on the guidance information set file and the pose information using a three-dimensional spatial projection algorithm; The projector is used to project the process guidance image onto the surface of the workpiece being projected. The fast positioning module is specifically used to: identify and match geometric features in the images before and after the movement using a deep learning template matching algorithm, calculate the homography matrix between images based on the matching results, and then map the preset target ball range box in the image after the movement. The fast positioning module is also specifically used to: perform super-resolution processing on the image within the target ball range frame, then perform image processing, calculate the coordinates of the target ball mark reference point according to the mark pattern and restore it to the original size, thereby calculating the two-dimensional coordinates of the target ball mark reference point in the camera image.

2. The optical projection system for rapid matching and positioning as described in claim 1, characterized in that, It also includes a system calibration module; the system calibration module is used to calibrate the internal parameters of the projector.

3. The optical projection system for rapid matching and positioning as described in claim 2, characterized in that, The system calibration module is specifically used to: control the projector to project structured light stripes onto the checkerboard optical calibration board, and control the camera to capture images of the structured light stripes each time they are projected; identify the corner points of the optical calibration board in the camera images, decode the image coordinates of the corner points of the optical calibration board in the projector images based on the image sequence recorded by the camera, and calculate the projector intrinsic parameters through a calibration algorithm.

4. The optical projection system for rapid matching and positioning as described in claim 1, characterized in that, The rapid positioning module is also specifically used to: establish a correspondence between the camera and projector two-dimensional image coordinate systems using structured light technology, and calculate the corresponding point of the target ball mark reference point in the camera image in the projector image based on the correspondence.

5. The optical projection system for rapid matching and positioning as described in claim 1, characterized in that, The rapid positioning module is also specifically used to: calculate the pose information of the projected workpiece relative to the projector's spatial coordinate system using the PNP algorithm based on the two-dimensional coordinates of the target ball mark reference point in the projector image and the measurement data of the three-dimensional coordinates of the target ball mark reference point.

6. The optical projection system for rapid matching and positioning as described in claim 1, characterized in that, The projection operation module is specifically used to: discretize the virtual process information into a three-dimensional point set, and generate the process guidance image through a three-dimensional spatial projection algorithm based on the pose matrix of the workpiece relative to the projector's spatial coordinate system, the projector's internal parameters, and the three-dimensional point set.

7. The optical projection system for rapid matching and positioning as described in claim 1, characterized in that, The projection operation module is also specifically used to enable the interaction between the operator and the optical projection system.

8. An optical projection method for rapid matching and positioning, characterized in that, Based on the optical projection system for rapid matching and positioning as described in any one of claims 1 to 7, including: The information editing module arranges the virtual process information required for each process according to the assembly process flow and generates a guide information set file; The rapid positioning module determines the two-dimensional coordinates of the preset target ball marker reference point in the camera image after the movement of the optical projection system based on the images of the projected workpiece captured by the camera before and after the movement of the optical projection system. It also determines the corresponding point of the target ball marker reference point in the camera image in the projector image through the structured light method. Combined with the measurement data of the three-dimensional coordinates of the target ball obtained by the measuring device, the module determines the pose information of the projected workpiece relative to the spatial coordinate system of the projector. The projection operation module generates process guidance images based on the guidance information set file and the pose information using a three-dimensional spatial projection algorithm; The projector projects the process guidance image onto the surface of the workpiece. The fast localization module specifically uses a deep learning template matching algorithm to identify and match geometric features in the images before and after the movement, and calculates the homography matrix between images based on the matching results, thereby mapping a preset target ball bounding box in the image after the movement. The fast positioning module also performs super-resolution processing on the image within the target ball's bounding box, and then performs image processing to calculate the coordinates of the target ball's marker reference point based on the marker pattern and restore it to its original size, thereby calculating the two-dimensional coordinates of the target ball's marker reference point in the camera image.

Images