Creating synthetic data for composite inspection

By generating synthetic data with introduced discrepancies in a 3D model of composite layup, the method addresses the scarcity of discordant data in training, reducing waste and time while improving the efficiency and accuracy of mismatch identifier programs in composite manufacturing.

JP2024159491APending Publication Date: 2024-11-08THE BOEING CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024021126
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-28
Filing Date
2024-02-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing methods for training classification models to identify discrepancies in composite manufacturing face challenges due to the scarcity and specificity of discordant data, leading to inefficiencies in resource usage and increased waste.

Method used

The generation of synthetic data through a 3D model of composite layup, where discrepancies are introduced, and captured images are modified to simulate various inspection types, providing a comprehensive training dataset for mismatch identifier programs.

Benefits of technology

This approach reduces training time and resource consumption while minimizing waste by using synthetic data to enhance the accuracy and adaptability of mismatch identifier programs, enabling efficient quality control in composite manufacturing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159491000001_ABST
    Figure 2024159491000001_ABST
Patent Text Reader

Abstract

To provide a method of generating synthetic data for composite inspection.SOLUTION: A method of generating synthetic data for composite manufacturing inspection comprises creating a three-dimensional model of a composite layup; generating an inconsistency in the three-dimensional model of the composite layup; setting a camera position and a camera orientation in the three-dimensional model space; generating a captured image of the three-dimensional model and the inconsistency from the camera position and the camera orientation; and creating a secondary image from the captured image, the secondary image having coloration of a preselected inspection type.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to testing, and more particularly, to data training sets for mismatch identification. [Background technology]

[0002] To identify discrepancies in production, a dataset is constructed to train a classification model, the quality of which depends on the amount of discrepancy data. Summary of the Invention [Problem to be solved by the invention]

[0003] It would therefore be desirable to have a method and apparatus that takes into account at least some of the above-mentioned problems, as well as other possible problems. [Means for solving the problem]

[0004] One embodiment of the present disclosure provides a method for generating synthetic data for composite manufacturing inspection. A three-dimensional model of a composite layup is created. A mismatch is generated in the three-dimensional model of the composite layup. A camera position and a camera orientation are set in the three-dimensional model space. A captured image of the three-dimensional model and the mismatch is generated from the camera position and the camera orientation. A secondary image is created from the captured image, the secondary image having a preselected inspection type coloration.

[0005] Another embodiment of the present disclosure provides a method for training a composite inspection system. Captured images of a three-dimensional model of a composite layup with discrepancies are generated at predetermined camera positions and predetermined camera orientations in the three-dimensional model space. The captured images are modified to form secondary images with preselected inspection-type coloring. A discrepancy identifier program is trained using the secondary images.

[0006] Yet another embodiment of the present disclosure provides a method of using synthetically generated training data. A discrepancy is generated in a three-dimensional model including a multiple layer composite layup. An image of the discrepancy and the three-dimensional model is generated at a predetermined camera position and a predetermined camera orientation based on a head orientation of an inspection system connected to the composite laying head. A secondary image is generated from the image, the secondary image having an inspection type coloring of the inspection system. The secondary image is provided as training data for a discrepancy identifier program.

[0007] The features and functions may be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments, further details of which can be seen with reference to the following description and drawings.

[0008] The novel features believed characteristic of the exemplary embodiments are set forth in the appended claims. However, the exemplary embodiments, as well as their preferred modes of use, further objects and modes, will be best understood by reference to the following detailed description of exemplary embodiments of the present disclosure, when considered in conjunction with the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] 1 is an illustration of an aircraft in accordance with an illustrative embodiment; [Diagram 2] FIG. 2 is a block diagram of data flow according to an exemplary embodiment. [Diagram 3] FIG. 1 is a block diagram of a manufacturing environment in accordance with an illustrative embodiment. [Figure 4] FIG. 1 is a side view of a composite material installation and inspection in accordance with an illustrative embodiment; [Diagram 5] FIG. 1 is an illustration of a display of a three-dimensional model in accordance with an illustrative embodiment. [Figure 6A] 10 is an illustration of a flowchart of a method for generating synthetic data for composite manufacturing inspection in accordance with an illustrative embodiment; [Figure 6B]10 is an illustration of a flowchart of a method for generating synthetic data for composite manufacturing inspection in accordance with an illustrative embodiment; [Figure 7] 10 is a flowchart of a method for training a composite inspection system in accordance with an illustrative embodiment. [Figure 8] 1 is a flowchart of a method for using synthetically generated training data in accordance with an example embodiment. [Figure 9] 1 is a block diagram of an aircraft manufacturing and service method in accordance with an illustrative embodiment; FIG. [Figure 10] FIG. 1 is a block diagram of an aircraft in which an illustrative embodiment may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1, a diagram of an aircraft is shown in accordance with an illustrative embodiment. Aircraft 100 has wing 102 and wing 104 attached to body 106. Aircraft 100 includes engine 108 attached to wing 102 and engine 110 attached to wing 104.

[0011] The body 106 has an aft section 112. A horizontal stabilizer 114, a horizontal stabilizer 116, and a vertical stabilizer 118 are attached to the aft section 112 of the body 106.

[0012] Aircraft 100 is an example of an aircraft that may have a composite inspection model trained according to the illustrative examples. Aircraft 100 is an example of an aircraft having composite parts that may be inspected using an inspection system trained with synthetic data.

[0013] Referring now to FIG. 2 , a block diagram of data flow is shown in accordance with an illustrative embodiment. Three-dimensional model 200 is a model of at least a portion of a part. Three-dimensional model 200 may be a model of at least a portion of at least one of wing 102, wing 104, or body 106 of aircraft 100. Three-dimensional model 200 has any desired amount of layers. Three-dimensional model 200 has been generated based on a design of the part to be manufactured. In some illustrative examples, three-dimensional model 200 has the same design as a layup of the part. In some illustrative examples, by being based on the layup of the part, the curvature of three-dimensional model 200 is the same as the part to be manufactured.

[0014] The three-dimensional model 200 is used to generate synthetic data 202. The synthetic data 202 has sufficient resolution to be used to train the discrepancy identifier program 204. By training the discrepancy identifier program 204 using the synthetic data 202 instead of or in addition to using the inspection data 206, a greater amount of training data can be provided to the discrepancy identifier program 204. By training the discrepancy identifier program 204 using the synthetic data 202 instead of or in addition to using the inspection data 206, fewer resources can be used. In some illustrative examples, training the discrepancy identifier program 204 using the synthetic data 202 reduces time. In some illustrative examples, training the discrepancy identifier program 204 using the synthetic data 202 reduces manufacturing trial and error, thereby reducing manufacturing time. In some illustrative examples, training the discrepancy identifier program 204 using the synthetic data 202 reduces manufacturing trial and error, thereby reducing material waste.

[0015] By training the discrepancy identifier program 204 using synthetic data 202 instead of or in addition to using inspection data 206, less waste is generated. For example, if physical discrepancies are intentionally created for training, this would generate physical waste. As another example, by utilizing synthetic data 202, less rework is performed, generating less waste. By training the discrepancy identifier program 204 using synthetic data 202 instead of or in addition to using inspection data 206, the discrepancy identifier program 204 can be trained before manufacturing the part. By training the discrepancy identifier program 204 using synthetic data 202 instead of or in addition to using inspection data 206, training time can be reduced.

[0016] The composite data 202 comprises a plurality of captured images 208 and a plurality of secondary images 210. The plurality of captured images 208 are images taken of the three-dimensional model 200 in a computer simulation. The plurality of captured images 208 can comprise any desired view within the three-dimensional model 200.

[0017] The multiple captured images 208 generated from the three-dimensional model 200 can be used for any desired inspection type. For example, the multiple captured images 208 can be generated for optical inspection, infrared (IR) inspection, laser inspection, or any other desired type of inspection. To create the multiple captured images 208 for a given inspection type, at least one of the camera orientation, camera position, field of view, material properties of the three-dimensional model 200, color of the three-dimensional model 200, or other aspects of the three-dimensional model 200 can be modified based on the given inspection type. In some illustrative examples, at least one of the camera orientation, camera position, or field of view can be modified based on the respective inspection system. In some illustrative examples, the multiple captured images 208 include a variety of camera orientations, camera positions, and fields of view to encompass different inspection system positioning options for a single type of given inspection type.

[0018] The plurality of secondary images 210 are images generated from the plurality of captured images 208. In some illustrative examples, the plurality of captured images 208 are modified to form the plurality of secondary images 210. In some illustrative examples, a filter is applied to the plurality of captured images 208 to generate the plurality of secondary images 210. The plurality of secondary images 210 can be provided to the mismatch identifier program 204 as training data. In some illustrative examples, the resolution of the plurality of captured images 208 is modified to create the plurality of secondary images 210. In some illustrative examples, the coloring of the plurality of captured images 208 is modified to create the plurality of secondary images 210. In some illustrative examples, the brightness of the plurality of captured images 208 is modified to create the plurality of secondary images 210. In some illustrative examples, only selected regions within the plurality of captured images 208 are modified. In some illustrative examples, a portion of the plurality of captured images 208 is sampled to determine whether modification should be performed. This sampling allows pixel values ​​of the plurality of captured images 208 to be compared to pixel values ​​of a plurality of inspection images captured during manufacturing. In some illustrative examples, a plurality of inspection images captured during manufacturing by an inspection system of a desired inspection type are used to define a pixel value profile that is applied to the plurality of captured images 208 to create a plurality of secondary images 210. For example, pixel value normalization and min / max rescaling may be performed on the plurality of captured images 208 to match the plurality of inspection images collected during manufacturing. In some illustrative examples, a portion of the plurality of inspection images 212 may be used to define the pixel value profile.

[0019] In some illustrative examples, the three-dimensional model 200 is created such that the plurality of captured images 208 have desired pixel values ​​and desired coloring without additional modification. In some illustrative examples, the material features in the three-dimensional model 200 are set to have a predetermined inspection type coloring. In these illustrative examples, the plurality of secondary images 210 are optional. In these illustrative examples, the plurality of captured images 208 can be sent to the discrepancy identifier program 204 as training data.

[0020] After sufficient synthetic data 202 has been generated to train the discrepancy identifier program 204, the discrepancy identifier program 204 can be used to evaluate a plurality of inspection images 212. The plurality of inspection images 212 is inspection data 206 generated during the manufacturing of a part. During inspection, the plurality of inspection images 212 is evaluated by the discrepancy identifier program 204. The discrepancy identifier program 204 locates discrepancies in the plurality of inspection images 212. The discrepancy identifier program 204 identifies a type of discrepancy in the plurality of inspection images 212.

[0021] With reference now to Figure 3, an illustration of a block diagram of a manufacturing environment is shown in accordance with an illustrative embodiment. Manufacturing environment 300 is an environment in which portions of aircraft 100 may be manufactured. Three-dimensional model 336 is an example of three-dimensional model 200 from Figure 2.

[0022] Composite laying head 302 lays composite tows 304 to form portion 306 of part 308. Part 308 is a component of platform 310. Platform 310 may take any desired form, such as a vehicle, a building, or a watercraft. Platform 310 may take the form of a car, a boat, a spacecraft, a train, a bus, or any other desired type of vehicle. In this illustrative example, platform 310 takes the form of an aircraft 312.

[0023] The inspection system 314 is mounted to the composite laying head 302. The inspection system 314 has an inspection type 316 and a head direction 318. The inspection type 316 comprises a technology utilized for the inspection system 314. The inspection type 316 may include IR inspection, laser inspection, optical inspection, or any other desired type of inspection. The inspection type 316 has a coloring 320. The coloring 320 includes saturation, brightness, and other values ​​of the pixels of the inspection type 316.

[0024] Inspection system 314 generates a plurality of inspection images 322 as composite laying head 302 lays composite tow 304. For quality control of platform 310, the plurality of inspection images 322 are evaluated for the presence of discrepancies. A machine vision system desirably identifies and classifies discrepancies in the plurality of inspection images 322. In these illustrative examples, discrepancy identifier program 324 reviews the plurality of inspection images 322 for discrepancies 326.

[0025] The mismatch identifier program 324 is an example of a machine vision / machine learning algorithm. Machine vision / machine learning algorithms require a large amount of training images to build a model. In an automated fiber placement (AFP) application, it is difficult to "force" a situation from real-world data that would provide the amount and type of data to train the mismatch identifier program 324. Synthetic data 332 is generated to support quality model training.

[0026] The illustrative examples recognize and take into account that the training data may be highly configuration specific, and that changes made to the layup 339 of the portion 306 of the part 308 or the layout of the part 308 may result in the generation of an entire new data set.

[0027] If training data is collected only from a single direction, at a single speed, or when the background temperature is a particular value, this leaves gaps in the training data that make it difficult to apply the same code to the mismatch identifier program 324 across multiple machines and use cases. Synthetic data 332 can be used to fill these data gaps. In some illustrative examples, synthetic data 332 can be used to train the mismatch identifier program 324 without real-world data. In some illustrative examples, synthetic data 332 can be used to train the mismatch identifier program 324 in conjunction with real-world data images.

[0028] The mismatch identifier program 324 is a classification model trained using training data 328. The training data 328 comprises at least one of a plurality of captured images 358 or a plurality of secondary images 330. The plurality of captured images 358 and the plurality of secondary images 330 are synthetic data 332 generated from a three-dimensional model space 334.

[0029] The three-dimensional model space 334 may be used to create a three-dimensional model for generating the synthetic data 332. The three-dimensional model space 334 may be any desired type of three-dimensional modeling software.

[0030] The three-dimensional model 336 comprises a composite layup 338. The three-dimensional model 336 is a representation of the portion 306 of the part 308. The composite layup 338 is based on a layup 339 of the portion 306 of the part 308. In some illustrative examples, the three-dimensional model 336 of the composite layup 338 is generated using the layup 339 of the portion 306 of the part 308. In some illustrative examples, the plurality of plies 340 represent each of the plurality of plies 341 of the portion 306 of the part 308. In some illustrative examples, the layup 339 comprises the composite material 303 of the plurality of plies 341, and the three-dimensional model 336 comprises the plurality of plies 340 arranged identically to the composite material 303 of the plurality of plies 341.

[0031] In some illustrative examples, the plurality of plies 340 forms a curvature 343 of the surface 342 of the composite layup 338. In some illustrative examples, the plurality of plies 340 is designed such that the curvature 343 is identical to the curvature of the portion 306 of the part 308. In some illustrative examples, the plurality of plies 340 has fewer plies than the plurality of plies 341 of the portion 306 of the part 308.

[0032] The 3D model 336 can be modified if the design for the part 308 changes. By modifying the 3D model 336, training data 328 can be generated for the discrepancy identifier program 324 to take into account the changes to the design of the part 308.

[0033] To form the synthetic data 332, inconsistencies 344 are generated in the three-dimensional model 336 of the composite layup 338. The inconsistencies 344 may take any desired form. In some illustrative examples, the inconsistencies 344 are one of overlapping tows 346, foreign object debris 348, or gaps 350. In some illustrative examples, the inconsistencies 344 are one of pilling, resin clumps, foreign object debris 348, edge creases / kinks, heat damage, tows that are still tacked, or missing tows.

[0034] The discrepancy 344 may be generated in the three-dimensional model 336 depending on the type of discrepancy 344. For example, if the discrepancy 344 is a hairball, the discrepancy 344 may be constructed in the three-dimensional model 336 as multiple randomly oriented lines of varying length that pass through a reference center point.

[0035] If the discrepancy 344 is an edge crease, the discrepancy 344 identifies a distance along the outer toe edge, and the optimization problem min||Δx|| 2 ,Σ||xi-xi+1||=d A three-dimensional model 336 can be constructed by solving: where x is the list of points that form the curve along the outer tow edge, d is the arc length required for the edge, and Δ is the umbrella Laplacian for the points that averages the neighbors of x.

[0036] If the mismatch 344 is a twist, the mismatch 344 can be constructed into the 3D model 336 by removing a portion of the tow over some distance and creating a new geometry by connecting the lower tow boundary to the upper tow boundary and vice versa. If the mismatch 344 is a tow that is still tacked, the mismatch 344 can be constructed into the 3D model 336 by identifying a distance along the tow at the tow start or tow end and lifting the tow off the 3D part surface along a quadratic curve while maintaining the tow length.

[0037] If the discrepancy 344 is one of an overlapping tow 346 or a gap 350, the discrepancy 344 can be constructed in the three-dimensional model 336 by identifying a portion of the tow over some distance and shifting the tow centerline up or down some amount. If the discrepancy 344 is a missing tow, the discrepancy 344 can be constructed in the three-dimensional model 336 by identifying the tow in the three-dimensional model 336 and removing it from the simulated geometry.

[0038] The virtual camera 352 is a representation of a field of view in the three-dimensional model space 334. The virtual camera 352 has a camera position 354 and a camera orientation 356 in the three-dimensional model space 334. The camera position 354 can be modified based on at least one of the inspection type 316 or the mounting position of the inspection system 314 to the composite laying head 302. The camera position 354 can be modified to change the amount of the three-dimensional model 336 that is visible in the multiple captured images 358. The camera position 354 can be modified to change the section of the three-dimensional model 336 that is visible in the multiple captured images 358.

[0039] In some illustrative examples, a head orientation 318 of an inspection system 314 attached to a composite laying head 302 in the manufacturing environment 300 is determined, and a camera position 354 and a camera orientation 356 are set in the three-dimensional model space 334 by setting a camera orientation 356 based on the head orientation 318.

[0040] From the camera positions 354 and camera orientations 356, a captured image 360 ​​of the three-dimensional model 336 and the mismatch 344 is generated. The captured image 360 ​​is one of a plurality of captured images 358. In some illustrative examples, the plurality of captured images 358 are taken from two or more camera positions 354. In some illustrative examples, the plurality of captured images 358 are taken from two or more camera orientations 356. The plurality of captured images 358 has sufficient resolution to be used as training data 328 for the mismatch identifier program 324.

[0041] In some illustrative examples, composite layup 338 is formed with material properties 345 such that multiple captured images 358 can be used as training data 328. In these illustrative examples, material properties 345 cause multiple captured images 358 to have the desired coloration, brightness, and other characteristics of a preselected inspection type 366, such as inspection type 316.

[0042] In other illustrative examples, processing is performed on the multiple captured images 358 to form multiple secondary images 330. In these illustrative examples, the multiple secondary images 330 are used as training data 328. In some illustrative examples, a secondary image 362 is created from the captured image 360, the secondary image 362 having a coloration of a preselected inspection type 366. In some illustrative examples, the captured image 360 ​​can be processed to form multiple secondary images 330 for multiple different inspection types.

[0043] In some illustrative examples, a number of filters 364 are applied to the captured image 360 ​​to create a secondary image 362. By applying the number of filters 364 to the captured image 360, a secondary image 362 is created that has characteristics of the images of the preselected exam type 366.

[0044] In some illustrative examples, a number of filters 364 are applied to the captured image 360 ​​to create a preselected inspection type 366 coloration 368 for the captured image 360 ​​to create a secondary image 362. In some illustrative examples, a number of filters 364 are applied to the captured image 360 ​​to create an IR image coloration 368.

[0045] In some illustrative examples, application of the filters 364 to the captured image 360 ​​creates in pixels of the secondary image 362 with the mismatch 344 a mismatch-type specific coloring 368 for a preselected inspection type 366. For example, if the mismatch 344 is foreign object debris 348, application of the filters 364 to the captured image 360 ​​creates in pixels of the secondary image 362 with the mismatch 344 a mismatch-type specific coloring 368 for the foreign object debris 348 for the preselected inspection type 366. As a specific example, if the mismatch 344 is foreign object debris 348 and the preselected inspection type 366 is IR inspection 370, application of the filters 364 to the captured image 360 ​​creates in pixels of the secondary image 362 with the mismatch 344 a mismatch-type specific coloring 368 for the foreign object debris 348 for the IR inspection 370.

[0046] In some illustrative examples, creating the secondary image 362 includes sampling color from pixels of the discrepancy 344 in the captured image 360 ​​and adjusting the color value of the pixel to a value within the range of the discrepancy type of the discrepancy 344 in the stock image of the preselected inspection type 366. After training the discrepancy identifier program 324, the discrepancy identifier program 324 can be used to evaluate multiple inspection images 322 of the composite tow 304 after the composite tow 304 has been laid.

[0047] As shown, computer system 372 includes three-dimensional model space 334 and discrepancy identifier program 324. In other illustrative examples, three-dimensional model space 334 and discrepancy identifier program 324 reside on separate computer systems that are in communication with each other.

[0048] Generating training data 328 from a 3D model space 334 reduces at least one of training time, costs, and operator time. The 3D model 336 can be used to generate training data 328 for multiple inspection types. The 3D model 336 can be used to generate training data 328 for multiple driving speeds. The 3D model 336 can be used to generate training data 328 for multiple different camera orientations.

[0049] The synthetic data 332 can also provide thermal information if the preselected inspection type 366 is an IR inspection 370. If the synthetic data 332 does not include thermal information such as coloration 368, a model such as the discrepancy identifier program 324 relies solely on contrast differences to extract discrepancies. If thermal data such as coloration 368 is present in the synthetic data 332, a shift in overall temperature or a camera being used in auto gain mode can be accepted.

[0050] The addition of thermal information adds content to the synthetic data 332 that is provided to machine learning algorithms such as the mismatch identifier program 324. Including thermal information allows training of the mismatch identifier program 324 to take into account overall environmental temperature when looking at contrast differences as well as tracking gradual temperature changes that are not evident in simple contrast-only images.

[0051] The three-dimensional model space 334 enables a detailed, simulated data set to be generated. In some illustrative examples, synthetic data can be generated using the three-dimensional model space 334 prior to first building a physical component in the manufacturing environment 300. The three-dimensional model space 334 can include material systems, machine configurations, part configurations, layup plans, heater plans, camera information, camera fields of view, and other desired parameters that enable a highly detailed, simulated data set.

[0052] The illustration of manufacturing environment 300 in FIG. 3 is not meant to imply physical or architectural limitations to the manner in which an example embodiment may be implemented. Other components in addition to or in place of the illustrated components may be used. Some components may be unnecessary. Also, the blocks are presented to illustrate some functional components. One or more of these blocks may be combined, divided, or combined and divided into different blocks when implemented in an example embodiment.

[0053] For example, the multiple secondary images 330 may be optional. In some illustrative examples, material characteristics of the multiple layers 340 are established such that the multiple captured images 358 may be used as training data 328. In these illustrative examples, no filtering or other modification is performed to generate the multiple secondary images 330.

[0054] 4, a side view of composite material laying and inspection is shown in accordance with an illustrative embodiment. View 400 is a view of laying a composite material 402 onto a substrate 404. In some illustrative examples, the composite material 402 is in the form of a composite tow. An inspection system 406 is mounted to a composite laying head 408. As the composite laying head 408 lays the composite material 402, the inspection system 406 generates multiple inspection images of the composite material 402. The field of view 410 of the inspection system 406 is affected by the inspection type of the inspection system 406, the head orientation of the inspection system 406, and the mounting position relative to the composite laying head 408.

[0055] The inspection images generated by inspection system 406 are examples of images that may be inspected by the discrepancy identifier program 324 of Figure 3. Inspection system 406 is an example of a physical implementation of inspection system 314 of Figure 3.

[0056] With reference now to Figure 5, an illustration of a representation of a three-dimensional model is shown in accordance with an illustrative embodiment. Three-dimensional model space 500 is a representation of three-dimensional model space 334 from Figure 3. In three-dimensional model space 500, a three-dimensional model 502 of a composite layup 504 exists. A mismatch 506 exists on a surface 507 of three-dimensional model 502.

[0057] Camera 508 is a representation of a viewpoint within the three-dimensional model space 500. The field of view 510 of camera 508 is shown. Camera 508 has a camera position 512 and a camera orientation 514 within the three-dimensional model space 500. As shown, the camera position and orientation have six dimensions. The camera position 512 has three dimensions x, y, and z coordinates. The camera orientation 514 has three dimensions (pitch, roll, yaw).

[0058] Synthetic data for different parts can be created by modifying the three-dimensional model 502 or creating different three-dimensional models. Synthetic data for different types of discrepancies can be created by correcting the discrepancies 506. Synthetic data for different types of inspection systems can be created by modifying at least one of the camera position 512, the camera orientation 514, and the material properties of the three-dimensional model 502. In this manner, the three-dimensional model space 500 can be used to generate synthetic data for any part, any desired type of inspection system, and any desired position of the inspection system.

[0059] 6A and 6B, a flow chart of a method for generating synthetic data for composite manufacturing inspection is shown in accordance with an exemplary embodiment. The method 600 can be used to generate synthetic data 202 for training the discrepancy identifier program 204 of FIG. 2. The method 600 can be used to generate synthetic data for training the discrepancy identifier program 324 of FIG. 3. The method 600 can be used to generate synthetic data that can be used to train the discrepancy identifier program for analyzing inspection data from the inspection system 406 of FIG. 4. The method 600 can be implemented in the three-dimensional model space 500.

[0060] The method 600 creates a three-dimensional model of the composite layup in a three-dimensional model space (operation 602). The three-dimensional model has material characteristics that can be modified as desired.

[0061] The method 600 generates discrepancies in the three-dimensional model of the composite layup (operation 604). The discrepancies can have any desired shape. In some illustrative examples, the discrepancies are on a surface of the three-dimensional model. In some illustrative examples, the discrepancies are below the surface and within the layers of the three-dimensional model.

[0062] Method 600 sets a camera position and a camera orientation in the three-dimensional model space (operation 606). Method 600 generates a captured image of the three-dimensional model and the mismatch from the camera positions and camera orientations (operation 608). Method 600 creates a secondary image from the captured image, the secondary image having a coloring of a preselected inspection type (operation 610). The preselected inspection type is an inspection type for which the secondary image is used to train the mismatch identifier program. In some illustrative examples, the preselected inspection type may be one of an IR inspection, a laser inspection, an optical inspection, or another desired form of inspection. Method 600 then ends.

[0063] In some illustrative examples, method 600 determines a layup of the portion of the part (operation 612). The portion of the part can be any desired section of the part. The part can be any desired component. The layup information includes an order of composite plies, a type of composite, a width of the composite, a shape of each layer of composite, and other variables for laying down the composite to form the portion of the part. In some illustrative examples, method 600 generates a three-dimensional model of the composite layup using the layup of the portion of the part (operation 614). In these illustrative examples, the three-dimensional model resembles the portion of the part. In some illustrative examples, the layup comprises multiple layers of composite material, and the three-dimensional model comprises multiple layers arranged identically to the multiple layers of composite material (operation 616).

[0064] In some illustrative examples, the inconsistency is one of overlapping tows, gaps between tows, or foreign object debris (operation 618). In some illustrative examples, the inconsistency is one of pilling, resin clumps, foreign object debris, creased / kinked edges, heat damage, tows that are still tacked, or missing tows.

[0065] In some illustrative examples, method 600 determines a head orientation of an inspection system attached to a composite laying head in the manufacturing environment and setting a camera position and orientation in the three-dimensional model space includes setting the camera orientation based on the head orientation (Operation 620). In these illustrative examples, the captured images have a field of view substantially similar to the field of view of an image captured by the inspection system attached to the composite laying head.

[0066] In some illustrative examples, creating the secondary image includes applying a number of filters to the captured image to create the secondary image (OPERATION 622). In some illustrative examples, applying a number of filters to the captured image creates a coloration of the IR image (OPERATION 624).

[0067] In some illustrative examples, the discrepancy has a discrepancy type, and applying a filter to the captured image creates a discrepancy type-specific coloring for a preselected inspection type in pixels of the secondary image with the discrepancy (operation 626). In some illustrative examples, for the preselected inspection type, each discrepancy type can have an associated color identified from previous real-world collected data. For example, a hairball in the IR image may have an associated color for the pixels in the discrepancy. As another example, a missing tow in the IR image may have its own associated color for the pixels in the discrepancy.

[0068] In some illustrative examples, applying the filter to the captured image modifies pixels of mismatches in the captured image such that the pixels have a color associated with a type of mismatch, and in some illustrative examples, applying the filter to the captured image generates a secondary image with mismatches having a color associated with a type of mismatch for a preselected inspection type.

[0069] In some illustrative examples, creating the secondary image includes sampling color from the mismatched pixels in the captured image and adjusting color values ​​for the pixels to values ​​within a range of mismatch types for mismatches in the on-hand image for the preselected inspection type (operation 628). In some illustrative examples, for the preselected inspection type, each mismatch type may have a range of pixel values ​​identified from previous real-world collected data. For example, a hairball in the IR image may have a range for values ​​for pixels in the mismatch. As another example, a missing tow may have a range for values ​​for pixels in the mismatch. In some illustrative examples, creating the secondary image includes adjusting values ​​for pixels of the missing tow in the captured image to have values ​​within a range of values ​​for pixels sampled in the real-world collected data.

[0070] In some illustrative examples, method 600 provides the secondary images to a discrepancy identifier program as training data (OPERATION 630). In some illustrative examples, method 600 generates a plurality of inspection images of the composite tow after the composite tow is laid (OPERATION 632). In some illustrative examples, method 600 evaluates the plurality of inspection images using the discrepancy identifier program trained using the plurality of secondary images (OPERATION 634).

[0071] Referring now to Figure 7, a flow chart of a method for training a composite inspection system is shown in accordance with an exemplary embodiment. The method 700 can be used to generate synthetic data 202 for training the discrepancy identifier program 204 of Figure 2. The method 700 can be used to generate synthetic data for training the discrepancy identifier program 324 of Figure 3. The method 700 can be used to generate synthetic data that can be used to train the discrepancy identifier program for analyzing inspection data from the inspection system 406 of Figure 4. The method 700 can be implemented in the three-dimensional model space 500 of Figure 5.

[0072] The method 700 generates an image of a three-dimensional model of a composite layup with discrepancies at a predetermined camera position and a predetermined camera orientation in a three-dimensional model space that contains the three-dimensional model (operation 702). The method 700 modifies the image to form a secondary image having a preselected inspection type staining (operation 704). The method 700 trains a discrepancy identifier program using the secondary image (operation 706). The method 700 then ends.

[0073] Although operations 702-706 describe a single captured image and secondary image, in some illustrative examples, multiple composite images can be used to train the discrepancy identifier program. In some illustrative examples, operations 702-706 are performed iteratively before using the discrepancy identifier program to evaluate multiple inspection images from production.

[0074] In some illustrative examples, modifying the captured image includes applying a number of filters to the captured image to create a secondary image (OPERATION 712). In some of these illustrative examples, the number of filters are applied to provide inspection-type features to the secondary image.

[0075] In some illustrative examples, correcting the captured image includes sampling color from the mismatched pixels in the captured image and adjusting the color values ​​of the pixels to values ​​within the range of the mismatch type in the handheld image for the preselected inspection type (ACT 714). In these illustrative examples, real world data in the form of the handheld image provides information for correcting the captured composite image.

[0076] In some illustrative examples, method 700 inspects the composite tow as it is laid to form a plurality of inspection images (OPERATION 716). In some illustrative examples, method 700 evaluates the plurality of inspection images using a discrepancy identifier program trained using the plurality of secondary images (OPERATION 718).

[0077] In some illustrative examples, the composite tow is inspected using an inspection system and the predefined camera orientation in the three-dimensional model space is based on a head orientation of the inspection system (operation 720). The three-dimensional model can be photographed from multiple different predefined camera orientations for the inspection system at different head orientations and / or different inspection types.

[0078] In some illustrative examples, inspecting the composite tow includes generating an IR image of the composite tow after deposition (operation 722). Applying the composite tow includes heating the composite tow prior to laying the composite tow on a substrate. The IR image provides information regarding the heat distribution in the composite tow after it is placed on the substrate. The substrate can be in the form of a prior layer of the composite tow.

[0079] In some illustrative examples, method 700 determines a head orientation of an inspection system attached to a composite laying head in the manufacturing environment (OPERATION 708). In some illustrative examples, method 700 sets a predetermined camera position and a predetermined camera orientation in the three-dimensional model space based on the head orientation (OPERATION 710).

[0080] Referring now to Figure 8, a flow chart of a method for using synthetically generated training data is shown in accordance with an example embodiment. The method 800 can be used to generate synthetic data 202 for training the mismatch identifier program 204 of Figure 2. The method 800 can be used to generate synthetic data for training the mismatch identifier program 324 of Figure 3. The method 800 can be used to generate synthetic data that can be used to train the mismatch identifier program for analyzing inspection data from the inspection system 406 of Figure 4. The method 800 can be implemented in the three-dimensional model space 500 of Figure 5.

[0081] The method 800 generates discrepancies in a three-dimensional model comprising a composite layup of multiple plies (operation 802). The discrepancies can have any desired shape. In some illustrative examples, the discrepancies are on a surface of the three-dimensional model. In some illustrative examples, the discrepancies are below the surface and within the plies of the three-dimensional model.

[0082] The method 800 generates captured images of the discrepancies and the three-dimensional model at a predetermined camera position and a predetermined camera orientation based on a head orientation of an inspection system connected to the composite laying head (operation 804). By generating the captured images at the predetermined camera position and predetermined camera orientation, the captured images resemble the field of view of the inspection system in a manufacturing environment.

[0083] The method 800 generates a secondary image from the captured image, the secondary image having an inspection type coloring of the inspection system (operation 806). The secondary image is generated by modifying the captured image in any desired manner. The method 800 provides the secondary image as training data for a mismatch identifier program (operation 808). The method 800 then ends.

[0084] In some illustrative examples, the method 800 creates a three-dimensional model based on a layup of the portion of the part to be manufactured (operation 810). In some illustrative examples, the plies in the three-dimensional model represent each of the plies of the layup of the portion of the part. In these illustrative examples, the three-dimensional model may be a representation of all composite tows of portion 306 of part 308 in FIG. 3. In some illustrative examples, the plies in the three-dimensional model represent a section of the plies of the layup of the portion of the part. In some illustrative examples, the plies in the three-dimensional model are selected to create the same curvature as the portion of the part. In some illustrative examples, the plies in the three-dimensional model are selected such that a surface of the composite layup of the three-dimensional model represents a surface of the portion of the part.

[0085] In some illustrative examples, generating the inconsistency includes moving at least one toe representation in the three-dimensional model (ACT 812). In some illustrative examples, moving at least one toe representation in the three-dimensional model generates at least one of overlapping toes, gaps, missing toes, or other toe position inconsistencies.

[0086] In some illustrative examples, generating the secondary image includes applying a number of filters to the captured image to create IR image coloring (ACT 814). The presence of IR image coloring causes thermal information to be present in the secondary image. The addition of thermal information adds content to the image provided to the machine learning algorithm. Adding thermal information using coloring allows for additional inconsistencies to be identified. Adding thermal information using coloring allows for temperature changes to be identified.

[0087] As used herein, the phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the listed items may be used, and only one of each item in the list may be required. For example, "at least one of item A, item B, or item C" may include, without limitation, item A, item A and item B, or item B. This example may also include item A, item B, and item C, or item B and item C. Of course, any combination of these items may be present. In other examples, "at least one of" may be, for example, without limitation, two items A, one item B, and ten items C, or four items B and seven items C, or any other suitable combination. An item may be a particular object, thing, or category. In other words, at least one of means that any combination of items and number of items may be used from the list, but not all of the items in the list are required.

[0088] As used herein, "some" when used in reference to an item means one or more of the item.

[0089] The flowcharts and block diagrams in the different depicted embodiments illustrate the architecture, functionality, and operation of some possible implementations of apparatus and methods in the illustrative embodiments. In this regard, each block in the flowcharts or block diagrams may represent at least one of a module, a segment, a function, or a portion of an operation or step.

[0090] In some alternative implementations of the exemplary embodiments, one or more functions noted in the blocks may occur out of the order noted in the figures. For example, in some cases, two blocks shown in succession may be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. Also, other blocks may be added in addition to the blocks illustrated in the flowcharts or block diagrams. Some blocks may be optional. For example, operations 612-634 may be optional. As another example, operations 708-722 may be optional. As yet another example, operations 810-814 may be optional.

[0091] An exemplary embodiment of the disclosure may be described in the context of an aircraft manufacturing and service method 900 shown in Figure 9 and an aircraft 1000 shown in Figure 10. Turning initially to Figure 9, an illustration of an aircraft manufacturing and service method in block diagram form is shown in accordance with an exemplary embodiment. During pre-production, the aircraft manufacturing and service method 900 may include specification and design 902 and material procurement 904 of the aircraft 1000 in Figure 10.

[0092] During production, component and subassembly manufacturing 906 and systems integration 908 of the aircraft 1000 takes place. The aircraft 1000 may then go through certification and delivery 910 to become in service 912. While in service 912 by a customer, the aircraft 1000 is scheduled for routine maintenance and service visits 914, which may include modification, reconfiguration, refurbishment, or other maintenance and service visits.

[0093] Each of the processes of aircraft manufacturing and service method 900 may be performed or implemented by a system integrator, a third party, and / or an operator. In these examples, the operator may be a customer. For purposes of this description, a system integrator may include, without limitation, any number of aircraft manufacturers and major system subcontractors, a third party may include, without limitation, any number of vendors, subcontractors, and suppliers, and an operator may be an airline, a leasing company, a military entity, a service organization, etc.

[0094] Referring now to Figure 10, an illustration of an aircraft in block diagram form is shown in which an illustrative embodiment may be implemented. In this example, aircraft 1000 is produced by aircraft manufacturing and service method 900 of Figure 9 and may include an airframe 1002 with a number of systems 1004 and an interior 1006. Example systems 1004 include one or more of a propulsion system 1008, an electrical system 1010, a hydraulic system 1012, and an environmental system 1014. Any number of other systems may be included.

[0095] Apparatus and methods embodied herein may be employed during at least one of the stages of aircraft manufacturing and service method 900. One or more illustrative embodiments may be manufactured or used during at least one of component and subassembly manufacturing 906, system integration 908, in-service 912, or maintenance and service 914 of FIG.

[0096] An illustrative example provides a method for generating synthetic data for training a mismatch identifier. In an illustrative example, the synthetic data is generated using a view of a three-dimensional model. Synthetic data for multiple different inspection types can be generated from the same three-dimensional model. The three-dimensional model can be modified based on design changes to the part.

[0097] In some illustrative examples, the synthetic data may be synthetic infrared data, which can improve and increase the feasibility of implementing IR inspections for AFP surveillance as well as general infrared assessment.

[0098] The addition of thermal data to the "contrast only" synthetic data provides context for infrared inspection. In some illustrative examples, the synthetic image is constructed with a visible light simulation with thermal data added in post-processing. The addition of thermal data to the synthetic data can be as simple as setting maximum and minimum temperature values ​​to bound a color palette or as complex as adding algorithms to account for the natural dissipation of heat as it flows through a material and is lost via convection / conduction. The addition of thermal data can be accomplished by setting material properties in the three-dimensional model. The addition of thermal data can be accomplished through processing the captured image to form a secondary image.

[0099] An illustrative example provides a virtual representation of a part that considers at least one of part layup, machine, material, and layup path. The virtual representation of the manufacturing environment allows for the simulation of multiple part configurations, camera positions, and layup programs / velocities.

[0100] The description of different exemplary embodiments is presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the disclosed forms of the embodiments. Many modifications and variations will be apparent to those skilled in the art. Furthermore, different exemplary embodiments may provide different forms compared to other exemplary embodiments. The selected one or more embodiments have been chosen and described to best explain the principles, practical applications of the embodiments, and to enable those skilled in the art to understand the present disclosure in terms of the various embodiments with various modifications suitable for the particular use contemplated. [Explanation of symbols]

[0101] 100 aircraft, 102 wing, 104 wing, 106 body, 108 engine, 110 engine, 112 tail section, 114 horizontal stabilizer, 116 horizontal stabilizer, 118 vertical stabilizer, 200 three-dimensional model, 202 synthetic data, 204 discrepancy identifier program, 206 inspection data, 208 multiple captured images, 210 multiple secondary images, 212 multiple inspection images, 300 manufacturing environment, 302 composite laying head, 303 composite material, 304 composite tow, 306 part, 308 part, 310 platform, 312 aircraft, 314 inspection system, 316 inspection type, 318 head orientation, 320 staining, 322 multiple inspection images, 324 discrepancy identifier program, 326 multiple discrepancies, 328 training data, 330 multiple secondary images, 332 synthetic data, 334 3D model space, 336 3D model, 338 composite layup, 339 layup, 340 multiple layers, 341 multiple layers, 342 surface, 343 curvature, 344 inconsistency, 345 material properties, 346 overlapping tows, 348 foreign debris, 350 gaps, 352 virtual camera, 354 camera position, 356 camera orientation, 358 multiple captured images, 360 captured image, 362 secondary image, 364 several filters, 366 preselected inspection type, 368 staining, 370 IR inspection, 372 computer system, 400 view, 402 composite material, 404 substrate, 406 inspection system, 408 composite laying head, 410 field of view, 500 3D model space, 502 3D model, 504 Composite layup, 506 Mismatch, 507 Surface, 508 Camera, 510 Field of view, 512 Camera position, 514 Camera direction, 600 Method, 602 Action, 604 Action, 606 Action, 608 Action, 610 Action, 612 Action, 614 Action, 616 Action, 618 Action, 620 Action, 622 Action, 624 Action, 626 Action, 628 Action, 630 Action, 632 Action, 634 Action, 700 Method, 702 Action, 704 Action, 706 Action, 708 Action, 710 Action, 712 Action, 714 Action, 716 Action, 718 Action, 720 Action, 722 Action, 800 Method, 802 operation, 804 operation, 806 operation, 808 operation, 810Operation, 812 Operation, 814 Operation, 900 Manufacturing and maintenance methods, 902 Specification and design, 904 Materials procurement, 906 Component and subassembly manufacturing, 908 System integration, 910 Certification and delivery, 912 In-service, 914 Maintenance and maintenance, 1000 Aircraft, 1002 Airframe, 1004 Systems, 1006 Interior, 1008 Propulsion systems, 1010 Electrical systems, 1012 Hydraulic systems, 1014 Environmental systems

Claims

1. 1. A method (600) for generating synthetic data (332) for composite manufacturing inspection, comprising: creating (602) a three-dimensional model (336) of a composite layup (338) in a three-dimensional model space (334); generating (604) a mismatch (344) in the three-dimensional model (336) of the composite layup (338); setting (606) a camera position (354) and a camera orientation (356) in the three-dimensional model space (334); generating (608) captured images (360) of the three-dimensional model (336) and the mismatch (344) from the camera positions (354) and the camera orientations (356); creating (610) a secondary image (362) from the captured image (360), the secondary image (362) having a coloring (368) of a preselected examination type (366); The method (600) comprising:

2. The method (600) of claim 1, wherein the inconsistency (344) is one of overlapping tows (346), a gap between tows (350), or foreign object debris (348).

3. determining (612) a layup (339) of a portion (306) of a part (308); generating (614) the three-dimensional model (336) of the composite layup (338) using the layup (339) of the portion (306) of the part (308); 10. The method of claim 1, further comprising:

4. 4. The method (600) of claim 3, wherein the layup (339) comprises a plurality of plies (341) of composite material (303), and the three-dimensional model (336) comprises a plurality of plies (340) co-located with the plurality of plies (339) of composite material (303).

5. 2. The method (600) of claim 1, wherein creating the secondary image (362) comprises applying (622) a number of filters (364) to the captured image (360) to create the secondary image (362).

6. 6. The method (600) of claim 5, wherein applying a number of the filters (364) to the captured image (360) produces (624) a coloration (368) of the IR image.

7. The method (600) of claim 5, wherein the discrepancy (344) has a discrepancy type, and the step of applying some of the filters (364) to the captured image (360) creates (626) a coloring specific to the discrepancy type for a preselected inspection type (366) within pixels of the secondary image (362) with the discrepancy (344).

8. 2. The method (600) of claim 1, wherein creating the secondary image (362) includes sampling color from pixels of the discrepancy (344) in the captured image (360) and adjusting color values ​​for the pixels to a value within a range of a discrepancy type of the discrepancy (344) in a stock image for the preselected inspection type (366).

9. 2. The method (600) of claim 1, further comprising determining a head orientation (318) of an inspection system (314) attached to a composite laying head (302) in the manufacturing environment (300), and wherein setting the camera position (354) and the camera orientation (356) in the three-dimensional model space (334) comprises setting (620) the camera orientation (356) based on the head orientation (318).

10. providing (630) said secondary images (362) as training data (328) to a mismatch identifier program (324); 10. The method of claim 1, further comprising:

11. generating (632) a plurality of inspection images (322) of the composite tow (304) after the composite tow (304) has been laid; evaluating (634) the plurality of test images (322) using the mismatch identifier program (324) trained using the plurality of secondary images (330); 11. The method (600) of claim 10, further comprising:

12. A method (700) for training a composite inspection system (314), comprising: generating (702) an image (360) of a three-dimensional model (336) of a composite layup (338) with a mismatch (344) at a predetermined camera position (354) and a predetermined camera orientation (356) in a three-dimensional model space (334) containing the three-dimensional model (336); Modifying (704) the captured image (360) to form a secondary image (362) having a preselected inspection type (366) coloration; training (706) a mismatch identifier program (324) using the secondary images (362); The method (700) comprising:

13. inspecting (716) the composite tow (304) as it is laid to form a plurality of inspection images (322); evaluating (718) the plurality of test images (322) using the mismatch identifier program (324) trained using the plurality of secondary images (330); 13. The method (700) of claim 12, further comprising:

14. 14. The method (700) of claim 13, wherein the composite tow (304) is inspected using an inspection system (314), and the predetermined camera orientation (356) in the three-dimensional model space (334) is based (720) on a head orientation (318) of the inspection system (314).

15. 14. The method (700) of claim 13, wherein inspecting the composite tow (304) comprises generating an IR image (722) of the composite tow (304) after deposition.

16. 13. The method (700) of claim 12, wherein modifying the captured image (360) comprises applying (712) a number of filters (364) to the captured image (360) to create the secondary image (362).

17. 13. The method (700) of claim 12, wherein modifying the captured image (360) includes sampling color from a pixel of the mismatch (344) in the captured image (360) and adjusting a color value of the pixel to a value within a range of a mismatch type of the mismatch (344) in an image of the preselected inspection type (366).

18. Determining (708) a head orientation (318) of an inspection system (314) attached to a composite laying head (302) in the manufacturing environment (300); setting (710) a predetermined camera position (354) and a predetermined camera direction (356) in the three-dimensional model space (334) based on the head direction (318); 13. The method (700) of claim 12, further comprising:

19. 1. A method (800) of using synthetically generated training data (328), comprising: generating (802) a mismatch (344) in a three-dimensional model (336) comprising a composite layup (338) of a plurality of plies (340); generating (804) captured images (360) of the discrepancy (344) and the three-dimensional model (336) at a predetermined camera position (354) and a predetermined camera orientation (356) based on a head orientation (318) of an inspection system (314) connected to the composite laying head (302); generating (806) a secondary image (362) from the captured image (360), the secondary image (362) having an inspection type coloring of the inspection system (314); providing (808) said secondary image (362) as training data (328) for a mismatch identifier program (324); The method (800) comprising:

20. 20. The method (800) of claim 19, wherein generating (814) the secondary image (362) includes applying a number of filters (364) to the captured image (360) to create a coloration of the IR image.

21. Creating (810) a three-dimensional model (336) based on a layup of the portion (306) of the part (308) to be manufactured.

20. The method (800) of claim 19, further comprising:

22. 20. The method (800) of claim 19, wherein generating the discrepancy (344) comprises displacing (812) at least one toe representation in the three-dimensional model (336).