Monitoring of composite part manufacturing

Abstract

A manufacturing monitoring system (200) and a method thereof for monitoring the manufacture of composite material parts are disclosed. The composite material parts are made of a plurality of structural elements (220) laid in a mold (210) in a predetermined order, and each structural element (220) has an identifier (224). Each identifier has a plurality of symbols (225a, 225b), which are different in shape and in at least one of position and orientation on the corresponding structural element (220). The system (200) includes at least one image sensor (120) and a monitoring subsystem. The at least one image sensor (120) is configured to acquire image data of the identifier when the structural element (220) is laid into the mold (210). The monitoring subsystem is configured to: receive image data from at least one image sensor (120); determine from the image data whether the structural element (220) in the mold (210) is correct by detecting an identifier (224) on the structural element (220) from the image data and comparing the detected identifier (224) with an expected identifier (220) obtained from a predetermined sequence; and output an indication of whether the structural element (220) is correct.

Description

Technical Field

[0001] The present invention relates to the monitoring of the manufacture of components made of composite materials, such as fiber-reinforced composite materials. Background Technology

[0002] As is known in the art, the manufacture of parts made of composite materials involves laying multiple ply sheets within a mold to shape the composite part into the desired form. The ply sheets are typically sheets made of woven fibers or unidirectional fibers, such as glass fibers or carbon fibers. These ply sheets can be pre-impregnated with a resin material, or once placed in the mold, the resin material can be infused through the ply to form a fiber-reinforced composite part. Techniques like these are widely used in the manufacture of large-scale components, such as wind turbine blades, aircraft structures, or marine components. In some applications, the layers of the ply sheets are separated by individual powder layers, and the entire mold is then heat-treated to produce the final composite part. In other applications, the resin is pre-impregnated with the fibers before curing in an oven or autoclave.

[0003] When manufacturing small composite parts, each ply sheet can be identical or easily distinguishable, making the layering of all ply sheets in the mold relatively simple. However, for large-scale parts, especially those with complex fiber orientation and shape requirements, each ply sheet must be laid in the mold in a specific location and sequence to achieve the desired shape and withstand structural loads. Ply sheets in different locations within the part can also have different shapes and sizes. The dimensions of these ply sheets make it difficult for personnel to visually identify different ply sheets or to manually check their correct positioning in the mold, especially since layups are often placed with the assistance of cranes. In such a manufacturing environment, each ply sheet needs to be individually identifiable to aid the layup process. This means that producing large parts with composites can be a labor-intensive process, requiring highly skilled and experienced operators.

[0004] One option to address this difficulty might be to add Quick Response (QR) codes, barcodes, or human-readable characters (i.e., numbers or letters) to the ply for part identification. However, these rely on precise identification at the local level of the ply, which the applicant has recognized can prove difficult or impractical for large-scale identification. For example, to read a QR code from 10m away, the QR code would need to be approximately 1m long. 2 Furthermore, the entire QR code needs to be visible. In large-scale manufacturing applications, even placing the camera 10m away from the laying area may not be feasible; instead, the camera may need to be placed approximately 20m away from where the lay-up sheet is inserted into the mold, above the operating height of the crane moving the lay-up sheet. Summary of the Invention

[0005] According to a first aspect of the invention, a manufacturing monitoring system is provided for monitoring the manufacture of a composite material component made of a plurality of structural elements laid in a mold in a predetermined order, each of the plurality of structural elements having an identifier; wherein each identifier includes a plurality of symbols, the plurality of symbols being different in shape and at least one of position and orientation on the respective structural element; wherein the system includes: at least one image sensor configured to acquire image data of the identifier when the structural element is laid in the mold; and a monitoring subsystem configured to: receive image data from the at least one image sensor; determine whether the structural element in the mold is correct by detecting the identifier on the structural element from the image data and comparing the detected identifier with a expected identifier obtained from the predetermined order; and output an indication regarding whether the structural element is correct.

[0006] According to another aspect of the present invention, a method for monitoring the laying of a composite material component is provided, the composite material component being made of a plurality of structural elements laid in a mold in a predetermined order, each of the plurality of structural elements having an identifier; wherein each identifier includes a plurality of symbols, the plurality of symbols being different in shape and at least one of position and orientation on the respective structural element; wherein the method includes: receiving image data of the identifier from at least one image sensor; determining whether the structural elements in the mold are correct from the image data by the steps of: detecting the identifier on the structural element from the image data; comparing the detected identifier with a expected identifier obtained from the predetermined order; and outputting an indication of whether the structural element is correct.

[0007] In another respect, the present invention provides a computer software product and a non-transitory computer-readable medium, the non-transitory computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the methods outlined above.

[0008] Therefore, those skilled in the art will see that, according to the present invention, it is possible to reliably monitor the placement of structural elements into the mold using appropriate identifiers to help confirm that the correct structural elements are being placed into the mold. The applicant has recognized that, when compared with other technologies such as QR codes or barcodes, the use of identifiers comprising multiple symbols with different shapes, positions, and / or orientations provides stable identification.

[0009] Ensuring that structural components are placed in the correct order makes the laying process more consistent, thus reducing the risk of component failure caused by incorrectly laid components. While components made from inconsistent laying processes can be detected during product testing, eliminating errors early in the manufacturing process also reduces overall product waste. The reduction in potential differences between components also contributes to improved overall product quality. By influencing the accuracy and time spent on the laying process, and by reducing the time wasted on remanufacturing components or relaying structural components, the entire manufacturing process can be made more efficient, thereby reducing the overall time cost of the process. It also allows for a reduction in the highly trained requirements for personnel operating the process.

[0010] It should be understood that structural elements can be any element used in the production of parts made of composite materials (i.e., composite components). Typically, such structural elements themselves will not possess any structural rigidity. For example, structural elements can be ply sheets (e.g., woven fiber sheets). Structural elements can be homogeneous materials, metal meshes, or even core materials. In the manufacture of some components, the layup process involves the layer-by-layer laying of elements already formed from multiple ply sheets, rather than laying single layers of ply sheets or anything other than single layers of ply sheets. Therefore, in some embodiments, structural elements can be prefabricated elements (e.g., structural elements made from multiple ply sheets previously cured together) that can possess a certain degree of structural rigidity. A wide variety of different structural elements can be used in the production of composite components. For a given structural element, performing a critical structural function in the finished composite component is not necessarily required. For example, it can provide useful thermal properties, electrical properties, or even aesthetic properties.

[0011] Composite material components can be any component made from a combination of different materials or layers. It should be understood that composite material components typically include other elements or materials (such as resins or powders) in addition to structural elements, as is well known to those skilled in the art.

[0012] During manufacturing, each structural element is laid into a mold to form the entire part, as is well known to those skilled in the art. Each structural element may have a different shape and size, or some structural elements may be identical throughout the part. Typically, each structural element needs to be laid in the mold in a specific order, position, and orientation to form the entire composite part. The laying process is established when designing the part prior to manufacturing. While in some embodiments the invention can be applied to the placement of a single structural element as part of the manufacturing process, the invention is generally most useful in the laying of multiple structural elements. Therefore, in some embodiments, the system is configured to look up the next identifier in a predetermined order after the correct structural element has been detected and an indication has been output. Thus, the monitoring system is able to look up each consecutive identifier in the predetermined order in order to monitor the entire laying process.

[0013] Because the order of the structural elements is predetermined, the corresponding set of identifiers used on the structural elements is also predetermined. This significantly simplifies the task of performing machine vision recognition, and in one set of embodiments, the monitoring system can be arranged to look up identifiers only from the predetermined set of identifiers.

[0014] Furthermore, given that even for large, complex components, the number of different structural elements may only be on the order of a few hundred, the degrees of freedom provided by having multiple symbols and multiple possible positions and / or orientations mean that the symbols themselves can be very simple, thus avoiding the need for high-resolution images capable of distinguishing minute changes from one identifier to the next. In fact, given that the monitoring subsystem can be configured with the order in which the structural elements should be laid out, and therefore with information about where each identifier is within the predetermined order, it can even sort the identifiers to enhance the differentiation between one identifier and the next in the sequence.

[0015] A predefined identifier set may include ten or fewer different symbol shapes, and the number of discrete orientations and positions in which they can be placed will be relatively limited. Therefore, the theoretical size of possible identifier sets in practice is relatively limited (e.g., on the order of several thousand), which contrasts with the virtually unlimited number of potential QR codes.

[0016] The monitoring subsystem can be configured to detect individual symbols from the set of symbols constituting the identifier, and then associate combinations of these symbols with a predefined set of identifiers. Alternatively, the complete identifier can be detected and compared directly with the expected identifier in a predetermined order or more broadly with the predefined set of identifiers. By having spatial requirements for the symbols of each identifier, the probability of misclassifying a correct structural element as an incorrect one is reduced. The position of a symbol can be determined by its position relative to the edge of the structural element or other visually significant features, and in some embodiments, the position of the symbol can also be determined relative to the layup area in the mold. The orientation of the symbol can be an angular position relative to a visually significant feature of the structural element, and / or an orientation relative to the layup area in the mold. If the structural element is symmetrical about any axis, the identifier can be symmetrical about the same axis, so the structural element can be placed in the mold with any correct orientation. In some composite parts, multiple structural elements of the same type may exist throughout the sequence, and these can all be identified from the same identifier.

[0017] It should be understood that the order in which structural elements need to be laid into the mold depends on various requirements of the component, as discussed above. Within a composite component, depending on the structural requirements of the component, portions of structural elements may overlap with one or more other structural elements to varying degrees. In some cases where there is only a small overlap between two given structural elements, it may be considered optimal for the overall structure that the upper structural element is placed into the mold before the lower structural element, despite the small overlap. In this case, the portion of the upper structural element that overlaps with the lower structural element (e.g., a protruding flange or end of the upper structural element) may need to be temporarily removed, for example by folding, to allow for the placement of the lower structural element. However, it is clearly important that once the lower element is placed, the portion of the upper structural element that overlaps with the lower structural element is returned to its original position. Thus, in one set of embodiments, multiple structural elements include upper and lower structural elements, with the upper structural element preceding the lower structural element in a predetermined order, wherein an overlap symbol is provided on either the upper or lower structural element, the presence or absence of which indicates whether the portion of the upper structural element that overlaps with the lower structural element has been returned to its original position after the lower structural element has been placed into the mold.

[0018] Determining whether a lower structural element is correctly placed can include detecting the presence or absence of overlapping symbols. That is, if an overlapping symbol is on an upper structural element, then detecting the overlapping symbol indicates that the lower structural element is correctly placed; and if an overlapping symbol is on a lower structural element, then not detecting the overlapping symbol indicates that the lower structural element is correctly placed. This allows the system to adapt to the situations described above, thus providing greater flexibility when dealing with more complex composite structures and sequences.

[0019] In one set of such embodiments, an overlap symbol is placed on the portion of the lower structural element that overlaps with the upper structural element. The identifier of the lower structural element can be defined in such a way that the absence of the overlap symbol indicates that the lower structural element is correctly placed. Alternatively, a separate follow-up check step can be performed to determine that the overlap symbol is not visible. If needed, this would allow the overlap symbol to perform additional functions as part of the normal identifier.

[0020] Therefore, in some embodiments, determining whether the lower structural element has been correctly placed is a two-step process. The first step includes determining whether the lower structural element itself is correct (correct element, correct position, etc.), and the second step includes determining whether the overlapping portion of the upper structural element has been correctly returned (i.e., once the lower structural element has been placed as discussed above, the overlapping portion of the upper structural element has been moved back to the top of the lower structural element). This two-stage check helps improve the efficiency and accuracy of the components, thereby ensuring that the overlapping portion is not returned until it has been confirmed that the overlapping portion is being placed on the correct lower structural element.

[0021] In another set of embodiments (not necessarily mutually exclusive), overlapping symbols are placed on the overlapping portions of the upper structural elements, and a separate inspection step is performed to determine whether the overlapping symbols are visible.

[0022] According to some embodiments, the system is configured to check whether structural elements are within a predetermined tolerance relative to a given metric. It should be understood that for tolerance checks, the image data may need to have sufficiently high resolution to allow precise measurement of the position and / or orientation of each symbol within the identifier relative to the element, a predefined laying area, or the mold. If all symbols of the identifier are detected to be within a predetermined tolerance compared to the expected position, the structural element can be determined to be correct. The tolerance may allow one or more of the following: angles relative to the nominal direction, shear angles, local deformation, deformation of the entire structural element, wrinkle formation, tilting of the structural element in the mold, etc. Data corresponding to the precise position and / or orientation of each symbol within the identifier can be acquired and analyzed. For example, data corresponding to the precise position and / or orientation of at least one symbol can be used to determine local deformation of the structural element due to wrinkles or other defects.

[0023] Different errors in different symbols within the identifier can represent different problems with structural elements or their placement. When monitoring problems in individual structural elements, the position of each symbol within each identifier determines which metric can be detected. Therefore, to monitor errors in individual structural elements, the symbols are preferably spaced apart throughout the structural element. In this way, wrinkles in individual structural elements can be detected, and local deformation and overall sheet deformation can be measured. For example, if a structural element has six symbols in its identifier, and if all symbols are of the correct shape, but only one symbol is in the wrong orientation, it can be determined that there is a local shear or wrinkle near that symbol. This is particularly advantageous when a single ply sheet or other thin and flexible structural element is being placed into a mold. Traditionally, wrinkles in sheets, especially when the sheet is light-colored, are very difficult to see. However, the variation in the distance between symbols in the identifier according to the invention can be used more easily to identify whether the sheet has wrinkled while being laid into the mold. The user can then be quickly notified of the problem, and the user can adjust the laid element to eliminate the wrinkles.

[0024] In some embodiments, the monitoring subsystem is configured to perform a two-stage determination: a first stage includes determining that the structural elements are in the correct order relative to a predetermined sequence; and a second stage includes determining that the structural elements are correctly positioned within the mold by the fact that the symbols of the structural elements are positioned within one or more predetermined tolerances of the expected locations.

[0025] While in some manufacturing environments, structural elements may be laid one at a time, in other environments, multiple structural elements may be laid adjacent to each other to form each layer of the finished composite component. Therefore, in some embodiments, one or more image sensors are configured to monitor the area where all structural elements may be laid. The monitoring subsystem is capable of detecting multiple identifiers from the image data.

[0026] In a predetermined sequence, any structural element belonging to the same layer of a component can be determined as correct when placed in any order, as long as they are placed in the correct position. In some embodiments, the system may wait until all structural elements in a given layer have been placed before determining that layer as correct. In another set of embodiments, the system may instead determine each structural element as correct. By allowing flexibility within a predefined sequence when multiple structural elements are part of the same layer, unnecessary “incorrect” structural elements will not be identified. This also means that, when laying is permitted, the user has flexibility in what order which elements are placed in a given layer, thereby increasing both ease of use and the efficiency of the laying procedure.

[0027] In some embodiments, a structural element can be considered correct as long as it is the next expected structural element to be placed in the mold relative to a predetermined order. However, in some embodiments, determining whether a structural element is correct also includes determining whether the structural element has been placed in the correct position within the mold—for example, by comparing the detailed position of one or more symbols of an identifier with one or more expected detailed positions. In these embodiments, information is thus provided to the monitoring subsystem regarding where the identifier(s)(s) should be located relative to the mold laying area.

[0028] In some embodiments, the system is arranged to provide an indication of a desired structural element—preferably including an identifier associated with the desired structural element from a predetermined order. This indication may be provided only when an incorrect structural element is detected, but in some embodiments, it is provided before determination. The indication of the desired structural element may include information about the identifier associated with the desired structural element. This indication may be provided on a user interface, such as a display screen, but in some (potentially overlapping) embodiments, the system includes at least one projector configured to project an image of the desired identifier for the correct structural element in the predetermined order onto a mold. Thus, the projected image of the identifier can be used to help place the correct structural element in the mold, thereby improving the efficiency of the laying process. In some embodiments, the indication of whether a structural element is correct includes the color of the projected image. In some embodiments, the color of the projected image is arranged to change when a structural element has been determined to be correct; that is, the projected image may include an indication of whether a structural element is correct, provided by a monitoring subsystem according to the invention. This simple visual indicator provides a clear confirmation to the user, allowing them to proceed to place the next structural element, thereby improving the ease of the laying process.

[0029] In some embodiments, when a structural element is determined to be incorrect, the indication in the output of the monitoring subsystem includes information about when the structural element was placed in a predetermined order. This can help guide the user and / or allow structural elements that are about to be needed to be temporarily stored near the mold, rather than being returned to the storage area.

[0030] In some embodiments that also monitor the detailed location of structural components within the mold, when the location of a structural component is determined to be incorrect, the indication in the output of the monitoring subsystem includes information about the correct positioning of the structural component. This allows the user to ensure that it is moved to the correct location. This can reduce the number of errors during the laying process.

[0031] When monitoring the placement of each structural element in a predetermined sequence, it is anticipated that the structural element will move across the field of view of one or more image sensors as it is moved into place. Therefore, in some embodiments, a determination of whether a structural element is correct is made after at least one of the following: a predetermined time; or an identifier has been detected exceeding a predetermined time; or an identifier has been detected exceeding a predetermined number of image frames. This way, although the structural element is still moved into place, the system will not output a negative result, thereby improving the accuracy of the monitoring system. The predetermined time can be the duration after a previous determination of a correct structural element, and / or the duration after user input. User input can be at least one of the following indications: an indication that the placement process has started, an indication that the sequence should proceed to the next step in the predetermined sequence, and / or any other suitable progress determination.

[0032] In other embodiments, the system is configured to make a determination on whether a structural element is correct only after the user provides input indicating that the structural element is considered ready.

[0033] To facilitate the output of instructions and / or any input from the user, in some embodiments, the system includes a user interface configured to display to the user instructions from the monitoring subsystem regarding the correctness of structural elements.

[0034] In many manufacturing environments, the laying area is surrounded by numerous large pieces of equipment, including cranes for lifting each structural element into place in a mold. In such environments, these pieces of equipment may obstruct some image sensors, preventing the acquisition of complete image data of the laying area. In such cases, the equipment may obscure some or all of the symbols that constitute an identifier. Therefore, in some embodiments, the monitoring subsystem is arranged to determine that a structural element is correct if at least a predetermined threshold percentage of the identifier has been determined to be correct compared to the expected identifier. The predetermined threshold percentage may be, for example, 80%. However, similarly, different thresholds may be used depending on the number and diversity of identifiers in the predetermined group. In some embodiments, an identifier is determined to be correct if a sufficient portion of the identifier has been detected to exclude the identifier from being any other identifier in the predetermined group of identifiers. It should be understood that, in this case, a partially obscured symbol can still be used to identify the identifier; that is, even when partially obscured, the shape of the symbol can be distinguished from other symbols, for example, a portion of a circle can always be distinguished from a portion of a triangle. In some embodiments, the system may be configured to allow an authenticated user to provide input to identify a structural element as correct.

[0035] It should be understood that image data can be acquired from a single sensor or from multiple sensors simultaneously. One or more image sensors can be cameras. One or more image sensors can be configured to monitor the entire mold, or they may only need to be able to image the portion of the mold where the structural elements will be laid (i.e., the laying area). In the case of multiple image sensors, they can each cover the entire monitoring area, or all or some image sensors may cover only a portion of the monitoring area. If one image sensor is obstructed by other devices, it may be advantageous to have multiple image sensors positioned at different angles to observe the laying area, as another sensor can provide data to locate identifiers.

[0036] In some embodiments, the monitoring subsystem includes a locator section configured to determine a detection area by searching for any candidate symbols throughout the image data. When a candidate symbol is located, the monitoring subsystem is configured to determine the shape, position, and / or orientation of the symbol. Using the locator section reduces the time spent detecting identifiers and determining whether an identifier is correct or incorrect by performing a low-resolution scan of the data before performing a more detailed search for the detailed shape, position, and / or orientation of the detected symbols. If the position of the symbol is determined to be incorrect, or if no symbol is detected in the detection area, the monitoring subsystem can immediately output an indication that the structural element is incorrect. This eliminates the need for a second-stage analysis, allowing for faster notification to the user that the structural element is incorrect, which in turn improves the efficiency of the manufacturing process.

[0037] In another set of embodiments, the locator portion is configured to determine detection regions in the image data, where one or more symbols in the identifier should be located for the structural elements expected to be correctly positioned in a predetermined order. Since the position of the symbol on the structural element is a parameter of the identifier, the locator portion allows the search for expected symbols requiring detailed imaging for detecting the shape (and possible orientation) of the symbols to be narrowed down to the corresponding detection regions. Therefore, the monitoring subsystem can be configured to detect the identifier by searching for symbols within the detection regions. One or more image sensors may only be required to provide image data of the detection regions. The monitoring subsystem can be configured to perform only a low-resolution analysis of the detection regions (one or more) before proceeding to perform more detailed analysis to identify the shape or location (including orientation) of the symbols, to determine only whether the symbol is present in any case.

[0038] The number of symbols in a given identifier can be chosen depending on the specific application, particularly considering the size of the structural elements and the quantity required to manufacture a particular part. As will be understood from the considerations described above, there may be good reasons to use more symbols than strictly required to distinguish the various structural elements used for a given part—for example, to provide error detection or robustness against visual occlusion. Having the same number of symbols for each identifier in a predetermined group is not necessary, but it is convenient.

[0039] The shape used for symbols is not required, but ideally they should be simple and well-defined with few edges. For example, use simple geometric shapes (such as squares, rectangles, circles, triangles) and familiar typographical symbols (such as +, ×, <, >, !) or simple combinations thereof.

[0040] In some embodiments, the system is configured to store information relating to the position of each detected symbol for each identifier. This allows for tracking variations in the placement of structural elements and / or the level of error within tolerances throughout the composite component. By storing and comparing data between the manufacture of consecutive components over time, variations between components can be monitored, allowing for better quality control between components. In some embodiments, the stored information can include information about which structural elements frequently exceed tolerances, and preferably, information about the classification of the type of tolerance exceedance, such as twisting, skew, tilting, or wrinkling. This level of monitoring allows for the marking of frequently exceeding tolerances to the production supervisor. If desired, for subsequent production runs, the production supervisor can adjust the placement of the symbols within the identifiers of the marked structural elements to make problems easier to identify during the laying process; for example, if a component frequently wrinkles in the middle, the symbol can be positioned near the problem area on the component to ensure that every instance of wrinkling exceeding tolerance is detected, so that it can be more easily corrected by the user.

[0041] The output from the monitoring subsystem regarding the correctness of structural components can be in the form of visual indicators. A visual indicator can be a two-state visual indicator to indicate whether the structural component is correct or incorrect. These two states can include the presence and absence of visual markers. Similarly, a visual indicator can have more than two states to convey further information. A visual indicator can be a three-state indicator. Visual indicators can include different color indicators, such as red, orange, and / or green (i.e., signal light states). Visual indicators are typically positioned so that they are visible to users in the manufacturing environment. These visual indicators can quickly indicate the current status of the laying process to the user. For example, a green light can be used to indicate that the current laying of the structural component is correct and complete; an orange light can be used to indicate that the system is waiting for the detection of an identifier; a flashing orange light can indicate that at least one symbol in the identifier has been detected, but not all symbols; and a red light can be used to indicate that the structural component is incorrect. Using a simple visual indicator system allows users to see the current status of the laying process without having to approach the user interface, thus allowing for a more efficient laying process. Of course, the indicators are not limited to visual indicators; for example, sound or tactile feedback can be used additionally or alternatively.

[0042] It should be understood that while most structural elements placed in the mold will be identified and required to be placed only in a specified predetermined order, there may be situations where experienced users (e.g., supervisors) wish to skip a structural element or wish for a structural element without an identifier acceptable to the system. Therefore, in some embodiments, overlay symbols can be placed by the user. The monitoring subsystem can be configured to detect at least one overlay symbol and follow predetermined instructions. The overlay symbol may indicate that the next structural element in the sequence should be skipped, or that the placed structural element will be considered correct. In some embodiments, there may be multiple different overlay symbols, each capable of indicating different information to the system, such as skipping the next element, the element being correct, or elements exceeding tolerances being acceptable. If an overlay symbol is detected, the projector (where provided) may stop projecting an image of the intended identifier as confirmation that an overlay symbol has been detected. If an overlay symbol is detected, the monitoring subsystem may also be configured to check a second placement of the overlay symbol on the structural element, or to check a second overlay symbol to confirm the instructions given by the overlay symbol.

[0043] Using overlay symbols means that users do not need to directly check progress or provide input to computing devices during operation. Instead, users can prevent unnecessary pauses in the manufacturing process directly from the laying area.

[0044] Those skilled in the art will understand that this invention can be used in many applications, particularly for large-scale manufacturing. This invention may be particularly useful in applications where quality control between components is critical, as variations in element orientation or within components can cause significant problems when the entire engineered article is being constructed. In some embodiments, the system can be configured to monitor the manufacturing of wind turbine blades. The dimensions within wind turbine blades, as well as the fiber orientation, distribution, and thickness, are extensively designed to ensure optimal energy transfer through the rotation of the blades, while also ensuring that strong winds do not damage the entire wind turbine. Therefore, this is an application where component manufacturing should be advantageously monitored. Furthermore, wrinkles and other defects in any structural element can lead to undesirable structural defects that require extensive repair or go undetected and subsequently require on-site repair or cause further damage. Similarly, the performance of components in ships and aircraft is also critical. Attached Figure Description

[0045] Certain embodiments of the invention will now be described by way of example only, with reference to the accompanying drawings, in which: Figure 1 The manufacturing environment employing an embodiment of the present invention is shown; Figure 2 It shows Figure 1 A schematic diagram of the monitoring system shown; Figure 3A and Figure 3B An example of a ply sheet that can be detected by this monitoring system is shown; Figure 4 An example laying sequence from a top-down view is shown in an embodiment of the present invention; Figure 5 It shows relative to Figure 4 Examples of incorrect laying order shown; Figure 6 An example laying sequence with overlay symbols used in embodiments of the present invention is shown.

[0046] Figure 7A shows a top view of the mold, illustrating the detection area used in some embodiments; Figure 7B A top view of the mold is shown, with symbol projections as used in some embodiments; Figure 8 This illustrates placing the ply sheet onto a surface with... Figure 7A and Figure 7B The detection area and the mold in which the symbol is projected; Figure 9A and Figure 9B The diagram shows a layup sheet laid out and overlapped in the correct order using the detection area of ​​FIG7A, as used in some embodiments; Figures 10A to 10C Examples of incorrectly laid sheets that can be detected in some embodiments are shown; Figures 11A to 11C Examples of out-of-tolerance calculations that can be analyzed in some embodiments are shown; Figure 12 A camera view shows the obstructed sheet material that can be detected; Figure 13 This is a flowchart illustrating the steps of a method for monitoring the manufacture of a component according to an embodiment of the present invention. Detailed Implementation

[0047] Figure 1 This is a diagram of a manufacturing environment 200 used for manufacturing fiber-reinforced composite material parts. Figure 1 A layup sheet 220 is shown above mold 210. A crane 250 is shown moving the layup sheet 220 from above into mold 210. The layup sheet 220 has an identifier 224, which includes two symbols 225a and 225b at specific locations on the layup sheet 220, as will be described in more detail below. Above the crane 250 (e.g., attached to the ceiling or suspended from a support above mold 210) are a camera 120 and a projector 130, which are part of a monitoring system.

[0048] In this example, the component being manufactured is a wind turbine blade, although those skilled in the art will understand that the manufacturing environment for other large composite components will be similar, and the present invention can be applied in the same way to a variety of manufacturing environments.

[0049] The turbine blades are made of a composite material manufactured by layering plywood 220 into a mold 210, then filling the mold 210 with resin and curing the entire component. This forms the desired specific shape and has the required complex fiber orientation. The mold 210 can be approximately 90m long, and the camera 120 and projector 130 are located approximately 10m above the mold 210. Although only five cameras 120 are shown in this figure for simplicity, in practice, more (e.g., thirty) cameras 120 are evenly distributed above the mold 210 to cover the area of ​​the mold. Only a single crane 250 is shown in this figure, but it should be understood that multiple cranes 250 or other mechanical lifting devices can be used.

[0050] Inside the die 210 is the layup area, where ply sheets 220 are laid layer by layer. In a typical die 210 of this size, two hundred or three hundred individual ply sheets 220 are laid layer by layer, and all must be carefully moved into place by a crane 250. Of these two hundred to three hundred individual ply sheets 220, many will have the same profile (i.e., size and shape). For this exemplary turbine blade, fifteen different profiles of ply sheets 220 are used, thus providing a predefined set of fifteen distinct identifiers.

[0051] Depending on the desired final shape, some layers may include multiple ply sheets 220 at different locations within the layer, while other layers may require only a single sheet. Each ply sheet 220 is specifically positioned to form the desired final shape and to provide it with a specific density distribution, depending on how and where each ply sheet 220 is placed within the die. The manner in which the ply sheets 220 are placed into the die 210 will vary depending on the size and type of the part being produced, as will be understood by those skilled in the art, and will not be further explained herein.

[0052] The camera 120 is positioned above the mold 210 to observe the interior of the mold 210, so as to image the layup sheet 220 as it is laid into the mold 210 and to identify the layup sheet 220 by its identifier 224. Figure 1 In one embodiment, the ply sheet 220 has an identifier 224 consisting of two symbols 225a and 225b. The combination of the shape, position, and orientation of both symbols 225a and 225b uniquely identifies the ply sheet 220 as having one of fifteen different sheet profiles used in this particular component. Typically, in each manufacturing environment, each ply sheet 220 with a different profile is assigned an identifier consisting of a different combination of symbols (i.e., for each different shape / size of ply sheet, the symbols change their position and / or orientation and / or shape). A camera 120 is positioned to generate image data including at least symbols 225a and 225b. This image data is then transmitted to a monitoring unit, which is configured as shown in the following reference. Figure 2 The monitoring subsystem will be discussed further.

[0053] Projector 130 is positioned to project the image of symbol 225 onto mold 210. Projector 130 is capable of projecting the image of symbol 225 onto mold 210 to indicate where the layup sheet 220 should be placed. Therefore, projector 130 is positioned above mold 210 to be able to project an image onto any location within mold 210, where, when correctly positioned, symbol 225 on any layup sheet 210 should be located during the layup process, as will be referred to below. Figure 6And a more detailed overview as shown in Figure 7.

[0054] Figure 2 It shows Figure 1 A schematic diagram of the manufacturing monitoring system 100 used in the process. The monitoring system 100 includes multiple image sensors, which are displayed as shown in the diagram. Figure 1 As shown, each of the cameras 120, along with multiple image sensors, sends image data to the monitoring unit 110. The monitoring unit 110 includes a locator section 112 and a tracking unit 114, which will be described below, and is capable of transmitting analyzed information to a user interface 142 and / or indicator lights 144. The monitoring unit 110 also communicates with a projector 130. The monitoring unit 110 is computer-implemented. The monitoring unit 110 includes a processor and memory, the memory containing computer-executable instructions that, when executed by the processor, cause the processor to perform the various steps outlined below.

[0055] Monitoring unit 110 has stored in its memory information about the correct order in which the layup sheets should be placed into the die according to their identifiers, as part of a predetermined sequence, as will be referred to below. Figure 4 and Figure 5 And described in more detail.

[0056] When manufacturing a part made of fiber-reinforced composite material, details of the part are provided to monitoring unit 110—for example, by downloading from a network (not shown). The details of the operation include a predetermined sequence in which the ply sheets should be placed into the die. After the operation is initiated by the operator, monitoring unit 110 provides the operator with information via user interface 142 about which ply sheet is first in the sequence. The operator can then select the correct ply sheet from the shelf area and move it to its position in the die.

[0057] The intended layup sheets from a predetermined sequence (or multiple sheets placed into a die for each layup of a single layer) are also indicated by projecting images of their identifiers onto the die using one or more projectors 130 to aid in the placement of the layup sheets into the die. The projected image shows the orange outline of the intended symbol before the layup sheet has been identified as correct or incorrect. During this process, indicator lights will also display orange to indicate that placement and identification are in progress. Once the layup sheet is placed into the die, image data is transmitted from camera 120 to monitoring unit 110.

[0058] The locator 112 is used to precisely locate the portions of the image data searched by the monitoring unit 110 for symbols. Before performing detailed analysis of the detection area to detect the identifier, the locator 112 identifies the general area (referred to as the detection area) where the symbol is located. See below for reference. Figure 7A and Figure 8 The operation of the locator 112 is described in more detail. Then, the monitoring unit 112 performs in-depth analysis only in these predefined detection areas.

[0059] The monitoring unit 110 then continues searching the detection area for identifiers required for a predefined sequence. Once an identifier is detected, the monitoring unit 110 compares it with the expected identifier in the predetermined sequence and outputs an indication of whether the placed ply sheet is correct. This indication can be conveyed through any combination of the user interface 142, indicator lights 144, and / or projector 130. If the ply sheet has been correctly placed, the projected image from projector 130 will turn green, indicator lights 144 will turn green, and detailed output on the user interface 142 will show that the ply sheet has been determined to be correct. The monitoring unit 110 then proceeds to the next identifier in the predetermined sequence, and the process outlined above is repeated for the next ply sheet.

[0060] If the sheet is identified as incorrect, the projector 130 will instead display a red projected image, the indicator light 144 will turn red, and the user interface will provide details about why the detected sheet is incorrect, as will be explained in more detail below.

[0061] At each step of the predetermined sequence, tracking unit 114 records details of the layup sheet being placed. This data is recorded separately for each manufactured part, enabling monitoring of the manufacturing process over time.

[0062] User interface 142 is a fixed or portable computing device, such as a fixed terminal, monitoring terminal, portable mobile device, remote terminal, or any other computing device known to those skilled in the art for running an automated manufacturing process. The user interface may be a dedicated terminal or implemented as software on another computing device, such as an application implemented on a tablet computer. Signal lights 144 are one or more light indicators placed around the manufacturing environment so that the operator can see the current status of the laying process without interacting with user interface 142.

[0063] Figure 3A and Figure 3B Two different ply sheets 220a and 220b are shown with different combinations of two symbols 225a, 225b, 225c, and 225d. The two symbols are combined to form the respective identifiers 224a and 224b of the two different ply sheets 220a and 220b. Figure 3BThe diagram also shows human-readable character 226, which allows users to quickly identify the ply sheet 220b without interpreting identifier 224b. Symbols 225a, 225b, 225c, and 225d differ in shape, position, and orientation to distinguish ply sheets 220a and 220b from one another. The orientation of a symbol is its angular rotation relative to ply sheets 220a and 220b and / or relative to die 210. The monitoring unit is capable of detecting the position of the symbols relative to the individual ply sheets 220a and 220b themselves, and also their position relative to die 210. For ply sheets 220 that are symmetrical about one or more axes, the placement of symbols 225 within identifier 224 can also be symmetrical about the same axis, ensuring that ply sheets 220 can be correctly identified in any correct orientation. It should be understood that while symbols 225 of any shape can be used, symbols 225 should be easily distinguishable from one another without requiring a high-resolution camera. Symbol 225 is integrated with each ply sheet 220 during the production of each ply sheet 220 to ensure consistent placement between production batches.

[0064] Figure 4 An example sequence for laying ply sheets 220c-220g into mold 210 is shown. The sequence comprises five steps. In the first step, three individual ply sheets 220c-220e are placed into mold 210; in the second step, two ply sheets 220c-220d are placed into the mold (they are the same as two of the three sheets placed in the first step); and in each of the last three steps, one ply sheet 220c, 220f, or 220g is placed. As shown, each ply sheet 220c-220g of a different shape / size has a different identifier 224c-224g, which is indicated by a combination of two symbols. Identical ply sheets 220 (like those seen in steps 1, 2, and 3 of the sequence) have the same identifier 224. Before the laying process begins, data including a predetermined order is provided to the monitoring unit so that the detected identifier 224 can be compared with those identifiers in the predetermined order as outlined above.

[0065] In some parts of the sequence, as shown in steps 3 through 5, a single layup sheet 220 is laid into the mold 210 in a single step of the sequence; however, in other parts of the sequence, as shown in steps 1 and 2, multiple layup sheets 220 are required in a single layer, so multiple layup sheets 220 are required in a single step of the sequence. The order of these layup sheets 220 within the sequence is not important if the layup sheets 220 in a single layer are not designed to overlap. The monitoring unit can be programmed such that a given step in the sequence is only determined to be correct once all the required layup sheets 220 have been placed, thus detecting all symbols within the combination of identifiers. However, if only one layup sheet 220 among multiple laid layup sheets 220 is incorrectly placed, it will be difficult to know which one it is. Therefore, it is advantageous to search for each identifier separately within the image data. In some implementations, separate cameras may be placed to locate each individual layup sheet 220, although in other implementations, multiple identifiers may be detected by a single camera, and / or a single identifier may be detected by more than one camera.

[0066] In some manufacturing environments, when moved to the correct position within mold 210, the layup sheet 220 may be dragged across the field of view of one or more cameras. For example, it may be necessary to... Figure 4 The layup sheets, placed in the rightmost position of the die 210, can be moved in from the right, left, top, or bottom of the field of view. To prevent incorrect identification of the layup sheet 220 as correct or incorrect when it is moved into place, the monitoring unit monitors for symbols across multiple image frames and determines whether the layup sheet 220 is correct or incorrect only if the identifier 224 has been detected for a certain duration (measured by the presence of the symbol in a certain number of image frames). It should be understood that the number of frames used will depend on the camera's frame rate. Since a symbol is only detected once the layup sheet 220 is stationary within the field of view, high frame rate video is not required; instead, it may be suitable to acquire image data only once per second or less.

[0067] Figure 5 Another iteration of a typical layup process is illustrated. In each of the first three steps of the sequence, the correct identifier has been detected; however, in the fourth step of the sequence, an incorrect identifier has been detected. At each step of the sequence, the monitoring system acquires image data provided by camera 120 and searches for symbols within the image. If the symbols constituting the identifiers for the correct layup sheet 220 are detected in the correct locations, the system provides an output indicating that the layup sheet is correct via indicator lights, a projector, and / or a user interface, as referenced above. Figure 2The monitoring unit then continues searching for the next layup sheet in the sequence. When an incorrect layup sheet 220g has been placed, as shown in step 4, it may simply be a matter of not giving the correct indication. However, it is more useful for the system to provide a clear output by projecting an image and / or turning a signal light red, declaring that the detected layup sheet 220 is incorrect. Furthermore, providing the user with additional information to ensure the correct sequence is restored as quickly as possible can be helpful. Therefore, the monitoring unit can provide output indicating which layup sheet 220 is expected to be placed, or when the current layup sheet 220 should be placed in the sequence.

[0068] exist Figure 5 In the example, step 5 of the sequence requires the ply sheet 220g that was placed in step 4. In this case, by providing the user with detailed output, they don't have to waste time returning the ply sheet 220g to storage; they simply realize that the ply sheet 220g is needed later. Instead, the correct ply sheet 220f can be introduced for step 4, and the previously incorrect ply sheet 220g can be quickly returned for placement in step 5.

[0069] The monitoring unit sequentially searches for each identifier compared to a predefined order. To avoid unnecessary indications that the layup sheet 220 is incorrect, the monitoring unit can wait a predetermined time before determining that the layup sheet is incorrect. For example, after the layup sheet 220 has been placed in step 3 of the order, the monitoring unit will search for identifiers such as... Figure 4 The correct identifier is shown in step 3. If the correct identifier is not placed after a set time period (set to be shorter than the minimum typical time between consecutive sheet placements), for example, one minute later, the monitoring unit will output that the placed ply sheet is incorrect. The set time period can be after a correct identification or after a user-initiated event (e.g., if the user has restarted the laying process after a break). In different embodiments, the sequence proceeds only to the next step after manual confirmation from the user. This can be used for evaluation only on demand.

[0070] Figure 6 The same usage was shown. Figure 4 Another laying process might work in the order of the steps. In this example, both steps 1 and 2 are correct, and... Figure 5The method shown is the same. In this example, the ply sheet 220c' placed in step 3 has the correct shape / size and is in the correct position, but it lacks its identifier. After ply sheet 220c' has been identified as incorrect by the monitoring unit, the user (e.g., a supervisor) has independently identified ply sheet 220c as correct, but it lacks its identifier 224. To avoid the need to replace the correct ply sheet 220c (which could significantly hinder production due to the production of another ply sheet), a cover symbol 226 has been placed on ply sheet 220c by the supervisor. This cover symbol 220, when detected by the monitoring unit, indicates that ply sheet 220 is correct despite its lack of identifier. To ensure that the cover symbol 226 is not an error, the monitoring unit waits for the cover symbol 226 to be placed a second time to confirm the cover, or waits for a second cover symbol. Those skilled in the art will understand that when instructed to do so by a supervisor, there can be various different types of user symbols (cover symbols), which can give the monitoring unit preset instructions to change a predefined order. Any use of the overlay symbol 226 will be recorded by the tracking unit as outlined above, along with the rest of the laying process.

[0071] In some example embodiments, the monitoring unit searches all acquired image data at a resolution that detects and distinguishes symbols 225 from one another to locate any symbol 225 within the image and determines which layup sheet 220 is present based on identifier 224. However, continuously monitoring all image data during production could require significant computing power. To accelerate the process of finding identifier 224 within image data frames, the monitoring unit includes a locator. As shown in FIG7A, the locator provides detection areas 122a, 122b for each possible symbol location within the mold 210.

[0072] Initially, the locator searches all image data at a low resolution, enabling the determination of approximate areas where symbols 225 may exist without resolving the precise shape, location, or orientation of each symbol 225. The locator identifies these approximate areas as detection regions 122a, 122b, within which data can be scanned at a higher resolution to determine the precise shape, location, and orientation of the symbols, thereby enabling the detection of the ply sheet 220 by its identifier. The two-stage search of the image data can use the same dataset (i.e., image frames), or it can use dynamic regions of interest (ROIs), where initial image frames are generated at a relatively low resolution, followed by higher-resolution imaging of the detection regions 122a, 122b.

[0073] In one alternative locator implementation, the locator can determine detection areas 122a, 122b around the location where the symbol should be located, based on a known layup order. These specific detection areas 122a, 122b are then analyzed to search for the symbol. Once a larger frame has been acquired, detection areas 122a, 122b can be formed in the image data, and only detection areas 122a, 122b are scanned to locate the symbol 224. The camera can be adjusted by the monitoring unit to acquire image data only within the defined detection areas 122a, 122b. If no symbol is detected in detection areas 122a, 122b, the monitoring unit can then output that the correct layup sheet 220 has not been found. Detection areas 122a, 122b can be dynamic, allowing them to be moved to find the precise location of the symbol 224 if no symbol is detected or not fully detected within the initial detection areas 122a, 122b.

[0074] The embodiments outlined above focus on the effective detection of identifiers to determine whether the layup sheet being placed is correct. The monitoring system can also be used by using, for example... Figure 1 and Figure 2 The projector shown is used to help place the layup sheet into the mold. Figure 7B This illustrates how projected images 132a and 132b can be seen in mold 210. For each stage in the sequence, the monitoring unit is comparing the detected symbol with the symbol expected from the identifier. The expected identifier can be projected into mold 210, thus providing projected images 132a and 132b at the positions where the symbols should be aligned during the laying process.

[0075] Figure 8 The detection areas 122a, 122b and 122h in Figure 7A are shown when the layup sheet 220h is placed in the mold 210. Figure 7BHow the projected images 132a and 132b interact. The ply sheet 220h has symbols 225a and 225b that form its identifier. When placing the ply sheet 220h, the user or machine placing the ply sheet 220h into the die 210 can aim to align the symbols 225a and 225b with the projected images 132a and 132b. This, in turn, can mean that any predefined detection areas 122a and 122b are more likely to contain the symbols 225a and 225b being detected. The signal light system outlined above can be integrated into the projection system such that the projected images 132a and 132b are orange when waiting for the correct symbol, red when an incorrect symbol is placed, and green when a given symbol is correct. When a single step in the sequence requires multiple layup sheets, each identifier will be read as correct in turn. So if the first layup has been correctly placed, the projected image of its symbol will turn green, while the projected image of the second layup sheet that has not yet been laid will still be indicated as orange, i.e., waiting for the identifier.

[0076] When using the locator, the system also adapts to the movement of the ply sheet 220h across the field of view, as discussed above, by making a determination on the correctness of the ply sheet 220 only if symbols 225a and 225b are present in all detection areas 122a and 122b for a given identifier. This means that if the first symbol 225a moves through the second detection area 122b at the same time the ply sheet 220h is moved into place, and therefore there is no symbol in the first detection area 122a, the monitoring unit will not begin the analysis on the correctness of the ply sheet 220. Instead, the monitoring unit will wait until the symbols are present in both detection areas 122a and 122b. At this stage, a detailed determination can be made to ensure that the ply sheet 220h is correct.

[0077] Figure 9A and Figure 9B A perspective view is shown illustrating how two ply sheets 220a and 220b are laid relative to each other and checked for correctness. In this example, the upper ply sheet 220a is laid in place first and is determined to be correct by checking its identifier symbols 224aa and 224ab as discussed above. Although the upper ply sheet 220a is defined first in the laying order due to other considerations (e.g., it is preferable to place it first for more efficient placement of other ply sheets in the sequence), the upper ply sheet 220a has a small overlap 221a that needs to be on top of the lower ply sheet 220b on the left.

[0078] The underlay sheet 220b has three symbols 224ba, 224bb, and 224bc, which together form its identifier as discussed above. The symbol 224bb on the portion 221b of the underlay sheet 220b that overlaps with a portion 221a of the overlay sheet 220a performs the additional function of an overlap symbol.

[0079] Before the lower layer sheet 220b enters the mold, the overlapping portion 221a on the upper layer sheet 220a is folded back to allow the lower layer sheet 220b to be laid correctly, such as... Figure 9A As indicated by the arrows in the diagram, while the overlapping portion 221a is folded back, the monitoring unit uses symbols 224ba, 224bb, and 224bc to determine that the underlay sheet 220b has been correctly placed, as discussed elsewhere in this document.

[0080] Figure 9B The diagram shows that the overlapping portion 221a of the upper ply sheet 220a has been placed back over the lower ply sheet 220b, thus covering the "overlap" symbol 224bb. A second-stage check is then performed. In this example, the locator has identified three detection areas 122a, 122b, and 122c, and is searching for the presence or absence of a symbol in these areas. The lower ply sheet 220b is now considered correctly placed because no symbol is detected in detection area 122b. Therefore, the monitoring subsystem has been able to infer that the overlapping portion 221a of the upper ply sheet 220a has been correctly placed back over the second ply sheet 220b. If symbol 224bc continues to be detected in detection area 112b, this would mean that the overlapping portion has not yet been correctly placed back.

[0081] Although Figure 9A and Figure 9B In one embodiment, the overlap symbol is located on the underlay sheet; however, in other embodiments, the overlap symbol may instead be placed on the overlapping portion of the upper lay sheet. In these embodiments, the first stage of confirming that the underlay sheet is correct will mean detecting the identifier on the second lay sheet in a normal manner, and the second stage will be further (definitely) detecting the overlap symbol from the upper lay sheet when the upper lay sheet is correctly placed back on top of the underlay sheet.

[0082] Figures 10A to 10CThe diagram illustrates some possible placements of the ply sheet 220i into the mold 210, and the system can determine if a placement is incorrect. For example, in a mold where the ply sheet 220i cannot be placed in the wrong position due to the molding of the mold itself, determining whether the placement is correct may be limited to whether the ply sheet has the correct shape / size. However, it is envisioned that in most applications, the ply sheet 220i being placed also needs to be in the correct position within the mold 210 to be indicated as correct. The correct placement of the ply sheet 220i in the mold 210... Figures 10A to 10C The figures are shown in dashed lines. In each of these examples, the layup sheet 220i being placed has the correct shape / size, but is incorrect due to another factor. These layup sheets 220i are not exactly rectangular, but rather taper slightly inward toward the right side of the die 210.

[0083] exist Figure 10A In the image, the 220i sheet appears to be correctly positioned, but it has been placed upside down, causing the left end to taper inwards. This is noticeable to the naked eye, especially to objects like... Figure 1 In such a large manufacturing environment, it would be difficult to determine that the ply sheet 220i is incorrect. However, for the monitoring system, symbols 225a and 225b are not in the correct position because they are oriented incorrectly relative to their positions on the detected ply sheet 220i. This means that the monitoring unit can easily determine that the ply sheet 220i has been placed incorrectly.

[0084] exist Figure 10B In this example, the ply sheet 220i has been positioned to the right and rotated away from its proper position within the die 210. In this example, the detected symbols 225a and 225b are in the correct position relative to the ply sheet 220i, but incorrect relative to the die 210. Furthermore, symbols 225a and 225b have the wrong orientation. Similarly, in this example, the monitoring unit is able to determine that the ply sheet is incorrect because the positions of symbols 225a and 225b within the identifier are incorrect.

[0085] exist Figure 10CIn this process, the ply sheet 220i is placed in the correct position, but it has been deformed relative to its intended shape / size. This deformation may be due to localized shearing on the ply sheet 220i after manufacturing, or it may be due to a manufacturing error. During the deformation of the ply sheet 220i, the symbols 225a and 225b constituting its identifiers also deform. Therefore, although the position and shape of the symbols 225a and 225b may appear correct from a distance, their shape, position, and orientation have been slightly altered by the deformation of the ply sheet 220i. In some embodiments, the monitoring unit can detect... Figure 10C The deformed ply sheet 220i shown is incorrectly detected. For example... Figure 10C The determination of the ply sheet 220i as shown will depend on the comparison between the resolution of the image data and the size of the features within the symbol shape. For example, since the deformation of the symbol is smaller than one pixel in the image, it will not be detected.

[0086] While the monitoring unit can determine whether a ply sheet is correct or incorrect by comparing the precise position and shape of the symbols in the identifier with a predetermined order, it is advantageous to determine that the ply sheet is correct if it falls within a predetermined tolerance range. The tolerance can be determined by errors in the shape and / or position and / or orientation of individual symbols, or by averaging across the entire identifier. Figures 10A to 10C Example errors in some symbol positions are shown, which can be used to determine whether the ply sheet is within tolerance. Figures 10A to 10C Each of these shows a simplified set of image data, thus illustrating the variation in symbol position.

[0087] exist Figure 11A In the diagram, four symbols, labeled A through D, are accompanied by vector arrows representing the difference between the actual position of a symbol and the position of the symbol in the identifier expected from its position in the sequence. Each symbol is detected at a position differing from its expected position by a different vector. The monitoring system can either record the positional error of each symbol individually, monitor tolerance by comparing the highest single error in either direction, or calculate an aggregated error metric.

[0088] exist Figure 11A In the example, symbol A has the highest single error along the X direction, and symbol D has the highest single error along the Z direction. Symbol D also has the highest overall error magnitude because it deviates from its expected position along both the X and Y directions. For a single identifier, the tolerance can be determined based on any one or all of these highest single errors being less than a predetermined error tolerance.

[0089] Figure 11BAnother error measurement method is shown, in which the angular error between the two most extreme errors relative to the nominal value is measured. In this example, symbol A has the maximum error along the positive X direction, and symbol C has the maximum error along the negative X direction. The angle theta between these two errors can represent the twist in the ply sheet.

[0090] Figure 11C Three symbols AC are shown again, illustrating the outline (e.g., projection) relative to where the symbols should be positioned. In this example, symbol C has the largest single error when compared to the expected (nominal) position of the symbols.

[0091] The monitoring system can be used to record information about the laying process via the tracking unit described above. The data stored in the tracking unit can include information relative to... Figures 10A to 10C The error in the nominal value shown is illustrated. This information can be used to monitor variations between components, ensuring consistency with production, and can provide feedback on the manufacturing of the ply sheet over time, thus aiding in quality control.

[0092] Figure 12 An example view of a ply sheet 220j in a mold (not shown) captured by a camera is shown, in which a portion of the ply sheet 220j is partially obscured by a crane 250. The ply sheet 220j has six symbols 225a-225f that constitute its identifier; however, one symbol 225b is hidden from the image sensor's line of sight by the crane 250. The ply sheet 220j is correctly positioned, but the entire identifier is not visible to the monitoring unit. To prevent this "correct" ply sheet 220j from being incorrectly identified as "incorrect," the monitoring unit can determine the ply sheet to be correct if at least a threshold percentage (e.g., >75%) of the identifier has been detected and determined to be correct.

[0093] The threshold percentage can be calculated by considering whether the threshold percentage of the detection area includes the symbol. If the monitoring unit can determine that a missing symbol is blocked by performing image analysis to identify a large, known obstruction (e.g., a crane component), the monitoring unit can use the threshold percentage. If the symbol is only partially blocked from view, the monitoring unit can use that portion of the symbol in determining whether the ply sheet 220 is correct. To prevent incorrect identification of the ply sheet as correct when a symbol is missing from the identifier, the identifier group is ideally designed such that each identifier is distinct from other identifiers in the manufacturing environment due to more than a single factor (e.g., having at least two different symbol shapes and / or positions between each identifier). Therefore, using... Figure 12For example, when symbol 225b is not visible due to being obstructed by crane 250, the monitoring unit eliminates the requirement that symbol 225b must be seen in the identifiers used for the ply sheet 220 to be considered correct. In a more complex implementation, the monitoring unit may consider which symbols can be detected and perform an analysis on whether they are sufficient to definitively identify the ply sheet compared to different ply sheet types (e.g., the fifteen in the example given above).

[0094] It should be understood that the various embodiments of the monitoring unit outlined above include steps performed on a computing device. In order to perform the invention, data about a predefined sequence is provided to the monitoring unit, and correct logical links are established between the steps in the sequence to ensure that the sequence is progressively passed as each layup sheet is laid. Therefore, the monitoring unit is pre-programmed with information about the predetermined sequence.

[0095] The predetermined sequence can be taught to the monitoring unit through a test layup procedure. In the test layup procedure, for each step, the correct layup sheet is carefully positioned before saving data based on provided image data (e.g., from each image sensor) for use in detecting symbols in the location. When the system is trained through the test layup procedure, all obstructions should be removed from the layup area before recording data to ensure that no symbols are missed.

[0096] A predetermined order can also be passed to the monitoring unit by directly inputting identifiers from other computing systems. For example, if the geometry of a component is defined or assisted by computer-aided design (CAD), the CAD geometry can be used to extract the predetermined order of identifiers. This can be achieved through full 3D calibration of the housing.

[0097] While two distinct methods for teaching the predetermined sequence have been outlined above, those skilled in the art will understand that a combination of the two methods can be used to ensure that any CAD input is correlated with the actual condition of the calibration layup procedure. The programming of the predetermined sequence can also be correlated with the production of the layup sheets themselves to ensure that the symbols of the layup sheets being produced match those anticipated by the monitoring unit.

[0098] After the manufacturing of multiple components has begun, the final predetermined order may need to be changed. The monitoring system may include, for example, facilities as part of a user interface, for allowing the user to change the predetermined order by, for example, adding, deleting, or moving sheets within the sequence.

[0099] Figure 13 This is a flowchart 300 illustrating a method for monitoring the laying of composite material components according to the present invention. It should be understood that this method is applicable to specific examples of the monitoring systems outlined above.

[0100] In step 310, image data is received from the image sensor. As summarized above, the image data can be for a specific area of ​​the laying area, such as the detection area, or it can be image data for the entire laying area. In some examples, image data is acquired after a predetermined time elapsed since the previous cycle of user input or method.

[0101] Steps 320 and 330 include a determination of whether the layup sheets have been correctly placed relative to a predefined order.

[0102] In step 320, an identifier is detected in the image data. The identifier is detected from a combination of symbols on each ply sheet. The identification can be performed, for example, via a two-step process using a locator as described above with reference to FIG. 7A. When the identification is detected via a two-step process, the first step is to locate the symbol in the image data, either through a known detection area or through a calculated detection area. The second step is to detect the symbol within the detection area. The identification detection may include detecting at least a predetermined percentage of the identifier to compensate for the above reference. Figure 12 Overview of situations where the image sensor's field of view is obstructed. Identifier detection can also wait until the symbol remains stationary for a predetermined number of image frames before proceeding to the next step, allowing the layup sheet to move within the image sensor's field of view, as referenced above. Figure 4 As outlined in Figure 7A.

[0103] In step 330, the detected identifiers are compared with the identifiers required for a predefined order. This comparison includes whether the correct combination of symbols was detected, and may also include determinations regarding whether the identifiers were detected in the correct position relative to the die, and / or whether the ply sheet was within tolerance. Therefore, it should be understood that the comparison of identifiers with order may include three separately determinable variables: the correct size / shape of the ply sheet, the correct position of the ply sheet, and the ply sheet being within tolerance (i.e., order, position, and accuracy).

[0104] In step 340, an output is generated indicating whether the ply sheet has been determined to be correct, i.e., relative to the variable being measured. This output may include information about which variable is incorrect, or information about which ply sheet is next, or which ply sheet should have been placed.

[0105] If the detected ply sheet is determined to be incorrect, the method returns to receiving image data and re-executes the process until the correct ply sheet has been placed. As outlined above, in some examples, overlay symbols may be used. According to the outlined method, if an identifier in the image data is determined to be an overlay symbol, the process continues according to the specific instructions indicated by the overlay symbol. For example, if the overlay symbol indicates that the ply sheet should be considered correct, the process will continue as if the correct ply sheet has been identified.

[0106] If the detected ply sheet is determined to be correct, the process can move to the next step in the predetermined sequence, and the process can restart from step 310.

[0107] The two ply sheets have the above reference. Figure 9A and Figure 9B In the case of the overlapping portion, if a two-step process is used, they can actually be considered as two independent steps in sequence for determining whether the underlay sheet is correct, such that in the first step image data is received and compared with the underlay sheet, and in the second step image data is received and compared with the underlay sheet having the portion overlapping with the top lay sheet on top. Alternatively, in some embodiments, these can be combined into a single step, in which the overlapping portion has been folded back into place (e.g., as shown in the image). Figure 9B (As shown in the diagram) In cases where overlapping symbols are blocked, the underlay sheet is analyzed without first making a separate determination about whether the underlay sheet is correct.

[0108] Those skilled in the art will understand that although the method steps are outlined herein in a specific order, some steps may be performed at different times within the process, or concurrently with other steps. For example, there may be a series of outputs in the form of projected images, indicating where the layup sheets should be placed, as referenced above. Figure 7B The image data reception can be continuous or can be performed only at predetermined intervals. Therefore, it should be understood that the outlined methods are merely illustrative, and embodiments may include additional steps to incorporate those features outlined with reference to the system described above.

[0109] The examples of the invention described above illustrate how ply sheets are placed into a mold during manufacturing and how these individual ply sheets are monitored. Those skilled in the art will understand that the above examples are equally applicable to any individually ply structural element, and the ply sheets in any individually ply structural element are merely one example.

[0110] Those skilled in the art will understand that the invention has been described by way of description of one or more specific aspects of the invention, but is not limited to these aspects; many variations and modifications are possible within the scope of the appended claims.

Claims

1. A manufacturing monitoring system for monitoring the manufacture of a composite material component, the composite material component being made from a plurality of structural elements laid in a mold in a predetermined order, each of the plurality of structural elements having an identifier; wherein, Each identifier includes multiple symbols that are different in shape and at least one of position and orientation on a corresponding structural element; The system includes: At least one image sensor, configured to acquire image data of the identifier when the structural element is laid into the mold; and The monitoring subsystem is configured as follows: Receive image data from the at least one image sensor; The correctness of the structural element in the mold is determined from the image data by detecting an identifier on the structural element from the image data and comparing the detected identifier with an expected identifier obtained from the predetermined sequence; and Output an indication of whether the structural elements are correct.

2. The manufacturing monitoring system according to claim 1, wherein, The manufacturing monitoring system is configured to look up the next identifier in the predetermined order after the correct structural element has been detected and the indication has been output.

3. The manufacturing monitoring system according to any one of the preceding claims, wherein, The monitoring subsystem is configured to determine that the structural element is correct by further determining that the structural element has been placed in the correct position within the mold.

4. The manufacturing monitoring system according to any one of the preceding claims, wherein, The manufacturing monitoring system is configured to check whether the structural element is within a predetermined tolerance relative to one or more of the following items: angle relative to the nominal direction, shear angle, local deformation, deformation of the entire structural element, wrinkle formation, and tilt of the structural element in the mold.

5. The manufacturing monitoring system according to claim 3 or 4, wherein, The manufacturing monitoring system is configured to determine that the structural element is correct if it detects that each symbol of the identifier is within a predetermined tolerance compared to the expected position.

6. The manufacturing monitoring system according to any one of the preceding claims, wherein, The system is configured to store information relating to the position of each detected symbol for each detected identifier.

7. The manufacturing monitoring system according to any one of claims 4 to 6, wherein, The monitoring subsystem is configured to perform a two-stage determination: the first stage includes determining that the structural elements are in the correct order relative to the predetermined sequence; The second stage includes determining that the structural element is correctly positioned within the mold by positioning the symbols of the structural element within one or more predetermined tolerances or corresponding expected positions.

8. The manufacturing monitoring system according to any one of the preceding claims, wherein, The monitoring subsystem includes a locator portion configured to determine at least one detection region in the image data, wherein one or more symbols in the identifier should be located in the at least one detection region for structural elements expected to be correct in the predetermined order. The monitoring subsystem is configured to detect the identifier by searching for symbols in the detection area.

9. The manufacturing monitoring system according to any one of claims 1 to 7, wherein, The monitoring subsystem includes a locator section configured to determine at least one detection region by searching for any candidate symbols in the image data. When a candidate symbol is located, the monitoring subsystem is configured to determine the shape and position of the symbol.

10. The manufacturing monitoring system according to any one of the preceding claims, wherein, The plurality of structural elements includes an upper structural element and a lower structural element, the upper structural element being earlier than the lower structural element in the predetermined order, and wherein an overlap symbol is provided on the upper structural element or the lower structural element, the presence or absence of the overlap symbol indicating whether the portion of the upper structural element overlapping with the lower structural element has been returned after the lower structural element is placed in the mold.

11. The manufacturing monitoring system according to claim 10, wherein, The overlapping symbol is placed on the portion of the lower structural element that overlaps with the upper structural element, and the monitoring subsystem is configured to perform an inspection step to determine that the overlapping symbol is not visible.

12. The manufacturing monitoring system according to any one of the preceding claims, wherein, The at least one image sensor is configured to monitor the area where all structural elements can be laid.

13. The manufacturing monitoring system according to any one of the preceding claims, wherein, The manufacturing monitoring system is configured to provide indications of the expected structural elements from the predetermined sequence.

14. The manufacturing monitoring system according to claim 13, wherein, The indication of the expected structural element includes information about the identifier associated with the expected structural element.

15. The manufacturing monitoring system according to any one of the preceding claims, wherein, The manufacturing monitoring system includes at least one projector configured to project an image of the intended identifier for the correct structural element in the predetermined sequence onto the mold.

16. The manufacturing monitoring system according to claim 13, wherein, The indication of whether the structural elements are correct includes the color of the projected image.

17. The manufacturing monitoring system according to any one of the preceding claims, wherein, When the structural element is determined to be incorrect, the indication in the output of the monitoring subsystem includes at least one of the following: information about when the structural element was placed in the predetermined order; And information regarding the correct positioning of the structural elements.

18. The manufacturing monitoring system according to any one of the preceding claims, wherein, The manufacturing monitoring system is configured to make a determination on whether a structural element is correct after at least one of the following items: a predetermined time; or an identifier has been detected exceeding a predetermined time; or an identifier has been detected exceeding a predetermined number of image frames.

19. The manufacturing monitoring system according to any one of the preceding claims, wherein, The manufacturing monitoring system is configured to make a determination on whether a structural element is correct only after the user provides input indicating that the structural element is considered ready.

20. The manufacturing monitoring system according to any one of the preceding claims, wherein, The system includes a user interface configured to display to the user indications from the monitoring subsystem regarding the correctness of the structural elements.

21. The manufacturing monitoring system according to any one of the preceding claims, wherein, The monitoring subsystem is configured to determine that the structural element is correct if at least a predetermined threshold percentage of the identifier has been determined to be correct compared to the expected identifier.

22. The manufacturing monitoring system according to any one of the preceding claims, wherein, The monitoring subsystem is configured to determine the identifier as correct if a sufficient portion of the identifier has been detected to rule out the identifier being any other identifier in a predetermined group of identifiers.

23. The manufacturing monitoring system according to any one of the preceding claims, wherein, The manufacturing monitoring system is configured to allow certified users to provide input to identify the structural element as correct.

24. The manufacturing monitoring system according to any one of the preceding claims, wherein, The monitoring subsystem is configured to detect at least one overlay symbol and follow predetermined instructions.

25. The manufacturing monitoring system according to claim 22, wherein, If an overlay symbol has been detected, the monitoring subsystem is also configured to check a second placement of the overlay symbol on the structural element, or to check a second overlay symbol to confirm the instructions given by the overlay symbol.

26. A monitoring method for monitoring the laying of a composite material component, the composite material component being made of a plurality of structural elements laid in a mold in a predetermined order, each of the plurality of structural elements having an identifier; wherein, Each identifier includes multiple symbols that are different in shape and at least one of position and orientation on a corresponding structural element; The method includes: Receive image data of the identifier from at least one image sensor; The following steps are used to determine whether the structural elements in the mold are correct from the image data: Detect identifiers on the structural elements from the image data; and The detected identifier is compared with the expected identifier obtained from the predetermined order; and Output an indication of whether the structural elements are correct.

27. A computer software product or non-transitory computer-readable medium, comprising instructions that, when executed by a processor, cause the processor to perform the method according to claim 24.

Images