Monitoring of composite part manufacture

Abstract

A manufacturing monitoring system (200) for monitoring manufacture of a composite part and a method relating to the same are disclosed. The composite part is made from a plurality of structural elements (220) laid into a mould (210) in a predetermined sequence, and each of the structural elements (220) have an identifier (224). Each identifier has a plurality of symbols (225a, 225b) varying by shape and at least one of position and orientation on respective structural elements (220). The system (200) includes at least one image sensor (120) configured to collect image data of the identifier when a structural element (220) is laid into the mould (210) and a monitoring sub-system. The monitoring sub-system is configured to: receive image data from the at least one image sensor (120); determine from the image data whether the structural element (220) in the mould (210) is correct by detecting an identifier (224) on the structural element (220) from the image data, and comparing the detected identifier (224) to an expected identifier (220) derived from the predetermined sequence; and output an indication as to whether the structural element (220) is correct.

Description

[0001] Monitoring of Composite Part Manufacture

[0002] This invention relates to the monitoring of the manufacture of parts made from composite materials, such as fibre-reinforced composite materials.

[0003] It is known in the art that manufacture of parts made from composite material includes the lay-up of multiple sheets of ply within a mould, to shape the composite part into its required shape. The sheets of ply are commonly sheets of woven or unidirectional fibres, such as fibre-glass or carbon-fibres. These can be pre-soaked in a resin material, or once placed into a mould, the resin material can be infused through the layers of ply to create a fibre-reinforced composite part. Techniques such as these are widely used in the manufacture of large-scale pieces, for example wind turbine blades, aircraft structures or parts for ships. In some applications the layers of sheets of ply are separated by layers of powder, and the whole mould then undergoes heat treatment to produce the final composite part. In other applications the resin is preimpregnated with the fibres before oven or autoclave curing.

[0004] When manufacturing small parts of composite material, each sheet of ply may be identical or easily distinguishable from each other, making the layering of all the sheets of ply into a mould relatively simple. However, for large-scale parts, especially those with complex fibre directions and shape requirements, each sheet of ply must be laid into the mould in specific locations and in a specific order to build up the required shape and carry the structural loads. Sheets of ply for different locations within the part may also be of different shapes and sizes. The scale of these sheets of ply makes it difficult for a person to visually identify different sheets of ply from one another, or be able to manually check if they are located correctly in a mould, especially as ply is commonly laid into place with the assistance of a crane. In such manufacturing environments each sheet of ply needs to be able to be individually recognisable to help with the lay-up process. This means that the production of large parts from composite materials can be a laborious process requiring a highly skilled and experienced operatives. One option for addressing this difficulty might be for parts to be identified via the addition of Quick Response (QR) codes, bar codes, or human readable characters (i.e. numbering or lettering) to plies. However these rely on accurate recognition locally to the ply, which the Applicant has recognised can prove difficult, or for the identification to be provided on very large scales. For example, for a QR code to be read from 10m away, the QR code would need to be around 1m2, and the whole of the QR code would need to be visible. In large scale manufacturing applications, placing of a camera even 10m away from the lay-up area may not be possible, instead they may be required to be nearer 20m away from where a sheet of ply is being inserted into a mould, to be above the level that cranes moving the sheets of ply operate.

[0005] According to a first aspect of the present invention there is provided a manufacturing monitoring system for monitoring manufacture of a composite part, the composite part being made from a plurality of structural elements laid into a mould in a predetermined sequence, each of the structural elements having an identifier; wherein each identifier comprises a plurality of symbols varying by shape and at least one of position and orientation on respective structural elements; wherein the system comprises: at least one image sensor configured to collect image data of the identifier when a structural element is laid into the mould; and a monitoring sub-system configured to: receive image data from the at least one image sensor; determine from the image data whether the structural element in the mould is correct by detecting an identifier on the structural element from the image data, and comparing the detected identifier to an expected identifier derived from the predetermined sequence; and output an indication as to whether the structural element is correct.

[0006] According to a further aspect of the present invention there is provided a method of monitoring composite part lay-up, the composite part being made from a plurality of structural elements laid into a mould in a predetermined sequence, each of the structural elements having an identifier; wherein each identifier comprises a plurality of symbols varying by shape and at least one of position and orientation on respective structural elements; wherein the method comprises: receiving image data of the identifier from at least one image sensor; determining from the image data whether the structural element in the mould is correct by: detecting an identifier on the structural element from the image data; and comparing the detected identifier to an expected identifier derived from the predetermined sequence; and outputting an indication as to whether the structural element is correct.

[0007] When viewed from a further the invention provides a computer software product and a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out the method outlined above.

[0008] Thus it will be seen by those skilled in the art that in accordance with the present invention, the lay-up of structural elements into a mould can be monitored reliably using the respective identifiers, to help confirm that the correct structural element is being placed into the mould. The Applicant has appreciated that the use of an identifier comprising a plurality of symbols with varying shapes, position and / or orientation can provide for robust identification when compared to other techniques such as QR or bar codes.

[0009] By ensuring that structural elements are placed in the correct order, the lay-up process can be made more consistent, reducing the risk of failure of the parts made from incorrectly laid structural elements. Whilst parts made from an inconsistent pay-up process may be detected in product testing, by eliminating errors earlier in the manufacturing process, overall product wastage is also reduced. Reduction in possible variation between different parts in turn also helps improve overall product quality. By impacting the accuracy and time taken for the lay-up process, the whole manufacturing process can be made more efficient, by reducing wasted time of remaking parts, or re-laying structural elements reducing the overall time cost for the process. It can also relax the requirements for those operating the process to be highly trained.

[0010] It will be appreciated that a structural element may be any element used in the production of a part of composite material (i.e. a composite part). Typically such structural elements would not perse have any structural rigidity. For example, the structural elements may be sheets of ply (e.g. a woven fibre sheet ). Structural elements may be homogenous materials, metallic meshes or even core materials. In the manufacture of some parts, the lay-up process includes layering of elements already formed from multiple sheets of ply instead of, or in addition to the lay-up of single sheets of ply. Therefore in some embodiments the structural element may be a prefabricated element (e.g. one made from multiple sheets of ply previously cured together) which may have some structural rigidity. In the production of a composite part a variety of different structural elements may be used. It is not essential for a given structural element to perform a critical structural function in the finished composite part. For example it could provide useful thermal, electrical or even aesthetic properties.

[0011] The composite part may be any part made from a combination of different materials or layers. It will be appreciated that the composite part will typically comprise other elements or materials in addition to the structural elements (e.g. resin or powder) as will be known to those skilled in the art.

[0012] During manufacture, each structural element is laid into a mould so as to form the whole part as will be known by those skilled in the art. Each structural element may be a different shape and size, or some structural elements may be the same within the whole part. Each structural element will typically be required to be laid in a certain order, position and orientation within the mould to create the whole composite part. The lay-up process is established when the part is designed prior to manufacture. Whilst in some embodiments the invention may be applied to the placement of single structural elements as part of a manufacturing procedure, the invention is typically likely to be most useful in the lay-up process of multiple structural elements. In some embodiments therefore, the system is configured to look for a next identifier in the predetermined sequence after a correct structural element has been detected, and the indication has been output. The monitoring system can therefore to look for each sequential identifier in the predetermined sequence, so as to be able to monitor the whole lay-up process.

[0013] As the sequence of structural elements is predetermined, the respective set of identifiers used on the structural elements is also predetermined. This significantly simplifies the task of performing machine visual identification and in a set of embodiments the monitoring system may be arranged only to look for identifiers from the predetermined set of identifiers.

[0014] Moreover, given that even for large, complex parts the number of different structural elements is only likely to be of the order of a few hundred, the degrees of freedom provided by having multiple symbols and multiple possible positions and / or orientations means that, the symbols themselves can be very simple, avoiding a requirement for high-resolution images which need to be able to distinguish small changes from one identifier to the next. Indeed given that the monitoring subsystem can be provided with the order in which structural elements should be laid, and thus information regarding where each identifier fits within the predetermined sequence, it is even possible to order the identifiers to enhance the difference between one identifier and the next in the sequence.

[0015] The predetermined set of identifiers may include ten or fewer different symbol shapes and the number of discrete orientations and positions in which they can be placed will be relatively limited so in practice the theoretical size of the possible set of identifiers is relatively limited (e.g. of the order of a few thousand) in contrast to the effectively unlimited number of potential QR codes.

[0016] The monitoring subsystem may be arranged to detect individual symbols from within the set of symbols which make up the identifiers, and then link the combination of these symbols to the pre-defined set of identifiers. Alternatively, complete identifiers could be detected and compared directly with the expected identifier in the predetermined sequence or more widely against the pre-defined set of identifiers. By having spatial requirements for the symbols of each identifier, the probability of false positive determinations of a correct structural element is reduced. The position of a symbol may be determined by its position relative to the edges or another visually distinctive feature of the structural element, and in some embodiments the position of a symbol may also be determined relative to a lay-up area in the mould. The orientation of a symbol may be an angular position relative to the visually distinctive feature of the structural element and / or the orientation relative to the lay-up area in the mould. If a structural element is symmetrical about any axis, the identifier can be symmetrical about the same axis, so the structural element can be placed into the mould in any correct orientation. In some composite parts there may be multiple structural elements of the same type during the whole sequence, and these can all be identified from the same identifier.

[0017] It will be appreciated that the order in which structural elements are required to be laid in the mould depends on various requirements of the part, as discussed above. Within a composite part, portions of a structural element may overlap with other structural element(s) to different degrees as required by the structure of the part. In some cases where there is only a small amount of overlap between two given structural elements, it may be considered best for the overall structure for the upper structural element to be placed into the mould before the lower one, notwithstanding the small amount of overlap. In such cases the portion of the upper structural element which overlaps the lower one, for example a protruding flap or end portion of the upper strucural element, may need to be temporarily moved out of the way, e.g. by folding, in order to place the lower structural element. Clearly however it is important that it is replaced once the lower element has been placed. In a set of embodiments therefore, the plurality of structural elements comprise an upper structural element and a lower structural element, the upper structural element being earlier in the predetermined sequence relative to the lower structural element, wherein an overlap symbol is provided on either the upper or lower structural element, the presence or absence of which indicates whether a portion of the upper structural element which overlaps the lower structural element has been replaced following placement of the lower structural element in the mould.

[0018] Determining whether the lower structural element is correctly placed may include detecting the presence or absence of the overlap symbol, i.e. if the overlap symbol is on the upper structural element, the lower structural element will be determined to have been correctly placed if the overlap symbol is detected, and if the overlap symbol is on the lower structural element, the lower structural element will be determined to be correctly placed if the overlap symbol is not detected. This allows the system to accommodate the cases set out above and thus provides greater flexibility in coping with more complex composite structures and sequences.

[0019] In one set of such embodiments the overlap symbol is provided on the lower structural element on a portion which is overlapped by the upper structural element. The identifier for the lower structural element could be defined in such a way as to require the absence of the overlap symbol for correct placement of the lower structural element to be indicated. Alternatively a separate subsequent checking step could be carried out to determine that the overlap symbol is not visible. This would allow the overlap sumbol to perform an additional role as part of the normal identifier if desired.

[0020] Thus in some embodiments, determining whether the lower strucural element has been correctly placed is a two-step process. The first step comprises determining whether the lower strucural elelemt is itself correct (correct element, correct location etc) and the second step comprises determining whether the overlapping portion of the upper structural element has been correctly replaced (i.e. that the overlapping portion of the upper strucural elelemt has been moved back on top of the lower strucural element once the lower structural element has been placed as discussed above). This two-stage check can help improve efficiency and accuracy of the part ensuring that the overlapping portion is not replaced until it has been confirmed that it is being placed over the correct lower structural element.

[0021] In another set of embodiments (not necessarily mutually exclusive) the overlap symbol is provided on the overlapping portion of the upper structural element and a separate checking step is carried out to determine that the overlap symbol is visible.

[0022] According to some embodiments, the system is configured to check whether said structural elements are within a predetermined tolerance with respect to a given metric. It will be appreciated that for tolerance checks, image data may need to be of a resolution high enough to allow for accurate measurements of the positions and / or orientations of symbols within identifiers relative to the element, pre-defined lay-up area or mould. A structural element may be determined to be correct if the symbols of the identifier are each detected at a position within a predetermined tolerance compared with an expected position. The tolerance may allow for one or more of: angles against a nominal direction, a shear angle, local distortion, distortion across a whole structural element, wrinkle creation, tilt of a structural element in the mould etc. Data corresponding to precise positions and / or orientation of each of the symbols within an identifier can be collected and analysed. For example, data corresponding to precise positon and / or orientation of at least one symbol may be used to determine local distortion of a structural element as a result of wrinkling or other defects. Different errors for different symbols within an identifier can represent different problems with structural elements or their placement. When monitoring for problems with individual structural elements, the position of the symbols within each identifier determine what sort of metric can be detected. Therefore, to monitor the errors in individual structural elements the symbols are preferably spaced across the whole of the structural element. In this manner, wrinkles in individual structural elements can be detected and local and whole sheet distortion can be measured. For example, if a structural element has six symbols in its identifier, and if all the symbols are of the correct shape, but only one of the symbols is in the wrong orientation, it may be determined that local shear or wrinkles are present close to that symbol. This is particularly advantageous when single sheets of ply, or other thin and flexible structural elements are being placed into a mould. Conventionally wrinkles in a sheet, particularly when the sheet is light in colour, are very difficult to see. Changes in the distance between symbols in the identifier in accordance with the invention however can be more easily used to identify that a sheet has been wrinkled whilst being placed into the mould. A user can then be swiftly informed of the problem, and can adjust the laid element to eliminate such wrinkles.

[0023] In some embodiments the monitoring subsystem is configured to make a two-stage determination: said first stage comprising determining that the structural element is correct relative to the predetermined sequence; and said second stage comprising determining that the structural element is correctly located within the mould by the symbols thereof being positioned within one or more predetermined tolerances of expected positions.

[0024] Whilst in some manufacturing environments, structural elements might only be able to be laid in one at a time, in some other environments multiple structural elements may be laid next to each other to form each layer of the completed composite part. Therefore, in some embodiments the image sensor(s) is / are configured to monitor an area where all structural elements might be laid. The monitoring subsystem may be able to detect multiple identifiers from the image data.

[0025] In the predetermined sequence, any structural elements which belong in the same layer of the part may be determined as correct when placed in any order, so long as they are placed in the correct location. In some embodiments the system may wait until all the structural elements in a given layer are placed before determining that the layer is correct. In another set of embodiments, the system may determine that each structural element is correct in turn. By allowing for flexibility within the predefined sequence when multiple structural elements are part of the same layer, unnecessary “incorrect” structural elements will be not be identified. This also means users will have flexibility in which element is placed in what order in a given layer, when the lay-up allows, increasing both ease of use, and the efficiency of the lay-up procedure.

[0026] In some embodiments, a structural element may be considered as correct as long as it is the next expected structural element to be placed into the mould relative to the predetermined sequence. In some embodiments however determining whether a structural element is correct also includes determining whether the structural element has been placed in a correct location within the mould - e.g. by comparing the detailed position of one or more symbols of the identifier to one or more expected detailed positions. In these embodiments the monitoring subsystem is therefore provided with information as to where, relative to the mould lay-up area, the symbol(s) for an identifier should be located.

[0027] In some embodiments the system is arranged to provide an indication of the expected structural element - preferably including the identifier relating to the expected structural element from the predetermined sequence. This indication could be provided only if an incorrect structural element is detected, but in some embodiments the indication is provided before a determination is made. The indication of the expected structural element may include information about the identifier relating to the expected structural element. The indication could be provided on a user interface such as a display screen but in some (potentially overlapping) embodiments the system comprises at least one projector configured to project an image of the expected identifier for the correct structural element in the predetermined sequence onto the mould. The projected images of the identifier can thus be used to help place the correct structural element in the mould, improving the efficiency of the lay-up process. In some embodiments the indication as to whether the structural element is correct comprises a colour of the projected image. In some embodiments, when it has been determined that a structural element is correct the colour of the projected image is arranged to change i.e. the projected image may comprise the indication as to whether the structural element is correct which is provided by the monitoring subsystem in accordance with the invention. This simple visual indicator can give a clear confirmation to the user that they can move on to placing the next structural element, improving the ease of the lay-up process.

[0028] In some embodiments, when the structural element has been determined to be incorrect, the indication in the output from the monitoring sub-system includes information about when that structural element should have been placed in the predetermined sequence. This may help to educate the user and / or may enable a structural element which is required soon to be stored temporarily near the mould rather than returning it to a storage area.

[0029] In some embodiments where the detailed location of the structural element within the mould is also monitored, when the location of the structural element has been determined to be incorrect, the indication in the output from the monitoring subsystem includes information on correct positioning of the structural element. This can enable a user to make sure it is moved to the correct position. This can reduce the number of errors in the lay-up process.

[0030] When monitoring the lay-up of each structural element in the predetermined sequence, it is anticipated that in the process of a structural element being moved into place it will move across the field of view of the image sensor(s). Therefore, in some embodiments a determination as to whether a structural element is correct is made after at least one of: a predetermined time; or an identifier has been detected for more than a predetermined time; or an identifier has been detected for more than a predetermined number of image frames. In this way the system will not output negative results whilst a structural element is still being moved into place, improving the accuracy of the monitoring system. The predetermined time may be a length of time after a previous determination of a correct structural element and / or a length of time after an input from a user. The input from a user may be at least one of an indication that the lay-up process has begun, an indication that the sequence should progress to the next step in the predetermined sequence, and / or any other suitable progress determination.

[0031] In other embodiments the system is configured to make the determination as to whether a structural element is correct only after a user provides an input to indicate that the structural element is considered to be ready. To facilitate the output indication and / or any inputs from a user, in some embodiments the system comprises a user interface configured to display the indication from the monitoring subsystem as to whether the structural element is correct to a user.

[0032] In many manufacturing environments, the lay-up area can be surrounded by lots of large pieces of equipment, including cranes which are used to lift each structural element into place in the mould. In such environments, pieces of equipment may get in the way of some of the image sensors, preventing complete image data from being obtained of the lay-up area. In such situations, the equipment may block out some of or all of one or more symbols that make up the identifier. In some embodiments therefore, the monitoring subsystem is arranged to determine that a structural element is correct if at least a predetermined threshold percentage of the identifier has been determined as correct relative compared to the expected identifier. The predetermined threshold percentage could, for example, be 80%. Equally however a different threshold could be used depending on the number and diversity of identifiers in the predetermined set. In some embodiments an identifier is determined to be correct if enough of it has been detected to be able to exclude the identifier being any other from the predetermined set. It will be appreciated in this context that a partially obscured symbol may still be able be used in recognising an identifier, i.e. where a shape of a symbol is distinguishable from other symbols even when partially obscured e.g. a portion of a circle is always distinguishable from a portion of a triangle. In some embodiments, the system may be configured to allow an authenticated user to provide an input to identify said structural element as correct.

[0033] It will be appreciated that the image data may be acquired from a single sensor or from multiple sensors simultaneously. The image sensor(s) may be cameras. The image sensor(s) may be configured to monitor the whole of the mould or they may only need to be able to image the part of the mould into which structural elements will be laid (i.e. the lay-up area). Where multiple image sensors are provided they may each cover the whole of the monitored area or all or some may cover only part of the monitored area. Multiple image sensors located to view the lay-up area from different angles may be advantageous if one of the image sensors is blocked by other equipment, as another sensor can provide the data to look for an identifier. In some of embodiments, the monitoring subsystem comprises a localiser portion configured to determine detection zones by searching the whole image data for any candidate symbols, wherein when a candidate symbol is located, the monitoring subsystem is arranged to determine the shape, position and / or orientation of the symbol. The use of a localiser portion can reduce the time taken for an identifier to be detected and determined to be correct or incorrect by only scanning a proportion of the image data, or performing a low resolution scan of the data before doing a more detailed search for detailed shape, position and / or orientation of the detected symbols. If it is determined that the position of a symbol is incorrect or there is no detected symbol in a detection zone, the monitoring subsystem can immediately output the indication that the strucutal element is incorrect. This can eliminate the need for a second stage of analysis informing the user more quickly that a structural element is incorrect, which in turn improves the efficiency of the manufacturing process.

[0034] In another set of embodiments, the localiser portion is configured to determine detection zones in the image data where one or more symbols in the identifier should be located for the expected correct structural element in the predetermined sequence. Since the positions of the symbols on a structural element is a parameter of the identifier, the localiser portion is able to allow narrowing of the search for an expected symbol needing detailed imaging for detecting shape (and potentially orientation) of the symbol to the corresponding detection zone. The monitoring subsystem may therefore be configured to detect the identifier by searching for symbols in the detection zones. The image sensor(s) may be only be required to provide image data of the detection zones. The monitoring subsystem could be configured to perform only a low resolution analysis of the detection zone(s) to determine only whether a symbol is present at all, before proceeding to carry out a more detailed analysis to identify the shape or position (including orientation) of the symbol.

[0035] The number of symbols in a given identifier can be chosen depending on the specific application, taking account particularly the size of the structural elements and how many are required to manufacture a particular part. As will be appreciated from the considerations set out hereinabove, there may be good reasons to employ more symbols than is strictly required to differentiate the structural elements for a given part - e.g. to provide error detection or robustness against visual occlusion. It is not essential for each identifier in a predetermined set to have the same number of symbols but this is convenient,

[0036] The shapes used for the symbols are not essential but ideally they are simple and well defined with a small number of edges. For example, simple geometric shapes (e.g. squares, rectangles, circles, triangles) and familiar typographic symbols (e.g. +, x, <,>,!) or simple combinations thereof are used.

[0037] In some embodiments, the system is configured to store information regarding the positions of detected symbols for each detected identifier. This can allow tracking of variation in placement of structural elements and / or level of error within tolerances across the whole composite part. By storing and comparing data between manufacture of successive parts over time, variations between parts can be monitored, allowing for better quality control between parts. In some embodiments, the stored information can include information about which structural elements are frequently out of tolerance, and preferably record information about a classification of types of out of tolerance e.g. twisted, skewed, tilted or wrinkled. This level of monitoring can allow elements which are regularly out of tolerance to be flagged to a production supervisor. If desirable, for a subsequent production run the production supervisor can adjust the placement of symbols within an identifier for a flagged structural element so that problems can more easily be identified during the lay-up process, e.g. if an element is regularly wrinkled in the middle the symbols can be positioned on the element in the vicinity of the problem area to ensure every instance of an out of tolerance wrinkle is detected, so it can be more readily corrected by a user.

[0038] The output from the monitoring subsystem as to whether the structural element is correct may be in the form of a visual indicator. The visual indicator may be a two-state visual indicator to indicate whether the structural element is correct or incorrect. The two states could comprise presence and absence of a visual sign. Equally the visual indicator may have more than two states to convey further information. The visual indicator may be a three-state indicator. The visual indicator may comprise different colour indicators, e.g. red, orange and / or green (i.e. a traffic light status). The visual indicator would typically be located so as to be seen by a user in the manufacturing environment. These visual indications can quickly indicate to a user the current status of the lay-up procedure. For example, a green light may be used to indicate that the lay-up of the current structural element is correct and completed; an orange light may be used to indicate that the system is awaiting detection of an identifier; a flashing orange light may indicate that at least one of the symbols in an identifier has been detected, but not all of the symbols; and a red light may be used to indicate that the structural element is incorrect. The use of a simple visual indication system can enable users to see the current status of the lay-up procedure without needing to be local to a user interface, allowing for more efficient lay-up procedure. Of course, the indication is not limited to be a visual one; sounds or haptic feedback could be used additionally or alternatively for example.

[0039] It will be appreciated that whilst most structural elements being placed in the mould will have been provided with identifiers, and be required to be placed only in the specified predetermined sequence, there may be situations where an experienced user (e.g. a supervisor) may wish for a structural element to be skipped, or for a structural element which does not have an identifier to be accepted by the system. Therefore, in some embodiments an override symbol may be placed by a user. The monitoring subsystem may be configured to detect at least one override symbol and follow a predetermined instruction. The override symbol may indicate that the next structural element in the sequence should be skipped, or that the placed structural element is to be treated as correct. In some embodiments there may be multiple different override symbols, and each can indicate to the system a different piece of information e.g. skip the next element, the element is correct, the out of tolerance element is acceptable etc. If an override symbol has been detected, the projector (where provided) may stop projecting an image of the expected identifier as confirmation that the override symbol has been detected. If an override symbol has been detected, the monitoring subsystem may be further configured check for a second placement of the override symbol on the structural element, or to check for a second override symbol to confirm the instruction given by the override symbol.

[0040] Use of an override symbol may mean that a user is not required to be checking progress or providing input to a computing device directly during operation, instead the user can prevent unnecessary pauses to the manufacturing procedure from the lay-up area directly. The skilled person will appreciate that the invention may be used in numerous applications, especially in large scale manufacture. The invention may be particularly useful in applications where quality control between parts is important, as variation in element orientation or presence within a part can cause significant problems when the whole engineered item is constructed. In some embodiments, the system may be configured to monitor manufacture of wind turbine blades. The dimensions, and fibre orientation and distribution and thickness within a wind turbine blade is heavily engineered to ensure optimum transfer of energy through the rotation of the blades, whilst also ensuring that heavy winds do not cause damage to the whole wind turbine. It is therefore an application where part manufacture should be advantageously monitored. Furthermore, wrinkles and other defects in any structural element can cause undesirable structural defects that need extensive repair or lay undetected and subsequently need field repair or cause damage. Similarly the performance of parts in ships and aircraft is also critical.

[0041] Certain embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0042] Figure 1 shows a manufacturing environment employing an embodiment of the present invention;

[0043] Figure 2 shows a schematic diagram of the monitoring system illustrated in Figure 1 ;

[0044] Figures 3A and 3B show example sheets of ply which can be detected by the monitoring system;

[0045] Figure 4 shows an example lay-up sequence from a top down perspective used in embodiments of the present invention;

[0046] Figure 5 shows an example of an incorrect lay-up sequence relative to that shown in Figure 4;

[0047] Figure 6 shows an example lay-up sequence with an override symbol used in embodiments of the present invention.

[0048] Figure 7A shows a top down perspective of a mould showing detection zones used in some embodiments;

[0049] Figure 7B shows a top down perspective of a mould showing symbol projection as used in some embodiments;

[0050] Figure 8 shows the placement of a sheet of ply into a mould with the detection zones and symbol projection of Figures 7A and 7B; Figures 9A and 9B show sheets of ply being laid in correct sequence with an overlap using the detection zones of Figures 7A as used in some embodiments;

[0051] Figures 10A - 10C show examples of incorrect sheets of ply which can be detected in some embodiments;

[0052] Figures 11 A - 11 C show examples of out of tolerance calculations which can be analysed in some embodiments;

[0053] Figure 12 shows a camera view of a sheet of ply being obstructed which can be detected;

[0054] Figure 13 is a flow diagram showing method steps for monitoring manufacture of parts according to an embodiment of the present invention.

[0055] Figure 1 is a diagram of a manufacturing environment 200 for making parts of fibre reinforced composite material. Figure 1 shows a sheet of ply 220 above a mould 210. A crane 250 is shown moving the sheet of ply 220 into a mould 210 from above. The sheet of ply 220 has an identifier 224 comprising two symbols 225a and 225b in specific positions on the sheet of ply 220 as will be described in more detail below. Above the crane 250 (e.g. attached to a ceiling or suspended from struts above the mould 210) are cameras 120 and projectors 130 which are part of a monitoring system.

[0056] In this example, the part being manufactured is a wind turbine blade, although a person skilled in the art will appreciate that the manufacturing environment for other large-scale composite parts will be similar, and that the invention can be equally applied to various manufacturing environments.

[0057] The turbine blade is made from composite material that is manufactured by layering sheets of ply 220 into a mould 210 before the mould 210 is filled with resin and the whole part is cured. This produces the specific shape required, with required complex fibre-orientation. The mould 210 may be around 90m in length, and the cameras 120 and projectors 130 are located at a distance of around 10m above the mould 210. Whilst only five cameras 120 are shown in this diagram for simplicity, to cover the area of the mould in practice many more (e.g. thirty) cameras 120 are evenly distributed above the mould 210. This diagram only shows a single crane 250, but it will be appreciated that multiple cranes 250, or other mechanical lifting means may be used. Inside the mould 210 is a lay-up area into which sheets of ply 220 of layered. In a typical mould 210 of this size, two or three hundred individual sheets of ply 220 are layered, and each must be carefully moved into place by the crane 250. Of the two to three hundred separate sheets of ply 220, many will have the same profile (i.e. size and shape). For this exemplary turbine blade fifteen different profiles of sheet of ply 220 are used, so there is a predefined set of fifteen different identifiers.

[0058] Depending on the desired final shape, some layers include multiple sheets of ply 220 in different locations within a layer, and other layers require only single sheets. Each sheet of ply 220 is specifically placed to produce the desired final shape and provide it with a specific density profile which depends on how and where each sheet of ply 220 is placed within the mould. The manner in which sheets of ply 220 are placed into a mould 210 will vary depending on the size and type of part being produced, as will be appreciated by those skilled in the art, and will not be further explained herein.

[0059] The cameras 120 are located above the mould 210 to be able to view the inside of the mould 210 so as to image the sheets of ply 220 as they are being laid into the mould 210, and to identify the sheet of ply 220 by its identifier 224. In the embodiment of Figure 1, the sheet of ply 220 has identifier 224 made from two symbols 225a, 225b. The combination of the shape, position and orientation of both symbols 225a, 225b uniquely identify the sheet of ply 220 as having one of the fifteen different sheet profiles used in this particular part. In general, within each manufacturing environment each sheet of ply 220 with a different profile is assigned an identifier made from different combinations of symbols (i.e. the symbols varying their positions and / or oreientations and / or shapes for each different shape / size sheet of ply). The cameras 120 are located to produce image data which at least includes the symbols 225a, 225b. This image data is then passed on to a monitoring unit forming a monitoring subsystem as discussed further below with reference to Figure 2.

[0060] The projectors 130 are located so as to project images of symbols 225 into the mould 210. The projectors 130 can project images of the symbols 225 into the mould 210 to indicate where a sheet of ply 220 should be placed. The projectors 130 are therefore placed above the mould 210 to be able to project images to any location within the mould 210 where a symbol 225 on any sheet of ply 210 during the lay-up procedure should be located when placed correctly as will be outlined in more detail below with reference to Figures 6 and 7.

[0061] Figure 2 shows a schematic diagram of the manufacturing monitoring system 100 employed in Figure 1. The monitoring system 100 includes several image sensors in the form of the cameras 120 as shown in Figure 1, which send image data to a monitoring unit 110. The monitoring unit 110 includes a localiser portion 112, and tracking unit 114 which will be described below, and can pass analysed information to a user interface 142 and / or a traffic light 144. The monitoring unit 110 is also in communication with the projectors 130. The monitoring unit 110 is computer implemented. The monitoring unit 110 includes a processor and a memory comprising computer-executable instructions that, when executed by the processor, cause the processor to perform the various steps outlined below.

[0062] The monitoring unit 110 has stored information in its memory regarding the correct order in which order sheets of ply should be placed into a mould from their identifiers as part of a predetermined sequence, as will be described in more detail below with reference to Figures 4 and 5.

[0063] When a part made of fibre-reinforced composite material is to be manufactured, details of the part are provided to the monitoring unit 110 - e.g. by downloading from a network (not shown). The details of the job include a predetermined sequence in which the sheets of ply should be placed into the mould. After the job is initiated by the operator, the monitoring unit 110 provides information about which sheet of ply is first in the sequence to the operator via the user interface 142. The operator can then select the correct sheet of ply from a shelving area and move it into position in the mould.

[0064] The expected sheet of ply from the predefined sequence (or plies where multiple sheets are placed in the mould for a single layer) is also indicated by projecting images of their identifiers into the mould with the projector(s) 130, to aid with the placement of the sheet of ply into the mould. Before a sheet of ply has been identified as correct or incorrect, the projected image shows an orange outline of expected symbols are the projected image. During this process the traffic light will also show orange, indicating that placement and identification is in process. Once the sheet of ply is placed into the mould, image data is sent from the cameras 120 to the monitoring unit 110.

[0065] The localiser 112 is used to refine the parts of the image data where symbols are searched for by the monitoring unit 110. The localiser 112 identifies general areas (known as detection zones) in which symbols are located before performing detailed analysis in the detection zones to detect an identifier.. The operation of the localiser 112 is described in more detail below with reference to Figure 7A and Figure 8. In- depth analysis is then performed by the monitoring unit 112 only in these predefined detection zones.

[0066] The monitoring unit 110 then proceeds to search the detection zones for the identifier required by the predefined sequence. Once the identifier has been detected, the monitoring unit 110 compares it to the expected identifier in the predetermined sequence and outputs an indication as to whether the sheet of ply placed was correct. This indication can be communicated via any combination of the user interface 142, the traffic light 144 and / or the projector 130. If the sheet of ply has been correctly placed, the projected image from the projector 130 will turn green, the traffic light 144 will turn green, and a detailed output on the user interface 142 will show that the sheet of ply was determined to be correct. The monitoring unit 110 will then progress to the next identifier in the predetermined sequence, and the above outlined process will be repeated for the next sheet of ply.

[0067] If the sheet of ply is identified as incorrect, instead the projector 130 will give a red projected image, the traffic light 144 will turn red, and the user interface will give details about why the detected ply is incorrect, as will be explained in more detail below.

[0068] At every step of the predetermined sequence, the tracking unit 114 records details of the sheets of ply being placed. This data is recorded separately for each part made, so that the manufacturing process can be monitored over time.

[0069] The user interface 142 is a fixed or portable computing device e.g. a fixed terminal running automated manufacture processes, a monitoring terminal, a portable mobile device, a remote terminal or any other computing device known to those skilled in the art. The user interface may be a dedicated terminal or be implemented as software on another computing device e.g. as an application on a tablet. The traffic light 144 is one or more lamp indicator placed around the manufacturing environment so operators can see the current state of the lay-up process without needing to handle the user interface 142.

[0070] Figures 3A and 3B show two different sheets of ply 220a, 220b with different combinations of two symbols 225a, 225b, 225c, 225d which combine to form their individual identifiers 224a, 224b. In Figure 3B human readable characters 226 are also shown, which allow a user to quickly identify a sheet of ply 220b without needing to interpret the identifier 224b. The symbols 225a, 225b, 225c, 225d vary by shape, position and orientation to identify the sheets of ply 220a, 220b from each other. The orientation of a symbol is its angular rotation with respect to the sheet of ply 220a, 220b and / or with respect to the mould 210.. The monitoring unit can detect positions of the symbols relative to the individual sheets of ply 220a, 220b themselves but also their positions relative to the mould 210. For sheets of ply 220 which are symmetric about one or more axes, the placement of the symbols 225 within the identifier 224 can also be symmetric about the same axes, so that the sheet of ply 220 can be recognised as correct in in any correct orientation. It will be appreciated that whilst any shape of symbol 225 could be used, the symbols 225 should be easily identifiable from one another without the need for a high-resolution camera. The symbols 225 are integrated with each sheet of ply 220 during the production of the individual sheets of ply 220 to ensure consistency of placement between production runs.

[0071] Figure 4 shows an example sequence for laying sheets of ply 220c-220g into a mould 210. The sequence includes five steps. In the first step three separate sheets of ply 220c-220e are placed into the mould 210, in the second step two sheets of ply 220c- 220d are placed into the mould (these are identical to two of the three sheets placed in the first step), and in each of the final three steps one sheet of ply 220c, 220f, 220g is placed. As shown, each different shape / size sheet of ply 220c-220g has a different identifier 224c-224g shown by a combination of two symbols. Identical sheets of ply 220, like those seen in steps 1, 2 and 3 of the sequence, have the same identifier 224. The monitoring unit is provided with data which includes this predetermined sequence before the lay-up process begins, so as to be able to compare detected identifiers 224 to those in the predetermined sequence as outlined above. In some parts of the sequence, one sheet of ply 220 is to be laid into the mould 210 in a single step of the sequence, as shown in steps 3-5, however in other parts of the sequence multiple sheets of ply 220 are required in a single layer so multiple are required for a single step of the sequence, as shown in steps 1 and 2. Where the sheets of ply 220 in a single layer are not designed to overlap, the order of those sheets of ply 220 within the sequence does not matter. The monitoring unit could be programmed so that a given step in the sequence will only be determined to be correct once all the required sheets of ply 220 are placed, so all the symbols within the combination of identifiers are detected. However, where only one of multiple laid sheets of ply 220 is incorrectly placed, it would be difficult to know which one. Therefore, advantageously, each identifier is searched for separately within the image data. In some implementations, separate cameras could be placed to look for each individual sheet of ply 220, although in other implementations, multiple identifiers may be detected by a single camera and / or a single identifier may be detected by more than one camera.

[0072] In some manufacturing environments a sheet of ply 220 may be dragged across the field view of one or more cameras when being moved into the correct location within the mould 210. For example, those required to be placed on the far right of the mould 210 in the sequence of Figure 4, may be moved in from the right, the left, above or below the field of view shown. In order to prevent misidentifying a sheet of ply 220 as correct or incorrect whilst it is being moved into position, the monitoring unit monitors symbols over multiple image frames, and only decides whether the sheet of ply 220 is correct or not when an identifier 224 has been detected for more than a certain length of time, measured by the symbols being present in a certain number of image frames. It will be appreciated that the number of frames used will depend on the frame rate of the camera. As symbols are only to be detected once the sheet of ply 220 is stationary within the field of view, high frame rate video is not necessary, instead it may be suitable to only take image data once every second, or longer.

[0073] Figure 5 shows another iteration of a typical lay-up process. In each of the first three steps of the sequence, the correct identifiers have been detected; however in the fourth step of the sequence the wrong identifier has been detected. At each step of the sequence, the monitoring system takes the image data provided by the cameras 120, and looks for symbols within the image. If symbols which make up the identifier for a correct sheet of ply 220 are detected in the correct positions, the system provides an output indicating that the sheet of ply is correct via the traffic light, projector and / or user interface, as was discussed above with reference to Figure 2. The monitoring unit then progresses to looking for the next sheet of ply within the sequence. When an incorrect sheet of ply 220g has been placed, like that shown in step 4, it could simply be that no correct indication is given. However more usefully the system provides a definite output stating that the detected sheet of ply 220 is incorrect by the projected image and / or traffic light turning red. Furthermore, it can be helpful to provide a user with additional information to ensure the correct sequence resumes as quickly as possible. The monitoring unit may therefore provide an output which indicates which sheet of ply 220 is expected instead, or when in the sequence the current sheet of ply 220 should be placed instead.

[0074] In the example of Figure 5, the sheet of ply 220g placed in step 4, is required in step 5 of the sequence. In this case, by providing a detailed output to the user they do not have to waste time in returning this sheet of ply 220g to storage, only to realise it is required again. Instead the correct sheet of ply 220f can be brought in for step 4, and the previously incorrect sheet of ply 220g can be quickly returned to be placed in step 5.

[0075] The monitoring unit looks for each identifier compared to the predefined sequence in turn. To avoid unnecessary indications that a sheet of ply 220 is incorrect, the monitoring unit can wait a predetermined length of time before making a determination that a sheet of ply in incorrect. For example, after the sheet of ply 220 has been placed in step 3 of the sequence, the monitoring unit will be looking for the correct identifier of step 3 as shown in Figure 4. If after a set period of time (set to shorter than the minimum typical time between placing of successive sheets), e.g. one minute, no correct identifier has been placed, the monitoring unit produces an output that the placed sheet of ply is incorrect. The set period of time can be after a correct identification, or it can be after a user initiated event, for example if a user has restarted the lay-up process after a break. In different embodiments the sequence will only progress onto the next step after manual confirmation from a user. This could be used to make an assessment on demand only. Figure 6 shows how another lay-up process might work, using the same Figure 4 sequence. In this example, steps 1 and 2 are both correct in the same way as shown in Figure 5. In this example, the sheet of ply 220c’ being placed in step 3 is the correct shape / size and in the correct location, however it is missing its identifier. After the sheet of ply 220c’ has been identified as incorrect by the monitoring unit, a user (e.g. a supervisor) has independently identified the sheet of ply 220c as correct, but that it is missing its identifier 224. To avoid needing to replace a correct sheet of ply 220c (potentially holding up production significantly to produce another sheet of ply), an override symbol 226 has been placed on the sheet of ply 220c by the supervisor. This override symbol 220, when detected by the monitoring unit, indicates that the sheet of ply 220 is correct despite its lack of identifier. To ensure that the override symbol 226 is not a mistake, the monitoring unit waits for the override symbol 226 to be placed a second time to confirm the override, or waits for a second override symbol. It will be appreciated by the skilled person that there may be multiple different types of user symbols (override symbols) which can give the monitoring unit pre-set instructions to alter the predefined sequence, when instructed to do so by a supervisor. Any use of the override symbol 226 will be recorded with the rest of the lay-up process by the tracking unit as outlined above.

[0076] In some example embodiments the monitoring unit searches all the acquired image data at a resolution where symbols 225 will be detected and distinguishable from one another to find any symbols 225 within the image, and determine which sheet of ply 220 is present from the identifier 224. However, continuous monitoring of all image data during production might require large amounts of computer processing power. To speed up the process of finding identifiers 224 within a frame of image data the monitoring unit includes a localiser. As shown in Figure 7A, the localiser implements detection zones 122a, 122b for each possible symbol location within the mould 210.

[0077] Initially, the localiser searches all of the image data at a lower resolution, so that general areas where symbols 225 might be present can be determined, without needing to resolve the exact shape, position or orientation of each symbol 225. The localiser identifies these general areas are as detection zones 122a, 122b, where the data can be scanned at higher resolution to determine exact shapes, positions and orientations of the symbols, enabling a sheet of ply 220 to be detected via its identifier. The two-stage search of image data may use the same data set (i.e. image frames), or a dynamic region of interest may be used, where an initial image frame is produced at a relatively low resolution, and the detection zones 122a, 122b are then imaged to a higher resolution.

[0078] In an alternative localiser implementation, the localiser could determine the detection zones 122a, 122b around the locations where symbols should be located according to the known lay-up sequence. These specific detection zones 122a, 122b would then be analysed to search for symbols. A detection zone 122a, 122b could be formed in the image data once a larger frame has already been acquired, and only the detection zones 122a, 122b would then be scanned to look for symbols 224. The cameras could be adjusted by the monitoring unit to acquire only image data of the defined detection zones 122a, 122b. If no symbol is detected in a detection zone 122a, 122b, the monitoring unit could then output that no correct sheet of ply 220 has been found. The detection zones 122a, 122b could be dynamic, so that if a symbol is not detected, or not fully detected within the initial detection zones 122a, 122b, the detection zone 122a, 122b can be moved to find the exact location of a symbol 224.

[0079] The above outlined embodiments focus on the efficient detection of identifiers to determine whether a sheet of ply being placed is correct. The monitoring system can also be used to help with placement of sheets of ply into a mould by using projectors as shown in Figures 1 and 2. Figure 7B shows how projected images 132a, 132b can be seen in the mould 210. For each stage in the sequence, the monitoring unit is comparing detected symbols against those expected from the identifier. The expected identifier can be projected into the mould 210 giving projected images 132a, 132b in the locations where symbols should be aligned during the lay-up process.

[0080] Figure 8 shows how the detection zones 122a, 122b of Figure 7A and the projected images 132a, 132b of Figure 7B interact when a sheet of ply 220h is placed in the mould 210. The sheet of ply 220h has symbols 225a, 225b forming its identifier. When placing the sheet of ply 220h, a user or machine which is placing the sheet of ply 220h into the mould 210 can aim to align the symbols 225a, 225b with the projected images 132a, 132b. This in turn can mean any predefined detection zones 122a, 122b are more likely to contain the symbols 225a, 225b being detected. The traffic light system outlined above can be integrated into the projection system, so that the projected images 132a, 132b are orange when awaiting a correct symbol, red when an incorrect symbol is placed, and green when a given symbol is correct. When multiple sheets of ply are required for a single step in the sequence, each identifier will read correct in turn, so if a first ply has been correctly placed, the projected images for its symbols will turn green, whilst the projected images for a second as yet unlaid sheet of ply will still indicate as orange i.e. awaiting an identifier.

[0081] When using the localiser, the system further accommodates movement of sheets of ply 220h across the field of view as discussed above, by only making a determination as to whether the sheet of ply 220 is correct when symbols 225a, 225b are present in all detection zones 122a, 122b for a given identifier. This means that if, whilst the sheet of ply 220h is being moved into place, the first symbol 225a moves through the second detection zone 122b, so there is no symbol in the first detections zone 122a, the monitoring unit will not start the analysis as to whether it is correct. Instead it will wait until a symbol is in both the detection zones 122a, 122b. At this stage, the detailed determination can be made, to ensure the sheet of ply 220h is correct.

[0082] Figures 9A and 9B show a perspective view of how two sheets of ply 220a, 220b may be laid relative to each other and checked to be correct. In this example, an upper sheet of ply 220a is laid into place first, is determined to be correct by the detection of its identifier symbols 224aa, 224ab as discussed above. Whilst the upper sheet of ply 220a was defined first in the lay-up sequence due to other considerations (e.g. it is best placed first for other sheets of ply in the sequence to be placed more efficiently), there is a small overlapping portion 221a of the upper sheet of ply 220a which is required to be on top of the lower sheet of ply 220b on the left.

[0083] The lower sheet of ply 220b has three symbols 224ba, 224bb, 224bc which make up its identifier as discussed above. The symbol 224bb located on the portion 221b of the lower sheet of ply 220b which is to be overlapped by part 221a of the upper sheet of ply 220a performs an additional role an overlap symbol.

[0084] Prior to the lower sheet of ply 220b being into the mould, the overlapping portion 221a on the upper sheet of ply 220a is folded back to allow the lower sheet of ply 220b to be laid correctly, as shown by the arrow in Figure 9A. Whilst the overlapping section 221a is folded back, the symbols 224ba, 224bb, 224bc are used by the monitoring unit to determine that the lower sheet of ply 220b has been correctly placed as discussed elsewhere.

[0085] Figure 9B shows the overlapping portion 221a of the upper sheet of ply 220a having been replaced over the lower sheet of ply 220b, covering the “overlap” symbol 224bb. A second stage of checking is then performed. In this example, the localiser has identified three detection zones 122a, 122b, 122c, where it is looking for the presence or absence of symbols. The lower sheet of ply 220b is now considered finally as correctly placed because no symbol is detected in detection zone 122b. The monitoring sub-system has therefore been able to make an inference that the overlapping portion 221a of the upper sheet of ply 220a has been correctly replaced over the second sheet of ply 220b. If the symbol 224bc continued to be detected in the detection zone 112b, it would mean that theoverlap portion had not been replaced correctly.

[0086] Whilst in the embodiment of Figures 9A and 9B the overlap symbol is located on the lower sheet of ply, in some other embodiments the overlap symbol may instead be placed on the overlap portion of the upper sheet of ply. In these embodiments, the first stage of confirming the lower sheet of ply as correct would mean the detection of the identifier on the second sheet in the normal way, and the second stage would be the further (positive) detection of the overlap symbol from the upper sheet of ply, when it is replaced correctly over the lower sheet of ply.

[0087] Figures 10A - 10C show some possible placements of sheets of ply 220i into a mould 210 which the system may determine to be incorrect. For example, in a mould where it is impossible to place sheets of ply 220i in the wrong location due to the shaping of the mould itself, determination of whether a placement is correct may be limited to whether the sheet of ply is the right shape / size. However, it is envisaged that in most applications the sheet of ply 220i being placed will also need to be in the correct position in the mould 210 to be indicated as correct. The correct placements for the sheets of ply 220i in the mould 210 is shown in Figures 10A - 10C is shown by the dashed lines. In each of these examples, the sheet of ply 220i being placed has is the correct shape / size, however is incorrect due to another factor. These sheets of ply 220i are not fully rectangular, instead taper slightly inwards towards the right-hand side of the mould 210. In Figure 10A, it might appear that the sheet of ply 220i is correctly located, however it has been placed upside-down so the left-hand end is tapered inwards. To the naked eye, especially in the large manufacturing environment like that of Figure 1 , it is difficult to make the determination that this sheet of ply 220i is incorrect. However, to the monitoring system, the symbols 225a, 225b are not in the correct positions, as both have the wrong orientation relative to their location on the detected sheet of ply 220i. This means the monitoring unit can easily determine that the sheet of ply 220i has been placed incorrectly.

[0088] In Figure 10B, the sheet of ply 220i has been placed displaced to the right as well as being rotated away from where it should be located within the mould 210. In this example, the detected symbols 225a, 225b are in the correct positions relative to the sheet of ply 220i, but at incorrect positions relative to the mould 210. In addition, the symbols 225a, 225b have the wrong orientation. Again, in this example the monitoring unit can determined that the sheet of ply is incorrect because the positions of the symbols 225a, 225b within the identifier are incorrect.

[0089] In Figure 10C, the sheet of ply 220i is placed in the correct location, however it has been warped relative to the shape / size it should be. The warping of the sheet of ply 220i may have been because of local shear on the sheet of ply 220i which has affected it after manufacture, or may have been due to a manufacturing error. During the warping of the sheet of ply 220i, the symbols 225a, 225b making its identifier have also warped. Therefore, whilst the position and shapes of the symbols 225a, 225b may look correct from a distance, their shape, position and orientation have been slightly altered by the warping of the sheet of ply 220i. In some embodiments, the monitoring unit can detect warped sheets of ply 220i like the one shown in Figure 10C as incorrect. The determination of a sheet of ply 220i like that shown in Figure 10C will depend upon the resolution of the image data vs the size of features within the shape of a symbol. For example, the warping of a symbol is not detected because it is less than a pixel on the image.

[0090] Whilst the monitoring unit could determine whether a sheet of ply is correct or not by comparing exact positions and shapes of symbols in identifiers to the predetermined sequence, it is advantageous to determine that a sheet of ply is correct if it falls within a predetermined tolerance range. The tolerance may be determined by the error in individual symbol’s shape and / or position and / or orientation or be averaged across an identifier. Figures 10A - 10C show some example error in symbol positions which can be used in determining whether a sheet of ply is within tolerance or not. Each of Figures 10A - 10C shows a simplified set of image data, showing variations in symbol positions.

[0091] In Figure 11A, four symbols are represented by labels A-D, with vector arrows representing the difference in the actual position of a symbol to the position of a symbol in the identifier expected by its location in the sequence. Each symbol has been detected at positions which vary from the expected position by a different vector. The monitoring system may record positional errors for each symbol individually, or may monitor tolerance by comparing against the highest single error in any one direction, or an aggregate error metric may be calculated.

[0092] In the example of Figure 11A, symbol A has the highest single error in the X direction, and symbol D has the highest single error in the Z direction. Symbol D also has the highest overall magnitude of error, as it deviates from its expected position in both the X and Y directions. Across a single identifier, tolerance can be determined based on any or all of these highest single errors being less than a predetermined error tolerance.

[0093] Figure 11 B shows another error measurement method, where angular error is measured between the two most extreme errors against a nominal. In this example, symbol A has the largest error in the positive X directions, and symbol C has the largest error in the negative X direction. The angle theta between these two symbol errors can represent twisting in a sheet of ply.

[0094] Figure 11C again shows three symbols A-C, shown against the outline (e.g. the projection) of where the symbols should be positioned. In this example, symbol C has the largest single error when compared to the expected (nominal) positions for the symbols.

[0095] The monitoring system can be used record information about the lay-up process via the tracking unit as described above. The data stored in the tracking unit can include the errors from the nominal shown in Figures 10A-10C. This information can be used to monitor variations between parts, which can ensure consistency with production, and can be useful to provide feedback to the manufacture of the sheets of ply over time, helping with quality control.

[0096] Figure 12 shows an example view from a camera of a sheet of ply 220j in a mould (not shown), where part of the sheet of ply 220j has been obstructed by part of a crane 250. The sheet of ply 220j has six symbols 225a-225f which make up its identifier, however one of the symbols 225b is hidden from view of the image sensor by the crane 250. This sheet of ply 220j has been correctly placed, however the monitoring unit cannot see the whole of the identifier. To prevent such “correct” sheets of ply 220j as being mis-identified as “incorrect”, the monitoring unit may determine a sheet of ply as correct if at least a threshold percentage (e.g. >75%) of the identifier has been detected and determined as correct.

[0097] The threshold percentage may be calculated by considering whether a threshold percentage of detections zones include symbols. The monitoring unit could employ the threshold percentage if it can determine that the missing symbols are obstructed by performing image analysis to identify large well-known obstructions such as crane parts. If a symbol is only partially obstructed from view, the monitoring unit may use that part of the symbol in the determination as to whether the sheet of ply 220 is correct. To prevent mis-identification of sheets of ply as correct when symbols are missing from an identifier, the set of identifiers is ideally designed so that each identifier varies from other identifiers in that manufacturing environment by more than a single factor (e.g. by having at least two symbols shapes and / or positions which differ between each identifier. Therefore, using the example of Figure 12, as the symbol 225b is not visible because it is blocked by the crane 250, the monitoring unit removes the requirement to see that symbol 225b within the identifier for this sheet of ply 220 to be determined as correct. In a more sophisticated implementation the monitoring unit could take account of which symbols can be detected and perform an analysis as to whether they are sufficient to identify the sheet of ply unambiguously compared to the different sheet of ply types (e.g. fifteen in the example given earlier).

[0098] It will be appreciated that the various embodiments of the monitoring unit outlined above include steps carried out on a computing device. To be able to carry-out the invention, the monitoring unit is provided with data about the predefined sequence, and be set with the correct logical links between steps in the sequence to ensure that the sequence is stepped through as each sheet of ply is laid. The monitoring unit is therefore pre-programmed with information about the predetermined sequence.

[0099] The predetermined sequence can be taught to the monitoring unit via a test lay-up procedure, where for each step, the correct sheet of ply is carefully positioned in place, before data is saved for the symbols in the detected positions based on provided image data (e.g. from the image sensors). When the system is trained via a test lay-up procedure, before data is recorded all obstructions should be removed from the lay-up area to ensure no symbol is not detected.

[0100] The predetermined sequence can also be taught to the monitoring unit by inputting identifiers directly from other computing systems. For example, if the geometry of a part is defined or aided by computer aided design (CAD), the CAD geometry can be used to extract a predetermined sequence of identifiers. This may be via a full shell 3D calibration.

[0101] Whilst two distinct methods of teaching the predetermined sequence are outlined above, the skilled person will appreciate that a combination of both methods may be used to ensure any CAD input links to the reality of a calibration lay-up procedure. The programming of the predetermined sequence may also be linked to the production of the sheets of ply themselves to ensure symbol match between those being produced and those expected by the monitoring unit.

[0102] The finalised predetermined sequence may require altering after manufacture of multiple parts has begun. The monitoring system may include, for example as part of the user interface, a facility for a user to alter the predetermined sequence by e.g. by adding, deleting or moving sheets of ply from within the sequence.

[0103] Figure 13 is a flow diagram 300 showing a method for monitoring composite part layup in accordance with the invention. It will be appreciated that the method is applicable to the specific examples of the monitoring system outlined above. At step 310 image data is received from image sensors. As outlined above, the image data may be for specific regions of the lay-up area, e.g. detection zones, or may be image data for the whole lay-up area. In some examples, image data is acquired after a predetermined time has passed since a user input or previous cycle of the method.

[0104] Steps 320 and 330 include the determination as to whether a correct sheet of ply has been placed relative to a predefined sequence

[0105] At step 320 an identifier is detected in the image data. The identifier is detected from combination of symbols on each sheet of ply. The detection of an identifier may take place via a two-step process for example through a localiser as described above with reference to Figure 7A. When detection of an identifier is via a two-step process the first step in the process is to work out where in the image data the symbols are located, either via known detection zones, or by calculated detection zones. The second step in the process is detection of symbols in the detection zones. The detection of an identifier may include detection of at least a predetermined percentage of the identifier, to compensate for the view of image sensors being blocked as outlined above with reference to Figure 12. The detection of an identifier may also wait until symbols have been stationary for a predetermined number of image frames before proceeding to the next step, to allow for movement of sheets of ply across the field of view of the image sensors as outlined above with reference to Figures 4 and 7A.

[0106] At step 330, the detected identifier is compared to that required by the predefined sequence. This comparison includes whether the correct combination of symbols is detected, and can also include a determination as to whether the identifier is detected in the correct position relative to a mould and / or whether the sheet of ply is within tolerance. It will be appreciated therefore that the comparison of the identifier to the sequence can include three separately determinable variables, correct size / shape sheet of ply, correct location of sheet of ply, and sheet of ply being within tolerance (i.e. sequence, location and precision).

[0107] At step 340 an output is produced which indicates whether the sheet of ply was determined as correct, i.e. relative to the variables being measured. The output can include information about which variable was incorrect, or include information about which sheet of ply is next, or which sheet of ply should have been placed.

[0108] If the detected sheet of ply is determined to be incorrect, the method returns to receiving image data, and runs through the process again until the correct sheet of ply has been placed. As outlined above, in some examples an override symbol may be used. According to the outlined method, if the identifier in the image data is determined to be an override symbol the process continues according to the specific instructions indicated by the override symbol. For example, if the override symbol indicated that the ply should be treated as correct, the process will continue as though the correct sheet of ply has been identified.

[0109] If the detected sheet of ply is determined to be correct, the process can move to the next step in the predetermined sequence, and the process can re-start from step 310.

[0110] Where two sheets of ply have an overlapping portion as described above with reference to Figures 9A and 9B, if a two-step process is used they may effectively be considered as two distrinct steps in the sequence for determining whether the lower sheet of ply is correct, so that in the first step image data is received and analysed in comparison to the lower sheet of ply, and in the second step image data is received and analysed in comparison to the lower sheet of ply with the overlapping portion from the upper sheet of ply on top. Alternatively, in some embodiments, these may be combined into a single step, where the lower sheet of ply is analysed in the context of the overlapping portion already being folded back into place (e.g. as shown in Figure 9B) so that the overlap symbol is obscured, without first separately making a determination as to whether the lower sheet of ply is correct.

[0111] It will be appreciated by the skilled person that whilst the method steps are outlined in a particular order here, some of the steps may take place at different times within the process, or take place simultaneously with other steps. For example, there may be a continuous output in the form of projected images indicating where sheets of ply should be placed as outlined above with reference to Figure 7B. The receipt of image data may also be continuous, or data may only be collected at predetermined intervals. It will therefore be appreciated that the outlined method is by way of example only, and that embodiments may include other steps to include those features outlined with reference to the system described above.

[0112] The examples of the invention illustrated above describe how sheets of ply are placed into a mould during manufacture and how these individual sheets of ply are monitored.

[0113] It will be appreciated by those skilled in the art that the above examples equally apply to any individually laid structural element, of which a sheet of ply is just one example.

[0114] It will be appreciated by those skilled in the art that the invention has been illustrated by describing one or more specific aspects thereof, but is not limited to these aspects; many variations and modifications are possible, within the scope of the accompanying claims.

Claims

Claims1. A manufacturing monitoring system for monitoring manufacture of a composite part, the composite part being made from a plurality of structural elements laid into a mould in a predetermined sequence, each of the structural elements having an identifier; wherein each identifier comprises a plurality of symbols varying by shape and at least one of position and orientation on respective structural elements; wherein the system comprises: at least one image sensor configured to collect image data of the identifier when a structural element is laid into the mould; and a monitoring sub-system configured to: receive image data from the at least one image sensor; determine from the image data whether the structural element in the mould is correct by detecting an identifier on the structural element from the image data, and comparing the detected identifier to an expected identifier derived from the predetermined sequence; and output an indication as to whether the structural element is correct.

2. A manufacturing monitoring system according to claim 1, configured to look for a next identifier in the predetermined sequence after a correct structural element has been detected, and the indication has been output.

3. A manufacturing monitoring system according to any preceding claim, wherein the monitoring sub-system is configured to determine a structural element as correct by further determining that the structural element has been placed in a correct location within the mould.

4. A manufacturing monitoring system according to any preceding claim, configured to check whether said structural elements are within a predetermined tolerance with respect to one or more of: angle against a nominal direction, a shear angle, local distortion, distortion across a whole structural element, wrinkle creation, and tilt of a structural element in the mould.

5. A manufacturing monitoring system according to claim 3 or 4, configured to determine that a structural element is correct if the symbols of the identifier are eachdetected at a position within a predetermined tolerance compared with an expected position.

6. A manufacturing monitoring system according to any preceding claim, wherein the system is configured to store information regarding positions of detected symbols for each detected identifier.

7. A manufacturing monitoring system according to any of claims 4 to 6, wherein the monitoring subsystem is configured to make a two-stage determination: said first stage comprising determining that the structural element is correct relative to the predetermined sequence; and the said second stage comprising determining that the structural element is correctly located within the mould by the symbols thereof being positioned within one or more predetermined tolerances or respective expected positions.

8. A manufacturing monitoring system according to any preceding claim, wherein the monitoring sub-system comprises a localiser portion configured to determine at least one detection zone in the image data where one or more symbols in the identifier should be located for the expected correct structural element in the predetermined sequence; wherein the monitoring sub-system is configured to detect the identifier by searching for symbols in the detection zones.

9. A manufacturing monitoring system according to any of claims 1 to 7, wherein the monitoring sub-system comprises a localiser portion configured to determine at least one detection zone by searching the image data for any candidate symbols; wherein when a candidate symbol is located, the monitoring sub-system is configured to determine the shape and position of the symbol.

10. A manufacturing monitoring system according to any preceding claim, wherein the plurality of structural elements comprises an upper structural element and a lower structural element, the upper structural element being earlier in the predetermined sequence relative to the lower structural element and wherein an overlap symbol is provided on either the upper or lower structural element, the presence or absence of said overlap symbol indicating whether a portion of the upper structural element whichoverlaps the lower structural element has been replaced following placement of the lower structural element in the mould.

11. A manufacturing monitoring system according to claim 10, wherein the overlap symbol is provided on the lower structural element on a portion which is overlapped by the upper structural element and the monitoring sub-system is configured to carry out a checking step to determine that the overlap symbol is not visible.

12. A manufacturing monitoring system according to any preceding claim, wherein the at least one image sensor is configured to monitor an area where all structural elements might be laid.

13. A manufacturing monitoring system according to any preceding claim, configured to provide an indication of the expected structural element from the predetermined sequence.

14. A manufacturing monitoring system according to claim 13, wherein the indication of the expected structural element includes information about the identifier relating to the expected structural element.

15. A manufacturing monitoring system according to any preceding claim, comprising at least one projector configured to project an image of the expected identifier for the correct structural element in the predetermined sequence onto the mould.

16. A manufacturing monitoring system according to claim 13, wherein said indication as to whether the structural element is correct comprises a colour of the projected image.

17. A manufacturing monitoring system according to any preceding claim, wherein when the structural element has been determined to be incorrect, the indication in the output from the monitoring sub-system includes at least one of: information about when that structural element should have been placed in the predetermined sequence; and information on correct positioning of the structural element.

18. A manufacturing monitoring system according to any preceding claim, configured to make the determination as to whether a structural element is correct is made after at least one of: a predetermined time; or an identifier has been detected for more than a predetermined time; or an identifier has been detected for more than a predetermined number of image frames.

19. A manufacturing monitoring system according to any preceding claim, configured to make said determination as to whether the structural element is correct only after a user provides an input to indicate that the structural element is considered to be ready.

20. A manufacturing monitoring system according to any preceding claim, wherein the system comprises a user interface configured to display the indication from the monitoring subsystem as to whether the structural element is correct to a user.

21. A manufacturing monitoring system according to any preceding claim, wherein the monitoring sub-system is configured to determine that a structural element is correct if at least a predetermined threshold percentage of the identifier has been determined as correct relative compared to the expected identifier.

22. A manufacturing monitoring system according to any preceding claim, wherein the monitoring sub-system is configured to determine an identifier as correct if enough of the identifier has been detected to be able to exclude the identifier being any other from a pre-determined set of identifiers.

23. A manufacturing monitoring system according to any preceding claim, configured to allow an authenticated user to provide an input to identify said structural element as correct.

24. A manufacturing monitoring system according to any preceding claim, wherein the monitoring sub-system is configured to detect at least one override symbol and follow a predetermined instruction.

25. A manufacturing monitoring system according to claim 22, wherein if an override symbol has been detected, the monitoring sub-system is further configuredcheck for a second placement of the override symbol on the structural element, or to check for a second override symbol to confirm the instruction given by the override symbol.

26. A method for monitoring of monitoring composite part lay-up, the composite part being made from a plurality of structural elements laid into a mould in a predetermined sequence, each of the structural elements having an identifier; wherein each identifier comprises a plurality of symbols varying by shape and at least one of position and orientation on respective structural elements; wherein the method comprises: receiving image data of the identifier from at least one image sensor; determining from the image data whether the structural element in the mould is correct by: detecting an identifier on the structural element from the image data; and comparing the detected identifier to an expected identifier derived from the predetermined sequence; and outputting an indication as to whether the structural element is correct.

27. A computer software product or a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out the method according to claim 24.

Images