Manufacturing System Change Classification

Abstract

The present invention provides a computing system including one or more processing units configured to receive manufacturing change specifications for proposed changes to a manufacturing system. [Solution] In the computing system 10, one or more processing units 12 have a first trained machine learning model 30 that calculates one or more predicted change categories 32 of the proposed change 22, a second trained machine learning model 40 that calculates a change size classification 42 related to the proposed change 22 and calculates the change size classification 42 based at least partially on one or more predicted change categories 32, and then output one or more predicted change categories 32 and the change size classification 42 to the user interface 50.

Description

Technical Field

[0001] The present disclosure relates to the development of manufacturing systems and processes.

Background Art

[0002] Aircraft manufacturing systems and processes are very complex and may include thousands of different equipment components and manufacturing steps. As a result, changes to a part of an aircraft manufacturing system or process can potentially affect many other components or steps. Before such changes are made, they are considered by Design Approval Engineers (DAEs) to determine their downstream effects. Considering changes to aircraft manufacturing and checking dependencies is a time-consuming task that can slow down the overall process of aircraft manufacturing system and process development.

[0003] Aircraft manufacturing is also an area where changes to manufacturing systems and processes may require regulatory approval. Due to the complexity of aircraft manufacturing systems and processes, determining whether a change requires regulatory approval can be a difficult and time-consuming task for the DAE.

Summary of the Invention

Means for Solving the Problems

[0004] According to one aspect of the present disclosure, a computing system is provided which includes one or more processing units configured to receive manufacturing change specifications for proposed changes to a manufacturing system. In a first trained machine learning model, one or more processing units are further configured to compute one or more predicted change categories for the proposed changes. In a second trained machine learning model, one or more processing units are further configured to compute a change magnitude classification associated with the proposed changes. The second trained machine learning model is configured to compute the change magnitude classification based at least in part on one or more predicted change categories. One or more processing units are further configured to output one or more predicted change categories and change magnitude classifications to a user interface. [Brief explanation of the drawing]

[0005] [Figure 1] An exemplary computing system in which one or more processing units receive manufacturing change specifications is schematically shown according to one exemplary embodiment. [Figure 2] The example in Figure 1 schematically illustrates a computing system in which one or more processing units are configured to train a first pre-trained machine learning model. [Figure 3] The example in Figure 2 schematically illustrates a computing system in which one or more processing units are configured to train a second pre-trained machine learning model. [Figure 4] Figure 1 shows an illustrative table of data included in the manufacturing change specification and data output by the first trained machine learning model. [Figure 5] Figure 1 illustrates a computing system in an example where additional training is performed on the first and second pre-trained machine learning models. [Figure 6A] Figure 1 shows a flowchart illustrating a method for classifying proposed changes to a manufacturing system in conjunction with a computing system. [Figure 6B]Figure 6A shows additional steps of the method that can be performed to train the first and second pre-trained machine learning models. [Figure 6C] The following shows additional steps to the method in Figure 6A, which can be performed in several examples to update a machine learning model. [Figure 7] A schematic diagram illustrates an exemplary computing environment in which the computing system in Figure 1 can be instantiated. [Modes for carrying out the invention]

[0006] To address the aforementioned challenges, a computing system 10 is provided, as shown in Figure 1. Figure 1 schematically shows an exemplary computing system 10 including one or more processing units 12 and one or more memory devices 14. The one or more processing units 12 may include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more dedicated hardware accelerators. The one or more memory devices 14 may include volatile memory and / or non-volatile storage devices. The computing system 10 can be implemented as a single physical computing device or as multiple communicably coupled physical computing devices, such as in a server-client configuration. The computing system 10 shown in Figure 1 further includes one or more input devices 16 and one or more output devices 18, which allow a user to interact with the computing system 10.

[0007] As shown in Figure 1, one or more processing units 12 are configured to receive a manufacturing modification specification 20 of a proposed modification 22 for a manufacturing system 24. For example, the proposed modification 22 may be a proposed modification for one or more manufacturing devices 25 included in the manufacturing system 24. Additionally or alternatively, the proposed modification 22 may be a proposed modification for one or more manufacturing steps 27 of a manufacturing process 26 performed in the manufacturing system 24. In the example in Figure 1, the manufacturing system 24 is configured to manufacture an aircraft 28. In this example, the proposed modification 22 may be a proposed modification for one or more components 29 of the aircraft 28 manufactured in the manufacturing system 24.

[0008] In the first trained machine learning model 30, one or more processing units 12 are further configured to compute one or more predicted change categories 32 of the proposed changes 22. For example, one or more predicted change categories 32 could specify one or more manufacturing devices 25 included in the manufacturing system 24. As another example, one or more predicted change categories 32 could specify one or more manufacturing steps 27 of the manufacturing process 26. In an example where the manufacturing system 24 is configured to manufacture an aircraft 28, one or more predicted change categories 32 could specify one or more components 29 of the aircraft 28 manufactured by the manufacturing system 24.

[0009] The first trained machine learning model 30 is configured to receive manufacturing change specifications 20 as input. The first trained machine learning model 30 shown in the example in Figure 1 is a multi-label classifier model that classifies the proposed changes 22 according to whether they belong to each of several potential change categories. The first trained machine learning model 30 could be, for example, an XGBoost model. As another example, the first trained machine learning model 30 could be a random forest model. In some examples, the first trained machine learning model 30 outputs one or more predicted change categories as an n-dimensional boolean vector, where n is the number of potential change categories.

[0010] Figure 2 schematically shows a computing system 10 when one or more processing units 12 are configured to train a first pre-trained machine learning model 30. The first pre-trained machine learning model 30 is trained using first training data 60 which includes a plurality of prior manufacturing change specifications 62. The plurality of prior manufacturing change specifications 62 represent respective prior changes to the manufacturing system 24. Furthermore, the first training data 60 includes a plurality of training change categories 64 for each of the prior changes. The training change categories 64 may be change categories previously selected by one or more DAEs that have reviewed the prior manufacturing change specifications 62.

[0011] In some examples, for each of the prior art modification specifications 62, one or more processing units 12 are further configured to compute one or more key phrases 66 associated with that prior art modification specification 62. In such examples, the first training data 60 further includes the key phrases 66. As shown in the example in Figure 2, the key phrases 66 can be computed by a natural language processing (NLP) model 68 by inputting the prior art modification specifications 62 into the NLP model 68, at least in part. Including the key phrases 66 in the first training data 60 can make the output of the first trained machine learning model 30 more consistent in examples where there are stylistic and / or template differences among the multiple prior art modification specifications 62.

[0012] In each of the multiple training iterations 78, one or more processing units 12 are further configured to input a batch of prior art modification specifications 62, along with the corresponding key phrases 66, into the first machine learning model 70. The first machine learning model 70 is configured to compute one or more respective training time predicted change categories 72 for each of the prior art modification specifications 62. One or more processing units 12 are further configured to compute each value of the first loss function 74, at least in part, based on the training time predicted change categories 72 and the training change categories 64. For example, the first loss function 74 may be a Hamming loss function.

[0013] One or more processors 12 are further configured to compute a model update 76 based at least in part on the value of a first loss function 74. This model update 76 can be computed via stochastic gradient descent (SGD). One or more processors 12 are further configured to apply the model update 76 to the first machine learning model 70. Thus, one or more processors 12 are configured to train the first machine learning model 70 over a plurality of training iterations 78 to obtain a first trained machine learning model 30.

[0014] Returning to the example in Figure 1, one or more processing units 12 are further configured to compute a change size classification 42 related to the proposed change 22 in a second trained machine learning model 40. In some examples, the change size classification 42 may be a binary classification. In such examples, the change size classification 42 may be a classification of whether the proposed change 22 is a major change (a change requiring regulatory approval) or a minor change (a change not requiring regulatory approval). In some examples, the change size classification 42 is a multidimensional binary classification indicating whether each of several categories (e.g., proposed change categories 32) is large or small.

[0015] The second trained machine learning model 40 is configured to receive the manufacturing change specification 20 as input. Furthermore, the second trained machine learning model 40 is configured to calculate a change magnitude classification 42 based at least in part on one or more predicted change categories 32 calculated for the proposed change 22 by the first trained machine learning model 30. Since changes to some manufacturing equipment 25, manufacturing steps 27, and / or components 29 are likely to be larger than changes to others, the second trained machine learning model 40 can calculate the change magnitude classification 42 more accurately by using one or more predicted change categories 32 as input. The second trained machine learning model 40 may be, for example, an XGBoost model.

[0016] Figure 3 schematically shows a computing system 10 during training of a second pre-trained machine learning model 40. One or more processors 12 are configured to train the second pre-trained machine learning model 40 using second training data 80 which includes a plurality of prior manufacturing change specifications 62. The second training data 80 further includes a plurality of training change categories 64 of the prior changes indicated in the prior manufacturing change specifications 62. In an example where one or more processors 12 are configured to compute key phrases 66 associated with the prior manufacturing change specifications 62, the key phrases 66 may also be included in the second training data 80. The second training data 80 further includes each of the plurality of prior change size classifications 82 associated with the prior changes. For example, the prior change size classifications 82 may be change size classifications assigned to the prior manufacturing change specifications 62 by one or more DAEs.

[0017] In each of the multiple training iterations 98, one or more processing units 12 are further configured to input batches of prior manufacturing change specifications 62, training change categories 64, and key phrases 66 into a second machine learning model 90, the second machine learning model being configured to compute each training time change magnitude classification 92 associated with the prior manufacturing change specifications 62. One or more processing units 12 are further configured to compute a value for a second loss function 94 based at least in part on the training time change magnitude classifications 92 and the prior change magnitude classifications 82.

[0018] One or more processors 12 are further configured to compute a model update 96 (e.g., via SGD) based at least in part on a second loss function 94 and to apply the model update 96 to the second machine learning model 90. Thus, one or more processors 12 are configured to update the second machine learning model 90 over a plurality of training iterations 98 in order to obtain a second trained machine learning model 40.

[0019] Returning to the example of FIG. 1, following the step of calculating a scale of change classification 42 associated with one or more predicted change categories 32 and change proposals 22, one or more processing devices 12 are further configured to output one or more predicted change categories 32 and scale of change classifications 42 to a user interface 50. For example, the user interface 50 can be a graphical user interface (GUI) displayed on one or more display devices included in one or more output devices 18 within the computing system 10.

[0020] FIG. 4 shows an exemplary mapping from the data included in the manufacturing change specifications 20 to predicted change categories 32 associated with those manufacturing change specifications 20. The first table 100 shown in FIG. 4 includes respective change identifiers associated with the change proposals 22. The first table 100 further includes a "payload keyword count" column indicating the number of key phrases 66 of the manufacturing change specifications 20 related to the aircraft payload. The first table 100 further includes a "change type" column including respective change type tags assigned to the change proposals 22 of the manufacturing change specifications 20. Further, the first table 100 includes an "ATA chapter" column indicating the specific provisions described in the manufacturing change specifications 20 as being related to the change proposals 22.

[0021] FIG. 4 further shows a second table 102 including a boolean indicator (either 0 or 1) for the change proposals 22. The boolean indicator for the change proposals 22 specifies whether the first trained machine learning model 30 identified that the change proposals 22 belong to each of the respective plurality of predicted change categories. Thus, in the example of FIG. 4, the rows of the second table 102 are n-dimensional vectors of boolean values output by the first trained machine learning model 30. The columns of the second table 102 shown in the example of FIG. 4 are "interior", "electrical", "EME", "cabin", "aerodynamics", "avionics".

[0022] The following table shows the experimental data regarding the performance of the first trained machine learning model 30. The following table compares the performance of multiple different statistical models including XGBoost (XGB), Random Forest (RF), k-Nearest Neighbors (KNN), Support Vector Machine (SVM), Decision Tree, and Logistic.

[0023]

Table 1

[0024] The above table shows that XGB has the highest performance regarding recall and F1, and the second highest performance regarding Hamming loss and precision. The RF model has the highest performance regarding Hamming loss and precision, and the second highest performance regarding recall and F1.

[0025] The following table shows the experimental data regarding the performance of the second trained machine learning model 40. The following table compares the performance of multiple different machine learning models including weighted XGB, unweighted XGB, Multilayer Perceptron (MLP), AdaBoost, weighted Support Vector Classifier (SVC), and balanced RF.

[0026]

Table 2

[0027] The above table shows that unweighted XGB has the highest performance regarding accuracy, precision, and F1, and the second highest performance regarding recall.

[0028] Figure 5 shows a computing system 10 in an example where additional training is performed on the first trained machine learning model 30 and the second trained machine learning model 40 after deployment. In the example of Figure 5, one or more processing units 12 are further configured to receive additional training data 114, which includes one or more ground truth change categories 110 and ground truth change magnitude classifications 112 associated with proposed changes 22. The one or more ground truth change categories 110 and ground truth change magnitude classifications 112 are received as user feedback via the user interface 50, following one or more processing units 12 that output one or more predicted change categories 32 and change magnitude classifications 42. The user can input one or more ground truth change categories 110 and ground truth change magnitude classifications 112 using one or more input devices 16. In some examples, one or more Grand Truth change categories 110 and Grand Truth change magnitude classifications 112 are endorsements of one or more predicted change categories and change magnitude classifications 42 calculated by the first trained machine learning model 30 and the second trained machine learning model 40, respectively.

[0029] One or more processors 12 are further configured to perform additional training on the first trained machine learning model 30 and the second trained machine learning model 40 using additional training data 114. Thus, one or more processors 12 are configured to compute the first further trained machine learning model 116 and the second further trained machine learning model 118, which are updated to reflect one or more ground truth change categories 110 and ground truth change magnitude classifications 112 selected by the user. The additional training of the first trained machine learning model 30 and the second trained machine learning model 40 can be performed as described above with reference to Figures 2 and 3.

[0030] Figure 6A shows a flowchart of Method 200 for use with a computing system to classify proposed changes to a manufacturing system. In step 202, Method 200 includes the step of receiving a manufacturing change specification for a proposed change to the manufacturing system. For example, a proposed change may be a change to one or more manufacturing devices and / or one or more manufacturing steps. In some examples, the manufacturing system is configured to manufacture an aircraft. A proposed change may be a change to one or more components of the aircraft. A manufacturing change specification may be a document uploaded in a user interface.

[0031] In step 204, method 200 further includes the step of computing one or more predicted change categories of proposed changes in a first trained machine learning model. For example, the first trained machine learning model may be an XGBoost model or a random forest model. In some examples, the first trained machine learning model outputs an n-dimensional vector of Boolean values, where n is the number of potential change categories. One or more predicted change categories may specify one or more manufacturing devices included in a manufacturing system, one or more manufacturing steps in a manufacturing process, or one or more components of an aircraft or other device manufactured in the manufacturing system.

[0032] In step 206, method 200 further includes the step of computing a change size classification related to the proposed change in a second trained machine learning model. The second trained machine learning model is configured to compute the change size classification based at least partially on one or more predicted change categories. Furthermore, the second trained machine learning model is configured to receive manufacturing change specifications as input. In some examples, the change size classification is a binary classification. For example, a binary classification could be a classification of whether the proposed change requires regulatory approval. In some examples, the change size classification may be a multidimensional binary classification computed on multiple different categories, such as the predicted change categories computed in step 204.

[0033] In step 208, method 200 further includes the step of outputting one or more predicted change categories and change magnitude classifications to the user interface. Thus, the DAE can consider the predicted change categories and change magnitude classifications predicted by the machine learning model.

[0034] Figure 6B shows additional steps of Method 200 that can be performed to train a first pre-trained machine learning model and a second pre-trained machine learning model. In step 210, Method 200 includes the step of training the first pre-trained machine learning model using the first training data, the first training data including multiple prior manufacturing change specifications and multiple training change categories for each of them. The prior manufacturing change specifications represent each prior change to the manufacturing system, and the training change categories are associated with the prior changes.

[0035] In some examples, step 210 further includes step 212. In step 212, for each prior manufacturing change specification, method 200 further includes the step of computing one or more key phrases related to that prior manufacturing change specification in a natural language processing model. The first training data further includes key phrases in the example in which step 212 is performed.

[0036] In step 214, method 200 further includes the step of training a second trained machine learning model using second training data. The second training data includes multiple prior manufacturing change specifications included in the first training data. Furthermore, the second training data includes multiple training change categories of prior changes indicated in the prior manufacturing change specifications. The second training data further includes multiple prior change size classifications for each of the prior changes associated with the prior changes. In the example in which step 212 is performed, key phrases may also be included in the second training data.

[0037] Figure 6C shows additional steps of Method 200 that can be performed in several examples to update a machine learning model. In step 216, Method 200 further includes receiving one or more Grand Truth Change Categories and Grand Truth Change Size Classifications via a user interface. The one or more Grand Truth Change Categories and Grand Truth Change Size Classifications are received, following the output of one or more Predicted Change Categories and Change Size Classifications.

[0038] In step 218, method 200 further includes the step of performing additional training on the first and second trained machine learning models using additional training data including one or more ground truth change categories and ground truth change magnitude classifications. Thus, the first and second trained machine learning models can be updated with user feedback.

[0039] Using the systems and methods described above, proposed changes to the manufacturing process are programmatically classified using machine learning models. These machine learning models predict which parts of the manufacturing system or process will be affected by the proposed changes, as well as the scale of the changes. The experimental results described above demonstrate that the machine learning models achieve high accuracy in these predictions. Therefore, the systems and methods described above can significantly reduce the amount of time spent by DAEs when determining the impact of proposed changes on the manufacturing system.

[0040] The methods and processes described herein can be linked to a computing system of one or more computing devices. In particular, such methods and processes can be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0041] Figure 7 schematically illustrates a non-limiting embodiment of computing system 300 that can perform one or more of the methods and processes described above. Computing system 300 is shown in a simplified form. Computing system 300 can embody computing system 10 described above and shown in Figure 1. Components of computing system 300 may include one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0042] The computing system 300 includes a processing circuit 302, a volatile memory 304, and a non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or other components not shown in Figure 7.

[0043] The processing circuit 302 typically includes one or more logic processors, which are physical devices that execute instructions. For example, a logic processor can be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical configurations. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise reach desired results.

[0044] A logic processor may include one or more physical processors that execute software instructions. Additionally or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware implementation logic or firmware instructions. The processor of the processing circuit 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential processing, parallel processing, and / or distributed processing. Individual components of the processing circuit 302 may optionally be distributed across two or more separate devices, which may be located remotely and / or configured for cooperative processing. For example, embodiments of the computing system 300 disclosed herein may be virtualized and executed by remotely accessible network computing devices configured in a cloud computing configuration. In such cases, it will be understood that these virtualized embodiments run on different physical logic processors on different machines. These different physical logic processors on different machines will be understood to be collectively encompassed by the processing circuit 302.

[0045] The non-volatile memory device 306 includes one or more physical devices that hold instructions executable by a processing circuit to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile memory device 306 can be transformed, for example, to hold different data.

[0046] The non-volatile storage device 306 may include a removable and / or built-in physical device. The non-volatile storage device 306 may include optical memory, semiconductor memory, and / or magnetic memory, or other mass storage technology. The non-volatile storage device 306 may include non-volatile devices, dynamic devices, static devices, read / write devices, read-only devices, sequential access devices, position-addressable devices, file-addressable devices, and / or content-addressable devices. It will be understood that the non-volatile storage device 306 is configured to retain instructions even when power to the non-volatile storage device 306 is cut off.

[0047] The volatile memory 304 may include a physical device that also contains random access memory. The volatile memory 304 is typically used by the processing circuit 302 to temporarily store information during the processing of software instructions. It will be understood that the volatile memory 304 typically does not continue to store instructions when power to the volatile memory 304 is cut off.

[0048] The processing circuit 302, the volatile memory 304, and the non-volatile storage device 306 can be integrated into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), programmable integrated circuits (PASICs) and application-specific integrated circuits (ASICs), programmable standard products (PSSPs) and application-specific standard products (ASSPs), systems on a chip (SOC), and complex programmable logic devices (CPLDs).

[0049] The terms “module,” “program,” and “engine” can be used to describe a mode of computing system 300 that is typically implemented in software by a processor to perform a specific function using a portion of volatile memory, the function including a transformation process that specifically configures the processor to perform that function. Thus, a module, program, or engine can be instantiated via a processing circuit 302 that executes instructions held by a non-volatile storage device 306 using a portion of volatile memory 304. It will be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated by different applications, services, code block, object, routine, API, function, etc. The terms “module,” “program,” and “engine” can encompass individual or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0050] If included, the display subsystem 308 can be used to present a visual representation of the data held by the non-volatile memory 306. The visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein modify the data held by the non-volatile memory and thus transform the state of the non-volatile memory, the state of the display subsystem 308 can also be transformed to visually represent the changes in the underlying data. The display subsystem 308 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the processing circuitry 302, volatile memory 304, and / or non-volatile memory 306 in a shared enclosure, or such display devices may be peripheral display devices.

[0051] If included, the input subsystem 310 may include or interface with one or more user input devices such as a keyboard, mouse, touchscreen, camera, or microphone.

[0052] If included, the communication subsystem 312 may be configured to connect the various computing devices described herein to each other and to other devices in a communicative manner. The communication subsystem 312 may include wired and / or wireless communication devices compliant with one or more different communication protocols. In non-limiting examples, the communication subsystem may be configured for communication over a wired or wireless local area network or wide area network, a broadband cellular network, etc. In some embodiments, the communication subsystem may enable the computing system 300 to send and / or receive messages to and from other devices over a network such as the Internet.

[0053] Furthermore, this disclosure includes configurations as follows:

[0054] Example 1. A computing system comprising one or more processing units, the one or more processing units receiving a manufacturing change specification for a proposed change to a manufacturing system, calculating one or more predicted change categories for the proposed change in a first trained machine learning model, and calculating a change magnitude classification related to the proposed change in a second trained machine learning model, wherein the second trained machine learning model is configured to calculate the change magnitude classification based at least partially on one or more predicted change categories, and the one or more processing units are configured to output one or more predicted change categories and change magnitude classifications to a user interface.

[0055] Example 2. The computing system described in Example 1, wherein one or more processing units are further configured to train a first trained machine learning model using first training data, the first training data comprising a plurality of prior manufacturing change specifications indicating each prior change to a manufacturing system, and a plurality of training change categories for each of the prior changes.

[0056] Example 3. The computing system as in Example 2, wherein for each of the prior art modification specifications, one or more processing units are further configured to compute one or more key phrases related to that prior art modification specification in a natural language processing model, and the first training data further comprises key phrases.

[0057] Example 4. One or more processing units are further configured to train a second trained machine learning model using second training data, wherein the second training data includes a plurality of prior manufacturing change specifications, a plurality of training change categories for the prior changes indicated in the prior manufacturing change specifications, and a plurality of prior change size classifications for each of the prior changes, as described in Example 2 or 3 of the computing system.

[0058] Example 5. The modified category classification is a multidimensional binary classification, as described in any of Examples 1 through 4 of the computing system.

[0059] Example 6. A computing system according to any one of Examples 1 to 5, wherein one or more processing units are further configured to receive one or more ground truth change categories and ground truth change size classifications, following the step of outputting one or more predicted change categories and change size classifications via a user interface, and to perform additional training on a first trained machine learning model and a second trained machine learning model using additional training data including one or more ground truth change categories and ground truth change size classifications.

[0060] Example 7. A computing system according to any one of Examples 1 to 6, in which one or more predictive change categories specify one or more manufacturing equipment included in the manufacturing system.

[0061] Example 8. A computing system as described in any of Examples 1 to 7, in which one or more predictive change categories specify one or more manufacturing steps in a manufacturing process.

[0062] Example 9. The computing system described in Example 1, wherein the manufacturing system is configured to manufacture an aircraft, and one or more predictive change categories specify one or more components of the aircraft to be manufactured by the manufacturing system.

[0063] Example 10. The first trained machine learning model outputs an n-dimensional vector of Boolean values, where n is the number of potential change categories, as described in any of Examples 1 through 9 of the computing system.

[0064] Example 11. A method for use with a computing system, comprising: receiving a manufacturing change specification for a proposed change to a manufacturing system; computing one or more predicted change categories for the proposed change in a first trained machine learning model; computing a change magnitude classification related to the proposed change in a second trained machine learning model, wherein the second trained machine learning model is configured to compute the change magnitude classification based at least partially on one or more predicted change categories; and outputting one or more predicted change categories and the change magnitude classification to a user interface.

[0065] Example 12. The method of Example 11, further comprising the step of training a first trained machine learning model using first training data, wherein the first training data includes a plurality of prior manufacturing change specifications indicating each prior change to a manufacturing system, and a plurality of training change categories for each of the prior changes.

[0066] Example 13. The method of Example 12, further comprising the step of computing one or more key phrases related to each of the prior manufacturing change specifications in a natural language processing model, wherein the first training data further comprises the key phrases.

[0067] Example 14. The method as in Example 12 or 13, further comprising the step of training a second trained machine learning model using second training data, wherein the second training data includes multiple prior manufacturing change specifications, multiple training change categories for the prior changes indicated in the prior manufacturing change specifications, and each of the multiple prior change size classifications associated with the prior changes.

[0068] Example 15. Change scale classification is a multidimensional binary classification, using any of the methods described in Examples 11 to 14.

[0069] Example 16. The method of any one of Examples 11 to 15, further comprising the steps of: outputting one or more predicted change categories and change magnitude classifications via a user interface; receiving one or more ground truth change categories and ground truth change magnitude classifications; and performing additional training on a first trained machine learning model and a second trained machine learning model using additional training data including one or more ground truth change categories and ground truth change magnitude classifications.

[0070] Example 17. The method according to any one of Examples 11 to 16, wherein one or more predicted change categories specify one or more manufacturing equipment included in the manufacturing system.

[0071] Example 18. One or more predicted change categories specify one or more manufacturing steps in the manufacturing process, as described in any of Examples 11 to 17.

[0072] Example 19. The method according to any one of Examples 11 to 18, wherein the manufacturing system is configured to manufacture an aircraft, and one or more predictive change categories specify one or more components of the aircraft to be manufactured by the manufacturing system.

[0073] Example 20. A computing system comprising one or more processing units, the one or more processing units being configured to train a first trained machine learning model using first training data comprising a plurality of prior manufacturing change specifications indicating each prior change to a manufacturing system, a plurality of training change categories for each prior change, and a plurality of key phrases associated with each prior manufacturing change specification; to train a second trained machine learning model using second training data comprising a plurality of prior manufacturing change specifications, a plurality of training change categories for the prior changes indicated in the prior manufacturing change specifications, a plurality of prior change magnitude classifications associated with each prior change, and a plurality of key phrases; to receive a manufacturing change specification for a proposed change to a manufacturing system; to calculate one or more predicted change categories for the proposed change in the first trained machine learning model; to calculate a change magnitude classification associated with the proposed change in the second trained machine learning model, at least partially based on one or more predicted change categories; and to output one or more predicted change categories and change magnitude classifications to a user interface.

[0074] As used herein, "and / or" is defined as inclusive or ∨, as specified by the truth table below.

[0075] [Table 3]

[0076] The configurations and / or methods described herein are illustrative in nature, and it will be understood that these particular embodiments or examples should not be considered restrictive, as numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or in any order. Similarly, the order of the processes described above may be changed.

[0077] The subject matter of this disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems, and configurations disclosed herein, as well as all other features, functions, actions, and / or characteristics, and all their equivalents. [Explanation of Symbols]

[0078] 10 Computing system, 12 Processing unit, 14 Memory unit, 16 Input unit, 18 Output unit, 20 Manufacturing change specifications, 22 Proposed changes, 24 Manufacturing system, 25 Manufacturing equipment, 26 Manufacturing process, 27 Manufacturing steps, 28 Aircraft, 29 Components, 30 First machine learning model, 32 Predicted change categories, 32 Proposed change categories, 40 Second machine learning model, 42 Change scale classification, 42 Change category classification, 50 User interface, 60 First training data, 62 Preceding manufacturing change specifications, 64 Training change categories, 66 Key phrases, 68 Model, 68 Natural language processing model, 70 First machine learning model, 72 Training time predicted change categories, 74 First loss function, 76 Model update, 78 Training iterations, 80 Second training data, 82 Preceding change scale classification, 90 Second machine learning model, 92 Training time change scale classification, 94 Second loss function, 96 Model update, 98 training iterations, 110 ground truth change categories, 112 ground truth change scale classification, 114 additional training data, 116 first further trained machine learning model, 118 second further trained machine learning model, 200 method, 300 computing system, 302 processing circuitry, 304 volatile memory, 306 non-volatile memory, 308 display subsystem, 310 input subsystem, 312 communication subsystem

Claims

1. A computing system (10), The system comprises one or more processing units (12), and the one or more processing units (12) are Having received the manufacturing change specifications (20) for the proposed changes (22) to the manufacturing system (24), In the first trained machine learning model (30), one or more predicted change categories (32) of the proposed changes (22) are calculated. In the second trained machine learning model (40), a change size classification (42) related to the proposed change (22) is calculated, wherein the second trained machine learning model (40) is configured to calculate the change size classification (42) based at least partially on one or more predicted change categories (32), and A computing system (10) configured to output one or more predicted change categories (32) and the change scale classification (42) to a user interface (50).

2. The one or more processing devices (12) are further configured to train the first trained machine learning model (30) using the first training data (60), and the first training data (60) is Multiple prior manufacturing change specifications (62) indicating each prior change to the manufacturing system (24), The computing system (10) according to claim 1, comprising each of the preceding multiple training modification categories (64) of the preceding modifications.

3. For each of the aforementioned prior manufacturing change specifications (62), the one or more processing units (12) are further configured to compute one or more key phrases (66) related to the prior manufacturing change specification (62) in a natural language processing model (68), The first training data (60) further includes the key phrase (66), The computing system (10) according to claim 2.

4. The one or more processing units (12) are further configured to train the second trained machine learning model (40) using the second training data (80), and the second training data (80) is The aforementioned multiple prior manufacturing change specifications (62), The preceding manufacturing change specifications (62) include a number of training change categories (64) of the preceding changes, The computing system (10) according to claim 2, comprising a plurality of prior change scale classifications (82) related to the prior changes.

5. The computing system (10) according to claim 1, wherein the change scale classification (42) is a multidimensional binary classification.

6. The one or more processing devices (12) are, Following the step of outputting one or more predicted change categories (32) and change magnitude classifications (42) via the user interface (50), the system receives one or more ground truth change categories (110) and ground truth change magnitude classifications (112). The computing system (10) according to claim 1, further configured to perform additional training on the first trained machine learning model (30) and the second trained machine learning model (40) using additional training data (114) including one or more ground truth change categories (110) and the ground truth change scale classification (112).

7. The computing system (10) according to claim 1, wherein the one or more predictive change categories (32) specify one or more manufacturing equipment (25) included in the manufacturing system (24).

8. The computing system (10) according to claim 1, wherein the one or more predictive change categories (32) specify one or more manufacturing steps (27) of a manufacturing process (26).

9. The manufacturing system (24) is configured to manufacture an aircraft (28), The one or more predicted change categories (32) specify one or more components (29) of the aircraft (28) manufactured by the manufacturing system (24), The computing system (10) according to claim 1.

10. The computing system (10) according to claim 1, wherein the first trained machine learning model (30) outputs an n-dimensional vector of Boolean values, where n is the number of potential change categories.

11. A method (200) for use with a computing system, Step (202) involves receiving the manufacturing change specifications for proposed changes to the manufacturing system, Step (204) of calculating one or more predicted change categories of the proposed changes in the first trained machine learning model, A step of calculating a change size classification related to the proposed change in a second trained machine learning model, wherein the second trained machine learning model is configured to calculate the change size classification based at least partially on one or more predicted change categories (206). A method (200) comprising the step of outputting the one or more predicted change categories and the change scale classification to a user interface (208).

12. The process further includes the step of training the first trained machine learning model using the first training data, wherein the first training data is: Multiple prior manufacturing change specifications, each indicating a prior change to the aforementioned manufacturing system, The method (200) of claim 11, comprising each of the multiple training modification categories of the preceding modification (210).

13. The method (200) of claim 12, further comprising the step of computing one or more key phrases related to each of the preceding manufacturing change specifications in a natural language processing model, wherein the first training data further comprises the key phrases (212).

14. The process further includes the step of training the second trained machine learning model using second training data, wherein the second training data is: The aforementioned multiple pre-production change specifications, The plurality of training change categories of the preceding changes as shown in the preceding manufacturing change specifications, The method (200) of claim 12, comprising a plurality of classifications of the magnitude of prior modifications related to the prior modifications (214) mentioned above.

15. The method according to claim 11 (200), wherein the change scale classification is a multidimensional binary classification.

16. The steps include: (216) receiving one or more ground truth change categories and ground truth change scale classifications, following the step of outputting one or more predicted change categories and the change magnitude classification via the user interface; The step (218) further includes performing additional training on the first trained machine learning model and the second trained machine learning model using additional training data including the one or more ground truth change categories and the ground truth change magnitude classification, The method according to claim 11 (200).

17. The method according to claim 11 (200), wherein the one or more predictive change categories specify one or more manufacturing equipment included in the manufacturing system.

18. The method according to claim 11 (200), wherein the one or more predicted change categories specify one or more manufacturing steps in a manufacturing process.

19. The manufacturing system is configured to manufacture aircraft, The one or more predicted change categories specify one or more components of the aircraft manufactured by the manufacturing system. The method according to claim 11 (200).

20. A computing system (10), The system comprises one or more processing units (12), The one or more processing devices (12) are, Multiple prior manufacturing change specifications (62) that show each prior change to the manufacturing system (24), Each of the preceding changes has multiple training change categories (64), Each of the aforementioned advance manufacturing change specifications (62) is associated with a number of key phrases (66), A first trained machine learning model (30) is trained using the first training data (60) which includes the following: The aforementioned multiple prior manufacturing change specifications (62), The preceding manufacturing change specifications (62) include the multiple training change categories (64) of the preceding changes, Each of the multiple classifications of the scale of the preceding changes related to the preceding changes (82), The aforementioned multiple key phrases (66), A second trained machine learning model (40) is trained using the second training data (80) which includes the following: Having received the manufacturing change specifications (20) for the proposed changes (22) to the manufacturing system (24), In the first trained machine learning model (30), one or more predicted change categories (32) of the proposed changes (22) are calculated. In the second trained machine learning model (40), a change size classification (42) related to the proposed change (22) is calculated based at least partially on one or more predicted change categories (32). The one or more predicted change categories (32) and the change scale classification (42) are output to the user interface (50). It is structured in such a way. Computing system (10).

Images