Neural network system for distributed lifting of programmable logic controllers using multiple processing units

Abstract

Distributed neural network boosting is performed by a neural network system through operation of at least one processor. A method includes providing a boosting algorithm that distributes a model among a plurality of processing units, each processing unit being a weak learner of a plurality of weak learners capable of performing computations while processing data independently of one another. The method further includes enabling distributed ensemble learning that enables a programmable logic controller (PLC) to scale an application using more than one of the plurality of processing units and to train the plurality of weak learners using the boosting algorithm. The plurality of weak learners are machine learning models that do not capture the entire data distribution and are purposefully designed for prediction with lower accuracy. The method further includes using the plurality of weak learners to vote for a final hypothesis based on feed-forward computations of the neural network.

Description

Technical Field

[0001] The present invention generally relates to a neural network system for distributing neural network enhancement of a programmable logic controller (PLC) using multiple processing units. Background Technology

[0002] Artificial intelligence (AI) is positioned to play a key role in reducing the programming and engineering effort required to create automated solutions. A crucial first step in integrating AI into automated environments is the Neural Processing Unit (NPU) module. A common challenge when working with edge devices such as Technology Module (TM) NPUs is their resource constraints. Processing speed, memory size, and storage can limit the applicability of the device. An NPU specializes in computations of feedforward neural networks (NNs). It takes tensor inputs and performs a series of tensor operations to compute the output. Using image detection as an example application, the input is an image, and the output is the bounding boxes and classifications of the detected objects. This large input space (number of pixels) and output space (number of possible bounding boxes and classes) often requires NN models to have high Vapnik-Chervonenkis (VC) dimensions to effectively capture the underlying distributions in the training data.

[0003] To support high VC dimensions, neural networks must have a large number of nodes and layers in their computational graph. However, there are three potential problems associated with having a large graph. First, the model can easily overfit the training data, making it less accurate and less generalizable. Second, computation speed may be insufficient for real-time applications due to processing limitations. Finally, edge devices may not even have enough computational resources to process the graph. The bottleneck is often the volatile memory capacity required to store the graph at runtime. As a result, the maximum VC dimension of deep neural networks is often limited by the available memory of the edge device.

[0004] Currently, NPUs are limited to running only a single neural network to execute a specific application. This neural network model is often intentionally designed to have just the right number of core count dimensions for the application. Therefore, scalability is not achieved due to computational resource constraints. Again using image detection as an example, this limitation effectively means that the NN model can only identify a limited number of categories with limited accuracy. MobileNet is currently the state-of-the-art pre-trained model that performs object detection in edge devices (i.e., mobile phones). It is trained on ImageNet, which has up to 1000 different categories. It presents global hyperparameters to the user to adjust the tradeoff between accuracy and latency due to resource constraints.

[0005] Therefore, a learning method is needed that enables the PLC to use more than one NPU to scale the application. Summary of the Invention

[0006] In short, aspects of this invention relate to a distributed ensemble learning method that enables a PLC to scale applications using more than one Neural Processing Unit (NPU). This is called Adaptive Boosting (AdaBoost). The advanced idea behind this algorithm is to train multiple weak learners, which are then used to "vote" for a final hypothesis. Weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with lower accuracy. In this case, a neural network is used as the weak learner. The advantage of this invention is that it leverages distributed computing methods and machine learning boosting algorithms to scale applications by using more than one NPU. Image detection, used as an example, has high-dimensional inputs and outputs. The existing solution Mobilenet provides users with the flexibility to modify the model size to fit the resource constraints of edge devices, but it trades model accuracy for a smaller model size. Its accuracy decreases as model complexity decreases. The proposed invention does not make this trade-off. AdaBoost combines multiple distributed neural network models to create more complex models without reaching resource constraints.

[0007] According to an illustrative embodiment of the present invention, a computer-implemented distributed neural network boosting method executed by a neural network system is provided. The method includes the step of providing a boosting algorithm by operating at least one processor, the boosting algorithm distributing a model across multiple processing units, each processing unit being a weak learner among multiple weak learners capable of independently performing computations and processing data simultaneously. The method further includes enabling distributed ensemble learning, which enables a programmable logic controller (PLC) to scale an application using more than one of the multiple processing units. The method further includes training multiple weak learners using the boosting algorithm. The multiple weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with lower accuracy. The method further includes using the multiple weak learners to vote on a final hypothesis using neural network-based feedforward computation.

[0008] According to another illustrative embodiment of the present invention, a neural network system for distributed neural network boosting is provided. The system includes a processor and an accessible memory storing a neural program comprising software instructions configured, when executed by the processor, to: provide a boosting algorithm that distributes a model across multiple processing units, each processing unit being a weak learner among multiple weak learners capable of independently performing computations and processing data simultaneously. The software instructions enable distributed ensemble learning, which allows a programmable logic controller (PLC) to scale an application using more than one of the multiple processing units. The software instructions use the boosting algorithm to train the multiple weak learners. The multiple weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with lower accuracy. The software instructions use the multiple weak learners to vote on a final hypothesis using neural network-based feedforward computation.

[0009] According to another illustrative embodiment of the invention, a non-transitory computer-readable storage medium encoded with executable instructions is provided. These instructions, when executed by at least one processor, operate one or more neural network systems. The instructions include providing a boosting algorithm that distributes a model across multiple processing units, each processing unit being a weak learner among multiple weak learners capable of independently performing computations and processing data simultaneously. The instructions include enabling distributed ensemble learning that allows a programmable logic controller (PLC) to scale an application using more than one of the multiple processing units. The instructions include using the boosting algorithm to train multiple weak learners. The multiple weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with lower accuracy. The instructions include using the multiple weak learners to vote on a final hypothesis using neural network-based feedforward computation. Attached Figure Description

[0010] Figure 1 A block diagram of a neural network system for distributed neural network enhancement of a programmable logic controller (PLC) using multiple processing units such as a neural processing unit (NPU) according to an exemplary embodiment of the present invention is shown.

[0011] Figure 2 A block diagram illustrating offline training of a weak neural network according to an exemplary embodiment of the present invention is shown.

[0012] Figure 3 A block diagram illustrating loading a neural network (NN) into an NPU according to an exemplary embodiment of the present invention is shown.

[0013] Figure 4 A feedforward operation for prediction according to an exemplary embodiment of the present invention is shown.

[0014] Figure 5 An overall production line according to an exemplary embodiment of the present invention is shown.

[0015] Figure 6 A schematic diagram of a computer-implemented distributed neural network enhancement method for an image detection task according to an exemplary embodiment of the present invention is shown.

[0016] Figure 7 A schematic diagram of a computer-implemented distributed neural network enhancement method executed by a neural network system according to an exemplary embodiment of the present invention is shown.

[0017] Figure 8 An example of a computing environment in which embodiments of the present disclosure can be implemented is shown. Detailed Implementation

[0018] To facilitate understanding of the embodiments, principles, and features of the present invention, they are explained below with reference to implementations in illustrative embodiments. In particular, they are described in the context of a neural network system for distributed neural network boosting of a programmable logic controller (PLC) using multiple processing units such as a neural processing unit (NPU). Distributed neural network boosting is provided within the NPU. The AdaBoost algorithm is used to boost multiple neural networks as weak learners, each of which can reside in a separate NPU, thus enabling them to utilize resources independently and perform computations simultaneously. Using AdaBoost training, each weak learner focuses on a resampled subset of the dataset. Combining all outputs allows the overall model to expand the Vapnik-Chervonenkis (VC) dimension, thereby effectively covering a larger underlying distribution of the data. Currently, the VC dimension of models implemented in NPUs is limited by their computational resources. Moreover, this limitation directly equates to a narrow applicability of the device, as the model's ability to learn complex distributions is limited. The present invention eliminates this limitation, enabling the NPU to scale to applications. However, embodiments of the present invention are not limited to use in the described devices or methods.

[0019] The components and materials described below as constituting various embodiments are intended to be illustrative and not restrictive. Many suitable components and materials that will perform the same or similar functions as the materials described herein are intended to be included within the scope of embodiments of the invention.

[0020] The following is a reference to the text. Figures 1 to 8 These and other embodiments of the neural network system according to this disclosure are described. The same reference numerals used throughout the figures identify similar or identical elements. The figures are not necessarily drawn to scale.

[0021] Consistent with one embodiment of the present invention, Figure 1 This diagram illustrates a neural network system 105 for distributed neural network enhancement according to an exemplary embodiment of the present invention. According to an exemplary embodiment of the present invention, distributed neural network enhancement is used in a programmable logic controller (PLC) system having a PLC 107 and multiple processing units such as neural processing units (NPUs) 110 (1-X). Users can connect compatible sensors, such as cameras or microphones, to the integrated interface of the NPU module. Data from the connected sensor systems, as well as information from the CPU program itself, can be processed using neural networks. The processing results are then evaluated in the CPU program. The NPU module of the controller and I / O system has a chip with artificial intelligence (AI) capabilities. For example, the NPU (neural processing unit) can be equipped with a vision processing unit, thus enabling efficient processing of the neural network. The functionality of the NPU module is derived from the supply of a trained neural network on an SD card and is equipped with a USB 3.1 interface and a Gigabit Ethernet port. Based on the neural network, data from the connected sensor systems or from the CPU program can be processed. For example, by using machine learning algorithms, visual quality inspection or image-guided robotic systems in a production line can be effectively implemented. This makes more efficient and more "human-like" behavior possible.

[0022] Distributed neural network boosting is a novel approach that distributes the computation of a machine learning model across multiple computers or hardware devices. This method utilizes boosting techniques to train more than one model (also known as a weak learner), which will then produce the final prediction. A model is a machine learning model that can be trained to map input data to output predictions. A weak learner is intentionally designed for lightweight and simple machine learning models. It is expected to learn only from a subset of the training data, so it will not become a powerful predictor on its own.

[0023] The neural network system 105 includes a processor 112 and an accessible memory 115 storing a neural program 117 including software instructions 120 configured, when executed by the processor 112, to provide a boosting algorithm 122 that distributes the model 125 across multiple processing units 110 (1-X), each of which is a weak learner among multiple weak learners (WL) 127 (1-X) capable of performing computations independently of each other while processing data.

[0024] Boosting Algorithm 122 is one of the ensemble algorithms in supervised learning used to reduce bias 162 and variance 160. It involves training weak learners (WL) 127(1-X) sequentially with different weight distributions in a sample dataset. Subsequently, all trained weak learners are able to produce a final prediction together. Variance 160 in probability theory and statistics is the expected value of the squared deviation of a random variable from its mean. Informally, it measures how far a set of numbers deviates from their mean. Bias 162 is a characteristic of statistical techniques or their results where the expected value of the result differs from the true underlying quantitative parameter being estimated.

[0025] Software instruction 120 enables distributed ensemble learning, which allows the programmable logic controller (PLC) 107 to scale the application 130 using more than one of a plurality of processing units 110 (1-X). Distributed ensemble learning is a series of algorithms in supervised learning in which multiple weak learners (WL) 127 (1-X) are trained individually, and then all of these weak learners are combined to produce predictions.

[0026] Software instruction 120 uses boosting algorithm 122 to train multiple weak learners (WL) 127 (1-X). The multiple weak learners (WL) 127 (1-X) are machine learning models (MODs) 135 (1-X) that do not capture the entire data distribution 137 and are intentionally designed to make predictions with lower accuracy 140. The entire data distribution 137 is a function or list showing all possible values ​​(or intervals) of the data. It also (and very importantly) shows the frequency of each value. Accuracy is the number of correctly predicted predictions divided by the total number of predictions. Weak learners do not need to have high accuracy. Lower accuracy 140 makes them lightweight and fast.

[0027] Software instruction 120 uses multiple weak learners (WL) 127 (1-X) to vote on a final hypothesis 142 using a feedforward computation 144 based on a neural network (NN) 150 (1-X). The final hypothesis 142 is the final prediction output by the machine learning model. In one embodiment, the neural network (NN) 150 (1-X) is used as multiple weak learners (WL) 127 (1-X). The feedforward computation 144 is a matrix computation (the opposite of backpropagation) performed by the neural network during the prediction phase.

[0028] The NPU 110 is an edge device designed for use in PLC systems. Edge devices are hardware devices capable of performing optimized matrix operations locally rather than relying on connections such as cloud computing. Figure 1The system contains the minimum components required for the operation of NPU 110. PLC 107 includes controller 146, which processes data from sensors or cameras and sends it to NPU 110, and subsequently summarizes predictions for a final hypothesis 142. Power supply 148 provides power to all components. Input / output (I / O) device interface 149 (1-X) allows NPU 110 to communicate with controller 146 via the Profinet protocol. Additional components such as I / O modules can also be added to system 105 to collect sensor data and issue control signals. Input / output (I / O) device interface 149 (1-X) is connected to PLC 107 via network switch 151. RGB camera 153 can be connected to PLC 107 for image detection.

[0029] For example, boosting algorithm 122 can be applied to image detection. A single-shot detector (SSD) can be used as a weak learner 127, while at least two hyperparameters 152 (1-2) are used to intentionally weaken model 125. The two hyperparameters 152 are adjustable to achieve different performance or results for machine learning model 125. The two hyperparameters 152 include a width multiplier 152 (1) and a resolution multiplier 152 (2), the former refining the neural network system 105 at each layer, and the latter reducing the resolution of the input image. The accuracy and speed of model 125 are tuned using the two hyperparameters 152. Accuracy is the number of correctly predicted predictions divided by the total number of predictions. In image detection, it can also be a metric measured by the overlap (IOU) of the bounding boxes of detected objects. Speed ​​is the computational speed of machine learning model 125 (i.e., the speed at which model 125 produces the final prediction).

[0030] Each model in the Model of Difference (MOD) 135 (1-X) of multiple Weak Learners (WL) 127 (1-X) returns a list of output bounding boxes 155 and their corresponding categories 157. The bounding boxes 155 capture objects of interest in the image. The boosting algorithm 122 can be used for typical regression or classification tasks. In classification tasks, the machine learning model 125 is used to identify the category 157 of the object. For example, an image detection model trained to identify dogs and cats in an image will produce bounding boxes labeled with the corresponding categories (dogs and cats). In regression tasks, the machine learning model 125 is used to predict continuous sample variables. For example, an image detection model trained to detect people will compute the center, width, and height of bounding boxes, which are continuous variables (as opposed to discrete categories).

[0031] Boosting algorithm 122 guarantees a reduction in variance 160 without increasing bias 162, thereby making model 125 more general. Boosting algorithm 122 combines multiple distributed neural network models (MODs) 135(1-X) to create more complex models without meeting resource constraints. For example, the computational resources of a computer or hardware device (such as processing power and memory) are limited. This resource constraint can impose an upper limit on speed and the accuracy of model predictions.

[0032] Boosting algorithm 122 combines all outputs to expand model 125 to a Vapnik-Chervonenkis (VC) dimension 165, thereby effectively covering a larger underlying distribution of the training data. The Vapnik-Chervonenkis (VC) dimension 165 is a measure of the capacity (complexity, expressiveness, richness, or flexibility) of the functional space that the machine learning model 125 can learn. Trained using boosting algorithm 122, each of the multiple weak learners (WL) 127 (1-X) focuses on a resampled subset of the dataset.

[0033] To apply the enhancement algorithm 122 to image detection, two main modifications are required. It enables the use of the MobileNet Single Detector (SSD) as a weak learner. Two hyperparameters 152 (1-2) can be used to intentionally "weaken" the model: a width multiplier 152 (1) and a resolution multiplier 152 (2), the former refining the neural network at each layer and the latter reducing the input image resolution. Based on Tables 1 and 2 below, the model's accuracy and speed can be easily tuned using these two hyperparameters 152 (1-2).

[0034] Table 1: MobileNet Width Multiplier

[0035] 1.0MobileNet-224 70.6％ 569 4.2 0.75MobileNet-224 68.4％ 325 2.6 0.5MobileNet-224 63.7％ 149 1.3 0.25MobileNet-224 50.6％ 41 0.5

[0036] Table 2: MobileNet Resolution

[0037] 1.0MobileNet-224 70.6％ 569 4.2 1.0MobileNet-192 69.1％ 418 4.2 1.0MobileNet-160 67.2％ 290 4.2 1.0MobileNet-128 64.4％ 186 4.2

[0038] Boosting algorithm 122 can train a weak learner (WL) 127(1-X), as described in Table 3 of the boosting neural network below. However, depending on the application, the voting on the final hypothesis (“output” segment) will require special attention.

[0039] Table 3: Boosting Algorithm (AdaBoost)

[0040]

[0041]

[0042] For typical regression or classification tasks, the boosting algorithm 122 detailed above can be used directly. For image detection tasks, the final output hypothesis cannot be a simple weighted sum, because each model 135 (1-X) (i.e., SSD) of the weak learner (WL) 127 (1-X) returns a list of bounding boxes 155 and their corresponding classes 157. To properly combine the results of the weak learner, all the output bounding boxes 155 and classes 157 should be grouped together into a set. This set can contain many low-confidence predictions and duplicates. Therefore, it is important to discard low-confidence predictions by a threshold 170 and then apply non-maximum suppression to reduce duplicates.

[0043] The boosting algorithm 122 ensures a reduction in variance 160 without increasing bias 162, making model 125 more general. This advantage addresses the first overfitting problem. Another advantage is the convenience of distributing model 125 across multiple processing units 110 (1-X). Since each weak learner 127 can perform computations independently, they can reside in separate NPUs. Utilizing multiple NPUs 110 (1-X) to process data simultaneously addresses two issues related to computational speed and memory limitations. This approach improves the scalability and flexibility of NPU 110, making it suitable for a wider range of tasks.

[0044] refer to Figure 2 , Figure 2 Offline training of a weak neural network (NN) 205 (1-X) according to an exemplary embodiment of the present invention is illustrated. The first step is to sequentially train multiple weak neural networks (NNs) 205 (1-X) using a boosting algorithm 122. These NNs 205 (1-X) are trained on different subsets of the sample data based on a sampling distribution. This distribution is updated each time an NN 205 is trained. The idea is that some data points may be difficult to learn and therefore frequently make incorrect predictions. Therefore, the boosting algorithm 122 intentionally increases the weights on these difficult data points, thereby forcing subsequent NNs to focus only on certain subsets of data.

[0045] Now transferred to Figure 3 , Figure 3 This illustrates loading a neural network (NN) 205 (1-X) into an NPU 305 (1-X) according to an exemplary embodiment of the present invention. Figure 2 After training, the graph and weights of the NN can be saved to files on the hard drive. These files can then be transferred to memory cards and loaded into the Neural Processing Unit (NPU) 305 (1-X).

[0046] Figure 4A feedforward for prediction is illustrated according to an exemplary embodiment of the present invention. The NPU is capable of independently performing feedforward calculations from input data 405 to generate predictions. When all predictions arrive at the PLC CPU memory, a voting heuristic aggregates these individual predictions to generate a final prediction 410. The voting heuristic is presented in boosting algorithm 122, but users can design their own heuristics depending on their application, performance, and prediction quality.

[0047] like Figure 5 What I saw Figure 5 An overall pipeline 505 according to an exemplary embodiment of the present invention is shown. The pipeline 505 typically comprises three stages 510 (1-3). In the first stage 510 (1), the user must train a weak neural network (NN) 515 (1-X) offline using a sample dataset according to the boosting algorithm 122. The number of iterations and the weak NN 515 (1-X) depend on the number of neural processing units (NPUs) 520 (1-X) to be distributed on it. Hyperparameters 152 are fine-tuned in this step to produce the desired prediction results. After the weak NNs 515 (1-X) are sufficiently trained, their corresponding graph structures and weights are loaded into individual NPUs in the second stage 510 (2). These NPUs 520 (1-X) and the weak NNs 515 (1-X) will be able to perform feedforward operations in the third stage 510 (3), and their outputs can be aggregated to form a final hypothesis 525.

[0048] like Figure 6 As shown, Figure 6 A schematic diagram illustrating a flowchart of a computer-implemented distributed neural network enhancement method 600 for an image detection task according to an exemplary embodiment of the present invention is shown. Reference is made to... Figures 1 to 5 The components and features described herein. It should be understood that some steps are not required to be performed in any particular order, and some steps are optional.

[0049] Method 600, performed by the neural network system, includes step 605 of applying boosting algorithm 122 to an image detection task. Each model of multiple weak learners (WL) 127 (1-X) returns a list of output bounding boxes 155 and their corresponding categories 157. Method 600 also includes step 610 of grouping all output bounding boxes 155 and all categories 157 together into a set such that the set contains many low-confidence predictions and duplicates. Method 600 further includes step 615 of discarding many low-confidence predictions by using a threshold 170. Method 600 also includes step 620 of applying non-maximum suppression to reduce duplicates.

[0050] exist Figure 7 middle, Figure 7A schematic diagram illustrating a flowchart of a computer-implemented distributed neural network enhancement method 700 executed by a neural network system according to an exemplary embodiment of the present invention is shown. Reference is made to... Figures 1 to 5 The components and features described herein. It should be understood that some steps are not required to be performed in any particular order, and some steps are optional.

[0051] Method 700 includes: step 705 of providing a boosting algorithm 122 by operating at least one processor, the boosting algorithm 122 distributing model 125 across a plurality of processing units 110 (1-X), each processing unit being a weak learner among a plurality of weak learners (WL) 127 (1-X) capable of independently performing computations and processing data simultaneously. Method 700 also includes: step 710 of enabling distributed ensemble learning, which enables a programmable logic controller (PLC) to scale the application using more than one of the plurality of processing units. Method 700 further includes: step 715 of training the plurality of weak learners using the boosting algorithm. The plurality of weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with lower accuracy. Method 700 further includes: step 720 of voting on a final hypothesis using the plurality of weak learners in a neural network-based feedforward computation.

[0052] While a programmable logic controller (PLC) has been described herein, the invention also contemplates a range of one or more other industrial controllers or other forms of industrial controllers. For example, other types of industrial controllers can be implemented based on one or more of the features presented above without departing from the subject matter of the invention.

[0053] The techniques described herein can be particularly useful for feedforward computation for prediction. While specific embodiments are described in relation to feedforward computation, the techniques described herein are not limited to feedforward computation and can also be used with other neural networks.

[0054] about Figure 8 , Figure 8 An example of a computing environment 800 in which embodiments of the present disclosure can be implemented is shown. For example, the computing environment 800 can be configured to perform the above-referenced... Figure 1 The neural network system discussed above, or the implementation of the above reference... Figure 7 The method 700 is described in part. Computers and computing environments (such as computer system 810 and computing environment 800) are known to those skilled in the art and are therefore briefly described herein.

[0055] like Figure 8As shown, the computer system 810 may include a communication mechanism such as a bus 821 or other communication mechanisms for communicating information within the computer system 810. The computer system 810 also includes one or more processors 820 coupled to the bus 821 for processing information. The processor 820 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processor known in the art.

[0056] Computer system 810 also includes system memory 830 coupled to bus 821 for storing information and instructions to be executed by processor 820. System memory 830 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 831 and / or random access memory (RAM) 832. System memory RAM 832 may include other(one or more) dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). System memory ROM 831 may include other(one or more) static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). Furthermore, system memory 830 may be used to store temporary variables or other intermediate information during instruction execution by processor 820. Basic input / output system (BIOS) 833 contains basic routines, such as those that facilitate the transfer of information between components within computer system 810 at startup, which may be stored in ROM 831. RAM 832 may contain data and / or program modules that are immediately accessible to and / or currently being operated by processor 820. System memory 830 may additionally include, for example, operating system 1034, application programs 835, other program modules 836, and program data 837.

[0057] The computer system 810 also includes a disk controller 840 coupled to a bus 821 to control one or more storage devices for storing information and instructions, such as hard disks 841 and removable media drives 842 (e.g., floppy disk drives, compact disk drives, tape drives, and / or solid-state drives). Storage devices can be added to the computer system 810 using appropriate device interfaces (e.g., Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0058] The computer system 810 may also include a display controller 865 coupled to a bus 821 to control a display 866 for displaying information to a computer user, such as a cathode ray tube (CRT) or liquid crystal display (LCD). The computer system includes an input interface 860 for interacting with the computer user and providing information to the processor 820, and one or more input devices, such as a keyboard 862 and a pointing device 861. The pointing device 861 may be, for example, a mouse, trackball, or pointing stick for communicating directional information and command selection to the processor 820 and for controlling cursor movement on the display 866. The display 866 may provide a touchscreen interface that allows input to supplement or replace the directional information and command selection communicated by the pointing device 861.

[0059] Computer system 810 is capable of performing some or all of the processing steps of embodiments of the present invention in response to processor 820 executing one or more sequences of one or more instructions contained in memory such as system memory 830. Such instructions can be read into system memory 830 from another computer-readable medium, such as hard disk 841 or removable media drive 842. Hard disk 841 can contain one or more data storage and data files used by embodiments of the present invention. Data storage contents and data files can be encrypted to improve security. Processor 820 can also be employed in a multiprocessing arrangement to execute one or more sequences of instructions contained in system memory 830. In alternative embodiments, hard-wired circuitry can be used instead of or in combination with software instructions. Therefore, embodiments are not limited to any specific combination of hardware circuitry and software.

[0060] As described above, computer system 810 can include at least one computer-readable medium or memory for storing instructions programmed according to embodiments of the present invention and for containing data structures, tables, records, or other data described herein. As used herein, the term "computer-readable medium" refers to any medium that participates in providing instructions to processor 820 for execution. Computer-readable media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical discs, solid-state drives, magnetic disks, and magneto-optical discs, such as hard disk 841 or removable media drive 842. Non-limiting examples of volatile media include dynamic memory, such as system memory 830. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including conductors constituting bus 821. Transmission media can also take the form of sound waves or light waves, such as sound waves or light waves generated during radio wave and infrared data communication.

[0061] The computing environment 800 may also include a computer system 810 that operates in a networked environment using a logical connection of one or more remote computers (such as remote computer 880). The remote computer 880 may be a personal computer (laptop or desktop), mobile device, server, router, network PC, peer-to-peer device, or other common network node, and typically includes many or all of the elements described above with respect to computer system 810. When used in a networked environment, computer system 810 may include a modem 872 for establishing communication on a network 871 (such as the Internet). Modem 872 may be connected to bus 821 via user network interface 870 or via another suitable mechanism.

[0062] Network 871 can be any network or system commonly known in the art, including the Internet, intranet, local area network (LAN), wide area network (WAN), metropolitan area network (MAN), direct connection or a series of connections, cellular telephone network, or any other network or medium that facilitates communication between computer system 810 and other computers (e.g., remote computer 880). Network 871 can be wired, wireless, or a combination thereof. Wired connections can be implemented using Ethernet, Universal Serial Bus (USB), RJ-11, or any other wired connection commonly known in the art. Wireless connections can be implemented using Wi-Fi, WiMAX and Bluetooth, infrared, cellular networks, satellite, or any other wireless connection technology commonly known in the art. Furthermore, several networks can operate independently or communicate with each other to facilitate communication within network 871.

[0063] In some embodiments, the computer system 810 can be utilized in conjunction with a parallel processing platform comprising multiple processing units. This platform allows for the parallel execution of one or more tasks associated with optimal design generation, as described above. For this example, in some embodiments, the execution of multiple product lifecycle simulations can be performed in parallel, thereby allowing for a reduction in the overall processing time for optimal design selection.

[0064] The embodiments of this disclosure can be implemented using any combination of hardware and software. Furthermore, the embodiments of this disclosure can be included in an article of manufacture (e.g., one or more computer program products) having, for example, a computer-readable, non-transitory medium. The medium already embodies computer-readable program code therein, for example, mechanisms for providing and facilitating the embodiments of this disclosure. The article of manufacture can be included as part of a computer system or can be sold separately.

[0065] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The aspects and embodiments disclosed herein are for illustrative purposes and are not intended to be limiting; the true scope and subject matter are indicated by the following claims.

[0066] As used herein, an executable application includes code or machine-readable instructions for instructing a processor (such as an operating system, a context data acquisition system, or the processor of another information processing system) to perform a predetermined function in response to user commands or input. An executable program is a piece of code or machine-readable instructions, a subroutine, or other distinct code segment or part of an executable application for performing one or more specific processes. These processes can include receiving input data and / or parameters, performing operations on the received input data and / or performing functions in response to received input parameters, and providing the obtained output data and / or parameters.

[0067] As used herein, a graphical user interface (GUI) includes one or more display images generated by a display processor and enabling user interaction with the processor or other devices, as well as associated data acquisition and processing functions. The GUI also includes an executable program or application. This executable program or application modulates the display processor to generate signals representing the GUI display images. These signals are supplied to a display device, which displays the images for the user to view. Under the control of the executable program or application, the processor manipulates the GUI display images in response to signals received from input devices. In this way, the user can interact with the display images using input devices, thereby enabling user interaction with the processor or other devices.

[0068] The functions and processing steps described herein can be executed automatically, or in whole or in part, in response to user commands. Automated activities (including steps) are performed in response to one or more executable instructions or device operations without requiring direct user initiation.

[0069] The systems and processes shown in the diagram are not exclusive. Other systems, processes, and menus can be derived from the principles of this invention to achieve the same objective. Although the invention has been described with reference to specific embodiments, it will be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the present design can be made by those skilled in the art without departing from the scope of the invention. As described herein, various systems, subsystems, agents, managers, and processes can be implemented using hardware components, software components, and / or combinations thereof.

[0070] Computer-readable medium instructions used to implement the operations of this disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions can be executed entirely on a user's computer, partially on a user's computer, partially on a user's computer as a standalone software package, and partially or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can execute the computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry and thereby perform aspects of this disclosure.

[0071] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable medium instructions.

[0072] It should be understood that Figure 8 The descriptions of program modules, applications, computer-executable instructions, code, etc., stored in system memory are merely illustrative and not exhaustive, and are described as meaning that processing supported by any particular module can alternatively be distributed across multiple modules or executed by different modules. Furthermore, various program modules, scripts, plugins, application programming interfaces (APIs), or any other suitable computer-executable code can be provided, hosted locally on computer system 810, on remote devices, and / or hosted on other computing devices(s) accessible via one or more networks(s) to support processing by… Figure 8 The functionality and / or additional or alternative functionality provided by the program modules, applications, or computer-executable code described herein. Furthermore, functionality can be modularized differently, such that it can be described as being provided by… Figure 8The processing collectively supported by the set of program modules described herein can be performed by fewer or more modules, or the functionality described as being supported by any particular module can be at least partially supported by another module. Furthermore, the program modules supporting the functionality described herein can form part of one or more applications that can be executed on any number of systems or devices according to any suitable computational model (such as, for example, client-server models, peer-to-peer models, etc.). Furthermore, the processing described as being supported by... Figure 8 Any functionality supported by any program module described herein can be implemented, at least in part, in hardware and / or firmware on any number of devices.

[0073] It should be further understood that, without departing from the scope of this disclosure, computer system 810 may include alternative and / or additional hardware, software, or firmware components beyond those described or depicted. More specifically, it should be understood that the software, firmware, or hardware components depicted as part of computer system 810 are merely illustrative, and some components may not be present, or additional components may be provided in various embodiments. While various illustrative program modules have been depicted and described as software modules stored in system memory, it should be understood that functionality described as supported by program modules can be enabled by any combination of hardware, software, and / or firmware. It should be further understood that each of the modules mentioned above can represent a logical partition of the supported functionality in various embodiments. This logical partition is depicted for ease of explanation of the functionality and does not represent the structure of the software, hardware, and / or firmware used to implement the functionality. Therefore, it should be understood that functionality described as provided by a particular module can be provided at least in part by one or more other modules in various embodiments. Furthermore, one or more depicted modules may not be present in some embodiments, while in other embodiments, additional modules not depicted may be present and capable of supporting at least a portion and / or additional functionality of the described functionality. Furthermore, while some modules can be described and depicted as submodules of another module, in some embodiments, such modules can be provided as standalone modules or as submodules of other modules.

[0074] Although specific embodiments of this disclosure have been described, those skilled in the art will recognize that numerous other modifications and alternative embodiments are within the scope of this disclosure. For example, any functionality and / or processing capabilities described with respect to a particular device or component can be performed by any other device or component. Furthermore, while various illustrative implementations and architectures have been described according to embodiments of this disclosure, those skilled in the art will understand that numerous other modifications to the implementations and architectures described herein are also within the scope of this disclosure. Moreover, it should be understood that any operation, element, component, data, etc., described herein as being based on another operation, element, component, data, etc., can be additionally based on one or more other operations, elements, components, data, etc. Therefore, the phrase "based on" or variations thereof should be interpreted as "at least partially based on".

[0075] Although embodiments have been described using language specific to structural features or methodological actions, it will be understood that this disclosure is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as illustrative forms for implementing embodiments. Conditional language, such as “can,” “may,” “may,” or “possibly,” and others, unless specifically stated otherwise or understood otherwise in the context in which they are used, is generally intended to convey that certain embodiments are capable of including certain features, elements, and / or steps that are not included in other embodiments. Therefore, this conditional language is not generally intended to imply features, elements, and / or steps that are claimed in any way by one or more embodiments, or that one or more embodiments must include logic for determining whether such features, elements, and / or steps are included in any particular embodiment or will be performed, with or without user input or prompting.

[0076] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram can represent a module, segment, or portion of instructions, including one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions indicated in the blocks may not appear in the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a system based on dedicated hardware that performs the specified functions or actions, or implements a combination of dedicated hardware and computer instructions.

[0077] While embodiments of this disclosure have been presented in an exemplary manner, it will be apparent to those skilled in the art that many modifications, additions, and deletions can be made therein without departing from the subject matter and scope of the invention and its equivalents, as set forth in the following claims.

[0078] The embodiments and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known starting materials, processing techniques, components, and apparatus have been omitted to avoid unnecessarily obscuring the embodiments in detail. However, it should be understood that the detailed descriptions and specific examples, while indicating preferred embodiments, are given by way of illustration only and not by way of limitation. Various substitutions, modifications, additions, and / or rearrangements within the subject matter and / or scope of the basic inventive concept will become apparent to those skilled in the art from this disclosure.

[0079] As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” or any other variation thereof are intended to cover non-exclusive inclusion. For example, a process, article, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent in the process, article, or apparatus.

[0080] Furthermore, none of the examples or illustrations given herein shall be construed in any way as a constraint, limitation, or explicit definition of any or more terms used with them. Rather, these examples or illustrations should be considered as descriptions relative to a particular embodiment and are for illustrative purposes only. Those skilled in the art will understand that any or more terms used with these examples or illustrations will cover other embodiments that may or may not be given in the specification or elsewhere, and all such embodiments are intended to be included within the scope of that one or more terms.

[0081] In the foregoing specification, the present invention has been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the invention. Therefore, the specification and figures are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the invention.

[0082] Although the invention has been described with respect to specific embodiments thereof, these embodiments are merely illustrative and not intended to be limiting. The description of the illustrated embodiments of the invention herein is not intended to be exhaustive, nor is it intended to limit the invention to the precise forms disclosed herein (and in particular, the inclusion of any particular embodiment, feature, or function is not intended to limit the scope of the invention to such embodiments, features, or functions). Rather, the description is intended to describe illustrative embodiments, features, and functions to provide context for those skilled in the art to understand the invention, without limiting the invention to any particular described embodiment, feature, or function. While specific embodiments and examples of the invention have been described herein for illustrative purposes only, various equivalent modifications are possible within the subject matter and scope of the invention, as will be recognized and understood by those skilled in the art. As indicated, such modifications can be made to the invention in light of the foregoing description of the illustrated embodiments, and such modifications will be included within the subject matter and scope of the invention. Therefore, although the invention has been described herein with reference to specific embodiments thereof, freedom of modification, various alterations and substitutions is contemplated in the foregoing disclosure, and it will be understood that in some cases, some features of embodiments of the invention will be employed without correspondingly using other features, without departing from the scope and subject matter of the invention as set forth. Thus, many modifications can be made to adapt particular situations or materials to the basic scope and subject matter of the invention.

[0083] The phrases “in one embodiment,” “in an embodiment,” or “in a specific embodiment,” or the corresponding appearance of similar terms throughout this specification, do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of any particular embodiment can be combined with one or more other embodiments in any suitable manner. It will be understood that, in light of the teachings herein, other variations and modifications of the embodiments described and illustrated herein are possible and should be considered part of the subject matter and scope of the invention.

[0084] Throughout this description, numerous specific details, such as examples of components and / or methods, are provided to provide a thorough understanding of embodiments of the invention. However, those skilled in the art will recognize that embodiments may be practiced without one or more specific details, or implemented with other means, systems, assemblies, methods, components, materials, parts, etc. In other instances, well-known structures, components, systems, materials, or operations are not specifically shown or described in detail to avoid obscuring aspects of embodiments of the invention. While the invention can be illustrated by using specific embodiments, this is not limiting and does not restrict the invention to any particular embodiment, and those skilled in the art will recognize that additional embodiments are readily understood and are part of the invention.

[0085] It will also be understood that one or more of the elements depicted in the figures / figures can also be implemented in a more individual or integrated manner, or even removed or rendered inoperable in some cases, which may be useful depending on the specific application.

[0086] Advantages, other benefits, and solutions to problems have been described above with respect to specific embodiments. However, the advantages, benefits, solutions to problems, and any component(s) that may cause any advantage, benefit, or solution to occur or become more apparent should not be construed as critical, required, or essential features or components.

Claims

1. A computer-implemented method for enhancing a distributed neural network, the method being executed by a neural network system comprising a programmable logic controller, at least one processor, and a plurality of processing units, and the method comprising: By operating the at least one processor: A boosting algorithm is provided, which distributes the model across the multiple processing units, each of which is a weak learner among multiple weak learners capable of performing computations and processing data independently of each other, wherein a neural network is used for the weak learners; Enable distributed ensemble learning, which enables the programmable logic controller to scale the application using one or more of the plurality of processing units; The boosting algorithm is used to train the plurality of weak learners, wherein the plurality of weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with low accuracy; and The multiple weak learners are used to vote on the final hypothesis using neural network-based feedforward computation. The enhancement algorithm is applied to image detection; and The single detector is used as a weak learner, while at least two hyperparameters are used to intentionally weaken the model. The at least two hyperparameters include a width multiplier that refines the neural network system at each layer and a resolution multiplier that reduces the resolution of the input image.

2. The method according to claim 1, wherein, Each of the multiple weak learners returns an output bounding box and a list of the model's corresponding categories.

3. The method according to claim 1, further comprising: The aforementioned boosting algorithm can be directly applied to regression or classification tasks.

4. The method according to claim 1, further comprising: The boosting algorithm is used for an image detection task, wherein each model of the plurality of weak learners returns an output bounding box and a list of the model's corresponding categories; Group all the output bounding boxes and all the categories together into a set such that the set contains many low-confidence predictions and copies; By using a threshold, many of the low-confidence predictions are discarded; and Nonmaximum suppression is applied to reduce the number of copies.

5. The method according to claim 1, wherein, The improvement algorithm ensures that variance is reduced without increasing bias, thereby making the model universal.

6. The method according to claim 1, wherein, The enhancement algorithm combines multiple distributed neural network models to create complex models.

7. The method according to claim 1, further comprising: All outputs are combined to allow the model to expand the Vapnik-Chervonenkis dimension, thereby effectively covering the large underlying distribution of the training data.

8. The method according to claim 1, further comprising: When trained using the boosting algorithm, each of the plurality of weak learners focuses on a resampled subset of the dataset.

9. A neural network system for improving distributed neural networks, the system comprising a programmable logic controller and multiple processing units. Processor; and Accessible memory storing a neural program including software instructions that, when executed by the processor, are configured to: A boosting algorithm is provided, which distributes the model across multiple processing units, each of which is a weak learner among multiple weak learners capable of independently performing computations and processing data simultaneously. The neural network is used for the weak learner; Enable distributed ensemble learning, which allows the programmable logic controller to scale the application using one or more of the plurality of processing units; The boosting algorithm is used to train the plurality of weak learners, wherein the plurality of weak learners are machine learning models that do not capture the entire data distribution and are intentionally designed to make predictions with low accuracy; and The multiple weak learners are used to vote on the final hypothesis using neural network-based feedforward computation. The enhancement algorithm is applied to image detection; and The single detector is used as a weak learner, while at least two hyperparameters are used to intentionally weaken the model. The at least two hyperparameters include a width multiplier that refines the neural network system at each layer and a resolution multiplier that reduces the resolution of the input image.

10. The neural network system according to claim 9, wherein, The accuracy and speed of the model are adjusted using the at least two hyperparameters.

11. The neural network system according to claim 9, wherein, Each of the multiple weak learners returns an output bounding box and a list of the model's corresponding categories.

12. A non-transitory computer-readable storage medium encoded with instructions that, when executed by at least one processor, a programmable logic controller, and a plurality of processing units of a neural network system according to claim 10, implement the method according to any one of claims 1 to 8.

Images