Systems and methods for virtual environments for reinforcement learning in manufacturing

Abstract

Systems, devices, and methods include receiving, by an interpreter component (104) having a processor with addressable memory, a first state (130) of a tool of a computer numerical control (CNC) machine; determining, by the interpreter component (104), a reward and a reward value based on the received first state (130), wherein the reward is at least one of a positive reward and a negative reward; transmitting, by the interpreter component, a set of information including the determined reward and reward value to an agent component (110); performing, by the agent component (110), at least one action (112) to generate a tool path (180, 182, 184) and enter a second state (132), wherein the second state (132) is in combination with the first state (130); and determining, by the agent component (110), the generated tool path based on the determined reward and value associated with the at least one action.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit from U.S. Provisional Patent Application No. 62 / 931,709, filed November 6, 2019, the contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The embodiments generally relate to the processing of materials in manufacturing, and more specifically to virtual environments for reinforcement learning during the processing.

[0004] background

[0005] Computer-aided manufacturing (CAM) software systems are used to program computer numerical control (CNC) machine tools, which are used in machine shops to produce discrete parts such as molds, dies, tools, prototypes, aerospace components, etc.

[0006] CNC machine tools run a machining program that executes a series of instructions that are the steps to manufacture a part. The CNC machine tool executes the machining program without knowing the context. The program execution is continuous, with limited foresight, and lacks a global overview of the process.

[0007] Overview

[0008] One method embodiment may include: receiving a first state of a tool of a computer numerical control (CNC) machine tool by an interpreter component having a processor with addressable memory; determining a reward and a reward value by the interpreter component based on the received first state, wherein the reward may be at least one of a positive reward and a negative reward; transmitting a set of information including the determined reward and reward value to an agent component by the interpreter component; performing at least one action by the agent component to generate a toolpath and enter a second state, wherein the second state may be combined with the first state; determining the generated toolpath by the agent component based on the determined reward and value associated with at least one action; and wherein at least one action may be determined based on at least one of: changing a tool, moving a tool, opening a spindle, and closing a spindle.

[0009] In another method embodiment, the first state includes information about at least one of the following: the state of the cutting tool, the state of the machine tool, and both the state of the cutting tool and the state of the machine tool constituting the entire system. In another method embodiment, the entire system can be represented by at least one of the following: cutting insert, discrete parts, cutting tool, and stock material. In another method embodiment, at least one action includes at least one of the following: turning operation, milling operation, additive directed energy deposition (DED), wire electrical discharge machining (EDM), and additive powder bed machining. In another method embodiment, at least one action can be part of a series of actions for generating a toolpath.

[0010] The computing device embodiment may include a processor and memory, and the computing device is configured to: receive a first state of a tool of a computer numerical control (CNC) machine tool via an interpreter component of the computing device; determine a reward and a reward value by the interpreter component based on the received first state, wherein the reward may be at least one of a positive reward and a negative reward; transmit a set of information including the determined reward and reward value to an agent component of the computing device by the interpreter component; perform at least one action by the agent component to generate a toolpath and enter a second state, wherein the second state may be combined with the first state; determine the generated toolpath by the agent component based on the determined reward and value associated with at least one action; and wherein at least one action may be determined based on at least one of the following: changing the tool, moving the tool, opening the spindle, and closing the spindle.

[0011] In another embodiment of the computing device, the first state includes information about at least one of the following: the state of the cutting tool, the state of the machine tool, and both the state of the cutting tool and the state of the machine tool constituting the entire system. In another embodiment of the computing device, the entire system may be represented by at least one of the following: cutting inserts, discrete parts, cutting tools, and blank material. In another embodiment of the computing device, at least one action includes at least one of the following: turning operation, milling operation, additive directed energy deposition (DED), wire electrical discharge machining (EDM), and additive powder bed machining. In another embodiment of the computing device, at least one action may be part of a series of actions for generating a toolpath.

[0012] An embodiment of a computer-aided manufacturing (CAM) system for a computer numerical control (CNC) machine tool may include an interpreter component and an agent component; wherein the interpreter component may be configured to: receive a first state of a tool of the CNC machine tool; determine a reward and a reward value based on the received first state, wherein the reward may be at least one of the following: positive and negative; transmit a set of information including the determined reward and reward value to the agent component; and wherein the agent component may be configured to: perform at least one action to generate a toolpath and enter a second state, wherein the second state may be combined with the first state; determine a generated toolpath based on the determined reward and value associated with at least one action; and wherein at least one action may be determined based on at least one of the following: changing a tool, moving a tool, opening a spindle, and closing a spindle. Brief description of the attached diagram

[0014] The components in the figures are not necessarily drawn to scale; rather, the emphasis is on illustrating the principles of the invention. In all the different views, similar reference numerals indicate corresponding parts. Embodiments are shown in the accompanying drawings by way of example, not limitation, wherein:

[0015] Figure 1 A system for using a virtual environment to perform reinforcement learning in manufacturing is described;

[0016] Figure 2 Depicting based on basic manufacturing operations Figure 1 The system;

[0017] Figure 3 Describing the use of in Figure 1 The process of reinforcement learning in manufacturing within a virtual environment;

[0018] Figure 4 Depicting the representation Figure 3 A pixelated vector representing a specific state during the process;

[0019] Figure 5 The pixelated vectors of the virtual cutting tool are depicted;

[0020] Figure 6 It describes the movement to Figure 4 In the state vector Figure 5 The pixelated vector of the virtual cutting tool;

[0021] Figure 7 Depicting Figure 5 Possible movement motions of the cutting tool;

[0022] Figure 8A The virtual state vector space in the first state is depicted;

[0023] Figure 8B Describes insertion into Figure 8A The virtual cutting tool in the virtual state vector space;

[0024] Figure 8C Depicting in Figure 8A In the virtual state vector space Figure 8B An example of the action of a virtual cutting tool moving from one state to another;

[0025] Figure 8D Depicting the second state after the action. Figure 8A The virtual state vector space;

[0026] Figure 9A Examples of different toolpaths for cutting virtual blanks using virtual cutting tools are depicted;

[0027] Figure 9B An example of an alternative toolpath for cutting a virtual blank using a virtual cutting tool is depicted;

[0028] Figure 9C An example of an alternative toolpath for cutting a virtual blank using a virtual cutting tool is depicted;

[0029] Figure 10 The optimal toolpath that will be executed by the agent is depicted;

[0030] Figure 11A-11B Another embodiment of a system for performing reinforcement learning in manufacturing using a vector environment is described;

[0031] Figure 12 A system for reinforcement learning in manufacturing using a virtual environment with multiple cutting tools is described.

[0032] Figures 13A-13D The protocol of a system for using a virtual environment in manufacturing to perform reinforcement learning is described;

[0033] Figure 14A The abstract actions of the system used in the virtual environment for reinforcement learning in manufacturing are described;

[0034] Figure 14B The magnet action of a system used in manufacturing a virtual environment for reinforcement learning is described.

[0035] Figure 14C The system for tracking actions in a virtual environment used for reinforcement learning in manufacturing is described;

[0036] Figure 15 An example top-level functional block diagram of a computing device embodiment is shown;

[0037] Figure 16 A high-level block diagram and process of a computing system for implementing embodiments of the system and process are shown;

[0038] Figure 17 A block diagram and process are shown in which an exemplary system in which embodiments can be implemented;

[0039] Figure 18 A cloud computing environment is described for implementing embodiments of the systems and processes disclosed herein; and

[0040] Figure 19 An exemplary method of the disclosed embodiments is depicted in the top-level flowchart.

[0041] Detailed description

[0042] The described techniques relate to one or more methods, systems, apparatuses, and media for training manufacturing machinery within a virtual environment for reinforcement learning in manufacturing, the media storing processor-executable process steps for training the manufacturing machinery. Training the manufacturing machinery may include computer-aided manufacturing (CAM) software systems for programming computer numerical control (CNC) machine tools. CNC machine tools can be used in a machine shop to produce discrete parts such as molds, dies, cutting tools, prototypes, aerospace components, etc. The techniques described below can be implemented through programmable circuits programmed or configured by software and / or firmware, or entirely through dedicated circuits, or a combination of these forms. Such dedicated circuits (if any) may take the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.

[0043] Figures 1-14C The following discussion provides a brief, general description of suitable computing environments in which the aspects of the described techniques can be implemented. While not strictly necessary, the aspects of this technique can be described in the general context of computer-executable instructions, such as routines executed by general-purpose or special-purpose data processing devices (e.g., server or client computers). The aspects of the techniques described herein can be stored or distributed on tangible computer-readable media, including magnetically or optically readable computer disks, hardwired or pre-programmed chips (e.g., EEPROM semiconductor chips), nanotechnology memories, biological memories, or other data storage media. Optionally, computer-implemented instructions, data structures, screen displays, and other data related to the technology can be distributed over a period of time via the Internet or other networks (including wireless networks) on a propagation medium (e.g., electromagnetic waves, sound waves, etc.). In some implementations, data can be provided on any analog or digital network (e.g., packet switching, circuit switching, or other schemes).

[0044] The described techniques can also be practiced in distributed computing environments, where tasks or modules are executed by remote processing devices linked via communication networks such as local area networks (“LANs”), wide area networks (“WANs”), or the Internet. In a distributed computing environment, program modules or subroutines can reside in both local and remote memory storage devices. Those skilled in the art will recognize that portions of the described techniques can reside on server computers, while corresponding portions can reside on client computers (e.g., PCs, mobile computers, tablets, or smartphones). Data structures and data transmissions specific to various aspects of this technology are also included within the scope of the described techniques.

[0045] refer to Figure 1 A system 100 for a virtual environment for reinforcement learning in manufacturing (VOM) is illustrated. The virtual environment for reinforcement learning in manufacturing system 100 provides intelligent automated machine operation and the production of discrete parts (e.g., molds, dies, cutting tools, prototypes, aerospace components, etc.). Generally, reinforcement learning is a field of machine learning used to take appropriate actions in a given situation to maximize rewards. In one embodiment, system 100 may execute a series of instructions that are steps for finding the best possible behavior or “path” that the machine should take in a given situation (e.g., the production of a machine tool). In one embodiment, the best possible behavior may include the shortest possible toolpath. In another embodiment, the best possible behavior may include or be based on the shortest possible toolpath without exceeding a certain threshold force limit, while also minimizing machine acceleration (or maintaining a certain temperature range in the case of directional energy deposition or DED additive manufacturing). Using the disclosed system 100, reinforcement agent components determine which steps to perform for a given task. In one embodiment, a training dataset may be lacking, so system 100 may have to learn from its own experience and history, for example, hysteresis defined by the dependence of the system state on its history.

[0046] In one embodiment, system 100 may include a virtual environment 102, a vector-type representation of environment state 106, a (finite) number of actions to be performed by action component 112, a vector-type representation of actions, an interpreter component 104, a policy component 114, an agent component 110, and a reward 108 of a reward system. In one embodiment, state 106 may be a representation of virtual environment 102. In one embodiment, state 106 may be a pixel matrix representing virtual environment 102. In one embodiment, the matrix may have the same number of rows and columns. In another embodiment, the matrix may have a different number of rows and columns. The pixels of state 106 may represent a portion of virtual environment 102, such as a cutting blade, discrete part, cutting tool, blank material, etc. In another embodiment, the pixels of state 106 may represent a combination of portions of virtual environment 102 (e.g., cutting tool and blank material). In yet another embodiment, the pixels of state 106 may represent all portions of virtual environment 102. In one embodiment, policy component 114 may pass a policy to agent component 110, which may include data about which actions 112 are possible. Examples of actions 112 that can be determined by agent component 110 based at least on policy component 114 include, for example, avoiding collisions and / or avoiding part damage. Agent component 110 may take actions in virtual environment 102, which can be interpreted by interpreter component 104 as representations of rewards 108 and states 106, which can be fed back to agent component 110. In one embodiment, interpreter component 104 may evaluate and determine whether reward 108 is negative or positive (and the value of each reward 108) based on what has been learned and an appropriate policy. Interpreter component 104 then forwards reward 108 and state 106 to agent component 110. More specifically, reward 108 of system 100 may include two types of reinforcement—positive and negative. System 100 learns through positive reinforcement such that when an event occurs due to a particular behavior, system 100 seeks to increase the strength and frequency of the behavior. This positive influence on the behavior of system 100 can, for example, lead to the repetition of positive steps, which helps to maximize machine performance. Conversely, negative reinforcement can provide reinforcing behavior by stopping or avoiding negative conditions.

[0047] Agent component 110 can interact with virtual environment 102 at discrete time steps. For example, at each time t, agent component 110 can receive observations that typically include a reward 108. Agent component 110 can then select an action 112 from a finite set of actions. In one embodiment, action 112 can be based on what agent component 110 has learned and a policy. Examples of actions that can be determined based on a policy include, for example, avoiding collisions and / or avoiding part damage. Action 112 can then be sent by agent component 110 to virtual environment 102. Virtual environment 102 can move to a new state and can determine a reward 108. In one embodiment, the goal of agent component 110 is to collect as many rewards 108 as possible. In one embodiment, agent component 110 can select any action based on the history of the process. In another embodiment, actions can be randomly selected by agent component 110. Thus, system 100 can operate on a trial-and-error basis, where system 100 continuously improves efficiency due to rewards obtained based on positive actions. Examples of negative rewards when machining via a tool in virtual space can include operating at temperatures above the desired temperature and / or under forces determined to be excessive, such as causing damage. In both cases, due to the force and temperature, tool cutting may not occur or inappropriate tool cutting may occur, thus a negative reward is given for the absence of tool cutting or the occurrence of inappropriate tool cutting at any given step. Conversely, each action can generate a positive reward when appropriate cutting occurs. That is, system 100 can learn an appropriate reward system based on representing the company's proprietary technology to cut the blank (or add material) more efficiently.

[0048] refer to Figure 2 This illustrates how it can be applied to basic manufacturing operations. Figure 1 The system can be based on basic manufacturing operations such as "tool movement," "tool change," "spindle opening," and "arc opening." The system is applicable to manufacturing operations, including but not limited to turning operation 120, milling operation 122, additive DED 124, wire electrical discharge machining (EDM), and additive powder bed machining 126. The system's agent component 110 can be based on machine learning algorithms (including artificial neural networks (ANN), X-GBoost, decision trees, genetic algorithms, etc.) to learn how to select the optimal set of actions to proceed from the initial state to the final state with optimal performance. In one embodiment, the optimal set of actions is the action that maximizes the total reward for a given portion. This is a set of actions that maximizes the performance of the machining operation. Actions that maximize performance can include, but are not limited to, the shortest cutting time, the highest quality cut, vibration-free operation, minimal machine acceleration, deposition or removal within the correct temperature range, etc.

[0049] refer to Figure 3 This illustrates a VOM system for reinforcement learning (e.g., Figure 2The iterative process of the exemplary VOM (Virtual Object Model) is described. The VOM process can begin in an initial state "0" 129, which is based on a virtual blank tool and a target tool representing the desired final manufactured tool (e.g., a final virtual rendering of the blank tool). In one embodiment, the VOM enters a state "s" 130, where s = 0 in the first step of the iterative process. In state "s" 130, information about the tool state is sent to the interpreter component 104 for evaluation. In the initial state "0", the agent component 104 has not yet performed any action, so the information about the tool state sent to the interpreter component 104 for evaluation indicates that no change has occurred in the virtual blank tool. In some embodiments, the interpreter component 104 evaluates and determines whether the reward 108 is negative or positive (and the value associated with the reward) based on what has been learned and an appropriate strategy, which can determine what action 112 is possible. The interpreter component 104 can then forward the reward 108 (positive or negative along with the value, if any) and state 130 back to the agent component 110. Agent component 110 can execute action 112 to enter a new state "s+1" 132. The VOM can have multiple possible actions 112 to be performed from state s 130 to state s+1 132, such as changing the tool, moving the tool, opening the spindle, and closing the spindle. In one embodiment, state s+1 can be a virtual representation of the stock tool after action 112 is performed on it. For example, after performing an action between states s and s+1, the virtual stock tool in state s+1 might look more like the target tool, thus generating a positive reward. State s+1 132 (combined with the original state s 130) is then sent to interpreter component 104 for evaluation. Again, interpreter component 104 forwards the reward 108 (positive or negative along with a value, if any) and the new state back to agent component 110. The process iterates (e.g., s+n times) until a final state 131 is reached, where there are no remaining blanks to be processed, no targets are empty and therefore to be added, and / or the number of trials / iterations performed is greater than a previously determined threshold. In one example embodiment, the number of trials performed is given as ten times the number of pixels, which represents the number of possible cuts as described in detail below. Other possible numbers of trials to be performed are possible and anticipated. After the maximum number of trials has been performed or there are no remaining blanks, the process may restart in initial state 129 for learning purposes until it is determined that learning is complete. In some embodiments, agent component 110 may learn by producing a series of virtual machine parts until agent component 110 successfully performs a performance evaluation that has a higher number of steps than any previous iteration (e.g., determined based on the highest total reward received for a given part). Optimal performance can be defined as the performance that will have the highest global reward (sum) for the machine part.In one embodiment, the reward takes into account the performance of each step, and a special reward can be given when the machine part is completed. For example, an additional reward can be given for the final step of the operation.

[0050] refer to Figure 4 The diagram illustrates a state vector 150 representing a specific state during the iterative process of a VOM system used for reinforcement learning. In this example, a turning operation of a virtual part is shown. Other possible state vector representations are also possible, such as for milling operations, additive DED operations, and additive powder bed operations. The state vector representation 150 of the virtual tool position in the VOM is shown in pixels 152 in two dimensions (e.g., x-axis 154 and z-axis 156). In another embodiment, the tool position can be shown in pixels of three dimensions: x, y, and z. The z-axis 156 represents the rotation that causes the part to turn, thus providing a rotational axis for cutting the virtual part.

[0051] In the virtual state vector space 150, at least five state vectors may exist. First, the blank virtual vector 160 occupies a pixel in the state vector space 150. Pixel 166 shows the blank space around the blank pixel 160. The target part state vector 158 is shown, which represents the desired position of the final part after the machining process. Pixel 162 represents tool damage (e.g., if the target part has appeared). Pixel 164 represents a collision between the blank and the shank of the virtual tool (e.g., when the shank overlaps with the blank of the target part). In one embodiment, each state vector 158, 160, 162, 164, 166 may be represented by an integer (e.g., 0, 1, 2, 3, or 4). In another embodiment, each state vector 158, 160, 162, 164, 166 may be represented as a table, with each table overlapping each other as a layer, where each layer represents one of the five possible state vectors.

[0052] refer to Figure 5 It shows the direction to move to. Figure 4 The virtual cutting tool in the virtual vector space 150 is pixelated, or more specifically, pixelated vector 170. Pixel 172 represents the cutting insert 172, which is the cutting part of the tool. Pixel 166 represents the blank space. Pixel 174 represents the shank that holds the cutting insert.

[0053] refer to Figure 6 , Figure 5 The pixelated vector of the cutting blade 172 can be moved to Figure 4 In the virtual state vector space 150, the blank 160 is cut to realize the target part state vector in the most efficient way. Figure 4 (158). The cutting insert 172 has a 2D x, z position 175 at a given time.

[0054] refer to Figure 7 The diagram illustrates the movement 176 of the cutting blade 172 and the shank 174. More specifically, starting from position 175, the cutting blade 172 can move in eight possible directions corresponding to eight consecutive units 178 surrounding the cutting blade position 175. In one embodiment, if the cutting blade 172 operates in 3-dimensional space (e.g., x-coordinate, y-coordinate, z-coordinate), 26 movements of the cutting blade 172 are possible. Therefore, to move from state s ( Figure 3 (130) moves to state s+1 ( Figure 3 132), such as Figure 3 The possible movement of the cutting insert 172, as described in 176, can be a movement in the direction of upward to the left, directly upward, upward to the right, right, downward to the right, downward, downward to the left, or left. Additional movements can also change the speed of movement of the cutting insert 172 and the feed rate. In one embodiment, the feed rate and speed are not continuous; instead, they are derived from a finite number of movements. Other possible movements may include changing the tool, moving the tool, opening the spindle, and closing the spindle.

[0055] Figures 8A-8D An example of the action of moving the cutting insert from state s 130 to state s+1 132 within a VOM system is shown. (Reference) Figure 8A This illustrates a virtual state vector space (e.g., virtual state vector space 150). Reference Figure 8A A virtual cutting tool 170, having a shank 174 and a cutting insert 172, is moved into the state vector space 150. More specifically, in state s 130, the cutting insert 172 occupies... Figure 8A The blank space 166. In one embodiment, in state s 130, information about the tool state is sent to the interpreter component ( Figure 3 104) for evaluation. Interpreter component ( Figure 3 104) Based on state changes and the reward system, evaluate and determine the reward ( Figure 3 (108). In some embodiments, the reward system reflects best practices derived from the practitioner's knowledge, while the policy, in turn, restricts possible actions. Rewards may consider cutting rate, force, machine dynamics, number of collisions, number of part failures, etc. The VOM system 100 can learn from the rewards and is restricted by the policy in terms of possible actions that can be taken. More specifically, the policy is a fixed set of rules, and the VOM system may not learn from the policy itself. Interpreter component ( Figure 3 104) and then the reward ( Figure 3 108 (positive or negative) and status s 130 forward back to the proxy component ( Figure 3 110). Proxy component ( Figure 3 (110) Then the action can be performed ( Figure 3 (112). In one embodiment, the action can be performed by a proxy component ( Figure 3 (110) Random selection. More specifically, during the learning phase, action selection can be based on the exploration rate, such that a higher exploration rate in the parameter space results in a higher proportion of random actions. When the learning phase ends and the operation moves to the execution phase, random actions are no longer executed. In another embodiment, the action can be based on the best estimate of the possible reward for possible actions regarding the current state.

[0056] In another embodiment, the proxy component of the VOM system ( Figure 3 (110) can be based on machine learning algorithms (such as ANN). As described in further detail below, learning can be at least partially based on so-called "Q-learning," where actions are performed by an agent component ( Figure 3 The 110) is randomly selected or chosen by the optimal "Q value" based on the toolpath exploration rate. In one embodiment, the ANN can determine the optimal Q value for the current state. The optimal Q value should represent the proxy component ( Figure 3 The optimal action (110) is chosen to obtain the highest global reward for a given tool part. The ANN can correctly "learn" this function (e.g., the Q-value function of the state).

[0057] refer to Figure 8C Action 112 could be moving the cutting tool 170 to the pixel at position 177, which is directly to the left of the initial position 175. Therefore, the cutting blade 172 will cut into the blank 160 at adjacent pixels marked "1" and "8", where pixel "8" is located at position 177. In this embodiment, the interpreter component ( Figure 3 104) Evaluate and determine the reward ( Figure 3 108). Interpreter component ( Figure 3 104) and then the reward ( Figure 3 108 (positive or negative, and reward value) and new state ( Figure 3 106) forwarded back to the proxy ( Figure 3 110). Proxy component ( Figure 3 (110) Then enter as Figure 8D The new state “s+1”132 is shown. The process can be iterated (e.g., s+n times) until a final state is reached where there are no remaining blanks to be processed or too many trials / iterations have been performed, where too many trials are determined based on a predetermined threshold.

[0058] refer to Figure 9A-C shows examples of different paths 180, 182, and 184 for cutting a blank using a virtual cutting tool. Figure 9A It has a large number of possible actions to create paths 180 using a virtual cutting tool 170, where the large number is based on and Figure 9B A comparison of path 182. The virtual environment used for reinforcement learning 100 provides a reward-based reduction in the number of required actions to reduce the overall toolpath 180, thereby making the process more efficient, for example, in terms of the distance the tool must travel. For example, with Figure 9A compared to, Figure 9B The number of actions targeting path 182 has decreased significantly because the VOM has received a positive reward. For example... Figure 9C As shown, with Figure 9A Path 180 and Figure 9B Compared to path 182, toolpath 184 has become even more efficient.

[0059] refer to Figure 10 When the reinforcement learning process is complete, the agent ( Figure 3 110) can execute the toolpath 186 that gives optimal performance. Learning can be at least partially based on so-called "Q-learning," where actions are performed by an agent ( Figure 3 The 110) is randomly selected or chosen by the optimal "Q value" based on the toolpath exploration rate. More specifically, Q-learning is a model-free reinforcement learning algorithm. The goal of Q-learning is to learn a policy that tells the agent ( Figure 3 Q-learning (110) determines what action to take under what circumstances. It does not require a model of the virtual environment and can handle stochastic transitions and rewards without adaptation. For any finite Markov decision process (FMDP), Q-learning finds a policy that is optimal in the sense that, starting from the current state, maximizes the expected value of the total reward in any and all successive steps. In general, Q-learning can identify the optimal action selection policy for any given FMDP, given infinite exploration time, and a partially stochastic policy. The term "Q" is named after the function that returns the reward used to provide reinforcement and can be said to represent the "quality" of the action taken in a given state. In one embodiment, the reward system needs to ensure convergence and provide results consistent with the knowledge provided to the VOM system.

[0060] Figure 11A and Figure 11B Another embodiment of a system 1100 for a virtual environment for reinforcement learning in manufacturing, utilizing a continuous, non-discrete, "contour-based" environment, is described. In one embodiment, the toolpath may follow a smooth trajectory rather than a route, a pixelated trajectory. For example, the environment can provide navigation in any vector direction, allowing the cutting tool to follow the real part surface. Figure 11AThis indicates the state of the machine and / or system before executing the cutting motion along the toolpath trajectory, and Figure 11B This indicates the state of the machine and / or system after the cutting motion. In one embodiment, in each iteration, the cutting tool may take an action and subsequently travel along the toolpath trajectory to remove blank material. Then, in the next iteration, the cutting tool may take a new action (based on the previous iteration) to continue removing blank material. Subsequent iterations continue to remove blank material until the remaining material follows the closed contour of the target material. Iterative or step actions can be performed in floating-point sizes and are not required to be limited to integer steps.

[0061] In one embodiment, the encoding can provide a non-discrete, smooth contour representation of the virtual environment state. In another embodiment, the encoding can provide a pixel matrix representation of the virtual environment state. In yet another embodiment, the system can provide a combination of smooth and pixelated representations of the virtual environment state. In one embodiment, the system can provide multiple options for viewing the system state. For example, the agent can use pixelated views at various resolutions to obtain an approximate view of the actual environment state. In another example, the agent can use a "zoom" view to enhance the view of the environment state. Other viewing options may also be used and are described below.

[0062] The system may include a cutting tool 1105 having a cutting tool holder 1104 and a cutting portion 1106 traveling along a tool path 1108 (e.g., a trajectory of the cutting tool 1105 at a specific angle relative to the target material 1112). The cutting portion 1106 can be used to remove material from the blank material 1110 to ultimately produce a blank material form 1110 identical to the target material 1112. In some embodiments, the system 1100 may use a continuous, non-discrete environment, abstract movement, and "diagonal portions." Abstract movement, or "abstraction," can remove physical, spatial, or temporal details or attributes in the study of an object or system to focus attention on more important details. In one embodiment, abstract movement may relate to the programmability of a virtual environment to provide an agent with action options to bring the cutting tool to a position in the virtual environment without precisely determining "partial step values ​​and angles."

[0063] In one embodiment, a diagonal portion refers to a portion of an angular surface that is not 90 degrees or 180 degrees. The non-discrete, contour-based environment allows the cutting tool to travel along any spatial vector.

[0064] More specifically, system 1100 can hide all data (except relevant data) about objects (e.g., blank material 1110 and target material 1112) to reduce complexity while improving efficiency. Therefore, abstract movement can remove data irrelevant to achieving the desired result (e.g., target material 1112), thereby reducing the complexity of coding within the virtual environment. The environment of system 1100 can be based on the closed profile of the part, blank, tool shank, and tool insert. In one embodiment, in each iteration, the cutting tool 1105 travels along a toolpath trajectory (e.g., toolpath trajectory 1108 in the first iteration) to remove blank material. The use of a vectorless virtual environment, abstract movement, and diagonal portions in system 1100 allows the cutting tool to remove blank material to ultimately conform to the closed profile of target material 1112. The virtual environment can learn to take the fewest steps (i.e., the fewest iterations) to achieve the final target material 1112.

[0065] Figure 12 A system 1200 is described, which utilizes a continuous, non-discrete, "contour-based" environment for reinforcement learning in manufacturing. System 1200 may include a blank material 1202 to be removed (leaving target material 1204). System 1200 may include multiple cutting tools 1206, 1208, 1210, and 1212. Cutting tools 1206, 1208, 1210, and 1212 may include a conventional vertical cutting tool 1206, a cutting / grooving tool 1208, a conventional horizontal tool 1210, and a boring tool 1212. System 1200 may include environment recoding to represent the part, workpiece, and cutting tools in vector / polygonal mathematics. In one embodiment, the calculation can be precisely defined using three tool movement possibilities: step move, angular move, and free move. Step move is based on a dictionary of up, down, left, and right. Angular move is assigned an angle and length of a certain degree. Free move is assigned arbitrary x and y lengths.

[0066] Figure 13A A system for performing reinforcement learning in manufacturing is described, which represents the target material, the blank material, and the cutting tool (e.g., ...). Figure 11A -Those shown in B). Figure 13B-13D "Protocol views" 1302, 1304, and 1306 are depicted for constructing target materials (e.g., target parts) using a system for reinforcement learning in manufacturing. The first protocol 1302 generates an image with a user-defined resolution that combines two main aspects of the overall view, such as an environmental screenshot, and a view centered on the cutting tool (e.g., a zoomed-in or enhanced view of the cutting insert). More specifically, the overall view ( Figure 13AThe image shown is displayed Figure 13B On the left side. Figure 13B The dashed box on the left shows the location where the cutting tool is inserted into the blank material, and this area is enhanced in the magnified view shown on the right. In one embodiment, the agent can use both the magnified and non-magnified views to make decisions. In one embodiment, the output of the agent is fed as an image with variable resolution. The enhanced view provides enhanced resolution, allowing the agent to make more accurate cutting decisions than a zoomed-out view.

[0067] The protocol views described above (the agent can use one or more of them) represent the state of the virtual environment at a given iteration. The agent can use these protocol views to "reason" about what actions to take next. Therefore, the agent can use one or more protocols to examine the environment state and determine what actions or actions need to be taken to ultimately bring the blank part to the final shape of the target part.

[0068] Figure 13C The second protocol 1304 shown generates an image with a user-defined resolution, which combines two different aspects: a view centered on the cutting tool (e.g., zoom or enhancement in the cutting insert), and a re-encoding based on the quantity and orientation of part and workpiece objects. The output of the feed agent is an image with variable resolution. In one embodiment, the generated image consists of a magnified view (e.g., an exaggerated view embedded in the blank material). Figure 13B The view consists of an enlarged view of the blank material, with the blank material surrounding the enlarged portion. This enlarged view of the blank material allows the agent to roughly understand the total amount of blank material surrounding the enlarged view portion.

[0069] Figure 13D The third protocol 1306 shown generates a table list of content: a two-column table representing the distance to be reached (or not reached) from a part object to a workpiece object. In one embodiment, the table list of content can be defined to generate small codes while significantly increasing the learning process of the system for reinforcement learning in a virtual environment during manufacturing. In one embodiment, the output of the feed agent is a table list with user-defined content. In one embodiment, the agent is a virtual operator.

[0070] In one embodiment, each row of the table list of contents may represent a single sector of the "radar" image shown on the left side of the table list. For example, the row for sector 0 is a table representation of sector 0 of the radar image. Each column represents the shortest distance from the center of the radar image to the part (first column) and the shortest distance from the center of the radar image to the workpiece material (second column). More specifically, each column represents the distance from a reference point in the cutting tool to the first and nearest segments of the part polygon (first column) and the workpiece material polygon (second column). If a part or workpiece material polygon segment is not present in the sector, no value is entered into the table list (e.g., for sectors 2, 3, 4, and 5). In one embodiment, sectors allow the agent to "see" the environment more precisely in order to take action. In one embodiment, more than six sectors are possible. In another embodiment, fewer than six sectors are possible.

[0071] Figure 14A Abstract actions for a system 1400, a virtual environment used for reinforcement learning in manufacturing, are described. The abstractions may include up (0,1), down (0,-1), left (-1,0), and right (1,0). Diagonals may include (-1,-1), (-1,1), (1,1), and (1,-1). In one embodiment, the virtual environment may have the ability to navigate in any vector direction, allowing for realistic diagonal movement; thus, the cutting tool may follow a smooth, non-discrete part surface. For example, the region 1401 to be cut by the cutting tool may require less movement than in a discretized, pixelated environment. For system 1400, diagonal movement may require only one action, while a pixelated environment may require two movements, such as moving up and then left.

[0072] Figure 14B The system 1402, used in a virtual environment for reinforcement learning in manufacturing, is described as having "magnetic movements." Magnetic movements can include up, down, left, and right. In one embodiment, the step size of the movement does not need to be predefined as a value. In one embodiment, the abstract movement of system 1402 can include "magnetic movement," where an agent can move a cutting tool as close as possible to the part without contacting it. In one embodiment, a magnet can attract the cutting tool to the target material when it approaches the target material.

[0073] In one embodiment, the agent can learn increasingly complex actions using a finite number of action possibilities. In this way, abstract movement can include moving the cutting tool as it "tracks" or "follows" the surface of the workpiece to be cut. Therefore, abstract movement can simplify the cutting process by reducing the number of available actions while still allowing for increasingly complex actions. More specifically, refer to... Figure 14CThe system 1404 for a virtual environment for reinforcement learning in manufacturing may have a set of tracking actions. The tracking actions may include clockwise or counter-clockwise directions for tracking a cutting tool along the contour of a target material, where there are intersections between the cutting tool and the target material.

[0074] Figure 15 An example top-level functional block diagram of computing device embodiment 400 is shown. The example operating environment is shown as a computing device 420 including a processor 424 (such as a central processing unit (CPU)), addressable memory 427, external device interface 426 (e.g., optional universal serial bus port and associated processing and / or Ethernet port and associated processing), and optional user interface 429 (e.g., status light array and one or more toggle switches, and / or display, and / or keyboard, and / or pointer-mouse system, and / or touchscreen). Optionally, the addressable memory may include any type of computer-readable medium capable of storing data accessible to the computing device 420, such as magnetic hard disk and floppy disk drives, optical disk drives, cassette tapes, tape drives, flash memory cards, digital video discs (DVDs), Bernoulli cassette tapes, RAM, ROM, smart cards, etc. In practice, any medium for storing or transmitting computer-readable instructions and data can be used, including connection ports or nodes to a network (such as a LAN, WAN, or the Internet). These components may communicate with each other via data bus 428. In some embodiments, via an operating system 425 (e.g., an operating system supporting a web browser 423 and an application 422), the processor 424 may be configured to perform the steps of establishing a communication channel and processing according to the above embodiments.

[0075] Figure 16This is a high-level block diagram 500 illustrating a computing system including a computer system for implementing embodiments of the systems and processes disclosed herein. Embodiments of this system can be implemented in various computing environments. The computer system includes one or more processors 502 and may further include an electronic display device 504 (e.g., for displaying graphics, text, and other data), main memory 506 (e.g., random access memory (RAM)), storage device 508, removable storage device 510 (e.g., removable storage drive, removable memory module, magnetic tape drive, optical disc drive, computer-readable medium storing computer software and / or data), user interface device 511 (e.g., keyboard, touchscreen, keypad, pointing device), and communication interface 512 (e.g., modem, network interface (such as an Ethernet card), communication port, or PCMCIA slot and card). The communication interface 512 allows software and data to be transferred between the computer system and external devices. The system also includes a communication infrastructure 514 (e.g., a communication bus, crossbar, or network) to which the aforementioned devices / modules are connected as shown.

[0076] Information transmitted via communication interface 514 may be in the form of signals, such as electronic, electromagnetic, optical signals, or other signals that can be received by communication interface 514 via signal-carrying communication link 516, and may be implemented using wires or cables, optical fibers, telephone lines, cellular / mobile phone links, radio frequency (RF) links, and / or other communication channels. Computer program instructions representing the block diagrams and / or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing device to cause a series of operations performed thereon to produce a computer-implemented process.

[0077] Several embodiments have been described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments. Each block of these illustrations / diagrams, or combinations thereof, can be implemented by computer program instructions. When provided to a processor, the computer program instructions generate machinery, such that the instructions, executed by the processor, create means for implementing the functions / operations specified in the flowcharts and / or block diagrams. Each block in the flowcharts / block diagrams may represent a hardware and / or software module or logic implementing an embodiment. In alternative implementations, the functions recorded in a block may not appear simultaneously in the order shown in the figures, etc.

[0078] The computer program (i.e., computer control logic) is stored in main memory and / or auxiliary memory. The computer program may also be received via communication interface 512. When executed, this computer program enables the computer system to perform the features of the embodiments discussed herein. Specifically, when executed, the computer program enables the processor and / or multi-core processor to perform features of the computer system. This computer program represents the controller of the computer system.

[0079] Figure 17 A block diagram of an example system 600 in which embodiments can be implemented is shown. System 600 includes one or more client devices 601, such as consumer electronic devices, connected to one or more server computing systems 630. Server 630 includes a bus 602 or other communication mechanism for transmitting information and a processor (CPU) 604 coupled to bus 602 for processing information. Server 630 also includes main memory 606 (such as random access memory (RAM) or other dynamic storage device) coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. Server computer system 630 also includes read-only memory (ROM) 608 or other static storage device coupled to bus 602 to store static information and instructions for processor 604. Storage device 610 (such as a disk or optical disk) is provided and coupled to bus 602 for storing information and instructions. Bus 602 may include, for example, thirty-two address lines for addressing video memory or main memory 606. Bus 602 may also include, for example, a 32-bit data bus for transferring data between components such as CPU 604, main memory 606, video memory, and storage device 610. Alternatively, multiplexed data / address lines may be used instead of separate data and address lines.

[0080] Server 630 can be coupled to display 612 via bus 602 for displaying information to a computer user. Input device 614 (including alphanumeric and other keys) is coupled to bus 602 for transmitting information and command selection to processor 604. Another type of user input device includes cursor control 616, such as a mouse, trackball, or cursor arrow keys for transmitting directional information and command selection to processor 604 and for controlling cursor movement on display 612.

[0081] According to one embodiment, the functionality is performed by a processor 604, which executes one or more sequences of instructions contained in main memory 606. Such instructions may be read into main memory 606 from another computer-readable medium (such as storage device 610). Execution of the sequence of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. One or more processors in a multiprocessor arrangement may also be used to execute the sequence of instructions contained in main memory 606. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the embodiment. Therefore, the embodiment is not limited to any particular combination of hardware circuitry and software.

[0082] The terms "computer program medium," "computer-usable medium," "computer-readable medium," and "computer program product" are generally used to refer to media such as main memory, secondary memory, removable storage drives, hard disks mounted in hard disk drives, and signals. These computer program products are means for providing software to a computer system. A computer-readable medium allows a computer system to read data, instructions, messages or message packets, and other computer-readable information from the computer-readable medium. For example, a computer-readable medium may include non-volatile memory such as floppy disks, ROM, flash memory, disk drive memory, CD-ROM, and other permanent storage devices. For example, it can be used to transfer information, such as data and computer instructions, between computer systems. Furthermore, a computer-readable medium may include computer-readable information in transient state media, such as network links and / or network interfaces, including wired or wireless networks that allow computers to read such computer-readable information. A computer program (also called computer control logic) is stored in main memory and / or secondary memory. A computer program may also be received via a communication interface. When executed, such a computer program enables a computer system to perform the features of the embodiments discussed herein. Specifically, when executed, the computer program enables a processor (multi-core processor) to perform the features of the computer system. Correspondingly, this computer program represents the controller of a computer system.

[0083] Generally, as used herein, the term "computer-readable medium" refers to any medium that participates in providing instructions to processor 604 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. For example, non-volatile media include optical discs or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 602. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0084] For example, common forms of computer-readable media include floppy disks, floppy disks, hard disks, magnetic tapes or any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette memory, carrier waves as described below, or any other media from which a computer can read.

[0085] Various forms of computer-readable media may involve carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions may initially be carried on the disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions via a telephone line using a modem. A modem local to server 630 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector coupled to bus 602 may receive the data carried in the infrared signal and place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. Instructions received from main memory 606 may optionally be stored on storage device 610 before or after execution by processor 604.

[0086] Server 630 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides bidirectional data communication coupled to network link 620, which connects to a global packet data communication network commonly referred to as the Internet 628. The Internet 628 uses electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks and on network link 620 and via communication interface 618 are exemplary forms or carriers for transmitting information, carrying digital data to and from server 630.

[0087] In another embodiment of server 630, interface 618 is connected to network 622 via communication link 620. For example, communication interface 618 may be an Integrated Services Digital Network (ISDN) card or modem to provide data communication connectivity to a corresponding type of telephone line, which may include part of network link 620. As another example, communication interface 618 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN. A wireless link may also be implemented. In any such implementation, communication interface 618 transmits and receives electromagnetic or photoelectric signals carrying digital data streams representing various types of information.

[0088] Network link 620 typically provides data communication to other data devices via one or more networks. For example, network link 620 may provide a connection to host computer 624 or data devices operated by an Internet Service Provider (ISP) via local area network 622. The ISP, in turn, provides data communication services via the Internet 628. Both local area network 622 and Internet 628 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks and on network link 620 and via communication interface 618 are exemplary forms or carriers for transmitting information, carrying digital data to and from server 630.

[0089] Server 630 can send / receive messages and data, including emails and program code, via a network, network link 620, and communication interface 618. Furthermore, communication interface 618 may include a USB / tuner, and network link 620 may be an antenna or cable for connecting server 630 to a cable provider, satellite provider, or other terrestrial transmission system for receiving messages, data, and program code from another source.

[0090] Example versions of the embodiments described herein can be implemented as logical operations in a distributed processing system, such as system 600 including server 630. The logical operations of the embodiments can be implemented as a series of steps performed in server 630, and as interconnected machine modules within system 600. The implementation method is a matter of choice and can depend on the performance of system 600 implementing the embodiments. Therefore, the logical operations constituting the example versions of the embodiments are referred to, for example, as operations, steps, or modules.

[0091] Similar to the server 630 described above, the client device 601 may include a processor, memory, storage device, display, input device, and communication interface (e.g., email interface) for connecting the client device to the Internet 628, ISP, or LAN 622 to communicate with the server 630.

[0092] System 600 may also include a computer (e.g., a personal computer, a computing node) 605 that operates in the same manner as client device 601, wherein a user may use one or more computers 605 to manage data in server 630.

[0093] Now for reference Figure 18The illustration depicts a cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, with local computing devices used by cloud consumers (such as, for example, personal digital assistants (PDAs), smartphones, smartwatches, set-top boxes, video game systems, tablets, mobile computing devices or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N) capable of communicating with these cloud computing nodes. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain resources on their local computing devices. It should be understood that... Figure 18 The types of computing devices 54A-N shown are merely illustrative, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0094] The disclosed embodiments described above may include exemplary methods for determining machining toolpaths, such as... Figure 19 The top-level flowchart is shown. Exemplary steps may include the following steps: (a) receiving a first state of the tool (130) of a computer numerical control (CNC) machine tool by an interpreter component (104) having a processor with addressable memory (step 810); (b) determining a reward and a reward value by the interpreter component (104) based on the received first state (130), wherein the reward is at least one of the following: positive and negative (step 820); (c) transmitting a set of information including the determined reward and reward value by the interpreter component to an agent component (110) (step 830); (d) performing at least one action (112) by the agent component (110) to generate a toolpath (180, 182, 184) and entering a second state (132), wherein the second state (132) is combined with the first state (130) (step 840); (e) determining the generated toolpath by the agent component (110) based on the determined reward and value associated with at least one action (step 850). Optionally, the method may further include: (f) at least one action (112) is determined based on at least one of the following: changing the tool, moving the tool, opening the spindle, and closing the spindle (step 860).

[0095] It is conceivable that various combinations and / or sub-combinations of the specific features and aspects of the above embodiments can be made, and still fall within the scope of the invention. Therefore, it should be understood that various features and aspects of the disclosed embodiments can be combined with or substituted for each other to form variations of the disclosed invention. Furthermore, the scope of the invention is intended to be disclosed herein by way of example and should not be limited to the specific disclosed embodiments described above.

Claims

1. A method for a virtual environment, comprising: The first state of the cutting tool of the computer numerical control (CNC) machine tool is received by a first computing device component having a processor and addressable memory; The first computing device component evaluates a reward and the value of the reward based on the received first state, wherein the reward is a positive reward or a negative reward; The first computing device component transmits a set of information, including the evaluated reward and the value of the reward, and the received first state, to the second computing device component; The second computing device component performs at least one action as part of generating a toolpath, wherein the at least one action includes at least one of changing the tool, moving the tool, turning on the spindle, and turning off the spindle, and the at least one action is constrained by a strategy that includes data about a set of actions available for the CNC machine tool; The second computing device component creates a subsequent state to be entered based on at least one action performed, the subsequent state being a second state, wherein the second state is a virtual representation of the blank after the at least one action is performed on the virtual blank; The second state is transmitted from the second computing device component to the first computing device component for the evaluation; The process of iteratively evaluating the reward and its value, transmitting the set of information, generating the toolpath, creating the second state, and transmitting the second state continues until a final state is reached, in which there is no remaining blank to be machined and / or the number of iterations performed exceeds a previously determined threshold. In each subsequent state, the second computing device component determines a series of actions for the toolpath based on the evaluated reward and value associated with at least one action performed, thereby generating steps for training the manufacturing machinery using a virtual environment for reinforcement learning in manufacturing; and Each of the series of actions is determined based on at least one of the following: changing the tool, moving the tool, opening the spindle, and closing the spindle.

2. The method according to claim 1, wherein, The first state includes the state of the cutting tool.

3. The method according to claim 2, wherein, The entire system is represented by at least one of the following: cutting inserts, discrete parts, cutting tools, and blank material.

4. The method according to claim 1, wherein, The at least one action includes at least one of the following: turning operation, milling operation, additive directional energy deposition (DED), wire electrical discharge machining (EDM), and additive powder bed.

5. The method according to claim 1, wherein, The at least one action is part of a series of actions used to generate a toolpath.

6. A computing device having a processor and a memory, said computing device being configured to: Receive the first status of the cutting tool from the computer numerical control (CNC) machine tool; The reward is evaluated based on the received first state and the value of the reward, wherein, The reward can be a positive reward or a negative reward; At least one action is performed as part of generating a toolpath, wherein the at least one action includes at least one of changing the tool, moving the tool, opening the spindle, and closing the spindle, and the at least one action is constrained by a strategy that includes data about a set of actions available for the CNC machine tool; Based on the execution of at least one action, a subsequent state is entered, which is a second state, wherein the second state is a virtual representation of the blank after the execution of the at least one action on the virtual blank; Transmit the second state for the evaluation; The process of iteratively evaluating the reward and its value, generating the toolpath, entering the second state, and transmitting the second state continues until a final state is reached, in which there is no remaining blank to be machined and / or the number of iterations performed exceeds a previously determined threshold. In each subsequent state, a series of actions for the toolpath are determined based on the evaluated reward and value associated with at least one action performed, thereby generating steps for training the manufacturing machinery using a virtual environment for reinforcement learning in manufacturing; and Each of the series of actions is determined based on at least one of the following: changing the tool, moving the tool, opening the spindle, and closing the spindle.

7. The computing device according to claim 6, wherein, The first state includes the state of the cutting tool.

8. The computing device according to claim 7, wherein, The entire system is represented by at least one of the following: cutting inserts, discrete parts, cutting tools, and blank material.

9. The computing device according to claim 6, wherein, The at least one action includes at least one of the following: turning operation, milling operation, additive directional energy deposition (DED), wire electrical discharge machining (EDM), and additive powder bed.

10. The computing device according to claim 6, wherein, The at least one action is part of a series of actions used to generate a toolpath.

11. A computer-aided manufacturing (CAM) system for a computer numerical control (CNC) machine tool, the system comprising a first computing device component, a second computing device component, and a third computing device component; in, The first computing device component is configured as follows: Receive the first state of the cutting tool of the CNC machine tool; The reward is evaluated based on the received first state and the value of the reward, wherein the reward is a positive reward or a negative reward; Transmit to the second computing device component a set of information including the evaluated reward and the value of the reward; and The second computing device component is configured as follows: Perform at least one action to generate a toolpath, wherein the at least one action includes at least one of changing the tool, moving the tool, turning on the spindle, and turning off the spindle; A strategy is received from a third computing device component, wherein at least one action is restricted by the strategy, and the strategy includes data about a set of actions that can be used on the CNC machine tool; Based on the execution of at least one action, a subsequent state is entered, which is a second state, wherein the second state is a virtual representation of the blank after the execution of the at least one action on the virtual blank; The second state is transmitted from the second computing device component to the first computing device component for the evaluation; The process of iteratively evaluating the reward and its value, transmitting the set of information, generating the toolpath, receiving the strategy, entering the second state, and transmitting the second state continues until a final state is reached, in which there is no remaining blank to be machined and / or the number of iterations performed exceeds a previously determined threshold. In each subsequent state, a series of actions for the toolpath are determined based on the evaluated reward and value associated with at least one action performed, thereby generating steps for training the manufacturing machinery using a virtual environment for reinforcement learning in manufacturing; and Each of the series of actions is determined based on at least one of the following: changing the tool, moving the tool, opening the spindle, and closing the spindle.

12. The system according to claim 11, wherein, The state of the cutting tools on the CNC machine tool is a representation of the virtual environment.

13. The system according to claim 12, wherein, The virtual environment includes the target part, the blank part, and the cutting tool.

14. The system according to claim 13, wherein, The final state is also based on the premise that no target part is empty and therefore needs to be added.

15. The system according to claim 11, wherein, The strategy is at least one of avoiding collisions and avoiding damage to parts.

16. The system according to claim 11, wherein, The strategy is based on a fixed set of rules to restrict the at least one action.

17. The system according to claim 11, wherein, The second computing device component is also configured to select subsequent actions based on the history of a previously executed set of actions.

18. The system according to claim 11, wherein, Virtual environments for reinforcement learning in manufacturing utilize continuous, non-discrete, contour-based environments.

Images