Management apparatus, processing system, management method, and article manufacturing method

Abstract

To provide a technology advantageous for efficiently performing control satisfying required accuracy of each of a plurality of drive shafts at a limited calculation cost.SOLUTION: Provided is a management apparatus for managing a processing apparatus including a driver configured to drive a target object in regard to a plurality of drive axes, and a controller configured to control the driver using a neural network for which a parameter for outputting a manipulated variable to the target object is decided by reinforcement learning. The management apparatus includes a learning unit configured to decide the parameter of the neural network by reinforcement learning. The learning unit performs the reinforcement learning by evaluating a reward obtained from a control result of the target object by the controller, and relatively adjusts rewards regarding the respective drive axes in accordance with required precision for the respective drive axes.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a management device, a processing system, a management method, and an article manufacturing method.

Background Art

[0002] For a control device that controls a physical quantity of a target object, a classical controller such as PID control is often used. In recent years, in addition to control systems based on classical control theory and modern control theory, control systems constructed using machine learning (including reinforcement learning) may be used. Further, a control system that combines a control system that does not include machine learning and a control system based on machine learning may also be used. In Patent Document 1, by using a feedback control device that combines a control system that does not include machine learning and a control system based on machine learning, a control deviation of an object that cannot be compensated only by a control system that does not include machine learning is reduced by the control system based on machine learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the control system performs control regarding a plurality of drive shafts, the required accuracies of the respective drive shafts may be different from each other. In that case, when performing feedback control regarding each drive shaft by the method shown in Patent Document 1, while there may be a drive shaft whose control deviation satisfies the required accuracy, there may also be a drive shaft whose control deviation cannot satisfy the required accuracy. In order to satisfy the required accuracy for all drive shafts, it is necessary to design the control system for each drive shaft in accordance with the drive shaft having the most stringent required accuracy. However, in that case, for a drive shaft whose required accuracy is not stringent, it becomes overspec, resulting in an unnecessary computational cost, which may be disadvantageous in terms of efficiency.

[0005] This invention provides a technology that is advantageous for efficiently controlling multiple drive axes while satisfying the required accuracy for each of them with limited computational cost. [Means for solving the problem]

[0006] According to one aspect of the present invention, a management device is provided for managing a processing device that includes a drive unit for driving an object with respect to a plurality of drive axes, and a control unit for controlling the drive unit using a neural network whose parameters for outputting an operation amount for the object are determined by reinforcement learning, wherein the management device comprises a learning unit that determines the parameters of the neural network by reinforcement learning, and the learning unit performs the reinforcement learning by evaluating the reward obtained from the control result of the object control by the control unit, and relatively adjusts the reward for each of the plurality of drive axes according to the required accuracy for each of the plurality of drive axes. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technology that is advantageous for efficiently controlling multiple drive axes while satisfying the required accuracy of each of them with limited computational cost. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example of a manufacturing system configuration. [Figure 2] A diagram showing an example of a stage setup configuration. [Figure 3] A block diagram showing an example of the control system configuration for a stage device. [Figure 4] A diagram showing an example configuration of the second control system. [Figure 5] A diagram showing an example of a neural network configuration. [Figure 6] A diagram showing the relationship between stage deviation and reward. [Figure 7] A flowchart showing how rewards are determined. [Figure 8] A diagram showing an example of a neural network configuration. [Figure 9] A diagram showing an example of a stage setup configuration. [Figure 10] A diagram showing an example of the configuration of a vibration isolation device. [Figure 11] A diagram showing an example of the configuration of the control system for a vibration isolation device. [Figure 12] A diagram showing an example configuration of an imprint device. [Figure 13] This diagram shows an example of the control system configuration for an imprinting device. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] <First Embodiment> Figure 1 shows the configuration of a manufacturing system MS (processing system) in one embodiment. The manufacturing system MS may include, for example, a processing device 1, a control device 2 that controls the processing device 1, and a management device 3 that manages the processing device 1 and the control device 2. The processing device 1 is a device that performs processing on an object to be processed, such as a manufacturing device, inspection device, or monitoring device. The concept of processing may include processing, inspecting, monitoring, and observing the object to be processed. Alternatively, the processing device 1 may be a stage device that holds and moves a substrate, or a vibration isolation device that reduces vibrations transmitted to an object such as a substrate, which are included in the above-mentioned manufacturing device, etc. The first embodiment is an embodiment in which the processing device 1 is a stage device in a lithography device. An embodiment in which the processing device 1 is a vibration isolation device will be described later as the fourth embodiment.

[0011] The processing device 1 includes a control target (object), and can control the control target by using a neural network for outputting an operation amount for the object. A plurality of parameter values of the neural network can be determined by reinforcement learning. The control device 2 can be configured to send a drive command to the processing device 1 and receive a drive result or a control result from the processing device 1. The management device 3 can perform reinforcement learning for determining a plurality of parameter values of the neural network of the processing device 1. Specifically, the management device 3 can determine the plurality of parameter values by repeating the operation of sending a drive command to the processing device 1 and receiving a drive result from the processing device 1 while changing all or part of the plurality of parameter values of the neural network. Therefore, the management device 3 may be understood as a learning device (learning unit).

[0012] All or part of the functions of the control device 2 may be incorporated into the management device 3. All or part of the functions of the control device 2 may be incorporated into the processing device 1. The processing device 1, the control device 2, and the management device 3 may be physically integrated, or may be physically configured as separate entities. The entire processing device 1 may be controlled by the control device 2, or may include components that are not controlled by the control device 2 in addition to the components controlled by the control device 2.

[0013] Hereinafter, an example in which the above manufacturing system MS is applied to a lithography apparatus will be described. A lithography apparatus generally includes a stage apparatus that holds and moves a substrate. Note that a lithography apparatus is an apparatus that forms a pattern on a substrate. Specific examples of the lithography apparatus include, for example, an exposure apparatus, an imprint apparatus, an electron beam drawing apparatus, etc. Hereinafter, the lithography apparatus will be described as an imprint apparatus. An imprint apparatus is an apparatus that forms a pattern on a substrate by curing an imprint material while bringing a mold (original plate) into contact with the imprint material supplied onto the substrate.

[0014] Figure 2 shows the configuration of the stage device 1000 used in the imprint apparatus as the processing apparatus 1 of this embodiment. In this specification and drawings, directions are indicated in an XYZ coordinate system in which the horizontal plane is the XY plane. The stage device 1000 holds the substrate W on the holding surface of the stage section 13 so that the surface of the substrate W is parallel to the horizontal plane (XY plane). Therefore, below, the directions that are orthogonal to each other in the plane along the holding surface of the stage section 13 are referred to as the X axis and the Y axis, and the direction perpendicular to the X axis and the Y axis is referred to as the Z axis. Also, below, the directions parallel to the X axis, Y axis and Z axis in the XYZ coordinate system are referred to as the X direction, Y direction and Z direction, and the direction of rotation around the Z axis is referred to as the θ direction.

[0015] The stage device 1000 comprises a main body 100 and a control system 200. The substrate W, which is the object to be controlled, is held by the stage unit 13 via a substrate chuck 11. The stage unit 13 moves the substrate W with sufficient stroke in the X and Y directions to perform pattern formation processing (imprint processing) on ​​each shot area of ​​the entire surface of the substrate W. The stage unit 13 also has sufficient stroke in the X and Y directions to move the substrate W to a replacement position where it is loaded and unloaded by a substrate replacement hand (not shown).

[0016] The stage unit 13 may include an X stage 13X, a Y stage 13Y, and a θ stage 13θ. The linear motor 19, which is a driving unit, may include an X linear motor 19X, a Y linear motor 19Y, and a θ linear motor 19θ. The X linear motor 19X, the Y linear motor 19Y, and the θ linear motor 19θ drive the X stage 13X, the Y stage 13Y, and the θ stage 13θ, respectively. The X stage 13X is guided to move freely in the X direction using hydrostatic guidance, and a driving force is applied in the X direction by the X linear motor 19X. On the X stage 13X, the Y stage 13Y can move in the Y direction by hydrostatic guidance and the Y linear motor 19Y. Further, on the Y stage 13Y, the θ stage 13θ can move in the θ direction by hydrostatic guidance and the θ linear motor 19θ. These linear motors are driven by a driver 14. The driver 14 supplies a current (electrical energy) corresponding to the instruction value given from the control system 200 to the linear motor 19. However, the configuration of the stage unit 13 is not limited to this. In particular, a more highly accurate positioning stage can be applied to the stage device in a lithography device such as an imprint device or an exposure device. An example of such a stage device will be described in the third embodiment.

[0017] The position measurement unit 18 may include an X stage position measurement unit 18X. The position of the X stage 13X in the X direction is measured by the X stage position measurement unit 18X. The X stage position measurement unit 18X is, for example, a linear encoder, and may include a scale (not shown) disposed on the surface plate 17 on the stage device 1000 and a head and a calculation unit on the X stage 13X. The position measurement unit 18 may further include a linear encoder (not shown) that measures the position of the Y stage 13Y in the Y direction and a linear encoder (not shown) that measures the position of the θ stage 13θ in the θ direction. Instead of these linear encoders, the measurement of the position of the stage unit 13 in each direction may be performed using a combination of an interferometer disposed in the main body structure of the imprint device and a reflection mirror disposed in the stage unit 13.

[0018] The control system 200 (control unit) can control the position or state of the stage unit 13, which is the object of control, using a neural network whose parameter values ​​are determined by reinforcement learning. Figure 3 is a block diagram showing the configuration of the control system 200. The area within the dashed lines is the control system 200. The control system 200 is composed of a digital computer to perform complex calculations. Such a digital computer may include arithmetic processing units such as a CPU or FPGA, and memory devices such as memory. The control system 200 may include a main control unit 206, a position command unit 203, and a stage control unit 201.

[0019] The main control unit 206 is responsible for sending commands to the stage control unit 201 and other operating units (not shown). The main control unit 206 may be a control unit that controls the entire imprint device. However, the functions of the main control unit 206 may also be implemented in the control device 2.

[0020] The stage position command unit 203 acquires and stores the coordinates indicating the target position of the stage unit 13 from the main control unit 206, and sends these values ​​to the stage control unit 201. The position measurement unit 18 measures the position of the stage unit 13 (stage position) at each sampling time and sends the stage position obtained from the measurement to the stage control unit 201.

[0021] The stage control unit 201 may include a deviation calculation unit 213, a first control system 211, a second control system 212, and an adder 214. The deviation calculation unit 213 calculates the difference between the stage position (measured value) received from the position measurement unit 18 and the stage position (target value) received from the stage position command unit 203. This difference is the control deviation, more specifically the position deviation, and will be referred to as the "stage deviation" below. The stage deviation is sent to the first control system 211 and the second control system 212. The first control system 211 is a first compensator that generates a first command value based on the stage deviation. The first control system 211 may include a PID controller. This PID controller takes information regarding the stage deviation as input and outputs a first manipulated variable U1 (first command value) for the stage unit 13.

[0022] The second control system 212 is a second compensator that generates a second command value based on the stage deviation. Figure 4 shows the configuration of the second control system 212. The second control system 212 may include a memory 215 that stores the history of stage deviations and a neural network (hereinafter referred to as "NN") 216. Based on the input stage deviation, the NN 216 outputs a value (second command value) that corresponds to the correction value of the first manipulated variable U1 output from the first control system 211.

[0023] Figure 5 shows an example of the NN216 configuration. The NN216 has hidden layers H1 and H2, which are intermediate layers, between the input layer I and the output layer O. The input layer I receives the stage deviations for the neighboring N samples of each axis. In hidden layer H1, calculations are performed on the input stage deviations for each axis, and in hidden layer H2, calculations are performed on the values ​​from hidden layer H1 for each axis. In other words, in hidden layer H2, calculations are performed on the stage deviations independently for each axis.

[0024] This allows NN216 to independently output the outputs U2_X, U2_Y, and U2_θ for each axis based on the stage deviation of each axis. Note that the NN216 shown in Figure 5 is just one example and is not limited to this form. For example, a configuration in which the stage deviations of the nearest N samples for each axis are concatenated before performing calculations on the hidden layer H1 is also possible.

[0025] The parameters of NN216 can be adjusted by reinforcement learning performed by the management device 3. Reinforcement learning is a type of machine learning method that represents the quality of an actor's actions with a numerical reward, and determines the actor's rules of action to maximize the cumulative value R of this reward over time. In other words, by appropriately defining the content of the reward, it is possible to obtain a superior actor that can reach a desired state as a result of accumulating actions toward the future.

[0026] The cumulative value R is expressed, for example, by the following formula. In the first embodiment, the second control system 212 is the main entity responsible for the action.

[0027]

number

[0028] The reinforcement learning method is not limited to a specific method, but for example, DQN (Deep Q-network) and PPO (Proximal Policy Optimization) can be applied. NN216 may be a network that directly outputs something corresponding to the dimension of the directive (policy network), or it may be a network that calculates the value of the directive (action-value network). In the case of an action-value network, a selection unit is added after the NN to choose the action with the maximum value, and the directive value selected by the selection unit becomes the output of the second control system 212 (second manipulated variable U2).

[0029] The adder 214 outputs the result (command value) of adding the first manipulated variable U1 (first command value) generated by the first control system 211 and the second manipulated variable U2 (second command value) generated by the second control system 212. The output of the adder 214 is sent to the driver 14 as an analog signal through a D / A converter (not shown). The driver 14 controls the current flowing through the coil of the linear motor 19 according to the value received from the adder 214. Since the thrust of the linear motor 19 is proportional to the current flowing through the coil, a force corresponding to the sum of the output values ​​of the first control system 211 and the second control system 212 is applied to the stage section 13.

[0030] In this configuration, the first control system 211 primarily handles the position feedback control system, while the second control system 212 further reduces stage deviations that cannot be fully compensated by the first control system 211. As a result, the stage deviation can be reduced compared to a control system using only the first control system 211. The first control system 211 may be, for example, a PID compensator, but it may also be another type of compensator. The first control system 211 is not necessarily required, and the second control system 212 alone may generate the command values ​​given to the driver 14.

[0031] Reinforcement learning is performed by evaluating the reward obtained from the control result of the control system 200. The reward in reinforcement learning is described below. In one example, the reward r in reinforcement learning is expressed by the following equation.

[0032] r=Gx·rx+Gy·ry+Gθ·rθ …(2) However, rx, ry, and rθ represent the rewards related to the X, Y, and θ axes, respectively, and Gx, Gy, and Gθ represent the weights of the rewards related to the X, Y, and θ axes, respectively.

[0033] Thus, the reward r in reinforcement learning is expressed as a weighted sum of the rewards for each axis. The rewards rx, ry, and rθ for each axis can be determined according to the magnitude of the stage deviation for each axis, as shown in Figure 6. According to Figure 6, the smaller the stage deviation, the greater the reward obtained.

[0034] The weights Gx, Gy, and Gθ for the reward on each axis are determined according to the required precision for each axis. For example, if the required precision for the X axis is stricter than that for the Y and θ axes, the value of the weight Gx is made larger than that of Gy and Gθ. In other words, equation (2) above is an equation that shows that a larger reward r can be obtained if the stage deviation of each axis is reduced according to the required precision.

[0035] Referring to the flowchart in Figure 7, the method for determining the reward weights Gx, Gy, and Gθ for each axis by the management device 3 will be explained. In S1, the management device 3 acquires information on the required accuracy for each axis. The required accuracy for each axis may be accuracy information input by the user as a request from the user. Then, in S2, the management device 3 determines the reward weights Gx, Gy, and Gθ for each axis based on the acquired required accuracy. In one example, if the required accuracy for the X axis is 3 nm and the required accuracy for the Y axis is 6 nm, the weight Gx is determined to be 2 and the weight Gy is determined to be 1. Such weights can be obtained, for example, by referring to a table that describes the pre-obtained correspondence between the required accuracy and the weights. Alternatively, a function (mathematical formula) representing the weights, with the required accuracy as a variable, may be determined in advance, and the weights can be obtained by applying the acquired required accuracy to this function. The management device 3 calculates the reinforcement learning reward using the determined weights for each axis. In this way, the reward for each of the multiple drive axes is adjusted relatively according to the acquired required accuracy for each of the multiple drive axes.

[0036] As described above, in this embodiment, the management device 3 (learning unit) performs reinforcement learning of the NN216 by evaluating the reward obtained from the control result of the stage unit 13 by the control system 200. The management device 3 can adjust the NN216 to satisfy the user's request. For example, according to this embodiment, the relative adjustment of the rewards according to the required accuracy for each of the multiple drive axes is performed by weighting the rewards for each axis in this way. This makes it possible to efficiently reduce the stage deviation of each axis with limited computational cost.

[0037] Furthermore, the required accuracy for each axis may be determined not only by the stage deviation of each axis, but also by the time it takes for the stage deviation of each axis to settle to a predetermined size. In that case, the reward for each axis will be determined according to the settling time of each axis. Also, in the above example, the stage deviation of each axis was used as the input to NN216, but other information related to each axis may also be used.

[0038] <Second Embodiment> In the second embodiment, an example of a different NN configuration from the NN216 of the first embodiment (Figure 5) is described. Figure 8 shows an example configuration of the NN216 according to the second embodiment. The NN216 has hidden layers H1 and H2, which are intermediate layers, between the input layer I and the output layer O. The input layer I receives the stage deviations for the neighboring N samples of each axis. In hidden layer H1, calculations are performed on the input stage deviations of each axis, and in hidden layer H2, calculations are performed on the values ​​from hidden layer H1 for all axes. In the hidden layer H2 of the first embodiment (Figure 5), calculations were performed on the stage deviations independently for each axis, but in the hidden layer H2 of the second embodiment, calculations are performed considering the stage deviations of all axes.

[0039] As a result, NN216 can output the outputs U2_X, U2_Y, and U2_θ for each axis, taking into account the magnitude of the stage deviation for all axes. For example, if there is a correlation between the stage deviations of the axes, using the NN216 of the second embodiment can reduce the stage deviation of each axis more efficiently and effectively than using the NN216 of the first embodiment. Note that the NN216 shown in Figure 8 is just one example and is not limited to this form. For example, a configuration in which the stage deviations of the neighboring N samples for each axis are concatenated before performing the calculation of the hidden layer H1 is also possible.

[0040] <Third Embodiment> In the third embodiment, an example of a stage device different from the stage device 1000 of the first embodiment (Figure 2) will be described. Figure 9 is a diagram showing the configuration of the stage device 2000 according to this embodiment. The stage device 2000 is a positioning device that moves a movable part as an object on a plane parallel to a first direction and a second direction which are mutually orthogonal.

[0041] The stage device 2000 is configured to move the stage section 2013 on the surface plate 2017 by a drive unit DP. The stage device 2000 is equipped with a single guide G as a guide that constrains the position of the stage section 2013 in the Y direction. This means that the stage device 2000 is equipped with only one guide G as a guide that constrains the position of the stage section 13 in the Y direction. The guide G has a guide surface 2026 parallel to the X direction, and the guide surface 2026 is a surface perpendicular to the surface plate 2017.

[0042] The stage section 2013 includes an X-movable body (first movable body) 2022, an X-beam (second movable body) 2012, and a Y-slider (third movable body) 2004. The X-movable body 2022 is movable in the X direction (first direction) while being guided by a guide G. The X-beam 2012 is supported and guided by a surface plate 2017 via a hydrostatic bearing composed of hydrostatic pads. The X-beam 2012 has a first end and a second end, the first end of which is connected to the X-movable body 2022 via a rotary bearing 2023 and moves on the surface plate 2017. The Y-slider 2004 is movable within a predetermined range between the first end and the second end by a drive unit (not shown) while being guided by the X-beam 2012.

[0043] The drive unit DP includes a first drive unit DP1 that drives the first end (the end on the +Y direction side) of the X-beam (second movable body) 2012 in the X direction (first direction), and a second drive unit DP2 that drives the second end (the end on the -Y direction side) of the X-beam 2012 in the X direction.

[0044] The first drive unit DP1 may consist of, for example, a linear motor. More specifically, the first drive unit DP1 may consist of a linear motor including a movable element (first movable element) 2024R and a stator (first stator) 2025R. The movable element 2024R may be connected to the first end of the X-beam 2012, and the stator 2025R may be connected to the side of the base plate 2017. The guide surface 2026 of the guide G may be positioned between the stator 2025R and the X-movable body 2022.

[0045] The second drive unit DP2 may consist of, for example, a linear motor. More specifically, the second drive unit DP2 may consist of a linear motor including a movable element (second movable element) 2024L and a stator (second stator) 2025L. The movable element 2024L may be connected to the second end of the X-beam 2012, and the stator 2025L may be connected to the side of the base plate 2017. There is no guide on the side of the second end of the X-beam 2012 that constrains the position of the second end in the Y direction.

[0046] In the first embodiment of the stage device 1000, the X stage 13X, Y stage 13Y, and θ stage 13θ could be positioned independently to position the substrate W on the stage section 13. However, in the third embodiment of the stage device 2000, the axes interfere with each other. Therefore, for example, to position the substrate on the stage section 2013 with respect to the X axis, it is necessary to simultaneously control not only the X position of the X movable body 2022, but also the angle θ of the X beam and the Y position of the Y slider. In cases where the axes interfere with each other, as in the stage device 2000, having a hidden layer H2 in NN216 that performs calculations considering the stage deviation of all axes, as shown in Figure 8, is particularly effective in effectively reducing the stage deviation of each axis with limited computational cost.

[0047] <Fourth Embodiment> In the fourth embodiment, an embodiment of a vibration isolation device that reduces vibrations transmitted to an object will be described. Figure 10 is a diagram showing an example of the configuration of the vibration isolation device 3000 according to this embodiment. The vibration isolation device 3000 has a main body 300 and a control system 200. The main body 100 of a stage device 1000, as shown in Figure 2, can be mounted on the main body 300. In the main body 300, the main body structure (vibration isolation platform) 101 is installed on the floor 103 via a tripod or quadruped vibration isolation mechanism 102 using air springs or the like. A linear motor 109, which is a drive unit, is attached to the main body structure 101. The linear motor 109 is configured to apply force in six axial directions to the main body structure 101. An accelerometer 48 is placed on the main body structure 101. The accelerometer 48 is configured to measure the acceleration of the main body structure 101 in six axial directions. The acceleration information of the main body structure 101 measured by the accelerometer 48 is sent to the control system 200.

[0048] Figure 11 is a block diagram showing the configuration of the control system 200. The area within the dashed lines represents the control system 200. The control system 200 may include a main control unit 206, a speed command unit 243, and a vibration isolation control unit 241.

[0049] The speed command unit 243 sends the target speed of the main structure 101 from the main control unit 206 to the vibration isolation control unit 241. The vibration isolation control unit 241 may include a deviation calculation unit 253, a first-order integrator 261, a first control system 251, a second control system 252, and an adder 254. The first-order integrator 261 calculates the speed of the main structure 101 by integrating the acceleration of the main structure 101 received from the accelerometer 48. The deviation calculation unit 253 calculates the difference (control deviation, hereinafter referred to as "speed deviation") between the speed of the main structure 101 obtained by the first-order integrator 261 and the speed of the main structure 101 (target value) received from the speed command unit 243. This speed deviation is sent to the first control system 251.

[0050] The first control system 251 is a first compensator that generates a first command value based on the speed deviation. For example, the first control system 251 may include a PID controller. This PID controller takes information about the speed deviation as input and outputs a first manipulated variable U11 (first command value) for the main structure 101.

[0051] The acceleration information of the main structure 101 obtained by the accelerometer 48 is also sent to the second control system 252. The second control system 252 is a second compensator that generates a second manipulated variable U12 (second command value) based on the acceleration of the main structure 101 (vibration isolation table) measured by the accelerometer 48. The second control system 252 includes a NN and outputs the second manipulated variable U12 based on the acceleration of the main structure 101 received from the accelerometer 48. Before being sent to the second control system 252, the acceleration of the main structure 101 may be processed by a cutoff filter, differentiator, integrator, etc. (not shown) to remove predetermined frequency components. The cutoff filter may be a low-pass filter, high-pass filter, band-pass filter, etc.

[0052] The adder 254 outputs the result (command value) of adding the first manipulated variable U11 (first command value) generated by the first control system 251 and the second manipulated variable U12 (second command value) generated by the second control system 252. The output of the adder 254 is sent to the driver 44 as an analog signal through a D / A converter (not shown). The driver 44 controls the current flowing through the coil of the linear motor 109 according to the value received from the adder 254. Since the thrust of the linear motor 109 is proportional to the current flowing through the coil, a force corresponding to the sum of the output values ​​of the first control system 251 and the second control system 251 is applied to the main structure 101.

[0053] In this configuration, the first control system 251 is primarily responsible for the velocity feedback control system, while the second control system 252 reduces the acceleration of the main structure 101 that cannot be fully compensated by the first control system 251. As a result, vibrations of the main structure 101 can be reduced compared to a control system consisting only of the first control system 251. The first control system 251 may be, for example, a PID compensator, but it may also be another type of compensator. The first control system 251 is not necessarily required, and the second control system 252 alone may generate the command values ​​given to the driver 44.

[0054] The parameters of the NN in the second control system 252 are adjusted by reinforcement learning. The reinforcement learning reward is calculated as a weighted addition to the reward for each axis, using weights corresponding to the required accuracy (acceleration, velocity) of the vibration of each axis of the main structure 101, similar to the first embodiment. This makes it possible to reduce the vibration of each axis according to the required accuracy of each axis with limited computational cost.

[0055] Furthermore, in the above example, the first-order integrator 261 integrates the acceleration of the main structure 101 received from the accelerometer 48 to obtain the velocity of the main structure 101. However, the velocity of the main structure 101 may also be measured directly using a speedometer. In this case, the acceleration of the main structure 101 may be obtained by taking the first derivative of the speedometer's measurement. Also, in the above example, the acceleration of each axis was used as input to NN216, but other information related to each axis may also be used.

[0056] <Fifth Embodiment> In the fifth embodiment, an imprint apparatus 4000, which is one aspect of a molding apparatus to which the present invention is applied, will be described. Figure 12 is a diagram showing the configuration of the imprint apparatus 4000 according to this embodiment. The imprint apparatus 4000 may include a stage apparatus as shown in any of the first to third embodiments described above. Furthermore, the imprint apparatus 4000 may also include a vibration isolation apparatus as shown in the fourth embodiment described above.

[0057] The imprint apparatus 4000 is a device having a forming section that brings the imprint material 7 supplied onto the substrate W into contact with a mold M and applies curing energy to the imprint material 7 to form a pattern on a cured material onto which the uneven pattern of the mold M has been transferred. For example, the imprint apparatus 4000 supplies the imprint material 7 onto the substrate W and cures the imprint material 7 with the mold M, which has an uneven pattern formed on it, in contact with the imprint material 7 on the substrate W. Then, by widening the gap between the mold M and the substrate W and peeling (release) the mold M from the cured imprint material 7, the pattern of the mold M can be transferred to the imprint material 7 on the substrate W. This series of processes is called the imprint process, and the imprint process is performed for each of the multiple shot regions on the substrate W. In other words, when performing the imprint process on each of the multiple shot regions on a single substrate W, the imprint process is repeated as many times as there are shot regions on that single substrate W.

[0058] The imprint material 7 can be a photocurable resin. Such imprint material 7 is supplied to the shot area of ​​the substrate W by the dispenser 107. Specifically, the stage unit 13 is positioned so that the position on the substrate W where the imprint material 7 should be supplied is directly below the dispenser 107. Next, the stage unit 13 is positioned so that the position on the substrate W where the imprint material 7 should be supplied is directly below the mold M. The mold M is held by the imprint head 23. The imprint head 23 can move the mold M in the Z direction by the actuator 29. The mold M waits at a position Z above the substrate W until the position of the shot area of ​​the substrate W is moved directly below the mold M. When the shot position on the substrate W is positioned directly below the mold M, the imprint head 23 lowers the mold M, bringing the pattern portion of the mold M into contact with the imprint material 7. When manufacturing semiconductor devices etc. with an imprint machine, alignment with previous layers is important when transferring the pattern of the mold M to the imprint material 7 on the substrate W. The alignment detector 106 optically detects alignment marks (not shown) provided on both the substrate W and the mold M, performs image processing, and detects the misalignment of the alignment marks in the X and Y directions (misalignment between the substrate W and the mold M). This misalignment information is sent to the control system 200, and alignment is performed by correcting the X, Y position and θ angle of the stage unit 13. Once alignment is complete, exposure light is irradiated onto the imprint material 7 from the illumination system 108 to cure the imprint material 7. After the imprint material 7 has cured, the imprint head 23 raises the mold M and releases the mold M from the imprint material 7 on the substrate W. Through this series of steps, a pattern corresponding to the pattern engraved on the mold M is transferred to the imprint material 7 on the substrate W. Similarly, the imprint process is sequentially performed on the other shot areas. Once the imprint process is complete for all shot areas of a single substrate, the stage unit 13 moves to the substrate exchange position. Then, the imprinted substrate is collected by a substrate exchange hand (not shown), and the next new substrate is supplied.

[0059] Figure 13 shows the configuration of the control system 200 of the imprint apparatus 4000. The alignment detector 106 measures the misalignment between the substrate W and the mold M and sends the measured misalignment information of the substrate W and the mold M to the control system 200. The alignment position command unit 270 acquires and stores the target value of the misalignment between the substrate W and the mold M from the main control unit 206. The deviation calculation unit 271 calculates the difference (control deviation, hereinafter referred to as "alignment error") between the misalignment received from the alignment detector 106 and the target value of the misalignment received from the alignment position command unit 270. The alignment error is sent to the alignment controller 272. The alignment controller 272 uses, for example, a PI controller and takes the alignment error received from the deviation calculation unit 271 as input and outputs a correction value that corrects the target value of the stage position sent from the position command unit 203.

[0060] In the fifth embodiment, the misalignment information between the substrate W and the mold M, obtained by the alignment detector 106, is sent to the second control system 212. The second control system 212 has an NN216, similar to the first embodiment. Based on the input misalignment information between the substrate W and the mold M from the alignment detector 106, the NN216 outputs a value corresponding to the correction value of the first manipulated variable U1 of the first control system 211. In the first embodiment, the second control system 212 worked to further reduce stage deviations that could not be compensated by the first control system 211, but in the fifth embodiment, it works to reduce misalignments between the substrate W and the mold M that could not be compensated by the first control system 211. The misalignment information between the substrate W and the mold M, obtained by the alignment detector 106, may be passed through a cutoff filter (not shown) that removes predetermined frequency components before being sent to the second control system 212. The cutoff filter may be a low-pass filter, a high-pass filter, a band-pass filter, etc.

[0061] Furthermore, in the fifth embodiment, the positional misalignment information between the substrate W and the mold M obtained by the alignment detectors 106 for each axis was used as the input to NN216, but other information related to each axis may also be used.

[0062] <Sixth Embodiment> In the fifth embodiment described above, an imprinting apparatus was described as one aspect of a molding apparatus, which transfers the pattern of a mold to an imprinting material by bringing the imprinting material and the mold into contact. However, the present invention can also be applied as another aspect of a molding apparatus to a planarization apparatus that forms a planarization film of a composition on a substrate by bringing a moldable material (composition) on the substrate into contact with a member (mold) having a flat surface.

[0063] The underlying pattern on the substrate has an uneven profile resulting from the pattern formed in the previous process, and especially with the recent increase in the multilayer structure of memory elements, process substrates are now sometimes exhibiting steps of around 100 nm. Steps caused by gentle undulations across the entire substrate can be corrected by the focus tracking function of the scan exposure equipment used in the photoprocessing stage. However, fine-pitched irregularities that fall within the exposure slit area of ​​the exposure equipment may fall outside the exposure equipment's DOF ​​(Depth Of Focus). Conventionally, methods such as SOC (Spin On Carbon) and CMP (Chemical Mechanical Polishing) have been used to smooth the underlying pattern of the substrate by forming a planarization layer. However, conventional techniques have problems in achieving sufficient planarization performance, and the unevenness of the substrate is expected to increase further with increasing multilayering.

[0064] To solve this problem, a planarization apparatus that uses the imprint technology described above to planarize substrates is being considered. The planarization apparatus brings a flat surface of a component, or a component without a pattern (a planar template), into contact with an uncured composition that has been supplied to the substrate in advance, thereby performing localized planarization within the substrate surface. Subsequently, the composition is cured while in contact with the planar template, and the planar template is separated from the cured composition. This forms a planarized layer on the substrate. Since the planarization apparatus using imprint technology drops an amount of composition corresponding to the step height of the substrate, it is expected to improve the accuracy of planarization compared to existing methods.

[0065] In the case of a planarization apparatus, a planarization film is formed on the entire surface of the substrate in one go. At this time, by applying the above embodiment, it is possible to reduce the overflow and incomplete filling of the composition from the substrate.

[0066] Furthermore, the present invention can be applied not only to lithography apparatus including molding apparatus, but also to measuring apparatus and processing apparatus. The measuring apparatus has a feedback control device to control the position of the target object, and also has a measuring unit that measures the object whose position has been controlled by the feedback control device. Examples of the measuring unit include a contact-type probe and a non-contact-type interferometer. The processing apparatus also has the above-mentioned feedback control device to control the position of the target object, and also has a processing unit that processes the object whose position has been controlled by the feedback control device. Examples of the processing unit include a cutting tool and a laser.

[0067] <Embodiment of Article Manufacturing Method> The article manufacturing method in this embodiment is suitable for manufacturing articles such as microdevices, semiconductor devices, and elements having microstructures. The article manufacturing method of this embodiment includes a transfer step of transferring a pattern from a master plate onto a substrate using the above-mentioned lithography apparatus (such as an exposure apparatus, imprint apparatus, or drawing apparatus), and a processing step of processing the substrate after the transfer step. Furthermore, such a manufacturing method includes other well-known steps (such as oxidation, film formation, vapor deposition, doping, planarization, etching, resist stripping, dicing, bonding, and packaging). The article manufacturing method of this embodiment is advantageous over conventional methods in at least one of the performance, quality, productivity, and production cost of the article.

[0068] Next, a method for manufacturing articles (semiconductor IC elements, liquid crystal display elements, color filters, MEMS, etc.) using the aforementioned planarization apparatus will be described. This manufacturing method includes the steps of: planarizing a composition placed on a substrate (wafer, glass substrate, etc.) by bringing the composition into contact with a mold using the aforementioned planarization apparatus; curing the composition; and separating the composition from the mold. This forms a planarized film on the substrate. Then, the substrate on which the planarized film has been formed is processed, such as forming a pattern using a lithography apparatus, and the processed substrate is processed in other well-known processing steps to manufacture an article. Other well-known processes include etching, resist stripping, dicing, bonding, packaging, etc. According to this manufacturing method, articles of higher quality than conventional methods can be manufactured.

[0069] <Other> The physical quantities used by the control device for control are not limited to those used in the following embodiments, but are of any type as long as they are capable of feedback control. For example, physical quantities include the displacement, velocity, or acceleration of an object in the linear and rotational directions, the flow rate, velocity, and pressure of a gas or fluid. Other physical quantities include the liquid level of a fluid, the temperature of an object, gas, or liquid, and the current, voltage, and charge of an electrical circuit. Other physical quantities include magnetic flux and magnetic flux density in a magnetic field, and sound pressure in a sound field. These physical quantities are measured by a detection unit using a known detector (sensor), and the measured values ​​are input to the control device. The drive unit is an active element that changes the physical quantities of the controlled object. When the controlled object is the position, velocity, or acceleration of an object, motors or piezoelectric elements are used. When the controlled object is a gas or fluid, pumps or valves are used. When the controlled object is an electrical system, drivers that manipulate current or voltage are used.

[0070] The disclosures herein include at least the following control devices, processing systems, and methods for manufacturing articles. (Item 1) A management device for managing a processing device which includes a drive unit that drives an object with respect to multiple drive axes, and a control unit that controls the drive unit using a neural network whose parameters for outputting an amount of manipulation applied to the object are determined by reinforcement learning, The neural network comprises a learning unit that determines the parameters of the neural network by reinforcement learning, The aforementioned learning unit, Reinforcement learning is performed by evaluating the reward obtained from the control result of the object by the control unit. The reward for each of the multiple drive shafts is adjusted relatively according to the required accuracy for each of the multiple drive shafts. A management device characterized by the following features. (Item 2) The reward evaluated in the reinforcement learning described above is represented by the weighted sum of the rewards for each of the multiple drive axes, The learning unit determines each weight in the weighted sum according to the required accuracy for each of the multiple drive shafts. The control device described in item 1, characterized by the features described herein. (Item 3) The management device according to item 2, characterized in that the learning unit acquires the required accuracy for each of the plurality of drive shafts and determines the weight for the acquired required accuracy based on the correspondence between the previously obtained required accuracy and the weight. (Item 4) A processing device including a drive unit that drives an object with respect to multiple drive axes, and a control unit that controls the drive unit using a neural network whose parameters for outputting an amount of manipulation applied to the object are determined by reinforcement learning, The learning device includes a device that determines the parameters of the neural network by reinforcement learning, The learning device is Reinforcement learning is performed by evaluating the reward obtained from the control result of the object by the control unit. The reward for each of the multiple drive shafts is adjusted relatively according to the required accuracy for each of the multiple drive shafts. A processing system characterized by the following: (Item 5) The reward evaluated in the reinforcement learning described above is represented by the weighted sum of the rewards for each of the multiple drive axes, The learning device determines each weight in the weighted sum according to the required accuracy for each of the multiple drive axes. The processing system described in item 4, characterized by the features described herein. (Item 6) The processing system according to item 5, characterized in that the learning device acquires the required accuracy for each of the plurality of drive axes and determines the weight for the acquired required accuracy based on the correspondence between the previously obtained required accuracy and the weight. (Item 7) The control unit is configured to generate command values ​​for the drive unit based on the control deviation. The control unit, A first compensator that generates a first command value based on the control deviation, A second compensator that generates a second command value based on the control deviation, Includes an adder that adds the first command value and the second command value to obtain the command value, The neural network is included in the second compensator. A processing system according to any one of items 4 to 6, characterized in that it is a processing system according to any one of items 4 to 6. (Item 8) The processing system according to any one of items 4 to 6, characterized in that the processing device is a positioning device that moves the movable part as the object on planes parallel to a first direction and a second direction which are orthogonal to each other. (Item 9) The positioning device includes a single guide that constrains the position of the movable part in the second direction, The aforementioned movable part is A first movable body that can move in a first direction while being guided by the aforementioned guide, A second movable body having a first end and a second end, the first end of which is connected to the first movable body via a rotating bearing, and which moves on the surface, Includes a third movable body that is guided by the second movable body and is movable within a predetermined range between the first end and the second end, The aforementioned drive unit is A first drive unit that drives the first end of the second movable body in the first direction, The second drive unit includes a second drive unit that drives the second end of the second movable body in the first direction, The processing system described in item 8, characterized by the features described above. (Item 10) The processing system according to any one of items 4 to 6, characterized in that the processing device is a vibration isolation device that reduces vibrations transmitted to an object. (Item 11) The vibration isolation device includes a vibration isolation platform on which the object to be mounted, and an accelerometer positioned on the vibration isolation platform. The drive unit is configured to drive the vibration isolation table, The control unit is configured to generate command values ​​for the drive unit based on the control deviation. The control unit, A first compensator that generates a first command value based on the speed deviation, A second compensator that generates a second command value based on the acceleration of the vibration isolation table measured by the accelerometer, Includes an adder that adds the first command value and the second command value to obtain the command value, The neural network is included in the second compensator. The processing system described in item 10, characterized by the features described herein. (Item 12) The processing system according to any one of items 4 to 6, characterized in that the processing apparatus is a lithography apparatus that performs a process of transferring the pattern of a master plate onto a substrate which is the object. (Item 13) The lithography apparatus includes a stage section on which the substrate, which is the object to be worked on, an alignment detector for measuring the positional misalignment between the master plate and the substrate, and a position measuring unit for measuring the position of the stage section. The drive unit is configured to drive the stage unit, The control unit is configured to generate command values ​​for the drive unit based on the control deviation. The control unit, A first compensator generates a first command value for the drive unit based on the position deviation, which is the difference between the measured value obtained by the position measuring unit and the target value. A second compensator that generates a second command value based on the positional deviation measured by the alignment detector, Includes an adder that adds the first command value and the second command value to obtain the command value, The neural network is included in the second compensator. The processing system described in item 12, characterized by the features described herein. (Item 14) A management method for managing a processing device that includes a drive unit that drives an object with respect to multiple drive axes, and a control unit that controls the drive unit using a neural network whose parameters for outputting an amount of manipulation applied to the object are determined by reinforcement learning, The neural network has a learning step of determining the parameters of the neural network by reinforcement learning, which includes evaluating the reward obtained from the control result of the object by the control unit, The aforementioned learning process is, A step of obtaining the required accuracy for each of the aforementioned multiple drive shafts, A step of relatively adjusting the reward for each of the plurality of drive shafts according to the acquired required accuracy for each of the plurality of drive shafts, A management method characterized by including (Item 15) A transfer step in which the pattern of the master plate is transferred to a substrate using the lithography apparatus in the processing system described in item 13, The process includes processing the substrate after the transfer process, A method for manufacturing an article, characterized by obtaining an article from the substrate that has undergone the processing steps described above. (Item 16) A management device for managing a processing device that includes a control unit for controlling an object using a neural network, The learning unit performs reinforcement learning of the neural network by evaluating the reward obtained from the control result of the object controlled by the control unit, The learning unit adjusts the neural network to satisfy the user's request. A management device characterized by the following features.

[0071] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0072] 13: Stage unit, 14: Driver, 18: Position measurement unit, 19: Linear motor, 200: Control system, 201: Stage control unit, 211: First control system, 212: Second control system

Claims

1. A management device that manages a processing device including a drive unit that drives an object about a plurality of drive axes, and a control unit that controls the drive unit using a neural network in which parameters for outputting an operation amount for the object are determined by reinforcement learning, a learning unit that determines parameters of the neural network by reinforcement learning; The learning unit performing the reinforcement learning by evaluating a reward obtained from a control result of the object by the control unit; Relatively adjusting the compensation for each of the plurality of drive axes according to the required accuracy for each of the plurality of drive axes. A management device characterized by:

2. the reward evaluated in the reinforcement learning is represented by a weighted sum of rewards for each of the plurality of drive axes; the learning unit determines each weight in the weighted sum according to a required accuracy for each of the plurality of drive axes. The management device according to claim 1 .

3. The management device according to claim 2, characterized in that the learning unit acquires the required accuracy for each of the plurality of drive axes and determines a weight for the acquired required accuracy based on a correspondence relationship between the required accuracy and a weight obtained in advance.

4. a processing device including a drive unit that drives an object about a plurality of drive axes, and a control unit that controls the drive unit using a neural network in which parameters for outputting an operation amount for the object are determined by reinforcement learning; a learning device that determines parameters of the neural network by reinforcement learning, The learning device performing the reinforcement learning by evaluating a reward obtained from a control result of the object by the control unit; Relatively adjusting the compensation for each of the plurality of drive axes according to the required accuracy for each of the plurality of drive axes. A processing system comprising:

5. the reward evaluated in the reinforcement learning is represented by a weighted sum of rewards for each of the plurality of drive axes; the learning device determines each weight in the weighted sum in accordance with a required accuracy for each of the plurality of drive axes.

5. The processing system according to claim 4.

6. The processing system described in claim 5, characterized in that the learning device acquires the required accuracy for each of the multiple drive axes and determines a weight for the acquired required accuracy based on a correspondence between the required accuracy and weight obtained in advance.

7. the control unit is configured to generate a command value for the drive unit based on a control deviation, The control unit a first compensator that generates a first command value based on the control deviation; a second compensator that generates a second command value based on the control deviation; an adder that adds the first command value and the second command value to obtain the command value, the neural network is included in the second compensator; 7. The processing system according to claim 4, wherein the processing system is a processing system for processing a semiconductor device.

8. 7. The processing system according to claim 4, wherein the processing device is a positioning device that moves a movable part as the object on a plane parallel to a first direction and a second direction that are perpendicular to each other.

9. the positioning device includes a single guide as a guide that constrains the position of the movable part in the second direction; The movable part is a first movable body that is movable in a first direction while being guided by the guide; a second movable body having a first end and a second end, the first end being connected to the first movable body via a rotary bearing, and adapted to move on the surface; a third movable body that is movable within a predetermined range between the first end and the second end while being guided by the second movable body, The drive unit is a first drive unit that drives the first end of the second movable body in the first direction; a second drive unit that drives the second end of the second movable body in the first direction, 9. The processing system according to claim 8.

10. 7. The processing system according to claim 4, wherein the processing device is a vibration isolation device that reduces vibrations transmitted to the object.

11. the vibration isolation device includes a vibration isolation table on which the object is mounted, and an accelerometer disposed on the vibration isolation table; the drive unit is configured to drive the vibration isolation table, the control unit is configured to generate a command value for the drive unit based on a control deviation, The control unit a first compensator that generates a first command value based on the speed deviation; a second compensator that generates a second command value based on the acceleration of the vibration isolation table measured by the accelerometer; an adder that adds the first command value and the second command value to obtain the command value, the neural network is included in the second compensator; The processing system of claim 10 .

12. 7. The processing system according to claim 4, wherein the processing device is a lithography device that performs a process of transferring a pattern of an original onto a substrate that is the object.

13. the lithography apparatus includes a stage unit on which the substrate, which is the object, is mounted, an alignment detector that measures a positional deviation between the original and the substrate, and a position measurement unit that measures a position of the stage unit; the drive unit is configured to drive the stage unit, the control unit is configured to generate a command value for the drive unit based on a control deviation, The control unit a first compensator that generates a first command value for the drive unit based on a position deviation that is a difference between a measurement value obtained by the position measurement unit and a target value; a second compensator that generates a second command value based on the positional deviation measured by the alignment detector; an adder that adds the first command value and the second command value to obtain the command value, the neural network is included in the second compensator; 13. The processing system of claim 12.

14. A management method for managing a processing device including a drive unit that drives an object about a plurality of drive axes, and a control unit that controls the drive unit using a neural network in which parameters for outputting an operation amount for the object are determined by reinforcement learning, the method comprising: a learning step of determining parameters of the neural network by reinforcement learning including evaluating a reward obtained from a control result of the object by the control unit, The learning step includes: obtaining a required accuracy for each of the plurality of drive axes; adjusting a compensation for each of the plurality of drive axes relatively in accordance with the acquired required accuracy for each of the plurality of drive axes; A management method comprising:

15. a transfer step of transferring a pattern of an original onto a substrate using the lithography apparatus in the processing system according to claim 13; a processing step of processing the substrate that has undergone the transfer step, A method for manufacturing an article, characterized in that an article is obtained from the substrate that has been subjected to the processing step.

16. A management device that manages a processing device including a control unit that controls an object using a neural network, a learning unit that performs reinforcement learning of the neural network by evaluating a reward obtained from a control result of the object by the control unit, the learning unit adjusts the neural network in response to a request from a user so as to satisfy the request. A management device characterized by:

Images