Simulation system
The simulation system addresses the challenge of coordinating consecutive operations by rendering model workers' actions in a virtual space, improving training efficiency through individualized task adaptation.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2023-06-12
- Publication Date
- 2026-06-23
Smart Images
Figure 0007878169000001 
Figure 0007878169000002 
Figure 0007878169000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to a technique for simulating a work line in a virtual space.
Background Art
[0002] Patent Document 1 discloses a learning support system. The learning support system overlays and displays a model video of an instructor, which serves as a model of the learner's operation, on the learner's visual video. The basic information of the learner includes the learner's age, dominant hand, eyesight, grip strength, etc. The learning support system changes the display content of the model video according to the basic information of the learner.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In a work line, there can be cooperation between consecutive operations. One object of the present disclosure is to provide a technique that can effectively learn the cooperation between consecutive operations.
Means for Solving the Problems
[0005] One aspect of the present disclosure relates to a simulation system that simulates a work line in a virtual space. The work line includes a first operation performed by a first target worker and a second operation performed by a second target worker on the result of the first operation. The first model operation includes an operation that serves as a model when the first target worker passes the result of the first operation to the second target worker. The second model operation includes an operation that serves as a model when the second target worker receives the result of the first operation from the first target worker. The simulation system comprises one or more processors. One or more processors determine a combination of first-model operation and second-model operation. When the first target worker performs an action related to the first task in the virtual space, one or more processors render the first model worker performing the first model action in the virtual space. When a second target worker performs actions related to the second task in the virtual space, one or more processors render the second model worker performing the second model actions in the virtual space. [Effects of the Invention]
[0006] According to this disclosure, a first model worker performing a first model action and a second model worker performing a second model action are rendered in a virtual space. The first model action includes exemplary actions when the first target worker hands over the result of the first action to the second target worker. The second model action includes exemplary actions when the second target worker receives the result of the first action from the first target worker. This enables the first and second target workers to effectively learn efficient coordination between the first and second actions. [Brief explanation of the drawing]
[0007] [Figure 1] This is a conceptual diagram illustrating the overview of the simulation system according to the embodiment. [Figure 2] This is a block diagram showing an example configuration of a simulation system according to an embodiment. [Figure 3] This is a conceptual diagram illustrating the coordination between consecutive tasks on a production line. [Figure 4] This is a conceptual diagram illustrating the simulation of the work line according to the embodiment. [Figure 5] This is a conceptual diagram illustrating a modified example of the simulation of the work line according to the embodiment. [Modes for carrying out the invention]
[0008] 1. Overview of the Simulation System Let's consider a specific task performed by an operator in a real space. For example, the task is performed in a real factory. The task is, for instance, the assembly of parts. According to this embodiment, simulation is used to efficiently train (learn) an operator to perform the task.
[0009] Figure 1 is a conceptual diagram illustrating the overview of the simulation system 100 according to this embodiment. The simulation system 100 performs a simulation of a predetermined task performed by an operator in a virtual space. To this end, the simulation system 100 virtually reproduces the work environment in which the operator performs the predetermined task within the virtual space. For example, the simulation system 100 reproduces (renders) the work environment within the virtual space based on DigitalTwin technology. For example, the simulation system 100 reproduces a virtual factory (DigitalTwin factory) that simulates a real factory within the virtual space.
[0010] The trainee worker (hereinafter referred to as "Target Worker TW") can experience a predetermined task in a virtual space by using an experience device. For example, the experience device is a wearable device such as a head-mounted display (HMD). The head-mounted display displays the virtual space reproduced by the simulation system 100. The movements of Target Worker TW in the real space (e.g., hand movements) are recognized by motion capture technology. For example, the movements of Target Worker TW are recognized by photographing them with a camera. As another example, inertial sensors (gyro sensors, accelerometers, etc.) may be attached to the body of Target Worker TW, and the movements of Target Worker TW may be recognized based on the detection results from these inertial sensors. The simulation system 100 superimposes and renders the recognized movements of Target Worker TW in the virtual space. For example, the simulation system 100 superimposes and renders the recognized hand movements of Target Worker TW in the virtual space. As a result, Target Worker TW can get the feeling that they are performing a predetermined task in the virtual space. In other words, Target Worker TW can experience a predetermined task in the virtual space.
[0011] Furthermore, the simulation system 100 may superimpose and render a virtual model worker MW performing model actions in the virtual space. Model actions are exemplary actions that the target worker TW should perform when performing a predetermined task. In the virtual space, the virtual model worker MW performs model actions. The model worker MW can also be called a "ghost worker". In the virtual space, the target worker TW performs the predetermined task by imitating the model actions performed by the model worker MW. This enables efficient and effective training.
[0012] Figure 2 is a block diagram showing an example configuration of the simulation system 100 according to this embodiment. The simulation system 100 includes a sensor 110, an input device 120, an output device 130, and an experience device 140.
[0013] Sensor 110 is arranged in the real space and detects various information. For example, Sensor 110 detects the movements of an operator in the real space. Examples of the sensor 110 for detecting the movements of an operator include a camera, an infrared sensor, an inertial sensor, etc. For example, by photographing the operator with a camera, the movements of the operator can be detected. As another example, an inertial sensor (gyro sensor, acceleration sensor, etc.) may be attached to the body of the operator, and the movements of the operator may be detected by that inertial sensor.
[0014] As another example, Sensor 110 may detect the physical characteristics of an operator in the real space. Examples of physical characteristics include the dominant hand, hand size, grip strength, height, muscle mass, eyesight, etc. For example, based on the movements of the operator photographed by a camera, the dominant hand of the operator is detected. As another example, based on an image of the operator photographed by a camera, the hand size, height, etc. are detected. In addition, Sensor 110 may include a height meter, a weight scale, a body composition meter, a grip strength meter, an eyesight meter, etc.
[0015] As yet another example, Sensor 110 may include a biosensor that detects the biological information of an operator in the real space. Examples of biological information include body temperature, heart rate, fatigue level, stress level, etc.
[0016] Examples of the input device 120 include a touch panel, a keyboard, a mouse, a microphone, etc. Examples of the output device 130 include a display, a touch panel, a speaker, etc.
[0017] The experience device 140 is used for the target operator TW to be trained to experience the working environment and a predetermined operation in the virtual space. For example, the experience device 140 is a wearable device such as a head-mounted display (HMD). The head-mounted display displays the virtual space reproduced by the simulation system 100.
[0018] The simulation system 100 further includes one or more processors 150 (hereinafter simply referred to as "processor 150") and one or more storage devices 160 (hereinafter simply referred to as "storage device 160"). The processor 150 executes various processes. Examples of the processor 150 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like. The storage device 160 stores various information. Examples of the storage device 160 include an HDD (Hard Disk Drive), an SSD (Solid State Drive), a volatile memory, a non-volatile memory, and the like.
[0019] The simulation program 170 is a computer program for performing simulation in the above-described virtual space and is executed by the processor 150. Various processes by the simulation system 100 may be realized by the cooperation of the processor 150 and the storage device 160 that execute the simulation program 170. The simulation program 170 is stored in the storage device 160. The simulation program 170 may be recorded on a computer-readable recording medium.
[0020] The work environment information 180 is information on a work environment (e.g., a real factory) reproduced in the virtual space. For example, the work environment information 180 indicates the three-dimensional arrangement of structures (lines, machines, walls, columns, etc.) in the work environment. For example, the three-dimensional arrangement of the structures is represented by CAD data. The work environment information 180 is stored in the storage device 160.
[0021] A database 200 is further stored in the storage device 160. The database 200 includes an operator characteristic database 210, a work performance database 220, and a model operation database 230.
[0022] The worker characteristics database 210 is a database that shows the physical characteristics of each worker. Examples of physical characteristics include dominant hand, hand size, grip strength, height, muscle mass, eyesight, etc. For example, each worker's physical characteristics are detected by the sensor 110 described above. Alternatively, each worker may input their own physical characteristics using the input device 120. Physical characteristics expressed numerically, such as height, may be grouped into predetermined intervals.
[0023] The work performance database 220 is a database that shows the performance of a given task performed by various workers. For example, the work performance database 220 includes video footage of a worker during a given task. The video footage of the worker is captured by a camera included in the sensor 110 described above. As another example, the work performance database 220 may show the details of the actions performed by the worker during the given task. As yet another example, the work performance database 220 may show the time it took for the worker to complete the given task. Such a work performance database 220 is generated based on the worker's actions detected by the sensor 110 described above.
[0024] The model motion database 230 is a database of model motions that serve as a guide for the target worker TW when performing a predetermined task. Model motions for predetermined tasks may be created in advance based on the work performance database 220.
[0025] The processor 150 recreates the work environment in which an operator performs a predetermined task within the virtual space, based on the work environment information 180. For example, the processor 150 recreates (renders) the work environment within the virtual space based on DigitalTwin technology.
[0026] Furthermore, the processor 150 uses the sensor 110 to recognize the movements of the target worker TW being trained. The movements of the target worker TW in real space are recognized using motion capture technology. For example, the movements of the target worker TW are recognized by photographing them with a camera. Alternatively, an inertial sensor may be attached to the body of the target worker TW, and the movements of the target worker TW may be recognized based on the detection results from the inertial sensor. The processor 150 superimposes and renders the recognized movements of the target worker TW in the virtual space. For example, the processor 150 superimposes and renders the recognized hand movements of the target worker TW in the virtual space. The target worker TW can get the feeling that they are performing a predetermined task in the virtual space via the experience device 140. In other words, the target worker TW can experience a predetermined task in the virtual space via the experience device 140.
[0027] Furthermore, the processor 150 determines, based on the model motion database 230, a model motion that the target worker TW should emulate when performing a predetermined task. The processor 150 then superimposes and renders a virtual model worker MW performing the model motion into the virtual space. In the virtual space, the target worker TW performs the predetermined task by mimicking the model motion performed by the model worker MW. This enables efficient and effective training.
[0028] 2. Simulation of the production line 2-1. Overview Next, let's consider the coordination between consecutive operations on a work line. Figure 3 is a conceptual diagram illustrating the coordination between consecutive operations on a work line. The work line includes an upstream first operation and a downstream second operation. The first worker W1 performs the upstream first operation. For example, the first worker W1 assembles a certain part. The first worker W1 hands over the result of the first operation (e.g., the assembled part) to the downstream second worker W2. The first worker W1 may hand over the result directly or indirectly via a conveyor belt or the like. The downstream second worker W2 receives the result of the first operation from the first worker W1. The second worker W2 may receive the result directly or indirectly via a conveyor belt or the like. Then, the second worker W2 performs the downstream second operation on the received result of the first operation.
[0029] It is desirable to effectively learn the coordination between the first and second tasks. In the following explanation, the first worker W1 being trained will be referred to as "First Target Worker TW1," and the second worker W2 being trained will be referred to as "Second Target Worker TW2."
[0030] Figure 4 is a conceptual diagram illustrating the simulation of the work line according to this embodiment. The simulation system 100 performs a simulation of the work line in a virtual space.
[0031] "First Model Action MO1" is a model action related to the first task. The first model action includes exemplary actions when the first target worker TW1 hands over the result of the first task to the second target worker TW2. Furthermore, the first model action may also include exemplary actions when the first target worker TW1 performs the first task.
[0032] On the other hand, "Second Model Action MO2" is a model action related to the second task. The second model action includes exemplary actions when the second target worker TW2 receives the result of the first task from the first target worker TW1. Furthermore, the second model action may also include exemplary actions when the second target worker TW2 performs the second task.
[0033] The simulation system 100 determines a combination of a first model motion MO1 and a second model motion MO2. Typically, the simulation system 100 determines a combination of the first model motion MO1 and the second model motion MO2 that satisfies predetermined conditions. Various examples of methods for determining the first model motion MO1 and the second model motion MO2 are described later. The combination of the first model motion MO1 and the second model motion MO2 for the combination of the first and second operations is registered in the model motion database 230.
[0034] The first model operator MW1 is a model operator MW that performs the first model action MO1. The second model operator MW2 is a model operator MW that performs the second model action MO2. The simulation system 100 renders the first model operator MW1 and the second model operator MW2 in the virtual space based on the model action database 230.
[0035] More specifically, when the first target worker TW1 performs an action related to the first task in the virtual space, the simulation system 100 renders a first model worker MW1 performing the first model action MO1 in the virtual space. The first target worker TW1 performs the action related to the first task by mimicking the first model action MO1 performed by the first model worker MW1. Similarly, when the second target worker TW2 performs an action related to the second task in the virtual space, the simulation system 100 renders a second model worker MW2 performing the second model action MO2 in the virtual space. The second target worker TW2 performs the action related to the second task by mimicking the second model action MO2 performed by the second model worker MW2. This enables the first target worker TW1 and the second target worker TW2 to effectively learn efficient coordination between the first and second tasks.
[0036] 2-2. Examples of methods for determining the first model behavior and the second model behavior. The following describes various examples of methods for determining the first model operation MO1 and the second model operation MO2. Typically, the combination of the first model operation MO1 and the second model operation MO2 is determined to satisfy "predetermined conditions".
[0037] 2-2-1. Example 1 In the first example, the simulation system 100 acquires characteristic information indicating the first characteristics of the first target worker TW1. The first characteristics of the first target worker TW1 include the physical characteristics of the first target worker TW1 (e.g., dominant hand, height, etc.). The physical characteristics of the first target worker TW1 are obtained from the worker characteristics database 210. Based on the physical characteristics of the first target worker TW1, the simulation system 100 determines a combination of the first model motion MO1 and the second model motion MO2.
[0038] For example, taking into account the dominant hand of the first target worker TW1, the first model action MO1 is determined so that the first target worker TW1 can easily hand over the result of the first operation to the second target worker TW2. Once the first model action MO1 is determined, the second model action MO2 is determined so that it can receive the result handed over by the first model action MO1.
[0039] As another example, taking into account the height of the first target worker TW1, a target position is determined where the first target worker TW1 can easily place the result of the first task. The first model action MO1 is determined to place the result of the first task at the target position. The second model action MO2 is determined to pick up the result of the first task from the target position.
[0040] The first characteristic of the first target worker TW1 may include the skill level of the first target worker TW1. The skill level of the first target worker TW1 is estimated based on the work performance database 220. For example, the faster the actions related to the first task, the higher the estimated skill level. If the skill level of the first target worker TW1 is low, the first model action MO1 is determined to be as simple as possible. Once the first model action MO1 is determined, the second model action MO2 is determined so that it can receive the result passed by the first model action MO1.
[0041] In general, the predetermined condition in the first example is that "the load on the first target worker TW1 when handing over the result of the first task is below a threshold." The load when handing over the result of the first task is calculated based on the degree of body twisting, movement time, etc. The load for various movement patterns can be obtained from the work performance database 220 or through simulation. Based on the first characteristics of the first target worker TW1, the simulation system 100 determines a combination of the first model movement MO1 and the second model movement MO2 such that the load on the first target worker TW1 when handing over the result of the first task is below a threshold. Because the first characteristics of the first target worker TW1 are taken into consideration, it becomes possible to learn more efficient coordination.
[0042] 2-2-2. Second Example In the second example, the simulation system 100 acquires characteristic information indicating the second characteristics of the second target worker TW2. The second characteristics of the second target worker TW2 include the physical characteristics of the second target worker TW2 (e.g., dominant hand, height, etc.). The physical characteristics of the second target worker TW2 are obtained from the worker characteristics database 210. Based on the physical characteristics of the second target worker TW2, the simulation system 100 determines a combination of the first model motion MO1 and the second model motion MO2.
[0043] For example, taking into account the dominant hand of the second target worker TW2, one of the first model action MO1 and the second model action MO2 is determined so that the second target worker TW2 can easily receive the result of the first action. Once one of the first model action MO1 and the second model action MO2 is determined, the other of the first model action MO1 and the second model action MO2 is determined.
[0044] As another example, the target position is determined to allow the second target worker TW2 to easily pick up the result of the first task, taking into account the height of the second target worker TW2. The first model action MO1 is determined to place the result of the first task at the target position. The second model action MO2 is determined to pick up the result of the first task from the target position.
[0045] The second characteristic of the second target worker TW2 may include the skill level of the second target worker TW2. The skill level of the second target worker TW2 is estimated based on the work performance database 220. For example, the faster the actions related to the second task, the higher the estimated skill level. If the skill level of the second target worker TW2 is low, the second model action MO2 is determined to be as simple as possible. Once the second model action MO2 is determined, the first model action MO1 is determined accordingly.
[0046] In general, the condition in the second example is that "the load on the second target worker TW2 when receiving the result of the first task is below a threshold." The load when receiving the result of the first task is calculated based on the degree of body twisting, movement time, etc. The load for various movement patterns is obtained from the work performance database 220 or through simulation. Based on the second characteristics of the second target worker TW2, the simulation system 100 determines the combination of the first model movement MO1 and the second model movement MO2 such that the load on the second target worker TW2 when receiving the result of the first task is below a threshold. Because the second characteristics of the second target worker TW2 are taken into consideration, it becomes possible to learn more efficient coordination.
[0047] 2-2-3. Third Example Combining the first and second examples described above is also possible. Since both the first characteristic of the first target worker TW1 and the second characteristic of the second target worker TW2 are taken into consideration, it becomes possible to learn even more efficient collaboration.
[0048] 2-2-4. The fourth example In the fourth example, the predetermined condition is that "the time it takes for the results of the first operation to be handed over from the first target worker TW1 to the second target worker TW2 is less than a predetermined time." The combination of the first model operation MO1 and the second model operation MO2 that satisfies this predetermined condition may be searched from the work performance database 220 or explored through simulation. The fourth example also makes it possible to learn efficient coordination.
[0049] 2-2-5. The fifth example In the fifth example, at least one of the first model action MO1 and the second model action MO2 is set to the actual action of the veteran worker. Both the first model action MO1 and the second model action MO2 may be set to the actual action of the veteran worker. The actual action of the veteran worker is obtained from the work performance database 220. According to the fifth example, it becomes possible to effectively learn efficient coordination by veteran workers.
[0050] 2-3. Effects As described above, according to this embodiment, a first model worker MW1 performing a first model action MO1 and a second model worker MW2 performing a second model action MO2 are rendered in the virtual space. The first model action MO1 includes exemplary actions when a first target worker TW1 hands over the result of the first task to a second target worker TW2. The second model action MO2 includes exemplary actions when a second target worker TW2 receives the result of the first task from a first target worker TW1. This enables the first target worker TW1 and the second target worker TW2 to effectively learn efficient coordination between the first and second tasks.
[0051] 3. Variant Figure 5 is a conceptual diagram illustrating a modified example. As described above, the first model operation MO1 and the second model operation MO2 are pre-set. The first target worker TW1 performs actions related to the first task by mimicking the first model operation MO1 performed by the first model worker MW1. However, the actual actions of the first target worker TW1 do not necessarily match the first model operation MO1. The actual actions of the first target worker TW1 may deviate from the first model operation MO1. In that case, the downstream second target worker TW2 will not be able to properly receive the result of the first task if the second model operation remains pre-set.
[0052] Therefore, in this modified example, the simulation system 100 dynamically corrects the second model operation MO2 according to the operation results of the first target worker TW1.
[0053] More specifically, the first action WO1 is the actual action taken by the first target worker TW1 in the virtual space when handing over the result of the first operation to the second target worker TW2. The simulation system 100 acquires information on the first action WO1 of the first target worker TW1 using the sensor 110. Subsequently, the simulation system 100 compares the first action WO1 with the first model action MO1 and calculates the degree of deviation of the first action WO1 from the first model action MO1. If the degree of deviation exceeds a threshold, the simulation system 100 corrects the second model action MO2 so that it can receive the result of the first operation handed over by the first action WO1. In other words, the simulation system 100 corrects the second model action MO2 to follow the first action WO1.
[0054] The corrected second model action MO2' is the corrected result of the second model action MO2. The simulation system 100 renders the second model worker MW2 performing the corrected second model action MO2' in a virtual space. The second target worker TW2 performs actions related to the second task by mimicking the corrected second model action MO2' performed by the second model worker MW2.
[0055] As explained above, according to the modified version, even if the first action of the first target worker TW1 on the upstream side deviates from the first model action MO1, it becomes possible to continue training the second target worker TW2 on the downstream side. [Explanation of symbols]
[0056] 100…Simulation system, 110…Sensor, 120…Input device, 130…Output device, 140…Experience device, 150…Processor, 160…Storage device, 170…Simulation program, 180…Work environment information, 200…Database, 210…Worker characteristics database, 220…Work performance database, 230…Model motion database
Claims
1. A simulation system that simulates a work line in a virtual space, The aforementioned work line includes a first task performed by a first target worker and a second task performed by a second target worker on the result of the first task, The first model operation includes an exemplary operation in which the first target worker hands over the result of the first operation to the second target worker. The second model operation includes a model operation in which the second target worker receives the result of the first operation from the first target worker, The simulation system comprises one or more processors, The one or more processors described above are: Characteristic information is obtained that shows at least one of the first characteristics of the first target worker and the second characteristics of the second target worker. Based on at least one of the first and second characteristics, the combination of the first model operation and the second model operation is determined. When the first target worker performs an action related to the first task in the virtual space, the first model worker performing the first model action is rendered in the virtual space. When the second target worker performs an action related to the second task in the virtual space, the second model worker performing the second model action is rendered in the virtual space. It is configured in such a way Simulation system.
2. A simulation system according to claim 1, The characteristic information indicates at least the first characteristics of the first target worker, The one or more processors determine the combination of the first model operation and the second model operation based on the first characteristics of the first target worker, such that the workload on the first target worker when delivering the result of the first operation is below a threshold. Simulation system.
3. A simulation system according to claim 1, The characteristic information indicates at least the second characteristics of the second target worker, The one or more processors determine the combination of the first model operation and the second model operation based on the second characteristics of the second target worker such that the load on the second target worker when receiving the result of the first operation is below a threshold. Simulation system.
4. A simulation system according to any one of claims 1 to 3, The one or more processors described above are: In the virtual space, information is obtained about the first action performed when the first target worker hands over the result of the first task to the second target worker. If the degree of deviation of the first operation from the first model operation exceeds a threshold, the second model operation is corrected to receive the result of the first operation passed by the first operation. Simulation system.