Information processing system and information processing method

Abstract

To accurately infer a higher-level operation from a recognition result of a series of atomic operations by a worker.SOLUTION: A production line analysis device 1 comprises: a data acquisition section 11 that acquires operation sequence data 31 representing a series of operations of a worker in a production line having a plurality of work stations; a noise detection section 12 that detects noise from the operation sequence data 31 on the basis of operation constitution information 32 representing a relationship between atomic operations by the worker and a higher-level operation comprising a combination of the plurality of atomic operations; a noise correction section 13 that corrects the operation sequence data 31 so as to reduce the noise; and an operation inference section 14 that infers, on the basis of the operation sequence data 34 after the correction by the noise correction section 13, the higher-level operation corresponding to a work item executed by the worker at each of the work stations of the production line.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing system and an information processing method used for analyzing work in each process of a production line. [Background technology]

[0002] Generally, production lines for manufacturing various products such as automobiles are composed of multiple work stations. For example, if the production volume per hour is 60 units, the production line will be composed of 60 work stations, each with a one-minute work time, and a product will be completed every minute on this production line. The work time for each work station can also be set to 5 or 10 minutes. At each work station, workers such as humans or robots perform specific work items related to the production of the product.

[0003] In a production line, if work is not properly allocated to each work station, variations in work time will occur between the stations. For example, if the average work time for all work stations is 5 minutes, one work station may take 7-8 minutes, while another may take 3-4 minutes. In this case, the former work station will be overloaded, causing in-process products to back up, while the latter work station will have to wait for work. This situation is called an imbalance in the production line, and it is best to avoid it as much as possible because it reduces productivity.

[0004] To improve product productivity, it is important to properly balance the work time at each work station. To achieve this, it is necessary to accurately know the work time for each work item at each work station. In recent years, work analysis methods have been devised that acquire worker motion data at each work station, and then use machine learning or other techniques to recognize the sequence of worker motions at each work station based on this motion data. This identifies the work items corresponding to the recognized motions and measures their work time. However, these work analysis methods have difficulty accurately recognizing complex, long-duration motions due to limitations in computer processing power and machine learning characteristics. Furthermore, different production lines and workers may perform different motions for the same work item. For these reasons, it is difficult to accurately recognize a sequence of worker motions from their motion data.

[0005] Therefore, a method has been proposed in which a series of actions performed by workers at each work station is broken down into short-duration actions (hereinafter referred to as "atomic actions") and action recognition is performed in atomic action units. This method makes it possible to realize action recognition using a computer. Furthermore, because individual atomic actions have many commonalities even across different production lines and manufacturers, even when action recognition in task units is difficult, correct action recognition in atomic action units is possible.

[0006] When recognizing actions in atomic operations, it is necessary to identify the start and end actions of each work item from the various recognized atomic operations, thereby setting boundaries for each work item in the series of atomic operations, and converting the sequence of atomic operations separated by these boundaries into higher-level actions corresponding to the work items at each work station. In this regard, for example, Patent Document 1 and Non-Patent Document 1 are known. Patent Document 1 describes a method for recognizing atomic operations from image data using a long short-term memory (LSTM) and a convolutional neural network (CNN), and then applying a mathematical model called a finite state automata (FSA) to the recognition results to identify higher-level actions consisting of a combination of multiple atomic operations. Non-Patent Document 1 also describes a method for inferring higher-level actions from a series of lower-level actions corresponding to atomic operations using a probabilistic model called a conditional random field (CRF). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent No. 11,054,811 [Non-patent literature]

[0008] [Non-Patent Document 1] Bordel, B., Alcarria, R., & Robles, T., "Recognizing human activities in Industry 4.0 scenarios through an analysis-modeling-recognition algorithm and context labels", Integrated Computer-Aided Engineering, 29 (2022), p. 83-103 Summary of the Invention [Problem to be solved by the invention]

[0009] The method of Patent Document 1 uses a rule-based model, FSA, to identify higher-level actions, and therefore is insufficient to correctly identify higher-level actions when the atomic action recognition results contain noise or deviate from predefined rules. In other words, the recognition accuracy of atomic actions is not always 100%, and workers may make operational errors during work. It is difficult for FSA to accurately define higher-level action boundaries for a series of atomic actions obtained under these circumstances. On the other hand, the method of Non-Patent Document 1 uses a probabilistic model to identify higher-level actions, and therefore is difficult to estimate higher-level actions with the desired accuracy when, for example, each work station on a production line has its own work manual or when there are common atomic actions across multiple higher-level actions.

[0010] The present invention has been made based on the above background, and aims to accurately estimate higher-level actions from the recognition results of a series of atomic actions performed by a worker. [Means for solving the problem]

[0011] The information processing system according to the present invention comprises a data acquisition unit that acquires action sequence data representing a series of actions performed by a worker on a production line having a plurality of work stations; a noise detection unit that detects noise from the action sequence data based on action configuration information representing the relationship between atomic actions performed by the worker and higher-level actions consisting of combinations of a plurality of the atomic actions; a noise correction unit that corrects the action sequence data so as to reduce the noise; and an action estimation unit that estimates the higher-level actions corresponding to the work items performed by the worker at each work station on the production line based on the action sequence data after correction by the noise correction unit. An information processing method according to the present invention involves using a computer to acquire action sequence data representing a series of actions performed by a worker on a production line having a plurality of work stations, detect noise from the action sequence data based on action configuration information representing the relationship between atomic actions performed by the worker and higher-level actions consisting of combinations of a plurality of the atomic actions, correct the action sequence data so as to reduce the noise, and estimate the higher-level actions corresponding to the work performed by the worker at each work station on the production line based on the corrected action sequence data. [Effects of the Invention]

[0012] According to the present invention, higher-level actions can be accurately estimated from the recognition results of a series of atomic actions performed by a worker. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram showing an outline of a production line to which the present invention is applied; [Figure 2] 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing the functional configuration of a production line analysis device according to an embodiment of the present invention. [Figure 4] 10 is a flowchart showing the flow of processing performed by the production line analysis device. [Figure 5] 10 is a flowchart showing the flow of noise detection processing. [Figure 6] 10 is a flowchart showing the flow of noise correction processing. [Figure 7] 10 is a flowchart showing the flow of a higher-level action estimation process. [Figure 8] 10 is a table showing an example of sensor characteristic information. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] Fig. 1 is a diagram showing an overview of a production line to which the present invention is applied. When applied to a production line such as that shown in Fig. 1, the present invention analyzes the work performed on the production line and, based on the results of the analysis, makes suggestions to users such as production line managers to improve productivity.

[0016] Figure 1(a) shows the state of the production line before adjustment. This production line is made up of multiple work stations, including work stations A, B, and C. At each work station, workers perform predetermined work items to produce products such as automobiles. At each work station, the predetermined work items are performed by either humans or robots, or by humans and robots working together. Hereinafter, all workers at each work station, regardless of whether they are humans or robots, will be collectively referred to as "workers."

[0017] In the production line shown in Figure 1(a), for example, at work station A, tasks A1, A2, A3, and A4 are performed in sequence, with the total work time being 60 minutes. When tasks A1 to A4 are completed at work station A, the in-process product is transported to the next work station B, where work begins.

[0018] At work station B, work items B1 and B2 are performed in sequence, with the total work time being 50 minutes. When work station B completes work B1 and B2, the product in progress is moved to the next work station, C, and work begins at work station C. At work station C, work items C1, C2, and C3 are performed in sequence, with the total work time being 60 minutes. Similarly, one or more specified work items are performed in sequence at each work station, and after they are completed, the work item at the next work station begins.

[0019] As shown above, in the production line before the adjustment, the working time for work stations A and C was 60 minutes, while the working time for work station B was 50 minutes. As a result, at work station B, after tasks B1 and B2 were completed, there was a waiting time of about 10 minutes until the next in-process product was delivered from work station A. Meanwhile, at work station C, the next in-process product was delivered from work station B before tasks C1 to C3 were completed, causing a backlog of in-process products. As a result, productivity across the entire production line decreased, and it was clear that the production line had become inefficient.

[0020] Figure 1(b) shows the state of the production line after adjustment. In this production line, among the work items of the production line before adjustment shown in Figure 1(a), work A4 at work station A has been moved to the next work station B, and work station C has been made more efficient overall by improving the efficiency of work C1 to C3, thereby standardizing the work time of each of work stations A, B, and C to 55 minutes. As a result, the waiting times and backlogs of products in process that occurred on the production line before adjustment have been eliminated, and productivity can be improved across the entire production line.

[0021] The present invention realizes an information processing system that provides users, such as production line managers, with information for adjusting the production line as described above. Specifically, the information processing system of the present invention detects a series of actions performed by workers at each work station on the production line and, based on the detection results, analyzes the work time for each work item at each work station. Then, based on the obtained analysis results, it develops a plan for balancing the work between each work station and proposes this plan to the user. This helps the user improve the production line and increase productivity.

[0022] 2 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. The information processing system 100 shown in FIG. 2 includes a production line analysis device 1, a non-deterministic sensor 2, a deterministic sensor 3, and an action recognition device 4.

[0023] The non-deterministic sensor 2 and the deterministic sensor 3 each detect the movements of a worker at each work station on the production line to be analyzed, and transmit sensing data corresponding to the detection results and the detection time to the movement recognition device 4. The non-deterministic sensor 2 is a sensor whose movement detection results may contain some error, and includes, for example, wearable sensors such as glove-type sensors or jacket-type sensors worn by human workers, inertial measurement units (IMUs) that detect the movements of workers, audio sensors, pressure sensors, image sensors, and distance sensors. The deterministic sensor 3 is a sensor whose movement detection results contain almost no error, and therefore the error in the sensing data can be ignored, and includes, for example, an area sensor that detects the area in the production line where the worker is located, and a control signal from a manufacturing execution system (MES) that manages the production line and gives instructions to workers. The non-deterministic sensor 2 and the deterministic sensor 3 may each include types of sensors other than those described above.

[0024] The action recognition device 4 recognizes the actions of the worker at each work station on the production line based on the sensing data input from the non-deterministic sensor 2 and the deterministic sensor 3. In this worker action recognition, the action recognition device 4 subdivides the series of actions performed by the worker during work and recognizes them in units of the aforementioned atomic actions. For example, actions such as "walking," "picking up a screw," "holding a tool (e.g., a screwdriver, a drill)," "tightening a screw," "holding a component (e.g., a panel, a connector)," and "attaching a component" are recognized as atomic actions. This enables recognition of individual atomic actions with higher accuracy than when recognizing a series of actions during work as a whole. Note that atomic actions may also be expressed in other terms, such as "low-level action," "lower-level action," "atomic action," "basic action," "elementary action," or "indivisible action." In either case, the worker's actions recognized by the action recognition device 4 are the series of actions performed by the worker during work subdivided into short-term intervals and can be easily recognized based on the sensing data.

[0025] The action recognition device 4 generates action sequence data representing a series of actions of the worker on the production line from the recognition results of a series of atomic actions for the actions of the worker at each work station. The generated action sequence data is then sent to the production line analysis device 1. This action sequence data describes the contents and execution times of multiple atomic actions recognized by the action recognition device 4 in the order in which they were performed.

[0026] The action recognition device 4 can recognize atomic actions from sensing data using various well-known methods, such as the method described in Patent Document 1. The action recognition device 4 may be realized as one function of the production line analysis device 1, without being separated from the production line analysis device 1.

[0027] The production line analysis device 1 estimates the work items to be performed at each work station on the production line based on the action sequence data input from the action recognition device 4. Then, based on the obtained work item estimation results, it analyzes the work time at each work station and plans a strategy for balancing the work among the work stations.

[0028] The production line analysis device 1 is configured using a computer such as a server or a PC, and includes a processor 10, a memory 20, a recording device 30, an input / output interface 40, an input device 50, and an output device 60. In the production line analysis device 1, these are connected to each other via a communication bus.

[0029] The processor 10 executes a predetermined program to control the operation of the production line analysis device 1 and to cause a computer such as a server or PC to function as the production line analysis device 1. The processor 10 is configured using, for example, a central processing unit (CPU), a micro processing unit (MPU), and a graphics processing unit (GPU). The memory 20 is a volatile storage medium used as a work area when the processor 10 executes a program. The recording device 30 is a recording medium capable of storing various programs and data and is configured using, for example, a hard disk drive (HDD) or a solid state drive (SSD). The input / output interface 40 is a part that performs interface processing for data input and output to the production line analysis device 1. Action sequence data transmitted from the action recognition device 4 to the production line analysis device 1 is input to the production line analysis device 1 via the input / output interface 40, recorded in the recording device 30, and used in processing performed by the processor 10. The input device 50 is a device that accepts data input from a user to the production line analysis device 1 and is configured using, for example, a keyboard, a mouse, a touch panel, a microphone, etc. The output device 60 is a device that outputs information to the user, and is configured using, for example, a display, a printer, a speaker, and the like.

[0030] Fig. 3 is a block diagram showing the functional configuration of a production line analysis device 1 according to one embodiment of the present invention. As shown in Fig. 3, the production line analysis device 1 includes the following functional blocks: a data acquisition unit 11, a noise detection unit 12, a noise correction unit 13, a motion estimation unit 14, an analysis unit 15, and a proposal unit 16. These functional blocks are realized by the processor 10 in Fig. 2 reading out a program stored in the recording device 30 and executing processing in accordance with this program.

[0031] The production line analysis device 1 also stores the following information: operation sequence data 31, operation configuration information 32, sensor characteristic information 33, corrected operation sequence data 34, and production line analysis data 35. These pieces of information are stored in the memory 20 and recording device 30 of FIG. 2 in the production line analysis device 1.

[0032] The data acquisition unit 11 acquires the action sequence data input from the action recognition device 4 via the input / output interface 40. The action sequence data acquired by the data acquisition unit 11 is stored in the recording device 30 as action sequence data 31 in FIG.

[0033] The noise detection unit 12 detects noise contained in the action sequence data 31 recorded in the recording device 30. Due to erroneous or non-recognition errors or non-recognition by the action recognition device 4, the action sequence data 31 may contain atomic actions that differ from the actions actually performed by the worker, atomic actions that the worker did not actually perform, or some of the atomic actions performed by the worker may be missing. Based on the action configuration information 32, the noise detection unit 12 detects mismatches or omissions of atomic actions in the action sequence data 31 relative to the actual actions of the worker as noise contained in the action sequence data 31. The action configuration information 32 is information indicating combinations of atomic actions corresponding to each work item required to produce a product on a production line, and is set in advance based on product specifications, production manuals, etc. Hereinafter, the combination of atomic actions represented by the action configuration information 32, i.e., the series of atomic actions corresponding to each work item on the production line, will be referred to as "higher-level actions."

[0034] The noise correction unit 13 corrects the operation sequence data 31 based on the sensor characteristic information 33 so as to reduce the noise detected by the noise detection unit 12. The sensor characteristic information 33 is information representing the characteristics of the non-deterministic sensor 2 and the deterministic sensor 3 in FIG. 2, and is set in advance based on the features and performance of the various sensors included therein. The correction result of the operation sequence data 31 by the noise correction unit 13 is stored in the recording device 30 as corrected operation sequence data 34.

[0035] The action estimation unit 14 estimates higher-level actions corresponding to the work items performed by the workers at each work station on the production line based on the action sequence data 34 corrected by the noise correction unit 13. This makes it possible to estimate which work items were performed at each work station.

[0036] The analysis unit 15 analyzes the work allocation for each work station based on the estimation results of the higher-level actions by the action estimation unit 14. For example, the sequence of atomic actions represented by the corrected action sequence data 34 is divided into units of higher-level actions estimated by the action estimation unit 14, thereby grouping the atomic actions for each work item in the corrected action sequence data 34. Then, the start and end times of the atomic actions before and after each division are identified from the execution times of each atomic action included in the corrected action sequence data 34, thereby identifying the start and end times of each work item at each work station. The execution time of each work item is calculated from the start and end times identified in this way, and the total execution time for each work station is calculated. Furthermore, the work times obtained for each work station can be compared to determine the balance between the work stations and analyze the work allocation. Furthermore, the presence or absence of unnecessary actions can be determined from the execution time of each work item to analyze the work allocation. Various other methods can also be used to analyze the work allocation.

[0037] The results of the estimation of higher-level actions by the action estimation unit 14, i.e., information on the work items being performed at each work station, and information on the results of the analysis of work allocation by the analysis unit 15, are stored in the recording device 30 as production line analysis data 35.

[0038] The suggestion unit 16 proposes improvement measures to the user based on the analysis results of the work allocation by the analysis unit 15. For example, the suggestion unit 16 proposes improvement measures for improving the productivity of the production line, such as moving a work item being performed at one of the work stations to another work station, changing the order of multiple work items being performed at one of the work stations, or changing or streamlining the work procedures for the work items, and presents these to the user using the output device 60 to propose improvement measures for the production line. In this way, it is possible to propose to the user measures for improving productivity by realizing a balanced work allocation between the work stations.

[0039] FIG. 4 is a flowchart showing the flow of processing carried out by the production line analysis device 1.

[0040] In step S10, the data acquisition unit 11 acquires the movement sequence data input from the movement recognition device 4 and stores it in the recording device 30 as movement sequence data 31.

[0041] In step S20, the noise detection unit 12 performs noise detection processing on the action sequence data 31 acquired in step S10. Details of the noise detection processing in step S20 will be described later with reference to the flowchart in FIG.

[0042] In step S30, the noise correction unit 13 performs noise correction processing on the operation sequence data 31 to reduce the noise detected in the noise detection processing in step S20. The obtained correction result is stored in the recording device 30 as corrected operation sequence data 34. Details of the noise correction processing will be described later with reference to the flowchart in FIG.

[0043] In step S40, the motion estimation unit 14 performs a higher-level motion estimation process for estimating higher-level motions corresponding to the work items performed by the workers at each work station on the production line on the corrected motion sequence data 34 obtained by the noise correction process in step S30. Details of the higher-level motion estimation process will be described later with reference to the flowchart in FIG.

[0044] In step S50, the analysis unit 15 analyzes the work allocation among the work stations based on the high-level motion estimation results obtained by the high-level motion estimation process in step S40. Here, the work allocation can be analyzed by determining the balance between the work stations and determining whether or not there is any unnecessary motion in each work item, for example, using the method described above.

[0045] In step S60, it is determined whether the work allocation to each work station on the production line is appropriate based on the results of the work allocation analysis obtained in step S50. If the work allocation is appropriate, i.e., if it is determined in step S50 that the balance between the work stations is sufficient or that there are almost no unnecessary actions in each work item, the process shown in the flowchart in Figure 4 ends. On the other hand, if the work allocation is inappropriate, i.e., if it is determined in step S50 that the balance between the work stations is insufficient or that each work item contains unnecessary actions, it is determined that there is room for improvement in the work allocation to each work station, and the process proceeds to step S70.

[0046] In step S70, the proposing unit 16 proposes improvement measures for the work allocation at each work station. Here, based on the analysis results of the work allocation obtained in step S50, improvement measures are proposed, for example, as described above, to balance the work allocation among the work stations and reduce the work time on the entire production line. Note that, for example, computational processing using machine learning can be used to propose these improvement measures.

[0047] In step S80, the suggestion unit 16 suggests the improvement measures obtained in step S70 to the user. Here, the improvement measures can be suggested to the user by, for example, displaying them on a screen on a display included in the output device 60.

[0048] When the process of step S80 is completed, the process shown in the flowchart of FIG. 4 ends.

[0049] FIG. 5 is a flowchart showing the flow of the noise detection process executed by the noise detection unit 12 in step S20 of FIG.

[0050] In step S201, the overall positioning of the action sequence data 31 and the action configuration information 32 is performed. Here, for example, the degree of agreement between the sequence of atomic actions represented by the action sequence data 31 and the sequence of atomic actions corresponding to the work order of each higher-level action represented by the action configuration information 32 is compared while shifting the positions of the atomic actions one by one. For this comparison, for example, a well-known method called the Needleman-Wunsch algorithm can be used. As a result, the relative positional relationship between these when the highest degree of agreement is obtained can be identified, allowing the processing of step S201 to be performed.

[0051] In step S202, gaps and mismatches of atomic actions in the action sequence data 31 are detected with respect to the action configuration information 32 based on the results of the overall alignment performed in step S201. Here, mismatches in the content of atomic actions between the action sequence data 31 and the action configuration information 32 after overall alignment are detected as "mismatches," and atomic actions that exist in one but not the other are detected as "gaps." Furthermore, among the gaps, atomic actions that exist in the configuration information 32 but not in the action sequence data 31 are classified as "omissions," and atomic actions that exist in the action sequence data 31 but not in the configuration information 32 are classified as "additions." This makes it possible to detect mismatches, omissions, and additions of atomic actions in the action sequence data 31 due to erroneous recognition or non-recognition by the action recognition device 4 as noise, based on the order of atomic actions according to the task order of each higher-level action represented by the action configuration information 32.

[0052] In step S203, information on the noise (gap / mismatch) detected in step S202 is output as noise information for the operation sequence data 31. This noise information includes information such as the type (gap / mismatch) of noise detected in the operation sequence data 31, its position (the sequential position in the operation sequence data 31 of the atomic operation corresponding to the noise; in the case of a gap, the sequential position in the operation sequence data 31 of the atomic operation next to the gap), and the atomic operation string before the noise occurrence (N consecutive atomic operations in the sequential position before the noise position, where N is a predetermined natural number). The noise information output from the noise detection unit 12 in step S203 is input to the noise correction unit 13.

[0053] After completing the process of step S203, the noise detection unit 12 ends the noise detection process shown in the flowchart of FIG.

[0054] FIG. 6 is a flowchart showing the flow of the noise correction process executed by the noise correction unit 13 in step S30 of FIG.

[0055] In step S301, any one of the noises included in the operation sequence data 31 is selected based on the noise information input from the noise detection unit 12. Here, the noise represented by the noise information is selected in atomic operation units.

[0056] In step S302, the type of noise selected in step S301 is determined. If the noise type is "addition", the process proceeds to step S303; if it is "mismatch" or "omission", the process proceeds to step S304.

[0057] In step S303, the noise selected in step S301 is removed from the action sequence data 31. As a result, atomic actions included in the action sequence data 31 that are not included in the action configuration information 32 are excluded as unnecessary atomic actions. After the processing of step S303 is performed, the process proceeds to step S309.

[0058] In step S304, an atomic action sequence that occurs before the noise selected in step S301 is obtained from the action sequence data 31. Here, the atomic action sequence before the noise can be obtained based on information about the atomic action sequence before the occurrence of the noise, which is included in the noise information input from the noise detection unit 12, for example.

[0059] In step S305, candidates for the correct atomic operation for the noise selected in step S301 are identified. If the noise is a "missing noise," an atomic operation that should actually exist, as indicated by the configuration information 32, is identified as a candidate for the correct atomic operation. If the noise is a mismatch, another atomic operation that may be erroneously recognized as the atomic operation corresponding to the noise is identified as a candidate for the correct atomic operation, for example, based on the sensor characteristic information 33.

[0060] FIG. 8 is a table showing an example of sensor characteristic information 33 for a certain type of sensor included in the non-deterministic sensor 2. The table shown in FIG. 8 is an example of a confusion matrix showing the relationship between the recognition results obtained by the action recognition device 4 from the sensing data of the sensor and the actual atomic actions for three types of atomic actions, "A," "B," and "C." For example, when the recognition result of the atomic action by the action recognition device 4 is "A," the probability that it matches the actual atomic action is 0.9, the probability that "B" is erroneously recognized as "A" is 0.07, and the probability that "C" is erroneously recognized as "A" is 0.03. Similarly, when the recognition result of the atomic action by the action recognition device 4 is "B," the probability that it matches the actual atomic action is 0.85, the probability that "A" is erroneously recognized as "B" is 0.15, and the probability that "C" is erroneously recognized as "B" is 0.05. Furthermore, when the recognition result of the atomic action by the action recognition device 4 is "C," the probability that it matches the actual atomic action is 0.9, the probability that "A" is misrecognized as "C" is 0.05, and the probability that "B" is misrecognized as "C" is 0.1.

[0061] 6, for example, if the atomic action corresponding to noise is "B," then the atomic actions "A" and "C" are identified as candidates for the correct atomic action. Note that the method for identifying candidates for the correct atomic action is not limited to this, and it is possible to identify candidates for the correct atomic action by any method as long as it is possible to identify another atomic action that may result in a recognition result of an atomic action corresponding to noise due to erroneous recognition or recognition omission by the action recognition device 4.

[0062] In step S306, a matching score is calculated for each candidate for the correct atomic action identified in step S305. The matching score calculated here is a value that represents the degree of certainty that each candidate for the correct atomic action is an atomic action (correct atomic action) actually performed by the worker. This matching score can be calculated, for example, based on the action configuration information 32, the sensor characteristic information 33, and the pre-noise atomic action sequence acquired in step S304. Specifically, the combination of atomic actions of each higher-level action represented by the action configuration information 32 is compared with the pre-noise atomic action sequence, and the probability that each candidate for the correct atomic action appears after the pre-noise atomic action sequence is calculated. If the noise type is "missing," this value is the matching score. Furthermore, if the noise type is "mismatch," the matching score for each candidate for the correct atomic action can be calculated by multiplying the calculated probability of each candidate by the probability of misrecognition of each candidate, as described above, represented by the sensor characteristic information 33. In addition, when the action recognition device 4 recognizes an atomic action based on sensing data from multiple types of sensors, it is preferable to multiply the probability of erroneous recognition for each sensor represented by the sensor characteristic information 33 by the probability of each candidate calculated from the atomic action sequence before noise.

[0063] In step S307, from among the candidates for the correct atomic action identified in step S305, the one with the highest matching score calculated in step S306 is selected as the correct atomic action.

[0064] In step S308, the noise type is determined. If the noise type is "mismatch", in step S309, the noise selected in step S301 is replaced with the correct atomic action selected in step S307 in the action sequence data 31. After the processing of step S309 is performed, the process proceeds to step S311. If the noise type determination result in step S308 is "omission", in step S310, the correct atomic action selected in step S307 is inserted before the noise position. This is because in the case of "omission", the atomic action immediately after the "omission" is set as the noise position. After the processing of step 310 is performed, the process proceeds to step S311. Through the above processing, the "mismatch" and "omission" in the action sequence data 31 are corrected.

[0065] In step S311, it is determined whether all noises included in the operation sequence data 31 have been selected in step S301. If there are any unselected noises, the process returns to step S301, and one of the unselected noises is selected in step S301, after which the processes of step S302 and subsequent steps are performed on that noise. As a result, the process of step S303 or steps S304 to S310 is performed on each atomic operation corresponding to the noise, and the noise is either removed from the operation sequence data 31, replaced with a correct atomic operation, or inserted. If it is determined in step S311 that all noises have been selected, the process proceeds to step S312.

[0066] In step S312, the correction results for each noise in the action sequence data 31 obtained by the processes in steps S301 to S311 are recorded in the memory 20 or the recording device 30 as corrected action sequence data .

[0067] After completing the process of step S312, the noise correction unit 13 ends the noise correction process shown in the flowchart of FIG.

[0068] FIG. 7 is a flowchart showing the flow of the higher-level motion estimation process executed by the motion estimator 14 in step S40 of FIG.

[0069] In step S401, one of the atomic operations included in the corrected operation sequence data 34 generated by the noise correction unit 13 is selected. Here, the unselected atomic operations are selected in order starting from the one closest to the top of the list.

[0070] In step S402, for the atomic action selected in step S401, a forward segment and a backward segment are set in the corrected action sequence data 34. A forward segment is a combination of atomic actions extracted in order from the selected atomic action to the atomic action that is a predetermined number of steps behind, in accordance with the order of the atomic actions in the corrected action sequence data 34. On the other hand, a backward segment is a combination of atomic actions extracted in order from the selected atomic action to the atomic action that is a predetermined number of steps before, in the reverse direction of the order of the atomic actions in the corrected action sequence data 34. The number of atomic actions included in each of the forward segment and the backward segment can be set arbitrarily, for example, to three.

[0071] In step S403, for the forward and backward segments set in step S402, a similarity score is calculated for each higher-level action defined in the action configuration information 32. Here, for each forward and backward segment, the similarity between the combination of atomic actions that make up these segments and the combination of atomic actions of each higher-level action represented by the action configuration information 32, i.e., the degree of matching of each atomic action, is calculated for each higher-level action.

[0072] In step S404, the higher-level action with the best similarity score calculated in step S403 is selected. Here, the higher-level action with the best score among the similarity scores calculated for each higher-level action for each forward segment and reverse segment, i.e., the higher-level action with the highest degree of match with the forward segment or the reverse segment, is selected as the higher-level action with the best similarity score.

[0073] In step S405, it is determined whether the difference between the similarity score of the higher-level action selected in step S404, i.e., the best similarity score, and the similarity scores of the other higher-level actions is equal to or greater than a predetermined value. If the difference is equal to or greater than the predetermined value, i.e., if the best similarity score can be considered to be sufficiently higher than the similarity scores of the other higher-level actions, the higher-level action selected in step S404 is estimated to be the higher-level action to which the atomic action selected in step S401 belongs, and the process proceeds to step S409. On the other hand, if the difference is less than the predetermined value, the process proceeds to step S406.

[0074] In step S406, candidates for the higher-level operation to which the atomic operation selected in step S401 belongs are identified based on the similarity score calculated in step S403. Here, for example, the higher-level operation with the best similarity score selected in step S404 and each higher-level operation whose difference in similarity score from that higher-level operation is within a predetermined value are identified as candidates for the higher-level operation to which the selected atomic operation belongs.

[0075] In step S407, for each combination of each candidate identified in step S406 and the previous estimation result, an alignment score is calculated for the operation configuration information 32. Here, for example, for the combination of each candidate for higher-level operation identified in step S406 and the previous estimation result, i.e., the higher-level operation estimated as the one to which the atomic operation selected in the previous step S401 belongs, the similarity of each higher-level operation with respect to the task order represented by the operation configuration information 32 is calculated as an alignment score. This makes it possible to use the previous estimation result of the higher-level operation for a certain atomic operation to estimate the higher-level operation corresponding to another atomic operation.

[0076] In step S408, one of the candidates is selected based on the alignment score of each candidate calculated in step S407. Here, the candidate for the higher-level operation that has the best alignment score is selected, that is, the candidate for the higher-level operation that has the highest degree of match with the operation configuration information 32 obtained for the combination with the previous estimation result. This allows the candidate for the higher-level operation to be estimated as the higher-level operation to which the atomic operation selected in step S401 belongs.

[0077] In step S409, it is determined whether all atomic operations included in the corrected operation sequence data 34 have been selected in step S401. If there are unselected atomic operations, the process returns to step S401, and one of the unselected atomic operations is selected in step S401, after which the processing from step S402 onwards is performed on that atomic operation. As a result, the processing of steps S402 to S404 is performed on each atomic operation included in the corrected operation sequence data 34, and further the processing of steps S406 to S408 is performed as necessary, thereby inferring higher-level operations corresponding to each atomic operation and inferring boundaries between each higher-level operation in the corrected operation sequence data 34 in step S404 or S408. If it is determined in step S409 that all atomic operations have been selected, the process proceeds to step S410.

[0078] In step S410, the movement estimation unit 14 outputs to the analysis unit 15 the estimation result of the higher-level movement corresponding to the corrected movement sequence data 34 obtained by the processes of steps S401 to S409.

[0079] After completing the process of step S410, the movement estimator 14 ends the higher-level movement estimation process shown in the flowchart of FIG.

[0080] Next, the planning of improvement measures performed in step S70 of FIG. 4 will be described.

[0081] In step S70, for example, a suggested improvement measure is to move one of the work items at a work station with a relatively long work time to another work station with a shorter work time, based on the work time at each work station calculated by the analysis unit 15. In this case, it is preferable that the work stations before and after the move are closely related to each other, such as being in a chronological order on the production line.

[0082] Furthermore, for work stations with long work times, the system may estimate the required work time. If the actual work time deviates significantly from the estimated time, the system may determine that the current work procedure is wasteful and propose improvements to improve work efficiency. For example, multiple atomic actions belonging to the higher-level actions corresponding to the work items of the work station may be classified into core actions that form the core of the work item and non-core actions that are not core actions. For example, core actions include screw tightening, while non-core actions include walking and gripping a tool. The system then obtains the time required for each atomic action from the execution time of each atomic action included in the corrected action sequence data 34 and compares these values. If the time required for the non-core actions is longer than the time required for the core actions, the system may determine that the work efficiency is low and propose improvements such as deleting or shortening some of the atomic actions included in the non-core actions. Specifically, improvements that can be proposed to improve work efficiency include, for example, changing the layout of the work station, changing the arrangement of tools used by workers at the work station, and changing the arrangement of the area where products in progress at the work station and parts to be assembled thereto are prepared.

[0083] Furthermore, improvements to improve work efficiency may be proposed, such as changing the order of multiple work items performed at a work station, or changing the order of execution of individual work items, i.e., changing the order of multiple atomic actions belonging to higher-level actions corresponding to each work item, etc. In addition to these, any measure can be proposed as an improvement measure as long as it can shorten the work time at a work station.

[0084] According to the embodiment of the present invention described above, the following advantageous effects can be obtained.

[0085] (1) The information processing system 100 includes a production line analysis device 1. The production line analysis device 1 includes a data acquisition unit 11 that acquires action sequence data 31 representing a series of actions performed by a worker on a production line having multiple work stations, a noise detection unit 12 that detects noise from the action sequence data 31 based on action configuration information 32 that represents the relationship between atomic actions performed by the worker and higher-level actions consisting of combinations of the multiple atomic actions, a noise correction unit 13 that corrects the action sequence data 31 to reduce noise, and an action estimation unit 14 that estimates higher-level actions corresponding to work items performed by the worker at each work station on the production line based on the action sequence data 34 corrected by the noise correction unit 13. This configuration makes it possible to accurately estimate higher-level actions from the recognition results of a series of atomic actions performed by the worker.

[0086] (2) The movement estimation unit 14 sets segments in the corrected movement sequence data 34, each combining a predetermined number of atomic movements, in the forward and backward directions (step S402), and estimates boundaries of higher-level movements based on these segments (steps S404, S408). This allows for highly accurate estimation of higher-level movements from the corrected movement sequence data 34.

[0087] (3) The movement estimation unit 14 identifies multiple candidates for higher-level movements for the sequence of atomic movements in the corrected movement sequence data 34 (step S406), and calculates an alignment score for the movement configuration information 32 for a combination of each identified candidate and the estimation result of the previous higher-level movement (step S407). Then, the movement estimation unit 14 estimates the higher-level movement corresponding to the work item by selecting one of the multiple candidates based on the calculated alignment score (step S408). As a result, even if there are multiple higher-level movements in the movement configuration information 32 that are highly similar to the sequence of atomic movements included in the corrected movement sequence data 34, it is possible to appropriately select one of them as the estimation result of the higher-level movement corresponding to the work item.

[0088] (4) The action sequence data 31 represents a sequence of multiple atomic actions each recognized by the action recognition device 4 based on sensor data acquired by a sensor that detects the worker's actions. Furthermore, the noise detected by the noise detection unit 12 is a discrepancy or gap in the recognition result of the atomic actions by the action recognition device 4 based on the sensor data, relative to the action configuration information 32. The noise correction unit 13 corrects the action sequence data 31 by correcting the noise using preset sensor characteristic information 33. As a result, even if discrepancies or gaps caused by erroneous or non-recognition by the action recognition device 4 are included as noise in the action sequence data 31, the action sequence data 31 can be corrected to reduce the noise.

[0089] (5) The production line analysis device 1 includes an analysis unit 15 that analyzes the start time and / or end time of work items corresponding to higher-level actions estimated by the action estimation unit 14 for each work station on the production line, and a proposal unit 16 that proposes improvement measures for the production line using the start time and / or end time for each work item at each work station obtained by the analysis unit 15. As a result, if there is room for improvement in the work allocation at each work station on the production line, it is possible to propose improvement measures to the user to improve productivity.

[0090] (6) The analysis unit 15 can classify multiple atomic actions included in the higher-level actions estimated by the action estimation unit 14 into core actions that form the core of the work items performed by the worker at the work station and non-core actions other than the core actions. In this case, the suggestion unit 16 can compare a first time required for the core action with a second time required for the non-core action and propose an improvement measure based on the comparison result. Specifically, for example, the core action includes screw tightening, and the non-core action includes walking and / or gripping a tool. In this way, it is possible to propose an appropriate improvement measure to improve work efficiency for a work station where the current work procedure is wasteful.

[0091] (7) The proposing unit 16 can also propose, as an improvement measure, a change to at least one of the order of atomic operations in the higher-level operations estimated by the operation estimating unit 14, the order of operations performed at the same work station, and the work station where the work item is performed. In this way, it is possible to propose an optimal improvement measure to eliminate imbalances in the production line and improve productivity.

[0092] (8) The proposing unit 16 can also propose, as an improvement measure, a change to at least one of the layout of a plurality of work stations in the production line, the arrangement of tools used by workers at the work stations, and the arrangement of products or parts at the work stations. In this way, it is possible to propose an improvement measure appropriate for shortening the work time of a work station that takes longer than other work stations.

[0093] (9) The action sequence data 31 represents a sequence of multiple atomic actions each recognized based on sensor data acquired by a sensor that detects the worker's actions while the worker is performing a task. The sensor may include, for example, at least one of an inertial measurement unit, a sound sensor, a pressure sensor, an image sensor, a distance sensor, and a wearable sensor as a non-deterministic sensor 2. This makes it possible to easily and reliably acquire the sensor data necessary to recognize the worker's atomic actions.

[0094] It goes without saying that the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above-described embodiments with other configurations.

[0095] Furthermore, the above-mentioned configurations, functional units, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-mentioned configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a hard disk, a recording device such as an SSD, an IC card, an SD card, a DVD, or other recording media.

[0096] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. For example, it can be considered that almost all components are actually connected to each other. [Explanation of symbols]

[0097] 1: Production line analysis equipment 2: Non-deterministic sensors 3: Deterministic Sensor 4: Motion recognition device 10: Processor 11: Data acquisition section 12: Noise detection section 13: Noise correction section 14: Motion estimation section 15:Analysis Department 16: Proposal Department 20: Memory 30: Recording device 31: Operation sequence data 32: Operational configuration information 33: Sensor characteristic information 34: Corrected motion sequence data 35: Production line analysis data 40: Input / output interface 50: Input device 60: Output device 100: Information Processing Systems

Claims

1. a data acquisition unit that acquires motion sequence data representing a series of motions of a worker in a production line having a plurality of work stations; a noise detection unit that detects noise from the action sequence data based on action configuration information that represents a relationship between the atomic action performed by the worker and a higher-level action that is a combination of a plurality of the atomic actions; a noise correction unit that corrects the operation sequence data so as to reduce the noise; an action estimation unit that estimates the higher-level actions corresponding to the work items performed by the workers at each work station on the production line based on the action sequence data corrected by the noise correction unit.

2. 2. The information processing system according to claim 1, The motion estimation unit sets segments in the corrected motion sequence data, each of which combines a predetermined number of atomic motions, in a forward direction and a reverse direction, and estimates boundaries of the higher-level motions based on the segments.

3. 2. The information processing system according to claim 1, the operation estimation unit identifies a plurality of candidates for the higher-level operation for the sequence of the atomic operations in the operation sequence data after correction, calculates an alignment score for the operation configuration information for a combination of each identified candidate and a past estimation result of the higher-level operation, and selects one of the plurality of candidates based on the alignment score, thereby estimating the higher-level operation corresponding to the work item.

4. 2. The information processing system according to claim 1, the action sequence data represents a sequence of the atomic actions each recognized based on sensor data acquired by a sensor that detects the action of the worker; The noise is a discrepancy or gap in the recognition result of the atomic action based on the sensor data with respect to the action configuration information; The noise correction unit corrects the operation sequence data by correcting the noise using preset characteristic information of the sensor.

5. 2. The information processing system according to claim 1, an analysis unit that analyzes the start time and / or end time of the work item corresponding to the higher-level action estimated by the action estimation unit for each work station on the production line; a proposal unit that proposes improvements to the production line using the start time and / or the end time for each work item at each work station.

6. 6. The information processing system according to claim 5, the analysis unit classifies the plurality of atomic actions included in the higher-level action into core actions that form the core of the work item performed by the worker at the work station and non-core actions other than the core actions; The suggestion unit compares a first time required for the core operation with a second time required for the non-core operation, and proposes the improvement measure based on the comparison result.

7. 7. The information processing system according to claim 6, the core operation includes screw tightening; The non-core motions include walking and / or gripping a tool.

8. 6. The information processing system according to claim 5, The information processing system is configured such that the proposal unit proposes, as the improvement measure, a change to at least one of the order of the atomic operations in the higher-level operations, the order of execution of multiple work items performed at the same work station, and the work station on which the work items are performed.

9. 6. The information processing system according to claim 5, The proposal unit is an information processing system that proposes, as the improvement measure, a change to at least one of the layout of the multiple work stations in the production line, the arrangement of tools used by the workers at the work stations, and the arrangement of products or parts at the work stations.

10. 2. The information processing system according to claim 1, the action sequence data represents a sequence of the atomic actions each recognized based on sensor data acquired by a sensor that detects an action of the worker while the worker is performing the work item; The information processing system, wherein the sensor includes at least one of an inertial measurement unit, an audio sensor, a pressure sensor, an image sensor, a distance sensor, and a wearable sensor.

11. By computer, acquiring motion sequence data representing a series of motions of a worker in a production line having a plurality of work stations; detecting noise from the action sequence data based on action configuration information representing a relationship between the atomic action performed by the worker and a higher-level action formed by a combination of a plurality of the atomic actions; correcting the motion sequence data so that the noise is reduced; an information processing method for estimating the higher-level actions corresponding to the operations performed by the workers at each work station on the production line based on the corrected operation sequence data;

12. 12. The information processing method according to claim 11, an information processing method for setting segments each combining a predetermined number of atomic actions in the corrected action sequence data in a forward direction and a backward direction, and estimating a boundary of the higher-level action based on the segments;

13. 12. The information processing method according to claim 11, An information processing method for estimating the higher-level operation corresponding to the work item by identifying multiple candidates for the higher-level operation for the sequence of atomic operations in the corrected operation sequence data, calculating an alignment score for the operation configuration information for a combination of each identified candidate and an estimation result of the previous higher-level operation, and selecting one of the multiple candidates based on the alignment score.

14. 12. The information processing method according to claim 11, the action sequence data represents a sequence of the atomic actions each recognized based on sensor data acquired by a sensor that detects the action of the worker; The noise is a discrepancy or gap in the recognition result of the atomic action based on the sensor data with respect to the action configuration information; An information processing method for correcting the operation sequence data by correcting the noise using predetermined characteristic information of the sensor.

15. 12. The information processing method according to claim 11, For each work station in the production line, performing an analysis of start and / or end times of the work items corresponding to the estimated high-level operations; An information processing method for proposing improvements to the production line using the start time and / or the end time for each work item at each work station.

16. 16. The information processing method according to claim 15, An information processing method that classifies the multiple atomic operations included in the higher-level operations into core operations that form the core of the work item performed by the worker at the work station and non-core operations other than the core operations, compares a first time required for the core operations with a second time required for the non-core operations, and proposes the improvement measures based on the comparison results.

17. 17. The information processing method according to claim 16, the core operation includes screw tightening; The non-core activity is an information processing method including walking and / or gripping a tool.

18. 17. The information processing method according to claim 16, An information processing method that proposes, as the improvement measure, a change to at least one of the order of the atomic operations in the higher-level operation, the order of execution of multiple work items performed at the same work station, and the work station on which the work items are performed.

19. 16. The information processing method according to claim 15, An information processing method that proposes, as the improvement measure, a change to at least one of the layout of the plurality of work stations in the production line, the arrangement of tools used by the workers at the work stations, and the arrangement of products or parts at the work stations.

20. 12. The information processing method according to claim 11, the action sequence data represents a sequence of the atomic actions each recognized based on sensor data acquired by a sensor that detects an action of the worker while the worker is performing the work item; The information processing method, wherein the sensor includes at least one of an inertial measurement unit, an audio sensor, a pressure sensor, an image sensor, a distance sensor, and a wearable sensor.

Images