Information processing system and information processing method

Abstract

This information processing system comprises: a data acquisition unit that acquires operation sequence data representing a series of operations of a worker in a production line having a plurality of work stations; a noise detection unit that detects noise from the operation sequence data on the basis of operation constitution information representing the relationship between atomic operations by the worker and a higher-level operation comprising a combination of the plurality of atomic operations; a noise correction unit that corrects the operation sequence data so as to reduce the noise; and an operation inference unit that infers, on the basis of the operation sequence data after the correction by the noise correction unit, the higher-level operation corresponding to a work item executed by the worker at each of the work stations of the production line.

Description

Information processing system and information processing method

[0001] The present invention relates to an information processing system and an information processing method used for analyzing work in each process of a production line.

[0002] Generally, production lines for manufacturing various products, such as automobiles, are composed of multiple work stations. For example, if the production volume per hour is 60 units, the production line is composed of 60 work stations, each with a one-minute work time, and a product is completed every minute on this production line. The work time for each work station can also be set to 5 or 10 minutes. At each work station, workers, such as humans or robots, perform specific work items related to the production of the product.

[0003] In a production line, if work is not properly allocated to each work station, variations in work time will occur between the work stations. For example, if the average work time for all work stations is 5 minutes, one work station may take 7-8 minutes, while another may take 3-4 minutes. In this case, the former work station will be overloaded, causing in-process products to back up, while the latter work station will have to wait for work. This situation is called an imbalance in the production line, and because it reduces productivity, it is best to avoid it as much as possible.

[0004] To improve product productivity, it is important to properly balance the work time at each work station. To achieve this, it is necessary to accurately know the work time for each work item at each work station. In recent years, work analysis methods have been devised that acquire worker motion data at each work station, and then use machine learning or other techniques to recognize the sequence of worker motions at each work station based on this motion data. This identifies the work items corresponding to the recognized motions and measures their work time. However, these work analysis methods have difficulty accurately recognizing complex, long-duration motions due to limitations in computer processing power and machine learning characteristics. Furthermore, different production lines and workers may perform different motions for the same work item. For these reasons, it is difficult to accurately recognize a sequence of worker motions from their motion data.

[0005] Therefore, a method has been proposed in which a series of actions performed by workers at each work station is broken down into short-duration actions (hereinafter referred to as "atomic actions") and action recognition is performed in atomic action units. This method makes it possible to realize action recognition using a computer. Furthermore, because individual atomic actions have many commonalities even across different production lines and manufacturers, even when action recognition in task units is difficult, correct action recognition in atomic action units is possible.

[0006] When recognizing actions in atomic operations, it is necessary to identify the start and end actions of each work item from among the various recognized atomic operations, thereby setting boundaries for each work item in the series of atomic operations, and converting the sequence of atomic operations separated by these boundaries into higher-level actions corresponding to the work items at each work station. In this regard, for example, Patent Literature 1 and Non-Patent Literature 1 are known. Patent Literature 1 describes a method for recognizing atomic operations from image data using a long short-term memory (LSTM) and a convolutional neural network (CNN), and then applying a mathematical model called a finite state automata (FSA) to the recognition results to identify higher-level actions consisting of a combination of multiple atomic operations. Non-Patent Literature 1 also describes a method for inferring higher-level actions from a series of lower-level actions corresponding to atomic operations using a probabilistic model called a conditional random field (CRF).

[0007] U.S. Pat. No. 11,054,811

[0008] Bordel, B., Alcarria, R., & Robles, T., "Recognizing human activities in Industry 4.0 scenarios through an analysis-modeling-recognition algorithm and context labels", Integrated Computer-Aided Engineering, 29 (2022), p. 83-103

[0009] The method of Patent Literature 1 identifies higher-level actions using FSA, a rule-based model, and therefore is insufficient to correctly identify higher-level actions when the obtained atomic action recognition results contain noise or deviate from predefined rules. In other words, the recognition accuracy of atomic actions is not always 100%, and workers may make operational errors during work. Therefore, it is difficult for FSA to set boundaries of higher-level actions with sufficient accuracy for a series of atomic actions obtained under these circumstances. On the other hand, the method of Non-Patent Literature 1 identifies higher-level actions using a probabilistic model, and therefore it is difficult to estimate higher-level actions with the desired accuracy when, for example, each work station on a production line has its own work manual or when there are common atomic actions across multiple higher-level actions.

[0010] The present invention has been made based on the above background, and aims to accurately estimate higher-level actions from the recognition results of a series of atomic actions performed by a worker.

[0011] An information processing system according to the present invention comprises: a data acquisition unit that acquires action sequence data representing a series of actions of a worker on a production line having a plurality of work stations; a noise detection unit that detects noise from the action sequence data based on action configuration information representing the relationship between atomic actions by the worker and higher-level actions consisting of combinations of a plurality of the atomic actions; a noise correction unit that corrects the action sequence data to reduce the noise; and an action estimation unit that estimates the higher-level actions corresponding to work items performed by the worker at each work station on the production line based on the action sequence data corrected by the noise correction unit. An information processing method according to the present invention comprises: acquiring action sequence data representing a series of actions of a worker on a production line having a plurality of work stations; detecting noise from the action sequence data based on action configuration information representing the relationship between atomic actions by the worker and higher-level actions consisting of combinations of a plurality of the atomic actions; correcting the action sequence data to reduce the noise; and estimating the higher-level actions corresponding to work items performed by the worker at each work station on the production line based on the corrected action sequence data.

[0012] According to the present invention, higher-level actions can be accurately estimated from the recognition results of a series of atomic actions performed by a worker.

[0013] A diagram showing an overview of a production line to which the present invention is applied. A diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. A block diagram showing the functional configuration of a production line analysis device according to an embodiment of the present invention. A flowchart showing the flow of processing performed by the production line analysis device. A flowchart showing the flow of noise detection processing. A flowchart showing the flow of noise correction processing. A flowchart showing the flow of higher-level operation estimation processing. A table showing an example of sensor characteristic information.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] Fig. 1 is a diagram showing an overview of a production line to which the present invention is applied. When applied to a production line such as that shown in Fig. 1, the present invention analyzes the work performed on the production line and, based on the results of the analysis, makes suggestions to a user, such as a production line manager, for improving productivity.

[0016] FIG. 1( a) shows the state of the production line before adjustment. This production line is composed of multiple work stations, including work stations A, B, and C. At each work station, workers perform predetermined work items to produce products such as automobiles. At each work station, the predetermined work items are performed by either humans or robots, or by a combination of humans and robots. Hereinafter, the workers at each work station, regardless of whether they are humans or robots, will be collectively referred to as "workers."

[0017] 1(a), for example, at work station A, tasks A1, A2, A3, and A4 are performed in sequence, with the total work time being 60 minutes. When tasks A1 to A4 are completed at work station A, the in-process product is transported to the next work station B, where work begins.

[0018] At work station B, work items B1 and B2 are performed in sequence, with the total work time being 50 minutes. When work items B1 and B2 are completed at work station B, the product in progress is moved to the next work station C, and work begins at work station C. At work station C, work items C1, C2, and C3 are performed in sequence, with the total work time being 60 minutes. Similarly, one or more predetermined work items are performed in sequence at each work station, and after they are completed, the work item at the next work station begins.

[0019] As described above, in the production line before adjustment, the working time for work stations A and C was 60 minutes, while the working time for work station B was 50 minutes. Therefore, at work station B, after tasks B1 and B2 were completed, there was a waiting time of about 10 minutes until the next in-process product was delivered from work station A. Meanwhile, at work station C, the next in-process product was delivered from work station B before tasks C1 to C3 were completed, causing a backlog of in-process products. As a result, it can be seen that productivity has decreased across the entire production line, making it an inefficient production line.

[0020] Figure 1(b) shows the state of the production line after adjustment. In this production line, among the work items of the production line before adjustment shown in Figure 1(a), work A4 at work station A has been moved to the next work station B, and work station C has been made more efficient overall by improving the efficiency of work C1 to C3, thereby standardizing the work time of each of work stations A, B, and C to 55 minutes. As a result, the waiting time and backlog of products in process that occurred on the production line before adjustment have been eliminated, and productivity of the entire production line can be improved.

[0021] The present invention realizes an information processing system that provides users, such as production line managers, with information for adjusting the production line as described above. Specifically, the information processing system of the present invention detects a series of actions performed by workers at each work station on the production line and, based on the detection results, analyzes the work time for each work item at each work station. Then, based on the obtained analysis results, it develops a plan for balancing the work between each work station and proposes this plan to the user. This helps the user improve the production line and increase productivity.

[0022] 2 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. The information processing system 100 shown in FIG. 2 includes a production line analysis device 1, a non-deterministic sensor 2, a deterministic sensor 3, and an action recognition device 4.

[0023] The non-deterministic sensor 2 and the deterministic sensor 3 each detect the movements of a worker at each work station on the production line to be analyzed and transmit sensing data corresponding to the detection results and the detection time to the movement recognition device 4. The non-deterministic sensor 2 is a sensor whose movement detection results may contain a certain degree of error. Examples include wearable sensors such as glove-type sensors or jacket-type sensors worn by human workers, inertial measurement units (IMUs) that detect worker movements, audio sensors, pressure sensors, image sensors, and distance sensors. The deterministic sensor 3 is a sensor whose movement detection results contain almost no error and therefore the error in the sensing data can be ignored. Examples include area sensors that detect the area in the production line where the worker is located, and control signals from a manufacturing execution system (MES) that manages the production line and gives instructions to workers. The non-deterministic sensor 2 and the deterministic sensor 3 may each include types of sensors other than those described above.

[0024] The action recognition device 4 recognizes the actions of a worker at each work station on the production line based on sensing data input from the non-deterministic sensor 2 and the deterministic sensor 3. In this worker action recognition, the action recognition device 4 subdivides a series of actions performed by the worker during work and recognizes them in units of the aforementioned atomic actions. For example, actions such as "walking," "picking up a screw," "holding a tool (e.g., a screwdriver, a drill)," "tightening a screw," "holding a component (e.g., a panel, a connector)," and "attaching a component" are recognized as atomic actions. This enables recognition of individual atomic actions with higher accuracy than when recognizing a series of actions performed during work as a whole. Note that atomic actions may also be expressed by other terms, such as "low-level action," "lower-level action," "atomic action," "basic action," "elementary action," or "indivisible action." In either case, the worker's actions recognized by the action recognition device 4 are a series of actions performed by the worker during work subdivided into short-term intervals and can be easily recognized based on sensing data.

[0025] The action recognition device 4 generates action sequence data representing a series of actions of the worker on the production line from the recognition results of a series of atomic actions for the actions of the worker at each work station, and then transmits the generated action sequence data to the production line analysis device 1. This action sequence data describes the contents and execution times of the multiple atomic actions recognized by the action recognition device 4 in the order in which they were performed.

[0026] The action recognition device 4 can recognize atomic actions from sensing data using various well-known methods, such as the method described in Patent Document 1. The action recognition device 4 may be realized as one function of the production line analysis device 1, without being separated from the production line analysis device 1.

[0027] The production line analysis device 1 estimates the work items to be performed at each work station on the production line based on the action sequence data input from the action recognition device 4. Then, based on the obtained work item estimation results, it analyzes the work time at each work station and plans a strategy for balancing the work among the work stations.

[0028] The production line analysis device 1 is configured using a computer such as a server or a PC, and includes a processor 10, a memory 20, a recording device 30, an input / output interface 40, an input device 50, and an output device 60. In the production line analysis device 1, these are connected to each other via a communication bus.

[0029] The processor 10 executes a predetermined program to control the operation of the production line analysis device 1 and to cause a computer such as a server or PC to function as the production line analysis device 1. The processor 10 is configured using, for example, a central processing unit (CPU), a microprocessing unit (MPU), or a graphics processing unit (GPU). The memory 20 is a volatile storage medium used as a work area when the processor 10 executes a program. The recording device 30 is a recording medium capable of storing various programs and data and is configured using, for example, a hard disk drive (HDD) or a solid state drive (SSD). The input / output interface 40 is a part that performs interface processing for data input and output to the production line analysis device 1. Action sequence data transmitted from the action recognition device 4 to the production line analysis device 1 is input to the production line analysis device 1 via the input / output interface 40, recorded in the recording device 30, and used in processing by the processor 10. The input device 50 is a device that accepts data input from a user to the production line analysis device 1 and is configured using, for example, a keyboard, mouse, touch panel, microphone, etc. The output device 60 is a device that outputs information to the user, and is configured using, for example, a display, a printer, a speaker, and the like.

[0030] Fig. 3 is a block diagram showing the functional configuration of a production line analysis device 1 according to an embodiment of the present invention. As shown in Fig. 3, the production line analysis device 1 includes the following functional blocks: a data acquisition unit 11, a noise detection unit 12, a noise correction unit 13, a motion estimation unit 14, an analysis unit 15, and a proposal unit 16. These functional blocks are realized by the processor 10 in Fig. 2 reading out a program stored in the recording device 30 and executing processing in accordance with this program.

[0031] The production line analysis device 1 also stores the following information: operation sequence data 31, operation configuration information 32, sensor characteristic information 33, corrected operation sequence data 34, and production line analysis data 35. These pieces of information are stored in the memory 20 and recording device 30 of FIG. 2 in the production line analysis device 1.

[0032] The data acquisition unit 11 acquires the action sequence data input from the action recognition device 4 via the input / output interface 40. The action sequence data acquired by the data acquisition unit 11 is stored in the recording device 30 as action sequence data 31 in FIG. 3 .

[0033] The noise detection unit 12 detects noise contained in the operation sequence data 31 recorded in the recording device 30. Due to erroneous or non-recognition errors or non-recognition by the operation recognition device 4, the operation sequence data 31 may contain atomic operations that differ from the operations actually performed by the worker, atomic operations that the worker did not actually perform, or some atomic operations performed by the worker may be missing. Based on the operation configuration information 32, the noise detection unit 12 detects mismatches or omissions of atomic operations in the operation sequence data 31 relative to the actual operations of the worker as noise contained in the operation sequence data 31. The operation configuration information 32 is information indicating combinations of atomic operations corresponding to each work item required to produce a product on a production line, and is set in advance based on product specifications, production manuals, etc. Hereinafter, the combinations of atomic operations represented by the operation configuration information 32, i.e., the series of atomic operations corresponding to each work item on the production line, will be referred to as "higher-level operations."

[0034] The noise correction unit 13 corrects the operation sequence data 31 based on the sensor characteristic information 33 so as to reduce the noise detected by the noise detection unit 12. The sensor characteristic information 33 is information representing the characteristics of the non-deterministic sensor 2 and the deterministic sensor 3 shown in FIG. 2 , and is set in advance based on the features and performance of the various sensors included therein. The correction result of the operation sequence data 31 by the noise correction unit 13 is stored in the recording device 30 as corrected operation sequence data 34.

[0035] The action estimation unit 14 estimates higher-level actions corresponding to the work items performed by the workers at each work station on the production line based on the action sequence data 34 corrected by the noise correction unit 13. This makes it possible to estimate which work items were performed at each work station.

[0036] The analysis unit 15 analyzes the work allocation for each work station based on the estimation results of the higher-level actions by the action estimation unit 14. For example, the sequence of atomic actions represented by the corrected action sequence data 34 is divided into units of higher-level actions estimated by the action estimation unit 14, thereby grouping the atomic actions for each work item in the corrected action sequence data 34. Then, the start and end times of the atomic actions before and after each division are identified from the execution times of each atomic action included in the corrected action sequence data 34, thereby identifying the start and end times of each work item at each work station. The execution time of each work item is calculated from the start and end times identified in this way, and the total execution time for each work item is calculated for each work station. Furthermore, the obtained execution times for each work station can be compared to determine the balance between the work stations and analyze the work allocation. Furthermore, the presence or absence of unnecessary actions can be determined from the execution time of each work item to analyze the work allocation. Various other methods can also be used to analyze the work allocation.

[0037] The results of the estimation of higher-level actions by the action estimation unit 14, i.e., information on the work items being performed at each work station, and information on the results of the analysis of work allocation by the analysis unit 15, are stored in the recording device 30 as production line analysis data 35.

[0038] The suggestion unit 16 proposes improvement measures to the user based on the analysis results of the work allocation by the analysis unit 15. For example, the suggestion unit 16 proposes improvement measures for improving the productivity of the production line, such as moving a work item being performed at one of the work stations to another work station, changing the order of multiple work items being performed at one of the work stations, or changing or streamlining the work procedures for the work items, and presents these to the user using the output device 60 to propose improvement measures for the production line. In this way, it is possible to propose to the user measures for improving productivity by realizing a balanced work allocation between the work stations.

[0039] FIG. 4 is a flowchart showing the flow of processing performed by the production line analysis device 1.

[0040] In step S10 , the data acquisition unit 11 acquires the movement sequence data input from the movement recognition device 4 and stores it in the recording device 30 as movement sequence data 31 .

[0041] In step S20, the noise detection unit 12 performs noise detection processing on the operation sequence data 31 acquired in step S10. Details of the noise detection processing in step S20 will be described later with reference to the flowchart in FIG.

[0042] In step S30, the noise correction unit 13 performs noise correction processing on the operation sequence data 31 to reduce the noise detected in the noise detection processing in step S20. The obtained correction result is stored in the recording device 30 as corrected operation sequence data 34. Details of the noise correction processing will be described later with reference to the flowchart in FIG.

[0043] In step S40, the motion estimation unit 14 performs a higher-level motion estimation process for estimating higher-level motions corresponding to the work items performed by the workers at each work station on the production line on the corrected motion sequence data 34 obtained by the noise correction process in step S30. Details of the higher-level motion estimation process will be described later with reference to the flowchart in FIG.

[0044] In step S50, the analysis unit 15 analyzes the work allocation among the work stations based on the high-level motion estimation results obtained by the high-level motion estimation process in step S40. Here, the work allocation can be analyzed by determining the balance between the work stations and determining whether or not there is any unnecessary motion in each work item, for example, using the method described above.

[0045] In step S60, it is determined whether the work allocation to each work station on the production line is appropriate based on the results of the work allocation analysis obtained in step S50. If the work allocation is appropriate, i.e., if it is determined in step S50 that the balance between the work stations is sufficient or that there are almost no unnecessary actions in each work item, the process shown in the flowchart of Figure 4 is terminated. On the other hand, if the work allocation is inappropriate, i.e., if it is determined in step S50 that the balance between the work stations is insufficient or that each work item contains unnecessary actions, it is determined that there is room for improvement in the work allocation to each work station, and the process proceeds to step S70.

[0046] In step S70, the proposing unit 16 proposes an improvement plan for the work allocation at each work station. Here, based on the analysis results of the work allocation obtained in step S50, for example, as described above, an improvement plan is proposed that will balance the work allocation among the work stations and reduce the work time for the entire production line. Note that, for example, computational processing using machine learning can be used to propose this improvement plan.

[0047] In step S80, the suggestion unit 16 suggests the improvement measures obtained in step S70 to the user. Here, the improvement measures can be suggested to the user by, for example, displaying them on a screen on a display included in the output device 60.

[0048] After the process of step S80 is completed, the process shown in the flowchart of FIG. 4 ends.

[0049] FIG. 5 is a flowchart showing the flow of the noise detection process executed by the noise detection unit 12 in step S20 of FIG.

[0050] In step S201, the overall positioning of the action sequence data 31 and the action configuration information 32 is performed. Here, for example, the degree of agreement between the sequence of atomic actions represented by the action sequence data 31 and the sequence of atomic actions corresponding to the work order of each higher-level action represented by the action configuration information 32 is compared while shifting the positions of the atomic actions one by one. This comparison can use, for example, a well-known technique called the Needleman-Wunsch algorithm. As a result, the relative positional relationship between these when the highest degree of agreement is obtained can be identified, allowing the processing of step S201 to be performed.

[0051] In step S202, gaps and mismatches of atomic actions in the action sequence data 31 are detected with respect to the action configuration information 32 based on the results of the overall alignment performed in step S201. Here, mismatches in the content of atomic actions between the action sequence data 31 and the action configuration information 32 after overall alignment are detected as "mismatches," and atomic actions that exist in one but not the other are detected as "gaps." Furthermore, among the gaps, atomic actions that exist in the configuration information 32 but not in the action sequence data 31 are classified as "omissions," and atomic actions that exist in the action sequence data 31 but not in the configuration information 32 are classified as "additions." In this way, mismatches, omissions, and additions of atomic actions in the action sequence data 31 due to erroneous recognition or non-recognition by the action recognition device 4 can be detected as noise based on the order of atomic actions according to the task sequence of each higher-level action represented by the action configuration information 32.

[0052] In step S203, information on the noise (gap / mismatch) detected in step S202 is output as noise information for the operation sequence data 31. This noise information includes information such as the type of noise (gap / mismatch) detected in the operation sequence data 31, its position (the sequential position in the operation sequence data 31 of the atomic operation corresponding to the noise; in the case of a gap, the sequential position in the operation sequence data 31 of the atomic operation following the gap), and the atomic operation string before the noise occurred (N consecutive atomic operations in the sequential position before the noise position, where N is a predetermined natural number). The noise information output from the noise detection unit 12 in step S203 is input to the noise correction unit 13.

[0053] After completing the process of step S203, the noise detection unit 12 ends the noise detection process shown in the flowchart of FIG.

[0054] FIG. 6 is a flowchart showing the flow of the noise correction process executed by the noise correction unit 13 in step S30 of FIG.

[0055] In step S301, any one of the noises included in the operation sequence data 31 is selected based on the noise information input from the noise detection unit 12. Here, the noise represented by the noise information is selected in atomic operation units.

[0056] In step S302, the type of noise selected in step S301 is determined. If the noise type is "addition", the process proceeds to step S303. If the noise type is "mismatch" or "omission", the process proceeds to step S304.

[0057] In step S303, the noise selected in step S301 is removed from the operation sequence data 31. As a result, atomic operations included in the operation sequence data 31 that are not included in the operation configuration information 32 are excluded as unnecessary atomic operations. After performing the processing of step S303, the process proceeds to step S309.

[0058] In step S304, an atomic action sequence that occurs before the noise selected in step S301 is obtained from the action sequence data 31. Here, the atomic action sequence before the noise can be obtained based on information about the atomic action sequence before the occurrence of the noise, which is included in the noise information input from the noise detection unit 12, for example.

[0059] In step S305, a candidate for the correct atomic operation for the noise selected in step S301 is identified. If the noise is "missing," an atomic operation that should actually exist, as indicated by the configuration information 32, is identified as a candidate for the correct atomic operation. If the noise is inconsistent, another atomic operation that may be erroneously recognized as the atomic operation corresponding to the noise is identified as a candidate for the correct atomic operation, for example, based on the sensor characteristic information 33.

[0060] FIG. 8 is a table showing an example of sensor characteristic information 33 for a certain type of sensor included in the non-deterministic sensor 2. The table shown in FIG. 8 is an example of a confusion matrix showing the relationship between the recognition results obtained by the action recognition device 4 from the sensing data of the sensor and the actual atomic actions for three types of atomic actions, "A," "B," and "C." For example, when the recognition result of the atomic action by the action recognition device 4 is "A," the probability that it matches the actual atomic action is 0.9, the probability that "B" is erroneously recognized as "A" is 0.07, and the probability that "C" is erroneously recognized as "A" is 0.03. Similarly, when the recognition result of the atomic action by the action recognition device 4 is "B," the probability that it matches the actual atomic action is 0.85, the probability that "A" is erroneously recognized as "B" is 0.15, and the probability that "C" is erroneously recognized as "B" is 0.05. Furthermore, when the recognition result of the atomic action by the action recognition device 4 is "C," the probability that it matches the actual atomic action is 0.9, the probability that "A" is misrecognized as "C" is 0.05, and the probability that "B" is misrecognized as "C" is 0.1.

[0061] 6, for example, if the atomic action corresponding to noise is "B," then the atomic actions "A" and "C" are identified as candidates for the correct atomic action. Note that the method for identifying candidates for the correct atomic action is not limited to this, and it is possible to identify candidates for the correct atomic action by any method as long as it is possible to identify another atomic action that may result in a recognition result of an atomic action corresponding to noise due to erroneous recognition or recognition omission by the action recognition device 4.

[0062] In step S306, a matching score is calculated for each candidate for the correct atomic action identified in step S305. The calculated matching score is a value representing the degree of likelihood that each candidate for the correct atomic action is an atomic action (correct atomic action) actually performed by the worker. This matching score can be calculated, for example, based on the action configuration information 32, the sensor characteristic information 33, and the pre-noise atomic action sequence acquired in step S304. Specifically, the combination of atomic actions of each higher-level action represented by the action configuration information 32 is compared with the pre-noise atomic action sequence, and the probability that each candidate for the correct atomic action appears after the pre-noise atomic action sequence is calculated. If the noise type is "missing," this value is the matching score. Furthermore, if the noise type is "mismatch," the matching score for each candidate for the correct atomic action can be calculated by multiplying the calculated probability of each candidate by the probability of misrecognition of each candidate, as described above, represented by the sensor characteristic information 33. In addition, when the action recognition device 4 recognizes atomic actions based on sensing data from multiple types of sensors, it is preferable to multiply the probability of erroneous recognition for each sensor represented by the sensor characteristic information 33 by the probability of each candidate obtained from the atomic action sequence before noise.

[0063] In step S307, from among the candidates for the correct atomic action identified in step S305, the one with the highest matching score calculated in step S306 is selected as the correct atomic action.

[0064] In step S308, the noise type is determined. If the noise type is "mismatch", in step S309, the noise selected in step S301 is replaced with the correct atomic action selected in step S307 in the operation sequence data 31. After the process of step S309 is performed, the process proceeds to step S311. If the result of the noise type determination in step S308 is "omission", in step S310, the correct atomic action selected in step S307 is inserted before the noise position. This is because in the case of "omission", the atomic action immediately after the "omission" is set as the noise position. After the process of step 310 is performed, the process proceeds to step S311. Through the above process, the "mismatch" and "omission" in the operation sequence data 31 are corrected.

[0065] In step S311, it is determined whether all noises included in the operation sequence data 31 have been selected in step S301. If there are any unselected noises, the process returns to step S301, and one of the unselected noises is selected in step S301, after which the processes of step S302 and subsequent steps are performed on that noise. As a result, the process of step S303 or steps S304 to S310 is performed on each atomic operation corresponding to the noise, and the noise is either removed from the operation sequence data 31, replaced with a correct atomic operation, or inserted. If it is determined in step S311 that all noises have been selected, the process proceeds to step S312.

[0066] In step S312, the correction results for each noise in the operation sequence data 31 obtained by the processes in steps S301 to S311 are recorded in the memory 20 or the recording device 30 as corrected operation sequence data .

[0067] After completing the process of step S312, the noise correction unit 13 ends the noise correction process shown in the flowchart of FIG.

[0068] FIG. 7 is a flowchart showing the flow of the higher-level motion estimation process executed by the motion estimator 14 in step S40 of FIG.

[0069] In step S401, one of the atomic operations included in the corrected operation sequence data 34 generated by the noise correction unit 13 is selected. Here, the unselected atomic operations are selected in order starting from the one closest to the top of the list.

[0070] In step S402, for the atomic action selected in step S401, a forward segment and a backward segment are respectively set in the corrected action sequence data 34. A forward segment is a combination of atomic actions extracted in order from the selected atomic action to the atomic action that is a predetermined number of steps behind, in accordance with the order of the atomic actions in the corrected action sequence data 34. On the other hand, a backward segment is a combination of atomic actions extracted in order from the selected atomic action to the atomic action that is a predetermined number of steps before, in the reverse direction of the order of the atomic actions in the corrected action sequence data 34. The number of atomic actions included in each of the forward segment and the backward segment can be set arbitrarily, for example, to three.

[0071] In step S403, for the forward and backward segments set in step S402, a similarity score is calculated for each higher-level action defined in the action configuration information 32. Here, for each forward and backward segment, the similarity between the combination of atomic actions that make up these segments and the combination of atomic actions of each higher-level action represented by the action configuration information 32, i.e., the degree of matching of each atomic action, is calculated for each higher-level action.

[0072] In step S404, the higher-level action having the best similarity score calculated in step S403 is selected. Here, the higher-level action having the best score among the similarity scores calculated for each higher-level action for each forward segment and reverse segment, i.e., the higher-level action having the highest degree of matching with the forward segment or the reverse segment, is selected as the higher-level action having the best similarity score.

[0073] In step S405, it is determined whether the difference between the similarity score of the higher-level action selected in step S404, i.e., the best similarity score, and the similarity scores of the other higher-level actions is equal to or greater than a predetermined value. If the difference is equal to or greater than the predetermined value, i.e., if the best similarity score can be considered to be sufficiently higher than the similarity scores of the other higher-level actions, the higher-level action selected in step S404 is estimated to be the higher-level action to which the atomic action selected in step S401 belongs, and the process proceeds to step S409. On the other hand, if the difference is less than the predetermined value, the process proceeds to step S406.

[0074] In step S406, candidates for the higher-level operation to which the atomic operation selected in step S401 belongs are identified based on the similarity score calculated in step S403. Here, for example, the higher-level operation with the best similarity score selected in step S404 and each higher-level operation whose difference in similarity score from that higher-level operation is within a predetermined value are identified as candidates for the higher-level operation to which the selected atomic operation belongs.

[0075] In step S407, for each combination of the candidates identified in step S406 and the previous estimation result, an alignment score is calculated for the operation configuration information 32. Here, for example, for each combination of the candidates for higher-level operations identified in step S406 and the previous estimation result, i.e., the higher-level operation estimated as the one to which the atomic operation selected in the previous step S401 belongs, the similarity of each higher-level operation with respect to the task order represented by the operation configuration information 32 is calculated as an alignment score. This makes it possible to use the previous estimation result of the higher-level operation for a certain atomic operation to estimate the higher-level operation corresponding to another atomic operation.

[0076] In step S408, one of the candidates is selected based on the alignment score of each candidate calculated in step S407. Here, the candidate for the higher-level operation that has the best alignment score is selected, that is, the candidate for the higher-level operation that has the highest degree of match with the operation configuration information 32 obtained for the combination with the previous estimation result. This allows the candidate for the higher-level operation to be estimated as the higher-level operation to which the atomic operation selected in step S401 belongs.

[0077] In step S409, it is determined whether all atomic operations included in the corrected operation sequence data 34 have been selected in step S401. If there are any unselected atomic operations, the process returns to step S401, and one of the unselected atomic operations is selected in step S401. After that, the process from step S402 onwards is performed on that atomic operation. As a result, the processes of steps S402 to S404 are performed on each atomic operation included in the corrected operation sequence data 34, and further the processes of steps S406 to S408 are performed as necessary. In step S404 or S408, higher-level operations corresponding to each atomic operation are estimated, and the boundaries of each higher-level operation in the corrected operation sequence data 34 are estimated. If it is determined in step S409 that all atomic operations have been selected, the process proceeds to step S410.

[0078] In step S 410 , the estimation results of the higher-level movements corresponding to the corrected movement sequence data 34 obtained by the processes in steps S 401 to S 409 are output from the movement estimation unit 14 to the analysis unit 15 .

[0079] After completing the process of step S410, the motion estimator 14 ends the higher-level motion estimation process shown in the flowchart of FIG.

[0080] Next, the planning of improvement measures performed in step S70 of FIG. 4 will be described.

[0081] In step S70, for example, a suggested improvement measure is to move one of the work items at a work station with a relatively long work time to another work station with a shorter work time, based on the work time at each work station calculated by the analysis unit 15. In this case, it is preferable that the work stations before and after the move are closely related to each other, such as being in a chronological order on the production line.

[0082] Furthermore, for work stations with long work times, the system may estimate the required work time. If the actual work time deviates significantly from this estimate, the system may determine that the current work procedure is wasteful and propose improvements to improve work efficiency. For example, multiple atomic actions belonging to higher-level actions corresponding to the work items of the work station may be classified into core actions that form the core of the work item and non-core actions that are not core actions. For example, core actions include screw tightening, while non-core actions include walking and gripping a tool. The system then obtains the time required for each core action and the time required for each non-core action from the execution time of each atomic action included in the corrected action sequence data 34, and compares these values. If the time required for the non-core actions is longer than the time required for the core actions, the system may determine that the work efficiency is low and propose improvements such as deleting or shortening some of the atomic actions included in the non-core actions. Specifically, improvements that can be proposed to improve work efficiency include, for example, changing the layout of the work station, changing the arrangement of tools used by workers at the work station, and changing the arrangement of the area where products in progress at the work station and parts to be assembled thereto are prepared.

[0083] Furthermore, improvements to improve work efficiency may be proposed, such as changing the order of multiple work items performed at a work station, or changing the order of execution of individual work items, i.e., changing the order of multiple atomic actions belonging to higher-level actions corresponding to each work item, etc. In addition to these, any measure can be proposed as an improvement measure as long as it can shorten the work time at a work station.

[0084] According to the embodiment of the present invention described above, the following advantageous effects can be obtained.

[0085] (1) The information processing system 100 includes a production line analysis device 1. The production line analysis device 1 includes a data acquisition unit 11 that acquires action sequence data 31 representing a series of actions performed by a worker on a production line having multiple work stations, a noise detection unit 12 that detects noise from the action sequence data 31 based on action configuration information 32 that represents the relationship between atomic actions performed by the worker and higher-level actions consisting of combinations of the multiple atomic actions, a noise correction unit 13 that corrects the action sequence data 31 to reduce noise, and an action estimation unit 14 that estimates higher-level actions corresponding to work items performed by the worker at each work station on the production line based on the action sequence data 34 corrected by the noise correction unit 13. This configuration allows the higher-level actions to be accurately estimated from the recognition results of a series of atomic actions performed by the worker.

[0086] (2) The movement estimation unit 14 sets segments in the corrected movement sequence data 34, each combining a predetermined number of atomic movements, in the forward and reverse directions (step S402), and estimates boundaries of higher-level movements based on these segments (steps S404 and S408). This allows for highly accurate estimation of higher-level movements from the corrected movement sequence data 34.

[0087] (3) The movement estimation unit 14 identifies multiple candidates for higher-level movements for the sequence of atomic movements in the corrected movement sequence data 34 (step S406), and calculates an alignment score for the movement configuration information 32 for each combination of the identified candidate and the estimation result of the previous higher-level movement (step S407). Then, by selecting one of the multiple candidates based on the calculated alignment score, the higher-level movement corresponding to the work item is estimated (step S408). As a result of this, even if there are multiple higher-level movements in the movement configuration information 32 that are highly similar to the sequence of atomic movements included in the corrected movement sequence data 34, any one of them can be appropriately selected as the estimation result of the higher-level movement corresponding to the work item.

[0088] (4) The movement sequence data 31 represents a sequence of multiple atomic movements each recognized by the movement recognition device 4 based on sensor data acquired by a sensor that detects the worker's movements. Furthermore, the noise detected by the noise detection unit 12 is a discrepancy or gap in the recognition result of the atomic movements by the movement recognition device 4 based on the sensor data, relative to the movement configuration information 32. The noise correction unit 13 corrects the movement sequence data 31 by correcting the noise using preset sensor characteristic information 33. In this way, even if the movement sequence data 31 contains discrepancies or gaps caused by erroneous or non-recognition errors by the movement recognition device 4 as noise, the movement sequence data 31 can be corrected to reduce this noise.

[0089] (5) The production line analysis device 1 includes an analysis unit 15 that analyzes the start time and / or end time of work items corresponding to higher-level actions estimated by the action estimation unit 14 for each work station on the production line, and a proposal unit 16 that proposes improvement measures for the production line using the start time and / or end time for each work item at each work station obtained by the analysis unit 15. As a result, if there is room for improvement in the work allocation at each work station on the production line, it is possible to propose improvement measures to the user to improve productivity.

[0090] (6) The analysis unit 15 can classify multiple atomic actions included in the higher-level actions estimated by the action estimation unit 14 into core actions that form the core of the work items performed by the worker at the work station and non-core actions other than the core actions. In this case, the suggestion unit 16 can compare a first time required for the core action with a second time required for the non-core action and propose an improvement measure based on the comparison result. Specifically, for example, the core action includes screw tightening, and the non-core action includes walking and / or gripping a tool. In this way, it is possible to propose an appropriate improvement measure to improve work efficiency for a work station where the current work procedure is wasteful.

[0091] (7) The proposing unit 16 can also propose, as an improvement measure, a change to at least one of the order of atomic operations in the higher-level operations estimated by the operation estimating unit 14, the order of operations performed at the same work station, and the work station at which the work item is performed. In this way, it is possible to propose an optimal improvement measure to eliminate imbalances in the production line and improve productivity.

[0092] (8) The proposing unit 16 can also propose, as an improvement measure, a change to at least one of the layout of a plurality of work stations in the production line, the arrangement of tools used by workers at the work stations, and the arrangement of products or parts at the work stations. In this way, it is possible to propose an improvement measure appropriate for shortening the work time of a work station that takes longer than other work stations.

[0093] (9) The action sequence data 31 represents a sequence of multiple atomic actions each recognized based on sensor data acquired by a sensor that detects the worker's actions while the worker is performing a task. The sensor may include, for example, at least one of an inertial measurement unit, a sound sensor, a pressure sensor, an image sensor, a distance sensor, and a wearable sensor as the non-deterministic sensor 2. This makes it possible to easily and reliably acquire the sensor data necessary to recognize the worker's atomic actions.

[0094] It goes without saying that the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above-described embodiments with other configurations.

[0095] Furthermore, the above-described configurations, functional units, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card, an SD card, or a DVD.

[0096] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. For example, it can be considered that almost all components are actually connected to each other.

[0097] 1: Production line analysis device 2: Non-deterministic sensor 3: Deterministic sensor 4: Action recognition device 10: Processor 11: Data acquisition unit 12: Noise detection unit 13: Noise correction unit 14: Action estimation unit 15: Analysis unit 16: Proposal unit 20: Memory 30: Recording device 31: Action sequence data 32: Action configuration information 33: Sensor characteristic information 34: Action sequence data after correction 35: Production line analysis data 40: Input / output interface 50: Input device 60: Output device 100: Information processing system

Claims

1. An information processing system comprising: a data acquisition unit that acquires action sequence data representing a series of actions performed by a worker on a production line having a plurality of work stations; a noise detection unit that detects noise from the action sequence data based on action configuration information that represents the relationship between atomic actions performed by the worker and higher-level actions consisting of combinations of a plurality of the atomic actions; a noise correction unit that corrects the action sequence data so as to reduce the noise; and an action estimation unit that estimates the higher-level actions corresponding to the work items performed by the worker at each work station on the production line based on the action sequence data after correction by the noise correction unit.

2. An information processing system according to claim 1, wherein the movement estimation unit sets segments in the corrected movement sequence data, each of which combines a predetermined number of atomic movements, in both the forward and reverse directions, and estimates the division of the higher-level movements based on the segments.

3. An information processing system according to claim 1, wherein the operation estimation unit identifies multiple candidates for the higher-level operation for the sequence of atomic operations in the corrected operation sequence data, calculates an alignment score for the operation configuration information for the combination of each identified candidate with the estimation result of the previous higher-level operation, and selects one of the multiple candidates based on the alignment score, thereby estimating the higher-level operation corresponding to the work item.

4. An information processing system according to claim 1, wherein the operation sequence data represents a sequence of a plurality of atomic operations each recognized based on sensor data acquired by a sensor that detects the worker's operations, the noise is a discrepancy or gap in the recognition results of the atomic operations based on the sensor data with respect to the operation configuration information, and the noise correction unit corrects the noise using pre-set characteristic information of the sensor, thereby correcting the operation sequence data.

5. An information processing system according to claim 1, comprising: an analysis unit that analyzes the start time and / or end time of the work item corresponding to the higher-level action estimated by the action estimation unit for each work station on the production line; and a proposal unit that uses the start time and / or end time for each work item at each work station to propose improvements to the production line.

6. An information processing system according to claim 5, wherein the analysis unit classifies the plurality of atomic operations included in the higher-level operations into core operations that form the core of the work items performed by the worker at the work station, and non-core operations other than the core operations, and the proposal unit compares a first time required for the core operations with a second time required for the non-core operations, and proposes the improvement measures based on the comparison results.

7. An information processing system according to claim 6, wherein the core motion includes screwing, and the non-core motion includes walking and / or gripping a tool.

8. An information processing system according to claim 5, wherein the suggestion unit proposes, as the improvement measure, a change to at least one of the order of the atomic operations in the higher-level operation, the order of execution of multiple work items performed at the same work station, and the work station on which the work items are performed.

9. An information processing system as described in claim 5, wherein the proposal unit proposes, as the improvement measure, a change to at least one of the layout of the plurality of work stations in the production line, the arrangement of tools used by the workers at the work stations, and the arrangement of products or parts at the work stations.

10. An information processing system according to claim 1, wherein the action sequence data represents a sequence of a plurality of the atomic actions each recognized based on sensor data acquired by a sensor that detects the actions of the worker while the worker is performing the work item, and the sensor includes at least one of an inertial measurement unit, a sound sensor, a pressure sensor, an image sensor, a distance sensor, and a wearable sensor.

11. An information processing method in which a computer acquires action sequence data representing a series of actions performed by a worker on a production line having a plurality of work stations, detects noise from the action sequence data based on action configuration information representing the relationship between atomic actions performed by the worker and higher-level actions consisting of combinations of a plurality of the atomic actions, corrects the action sequence data so as to reduce the noise, and estimates the higher-level actions corresponding to the work performed by the worker at each work station on the production line based on the corrected action sequence data.

12. An information processing method according to claim 11, wherein segments each combining a predetermined number of atomic actions are set in the corrected action sequence data in the forward and reverse directions, and the division of the higher-level actions is estimated based on the segments.

13. An information processing method according to claim 11, wherein a plurality of candidates for the higher-level operations are identified for the sequence of atomic operations in the corrected operation sequence data, an alignment score for the operation configuration information is calculated for each combination of the identified candidate and the estimation result of the previous higher-level operation, and one of the plurality of candidates is selected based on the alignment score, thereby estimating the higher-level operation corresponding to the work item.

14. An information processing method according to claim 11, wherein the action sequence data represents a sequence of a plurality of the atomic actions each recognized based on sensor data acquired by a sensor that detects the worker's actions, the noise is a discrepancy or gap in the recognition results of the atomic actions based on the sensor data with respect to the action configuration information, and the information processing method corrects the action sequence data by correcting the noise using predetermined characteristic information of the sensor.

15. An information processing method as claimed in claim 11, comprising: analyzing the start time and / or end time of the work item corresponding to the estimated higher-level operation for each work station on the production line; and using the start time and / or end time for each work item at each work station to propose improvement measures for the production line.

16. An information processing method as claimed in claim 15, wherein the atomic operations included in the higher-level operations are classified into core operations that form the core of the work item performed by the worker at the work station and non-core operations other than the core operations, a first time required for the core operations is compared with a second time required for the non-core operations, and the improvement measures are proposed based on the comparison results.

17. An information processing method according to claim 16, wherein the core motion includes screwing, and the non-core motion includes walking and / or gripping a tool.

18. An information processing method according to claim 16, wherein the improvement measure is a change to at least one of the order of the atomic operations in the higher-level operation, the order of execution of multiple work items performed at the same work station, and the work station on which the work items are performed.

19. An information processing method as defined in claim 15, wherein the improvement measure is a change to at least one of the layout of the plurality of work stations in the production line, the arrangement of tools used by the workers at the work stations, and the arrangement of products or parts at the work stations.

20. An information processing method according to claim 11, wherein the action sequence data represents a sequence of a plurality of the atomic actions each recognized based on sensor data acquired by a sensor that detects the actions of the worker while the worker is performing the work item, and the sensor includes at least one of an inertial measurement unit, a sound sensor, a pressure sensor, an image sensor, a distance sensor, and a wearable sensor.

Images