Information processing apparatus, program, and detection method

Abstract

To provide an information processing apparatus for efficiently indicating, to a beginner and a person with less work experience, the relationship of the content of their work operations with the work operations of a skilled worker, a program, and a detection method.SOLUTION: An information processing apparatus related to an embodiment can detect whether there is a temporal difference between a first operation and a second operation different from the first operation, on the basis of the presence or absence of a frame included in only one of first time-series data related to the first operation and second time-series data related to the second operation, which are associated for each time-series frame.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present embodiment relates to an information processing device, a program, and a detection method. [Background technology]

[0002] Recently, there has been a growing interest in technology that analyzes worker work using camera-based image data and skeletal position information obtained by human posture detectors and motion sensors. There is a growing demand for technology that utilizes this work analysis to efficiently show, for example, beginners and those with little work experience the relationship between their own work movements and the work movements of more experienced workers through images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-77218 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the present invention aims to solve is to provide an information processing device, program, and detection method for efficiently showing, for example, to a beginner or a person with little work experience, the relationship between the content of their own work actions and the work actions of a skilled worker. [Means for solving the problem]

[0005] The information processing device according to the embodiment can detect whether there is a time difference between a first action and a second action based on the presence or absence of a frame that is included only in either first time series data relating to the first action or second time series data relating to a second action different from the first action, which are associated on a time-series frame-by-frame basis. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a schematic diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Figure 2] 1 is a schematic diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Figure 3] 1 is a schematic diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of joint position extraction according to the embodiment. [Figure 5] 1 is an example of an association graph obtained by the DTW method. [Figure 6] 1 is an example of an association graph obtained by the DTW method. [Figure 7] 3 is an example of a display unit according to the first embodiment. [Figure 8] FIG. 10 is an example of a schematic diagram of an information processing system according to a first modified example. [Figure 9] FIG. 10 is an example of a schematic diagram of an information processing system according to a second modified example. [Figure 10] 10 is an example of an association graph represented as a line graph. [Figure 11] 10 is an example of a display image on a display unit in the second embodiment. [Figure 12] 10 is an example of a display image on a display unit in the third embodiment. [Figure 13] 13 is an example of an association graph according to the fourth embodiment. [Figure 14] 13 is an example of an association graph according to the fourth embodiment. [Figure 15] FIG. 20 is an example of a schematic diagram of an information processing system according to a seventh embodiment. [Figure 16] FIG. 20 is an example of a schematic diagram of an information processing system according to an eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments will be described with reference to the accompanying drawings. In each embodiment, substantially identical components are denoted by the same reference numerals, and some of their descriptions may be omitted. The drawings are schematic, and the relationship between the thickness of each component and the planar dimensions, the thickness ratio of each component, etc. may differ from the actual ones.

[0008] (First embodiment) 1 is a schematic diagram showing an example of the configuration of an information processing system 10 according to this embodiment. The information processing system includes an information acquisition terminal 200 and an information processing device 100. The information acquisition terminal 200 and the information processing device 100 are connected to each other via a network so that they can communicate with each other. The network may be wired or wireless, and may be any of a variety of communication networks such as the Internet.

[0009] The information acquisition terminal 200 is a terminal that acquires the movements of the worker P (including P1 and P2). The information acquisition terminal 200 may be a mobile terminal such as a smartphone or tablet PC, a stationary terminal, or a terminal installed on the premises of a factory or business. The information acquisition terminal 200 has a camera function for capturing the movements of the worker P, or is connected to a camera device to capture the movements of the worker P. Alternatively, the information acquisition terminal 200 has a sensor function for detecting the movements of the worker P, and is connected to a sensor device to detect the movements of the worker P. Note that the range of movement capture and detection is not limited to the entire body of the worker P, but may be partial, such as the upper body or only the hands. Furthermore, by capturing the same movement of the worker P from different angles using multiple information acquisition terminals 200, it is possible to analyze the differences in the movements in more detail and provide correction instructions. Note that the information acquisition terminal 200 does not need to be a terminal owned by the worker P. For example, it may be a terminal owned by a service provider that provides a skill acquisition service. The information acquisition terminal 200 is capable of photographing and / or detecting the actions of a worker P who is accustomed to the work (hereinafter referred to as "expert worker P1") and a worker P who is not accustomed to the work (hereinafter referred to as "beginner P2"). Hereinafter, photographing the actions of the worker P by a camera and detecting them by a sensor will be referred to as "photographing actions." In the information processing system, the information acquisition terminal 200 that photographs the expert worker P1 and the terminal that photographs the beginner P2 may be the same terminal, or, as shown in FIG. 2, the information processing system may include a plurality of information acquisition terminals 200, and the information acquisition terminal 200 that photographs the expert worker P1 and the information acquisition terminal 200 that photographs the beginner P2 may be different terminals.

[0010] The information acquisition terminal 200 includes a communication unit 210 for communicating with the information processing device 100. The communication unit 210 is realized by, for example, a network interface card (NIC).

[0011] The information processing device 100 analyzes the differences in work between the skilled worker P1 and the beginner P2 based on the actions of the workers acquired by the information acquisition terminal 200.

[0012] 3 is a schematic diagram illustrating an example of the configuration of an information processing device 100 according to this embodiment. The information processing device 100 is capable of detecting whether there is a time difference between a first action and a second action based on the presence or absence of a frame included only in either first time-series data relating to a first action or second time-series data relating to a second action different from the first action, which are associated on a time-series frame-by-frame basis. Specifically, the information processing device 100 includes an association unit 132 that associates frames included in data including a time series recording a first action (here, a sample action) of a first subject (here, an expert P1) with frames included in data including a time series recording a second action (here, a target action) of a second subject (here, a beginner P2), and a detection unit 133 that detects a difference in progress between the first action and the second action based on the presence or absence of a frame included only in either the data including a time series recording the first action or the data including a time series recording the second action. The difference in progress may occur, for example, when one worker skips a task step or when a second worker falls behind because he or she is unfamiliar with the task.The model task (first action) is the action that beginner P2 (second action subject) uses as a model when performing the target task (second action).

[0013] Here, a frame refers to one still image that constitutes a video recording the movements of an expert P1 or a beginner P2. However, the video here is not limited to the original raw data that was shot, but also includes processed videos and a series of data extracted from the video, such as the skeletal coordinates of the subject.

[0014] The information processing device 100 is, for example, a server device that executes task variance analysis. The information processing device 100 includes a communication unit 110, a storage unit 120, a control unit 130, and a display unit 140.

[0015] The communication unit 110 is realized by, for example, a network interface card (NIC). The information processing device 100 is connected to the information acquisition terminal 200 by wire or wirelessly, and transmits and receives information to and from the information acquisition terminal 200 via the communication unit 110. The information transmitted and received here includes information about the movements of the worker P (including an experienced worker P1 and a beginner P2) acquired by the information acquisition terminal 200, information about the control of the information acquisition terminal 200, and the like. The information about the movements of the worker P includes a video showing a series of movements of the worker P, data including a series of movements of the worker P captured by the information acquisition terminal 200 as a sensor, and the like. The information about the control of the information acquisition terminal 200 includes control information about ON / OFF of the information acquisition terminal 200 and control information about parameters such as the video frame rate, image quality, and brightness when recording.

[0016] The memory unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The memory unit 120 stores time-series data of the posture coordinates of the skilled worker P1 included in the time-series data recording the sample task, time-series data of the posture coordinates of the beginner P2 included in the time-series data recording the target task, various programs to be executed by the control unit 130, and various data output by the control unit 130. Here, "posture coordinates" refers to data that digitizes the posture of the worker P and indicates it as coordinates. As shown in FIG. 3, the memory unit 120 includes a sample task memory unit 121, a target task memory unit 122, an association memory unit 123, a detection memory unit 124, and a program memory unit 125. Each memory unit included in the memory unit 120 will be described below in order.

[0017] The sample task memory unit 121 stores the target task as video data acquired by the information acquisition terminal 200 or data along a time series of posture coordinates representing postures. The target task is, for example, the movements of the skilled person P1. The data stored in the sample task memory unit 121 is, for example, video data capturing the movements of the skilled person P1. Alternatively, the data stored in the sample task memory unit 121 may be time series data of posture coordinates representing the position coordinates and rotation direction of each part of the skilled person P1's body, acquired by the estimation unit 131 through skeletal estimation of the skilled person P1 from the video data. This allows the size of the data to be smaller than that of video data, thereby reducing the consumption of various resources.

[0018] The target task memory unit 122 stores the target task as video data or time-series data of posture coordinates representing postures acquired by the information acquisition terminal 200. The target task is, for example, the movements of the beginner P2. The data stored in the target task memory unit 122 may be, for example, video data capturing the movements of the beginner P2. Alternatively, the data stored in the target task memory unit 122 may be time-series data of posture coordinates representing the position coordinates and rotation direction of each part of the beginner P2's body, acquired by the estimation unit 131 through skeletal estimation of the beginner P2 from the video data, similar to the data stored in the sample task memory unit 121. This allows for a smaller data size to be handled compared to video data, thereby reducing the consumption of various resources. The data stored in the sample task memory unit 121 and the data stored in the target task memory unit 122 may be in the same format or different formats.

[0019] The user can specify whether to store the video data acquired by the information acquisition terminal 200 and the posture coordinates estimated by the estimation unit 131 in the sample task memory unit 121 or the target task memory unit 122. For example, the sample task memory unit 121 can store a worker who has worked for a certain task for a shorter time than the target worker, thereby improving the work of the target worker. Alternatively, the sample task memory unit 121 may be determined based on attribute data of a worker P (including an experienced worker P1 and a beginner P2) input in advance. In this case, the attribute data of the worker P, such as age, years of service, qualifications held, and data scoring the work of the worker P, is stored in the memory unit 120. Here, "scoring" a task refers to quantifying the feature quantities of the worker P's movements. The feature quantities include vectors output by a machine learning model that inputs joint positions and image information. Examples of machine learning models include a convolutional neural network (CNN) and a graph convolutional network (GCN), and the output vectors can be expressed in high dimensions, such as 512 dimensions. These attributes include coordinates representing the positions of joints, angles of tilting the head or bending the elbow, the time the hands are placed in a specific position, the speed of eye movement, and the direction of the body. Since each of the position coordinates, angle, time, and speed is a quantifiable feature, representative values ​​such as the mean, median, and mode can be calculated. If these attribute data exceed a predetermined evaluation value, the worker P is treated as an expert P1, and the video data and posture coordinates of the worker P are stored in the sample task memory unit 121. If the attribute data are equal to or less than the predetermined evaluation value, the worker P is treated as a beginner P2, and the video data and posture coordinates of the worker P are stored in the target task memory unit 122. Alternatively, data of a worker P who completed a task faster than the predetermined time may be stored in the sample task memory unit 121, and data of a worker P who took longer than the predetermined time may be stored in the target task memory unit 122. The reference values ​​for these attribute data and task times may be preset fixed values, values ​​that change based on the performance of the worker P, or values ​​arbitrarily set by the user of the information processing system 10.

[0020] The association storage unit 123 stores the results output by the association unit 132, which associates the sample work with the target work for each motion. Here, "motion" refers to a simple action included in the sample work and the target work, such as raising a hand, changing the direction of the body, pausing a motion, etc. The data stored in the association storage unit 123 is in a graph format, such as the association graph obtained by the association unit 132 (described later using FIG. 5). Alternatively, the data may be in a table format containing equivalent data, or in another format.

[0021] The detection storage unit 124 stores data output by the detection unit 133 regarding the time difference of the target work detected.

[0022] The program storage unit 125 stores various programs executed by the control unit 130.

[0023] The control unit 130 is a processing unit that controls the entire information processing device, and is, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) (so-called processor). The control unit 130 loads various programs (for example, the information processing program according to the present application) stored in the storage unit 120 into RAM, which serves as a working area, and executes the programs. The control unit 130 is realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0024] The control unit 130 has an estimation unit 131, a correlation unit 132, and a detection unit 133, and realizes or executes each function and action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 3, and may have any other configuration as long as it is configured to execute the process of analyzing the difference between the actions of the skilled person P1 and the actions of the beginner P2, which will be described later. Furthermore, the connection relationship between the processing units included in the control unit 130 is not limited to the connection relationship shown in Fig. 3, and may be other connection relationships.

[0025] The estimation unit 131 estimates the posture coordinates of the skilled worker P1 from time-series data recording the sample task, and estimates the posture coordinates of the beginner P2 from time-series data recording the target task. The estimated posture coordinates are stored in the storage unit 120. The estimation unit 131 executes an estimation program stored in the program storage unit 125 to identify the positions of each body part and joint of the skilled worker P1 and the beginner P2 through image recognition and image processing from videos capturing the movements of the skilled worker P1 and the beginner P2, and estimates the posture coordinates required for subsequent processing. FIG. 4 is a diagram illustrating an example of joint position extraction according to an embodiment. The positions of each body feature point, such as joints (e.g., elbows and knees), head, and palms, are represented by diamonds, and these feature points are connected by straight lines. The estimation unit 131 can estimate the skeleton of the worker P as shown in the figure from videos captured of the worker P. At this time, the body parts and joint positions estimated for the skilled person P1 and the beginner P2 are of the same type. The estimation unit 131 does not have to focus on the entire body of the skilled person P1 and the beginner P2, but may focus on a part of the body. For example, it may focus on the fine movements of the fingers or the movement of the eyeballs (gaze).

[0026] The association unit 132 uses dynamic programming to associate, for each motion, frames included in the time-series data recording the sample task with frames included in the time-series data recording the target task. By executing an association program stored in the program storage unit 125, the association unit 132 associates, for each motion, the sample task stored in the sample task memory unit 121 with the time-series data of posture coordinates of the target task stored in the target task memory unit 122. The association unit 132 outputs a graph showing the correspondence for each motion between the frames included in the time-series data recording the sample task and the frames included in the time-series data recording the target task. FIG. 5 shows an example of an association graph obtained by the DTW (Dynamic Time Warping) method of dynamic programming used for time association. In the association graph shown in FIG. 5, the vertical axis represents the progress of the sample task by the expert P1 along the timeline of the sample task, and the horizontal axis represents the progress of the target task by the beginner P2 along the timeline of the target task. The black dots on the correspondence graph indicate the corresponding points between the actions of each task. The correspondence graph shown in Figure 5(a) is for the case where beginner P2 has improved and is able to work at the same pace as expert P1. Because beginner P2's movements are not lagging behind expert P1, the progress of expert P1 on the sample task and beginner P2 on the target task match at each point in time. On the other hand, the correspondence graph shown in Figure 5(b) is for the case where beginner P2 is lagging behind expert P1. Areas with horizontally aligned black dots, such as the third to sixth columns from the left, indicate that beginner P2's target task takes longer than expert P1's sample task. This means that beginner P2 is making unnecessary motions during this time. The illustrated correspondence graph is a discrete graph using dots, but it can also be represented as a continuous line graph, curve, or other graph.

[0027] The detection unit 133 uses the association graph obtained by the association unit 132 to detect where the target task is delayed relative to the sample task. The detection unit 133 scans a fixed window in the graph and compares the value indicating the slope of the graph within the fixed window with a threshold to detect where a time difference occurs. In particular, the detection unit 133 detects where the slope of the graph within the fixed window is below a certain level as where the second action is delayed relative to the first action. Specifically, as shown in Figures 5(b) and 6, among the points where black dots are lined up horizontally, delays of a certain level or more are detected as delays by beginner P2. Even if a momentary delay occurs, if the delay is below the threshold, it is not considered to be a delay by beginner P2. The threshold for detecting a delay by beginner P2 may be the number of black dots lined up horizontally or may be based on time. In Figure 6, points detected as delays are shown in gray when four or more consecutive black dots lined up horizontally are considered to be a delay by beginner P2. In Figure 6, the areas where six black dots are lined up horizontally, such as the third to eighth columns from the left, are detected as "delays," but the areas where three black dots are lined up horizontally, such as the eleventh to thirteenth columns from the left, are not detected as delays by beginner P2. The threshold for whether or not to detect a delay may be set in advance, or the user may be able to set an arbitrary threshold.

[0028] The display processing unit 134 executes a process of simultaneously displaying on the display unit 140 frames included in the time-series data recording the sample task and frames included in the time-series data recording the target task that are associated with the frames by the association unit 132. The display processing unit 134 determines the information to be displayed on the display unit 140 using the delay results detected by the detection unit 133. Based on the association graph and the results of the detection unit 133, the display processing unit 134 adjusts the playback speed of the sample task video and the target task video so that the display unit 140 simultaneously displays the same tasks. When displaying frames included only in either the time-series data recording the sample task or the time-series data recording the target task, the display processing unit 134 executes a process of stopping the other frame and displaying it on the display unit 140. Specifically, when the black circles are side-by-side in the association graph, i.e., while the beginner P2 is making unnecessary movements, the playback speed of the two videos is determined so that the movements of the expert P1 are stopped. When the work of the beginner P2 catches up with the work of the expert P1, the pause of the sample work video is released.

[0029] In addition, the display processing unit 134 may execute a process to display on the display unit 140 the correspondence between each motion between the frames included in the time series data recording the sample work and the frames included in the time series data recording the target work.

[0030] The display unit 140 provides the user with feedback on information about temporal task differences, for example, by using the association graph obtained by the association unit 132 and the results of the delay detected by the detection unit 133. The display unit 140 displays information about temporal differences based on processing by the display processing unit 134. The display unit 140 is a display capable of displaying images. The display unit 140 may have buttons or the like for operation around it, or may have a touch panel function that allows the user to operate it by touching the screen.

[0031] FIG. 7 illustrates an example of the display unit 140 in this embodiment, where the display unit 140 is a display screen and displays the analysis results of the work differences as a video. A specific example is shown. Note that FIG. 7(a) illustrates an example of the display content of the display unit 140 displaying information when the skilled worker P1 and the beginner P2 are performing the same work, and FIG. 7(b) illustrates an example of the display content of the display unit 140 displaying information when the beginner P2 is lagging behind the skilled worker P1. In this embodiment, the display unit 140 displays a sample work video and a target work video. The display unit 140 displays a seek bar and a seek bar below the sample work video and the target work video, respectively. The user can operate the seek bar to play the sample work video and the target work video from any point. In FIG. 7, the black circle cursor indicates the portion of the video currently being played that corresponds to each motion. Furthermore, to indicate the time period in which the delay of the beginner P2 was detected, the seek bar may display a delay indicator at the point corresponding to that time period. Specifically, the seek bar portion corresponding to the time period when beginner P2 was late can be thickened or colored. In FIG. 7, the seek bar portion corresponding to the time period when beginner P2 was late is displayed in bold. Also, in FIG. 7(b), the frame of the image is highlighted while displaying information about when beginner P2 is behind the expert P1. The information displayed on the display unit 140 is not limited to these images, and may also display a correlation graph or text including the number of seconds, number of times, and the delayed task content. The display unit 140 may be a screen that displays the above-mentioned images, or may be a lamp that indicates the task difference with light or a sound source that indicates the task difference with sound. If the display unit 140 is a lamp or sound source, it may emit a light or sound when a delay in beginner P2's task is detected.

[0032] In the above embodiment, each piece of data is transmitted and received over a network, and each part of the data is processed on an edge terminal or on the cloud.

[0033] In the above, an example has been described in which the information processing device 100 includes the storage unit 120 and the display unit 140 in addition to the association unit and the detection unit 133, but these do not necessarily have to be included in the information processing device 100. For example, the storage unit 120 may be stored in a server external to the information processing device 100, and necessary data may be transmitted and received to and from the external server via the communication unit 110 included in the information processing device 100. Similarly, the display unit 140 may be provided external to the information processing device 100, and the information processing device 100 may transmit image data to be displayed on the display unit 140 or instruct the display unit 140 on the timing of emitting light or sound via the communication unit 110 included in the information processing device 100 and a communication device (not shown) included in the display unit 140.

[0034] The information processing system 10 of this embodiment can analyze the temporal differences in work between the skilled person P1 and the beginner P2 based on the time-series data of the posture coordinates of the two people, and provide feedback to the user.

[0035] (First Modification) Fig. 8 is an example of a schematic diagram of the information processing system 10 in this modification. In Fig. 3, the estimation unit 131 has been described as being included in the information processing device 100, but as shown in Fig. 8, the estimation unit 131 may be included in the information acquisition terminal 200. By estimating the posture coordinates of the expert P1 and the beginner P2 from images captured by a camera in the information acquisition terminal 200 and transmitting the results from the communication unit 210 included in the information acquisition terminal 200 to the communication unit 110 included in the information processing device 100, it is possible to reduce the amount of data exchanged between the communication units compared to when data of the captured video is transmitted as is.

[0036] (Second Modification) Fig. 9 is an example of a schematic diagram of the information processing system 10 in this modified example. Although Fig. 3 illustrates a case where the sample task memory unit 121 and the target task memory unit 122 are different, as shown in Fig. 9, the video data and posture coordinates of the worker P may be stored collectively in the task memory unit 20 without separating the sample task memory unit 121 and the target task memory unit 122.

[0037] In this modified example, the user can arbitrarily select who among multiple workers P is to be the skilled worker P1, or the skilled worker P1 may be determined automatically based on the attribute data of the worker P. In this case, the attribute data of the worker P, such as age, years of service, qualifications held, and data that scores the work of the worker P, are stored in the storage unit 120. The attribute data of two people whose work differences are compared is compared, and for example, the older person is determined to be the skilled worker P1, and this can be used for subsequent processing in the association unit 132, detection unit 133, and display unit 140.

[0038] (Third Modification) FIG. 10 is an association graph represented by a line graph. When a line graph is used as the association graph in this way, the delay in the target task of beginner P2 may be determined based on the slope of the graph. Specifically, beginner P2 is determined to be behind in the target task in a section where the slope of the graph is below a predetermined threshold. This threshold may be set in advance or may be set by the user. Furthermore, beginner P2's target task is determined to be behind not only when the slope of the graph is "less than" the threshold, but also when the slope of the graph is "equal to or less than" the threshold. In FIG. 10, the sections where beginner P2 is determined to be behind are shown in gray.

[0039] (Second embodiment) In this embodiment, the same aspects as in the first embodiment will be omitted, and only the differences from the first embodiment will be described. Also, the same names and symbols as in the first embodiment will be used in the description.

[0040] FIG. 11 is an example of a display image on the display unit 140 in this embodiment. In this embodiment, instead of videos of the skilled worker P1 and the beginner P2, the display unit 140 displays an image of the beginner P2 when he / she was behind in the target task. The display unit 140 displays a still image selected from frames included only in either the time-series data recording the sample task or the time-series data recording the target task. The image displayed here is a representative frame. The representative frame may be a frame located in the center of the frame corresponding to the section where the beginner P2 is behind detected by the detection unit 133, or a frame when the beginner P2 begins to fall behind, and is selected by the display processing unit 134. Which frame of the frames constituting the video of the beginner P2 when he / she is behind may be set in advance or may be set and changeable by the user. Note that the number of representative frames is not limited to one, and multiple frames may be selected. FIG. 11 shows a case where three images are displayed on the display unit 140 as representative frames. In the figure, the frame located in the middle of the frames corresponding to the section where beginner P2 is delayed is displayed in the center of the screen as a representative frame, with the frame one second earlier displayed on the left and the frame one second later than the representative frame displayed on the right. This allows beginner P2 to see what he did in the one second before and after the task that took him so long, and clearly understand the cause of the increase in the required time. By displaying only the representative frame selected by the display processing unit 134 on the display unit 140 instead of displaying a series of moving images, the time required to provide feedback to the user about beginner P2's delay can be shortened.

[0041] (Third embodiment) In this embodiment, the same aspects as the first and second embodiments will be omitted, and only the differences from the first and second embodiments will be described. Also, the same names and symbols as those of the first and second embodiments will be used in the description.

[0042] FIG. 12 is an example of a display image on the display unit 140 in this embodiment. In this embodiment, instead of a video showing the skilled person P1 and the beginner P2, the display unit 140 displays a schematic diagram connecting the joint positions of the skilled person P1 and the beginner P2 with lines, or an avatar. The display unit 140 displays a sample task using only the skeleton of the skilled person P1, and a target task using only the skeleton of the beginner P2. The display processing unit 134 can generate avatars for the skilled person P1 and the beginner P2 based on the posture coordinates and joint positions estimated by the estimation unit 131. Alternatively, as shown in FIG. 12, a rough schematic diagram of the skilled person P1 and the beginner P2, which is simpler than an avatar and consists of points and lines representing joint positions, may be used. By displaying an avatar on the display unit 140 instead of a video showing the skilled person P1 and the beginner P2, the user can receive feedback on the beginner P2's delay while protecting the privacy of the skilled person P1 and the beginner P2.

[0043] (Fourth embodiment) In this embodiment, the same aspects as the first to third embodiments will be omitted, and only the differences from the first to third embodiments will be described. The same names and symbols will be used for the parts that are common to the first to third embodiments.

[0044] In this embodiment, the association unit 132 breaks down the sample work and the target work into a first section, which corresponds to the work that comes earlier in time, and a second section, which corresponds to the work that comes later than the first section. The association unit 132 associates frames included in the first section of the time-series data recording the sample work with frames included in the first section of the time-series data recording the target work, and associates frames included in the second section of the time-series data recording the sample work with frames included in the second section of the time-series data recording the target work. FIG. 13 shows an example of an association graph in this embodiment. The association unit 132 breaks down the sample work of the skilled worker P1 and the target work of the beginner P2 by task content. FIG. 13 illustrates a case in which the skilled worker P1 and the beginner P2 perform task B after task A. The association unit 132 associates task A of the skilled worker P1 with task A of the beginner P2, and associates task B of the skilled worker P1 with task B of the beginner P2 separately from task A. The work content may be broken down by a user giving instructions to divide the work content at any point based on information about the work content that he or she has obtained in advance, or the information processing device 100 may determine and break down the movements of the worker P. By breaking down the work content into work content and associating each motion in this way, the accuracy of the association can be improved, and furthermore, the amount of calculation required for association using dynamic programming such as the DTW method can be reduced.

[0045] FIG. 14 shows an example of a correspondence graph in this embodiment. A skilled worker P1 performs tasks A, B, C, and D, while a beginner P2 performs tasks A, B, and D. The sequence of actions is broken down into corresponding tasks, and the correspondence unit 132 performs correspondence while skipping task C, which the beginner P2 does not perform. This situation may occur when the skilled worker P1 can perform an additional, highly sophisticated task that requires a little extra effort, while the beginner P2 only performs necessary and sufficient tasks. If the correspondence unit 132 performs correspondence without skipping task C, the detection unit 133 will determine that the skilled worker P1 is behind the beginner P2 by task C, resulting in noise in the feedback to the user that is different from the process that is actually being detected. Therefore, by performing task difference analysis on only tasks common to the skilled worker P1 and the beginner P2, more accurate task difference analysis results can be obtained.

[0046] (Fifth embodiment) In this embodiment, the same aspects as the first to fourth embodiments will be omitted, and only the differences from the first to fourth embodiments will be described. Also, the same names and symbols will be used for the parts that are common to the first to fourth embodiments.

[0047] In this embodiment, task variance analysis can be performed in real time on the target task of beginner P2. Here, "real time" is not limited to the narrow definition of the moment when beginner P2 performs a certain action in the target task and the moment when the information processing device 100 analyzes and processes that action simultaneously. It is assumed that the association unit 132 and the detection unit 133 start performing association and detection of whether there is a time difference, respectively, in the information processing device 100 before the information acquisition terminal 200 completes measurement of the series of target tasks of beginner P2. Furthermore, the display processing unit 134 may start display processing before the information acquisition terminal 200 completes measurement of the series of target tasks of beginner P2. That is, rather than using prerecorded data as target task data, recorded data of beginner P2's movements is quickly used as target task data for association in the association unit 132. However, in this description, it is assumed that the sample task of expert P1 is recorded in advance and stored in the sample task memory unit 121. For the target task of beginner P2, real-time captured data is used. Using the data on the sample task and the target task, a program is executed in the estimation unit 131, the association unit 132, the detection unit 133, and the display processing unit 134. Due to the time required for each process, a time lag of several seconds may occur between the actual task performed by the beginner P2 and its display on the display unit 140; however, it is possible to reduce the processing load and shorten the time lag by reducing the frame rate of the video used for analysis, reducing the number of joints in the posture coordinates to be input, and narrowing the search range for association in the dynamic programming (DTW) method.

[0048] One method for visualizing the results of real-time task difference analysis is to display the results of matching and synchronizing the sample task of the expert P1 and the target task actions of the beginner P2 for each motion on the display unit 140. At this time, in order to ensure the search range of the DTW method, a buffer is provided for storing the data on the target task of the beginner P2, which is input in real time, in the target task memory unit 122 for each predetermined time or motion. The DTW method is used to match the tasks with the sample tasks of the beginner stored in the buffer in the target task memory unit 122 for each predetermined time or motion.

[0049] The association unit 132 uses dynamic programming to associate frames included in the data recording the sample task with frames included in the data recording the target task for each motion. This is the same as in the first embodiment. However, in this embodiment, data related to the movements of the beginner P2 acquired in real time is used as the data for the target task. The association unit 132 reads a certain amount of time-series data related to the latest movements of the beginner P2 (for example, several seconds) from the target task memory unit 122 and associates it as data for the target task. Then, when new time-series data is added, the association unit 132 may delete older time-series data from the target task memory unit 122.

[0050] The detection unit 133 detects where the target task is delayed relative to the sample task, using the association graph obtained by the association unit 132. The time width of the fixed window may be the same as the time included in the time series data that the association unit 132 reads out each time from the target task memory unit 122, or it may be shorter than that.

[0051] In this way, by performing a real-time task difference analysis on the target task of the beginner P2, the beginner P2 can obtain feedback regarding the differences between himself and the expert P1 before he forgets the task he has performed.

[0052] (Sixth embodiment) In this embodiment, the same aspects as the first to fifth embodiments will be omitted, and only the differences from the first to fifth embodiments will be described. The same names and symbols will be used for the parts that are common to the first to fifth embodiments.

[0053] In the first to fifth embodiments, the person performing the sample task (expert P1) and the person performing the target task (beginner P2) are described as different people. However, in this embodiment, the person performing the sample task and the person performing the target task can be the same person. A previously acquired series of actions of beginner P2 is used as the target task, and a newly acquired series of actions of beginner P2 is used as the sample task, and differences between these tasks are analyzed. The association unit 132 acquires, from the target task memory unit 122, data from when beginner P2 first attempted the series of actions as the target task, and acquires, from the target task memory unit 122, the latest data from when beginner P2 attempted the series of actions as the sample task. The detection unit 133 detects where the beginner P2's initial data lags behind the latest data of beginner P2.

[0054] In this way, by performing a task difference analysis using a past task as the target task and a later task by the same person as the sample task, it is possible to ascertain the degree of skill improvement of the same person.

[0055] The data selected as the target work in the association unit 132 does not necessarily have to be the first data, and the data selected as the sample work does not necessarily have to be the latest data.

[0056] In addition, instead of detecting the point where the first data of beginner P2 is behind the latest data of beginner P2, the detection unit 133 may detect the point where the latest data of beginner P2 is ahead of the first data of beginner P2, that is, the point where beginner P2's work has improved.

[0057] The subject of the sample task or target task is not limited to a person, but may be an animal or a machine such as a robot. Furthermore, the subject does not have to be an entity that moves independently, and the data such as video data that is the subject of task variance analysis may be data that records the movement of a fluid.

[0058] (Seventh embodiment) In this embodiment, the same aspects as the first to sixth embodiments will be omitted, and only the differences from the first to sixth embodiments will be described. The same names and symbols will be used for the parts that are common to the first to sixth embodiments.

[0059] In this embodiment, less important parts of a task are excluded from the task variance analysis, and only more important parts are selectively included in the task variance analysis. FIG. 15 is an example of a schematic diagram of the information processing system 11 in this embodiment. The information processing device 101 includes a target selection unit 135 that selects a target range for analyzing the differences between the sample task and the target task from images included in the time-series data recording the sample task and the time-series data recording the target task. The target selection unit 135 executes a target selection program stored in the program storage unit 125 to select a range of images to be used in the task variance analysis, for example, based on coordinates. The range may be specified by the user or determined by the target selection unit 135. When the target selection unit 135 determines the range, it can select a range appropriate for the task content specified or determined by the information processing device 101 as the target range for the task variance analysis. For example, in the case of variance analysis of assembly work on a conveyor belt, the entire body of worker P is the target range, whereas in the case of detailed assembly work on a desk, only the worker P's hands are the target range. Alternatively, the target selection unit 135 may determine the target range from the movement of the worker P. For example, if there is very little movement of the lower body, the lower body of the worker P can be excluded from the target range. The video data and posture coordinates of the worker P whose target range has been limited by the target selection unit 135 are stored in the sample work storage unit 121 and the target work storage unit 122. The subsequent processing in the estimation unit 131 to the display processing unit 134 also uses the data whose target range has been limited.

[0060] In this way, by performing the task variance analysis while excluding parts with low importance, it is possible to reduce the amount of storage space used by the storage unit 120 and improve the speed and accuracy of the analysis.

[0061] The target range may be limited by the target selection unit 135 before or after the processing by the estimation unit 131.

[0062] (Eighth embodiment) In this embodiment, the same aspects as the first to seventh embodiments will be omitted, and only the differences from the first to seventh embodiments will be described. Also, the same names and symbols will be used for the parts that are common to the first to seventh embodiments.

[0063] In this embodiment, the feature values ​​of the movements of the skilled person P1 are quantified, and a sample movement is generated based on a representative value of these feature values. FIG. 16 is an example of a schematic diagram of the information processing system 12 in this embodiment. In the information processing device 102, the sample movement is generated based on a representative value contained in time-series data of the posture coordinates of multiple skilled people P1. The information processing device 102 includes a sample generation unit 136. The sample generation unit 136 can generate the sample movement by executing a sample generation program stored in the program storage unit 125. The feature values ​​of the movements of the skilled person P1 include, for example, coordinates representing the height and position of the hands, arms, shoulders, head, etc., the angle of the head tilt, the time the hands are placed in a specific position, the speed of eye movement, the direction of the body, and the angle of the elbows. Since the position coordinates, angle, time, and speed are each quantifiable feature values, representative values ​​such as the mean, median, and mode can be calculated. Based on the calculated representative value, the sample generation unit 136 generates a sample task, and the generated sample task is stored in the sample task storage unit 121. The subsequent processing related to task variance analysis is also performed using the generated sample task.

[0064] In this way, by generating a sample task based on representative values ​​of multiple data, even if the skilled worker P1 makes a single mistake or delay, the outliers of those actions can be reduced.

[0065] The person who appears as the skilled worker P1 in the multiple videos input to generate the sample work does not have to be the same person. By using different people as the skilled workers P1 and generating sample work based on representative values ​​from multiple data, it is possible to perform work variance analysis using sample work in which individual habits are reduced.

[0066] (Ninth embodiment) In this embodiment, the same aspects as the first to eighth embodiments will be omitted, and only the differences from the first to eighth embodiments will be described. Also, the same names and symbols will be used for the parts that are common to the first to eighth embodiments.

[0067] In this embodiment, the sample work is represented by a CG model. When a program related to the sample work is input, the sample generation unit 136 can generate a CG model of the sample work. By using data generated from a program as the sample work instead of using data of an actual person, it is possible to compare the analysis target with perfect work without any errors.

[0068] Although several embodiments have been described above, the present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0069] Furthermore, the present disclosure includes examples according to the following appendices.

[0070] [Appendix 1] An information processing device capable of detecting whether or not there is a time difference between a first action and a second action, based on the presence or absence of a frame included in only one of first time series data relating to the first action and second time series data relating to a second action different from the first action, the first time series data being associated on a time-series frame-by-frame basis.

[0071] [Appendix 2] a storage unit that stores time-series data of posture coordinates of a first actor performing the first action, which is included in the first time-series data, and time-series data of posture coordinates of a second actor performing the second action, which is included in the second time-series data; 10. The information processing device according to claim 1.

[0072] [Appendix 3] an estimation unit that estimates posture coordinates of the first subject from the first time-series data and estimates posture coordinates of the second subject from the second time-series data; the storage unit stores the posture coordinates; 3. The information processing device according to claim 2.

[0073] [Appendix 4] a display processing unit that executes processing to simultaneously display, on a display unit, a frame included in the first time series data and a frame included in the second time series data that is associated with the frame; 4. An information processing device according to any one of claims 1 to 3.

[0074] [Appendix 5] the display processing unit, when displaying a frame included in only one of the first time series data or the second time series data, executes a process of stopping the other frame and displaying it on the display unit. 5. The information processing device according to claim 4.

[0075] [Appendix 6] a display processing unit that executes a process of displaying, on a display unit, a correspondence relationship for each motion between a frame included in the first time-series data and a frame included in the second time-series data; 2. The information processing device according to claim 1.

[0076] [Appendix 7] a display unit that displays information related to the time difference based on processing by the display processing unit; 7. An information processing device according to any one of appendices 4 to 6.

[0077] [Appendix 8] a correspondence unit that associates frames included in the first time series data with frames included in the second time series data for each motion using dynamic programming; 8. An information processing device according to any one of appendices 1 to 7.

[0078] [Appendix 9] a correspondence unit that outputs, as a graph, a correspondence relationship for each motion between a frame included in the first time series data and a frame included in the second time series data; a detection unit that scans a fixed window in the graph and compares a value indicating the slope of the graph within the fixed window with a threshold value to detect a point where the time difference occurs; 9. The information processing device according to any one of Supplementary Notes 1 to 8, comprising:

[0079] [Appendix 10] the association unit outputs a correspondence relationship for each motion between the frames included in the first time-series data and the frames included in the second time-series data as a graph with the first action on the vertical axis and the second action on the horizontal axis; the detection unit scans a fixed window in the graph and detects a point within the fixed window where the slope of the graph is equal to or less than a certain value as a point where the second action is delayed relative to the first action. 10. The information processing device according to claim 9.

[0080] [Appendix 11] the display unit displays a still image selected from frames included in only one of the first time series data or the second time series data. 11. An information processing device according to any one of appendices 7 to 10.

[0081] [Appendix 12] the display unit displays the first action in a general form of only the skeleton of a first actor performing the first action. 12. An information processing device according to any one of appendices 7 to 11.

[0082] [Appendix 13] the associating unit decomposes the first motion and the second motion into a first section relating to a task that comes earlier in time and a second section relating to a task that comes later than the first section; Correlating frames included in a first interval of the first time series data with frames included in a first interval of the second time series data; Correlating frames included in the second interval of the first time series data with frames included in the second interval of the second time series data; 13. An information processing device according to any one of appendices 8 to 12.

[0083] [Appendix 14] further comprising an information acquisition terminal for acquiring the second action; before the information acquisition terminal finishes acquiring the series of the second actions, the information acquisition terminal starts the frame-by-frame correspondence and the detection of whether or not there is a time difference. 14. An information processing device according to any one of appendices 1 to 13.

[0084] [Appendix 15] a first subject performing the first action and a second subject performing the second action are different; 15. An information processing device according to any one of appendices 1 to 14.

[0085] [Appendix 16] The first actor performing the first action and the second actor performing the second action are the same. 16. An information processing device according to any one of appendices 1 to 15.

[0086] [Appendix 17] a target selection unit that selects a target range for analyzing a difference between the first action and the second action from images included in the first time-series data and the second time-series data; 17. An information processing device according to any one of appendices 1 to 16.

[0087] [Appendix 18] The first action is generated based on a representative value included in a plurality of time-series data of posture coordinates of the first subject of action. An information processing device according to any one of appendices 2, 3, 12, 15, and 16.

[0088] [Appendix 19] the first motion is represented by a CG model; 19. An information processing device according to any one of appendices 1 to 18.

[0089] [Appendix 20] The first action is a model action used by a second subject performing the second action when performing the second action. 20. An information processing device according to any one of appendices 1 to 19.

[0090] [Appendix 21] A program capable of detecting whether or not there is a time difference between a first action and a second action, based on the presence or absence of a frame included in only one of first time series data relating to the first action and second time series data relating to a second action different from the first action, the first time series data and the second time series data being associated on a time-series frame-by-frame basis.

[0091] [Appendix 22] A detection method in which an information processing device detects whether there is a time difference between a first action and a second action, based on the presence or absence of a frame that is included in only one of first time series data relating to the first action and second time series data relating to a second action different from the first action, the first time series data being associated on a time-series frame-by-frame basis. [Explanation of symbols]

[0092] 10 Information Processing Systems 11 Information Processing Systems 12 Information Processing Systems 20 Working Memory 100 Information processing device 101 Information processing equipment 102 Information processing equipment 110 Communications Department 120 Storage section 121 Sample Working Memory 122 Target Working Memory 122 Working Memory for Each Task 123 Storage section 124 Detection memory unit 125 Program Memory Unit 130 Control Unit 131 Estimation Department 132 Mapping section 133 Detector 134 Display processing section 135 Target Selection Section 136 Sample generation section 140 Display section 200 Information Acquisition Terminal 210 Communications Department

Claims

1. An information processing device capable of detecting whether or not there is a time difference between a first action and a second action based on the presence or absence of a frame that is included only in either first time series data relating to the first action or second time series data relating to a second action different from the first action, the frames being associated on a time series frame basis.

2. a storage unit configured to store time-series data of posture coordinates of a first actor performing the first action included in the first time-series data, and time-series data of posture coordinates of a second actor performing the second action included in the second time-series data; The information processing device according to claim 1 .

3. an estimation unit that estimates posture coordinates of the first subject from the first time-series data and estimates posture coordinates of the second subject from the second time-series data; the storage unit stores the posture coordinates; The information processing device according to claim 2 .

4. a display processing unit that executes processing to simultaneously display, on a display unit, a frame included in the first time series data and a frame included in the second time series data that is associated with the frame; The information processing device according to claim 1 .

5. the display processing unit, when displaying a frame included in only one of the first time series data or the second time series data, executes a process of stopping the other frame and displaying it on the display unit. The information processing device according to claim 4 .

6. a display processing unit that executes a process of displaying, on a display unit, a correspondence relationship for each motion between a frame included in the first time-series data and a frame included in the second time-series data; The information processing device according to claim 1 .

7. a display unit that displays information related to the time difference based on processing by the display processing unit; The information processing device according to claim 4 .

8. a correspondence unit that associates frames included in the first time series data with frames included in the second time series data for each motion using dynamic programming; The information processing device according to claim 1 .

9. a correspondence unit that outputs, as a graph, a correspondence relationship for each motion between a frame included in the first time series data and a frame included in the second time series data; a detection unit that scans a fixed window in the graph and compares a value indicating the slope of the graph within the fixed window with a threshold value to detect a point where the time difference occurs; The information processing device according to claim 1 , comprising:

10. the association unit outputs a correspondence relationship for each motion between the frames included in the first time-series data and the frames included in the second time-series data as a graph with the first action on the vertical axis and the second action on the horizontal axis; the detection unit scans a fixed window in the graph and detects a point within the fixed window where the slope of the graph is equal to or less than a certain value as a point where the second action is delayed relative to the first action; The information processing device according to claim 9 .

11. the display unit displays a still image selected from frames included in only one of the first time series data or the second time series data. The information processing device according to claim 7 .

12. the display unit displays the first action in a general form of only the skeleton of a first actor performing the first action; The information processing device according to claim 7 .

13. the associating unit decomposes the first motion and the second motion into a first section relating to a task that comes earlier in time and a second section relating to a task that comes later than the first section; Correlating frames included in a first interval of the first time series data with frames included in a first interval of the second time series data; Correlating frames included in a second section of the first time series data with frames included in a second section of the second time series data; The information processing device according to claim 8 .

14. further comprising an information acquisition terminal for acquiring the second action; before the information acquisition terminal finishes acquiring the series of second actions, the information acquisition terminal starts the frame-by-frame correspondence and the detection of whether or not there is a time difference. The information processing device according to claim 1 .

15. a first subject performing the first action and a second subject performing the second action are different from each other; The information processing device according to claim 1 .

16. a first subject performing the first action and a second subject performing the second action are the same; The information processing device according to claim 1 .

17. a target selection unit that selects a target range for analyzing a difference between the first action and the second action from images included in the first time-series data and the second time-series data; The information processing device according to claim 1 .

18. the first action is generated based on a representative value included in a plurality of pieces of time-series data of posture coordinates of the first performer; The information processing device according to claim 2 .

19. the first action is represented by a CG model; The information processing device according to claim 1 .

20. The first action is a model action used by a second subject performing the second action when performing the second action. The information processing device according to claim 1 .

21. The processing circuit A program capable of detecting whether or not there is a time difference between a first action and a second action, based on the presence or absence of a frame included in only one of first time series data relating to the first action and second time series data relating to a second action different from the first action, the first time series data being associated on a frame-by-frame basis in a time series.

22. A detection method in which an information processing device detects whether there is a time difference between a first action and a second action based on the presence or absence of a frame that is included in only one of first time series data relating to the first action and second time series data relating to a second action different from the first action, the frames being associated on a chronological frame basis.

Images