Evaluation method and data generation method

Abstract

To appropriately evaluate the accuracy of recognition results obtained by the recognition model. [Solution] The method for evaluating the accuracy of work recognition comprises: a result acquisition step of acquiring a recognition result by a recognition model configured to recognize the type of work of a worker using data related to the worker's work as input; a correct answer acquisition step of acquiring a correct answer type that corresponds to the recognition result and represents the actual type of work; and an evaluation step of evaluating the accuracy of the recognition result using similarity data, which includes the similarity between the work type as a recognition result and the correct answer type, and which has been defined in advance for the similarity between a plurality of work types.

Description

[Technical Field]

[0001] The present disclosure relates to an evaluation method and a data generation method. [Background Art]

[0002] Conventionally, recognition technology for recognizing the work of an operator is known. For example, in Patent Document 1, work is recognized by inputting data based on sensor data detected by various sensors into a previously prepared work recognition model. [Prior Art Literature] [Patent Literature]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 2022-018180 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] The recognition accuracy of work recognition by a work recognition model may be evaluated. Here, in an evaluation method that evaluates recognition accuracy only from the perspective of whether the recognition result by the work recognition model matches the actual work, the same evaluation result is obtained for two cases: one where the recognition result does not match the actual work but is relatively similar, and the other where the recognition result is completely different from the actual work. Therefore, such an evaluation method cannot appropriately evaluate recognition accuracy. [Means for Solving the Problem]

[0005] The present disclosure can be implemented in the following modes.

[0006] (1) According to a first embodiment of the present disclosure, a method for evaluating the accuracy of work recognition is provided. This evaluation method comprises: a result acquisition step of acquiring a recognition result by a recognition model configured to recognize the type of work of a work based on data relating to a worker's work as input; a correct answer acquisition step of acquiring a correct answer type that corresponds to the recognition result and represents the actual type of work; and an evaluation step of evaluating the accuracy of the recognition result using similarity data, which includes the similarity between the work type as a recognition result and the correct answer type, with similarity data having predefined similarities between a plurality of work types. This approach allows for an appropriate evaluation of the accuracy of the recognition results by taking into account the similarity between the type of work performed as a recognition result and the correct answer type. (2) In the above configuration, in the result acquisition step, a series of recognition results including a plurality of recognition results that are consecutive in time series are acquired, in the correct answer acquisition step, a series of correct answer types including a plurality of correct answer types that are consecutive in time series are acquired, and in the evaluation step, the similarity is acquired for each of the plurality of recognition results using the similarity data, and the accuracy of the series of recognition results is evaluated according to an accuracy score obtained by prorating and adding each of the acquired similarity values ​​according to the number of the plurality of recognition results. With this configuration, the accuracy of the series of recognition results can be evaluated simply and appropriately. (3) The above embodiment further includes a display step of displaying the recognition result and the correct answer type on the display screen of the display device, wherein in the display step, the recognition result and the correct answer type may be displayed on the display screen such that the degree of color of the recognition result and the correct answer type differs for each type of work. According to this embodiment, the accuracy of the recognition result can be evaluated while using color to display the recognition result and the correct answer type on the display screen in a visually easy-to-understand manner. (4) In the above embodiment, in the display step, the recognition result and the correct answer type may be displayed on the display screen such that the work types with a first similarity are represented by a combination of colors that have a shorter distance in the Lab color space compared to the work types with a second similarity that is lower than the first similarity. In this embodiment, the similarity between the recognition result and the correct answer type can be visually represented on the display screen in an easy-to-understand manner using color. (5) A second embodiment of the present disclosure provides a data generation method. This data generation method obtains feature quantities for each of a plurality of tasks, plots the plurality of tasks in a feature space according to the obtained feature quantities, obtains a representative position for each task type associated with each of the plurality of tasks in the feature space using the positions of the plurality of tasks plotted in the feature space, and generates similarity data that defines the similarity between the plurality of task types according to the representative position obtained for each task type. This disclosure can be implemented in forms other than the evaluation method and data generation method described above, such as an evaluation device, a data generation device, a work recognition device, a work recognition method, a program, a non-temporary recording medium on which the program is recorded, or a program product. The program product may be provided, for example, as a recording medium on which the program is recorded, or as a program product that can be distributed via a network. [Brief explanation of the drawing]

[0007] [Figure 1] This is an explanatory diagram showing the schematic configuration of the work recognition system. [Figure 2] This is a diagram illustrating an example of similarity data. [Figure 3] This is a flowchart of the data generation process. [Figure 4] This is a diagram illustrating the first display process. [Figure 5] This is a flowchart of the evaluation process. [Modes for carrying out the invention]

[0008] A. First Embodiment: Figure 1 is an explanatory diagram showing the schematic configuration of the work recognition system 10 in the first embodiment. The work recognition system 10 is used to recognize work performed by worker WK. The work to be recognized by the work recognition system 10 is also called the "target work".

[0009] The work recognition system 10 is used in the work area where worker WK performs work. In this embodiment, the work area is a factory FC for manufacturing vehicles. In this embodiment, the work is a variety of tasks for manufacturing vehicles, including, for example, tasks related to vehicle assembly, tasks related to attaching parts to vehicles, and tasks related to vehicle inspection. A task includes one or more actions to accomplish that task. A task usually includes a series of actions.

[0010] In this disclosure, "recognizing a task" more specifically means recognizing a task type, which is the type of task. Task types are used to classify tasks according to their purpose, characteristics, etc. The number of task types used in the task recognition system 10, i.e., the number of task types, is predetermined. In this embodiment, the task types include "removal," "pre-tightening," "tightening," "walking," "fitting," and "returning." Note that "tightening" means the task of tightening a screw. "Pre-tightening" means the task of lightly tightening a screw before "tightening." Pre-tightening is performed, for example, to temporarily fasten a part. While "tightening" and "pre-tightening" are similar in that they both involve tightening a screw, there are differences between them, for example, in the magnitude of the tightening torque and the tightening time. "Returning" means the task of moving a completed assembled product to a predetermined location for subsequent processes. In other embodiments, the task types may include, in addition to or in place of these types, any other arbitrary types.

[0011] The work recognition system 10 comprises a sensor group 60 including one or more sensors 50, and a control device 100. In this embodiment, the control device 100 functions as a work recognition device that performs work recognition, an evaluation device that evaluates the accuracy of work recognition, and a data generation device that generates similarity data SD, which will be described later.

[0012] Sensor 50 observes the work performed by worker WK. "Observing work with sensor 50" means observing at least one of the following regarding the work being observed: worker WK, who is the subject of the work; the work object, which is the object processed by the work; the equipment EQ used for the work; the tools TL used for the work; and the work environment in which the work is performed. The information obtained by observing the target work with sensor 50 is also called observation information. Sensor 50 transmits the observation information from sensor 50, that is, the detection result from sensor 50, to the control device 100. The observation information is associated with information indicating the detection timing at which the observation information was detected.

[0013] Sensor 50 includes various sensors such as cameras (e.g., surveillance cameras, first-person cameras), microphones, inertial measurement units (IMUs), bending sensors, vibration sensors, vital sensors, area sensors, and pressure sensors. An IMU may include, for example, a 3-axis accelerometer, a 3-axis gyroscope, and a geomagnetic sensor. Sensor 50 can be installed, for example, in various locations in the factory FC, on equipment EQ used for work, on tools TL used for work, and on wearables worn by workers WK. Wearables may include, for example, eyewear EW such as goggles or glasses, work clothes WW, and gloves WG. Sensor 50 may be installed in various locations in the factory FC, for example, surveillance cameras and area sensors. Eyewear EW may also be equipped with, for example, a first-person camera and a microphone for detecting sounds around the worker WK. Furthermore, work clothes WW and gloves WG may be equipped with, for example, an IMU for detecting acceleration and angular velocity in the worker WK, a bending sensor for detecting the bending of the worker WK's fingers and wrists, a vibration sensor for detecting vibrations associated with work, a pressure sensor for detecting pressure on the fingers associated with work, a sound detection sensor for detecting sounds associated with work, and a vital sensor for detecting the worker WK's vital signs. Note that the types and combinations of sensors 50 are not limited to those described above.

[0014] The control device 100 is composed of a computer comprising a processor 101, a memory 102 including ROM and RAM, an input / output interface 103, and an internal bus 104. The processor 101, memory 102, and input / output interface 103 are connected via the internal bus 104 to enable bidirectional communication. A communication device 105 and a display device 106 are connected to the input / output interface 103. The communication device 105 can communicate directly or indirectly with each sensor 50 by wired or wireless communication. The display device 106 is composed of, for example, a liquid crystal display and displays various information such as information related to the work recognition results by the work recognition system 10. Various information such as program PG1, recognition model 220, similarity data SD, and history data HD are stored in memory 102. The processor 101 executes the program PG1 to realize various functions, including those of a work recognition unit 120, an acquisition unit 125, an evaluation unit 130, an output processing unit 135, and a data generation unit 140.

[0015] The work recognition unit 120 uses the recognition model 220 to recognize the type of work to be performed.

[0016] The recognition model 220 is configured to recognize the type of work of a target work by taking work data related to the target work as input. In this embodiment, the recognition model 220 is a machine learning model that has been pre-trained to recognize the type of work of a target work by taking work data as input. That is, the recognition model 220 is both a model that estimates the type of work and a model that classifies work. In this embodiment, at least one of primary data representing observational information and secondary data based on the primary data is used as the work data. The secondary data may be, for example, an estimation result obtained by inputting the primary data into a predetermined machine learning model.

[0017] In the present embodiment, a machine learning model using a neural network is used as the recognition model 220. The neural network includes a convolutional neural network (CNN) and a recurrent neural network (RNN). Also, in the present embodiment, the recognition model 220 is trained through supervised learning. In the supervised learning of the recognition model 220, work data is used as an explanatory variable, and the work type of the target work is used as an objective variable, that is, a label. Note that in other embodiments, various machine learning models such as random forest, support vector machine (SVM), etc., may be used as the recognition model 220, for example. Also, in other embodiments, the training method for the recognition model 220 is not limited to supervised learning.

[0018] The work recognition unit 120 acquires observation information from the sensor group 60, inputs work data based on the observation information to the recognition model 220, thereby causing the recognition model 220 to output a recognition result. Then, the work recognition unit 120 associates the output recognition result, the work data used for work recognition, and timing information, and records the associated data as history data HD in the memory 102. As the timing information, for example, information indicating the timing at which work recognition is performed may be used, or information indicating detection timing may be used. By repeatedly performing work recognition by the work recognition unit 120, a plurality of recognition results are recorded in the history data HD. Note that the history data HD may be stored in, for example, a computer external to the control device 100 or a recording medium.

[0019] The acquisition unit 125 acquires predetermined information used for an evaluation process described later. The predetermined information includes the recognition result obtained by the above-described recognition model 220 and a correct type corresponding to the recognition result. The correct type indicates the actual type of the target work. The correct type may match the work type as the recognition result, or may differ from the work type as the recognition result. In the present embodiment, the correct type is associated in advance with the recognition result in the history data HD. The acquisition unit 125 acquires the predetermined information by referencing the history data HD.

[0020] Note that the determination of the correct type may be performed manually, for example. Further, for example, when the recognition model 220 is configured as a recognition model that prioritizes processing speed, the correct type may be determined based on a recognition result obtained by another machine learning model that prioritizes recognition accuracy over the recognition model 220. With this configuration, for example, while effectively performing real-time work recognition using the recognition model 220, the correct type can be effectively determined using another machine learning model.

[0021] In the present embodiment, the acquisition unit 125 acquires a series of recognition results and a series of correct types. The series of recognition results includes a plurality of time-series consecutive recognition results. The series of correct types includes a plurality of time-series consecutive correct types. Each correct type included in the series of correct types corresponds to each recognition result included in the series of recognition results. The acquisition unit 125 acquires such a series of correct types and a series of recognition results by referring to the history data HD. Hereinafter, the number of recognition results included in a series of recognition results is also referred to as the number of results.

[0022] The evaluation unit 130 executes an evaluation process. The evaluation process is a process for evaluating the accuracy of recognition results by using the recognition results and correct types acquired by the acquisition unit 125 and the similarity data SD. It can also be said that the evaluation process is a process for evaluating the accuracy of the recognition model 220.

[0023] The similarity data SD is data in which similarities between a plurality of work types are defined in advance. The similarity data SD includes similarities for all combinations of predetermined work types. Therefore, the similarity data SD includes the similarity between the work type as a recognition result and the correct type.

[0024] In the present embodiment, the similarity in the similarity data SD is expressed as a distance between work types, that is, a degree of deviation between work types. More specifically, the similarity in the present embodiment is expressed by Euclidean distance or Manhattan distance. Therefore, in the present embodiment, the smaller the similarity value, the higher the similarity.

[0025] Figure 2 is a diagram illustrating an example of similarity data SD. Figure 3 is a flowchart of the data generation process for realizing the data generation method in this embodiment. The data generation process is performed to generate similarity data SD. In this embodiment, the data generation process is performed by the data generation unit 140 prior to the evaluation process.

[0026] In step S105 of Figure 3, the data generation unit 140 acquires the feature quantities of each of the multiple tasks. In this embodiment, the feature quantities of a task are acquired using the task data for that task. Each task acquired in step S105 is pre-associated with a task type. More specifically, each task is pre-associated with the actual task type, i.e., the correct task type. In step S105 of this embodiment, the data generation unit 140 acquires the task data and correct task type for each of the multiple tasks performed during a predetermined period from the historical data HD. Then, the data generation unit 140 acquires the feature quantities of each task using the acquired task data for each task.

[0027] For extracting features using work data, an extraction model utilizing a machine learning model such as a neural network may be used. For example, the extraction model may be generated using a neural network (hereinafter referred to as a clustering model) that has been trained to cluster each work into a predetermined number of clusters using the work data as input. In this case, for example, the extraction model can be generated by taking at least some layers of the clustering model and using them as the extraction model. Alternatively, for example, the extraction model may be generated using a neural network (hereinafter referred to as a classification model) that has been trained to classify each work using the work data as input. In this case, for example, the extraction model can be generated by taking at least some layers of the classification model and using them as the extraction model. In these cases, the data generation unit 140 can obtain the features of each work by inputting the work data of each work into the extraction model. Furthermore, not limited to the above, the data generation unit 140 may, for example, use the work data of each work itself as the features of each work.

[0028] In step S110, the data generation unit 140 plots each operation in the feature space FS according to the features acquired in step S105. The number of dimensions of the feature space FS corresponds to the number of types of features acquired in step S105. It is preferable that the number of dimensions of the feature space FS be 2 or more. In Figure 2, the feature space FS is shown as a 2-dimensional feature space to facilitate understanding of the technology.

[0029] In step S115, the data generation unit 140 obtains a representative position for each work type associated with each work in the feature space FS. More specifically, in step S115, work groups are defined in the feature space FS by dividing the work according to work type, and a representative position in the feature space FS is obtained for each work group. In this embodiment, the representative position is the centroid of the plotted position on the feature space FS for each work included in the work group. That is, the representative position can be calculated, for example, by averaging the coordinates of the plotted positions of each work included in the work group. Figure 2 shows an example in which work groups GR1, GR2, GR3, and GR4 corresponding to "tightening," "temporary tightening," "fitting," and "walking" are defined, and representative positions DP1, DP2, DP3, and DP4 are obtained for each work group.

[0030] In step S120, the data generation unit 140 generates similarity data SD by defining the similarity between work types according to each representative position obtained in step S115. More specifically, in step S120, the similarity between work types is defined according to each representative position to cover all combinations. That is, in step S120, the degree of deviation between work types is calculated using a brute-force method. As shown in Figure 2, the similarity data SD is defined, for example, as table data. In this embodiment, the degree of deviation between identical work types is defined as zero.

[0031] In step S105 described above, similar features tend to be obtained for tasks belonging to the same work type. Furthermore, for tasks belonging to work types with similar purposes and characteristics, relatively similar features tend to be obtained, though not to the same extent as for tasks belonging to the same work type. For example, the features obtained for tasks belonging to "tightening" and "pre-tightening," both of which involve tightening screws, tend to be relatively similar. On the other hand, the features obtained for tasks belonging to "tightening" and "walking" tend to be relatively dissimilar. The similarity data SD described above is defined according to these features, and therefore appropriately reflects the similarity and differences between work types as high or low similarity scores. Note that the method for generating the similarity data SD is not limited to the above.

[0032] In the evaluation process, the evaluation unit 130 uses the acquired recognition result and correct answer type to obtain the similarity between the recognition result and the correct answer type by referring to the similarity data SD defined as described above. Then, it evaluates the accuracy of the recognition result according to the acquired similarity.

[0033] In the evaluation process of this embodiment, the evaluation unit 130 uses similarity data SD to obtain similarity for each recognition result included in a series of recognition results, and calculates an accuracy score by prorating and adding each obtained similarity according to the number of results. Then, the evaluation unit 130 evaluates the accuracy of the series of recognition results collectively according to the calculated accuracy score. In this embodiment, as described above, similarity is expressed as deviation, so the smaller the accuracy score, the more accurate the recognition result is evaluated to be. For example, in the example in Figure 2, if the recognition results for a series of operations including a first operation and a second operation, whose actual work types are "tightening" and "fitting" respectively, are first operation: "temporary tightening" and second operation: "fitting", the accuracy score is calculated as 0.5 × 1 + 0.5 × 0 = 0.5. On the other hand, if the recognition results for the same series of operations are first operation: "walking" and second operation: "fitting", the accuracy score is calculated as 0.5 × 40 + 0.5 × 0 = 20. In this case, the accuracy of the latter recognition result is lower than that of the former recognition result.

[0034] Let's return to the explanation of Figure 1. In this embodiment, the output processing unit 135 performs a first display process. The first display process is the process of displaying the recognition result and the correct answer type on the display screen of the display device 106. In the first display process in this embodiment, the recognition result and the correct answer type are displayed on the display screen simultaneously for at least a portion of the time. In addition, in this embodiment, the output processing unit 135 performs a second display process. The second display process is the process of displaying evaluation result information related to the evaluation result on the display device 106. The evaluation result information may be displayed on the display screen simultaneously with the recognition result and correct answer type displayed by the first display process for at least a portion of the time.

[0035] Figure 4 is a diagram illustrating the first display process in this embodiment. In this embodiment, during the first display process, the display screen 107 of the display device 106 displays the recognition result and the correct answer type such that the degree of color of the recognition result and the correct answer type differs for each type of work. In the example in Figure 4, the recognition result is displayed in the first display area AR1 of the display screen 107, and the correct answer type is displayed in the second display area AR2 of the display screen 107. In Figure 4, the difference in the degree of color is schematically represented by differences in the type, direction, and density of hatching. Note that "different degrees of color" means that at least one of the hue, saturation, and brightness is different.

[0036] Furthermore, in the first display processing in this embodiment, the display screen 107 displays the recognition results and correct answer types such that work types with a first similarity level are represented by color combinations with shorter distances in the Lab color space compared to work types with a second similarity level. The second similarity level is lower than the first similarity level. In other words, in this embodiment, the greater the similarity between the recognition result and the correct answer type, the closer the colors used to display the recognition result and the correct answer type. For example, in this embodiment, the similarity between "temporary fastening" and "tightening" is higher than the similarity between "temporary fastening" and "insertion," so this display configuration can be achieved by displaying "temporary fastening," "tightening," and "insertion" in blue, light blue, and yellow, respectively, which have similar brightness and saturation levels.

[0037] As shown in Figure 4, in this embodiment, the recognition results and correct answer types are represented by a Gantt chart on the display screen 107. In the Gantt chart of Figure 4, predetermined sets of tasks are displayed on the same row. Furthermore, tasks that occur later in chronological order are displayed further to the right and further down on the same row. In the example of Figure 4, the recognition result of task RR1 is "fit," while the correct answer type for task RR1 is "tighten." Similarly, the recognition result of task RR2 is "temporarily tightened," while the correct answer type for task RR2 is "tighten." In this case, on the display screen 107, each recognition result and each correct answer type is displayed such that the difference in the degree of color between the recognition result and the correct answer type of task RR1 is greater than the difference in the degree of color between the recognition result and the correct answer type of task RR2. Here, "difference" refers to the difference in the Lab color space. Note that the display method of the recognition results and correct answer types on the display screen 107 is not limited to the stepped Gantt chart described above. For example, the recognition result and the correct answer type may be represented by a time chart in the shape of a band, which displays each task, color-coded according to the task type, arranged in chronological order.

[0038] Figure 5 is a flowchart of the evaluation process for implementing the evaluation method in this embodiment. In step S205, the acquisition unit 125 acquires the recognition result. A step in which a recognition result is acquired, such as step S205, is also called a result acquisition step.

[0039] In step S210, the acquisition unit 125 acquires the correct answer type. A step in which the correct answer type is acquired, such as in step S210, is also called the correct answer acquisition step. As described above, in steps S205 and S210 of this embodiment, the acquisition unit 125 acquires a series of recognition results and a series of correct answer types by referring to the history data HD.

[0040] In step S215, the evaluation unit 130 performs an evaluation process to evaluate the accuracy of the recognition result obtained in step S205. More specifically, the evaluation unit 130 uses the recognition result obtained in step S205 and the correct answer type obtained in step S210 to refer to the similarity data SD to obtain the similarity between the work type as a recognition result and the correct answer type. Then, using the obtained similarity, the accuracy of the recognition result is evaluated. A step in which the accuracy of the recognition result is evaluated using the similarity data SD, as in step S215, is also called an evaluation step.

[0041] In step S220, the output processing unit 135 executes the first display process to display the recognition result obtained in step S205 and the correct answer type obtained in step S210 on the display screen 107 of the display device 106. A step in which the recognition result and the correct answer type are displayed on the display screen 107, as in step S220, is also called a display step.

[0042] In step S225, the output processing unit 135 executes a second display process to display the evaluation result from step S215 on the display screen 107. In other embodiments, the output processing unit 135 may, instead of or in addition to the first and second display processes, execute a process to output the evaluation result to an external computer or the like.

[0043] According to the evaluation method in this embodiment described above, the recognition result of work types is evaluated using similarity data SD, which has a predefined similarity between work types. Therefore, the accuracy of the recognition result can be appropriately evaluated by taking into account the similarity between the recognized work type and the correct work type. Furthermore, according to this embodiment, the accuracy of the recognition model 220 can be appropriately evaluated. For example, the first model that recognizes "tightening" as "temporary tightening" can be evaluated as having higher accuracy than the second model that recognizes "tightening" as "walking". As a result, compared to conventional evaluation methods that evaluate recognition accuracy only from the perspective of whether the recognition result matches the actual work type, for example, it is possible to more appropriately determine whether the direction of learning and construction of the recognition model 220 is correct. Furthermore, the accuracy of the recognition result by the recognition model 220 can be evaluated with higher sensitivity, and maintenance such as retraining, additional learning, reconstruction, and switching to a new recognition model of the recognition model 220 can be performed more appropriately.

[0044] Furthermore, in this embodiment, a similarity score is obtained for each of the multiple recognition results included in the series of recognition results, and the accuracy of the series of recognition results is evaluated collectively according to the accuracy score obtained by prorating and adding each of the obtained similarity scores according to the number of recognition results. Therefore, the accuracy of the series of recognition results can be evaluated simply and appropriately.

[0045] Furthermore, in this embodiment, in the display step of displaying the recognition result and correct answer type on the display screen 107, the recognition result and correct answer type are displayed on the display screen 107 in such a way that the degree of color of the recognition result and correct answer type differs for each type of work. Therefore, while evaluating the accuracy of the recognition result, the recognition result and correct answer type can be displayed on the display screen 107 in a visually easy-to-understand manner using color. In particular, in this embodiment, the recognition result and correct answer type are displayed on the display screen 107 in such a way that types with a similarity of first degree are represented by color combinations with shorter distances in the Lab color space compared to types with a similarity of second degree. Therefore, the similarity between the recognition result and the correct answer type can be visually expressed in a visually easy-to-understand manner on the display screen 107 using color.

[0046] B. Other embodiments: (B1) In the above embodiment, the recognition model 220 is configured as a machine learning model, but is not limited thereto. For example, the recognition model 220 may be a rule-based model constructed to recognize the type of target work using work data as input.

[0047] (B2) In the above embodiment, a series of recognition results are obtained in the result acquisition step, but this is not limited to this. For example, only one recognition result may be obtained in the result acquisition step, or multiple recognition results that are not consecutive in time may be obtained. Also, when multiple recognition results are obtained, the accuracy of the multiple recognition results does not have to be evaluated all at once, and the accuracy of each recognition result may be evaluated individually.

[0048] (B3) In the above embodiment, the similarity is expressed as the degree of deviation, but is not limited to this. For example, the similarity may be expressed as the reciprocal of the Euclidean distance or Manhattan distance, or as the cosine similarity or Jacquard similarity.

[0049] (B4) In the above embodiment, in the display step, the recognition results and correct answer types are displayed such that work types with a first similarity level are represented by color combinations with shorter distances in the Lab color space compared to work types with a second similarity level. However, the display manner of the recognition results and correct answer types in the display step is not limited to this. For example, in the display step, the recognition results and correct answer types may be displayed simply so that the degree of color differs for each work type. Alternatively, in the display step, the recognition results and correct answer types may be displayed without differentiating the degree of color for each work type. Furthermore, in the above embodiment, the display step may not be performed at all.

[0050] (B5) In the above embodiment, the control device 100 functions as a work recognition device, an evaluation device, and a data generation device, but it is sufficient for it to function as at least an evaluation device. If the control device 100 does not function as a work recognition device, the control device 100 does not need to have a work recognition unit 120. Also in this case, the memory 102 of the control device 100 does not need to have a recognition model 220 recorded in it. If the control device 100 does not function as a data generation device, the control device 100 does not need to have a data generation unit 140. Also in this case, the control device 100 may acquire and use similarity data SD prepared in advance by, for example, an external computer or the like.

[0051] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0052] 10...Work recognition system, 50...Sensor, 60...Sensor group, 100...Control device, 101...Processor, 102...Memory, 103...Input / output interface, 104...Internal bus, 105...Communication device, 106...Display device, 107...Display screen, 120...Work recognition unit, 125...Acquisition unit, 130...Evaluation unit, 135...Output processing unit, 140...Data generation unit, 220...Recognition model

Claims

1. A method for evaluating the accuracy of work recognition, A result acquisition step involves obtaining a recognition result from a recognition model configured to recognize the type of work performed by the worker, using data related to the worker's work as input. A correct answer acquisition step that acquires a correct answer type that corresponds to the recognition result and represents the actual work type, An evaluation method comprising: an evaluation step of evaluating the accuracy of the recognition result using similarity data, which is a set of similarity data in which the similarity between a plurality of work types is defined in advance, and which includes the similarity between the work type as a recognition result and the correct answer type.

2. The evaluation method according to claim 1, In the result acquisition step, a series of recognition results including multiple recognition results that are sequentially consecutive are acquired. In the correct answer acquisition step, a series of correct answer types, including multiple correct answer types that are consecutive in time, are acquired. An evaluation method comprising the evaluation step of obtaining the similarity score for each of the multiple recognition results using the similarity data, and evaluating the accuracy of the series of recognition results according to an accuracy score obtained by proportionally allocating and adding each of the obtained similarity scores according to the number of the multiple recognition results.

3. The evaluation method according to claim 1 or 2, further, The system includes a display step that displays the recognition result and the correct answer type on the display screen of the display device, An evaluation method in which, in the display step, the recognition result and the correct answer type are displayed on the display screen such that the degree of color of the recognition result and the correct answer type differs for each type of work.

4. The evaluation method according to claim 3, An evaluation method in which, in the display step, the recognition results and the correct answer type are displayed on the display screen such that the work types with a first similarity are represented by a combination of colors with a shorter distance in the Lab color space compared to the work types with a second similarity, which is lower than the first similarity.

5. A data generation method, By obtaining the features of each of the multiple tasks, Depending on the acquired features, the multiple operations are plotted in the feature space. Using the positions of the multiple tasks plotted in the feature space, a representative position of the task type associated with each of the multiple tasks is obtained in the feature space. A data generation method that generates similarity data defining the similarity between multiple work types according to the representative position obtained for each work type.

Images