State recognition program, state recognition method, and information processing device

Abstract

This information processing device: acquires image data of a worker performing an assembly operation; enters the image data into a first machine learning model that has been trained using, as correct answer data, each state of a specific object until a specific operation among a series of operations, in which the specific operation is performed on the specific object, is completed; and recognizes whether or not the specific operation is being performed by the worker, on the basis of the results output by the first machine learning model in response to the input of the image data.

Description

Status recognition program, status recognition method, and information processing device

[0001] The present invention relates to a status recognition program, a status recognition method, and an information processing device.

[0002] Although assembly work is becoming increasingly automated, much manual work remains, and in order to improve the quality and productivity of manual work, it is important to ensure that the work is being carried out in the correct process and to transfer the know-how of experts to less skilled workers.Video analysis is being used as a technology to achieve this.

[0003] As a video analysis technique, a work status determination technique is known that determines whether a specific task has been performed on a specific object from video data of assembly work. For example, the work status is determined by partially separating each image data included in the video data, extracting the area where the worker is working, detecting whether the extracted area contains scratches, and outputting a determination result based on the original image. In recent years, as a method for setting the area used to determine the work status, object detection techniques and semantic segmentation are used to set the area for each image data.

[0004] Japanese Patent Application Publication No. 2018-81629 International Publication No. 2021 / 235061 U.S. Patent Application Publication No. 2019 / 0377944 U.S. Patent Application Publication No. 2023 / 0134508

[0005] However, the above technology can sometimes result in low accuracy in determining the work state. For example, the area in which a specific work has been performed is small, and image classification using the entire object area results in low accuracy. Also, information obtained using only the object area may not be sufficient to determine whether a specific work is being performed. In particular, when the change in the object due to the work is limited, such as when applying a chemical to only part of the object, the accuracy of determining the work state is low.

[0006] In one aspect, an object of the present invention is to provide a status recognition program, a status recognition method, and an information processing device that can accurately determine whether or not a task is in progress.

[0007] In the first proposal, the state recognition program causes a computer to acquire image data of a worker performing assembly work, input the image data into a first machine learning model that has been trained using correct data representing each state of a specific object until the specific task is completed in a series of tasks in which the specific task is performed on the specific object, and recognize whether the specific task is being performed by the worker based on the results output by the first machine learning model in response to the input of the image data.

[0008] According to one embodiment, it is possible to accurately determine whether or not work is being performed.

[0009] FIG. 1 is a diagram illustrating an activity recognition system according to a first embodiment. FIG. 2 is a diagram illustrating an example of an activity in which the state of an object changes before and after the activity. FIG. 3 is a diagram illustrating a problem in the reference technology. FIG. 4 is a diagram illustrating image classification according to the first embodiment. FIG. 5 is a block diagram illustrating an example of a functional configuration of an information processing device according to the first embodiment. FIG. 6 is a diagram illustrating training data of a first classification model according to the first embodiment. FIG. 7 is a diagram illustrating training data of a second classification model according to the first embodiment. FIG. 8 is a diagram illustrating a video information DB. FIG. 9 is a diagram illustrating an image information DB. FIG. 10 is a diagram illustrating a determination result DB. FIG. 11 is a diagram illustrating training of a first classification model according to the first embodiment. FIG. 12 is a diagram illustrating training of a second classification model according to the first embodiment. FIG. 13 is a diagram illustrating an example of an input range to a classification model according to the first embodiment. FIG. 14 is a diagram illustrating a determination method using two models according to the first embodiment. FIG. 15 is a diagram illustrating an example of activity recognition according to the first embodiment. FIG. 16 is a flowchart illustrating a flow of machine learning processing of a model according to the first embodiment. FIG. 17 is a flowchart illustrating a flow of determination processing according to the first embodiment. Fig. 18 is a diagram illustrating a comparison with the conventional technology according to Example 1. Fig. 19 is a diagram illustrating a verification result of the determination result by the information processing device 10 according to Example 1. Fig. 20 is a diagram illustrating a verification result of the determination result by the information processing device 10 according to Example 1. Fig. 21 is a diagram illustrating a hardware configuration.

[0010] The following describes in detail embodiments of the present invention, a status recognition program, a status recognition method, and an information processing device, with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the embodiments can be combined as appropriate within a consistent range.

[0011] <Overall Configuration> Fig. 1 is a diagram illustrating a task recognition system according to Example 1. As shown in Fig. 1, in this system, each device in a factory 2 is connected to an information processing device 10 via a network N. Note that the network N can be any of various communication networks such as the Internet or a dedicated line.

[0012] The factory 2 is a factory that produces various products, and cameras 2a are installed in each workplace where workers perform assembly work. Note that the type of factory and the products produced are not limited, and the system can be applied to various fields, such as factories that manufacture processed products, factories that manage the distribution of goods, and automobile factories.

[0013] The camera 2a is set to capture an arbitrary area, such as the entire worker or the area where the worker is working on an object, and records the captured data as video data. The video data is composed of multiple images (frame images) captured by the camera 2a, i.e., a series of frames of a video. The number of factories 2 and cameras 2a is not limited to that shown in FIG. 1.

[0014] The information processing device 10 is an example of a computer that is connected to a camera 2a installed in the factory 2, acquires video data captured by the camera 2a, and performs video analysis. Specifically, the information processing device 10 acquires frame images (image data) from the video data and detects the work of a worker based on the results of inputting each image data into a classification model. For example, the information processing device 10 improves the quality and productivity of manual work by recognizing whether the worker has completed the work correctly and whether the worker performed the work in the correct order.

[0015] (Reference Technology) Here, we will explain video analysis of commonly performed assembly work. In recent years, much manual work remains, especially in assembly work involving precision equipment, and video analysis has become important in order to improve the quality and productivity of manual work.

[0016] General video analysis detects objects from image data (frame images) using models and analytical techniques, and can detect tasks where the state of an object changes before and after the task, such as applying chemicals or installing parts. When object detection and image recognition are combined, objects related to the task are detected, and image recognition is performed on the image area where the object appears to determine whether the task was completed properly. In other words, general video analysis checks whether the task is being performed properly by detecting changes in the state of an object from frame images.

[0017] FIG. 2 is a diagram illustrating an example of a task in which the state of an object changes before and after the task. Here, as an example, a coating task is described in which a chemical agent is applied to a target object, such as an electronic component, using a syringe. As shown in FIG. 2, in such a coating task, the state of the object changes not only before and after the task but also during the task. For example, the target object before the task is in a state where no work is being performed by the worker. The target object during the task is in a state where the chemical agent is being applied to the target object using the syringe being used by the worker. The target object after the task is in a state where the chemical agent has been applied using the syringe being used by the worker.

[0018] Next, a case where work detection is performed by general image analysis for the application work shown in Fig. 2 will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining work detection by the reference technology.

[0019] As shown in Figure 3, in general video analysis, when image data that does not show the worker's hands or image data that does not have a chemical applied to the target object is input, such as the pre-work state in Figure 2, it is detected as the pre-work state. In other words, the pre-work state is correctly detected as the pre-work state.

[0020] Next, general video analysis detects that work is in progress when image data showing the worker's hands or image data showing a place where a drug has been applied to a target object is input, such as the before-work state in Figure 2. In other words, the state of work is correctly detected as work in progress.

[0021] Next, general video analysis will detect the end of work when it receives image data showing the latter half of the work of applying medicine to the target object, or when the worker's hand leaves the syringe during the work. In other words, it will falsely detect the end of work even when the work is still in progress.

[0022] General video analysis detects the state as being after work when image data that does not show the worker's hands or image data in which the medicine is completely applied to the target object is input, such as the state after work in Figure 2. In other words, the state after work is correctly detected as being after work.

[0023] As described above, general video analysis has difficulty accurately detecting work when the change in the target object due to the work is limited, such as when applying a medicine to a part of the target object. In the example of Figure 3, image data showing the latter half of the work of applying the medicine or image data in which the worker's hand has removed the syringe during the work may be detected as having finished the work, even though the work is still in progress. In other words, general video analysis has difficulty recognizing the state when there is a work-in-progress state, which is an intermediate state between before and after the work.

[0024] Therefore, the information processing device 10 according to the first embodiment acquires image data of a worker performing assembly work, and inputs the image data to a first machine learning model that has been trained using, as correct answer data, each state of a specific object until the specific work is completed among a series of work in which the specific work is performed on the specific object. Then, the information processing device 10 recognizes whether the specific work is being performed by the worker based on the result output by the first machine learning model in response to the input of the image data.

[0025] Furthermore, the information processing device 10 according to the first embodiment inputs image data to a second machine learning model that has been trained using the state of a specific object before the start of a specific task and the state of the specific object when the specific task is completed as correct answer data. Then, the information processing device 10 recognizes whether the specific task performed by the worker is before or after the task based on the result output by the second machine learning model in response to the input of image data.

[0026] FIG. 4 is a diagram illustrating image classification according to the first embodiment. As shown in FIG. 4 , in the inference phase, each time the information processing device 10 acquires video data, it determines whether a frame image (image data) in the video data represents work in progress. For example, the information processing device 10 inputs image data into a first classification model that classifies the image data as either before or after work, and a second classification model that classifies the image data as either during work or other activities. The information processing device 10 then integrates the classification results of the first and second classification models to determine whether the image data represents work in progress.

[0027] In other words, the information processing device 10 can accurately determine whether or not work is being performed using the second classification model. Furthermore, by using the first classification model and the second classification model, the information processing device 10 can determine the state after work in two stages, thereby further improving accuracy.

[0028] <Functional Configuration> Here, the functional configuration of the information processing device 10 according to Example 1 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the functional configuration of the information processing device 10 according to Example 1. As shown in Fig. 5, the information processing device 10 includes a communication unit 11, a control unit 12, and a storage unit 13.

[0029] The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface, etc. For example, the communication unit 11 receives video data from a camera 2 a installed inside the factory 2.

[0030] The storage unit 13 is an example of a processing unit that stores various data and programs executed by the control unit 12, and is realized by, for example, a memory, a hard disk, etc. The storage unit 13 stores a first training data DB 13a, a second training data DB 13b, a first classification model 13c, a second classification model 13d, a video information DB 13e, an image information DB 13f, and a judgment result DB 13g.

[0031] The first training data DB 13a is a database that stores training data used in machine learning of the first classification model 13c that determines whether an object is before or after work. FIG. 6 is a diagram illustrating training data for the first classification model 13c according to Example 1. As shown in FIG. 6, the training data stored in the first training data DB 13a is data in which "image data," which serves as an explanatory variable during machine learning, is associated with "before work, after work," which serves as a target variable during machine learning. "Image data" indicates image data that captures a target object such as an electronic component. "Before work" indicates the state of the target object before work, and "after work" indicates the state of the target object after work.

[0032] 6 shows that "before work" is set as the objective variable for "image data on which chemicals have not been applied to electronic components," and that "after work" is set as the objective variable for "image data on which chemicals have been applied to electronic components."

[0033] The second training data DB 13b is a database that stores training data used for machine learning of the second classification model 13d, which determines whether an object has been worked on or not. FIG. 7 is a diagram illustrating training data for the second classification model 13d according to Example 1. As shown in FIG. 7, the training data stored in the second training data DB 13b is data in which "image data," which serves as an explanatory variable during machine learning, is associated with "after work, other" which serves as a target variable during machine learning. The "image data" indicates image data of an electronic component, which is the target object, and the worker's hand or the syringe used by the worker. "Working" indicates a state of the target object during work, and "other" indicates a state of the target object other than the state after work.

[0034] 7 shows that "in progress" is set as the objective variable for "image data showing the state in which the application of the medicine is in progress," and that "other" is set as the objective variable for "image data showing the state in which the application of the medicine is complete." Note that image data with the objective variable "in progress" can be image data that does not show hands or syringes in the middle of work, or image data showing the latter half of the work of applying the medicine.

[0035] The first classification model 13c is a machine learning model that is trained using training data stored in the first training data DB 13a. For example, the first classification model 13c is trained using the state of a specific object before a specific task is started and the state of the specific object when the specific task is completed as correct answer data, and determines whether the state is the pre-task state or the post-task state in response to input image data and outputs the determined state.

[0036] The second classification model 13 d is a machine learning model that is trained using training data stored in the second training data DB 13 b. For example, the second classification model 13 d is trained using correct answer data that represent each state of a specific object until a specific task is completed among a series of tasks in which a specific task is performed on a specific object, and determines whether the object is in a state after the task or in another state in response to input image data, and outputs the determined result.

[0037] The video information DB 13e is a database that stores video data of the inside of the factory transmitted from the camera 2a installed in the factory 2. For example, the control unit 12 acquires video data from the camera 2a and stores the video data in the video information DB 13e.

[0038] 8 is a diagram illustrating the video information DB 13e according to the first embodiment. As shown in FIG. 8, the video information DB 13e stores items of "factory" and "video." Here, the stored "factory" is information specifying the factory where the camera 2a is installed. The "video" is information specifying the video captured in the factory.

[0039] 8, the video information DB 13e stores "Factory A, Video A" as "Factory, Video". That is, the video information DB 13e stores that the video captured in "Factory A" is "Video A". Note that if there are multiple cameras, the video information DB 13e may store video data for each camera or for each worker.

[0040] The image information DB 13f is a database that stores image data showing a state in which a worker is working. For example, the image information DB 13f stores each image data (frame image) that constitutes video data acquired by the control unit 12 or manually by an administrator.

[0041] 9 is a diagram illustrating the image information DB 13f according to the first embodiment. As shown in Fig. 9, the image information DB 13f stores items of "video" and "image." Here, the stored "video" is the same as in Fig. 8, and the "image" is information specifying image data converted from video data.

[0042] Explaining using the example of FIG. 9, the image information DB 13f stores "video A, image A" and the like as "video, image." That is, the image information DB 13f stores that image data converted from "video A" is "image A." Note that, if there are multiple cameras, the image information DB 13f may store image data for each camera, or may store image data for each worker.

[0043] The determination result DB 13g is a database that stores a determination result obtained by integrating the classification results of the first classification model 13c and the second classification model 13d. FIG. 10 is a diagram illustrating the determination result DB 13g according to the first embodiment. As shown in FIG. 10, the determination result DB 13g stores items of "image," "working," and "other." Here, the stored items of "image," "working," and "other" are the same as those in the above-described example.

[0044] 10 , the determination result DB 13g stores "Image A, X, O," "Image B, O, X," and "Image C, X, O" as "Image, Working, Other." That is, as a result of integrating the classification results of the first classification model 13c and the second classification model 13d, the determination result DB 13g stores information that "Image A" is not working and is marked with "X," and that "otherwise" is marked with "O."

[0045] Next, returning to Fig. 3, the control unit 12 will be described. The control unit 12 is a processing unit that controls the entire information processing device 10, and is realized by, for example, a processor. This control unit 12 has a training unit 14 and an inference unit 15. The training unit 14 and the inference unit 15 are realized by electronic circuits included in the processor, processes executed by the processor, etc.

[0046] The training unit 14 has a first classification model training unit 14a and a second classification model training unit 14b, and is a processing unit that generates a first classification model 13c and a second classification model 13d prior to determining image data containing a target object.

[0047] The first classification model training unit 14a is a processing unit that generates the first classification model 13c through machine learning using training data stored in the first training data DB 13a. Specifically, the first classification model training unit 14a generates feature data indicating whether or not a chemical has been applied to the target object, that is, an electronic component, from image data. The first classification model training unit 14a then inputs correct answer data, in which the feature data indicating whether or not a chemical has been applied is used as an explanatory variable and information before and after the task is used as a target variable, into the first classification model 13c, and performs machine learning so as to minimize the error between the output result of the first classification model 13c and the target variable.

[0048] 11 is a diagram illustrating machine learning of the first classification model 13c. For example, as shown in FIG. 11, the first classification model training unit 14a inputs "(1) image data of electronic components with no chemicals applied," which is image data labeled "before work," into the first classification model 13c and obtains an output result from the first classification model 13c. The first classification model training unit 14a then updates the parameters of the first classification model 13c so that the output result of the first classification model 13c becomes "before work."

[0049] Similarly, the first classification model training unit 14a inputs "(2) Image data of an electronic component having a chemical applied thereto," which is image data labeled "after work," into the first classification model 13c, and obtains an output result from the first classification model 13c. The first classification model training unit 14a then updates the parameters of the first classification model 13c so that the output result of the first classification model 13c becomes "after work."

[0050] The second classification model training unit 14b is a processing unit that generates the second classification model 13d through machine learning using training data stored in the second training data DB 13b. The second classification model training unit 14b trains the second classification model 13d by using, as correct answer data, the initial state of a specific object when a specific task is started, the intermediate state of the specific object while the specific task is being performed, and the final state of the specific object when the specific task is completed. Specifically, the second classification model training unit 14b generates, from image data, feature data representing a state during the process of applying a chemical to an electronic component, which is a target object. The second classification model training unit 14b then inputs the correct answer data, which uses the feature data representing the state during the process of applying the chemical as an explanatory variable and information indicating whether the task is in progress or not as an objective variable, into the second classification model 13d, and performs machine learning so as to minimize the error between the output result of the second classification model 13d and the objective variable.

[0051] 12 is a diagram illustrating machine learning of the second classification model 13d. For example, as shown in FIG. 12, the second classification model training unit 14b inputs "(1) image data of electronic components with no chemicals applied," which is image data labeled "other," into the second classification model 13d and obtains an output result from the second classification model 13d. The second classification model training unit 14b then updates the parameters of the second classification model 13d so that the output result of the second classification model 13d becomes "other."

[0052] Similarly, the second classification model training unit 14b inputs "(2) Image data of the state in which the application of the medicine is completed," which is image data labeled "other," into the second classification model 13d, and obtains an output result from the second classification model 13d. The second classification model training unit 14b then updates the parameters of the second classification model 13d so that the output result of the second classification model 13d becomes "other."

[0053] Similarly, the second classification model training unit 14b inputs "(3) Image data of a state in which a medicine is being applied," which is image data labeled "in progress," into the second classification model 13d and obtains an output result from the second classification model 13d. The second classification model training unit 14b then updates the parameters of the second classification model 13d so that the output result of the second classification model 13d becomes "in progress." Note that the image data of a state in which a medicine is being applied in progress includes image data in the middle of the process that does not show hands or a syringe, image data in which the medicine application process is in the latter half, and the like.

[0054] Returning to Figure 5, the inference unit 15 is a processing unit that has a reception unit 15a, an acquisition unit 15b, a classification unit 15c, a judgment unit 15d, and an output unit 15e, and recognizes (infers) the work state from video data using each trained model.

[0055] The receiving unit 15a is a processing unit that receives video data from the camera 2a and stores the video data in the video information DB 13e. Specifically, the receiving unit 15a receives video data that captures the area around the hands of a worker performing a task and the entire process from the start to the end of the task.

[0056] The acquisition unit 15b is a processing unit that acquires image data to be input to each model from the video data accepted by the acceptance unit 15a. Specifically, the acquisition unit 15b generates image data of sizes for the first classification model 13c and image data of sizes for the second classification model 13d from each image data that is a frame image constituting the video data.

[0057] For example, the acquisition unit 15b extracts an object region (rectangular region) from the image data using an object detection model or a known object detection technology, and generates region image data including the region. Next, the acquisition unit 15b identifies range adjustment parameters (range adjustment parameter 1, range adjustment parameter 2) corresponding to the target part and the target operation from a list of range adjustment parameters that are set for each operation and part and that identify the extracted region. Note that the range adjustment parameter 1 is set to specify a larger region than the range adjustment parameter 2.

[0058] The acquisition unit 15b then multiplies the area image data by a predetermined range adjustment parameter 1 to generate first partial image data that is cut out from the surrounding area including the object (such as the object part and the tool (or the worker's hand) used by the worker for work) for use in determining whether work is being performed and input to the second classification model 13d. Meanwhile, the acquisition unit 15b multiplies the area image data by a range adjustment parameter 2 to generate second partial image data that is cut out from only the area where the state changes for use in determining whether work is being performed and input to the first classification model 13c.

[0059] 13 is a diagram illustrating the input range for each model. As shown in FIG. 13, the acquisition unit 15b performs object detection on the object detection image 41 to generate an object detection image. Then, the acquisition unit 15b performs parameter adjustment on the object detection image for the second classification model 13d to generate first partial image data 43 including the work target parts and the tool. Furthermore, the acquisition unit 15b performs parameter adjustment on the object detection image 41 for the first classification model 13c to generate second partial image data 42 including only the work target parts.

[0060] The classification unit 15c is a processing unit that determines the work state from the video data using each model that classifies the work state. For example, the classification unit 15c inputs the first partial image data 43 generated by the acquisition unit 15b into the second classification model 13d to determine whether the work is being done or not. Furthermore, the classification unit 15c inputs the second partial image data 42 generated by the acquisition unit 15b into the first classification model 13c to determine whether the work is being done or not.

[0061] The determination unit 15d is a processing unit that recognizes whether a specific task is being performed by a worker and determines the status of the task based on the result output by the second classification model 13d in response to input image data. Specifically, the determination unit 15d determines the status of the task using the determination result of the classification unit 15c. For example, if the output result of the second classification model 13d to which the first partial image data 43 generated from image A is input is "working," the determination unit 15d determines that the task is being performed, and if the output result is "other," the determination unit 15d determines that the task has been completed. Then, as shown in FIG. 10 , the determination unit 15d stores the determination result for image A, such as "◯" for "working" or "×" for "other."

[0062] 14 is a diagram illustrating a determination method using two models. As shown in FIG. 14, the determination unit 15d inputs first partial image data 43 generated from a frame image into the second classification model 13d and obtains a determination result indicating whether the target task is being performed or not. The determination unit 15d also inputs second partial image data 42 generated from the frame image into the first classification model 13c and obtains a determination result indicating whether the target task is being performed before or after the task. The determination unit 15d then recognizes the state of the target task using the output results of the second classification model 13d and the output results of the first classification model 13c.

[0063] 15 is a diagram showing an example of task recognition. As shown in FIG. 15, if the output result of the second classification model 13d is "working," the determination unit 15d recognizes the target task as "working." On the other hand, if the output result of the second classification model 13d is "other" and the output result of the first classification model 13c is "before work," the determination unit 15d recognizes the target task as "before work." Similarly, if the output result of the second classification model 13d is "other" and the output result of the first classification model 13c is "after work," the determination unit 15d recognizes the target task as "after work."

[0064] The output unit 15e is a processing unit that outputs the determination result and the recognition result by the determination unit 15d. For example, the output unit 15e displays the final recognition result on a display or transmits it to a specified terminal.

[0065] (Flow of Machine Learning Processing) Fig. 16 is a flowchart showing the flow of machine learning processing of a model according to Example 1. As shown in Fig. 16 , the training unit 14 collects training data (S1) and inputs it into the object detection model to perform training (S2).

[0066] The training unit 14 then considers the input range of image data for each of the first classification model 13c and the second classification model 13d (S3). The training unit 14 then trains the first classification model 13c (S4) and the second classification model 13d (S5). Finally, the training unit 14 deploys each model (S6), and the process ends.

[0067] (Flow of Determination Process According to Example 1) Fig. 17 is a flowchart showing the flow of the determination process according to Example 1. As shown in Fig. 20, the inference unit 15 executes the following loop process (S200 to S213) for each frame image.

[0068] Specifically, when the inference unit 15 acquires frame image data (S201), it performs object detection (S202). Next, if an object is detected by object detection (S203, Yes), the inference unit 15 acquires an object image (S204). Here, if an object is not detected by object detection (S203, No), the inference unit 15 executes processing on the next frame image.

[0069] Next, the inference unit 15 adjusts the image area of ​​the acquired object image (S205), and performs a determination using the second classification model 13d (S206). If the determination result using the second classification model 13d indicates that work is in progress (Yes in S207), the inference unit 15 outputs that work is in progress (S208), and performs processing on the next frame.

[0070] On the other hand, if the determination result by the second classification model 13d is not "work in progress" (No in S207), the inference unit 15 performs a determination using the first classification model 13c (S209). Here, if the determination result by the first classification model 13c is "before work has been completed" (Yes in S210), the inference unit 15 outputs "before work has been completed" (S211) and executes processing for the next frame. On the other hand, if the determination result by the first classification model 13c is not "before work has been completed" (No in S210), the inference unit 15 outputs "after work has been completed" (S212) and executes processing for the next frame.

[0071] (Effect) As described above, the information processing device 10 can realize a classification algorithm that can recognize the state of an object at a fine level, and can also realize an algorithm that can accurately obtain the timing of task completion even when an intermediate state exists.

[0072] Furthermore, the information processing device 10 extracts only the areas where the state changes from the object image data acquired by object detection and inputs the extracted areas into the first classification model 13c, thereby being able to focus only on the areas where the state changes, thereby enabling fine-grained state recognition to be performed.

[0073] In addition, the information processing device 10 extends the method using object detection and image recognition, and can determine whether a person is working or not by inputting image data including the surrounding area of ​​the object image data obtained by object detection into the second classification model 13d2.

[0074] Furthermore, through the processing described above, the information processing device 10 can realize a classification algorithm and provide accurate task completion timing when an intermediate state exists, thereby realizing accurate and fast recognition processing by the information processing device 10.

[0075] Furthermore, by combining the first classification model 13c and the second classification model 13d, the information processing device 10 can determine the timing of task completion accurately by using two methods to determine before and after the task and combining the results. Furthermore, by combining the first classification model 13c and the second classification model 13d, the information processing device 10 can classify an intermediate state as being in progress, thereby preventing erroneous detection of task completion during the task.

[0076] FIG. 18 is a diagram illustrating a comparison with the reference technology described in FIG. 3. FIG. 18 differs from FIG. 3 in that an intermediate state can be accurately determined as "working." This is because the information processing device 10 performs the discrimination process using the second classification model 13d, which is trained using training data including the "working" state. For example, the information processing device 10 can correctly determine that the state is "working" by inputting image data of a worker's hand and a syringe in the middle of applying a medicine into the second classification model 13d.

[0077] Furthermore, the information processing device 10 inputs image data of the worker's hand and syringe during application of the medication into the second classification model 13d and determines the state as "other." In this case, the state would normally be recognized as "after work." However, the information processing device 10 according to Example 1 can ensure the reliability of the determination result of the second classification model 13d by using the determination of the first classification model 13c. In other words, even if the determination result of the second classification model 13d is "other (= after work)," if the first classification model 13c determines the state as "before work," the information processing device 10 prioritizes the determination result of the second classification model 13d and determines the state as "before work." Therefore, the information processing device 10 can improve the reliability of the determination result. Furthermore, even if an erroneous determination occurs, the information processing device 10 can ultimately output a highly accurate determination result without outputting the erroneous determination.

[0078] Next, the verification results of the determination results by the information processing device 10 will be described using Figures 19 and 20. Here, the target task is selected as the application of a protective agent (drug) using a syringe among the product assembly tasks. The training data and evaluation data are the same data, and video data of assembly tasks performed by workers A and B was used, but only some frames of the video data were used for training. Figure 19 is a diagram explaining the determination of completion of drug application according to Example 1. Figure 20 will be described using graphs 44 and 45.

[0079] Graph 44 represents a conventional method for determining whether or not drug application is complete, and graph 45 represents a method for determining whether or not drug application is complete using the current model. The vertical axes of graphs 44 and 45 represent "in progress" and "completed," while the horizontal axes represent the number of frames. The black portions of graphs 44 and 45 represent the output of the model, with solid lines indicating when the work is complete and dotted lines indicating ideal timing for determination.

[0080] In graph 44, a false positive is detected when the task is either before or during the task, resulting in a determination that the task has been completed. However, in graph 45, no false positive is detected when the task is either before or during the task, resulting in an improvement in the false positive. Furthermore, in graph 45, a completion determination is output at an ideal timing.

[0081] FIG. 20 is a diagram illustrating the determination of drug application completion according to Example 1. In FIG. 20, explanation is given using graphs 46 and 47. Graph 46 is a graph representing a conventional drug application completion determination, and graph 47 is a graph representing a drug application completion determination using the current model. The vertical axes of graphs 46 and 47 indicate "in progress" and "completed," and the horizontal axes represent the number of frames. Furthermore, the black portions of graphs 46 and 47 represent the output of the model, the solid lines indicate that the work has been completed, and the dotted lines indicate the ideal timing of determination.

[0082] Graph 46 shows the phenomenon where a task is judged as incomplete immediately after it is finished. However, graph 47 shows that this phenomenon is not observed and has been improved. Also, graph 47 shows that the completion judgment is output at an ideal timing, though a little early.

[0083] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0084] (Use of First Classification Model) The acquisition unit 15b of the information processing device 10 can perform object recognition processing on image data of a worker performing assembly work and extract second partial image data from the image data using the results of the object recognition processing. For example, the information processing device 10 can determine whether the image data shows work in progress only based on the determination using the second classification model 13d, without performing a pre-work and post-work determination using the first classification model 13c. Furthermore, by replacing the first classification model 13c with a timestamp or the like, the information processing device 10 can integrate the pre-work and post-work determination results using the timestamp and the determination result using the second classification model 13d to determine whether the image data shows work in progress.

[0085] (Determination based on difference between frames) The information processing device 10 uses range adjustment parameters to identify in advance from image data an area where the state of the target object changes, and performs adjustment. However, the information processing device 10 can also check the difference between frames of image data and extract an area where the state of the target object changes.

[0086] (Object Detection Model) The information processing device 10 has been described as performing region adjustment after detecting an object from image data using an object detection model, but is not limited to this. For example, the information processing device 10 can use a publicly known object detection model, or can generate an object detection model by machine learning using image data used for training each model as training data.

[0087] (System) The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0088] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown. In other words, all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. For example, the training unit 14 and the inference unit 15 can be realized as separate computers (housings). In other words, they can be realized as an information processing device that performs the same functions as the training unit 14 and an information processing device that performs the same functions as the inference unit 15.

[0089] Furthermore, all or any part of the processing functions performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0090] (Hardware) Fig. 21 is a diagram illustrating an example of a hardware configuration. As shown in Fig. 21, the information processing device 10 includes a communication device 100a, a hard disk drive (HDD) 100b, a memory 100c, and a processor 100d. The components shown in Fig. 21 are connected to each other via a bus or the like.

[0091] The communication device 100a is a network interface card or the like, and communicates with other devices. The HDD 100b stores programs and databases that operate the functions shown in FIG.

[0092] The processor 100d reads out a program that executes the same processes as the respective processing units shown in Fig. 5 from the HDD 10b or the like and loads it into the memory 100c, thereby operating a process that executes the respective functions described in Fig. 5 or the like. For example, this process executes the same functions as the respective processing units of the information processing device 10. Specifically, the processor 100d reads out a program that has the same functions as the training unit 14, the inference unit 15, or the like from the HDD 100b or the like. Then, the processor 100d executes a process that executes the same processes as the training unit 14, the inference unit 15, or the like.

[0093] In this way, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 10. For example, the above-described embodiment may also be applied in the same way when another computer or server executes the program, or when these execute the program in cooperation with each other.

[0094] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and may be read out from the recording medium and executed by a computer.

[0095] REFERENCE SIGNS LIST 10 Information processing device 11 Communication unit 12 Control unit 13 Storage unit 13a First training data DB 13b Second training data DB 13c First classification model 13d Second classification model 13e Video information DB 13f Image information DB 13g Judgment result DB 14 Training unit 14a First classification model training unit 14b Second classification model training unit 15 Inference unit 15a Reception unit 15b Acquisition unit 15c Classification unit 15d Judgment unit 15e Output unit

Claims

1. A state recognition program that causes a computer to execute the following process: acquire image data of a worker performing assembly work; input the image data into a first machine learning model that has been trained using correct answer data for each state of a specific object until the specific work is completed among a series of work in which the specific work is performed on the specific object; and recognize whether the specific work is being performed by the worker based on the results output by the first machine learning model in response to the input of the image data.

2. The state recognition program according to claim 1, characterized in that the computer is caused to execute the following process: inputting the image data into a second machine learning model that has been trained using the state of the specific object before the specific work begins and the state of the specific object when the specific work is completed as correct answer data; and recognizing whether the specific work performed by the worker is before or after the work based on the results output by the second machine learning model in response to the input of the image data.

3. The state recognition program according to claim 2, characterized in that the computer is caused to execute a process in which, when the specific task by the worker is recognized as not being performed based on the output result of the first machine learning model and the specific task by the worker is recognized as having been completed based on the output result of the second machine learning model, the specific task by the worker is recognized as having been completed.

4. The state recognition program of claim 2, characterized in that the first machine learning model is trained using, as the correct answer data, the initial state of the specific object when the specific task is started, the intermediate state of the specific object while the specific task is being performed, and the final state of the specific object when the specific task is completed.

5. The state recognition program of claim 1, characterized in that the acquiring process extracts first partial image data of an area including the specific object being worked on and the worker's hands from image data of the worker performing the assembly work, and the inputting process inputs the first partial image data into the first machine learning model.

6. The state recognition program according to claim 5, characterized in that the acquisition process identifies parameters corresponding to the specific object and the specific task from a list of parameters that identify the extracted area, which are set for each task and object, and extracts the first partial image data from the image data using the identified parameters.

7. The state recognition program according to claim 2, characterized in that the acquiring process extracts second partial image data of an area including the specific object being the work target from image data of the worker performing the assembly work, and the inputting process inputs the second partial image data into the second machine learning model.

8. The state recognition program according to claim 7, characterized in that the acquiring process comprises: executing an object recognition process on image data of the worker performing the assembly work; and extracting the second partial image data from the image data using the results of the object recognition process.

9. A state recognition method characterized by having a computer execute the following process: acquire image data of a worker performing assembly work; input the image data into a first machine learning model that has been trained using correct answer data for each state of a specific object until the specific work is completed out of a series of work in which the specific work is performed on the specific object; and recognize whether the specific work is being performed by the worker based on the results output by the first machine learning model in response to the input of the image data.

10. An information processing device characterized by having a control unit that acquires image data of a worker performing assembly work, inputs the image data into a first machine learning model that has been trained using correct data representing each state of a specific object until the specific task is completed in a series of tasks in which the specific task is performed on the specific object, and recognizes whether the specific task is being performed by the worker based on the results output by the first machine learning model in response to the input of the image data.

Images