Information processing device, control method for information processing device and program

The information processing device automates task naming in manufacturing processes by analyzing images to quickly generate new work procedure information, addressing the inefficiency of manual task naming changes.

JP2024168660APending Publication Date: 2024-12-05CANON MARKETING JAPAN INC +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023085529
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing systems require time-consuming manual effort to create new work procedure information when operations are added or deleted in manufacturing processes, as task naming is labor-intensive.

Method used

An information processing device that quickly generates new work procedure information by analyzing reference and new images, identifying similarities, and automatically assigning task names, with manual verification and correction options.

Benefits of technology

Enables rapid generation of accurate new work procedure information, even with changes in work processes, by automating task identification and allowing manual adjustments, thus improving efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168660000001_ABST
    Figure 2024168660000001_ABST
Patent Text Reader

Abstract

To provide an information processing device capable of quickly generating new work procedure information to be work reference even in the case that a change in a work procedure of, for example, adding or deleting work or the like occurs, a control method for an information processing device, and a program.SOLUTION: In an information processing system 1, an information processing device 2 includes acquisition means (photographing means 3, operation means 5) for acquiring work procedure information about a plurality of work procedures, a reference image obtained by photographing each work, and a new image obtained by photographing new work, specification means (control means 21) capable of specifying the new work included in the new image on the basis of similarity between the new image and the reference image, and control means 21 for generating new work procedure information on the basis of the new work specified by the specification means and the work procedure information.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In the manufacturing process in the manufacturing industry, various tasks such as assembly work for assembling parts together are performed, but sometimes some tasks may be omitted or omitted, resulting in a defective product. In order to reduce such omissions or omissions, it has been conventionally practiced to detect omissions or omissions of tasks by analyzing images of each task using machine learning. For example, Patent Document 1 discloses a work support system that analyzes images of a workplace using machine learning to understand the work process and support the worker. In the work support system described in Patent Document 1, when generating teacher data for machine learning, highly accurate teacher data can be generated with little effort even if the work location, worker, etc. are changed in the captured images.

[0003] Incidentally, such a work support system uses work procedure information as a reference for judging whether or not a worker has performed each task in the correct procedure. The work procedure information is information in which a work process is divided into several tasks based on a photographed image of the reference work process, and the names of each task (task name) are arranged according to the work process. In actual work support, the task names of each task obtained by analyzing the photographed image of the worker's actual work are compared with the task names in the work procedure information to detect omissions or omissions of tasks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-12586 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a new task is added or an existing task is deleted in the manufacturing process, new work procedure information must be created. At that time, new work procedure information is created based on a photographed image (new image) of the new work process, but it is time-consuming to divide the work process into several tasks and give each task a name from scratch in the new image.

[0006] The present invention has been made in view of the above problems, and aims to provide an information processing device, a control method for the information processing device, and a program that can quickly generate new work procedure information even when a change occurs in the work procedure, for example, when a task is added or deleted. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing device of the present invention is characterized by comprising: an acquisition means for acquiring work procedure information relating to the procedures of a plurality of tasks, a reference image of each of the tasks, and a new image of a new task; an identification means for identifying the new task contained in the new image based on the similarity between the new image and the reference image; and an information processing means for generating new work procedure information based on the new task identified by the identification means and the work procedure information. Effect of the Invention

[0008] According to the present invention, even when a change occurs in the work procedure, for example, when a task is added or deleted, new work procedure information can be quickly generated. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an external view showing an example of the external appearance of an information processing system. [Diagram 2] 2 is a block diagram showing an example of a hardware configuration of the information processing system shown in FIG. 1. [Diagram 3]4 is a flowchart showing a process executed by the information processing device. [Figure 4] 4 is a flowchart showing the process in step S0103 (subroutine) of the flowchart shown in FIG. [Diagram 5] 5 is a flowchart showing the process in step S0216 (subroutine) of the flowchart shown in FIG. [Figure 6] 4 is a flowchart showing the process in step S0104 (subroutine) of the flowchart shown in FIG. [Figure 7] FIG. 13 is a diagram showing an example of a first screen in a mode in which a work procedure update process is executed. [Figure 8] FIG. 13 is a diagram showing an example (modification) of a first screen in a mode for executing a work procedure update process. [Figure 9] FIG. 13 is a diagram showing an example of a second screen in a mode in which a work procedure update process is executed. [Figure 10] FIG. 13 is a diagram showing an example (modification) of a second screen in a mode for executing a work procedure update process. [Figure 11] FIG. 13 is a diagram showing an example of a screen in a mode for executing an operation error detection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments. For example, each part constituting the present invention can be replaced with any configuration that can exert the same function. In addition, any configuration may be added.

[0011] FIG. 1 is an external view showing an example of the external appearance of an information processing system. FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing system shown in FIG. 1. As shown in FIGS. 1 and 2, the information processing system 1 has an information processing device 2 capable of information processing, a camera 31 as an image capturing means 3 for capturing an image of a worker WK performing various tasks, a display (monitor) 41 as a display means 4 for displaying images, etc., and a keyboard 51 and a mouse 52 as an operation means 5 for performing input operations, etc., which are connected to each other so as to be able to communicate with each other. The camera 31 is a digital camera capable of capturing moving images. The information processing device 2 is a computer (server) having a control means 21 and a storage means 22. The control means 21 has, for example, a CPU, and can control the operation of the image capturing means 3, the display means 4, and the operation means 5. The control means 21 may further have a GPU, etc. The storage means 22 has, for example, a HDD, a ROM, a RAM, etc. Furthermore, the storage means 22 stores various programs, images captured by the camera 31, etc. The various programs include, for example, a program for causing a computer to execute each means of the information processing device 2 (a control method for the information processing device).

[0012] Fig. 3 is a flowchart showing the processing executed by the information processing device. As shown in Fig. 3, in step S0101, a mode for executing various processes is input to the information processing device 2 via the operation means 5. This mode includes a mode for executing a work procedure update process and a mode for executing a work error detection process. The "work procedure update process" is a process for updating (changing) the work procedure up to now (up to the present) to a new work procedure. The "work error detection process" is a process for detecting, i.e., confirming, whether or not the worker WK has been performing work in accordance with the work procedure up to now or the new work procedure.

[0013] In step S0102, the control means 21 judges whether the mode input in step S0101 is a mode for executing the work procedure update process. If it is judged as a result of the judgment in step S0102 that it is a mode for executing the work procedure update process, the process proceeds to step S0103. If it is judged as a result of the judgment in step S0102 that it is not a mode for executing the work procedure update process, the process proceeds to step S0104.

[0014] In step S0103, the control means 21 executes a work procedure update process. In the work procedure update process, work procedure information on the procedure of a plurality of tasks before the change (old procedure), a reference image of each task, and a new image of a plurality of tasks after the change, including a new task, are used. These are acquired in advance (acquisition step) prior to the execution of the work procedure update process, and stored in the storage means 22. The work procedure information includes, for example, a task name (name) associated with each task. This work procedure information is acquired, for example, by an input operation using the operation means 5 (acquisition means). Each image is acquired by shooting using the shooting means 3 (acquisition means). In this embodiment, the reference image and the new image may each be a still image, but are preferably a video. In the case of a video, the task included in the video can be accurately identified.

[0015] In step S0104, the control means 21 executes a work error detection process. As described below, if the work error detection process detects that the work is not being performed according to the work procedure, the fact is notified. In the work error detection process, the worker WK actually performs the work according to the work procedure.

[0016] Fig. 4 is a flowchart showing the process in step S0103 (subroutine) of the flowchart shown in Fig. 3. As shown in Fig. 4, in step S0201, the control means 21 of the information processing device 2 reads out new images (moving images of a new procedure) captured of a new task, which are stored in the storage means 22. As described above, in this embodiment, the new images are moving images, that is, an image group consisting of a plurality of still images.

[0017] In step S0202, the control means 21 repeats the processes of steps S0203 to S0214 for all frames of the new image.

[0018] In step S0203, the control means 21 obtains one frame of an image from the new image.

[0019] In step S0204, the control means 21 initializes the similarity value for the frame acquired in step S0203. Here, "similarity" refers to the similarity of the new image with the reference image. The reference image and the new image each include a worker WK. In this embodiment, the posture of the worker WK in the reference image and the posture of the worker WK in the new image are detected, and the degree of agreement between the two postures is taken as the similarity. Then, in a later process, the control means 21 (identification means) can identify a new task included in the new image based on this similarity (identification process).

[0020] In step S0205, the control means 21 reads out a reference image (moving image of the old procedure) of the work before the change, which is stored in the storage means 22. The reference image is also a moving image, like the new image.

[0021] In step S0206, the control means 21 repeats the processes of steps S0207 to S0211 for all frames of the reference image.

[0022] In step S0207, the control means 21 obtains one frame of an image from the reference image.

[0023] In step S0208, the control means 21 calculates the similarity between the new image acquired in step S0203 and the reference image acquired in step S0207. As described above, in this embodiment, the similarity is the degree of agreement between the posture of the worker WK in the reference image and the posture of the worker WK in the new image, but is not limited to this.

[0024] In step S0209, the control means 21 judges whether the similarity calculated in step S0208 is higher (better) than the best similarity for the new image acquired in step S0203. For example, first, the first work procedure of the new image is compared with the first work procedure of the reference image. In this first comparison, the similarity between the new image and the reference image can be acquired. Next, the first work procedure of the new image is compared with the second work procedure of the reference image. In this second comparison, the similarity between the new image and the reference image can also be acquired, and the similarity is assumed to be higher than the previous similarity. In this case, the similarity acquired in the second comparison is regarded as the best similarity. Such comparison is repeated to search for the best similarity. In addition, if the control means 21 has a GPU, for example, machine learning can be used for the comparison in step S0209. Then, if it is judged that the similarity is higher than the best similarity as a result of the judgment in step S0209, the process proceeds to step S0210. On the other hand, if it is determined in step S0209 that the similarity is not higher than the best similarity, the process proceeds to step S0212.

[0025] In step S0210, the control means 21 updates the best similarity score for the new image to the similarity score calculated in step S0208.

[0026] In step S0211, the control means 21 stores in the storage means 22 the action information (task name) attached to the reference image.

[0027] In step S0212, the control means 21 judges whether or not the best similarity for the new image acquired in step S0203 satisfies the threshold condition, i.e., whether or not the similarity is equal to or greater than the threshold. The threshold is pre-stored in the storage means 22. The threshold can be changed as appropriate by operating the operation means 5. If it is determined in step S0212 that the threshold condition is satisfied, the process proceeds to step S0213. On the other hand, if it is determined in step S0212 that the threshold condition is not satisfied, the process proceeds to step S0214.

[0028] In step S0213, the control means 21 assigns the action information saved in step S0211 to the current new image. Specifically, for example, it is assumed that one scene in the reference image includes "screw tightening" and one scene in the new image also includes "screw tightening". At this time, a task name has not yet been assigned to the one scene "screw tightening" in the new image. And it is assumed that both "screw tightening" have the best similarity. In this case, the task name "screw tightening" in the reference image is assigned to "screw tightening" (information on the new task) in the new image. Thus, in step S0213, first, the control means 21 identifies a new task included in the new image whose best similarity satisfies the threshold condition. In the above specific example, the identification of this new task corresponds to one scene "screw tightening" in the new image. Next, the control means 21 generates new task procedure information based on this identified new task and the task procedure information in the old procedure (information processing step). In the specific example, the work procedure information in the old procedure corresponds to the work name "screw tightening" in the reference image. Also, generating new work procedure information corresponds to assigning the work name "screw tightening" in the reference image to "screw tightening" in the new image.

[0029] In step S0214, the control means 21 does not assign an action to the current new image. Specifically, the control means 21 does not assign a specific action to the current new image, and the action is manually assigned in a later process.

[0030] In step S0215, the control means 21 judges whether or not the confirmation button 0505 (see FIG. 7) for the assignment operation has been pressed (operated). If it is judged in step S0215 that the button has been pressed, the process proceeds to step S0216. On the other hand, if it is judged in step S0215 that the button has not been pressed, the process remains in step S0215 and waits.

[0031] In step S0216, the control means 21 (verification means) checks and modifies the actions assigned in steps S0213 and S0214, that is, verifies the accuracy of the new work procedure information. In this assigned action checking process, for example, an already assigned action is changed to another action, an action is assigned to an unassigned part, and a separator position of the action is changed and a new action is added.

[0032] Fig. 5 is a flowchart showing the process in step S0216 (subroutine) of the flowchart shown in Fig. 4. As shown in Fig. 5, in step S0301, the control means 21 reads out a new moving image and the action information currently assigned to the new moving image.

[0033] In step S 0302 , the control means 21 reads out a list of operation information stored in the storage means 22 .

[0034] In step S0303, the control means 21 repeats the processes of steps S0304 to S0321 until a registration work end button 0703 (see FIG. 9) is pressed. The registration work end button 0703 is a button pressed when a user such as a worker WK completes the action confirmation and further completes the action assignment work.

[0035] In step S0304, the control means 21 accepts operations on the action name button 0701, the action addition button 0702, the registration work end button 0703, the split button 0707, the merge button 0708, and the seek bar 0706 (see FIG. 9) as input operations. Note that the action name button 0701, the action addition button 0702, the registration work end button 0703, the split button 0707, the merge button 0708, and the seek bar 0706 are displayed on the display 41 (display means 4) under the control of the control means 21 (display control means).

[0036] In step S0305, the control means 21 judges whether or not the input in step S0304 was a click (press) of the action name button 0701. If it is judged as a result of the judgment in step S0305 that the action name button 0701 was clicked, the process proceeds to step S0306. On the other hand, if it is judged as a result of the judgment in step S0305 that the action name button 0701 was not clicked, the process proceeds to step S0307.

[0037] In step S0306, the control means 21 assigns the action corresponding to the action name button 0701 determined in step S0305 to the current section specified by the arrow 0709 of the seek bar 0706. Hereinafter, a group of images to which the same action is assigned for one or more consecutive frames is referred to as a "section." Note that an unassigned action is also treated as one type of action.

[0038] In step S0307, ​​the control means 21 judges whether or not the input in step S0304 was a click of the add action button 0702. If it is judged as a result of the judgment in step S0307 that the input was a click of the add action button 0702, the process proceeds to step S0308. On the other hand, if it is judged as a result of the judgment in step S0307 that the input was not a click of the add action button 0702, the process proceeds to step S0313.

[0039] In step S0308, the control means 21 displays a pop-up 0801 (see FIG. 10) on the display 41 (display means 4). The pop-up 0801 includes an action name input section 0802, an OK button 0803, and a cancel button 0804. The action name input section 0802 is an operation means for inputting a new action name (action name) as information on a new action, for example, when a new action has not been specified. This allows a desired new action name to be entered in the action name input section 0802. The OK button 0803 is pressed when deciding on the new action name entered in the action name input section 0802. The cancel button 0804 is pressed when canceling the input of the new action name into the action name input section 0802.

[0040] In step S0309, the control means 21 accepts the new work name input on the pop-up 0801 displayed in step S0308.

[0041] In step S0310, the control means 21 accepts an operation on the OK button 0803 or the cancel button 0804 in the pop-up 0801 as an input operation.

[0042] In step S0311, the control means 21 judges whether or not the input in step S0310 is a click of the OK button 0803. If it is judged as a result of the judgment in step S0311 that the input is a click of the OK button 0803, the process proceeds to step S0312. On the other hand, if it is judged as a result of the judgment in step S0311 that the input is not a click of the OK button 0803, the process proceeds to the next step of step S0312.

[0043] In step S0312, the control means 21 registers a new action with the action name input in step S0309. This registration result is stored in the storage means 22. Furthermore, when a list of action information is displayed on the display 41, the control means 21 reflects (adds) the registration result to the list of action information, and updates the list of action information on the display 41.

[0044] In step S0313, the control means 21 judges whether or not the input in step S0304 was a click of the split button 0707. If it is judged as a result of the judgment in step S0313 that the input was a click of the split button 0707, the process proceeds to step S0314. On the other hand, if it is judged as a result of the judgment in step S0313 that the input was not a click of the split button 0707, the process proceeds to step S0315.

[0045] In step S0314, the control means 21 divides the section specified by the seek bar 0706 into left and right sections based on the current position of the arrow 0709 of the seek bar 0706. The left and right sections after division become separate sections, but the same operations as those of the original section are assigned to them.

[0046] In step S0315, the control means 21 judges whether or not the input in step S0304 was a click of the Merge button 0708. If it is judged as a result of the judgment in step S0315 that the input was a click of the Merge button 0708, the process proceeds to step S0316. On the other hand, if it is judged as a result of the judgment in step S0315 that the input was not a click of the Merge button 0708, the process proceeds to step S0318.

[0047] In step S0316, the control means 21 judges whether the left and right partitions are different partitions via the arrow 0709 of the seek bar 0706. As described above, a "partition" is a group of images to which the same action is assigned for one or more consecutive frames. Then, if it is judged in step S0316 that the partitions are different partitions, the process proceeds to step S0317.

[0048] In step S0317, the control means 21 performs a merging process to merge the left and right partitions into one partition. The control means 21 also assigns the action that was assigned to the earlier partition of the left and right partitions to the merged partition.

[0049] If it is determined in step S0316 that the sections are not separate, the process proceeds to the next step, step S0317. In the next step, the left section and the right section are the same section, so that the left section and the right section cannot be integrated into one section. Therefore, for example, it is confirmed that the left section is "section A" and the right section is "section B". It does not matter what type of operation is assigned to section A and section B. In this way, in step S0316, it is determined whether the arrow 0709 of the seek bar 0706 is located at the boundary between the left section and the right section, which are separate sections.

[0050] In step S0318, the control means 21 judges whether or not the input in step S0304 is an operation of the seek bar 0706. If it is judged as an operation of the seek bar 0706 as a result of the judgment in step S0318, the process proceeds to step S0319. On the other hand, if it is judged as not an operation of the seek bar 0706 as a result of the judgment in step S0318, the process proceeds to the next step of step S0321.

[0051] In step S0319, the control means 21 moves the position of the arrow 0709 of the seek bar 0706 to the position operated in step S0318.

[0052] In step S 0320 , the control means 21 changes the image 0704 displayed on the display 41 to an image corresponding to the current position of the arrow 0709 of the seek bar 0706 .

[0053] In step S0321, the control means 21 displays the action assigned to the image displayed in step S0320 in the action display position 0705 of the current image.

[0054] By the above-mentioned process, it is possible to confirm that the allocation in step S0213 and step S0214 is correct. And, in the information processing device 2 configured as above, even if a change occurs in the work procedure, for example, when a task is added or deleted, accurate new work procedure information that serves as a reference for the work, that is, new work procedure information with correct allocation, can be quickly generated and stored in the storage means 22. In particular, in the information processing system 1 (information processing device 2), if there is any task in the work video of the new process that is similar to a task in the old procedure information (reference video), the task name of the task in the old procedure information (reference video) can be assigned to the similar task in the work video of the new process. In this way, the task name of the similar task in the work video of the new process is quickly set (determined). Also, the worker WK, the supervisor, or the like can manually assign a task name to the task not similar to the task in the old procedure information (reference video) among the tasks in the work video of the new process. In this way, the task name of the task in the new process can be set regardless of whether the task in the work video of the new process is similar to the task in the old procedure information (reference video). The work name can then be used in the work error detection process.

[0055] FIG. 6 is a flowchart showing the process in step S0104 (subroutine) of the flowchart shown in FIG. 3. In step S0401, the control means 21 reads out work procedure information (new work procedure information) from the storage means 22. For example, in the case of an assembly operation for assembling parts, the work procedure information includes step 1 "take out a screwdriver", step 2 "take out a screw", step 3 "tighten the screw", etc. The work procedure information also includes the start time of each operation. For example, the start time of step 1 "take out a screwdriver" can be 0 seconds after the start of the part assembly operation. The start time of step 2 "take out a screwdriver" can be 25 seconds after the start of the part assembly operation.

[0056] In step S0402, the control unit 21 reads out a video that is to be subjected to the work error detection. This target video may be, for example, a video that has already been stored in the storage unit 22, or a video that is being captured by the imaging unit 3.

[0057] In step S0403, the control means 21 repeats the processes of steps S0404 to S0414 while the test end button 0906 (see FIG. 11) is not pressed.

[0058] In step S0404, the control means 21 accepts an operation on the work start button 0904 (see FIG. 11) as an input operation.

[0059] In step S0405, the control means 21 sets the current time as the start time of the work.

[0060] In step S0406, the control means 21 reads out the next frame of the target moving image, and repeats the processing of steps S0407 to S0413 while the work end button 0905 is not pressed.

[0061] In step S0407, the control means 21 obtains the current frame of the target moving image.

[0062] In step S0408, the control means 21 recognizes the action in the current frame, that is, the task, based on the image features of the current frame and the image features of the past frame. For this recognition method, for example, the above-mentioned method of identifying a new task based on similarity can be used.

[0063] In step S0409, the control means 21 acquires past action recognition results (task information). This, together with the current action recognition result, makes it possible to accurately determine whether or not there is an error, such as an omission to perform a task.

[0064] In step S0410, the control means 21 (determination means) determines whether or not there is a mistake in the work, that is, whether the work is correct or incorrect, based on new work sequence information (including time information), the current action recognition result, and past performed work information.

[0065] In step S0411, the control means 21 judges whether or not an alert, i.e., a notification that an operation error has occurred, is necessary based on the judgment result of step S0410. If it is judged as a result of the judgment in step S0411 that an alert notification is necessary, the process proceeds to step S0412. On the other hand, if it is judged as a result of the judgment in step S0411 that an alert notification is not necessary, the process proceeds to step S0413.

[0066] In step S0412, the control means 21 issues an alert notification. This notification method is not particularly limited, and examples include a method of displaying a pop-up on the display 41, a method of emitting a sound from a speaker, and a method of emitting a red light from a signal lamp. In this way, the display 41, the speaker, the signal lamp, and the like function as notification means for notifying the result of the determination in step S0410. In addition, these notification means are not limited to notification that a work error has occurred, and may, for example, be notification that no work error has occurred.

[0067] In step S0413, the control means 21 updates the past task information based on the action recognition result and task error judgment for the current frame. This makes it possible to use the updated task information when making task error judgments in the next and subsequent frames.

[0068] In step S0414, the control means 21 sets the current time as the work end time.

[0069] When a work mistake occurs as described above, the information processing device 2 can accurately detect the work mistake and notify the worker WK of the detection result. This allows the worker WK to correct the work mistake. As described above, in the work procedure update process, it is determined which work in the procedure information (reference video) each work in the actual work video is similar to, and the work name of each work in the procedure information is given to each work in the actual work video. Then, in the work mistake detection process, it is confirmed whether or not the sequence of each work name in the actual work video matches the sequence of each work name in the procedure information (reference video), and whether or not there is any omission in each work name given to the actual work video by comparing with each work name in the procedure information (reference video). Based on this confirmation result, a work mistake can be detected. For example, if a missing work name is identified, it can be detected that the work of the work name is missing.

[0070] FIG. 7 is a diagram showing an example of a first screen in a mode for executing a work procedure update process. The work procedure update process mode screen 0500 shown in FIG. 7 includes an image 0501, an image 0502, an old procedure action list 0503, a new procedure action list 0504, and a confirmation button 0505. The image 0501 is an image of one frame in a video of an old procedure, and the image 0502 is an image of one frame in a video of a new procedure. The image 0501 is an image determined to be most similar to the image 0502, and an action name (task name) 0506 is displayed on the upper left side. By displaying the image 0502, it is possible to grasp that an analysis of an action to be assigned to the image 0502 is being performed. In this way, the information processing system 1 can display the image 0501, which is a reference image, and the image 0502, which is a new image, below the image 0501, on the display 41. This makes it possible to grasp that an image similar to the image 0502 exists and that the image is the image 0502.

[0071] The old procedure action list 0503 shows a list of action names of the old procedure, for example, action name 0511 of "move base", action name 0512 of "take screwdriver", action name 0513 of "take screw", action name 0514 of "tighten screw", action name 0515 of "pointing and checking". The new procedure action list 0504 shows a list of action names assigned to one or more frames of the new procedure video in the analysis up to now, for example, action name 0521 of "move base", action name 0522 of "take screwdriver", action name 0523 of "take screw", action name 0524 of "tighten screw". In this way, the information processing system 1 can display the action name of the old procedure and the action name of the new procedure to the right of the action name of the old procedure on the display 41. This allows the action name of the old procedure to be compared with the action name of the new procedure. For example, in the case of FIG. 7, it can be seen that the old procedure operation list 0503 contains operation name 0515, but the new procedure operation list 0504 does not contain operation name 0515.

[0072] The Confirm button 0505 is a button that is pressed after the analysis of the video of the new procedure is completed. By pressing the Confirm button 0505, it is possible to transition to a page where the assigned actions can be confirmed or changed, new actions can be added, etc.

[0073] Fig. 8 is a diagram showing an example (modified example) of the first screen in the mode for executing the work procedure update process. Here, differences from Fig. 7 described above will be mainly described, and similar matters will not be described. The work procedure update process mode screen 0500 includes an image 0601 instead of the image 0501. The image 0601 is displayed when there is no similar old procedure video (image 0501) for the image 0502. In the image 0601, the action name 0506 is left blank. Furthermore, when the image 0601 is displayed, the image 0502 is in an unassigned action state.

[0074] FIG. 9 is a diagram showing an example of a second screen in a mode for executing a work procedure update process. The work procedure update process mode screen 0700 shown in FIG. 9 includes an action name button 0701, an action addition button 0702, a registration operation end button 0703, an image 0704, a seek bar 0706, a split button 0707, and an integration button 0708. The action name button 0701 is operated when allocating an action to a section selected by the seek bar 0706. The action name button 0701 includes, for example, an action name 0711 indicating "move the base", an action name 0712 indicating "take a screwdriver", an action name 0713 indicating "take a screw", an action name 0714 indicating "tighten the screw", an action name 0715 indicating "point and check", and an action name 0716 indicating "new action". The action addition button 0702 is operated when adding a new action. By operating the action addition button 0702, a pop-up 0801 (see FIG. 10) is displayed. A registration operation end button 0703 is operated when operation registration is completed and confirmation is to be ended.

[0075] An image 0704 is an image of one frame of the new procedure video. This image 0704 is an image at a position specified by an arrow 0709 of a seek bar 0706. In an action display position 0705 below the image 0704, the name of the action (task name) assigned to the image 0704 is displayed. Note that the action display position 0705 may also display an indication that the action is unassigned. The seek bar 0706 is operated when determining which frame of the image / action of the new procedure video is to be displayed. The seek bar 0706 is divided into groups of images (sections) to which the same action is assigned for one or more consecutive frames. This seek bar 0706 is previously set into sections for consecutive actions assigned in the initial state, and the sections can be changed. The seek bar 0706 is colored, for example, in red (hatched portion inclined from the upper left to the lower right), green (hatched portion inclined from the upper right to the lower left), or orange (cross-hatched portion). Red indicates that the action is unassigned and that the action name button 0701 must be operated to assign the action. Green indicates that the action has been assigned manually. Orange indicates that the action has been assigned by automatic action assignment in the work procedure update mode. When changing an automatically assigned action, the arrow 0709 of the seek bar 0706 is moved to the section to which the automatically assigned action is to be changed, and the action name button 0701 is operated. This changes the color of the section to which the arrow 0709 is moved to green. In this way, the information processing system 1 can display on the display 41 a time period in which an action has been assigned automatically or manually (a new task has been identified) and a time period in which an action has not been assigned (no new task has been identified) in different colors. This allows the user to quickly grasp the action assignment status by visually checking the seek bar 0706.

[0076] The split button 0707 is an operating means for splitting the section specified by the seek bar 0706 into left and right sections based on the current position of the seek bar 0706. After splitting, the left and right sections become separate sections, but the same action as that of the original section is assigned to them. The combine button 0708 is an operating means for merging the left and right sections specified by the seek bar 0706 into one section. As described above, the action assigned to the earlier section is assigned to the section after the merger. This makes it possible to organize sections by splitting and merging them, which contributes to the generation of work procedure information. In the present embodiment, the information processing system 1 can display both a split button 0707 for splitting a partition and an integration button 0708 for integrating partitions, but is not limited to this. For example, depending on the type of work, one of the split button 0707 and the integration button 0708 may be displayed.

[0077] FIG. 10 is a diagram showing an example (modification) of the second screen in the mode for executing the work procedure update process. Here, the differences from FIG. 9 described above will be mainly described, and the description of the similar matters will be omitted. The work procedure update process mode screen 0700 shown in FIG. 10 includes a pop-up 0801. The pop-up 0801 is displayed by operating the action addition button 0702. As described above, the pop-up 0801 includes an action name input section 0802, an OK button 0803, and a cancel button 0804. By inputting a new work name in the action name input section 0802 and operating the OK button 0803, the new work name is added to the action name 0716. This allows the new work name (information on a new work) to be reflected in new work procedure information, and the work procedure information can be used for work error detection processing. Note that when the cancel button 0804 is operated, the total number of buttons of the action name button 0701 does not change.

[0078] Fig. 11 is a diagram showing an example of a screen in a mode for executing a work error detection process. A screen 0900 shown in Fig. 11 includes an image 0901, a work list 0902, an alert 0903, a work start button 0904, a work end button 0905, and an inspection end button 0906. The image 0901 is an image in which an action recognition result (for example, "pointing and checking") is superimposed on the current frame of a video being analyzed. The work list 0902 is a list of work names, and for example, an action name 0911 of "moving the base", an action name 0912 of "taking out a screwdriver", an action name 0913 of "taking out a screw", an action name 0914 of "tightening a screw", and an action name 0915 of "pointing and checking" are listed in the order of the work.

[0079] The alert 0903 indicates the alert state of each operation by color. For example, green (hatched portion inclined from upper left to lower right) indicates a state in which the operation (task) is completed normally. In FIG. 9, the operation name 0911 and the operation name 0912 are green alerts. Red (hatched portion inclined from upper right to lower left) indicates a state in which the operation has been omitted. In FIG. 9, the operation name 0913 is a red alert. Yellow (cross-hatched portion) indicates a state in which the operation was performed outside the scheduled time. In FIG. 9, the operation name 0914 is a red alert. White indicates a state before the operation was performed. In FIG. 9, the operation name 0915 is a white alert. By operating the work start button 0904, the start time of a series of operations is recognized, and the analysis of the operations begins. By operating the work end button 0905, the end time of a series of operations is recognized, and the analysis of the operations ends. The inspection end button 0906 is operated when the execution of the operation error detection process is ended.

[0080] From the screen 0900 as described above, the status of each task can be ascertained.

[0081] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiment to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0082] 1. Information Processing Systems 2. Information processing equipment 3. Shooting Method 4 Display means 5 Operating means 21 Control Means 22 Memory means

Claims

1. An acquisition means for acquiring work procedure information relating to the procedures of a plurality of tasks, a reference image of each of the tasks, and a new image of a new task; an identification means for identifying the new work included in the new image based on a similarity between the new image and the reference image; an information processing device that generates new work procedure information based on the new work identified by the identification device and the work procedure information,

2. 2 . The information processing apparatus according to claim 1 , wherein, when generating the new work procedure information, the information processing means sets information about the new work identified by the identifying means based on the work procedure information.

3. The work procedure information includes a name of each of the works, 3. The information processing apparatus according to claim 2, wherein said information processing means assigns one of the names of said tasks as the information regarding said new task.

4. 2. The information processing apparatus according to claim 1, further comprising an operation unit for performing an operation to input information about a new task when the new task is not specified by the specifying unit.

5. 5. The information processing apparatus according to claim 4, wherein said information processing means reflects the information on the new work in the new work procedure information.

6. 2. The information processing apparatus according to claim 1, further comprising a display control means for controlling display of the work procedure information and the new work procedure information on a display means.

7. 7. The information processing apparatus according to claim 6, wherein the display control means controls the display means to display the work procedure information and the new work procedure information in a manner that allows them to be compared.

8. 2. The information processing apparatus according to claim 1, further comprising a display control means for controlling display of the reference image and the new image on a display means.

9. 9. The information processing device according to claim 8, characterized in that the display control means controls the display means to display a time period during which the identification means identified the new work and a time period during which the identification means did not identify the new work.

10. The new image is an image group consisting of a plurality of images, 10. The information processing apparatus according to claim 9, further comprising an operation unit for performing at least one of an operation for dividing the group of images and an operation for integrating the group of images.

11. each of the reference image and the new image includes a worker; 2 . The information processing apparatus according to claim 1 , wherein the specifying means determines the degree of similarity between the posture of the worker in the reference image and the posture of the worker in the new image.

12. 2. The information processing apparatus according to claim 1, further comprising a verification unit for verifying the accuracy of the new work procedure information.

13. 2. The information processing apparatus according to claim 1, further comprising a determination unit that determines whether new tasks including the new task are correct or incorrect based on the new task procedure information.

14. 14. The information processing apparatus according to claim 13, further comprising a notification unit that notifies a result of the determination made by said determination unit.

15. The information processing apparatus according to claim 1 , wherein the reference image and the new image are each a moving image.

16. A method for controlling an information processing device capable of processing information, comprising: an acquisition step of acquiring work procedure information relating to the procedures of a plurality of tasks, a reference image of each of the tasks, and a new image of a new task; an identifying step of identifying the new work included in the new image based on a similarity between the new image and the reference image; a step of generating new work procedure information based on the new work identified in the identification step and the work procedure information,

17. 2. A program for causing a computer to execute each of the means of the information processing apparatus according to claim 1.

Citation Information

Patent Citations

  • Data generator, data generation system, and method for generating data

    JP2021012586A