Information processing device, improvement proposal method, and program

Abstract

An information processing device comprises: a video acquisition unit that acquires a plurality of videos obtained by capturing images of work states of a plurality of workers; a work recognition unit that analyzes the plurality of videos and recognizes a plurality of types of work performed by each worker; an identification unit that identifies one or more types of work of a first worker to be improved from among the plurality of types of work on the basis of recognition results from the work recognition unit; a generation unit that generates proposal information indicating an improvement plan for each of the identified one or more types of work; and an output unit that outputs the proposal information to the first worker.

Description

Information processing device, improvement suggestion method, and program

[0001] The present disclosure relates to an information processing device, an improvement suggestion method, and a program.

[0002] 2. Description of the Related Art Conventionally, measures have been taken at production sites of various products, such as factories, to improve the efficiency of production work, such as manufacturing, processing, and assembly of products, carried out by workers.

[0003] For example, Patent Literature 1 discloses a technology in which, each time a worker performing production work on a certain product updates the shortest time for that production work, an optimal work procedure is extracted from a video of the production work situation using a trained model, and by comparing it with the work procedures of other workers, a proposal for improving the work procedures of the other workers is presented. Note that the trained model has learned about the standard work time and suitability of the work for the production work.

[0004] Japanese Patent Application Laid-Open No. 2022-21437

[0005] However, with the above-described conventional technology, a trained model cannot be prepared unless the tasks requiring improvement are known. In other words, the above-described conventional technology is based on the premise that the tasks requiring improvement are known. Therefore, it is difficult to apply the above-described conventional technology in cases where it is difficult to identify tasks requiring improvement, such as in on-site work where a wide variety of tasks must be performed and new types of tasks frequently arise.

[0006] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device, an improvement suggestion method, and a program that can support the improvement of work even when the work requiring improvement is unknown.

[0007] One aspect of the information processing device disclosed herein includes an image acquisition unit that acquires multiple videos capturing the work situations of multiple workers; a task recognition unit that analyzes the multiple videos and recognizes multiple types of tasks performed by each worker; an identification unit that identifies one or more types of tasks from the multiple types of tasks to be improved by a first worker based on the recognition results of the task recognition unit; a generation unit that generates proposal information indicating improvement plans for each of the one or more types of tasks identified; and an output unit that outputs the proposal information to the first worker.

[0008] One aspect of the improvement suggestion method disclosed herein includes a video acquisition step in which a video acquisition unit acquires multiple videos capturing the work situations of multiple workers; a task recognition step in which a task recognition unit analyzes the multiple videos and recognizes multiple types of tasks performed by each worker; an identification step in which an identification unit identifies one or more types of tasks to be improved by a first worker from among the multiple types of tasks based on the recognition results of the task recognition unit; a generation step in which a generation unit generates proposal information indicating improvement plans for each of the one or more types of tasks identified; and an output step in which an output unit outputs the proposal information to the first worker.

[0009] One aspect of the program disclosed herein is for causing a computer to execute the following steps: an image acquisition step of acquiring multiple videos capturing the work situations of multiple workers; an operation recognition step of analyzing the multiple videos to recognize multiple types of work performed by each worker; an identification step of identifying one or more types of work from the multiple types of work that are to be improved by a first worker based on the recognition results of the operation recognition unit; a generation step of generating proposal information indicating improvement plans for each of the one or more types of work that have been identified; and an output step of outputting the proposal information to the first worker.

[0010] According to the present disclosure, it is possible to provide an information processing device, an improvement suggestion method, and a program that can support the improvement of an operation even when the operation requiring improvement is unknown.

[0011] FIG. 1 is a block diagram showing an example of the system configuration of a work analysis system according to this embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing device and a worker terminal according to this embodiment. FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. FIG. 4 is a diagram showing an example of an image captured by an imaging device according to this embodiment and acquired by an image acquisition unit according to this embodiment. FIG. 5 is a diagram showing an example of a work history according to this embodiment. FIG. 6 is a diagram showing an example of a recommendation screen according to this embodiment. FIG. 7 is a diagram showing an example of a work history according to this embodiment. FIG. 8 is a diagram showing an example of a recommendation screen including proposal information according to this embodiment. FIG. 9 is a flowchart showing an example of work recognition processing performed by the information processing device according to this embodiment. FIG. 10 is a sequence diagram showing an example of a condition setting processing performed by the work analysis system according to this embodiment. FIG. 11 is a sequence diagram showing an example of an improvement proposal processing performed by the work analysis system according to this embodiment.

[0012] Hereinafter, an embodiment of the present disclosure (hereinafter simply referred to as "the present embodiment") will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiment. Furthermore, the following embodiment and modified examples can be combined as appropriate.

[0013] The work analysis system of this embodiment will be described below using warehouse work performed in a logistics center (warehouse) that delivers parts (repair parts) as an example.

[0014] Generally, logistics centers that handle parts handle a huge variety of parts and tools, and the parts they handle change over frequently. Furthermore, logistics centers must perform warehouse tasks such as inspection, receiving, picking, packaging, and removal for each part. In particular, the tasks involved in inspection, receiving, picking, and packaging often differ for each part. As such, logistics centers that deliver parts not only perform a wide variety of warehouse tasks, but also frequently encounter new types of tasks (for parts), making it difficult to identify tasks that require improvement. In particular, warehouse tasks often involve simple tasks that can be performed at the discretion of the workers, making it difficult to identify tasks that require improvement.

[0015] An example of a task that requires improvement is one that takes a worker longer to complete than other workers, but where the worker can reduce the time required by understanding the optimal way to do the task.

[0016] The work analysis system of this embodiment analyzes footage of the work being performed by each worker, uses the analysis results to identify tasks that require improvement by the worker, and makes suggestions to the worker about improving the identified tasks, thereby supporting the improvement of tasks that were initially unclear (unknown) and require improvement.

[0017] The following describes in detail the work analysis system of this embodiment, taking as an example the packaging work at a logistics center (warehouse) that handles parts. However, the work that the work analysis system of this embodiment targets is not limited to this, and can target any of the warehouse work described above. It can also target general work at logistics centers that handle finished products, rather than parts.

[0018] Packaging refers to wrapping each individual part individually, such as wrapping parts in cushioning material and then packing them in an outer box. The cushioning material, outer box, tape used when wrapping in cushioning material, and tape used to seal the lid of the outer box vary depending on the type (size, shape, etc.) of parts to be individually packaged. Generally, parts are not stored individually at a logistics center, but are stored in batches of a certain number. For this reason, when shipping parts, the parts to be shipped are picked and individually packaged, and then packed together with other individually packaged parts and transported.

[0019] First, the configuration of the work analysis system of this embodiment will be described.

[0020] 1 is a block diagram showing an example of the system configuration of this embodiment of the work analysis system 1. As shown in Fig. 1, the work analysis system 1 includes multiple image capture devices 10-1 to 10-n (n is a natural number), an information processing device 20, a generative model 30, and multiple worker terminals 40-1 to 40-m (m is a natural number).

[0021] The multiple image capture devices 10-1 to 10-n, the information processing device 20, the generative model 30, and the multiple worker terminals 40-1 to 40-m are connected via a network 2. The network 2 can be realized by, for example, at least one of the Internet and a local area network (LAN). The network 2 may be a wired network, a wireless network, or a mixture of wired and wireless networks.

[0022] In the following description, when it is not necessary to distinguish between the image capture devices 10-1 to 10-n, they may be simply referred to as image capture devices 10. Similarly, when it is not necessary to distinguish between the worker terminals 40-1 to 40-m, they may be simply referred to as worker terminals 40.

[0023] The image capturing devices 10-1 to 10-n are installed inside and outside the warehouse, capture images of the work status of workers performing work inside and outside the warehouse, and output the captured images to the information processing device 20. Each image capturing device 10 is installed inside or outside the warehouse so as to capture, for example, the work area where the workers perform their work (for example, a workbench where individual packaging work is performed). Examples of the image capturing device 10 include a video camera.

[0024] The information processing device 20 analyzes multiple images captured by multiple image capture devices 10 and recognizes various tasks performed by workers from each image. The information processing device 20 also identifies tasks that require improvement from the recognized tasks and makes improvement suggestions for the identified tasks using, for example, a generative model 30. The information processing device 20 can be realized, for example, by a server device or a cloud service.

[0025] The generative model 30 is a natural language processing model known as generative AI (generative artificial intelligence), and provides a service using natural language processing. In this embodiment, the generative model 30 is described taking as an example an AI chat service in which large language models (LLMs) are fine-tuned, but is not limited to this. For example, the generative model 30 may be realized by small language models (SLMs).

[0026] The worker terminal 40 is a terminal device used by a worker, and examples thereof include a tablet terminal, a smartphone, or a PC (Personal Computer). The worker terminal 40 is used, for example, by the worker to check improvement suggestions made by the information processing device 20. The worker terminal 40 may be held by the worker himself or may be installed in a workplace where the worker performs work.

[0027] FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing device 20 and the worker terminal 40 according to this embodiment.

[0028] First, a description will be given of the hardware configuration of the information processing device 20. As shown in Fig. 2, the information processing device 20 includes a control device 21, a main memory device 22, an auxiliary memory device 23, a communication device 24, and various buses 29. The control device 21, the main memory device 22, the auxiliary memory device 23, and the communication device 24 are connected via the various buses 29. As such, the information processing device 20 of this embodiment has a general hardware configuration using a normal computer.

[0029] The control device 21 controls the overall operation of the information processing device 20. Examples of the control device 21 include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), but are not limited to these. There may be any number of CPUs or GPUs as long as there is one or more, and they may be single-core or multi-core.

[0030] Examples of the main storage device 22 include, but are not limited to, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores various programs, such as a program for controlling the information processing device 20, a program for recognizing the work of a worker from video, and a program for making improvement suggestions. The RAM is used as a work area when the control device 21 performs various controls based on the programs stored in the ROM.

[0031] The auxiliary storage device 23 stores various data, such as the various programs described above and a recognition model (trained interaction recognition model) for recognizing the worker's work from video. The various programs described above may be stored in at least one of the main storage device 22 and the auxiliary storage device 23. Examples of the auxiliary storage device 23 include, but are not limited to, existing storage devices capable of magnetic, electrical, or optical storage, such as a hard disk drive (HDD), a solid state drive (SSD), and a digital versatile disc (DVD). The auxiliary storage device 23 may be built into the information processing device 20 or externally connected to the information processing device 20 via an interface such as a universal serial bus (USB). The auxiliary storage device 23 may also be a network-attached storage (NAS) connected via a network, such as a LAN or a wide area network (WAN).

[0032] The communication device 24 is used to communicate with the worker terminal 40 and the like via the network 2. Examples of the communication device 24 include, but are not limited to, a communication device for a wired LAN and a wireless communication device for a wireless LAN.

[0033] In addition to the above configuration, the information processing device 20 may further include hardwired circuits such as an IC (Integrated Circuit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array) that are specific to the information processing device 20.

[0034] Next, the hardware configuration of the worker terminal 40 will be described. As shown in Fig. 2, the worker terminal 40 includes a control device 41, a main memory device 42, an auxiliary memory device 43, a communication device 44, an input device 45, a display device 46, and various buses 49. The control device 41, the main memory device 42, the auxiliary memory device 43, the communication device 44, the input device 45, and the display device 46 are connected via the various buses 49. As such, the worker terminal 40 of this embodiment has a general hardware configuration such as a PC or a tablet terminal.

[0035] The control device 41 controls the overall operation of the worker terminal 40. The control device 41 is realized in the same manner as the control device 21.

[0036] The method for realizing the main memory device 42 is the same as that for the main memory device 22, and therefore detailed description thereof will be omitted. The ROM of the main memory device 42 stores various programs, such as a program for controlling the worker terminal 40.

[0037] The auxiliary storage device 43 stores the various programs and data described above. The various programs described above may be stored in at least one of the main storage device 42 and the auxiliary storage device 43. The method for realizing the auxiliary storage device 43 is similar to that for the auxiliary storage device 23, and therefore a detailed description thereof will be omitted.

[0038] The communication device 44 is used to communicate with the information processing device 20 and the like via the network 2. The method for realizing the communication device 44 is similar to that of the communication device 24, and therefore a detailed description thereof will be omitted.

[0039] The input device 45 is used for various inputs, selections, and specifications for confirming improvement proposals, and serves as a user interface between the worker and the input device 45. Examples of the input device 45 include, but are not limited to, a keyboard, a mouse, and a touch panel. The input device 45 may be built into the worker terminal 40 or may be externally connected to the input device 45 via an interface such as a USB.

[0040] The display device 46 displays various screens, such as a display screen for improvement proposals, and serves as a user interface between the worker and the display device 46. Examples of the display device 46 include, but are not limited to, various displays such as a liquid crystal display, an organic electroluminescence (EL) display, and a touch panel display. The display device 46 may be a built-in display built into the worker terminal 40, or an external display connected to the worker terminal 40 via a display interface such as HDMI (registered trademark).

[0041] 3 is a block diagram showing an example of the functional configuration of the information processing device 20 according to this embodiment. As shown in FIG. 3, the information processing device 20 includes an image acquisition unit 201, an image storage unit 203, a recognition model storage unit 207, an activity recognition unit 209, an activity history storage unit 211, an input / output control unit 221, a management unit 231, a condition storage unit 233, an identification unit 235, and a generation unit 237. The input / output control unit 221 includes a reception unit 223 and an output unit 225.

[0042] The video acquisition unit 201, the task recognition unit 209, the input / output control unit 221, the reception unit 223, the output unit 225, the management unit 231, the identification unit 235, and the generation unit 237 can be realized, for example, by the control device 21, the main memory device 22, and the communication device 24 described in FIG. 2 . For example, the control device 21 reads various programs stored in the main memory device 22 (ROM) or the auxiliary memory device 23 and loads them into the main memory device 22 (RAM). The control device 21 realizes the above-mentioned functional units by executing various processes in accordance with the loaded programs. Here, the above-mentioned functional units are realized as software, but at least some of the above-mentioned functional units may be realized as hardware. In this case, the functional units realized as hardware may be realized, for example, by the above-mentioned hardwired circuit. Furthermore, any of the above-mentioned functional units may be realized by a combination of software and hardware.

[0043] The image storage unit 203, the recognition model storage unit 207, the work history storage unit 211, and the condition storage unit 233 can be realized by, for example, the auxiliary storage device 23 described with reference to FIG.

[0044] The video acquisition unit 201 acquires a plurality of videos capturing the work situations of a plurality of workers from a plurality of image capture devices 10-1 to 10-n, and stores (accumulates) the videos in the video storage unit 203. FIG. 4 is a diagram showing an example of a video 501 captured by the image capture device 10 of this embodiment and acquired by the video capture unit 201. The video 501 shown in FIG. 4 is a video captured by the image capture device 10 installed on the ceiling, showing a scene in which a worker W is performing individual packaging work on a workbench 51. Specifically, on the workbench 51, the worker W wraps components in cushioning material using tapes 61 and 63 (not shown), packs the cushioning-wrapped components into an outer box 71, and seals the outer box 71 with tape 65 attached to the lid of the outer box 71.

[0045] In this embodiment, the multiple image capture devices 10 are arranged in the warehouse so as to capture images of the work benches where workers perform the packaging work. Accordingly, the video capture unit 201 captures, from each image capture device 10, videos of each worker performing the packaging work, as shown in FIG. 4 , and stores (accumulates) the multiple captured videos in the video storage unit 203. Therefore, the multiple videos stored in the video storage unit 203 are expected to include videos of each worker sequentially performing the packaging work of various parts, such as a video of a scene in which worker A performs the packaging work of part X and then the packaging work of part Y, or a video of a scene in which worker B performs the packaging work of part Z and then the packaging work of part Y.

[0046] The recognition model storage unit 207 stores a recognition model for recognizing the work being performed by the worker from the video stored (accumulated) in the video storage unit 203. The recognition model is, for example, a worker interaction recognition model, and is a trained model generated by a learning device (not shown) learning training data.

[0047] The learning data may be data in which annotations (labels) are added to work videos, such as live-action videos of workers performing individual packaging work or CG videos of CG models (computer graphics) of workers performing individual packaging work. The annotations indicate, for each frame of the work video, whether or not there is interaction between the worker's hand and an object (tool), and if there is interaction, the name of the object with which there is interaction and the action using the object.

[0048] A learning device (not shown) learns such learning data and generates a recognition model that outputs recognition results for each frame of the input video that indicate what object (tool) the worker is using and what action he or she is performing in the individual packaging work.

[0049] The task recognition unit 209 analyzes the multiple videos acquired by the video acquisition unit 201 (stored in the video storage unit 203) to recognize the multiple types of tasks performed by each worker. Specifically, the task recognition unit 209 analyzes the multiple videos to recognize each of the multiple types of tasks performed by each worker on a task-by-task basis. The task recognition unit 209 also recognizes the task time required for each of the multiple types of tasks performed by each worker. The task recognition unit 209 stores the task recognition results as a task history in the task history storage unit 211.

[0050] In this embodiment, the work recognition unit 209 acquires video from the video storage unit 203 and inputs the acquired video into a recognition model stored in the recognition model storage unit 207. The work recognition unit 209 acquires from the recognition model recognition results indicating, for each frame of the video, what object (tool) the worker is using and what action in the individual packaging work he is performing, thereby recognizing the work being performed by the worker shown in the video in work process units and storing this as a work history in the work history storage unit 211. By performing the above processing on all of the video stored in the video storage unit 203, the work recognition unit 209 can break down and recognize each of the multiple types of work performed by each worker into work process units.

[0051] Fig. 5 is a diagram showing an example of a work history in this embodiment. The work history shown in Fig. 5 shows the recognition result of the work of worker A, which is obtained by recognizing the work of worker A from a video of a scene in which worker A performs the work of packaging component X and then the work of packaging component Y, as described above. In addition, in the work history shown in Fig. 5, frames of the same work process are displayed together. In the example shown in Fig. 5, the work history includes the worker, the work type, the start date and time, the end date and time, the frame, the work process, and the frame video.

[0052] The worker refers to the worker performing the task recognized by the task recognition unit 209 ("Worker A" in the example shown in FIG. 5). The task recognition unit 209 may recognize the worker from video using a recognition model, or may set the worker using information such as who will be performing the task and which workbench they will be using, input into the worker terminal 40 before the worker performs the task. In the latter case, the task recognition unit 209 simply acquires the information from the worker terminal 40.

[0053] The task type indicates the type of task recognized by the task recognition unit 209 ("packaging component X" in the example shown in FIG. 5). The task type may be recognized from an image by the task recognition unit 209 using a recognition model, or may be set using information such as component code information input to the worker terminal 40 before the worker performs each task. Generally, before the individual packaging task, code information such as a barcode or two-dimensional code attached to the component to be individually packaged is read by the worker terminal 40, and after the individual packaging task for the component is completed, the worker inputs the completion of the task from the worker terminal 40. Therefore, in the latter case, the task recognition unit 209 only needs to acquire information from the worker terminal 40.

[0054] The start date and time indicates the date and time when the work process started, the end date and time indicates the date and time when the work process ended, and the frame indicates the frame number from the start to the end of the work process. Note that the work time required for the work may be, for example, the total time obtained by adding up the difference between the end date and time and the start date and time for each work process, or may be the total frames required for the work divided by the frame rate (e.g., 60 fps).

[0055] The work process indicates the work process for packaging part X that is recognized from the video using the recognition model by the work recognition unit 209. Specifically, the work process indicates what objects (tools) the worker is using and what actions he or she is performing in the packaging work.

[0056] For example, suppose that during the time period from "5 / 10 12:00:00" to "5 / 10 12:00:05" in the video, worker A is placing cushioning material on a workbench, and an interaction occurs between worker A's hand and the cushioning material. The recognition model has learned learning data in which, in a frame video of the worker placing the cushioning material, an annotation is added that defines the action of placing the cushioning material when the object interacting with the worker's hand is the cushioning material. In this case, using the recognition model, the task recognition unit 209 recognizes the task process during the above time period in the video as an object "cushioning material" and an action "placing."

[0057] For example, suppose that during the time period from "5 / 10 12:00:05" to "5 / 10 12:00:10" in the video, worker A is fixing component X placed on cushioning material with tape α, and an interaction occurs between worker A's hand and the tape α and component X. The recognition model has learned learning data in which, for frame video of the worker fixing component X placed on cushioning material with tape α, an annotation is added that defines an action of fixing component X with tape α when the tape α and component X are objects interacting with the worker's hand. In this case, using the recognition model, the task recognition unit 209 recognizes the task process during the above time period in the video as consisting of the objects "tape α, component X" and "fixing."

[0058] For example, suppose that during the time period from "5 / 10 12:00:10" to "5 / 10 12:00:20" in the video, worker A is wrapping part X in cushioning material and securing it with tape β, and interactions occur between worker A's hand and the tape β and the cushioning material. The recognition model has learned learning data in which, in a frame video of the worker wrapping part X in cushioning material and securing it with tape β, an annotation is added that defines the action of wrapping in cushioning material and securing it with tape β when the objects interacting with the worker's hand are tape β and the cushioning material. In this case, using the recognition model, the task recognition unit 209 recognizes the tasks during the above time period in the video as consisting of the objects "tape β, cushioning material" and the action "wrapping."

[0059] For example, suppose that during the time period from "5 / 10 12:00:20" to "5 / 10 12:00:30" in the video, worker A was assembling an outer box, and interaction occurred between worker A's hand and the outer box. The recognition model has learned learning data in which, in a frame video of the worker assembling the outer box, an annotation is added that defines the action of assembling the outer box when the object interacting with the worker's hand is the outer box. In this case, using the recognition model, the task recognition unit 209 recognizes the task process during the above time period in the video as an object "outer box" and an action "assembly."

[0060] For example, suppose that during the time period from "5 / 10 12:00:30" to "5 / 10 12:00:35" in the video, worker A is attaching a label to an outer box, and an interaction occurs between worker A's hand and the label. Furthermore, suppose that the recognition model has learned learning data in which, in a frame video of the worker attaching a label to an outer box, an annotation is added that defines the action of attaching a label when the object interacting with the worker's hand is a label. In this case, using the recognition model, the task recognition unit 209 recognizes the task process during the above time period in the video as the object "label" and the action "attach."

[0061] For example, suppose that during the time period from "5 / 10 12:00:35" to "5 / 10 12:00:40" in the video, worker A is packing cushioning material wrapped around part X into an outer box, and interactions occur between worker A's hand and the cushioning material and the outer box. The recognition model has learned learning data in which, in a frame video of the worker packing cushioning material wrapped around part X into an outer box, an annotation defining an action of "packing" is added when the objects interacting with the worker's hand are the cushioning material and the outer box. In this case, using the recognition model, the task recognition unit 209 recognizes the objects as "cushioning material, outer box" and the action as "packing" for the task during the above time period in the video.

[0062] For example, suppose that during the time period from "5 / 10 12:00:40" to "5 / 10 12:00:50" in the video, worker A is sticking tape γ to the lid of the outer box and sealing the outer box, and interactions occur between worker A's hand and the tape γ and the outer box. Furthermore, suppose that the recognition model has learned learning data in which, in a frame video of sticking tape γ to the lid of the outer box and sealing the outer box, the objects interacting with the worker's hand are the tape γ and the outer box, and an annotation defining an action of "sealing" is added. In this case, using the recognition model, the task recognition unit 209 recognizes the task process during the above time period in the video as the objects "tape γ, outer box" and the action "sealing."

[0063] It is assumed that before the work process of the object "cushioning material" and the action "place," worker A reads the code information assigned to part X using worker terminal 40, and after the work process of the object "tape γ, outer box" and the action "seal," worker A completes the work of packaging part X using worker terminal 40. Therefore, the work recognition unit 209 can recognize that the work type of each work process performed from "5 / 10 12:00:00" to "5 / 10 12:00:50" is packaging part X.

[0064] The frame video is video data of the frame number indicated by the frame, and is video data extracted from the video used by the task recognition unit 209 to recognize the task.

[0065] The input / output control unit 221 controls the input and output of various information to and from the worker terminal 40. Hereinafter, a worker using the worker terminal 40 will be referred to as a first worker. The reception unit 223 included in the input / output control unit 221 receives an output request for a recommendation screen (improvement proposal screen) for making a recommendation (improvement proposal) from the worker terminal 40 based on an instruction from the first worker to the worker terminal 40. When the reception unit 223 receives the output request for the recommendation screen, the output unit 225 included in the input / output control unit 221 outputs the recommendation screen to the worker terminal 40, and the control device 41 of the worker terminal 40 displays the recommendation screen on the display device 46. Note that the display of the recommendation screen by the worker terminal 40 may be premised on login authentication of the first worker.

[0066] The recommendation screen is a screen for setting predetermined conditions used by the identification unit 235 (described later) to identify one or more types of work to be improved by the first worker from among multiple types of work recognized by the work recognition unit 209. The recommendation screen is also a screen for displaying proposal information indicating improvement plans for each of the one or more types of work by the generation unit 237 (described later).

[0067] FIG. 6 is a diagram showing an example of a recommendation screen 601 according to this embodiment. The example shown in FIG. 6 shows the initial screen of the recommendation screen 601, and does not include any proposed information. On the recommendation screen 601 shown in FIG. 6, filter conditions (an example of predetermined conditions) can be set in a check box 611, and the number of improvement proposals to be displayed for work can be set in an input box 621. Examples of filter conditions include, but are not limited to, sorting by difference, top 10% of difference, and difference of 20 seconds or more.

[0068] The difference may be the difference between the work time of a first worker and the shortest work time for that work (the work time of the worker who performed that work in the shortest time). If a worker performs that work multiple times, the difference may be the average value of the differences. When the difference is the average value of the differences, the average value may be calculated by excluding the maximum and minimum values ​​of the differences in order to remove abnormal values.

[0069] The order of difference is, for example, a condition under which improvement plans are generated for all task types recognized by the task recognition unit 209 and displayed in descending order of the difference. The top 10% difference is, for example, a condition under which improvement plans are generated for task types whose differences are within the top 10% of all task types recognized by the task recognition unit 209 and displayed in descending order of the difference. The difference of 20 seconds or more is, for example, a condition under which improvement plans are generated for task types whose differences in task time are 20 seconds or more of all task types recognized by the task recognition unit 209 and displayed in descending order of the difference.

[0070] The first worker uses the input device 45 to set the filter conditions and the number of items to be displayed on the recommendation screen 601, and when the first worker selects a setting button (not shown), the worker terminal 40 outputs the set conditions to the information processing device 20. The reception unit 223 receives input of predetermined conditions from the worker terminal 40. Specifically, the reception unit 223 receives the filter conditions (e.g., top 10% of differences) and the number of items to be displayed (e.g., 10) set on the recommendation screen 601 from the worker terminal 40.

[0071] When the receiving unit 223 receives the filter conditions and the number of items to be displayed, the management unit 231 stores them in the condition storage unit 233 and manages these conditions on the condition storage unit 233 .

[0072] The identification unit 235 identifies one or more types of work to be improved by the first worker from among multiple types of work, based on the recognition result of the work recognition unit 209 (the work history stored in the work history storage unit 211). Specifically, the identification unit 235 identifies one or more types of work based on the work time included in the recognition result of the work recognition unit 209 (the work history stored in the work history storage unit 211). For example, the identification unit 235 calculates, for each type of work, the difference between the shortest work time for that work and the work time of the first worker, and identifies, as one or more types of work, work types for which the difference satisfies the predetermined condition received by the receiving unit 223 (stored in the condition storage unit 233). Note that the identification unit 235 re-identifies one or more types of work each time input of the predetermined condition is received (each time the filter condition is reset).

[0073] The generation unit 237 generates proposal information indicating improvement plans for each of the one or more types of work identified by the identification unit 235. Specifically, the generation unit 237 generates the proposal information for each of the one or more types of work using the recognition result of the worker who will take the shortest task time. For example, the generation unit 237 generates a prompt for each of the one or more types of work using at least the recognition result of the worker who will take the shortest task time, and inputs the generated prompt into the generative model 30 to generate the proposal information.

[0074] The proposal information is a proposal to improve at least a part of the work steps of one or more types of work. In this embodiment, the proposal information is a proposal to rearrange the order of the work steps of one or more types of work, but is not limited to this.

[0075] The processing of the identification unit 235 and the generation unit 237 of this embodiment will be described in detail below. In this embodiment, the identification unit 235 calculates, for each type of task recognized by the task recognition unit 209, the difference between the shortest task time for that task and the task time of the first worker. FIG. 7 is a diagram showing an example of the task history of this embodiment, showing the task time of the first worker packaging component X. It is also assumed here that the task time of worker A packaging component X described in FIG. 5 is the shortest task time. In this case, the task time of worker A packaging component X is 50 seconds (s) as shown in FIG. 5, and the task time of the first worker packaging component X is 80 seconds (s) as shown in FIG. 7, resulting in a difference of 30 seconds (s).

[0076] 7, the first worker affixes a label to the outer box before assembling it, and is unsure of where on the outer box to affix the label, which takes time. Therefore, in the packaging of part X by the first worker shown in Fig. 7, the order of the work step of the object "outer box" and the action "assembly" and the work step of the object "label" and the action "affix" is reversed compared to the packaging of part X by worker A shown in Fig. 5, and the work time for the work step of the object "label" and the action "affix" takes longer.

[0077] The identification unit 235 performs the above-described processing for other types of work recognized by the work recognition unit 209, and calculates, for each type of work, the difference between the shortest work time for that work and the work time of the first worker.

[0078] In addition, here, it is assumed that the predetermined condition accepted by the accepting unit 223 (stored in the condition storage unit 233) is the top 10% of differences, and the difference of the first worker who individually packages the part X described above is within the top 10% of differences.

[0079] Therefore, the generation unit 237 generates an improvement plan for each task that has a difference within the top 10% and includes the packaging of part X. For example, the generation unit 237 generates an improvement plan for the first worker for packaging part X using the task recognition result (task history) for packaging part X by worker A described in FIG. 5 . For example, the generation unit 237 extracts information about the task from the task recognition result (task history) for packaging part X by worker A described in FIG. 5 , generates a prompt including an instruction to put the extracted information about the object and action in the task into a sentence, and inputs the generated information into the generative model 30. As a result, the generation unit 237 obtains a sentence (improvement plan) from the generative model 30, such as, for example, "Perform part X in the following order: step 1 → step 2 → step 3 → step 4 → step 5 → step 6 → step 7."

[0080] An example of step 1 is a sentence such as "1. Place the cushioning material on the workbench." An example of step 2 is a sentence such as "2. Place part X on the cushioning material and secure it with tape α." An example of step 3 is a sentence such as "3. Wrap part X in the cushioning material and secure it with tape β." An example of step 4 is a sentence such as "4. Assemble the outer box." An example of step 5 is a sentence such as "5. Attach a label to the outer box." An example of step 6 is a sentence such as "6. Pack the cushioning material that has wrapped part X into the outer box." An example of step 7 is a sentence such as "7. Attach tape γ to the lid of the outer box and seal it."

[0081] Furthermore, for example, the generation unit 237 may extract information about the work process from the work recognition result (work history) of the first worker for packaging part X individually described in Fig. 7 , and may generate a prompt that proposes an improvement plan for the work of packaging part X individually described in Fig. 7 by the first worker, including information about the objects and actions in the work process, with reference to the work recognition result of the worker A for packaging part X individually described in Fig. 5 , and input the generated prompt to the generative model 30. In this case, it is expected to acquire a sentence (improvement plan) such as "Instead of attaching a label to the outer box and then assembling the outer box, it is better to assemble the outer box and then attach a label to the outer box."

[0082] Furthermore, for example, the generation unit 237 may extract the contents of the objects and actions in the work process without using the generative model 30, and generate an improvement plan such as, "For part X, please carry out the following in the order of process 1 (object "cushioning material", action "place") → process 2 (object "tape α, part X", action "fix")... → process 7 (object "tape γ, outer box", action "seal")."

[0083] The generation unit 237 generates proposal information by similarly generating improvement plans for each operation other than the part X packaging that has a difference within the top 10% and generates a recommendation screen including the proposal information.

[0084] 8 is a diagram showing an example of a recommendation screen 601 including proposal information 631 of this embodiment. In the example shown in FIG. 8, the recommendation screen 601 includes proposal information 631 with items such as priority, part, shortest time, average difference, and improvement plan for each task that is within the top 10% of differences including the packaging of part X. Note that the item "procedure" included in the improvement plan has a link to a frame image of the task with the shortest work time used to generate the improvement plan.

[0085] The output unit 225 outputs the proposal information generated by the generation unit 237 to the first worker. Specifically, the output unit 225 outputs a recommendation screen including the proposal information generated by the generation unit 237 to the worker terminal 40 of the first worker, and the control device 41 of the worker terminal 40 displays, for example, the recommendation screen shown in FIG. 8 on the display device 46.

[0086] Next, the operation of the work analysis system of this embodiment will be described.

[0087] 9 is a flowchart showing an example of task recognition processing performed by the information processing device 20 of this embodiment. The task recognition processing shown in FIG. 9 may be performed, for example, sequentially or periodically.

[0088] First, the video acquisition unit 201 acquires a plurality of videos capturing the working situations of a plurality of workers from the plurality of image capture devices 10-1 to 10-n, and stores (accumulates) the videos in the video storage unit 203 (step S101).

[0089] Next, the work recognition unit 209 acquires the video from the video storage unit 203, inputs the acquired video into the recognition model stored in the recognition model storage unit 207, and acquires the recognition result, thereby recognizing the work being performed by the worker shown in the video on a work process basis (step S103).

[0090] Next, the task recognition unit 209 stores the task recognition result as a task history in the task history storage unit 211 (step S105).

[0091] 10 is a sequence diagram showing an example of a condition setting process performed by the work analysis system 1 of this embodiment. The condition setting process shown in FIG. 10 will be described below using the setting of a filter condition and the number of items to be displayed as an example.

[0092] First, the receiving unit 223 receives a request to output a recommendation screen (improvement proposal screen) for making a recommendation (improvement proposal) from the worker terminal 40 based on an instruction from the first worker to the worker terminal 40 (step S201).

[0093] Next, the output unit 225 outputs the recommendation screen to the worker terminal 40 (step S203), and the control device 41 of the worker terminal 40 displays the recommendation screen on the display device 46 (step S205).

[0094] Next, the first worker uses the input device 45 to set the filter conditions and the number of items to be displayed on the recommendation screen 601 (step S207), and when the first worker selects a setting button (not shown), the worker terminal 40 outputs these set conditions to the information processing device 20 (step S209).

[0095] Next, the receiving unit 223 receives these conditions from the worker terminal 40, and the management unit 231 stores these conditions in the condition storage unit 233 and manages these conditions on the condition storage unit 233 (step S211).

[0096] 11 is a sequence diagram showing an example of the improvement proposal process performed by the work analysis system 1 of this embodiment. The improvement proposal process shown in FIG. 11 may be performed, for example, following the process shown in FIG.

[0097] First, the identification unit 235 calculates the difference between the shortest work time for each type of work and the work time of the first worker, and identifies one or more types of work for which the difference satisfies the specified conditions accepted by the acceptance unit 223 (stored in the condition memory unit 233) (step S301).

[0098] Next, the generation unit 237 generates a prompt using at least the recognition result of the worker who will have the shortest task time for each of one or more types of tasks (step S303), and obtains suggested information by inputting the generated prompt into the generation model 30 (step S305).

[0099] Next, the generating unit 237 generates a recommendation screen including the proposal information (step S307).

[0100] Next, the output unit 225 outputs the recommendation screen to the worker terminal 40 (step S309), and the control device 41 of the worker terminal 40 displays the recommendation screen on the display device 46 (step S311).

[0101] In this embodiment, the setting of predetermined conditions and the display of proposed information are performed on the recommendation screen 601 (see FIGS. 6 and 8 ), but the setting screen for predetermined conditions and the display screen for proposed information may be displayed separately. In this case, the setting screen for predetermined conditions may be provided with input boxes for setting other conditions, not limited to the filter and the number of items to be displayed.

[0102] As described above, the work analysis system of this embodiment analyzes video footage of the work being performed by each worker, identifies tasks that require improvement using the analysis results, and makes suggestions for improving the identified tasks, thereby supporting the improvement of tasks that were initially unclear (unknown) and requiring improvement. As a result, even if a worker takes more time to complete a task than other workers, the worker can understand the optimal way to do it, thereby reducing the time required for the task.

[0103] (Variation 1) In the above embodiment, the proposal information is described as a proposal for rearranging the order of work steps for one or more types of work, but the present invention is not limited to this. The proposal information may be, for example, at least one of a proposal regarding the placement of items used in at least some of the work steps and a proposal regarding work actions in at least some of the work steps. For example, if the long work time is caused by a tool such as tape being placed far away, the proposal information may suggest placing the tool closer to the worker. Furthermore, for example, if the long work time is caused by overlapping actions, such as applying tape twice, the proposal information may suggest improving the overlapping actions (performing them all at once).

[0104] (Program) The program executed by the information processing device 20 in the above embodiment and modified example is provided as a file in an installable or executable format stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0105] The programs executed by the information processing device 20 of the above embodiment and modified example may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the information processing device 20 of the above embodiment and modified example may be provided or distributed via a network such as the Internet. The programs executed by the information processing device 20 of the above embodiment and modified example may be provided by being pre-installed in a ROM or the like.

[0106] The programs executed by the information processing device 20 of the above embodiment and the above modification have a modular configuration for implementing the above-mentioned units on a computer. In actual hardware, for example, the CPU reads the learning program from the HDD onto the RAM and executes it, thereby implementing the above-mentioned units on the computer.

[0107] The above-described embodiment and modifications merely illustrate examples of specific embodiments of the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these. Therefore, the present disclosure can be implemented in various forms without departing from the spirit or main features thereof. For example, the above-described embodiment and modifications may be appropriately combined in their respective constituent units. Furthermore, for example, some components may be deleted from all components in the above-described embodiment and modifications.

[0108] The present disclosure includes the following aspects.

[0109] (1) An information processing device comprising: a video acquisition unit that acquires multiple videos capturing the work situations of multiple workers; a task recognition unit that analyzes the multiple videos and recognizes multiple types of tasks performed by each worker; an identification unit that identifies one or more types of tasks to be improved by a first worker from among the multiple types of tasks based on the recognition results of the task recognition unit; a generation unit that generates proposal information indicating improvement plans for each of the one or more types of tasks identified; and an output unit that outputs the proposal information to the first worker.

[0110] (2) The information processing device according to (1), wherein the task recognition unit further recognizes task time required for each task, and the identification unit identifies the one or more types of tasks based on task time included in the recognition result.

[0111] (3) The information processing device according to (2), wherein the identification unit calculates, for each type of work, a difference between a shortest work time for the work and a work time of the first worker, and the one or more types of work are types of work for which the difference satisfies a predetermined condition.

[0112] (4) The information processing apparatus according to any one of (3) above, further comprising: a receiving unit that receives input of the predetermined condition from a worker terminal used by the first worker.

[0113] (5) The information processing device according to (4), wherein the specifying unit re-specifies the one or more types of work each time an input of the predetermined condition is accepted.

[0114] (6) The information processing device according to any one of (1) to (5), wherein the task recognition unit recognizes each task in units of task steps, and the proposal information is a proposal to improve at least a part of each task step of the one or more types of tasks.

[0115] (7) The information processing device according to (6), wherein the proposal information is at least one of a proposal for rearranging the order of work steps for each of the one or more types of work, a proposal for arranging items used in at least some of the work steps, and a proposal for work movements in at least some of the work steps.

[0116] (8) The information processing device according to any one of (1) to (5), wherein the task recognition unit further recognizes task time required for each task, and the generation unit generates the proposal information using a recognition result of a worker who takes the shortest task time for each of the one or more types of tasks.

[0117] (9) The information processing device according to (8), wherein the task recognition unit further recognizes task time required for each task, and the generation unit generates a prompt for each of the one or more types of tasks using at least the recognition result of the worker who has the shortest task time, and generates the suggested information by inputting the generated prompt into a generation model.

[0118] (10) An improvement suggestion method including: a video acquisition step in which a video acquisition unit acquires multiple videos capturing the work situations of multiple workers; a task recognition step in which a task recognition unit analyzes the multiple videos and recognizes multiple types of tasks performed by each worker; an identification step in which an identification unit identifies one or more types of tasks to be improved by a first worker from the multiple types of tasks based on the recognition result of the task recognition unit; a generation step in which a generation unit generates proposal information indicating improvement plans for each of the one or more types of tasks identified; and an output step in which an output unit outputs the proposal information to the first worker.

[0119] (11) A program for causing a computer to execute the following steps: an image acquisition step for acquiring multiple images capturing the work situations of multiple workers; an operation recognition step for analyzing the multiple images and recognizing multiple types of operations performed by each worker; an identification step for identifying one or more types of operations to be improved by a first worker from among the multiple types of operations based on the recognition results of the operation recognition unit; a generation step for generating proposal information indicating improvement plans for each of the one or more types of operations identified; and an output step for outputting the proposal information to the first worker.

[0120] REFERENCE SIGNS LIST 1 Work analysis system 2 Network 10, 10-1 to 10-n Multiple image capture devices 20 Information processing device 30 Generative model 40, 40-1 to 40-m Multiple worker terminals 201 Video acquisition unit 203 Video storage unit 207 Recognition model storage unit 209 Work recognition unit 211 Work history storage unit 221 Input / output control unit 223 Reception unit 225 Output unit 231 Management unit 233 Condition storage unit 235 Identification unit 237 Generation unit

Claims

1. An information processing device comprising: a video acquisition unit that acquires multiple videos capturing the work situations of multiple workers; a task recognition unit that analyzes the multiple videos and recognizes multiple types of tasks performed by each worker; an identification unit that identifies one or more types of tasks that are to be improved by a first worker from among the multiple types of tasks based on the recognition results of the task recognition unit; a generation unit that generates proposal information indicating improvement plans for each of the one or more types of tasks identified; and an output unit that outputs the proposal information to the first worker.

2. The information processing device according to claim 1, wherein the task recognition unit further recognizes the task time required for each task, and the identification unit identifies the one or more types of tasks based on the task time included in the recognition result.

3. The information processing device according to claim 2, wherein the identification unit calculates the difference between the shortest work time for each type of work and the work time of the first worker, and the one or more types of work are types of work for which the difference satisfies a predetermined condition.

4. The information processing device according to claim 3, further comprising a reception unit that receives input of the predetermined conditions from a worker terminal used by the first worker.

5. The information processing device according to claim 4, wherein the specifying unit re-specifies the one or more types of work each time an input of the predetermined condition is received.

6. An information processing device according to any one of claims 1 to 5, wherein the task recognition unit recognizes each task in units of task steps, and the proposal information is a proposal to improve at least a part of each task step of the one or more types of tasks.

7. The information processing device according to claim 6, wherein the proposal information is at least one of a proposal for rearranging the order of work processes for each of the one or more types of work, a proposal regarding the placement of items used in at least some of the work processes, and a proposal regarding work actions in at least some of the work processes.

8. An information processing device according to any one of claims 1 to 5, wherein the task recognition unit further recognizes the task time required for each task, and the generation unit generates the proposed information using the recognition result of the worker who takes the shortest task time for each of the one or more types of tasks.

9. The information processing device according to claim 8, wherein the task recognition unit further recognizes the task time required for each task, and the generation unit generates a prompt for each of the one or more types of tasks using at least the recognition result of the worker who takes the shortest task time, and generates the proposed information by inputting the generated prompt into a generation model.

10. An improvement suggestion method including: a video acquisition step in which a video acquisition unit acquires multiple videos capturing the work situations of multiple workers; a task recognition step in which a task recognition unit analyzes the multiple videos and recognizes multiple types of tasks performed by each worker; an identification step in which an identification unit identifies one or more types of tasks to be improved by a first worker from among the multiple types of tasks based on the recognition results of the task recognition unit; a generation step in which a generation unit generates proposal information indicating improvement plans for each of the one or more types of tasks identified; and an output step in which an output unit outputs the proposal information to the first worker.

11. A program for causing a computer to execute the following steps: an image acquisition step for acquiring multiple images capturing the work situations of multiple workers; an operation recognition step for analyzing the multiple images and recognizing multiple types of operation performed by each worker; an identification step for identifying one or more types of operation to be improved by a first worker from among the multiple types of operation based on the recognition results of the operation recognition unit; a generation step for generating proposal information indicating improvement plans for each of the one or more types of operation identified; and an output step for outputting the proposal information to the first worker.

Images