Information processing device, improvement suggestion method, and program

Abstract

This invention provides an information processing device, an improvement suggestion method, and a program that can support the improvement of a task even when the task requiring improvement is unknown. [Solution] One aspect of the information processing device of the present disclosure includes: an image acquisition unit that acquires multiple images of the work status of multiple workers; an action recognition unit that analyzes the multiple images and recognizes multiple types of work performed by each worker; an identification unit that identifies one or more types of work from among the multiple types of work to be improved by the first worker based on the recognition results of the action recognition unit; a generation unit that generates proposal information indicating improvement plans for each of the identified one or more types of work; and an output unit that outputs the proposal information to the first worker.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an improvement proposal method, and a program.

Background Art

[0002] Conventionally, at production sites of various products such as factories, measures have been taken to improve the efficiency of production operations such as product manufacturing, processing, and assembly by workers.

[0003] For example, in Patent Document 1, every time the shortest time of a production operation is updated by a worker performing the production operation of a certain product, an optimal operation procedure is extracted from a video obtained by photographing the operation status of the production operation using a learned model, and compared with the operation procedures of other workers, thereby presenting improvement proposals for the operation procedures of other workers. Note that the learned model is one that has learned about the standard operation time and the suitability of the operation of the production operation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the above-described conventional technology, a learned model cannot be prepared unless the operation that requires improvement is known. That is, the above-described conventional technology is premised on the fact that the operation that requires improvement is known. Therefore, in a situation where it is necessary to perform a wide variety of operations and new types of operations frequently occur, and it is difficult to identify the operation that requires improvement itself, as in the case of operations at a site, it is also difficult to apply the above-described conventional technology.

[0006] This disclosure is made in light of the above circumstances and aims to provide an information processing device, an improvement suggestion method, and a program that can support the improvement of work even when the work requiring improvement is unknown. [Means for solving the problem]

[0007] One embodiment of the information processing device of the present disclosure includes: an image acquisition unit that acquires multiple images of the work status of multiple workers; a work recognition unit that analyzes the multiple images and recognizes multiple types of work performed by each worker; an identification unit that identifies one or more types of work from among the multiple types of work that are subject to improvement for the first worker based on the recognition results of the work recognition unit; a generation unit that generates proposal information indicating improvement plans for each of the identified one or more types of work; and an output unit that outputs the proposal information to the first worker.

[0008] One embodiment of the improvement proposal method of the present disclosure includes: a video acquisition step in which a video acquisition unit acquires multiple videos of the work status of multiple workers; a work recognition step in which a work recognition unit analyzes the multiple videos and recognizes multiple types of work performed by each worker; a specification step in which a specification unit identifies one or more types of work that are subject to improvement for a first worker from among the multiple types of work based on the recognition results of the work recognition unit; a generation step in which a generation unit generates proposal information indicating improvement proposals for each of the one or more types of work identified; and an output step in which an output unit outputs the proposal information to the first worker.

[0009] One aspect of the program of this disclosure involves causing a computer to execute the following steps: an image acquisition step of acquiring multiple images of the work status of multiple workers; a work recognition step of analyzing the multiple images to recognize multiple types of work performed by each worker; an identification step of identifying one or more types of work from among the multiple types of work to be improved by the first worker based on the recognition results of the work recognition unit; a generation step of generating proposal information showing improvement plans for each of the identified one or more types of work; and an output step of outputting the proposal information to the first worker. [Effects of the Invention]

[0010] According to this disclosure, even when the work requiring improvement is unknown, it is possible to provide an information processing device, an improvement suggestion method, and a program that can support the improvement of said work. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a block diagram showing an example of the system configuration of the work analysis system of this embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device and worker terminal of this embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. [Figure 4] Figure 4 shows an example of an image captured by the imaging device of this embodiment and acquired by the image acquisition unit. [Figure 5] Figure 5 shows an example of the work history in this embodiment. [Figure 6] Figure 6 shows an example of the recommendation screen in this embodiment. [Figure 7] Figure 7 shows an example of the work history in this embodiment. [Figure 8] Figure 8 shows an example of a recommendation screen that includes the proposed information of this embodiment. [Figure 9]Figure 9 is a flowchart showing an example of the work recognition process performed in the information processing device of this embodiment. [Figure 10] Figure 10 is a sequence diagram showing an example of the condition setting process performed in the work analysis system of this embodiment. [Figure 11] Figure 11 is a sequence diagram showing an example of the improvement suggestion process performed in the work analysis system of this embodiment. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present disclosure (hereinafter simply referred to as "these embodiments") will be described in detail with reference to the drawings. However, this disclosure is not limited to the following embodiments. Furthermore, the following embodiments and modifications can be combined as appropriate.

[0013] In the following section, we will explain the work analysis system of this embodiment using warehouse operations performed at a logistics center (warehouse) that delivers parts (repair parts) as an example.

[0014] Generally, logistics centers that handle parts deal with a vast and diverse range of parts and tools, and the parts they handle change frequently. Furthermore, logistics centers need to perform warehouse operations such as receiving, inspection, warehousing, picking, individual packaging, packing, and shipping for each part, and the work content for receiving, inspection, warehousing, picking, and individual packaging often differs depending on the part. Thus, logistics centers that distribute parts face the challenge of not only performing a wide variety of warehouse operations, but also frequently encountering new types of parts, making it difficult to identify operations that require improvement. In particular, warehouse operations often involve simple tasks that can be performed to a certain extent at the discretion of the worker, which makes it difficult to identify operations that require improvement.

[0015] Tasks requiring improvement include, for example, tasks where a worker takes longer to complete than other workers, but where the worker can shorten the time by understanding the optimal method.

[0016] In the work analysis system of this embodiment, videos of the work performed by each worker are analyzed, and using the analysis results, the work that requires improvement for the worker is identified, and improvement proposals for the identified work are provided to the worker, thereby assisting in improving the work that requires improvement, which was initially unclear (unknown).

[0017] Hereinafter, the work analysis system of this embodiment will be specifically described while appropriately citing examples of individual operations in a logistics center (warehouse) that handles parts. However, the work targeted by the work analysis system of this embodiment is not limited to this, and can be targeted at any warehouse work described above. Also, it can be targeted at all work in a logistics center that handles products (finished products) instead of parts.

[0018] Note that individual packaging means individually packaging each part. For example, wrapping the part with a cushioning material and boxing it in an outer box can be cited. The cushioning material, outer box, tape used when wrapping with the cushioning material, or the tape used when sealing the lid portion of the outer box differ depending on the type (size, shape, etc.) of the part to be individually packaged. Generally, parts are not individually stored in the logistics center but are stored in a certain quantity together. Therefore, when shipping parts, after picking and individually packaging the parts to be shipped, they are packaged together with other individually packaged parts and then carried out.

[0019] First, the configuration of the work analysis system of this embodiment will be described.

[0020] FIG. 1 is a block diagram showing an example of the system configuration of the work analysis system 1 of this embodiment. As shown in FIG. 1, the work analysis system 1 includes a plurality of imaging devices 10-1 to 10-n (n is a natural number), an information processing device 20, a generation model 30, and a plurality of worker terminals 40-1 to 40-m (m is a natural number).

[0021] Multiple imaging devices 10-1 to 10-n, an information processing device 20, a generation model 30, and multiple worker terminals 40-1 to 40-m are connected via network 2. Network 2 can be implemented by at least one of the following: the Internet and / or a LAN (Local Area Network). Network 2 may be a wired network, a wireless network, or a mixture of wired and wireless networks.

[0022] In the following explanation, if it is not necessary to distinguish between the imaging devices 10-1 to 10-n, they may simply be referred to as imaging device 10. Similarly, if it is not necessary to distinguish between the worker terminals 40-1 to 40-m, they may simply be referred to as worker terminal 40.

[0023] The imaging devices 10-1 to 10-n are installed inside and outside the warehouse to photograph the work of workers performing tasks inside and outside the warehouse, and output the captured images to the information processing device 20. Each imaging device 10 is installed inside or outside the warehouse to photograph, for example, the work area where the worker is performing tasks (for example, a workbench where individual packaging work is performed). The imaging devices 10 may include, for example, video cameras.

[0024] The information processing device 20 analyzes multiple images captured by multiple imaging devices 10 and recognizes various tasks performed by workers from each image. The information processing device 20 also identifies types of tasks requiring improvement from the recognized tasks and makes improvement suggestions for those tasks, for example, using a generative model 30. The information processing device 20 can be implemented, for example, by a server device or a cloud service.

[0025] The generative model 30 is a natural language processing model, also known as generative AI (Generative Artificial Intelligence), and provides services using natural language processing. In this embodiment, the generative model 30 is described as an AI chat service that has been fine-tuned from a large language model (LLM), but it is not limited to this. For example, the generative model 30 may be implemented using a small language model (SLM).

[0026] The worker terminal 40 is a terminal device used by the worker, and examples include a tablet, smartphone, or PC (Personal Computer). The worker terminal 40 is used, for example, for the worker to check improvement suggestions made by the information processing device 20. The worker terminal 40 may be held by the worker themselves, or it may be installed in the workplace where the worker performs their work.

[0027] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 20 and the worker terminal 40 in this embodiment.

[0028] First, the hardware configuration of the information processing device 20 will be described. As shown in Figure 2, the information processing device 20 comprises a control device 21, a main memory 22, an auxiliary storage device 23, a communication device 24, and various buses 29. The control device 21, the main memory 22, the auxiliary storage device 23, and the communication device 24 are connected via the various buses 29. Thus, the information processing device 20 of this embodiment has a general hardware configuration using a normal computer.

[0029] The control device 21 controls the overall operation of the information processing device 20. The control device 21 may be, for example, at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), but is not limited to these. There may be one or more CPUs or GPUs, and they may be single-core or multi-core.

[0030] Examples of main memory 22 include, but are not limited to, ROM (Read Only Memory) and RAM (Random Access Memory). ROM stores various programs, such as programs for controlling the information processing device 20, programs for recognizing the operator's work from video, and programs for suggesting improvements. RAM is used as a workspace for the control device 21 to perform various controls based on the programs stored in ROM.

[0031] The auxiliary storage device 23 stores various data, including the various programs mentioned above and a recognition model (a trained interaction recognition model) for recognizing the worker's work from the video. The various programs mentioned above only need to be stored in at least one of the main storage device 22 and the auxiliary storage device 23. The auxiliary storage device 23 may be, but is not limited to, at least one of existing storage devices capable of magnetic, electrical, or optical storage, such as an HDD (Hard Disk Drive), SSD (Solid State Drive), and DVD (Digital Versatile Disc). The auxiliary storage device 23 may be built into the information processing device 20 or externally connected to the information processing device 20 via an interface such as USB (Universal Serial Bus). Furthermore, the auxiliary storage device 23 may be a NAS (Network Attached Storage) connected via a network such as a LAN or WAN (Wide Area Network).

[0032] The communication device 24 is used to communicate with worker terminals 40 and the like via the network 2. Examples of the communication device 24 include, but are not limited to, wired LAN communication devices and wireless LAN wireless communication devices.

[0033] In addition to the above configuration, the information processing device 20 may further include hardwired circuits such as ICs (Integrated Circuits), ASICs (Application Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays) that are specific to the information processing device 20.

[0034] Next, the hardware configuration of the worker terminal 40 will be described. As shown in Figure 2, the worker terminal 40 comprises a control device 41, a main memory 42, an auxiliary storage device 43, a communication device 44, an input device 45, a display device 46, and various buses 49. The control device 41, main memory 42, auxiliary storage device 43, communication device 44, input device 45, and display device 46 are connected via various buses 49. Thus, the worker terminal 40 in this embodiment has a general hardware configuration, such as that of a PC or tablet terminal.

[0035] The control device 41 controls the overall operation of the worker terminal 40. The implementation method of the control device 41 is the same as that of the control device 21.

[0036] The implementation method for the main memory 42 is the same as that for the main memory 22, so a detailed explanation will be omitted. The ROM of the main memory 42 stores various programs, including a program for controlling the worker terminal 40.

[0037] The auxiliary storage device 43 stores the various programs and data described above. The various programs described above only need to be stored in at least one of the main storage device 42 and the auxiliary storage device 43. The implementation method for the auxiliary storage device 43 is the same as that for the auxiliary storage device 23, so a detailed explanation is omitted.

[0038] The communication device 44 is used to communicate with the information processing device 20 and other devices via the network 2. Since the implementation method for the communication device 44 is the same as that for the communication device 24, a detailed explanation is omitted.

[0039] The input device 45 is used for various inputs, selections, and specifications to confirm improvement suggestions, and serves as a user interface between the operator and the device. Examples of input devices 45 include, but are not limited to, keyboards, mice, and touch panels. The input device 45 may be built into the operator terminal 40 or it may be externally connected via an interface such as USB.

[0040] The display device 46 displays various screens, such as improvement suggestion screens, and serves as a user interface between the device and the worker. Examples of the display device 46 include, but are not limited to, liquid crystal displays, organic electro-luminescence (OLED) displays, and touch panel displays. The display device 46 may be an internal display built into the worker terminal 40, or an external display connected to the worker terminal 40 via a display interface such as HDMI®.

[0041] Figure 3 is a block diagram showing an example of the functional configuration of the information processing device 20 in this embodiment. As shown in Figure 3, the information processing device 20 includes an image acquisition unit 201, an image storage unit 203, a recognition model storage unit 207, a work recognition unit 209, a work history storage unit 211, an input / output control unit 221, a management unit 231, a condition storage unit 233, a specification unit 235, and a generation unit 237. The input / output control unit 221 includes a reception unit 223 and an output unit 225.

[0042] The video acquisition unit 201, the work recognition unit 209, the input / output control unit 221, the reception unit 223, the output unit 225, the management unit 231, the identification unit 235, and the generation unit 237 can be realized, for example, by the control device 21, main memory 22, and communication device 24 described in Figure 2. For example, the control device 21 reads various programs stored in the main memory 22 (ROM) or auxiliary memory 23 and expands them into the main memory 22 (RAM). The control device 21 realizes each of the above-mentioned functional units by executing various processes according to the expanded programs. Here, the case in which each of the above-mentioned functional units is realized as software has been explained as an example, but at least a part of each of the above-mentioned functional units may be realized as hardware. In this case, the functional unit to be realized as hardware can be realized, for example, by the hardwired circuit described above. Alternatively, any of the above-mentioned functional units may be realized through the cooperation of software and hardware.

[0043] The video memory unit 203, the recognition model memory unit 207, the work history memory unit 211, and the condition memory unit 233 can be implemented, for example, by the auxiliary storage device 23 described in Figure 2.

[0044] The video acquisition unit 201 acquires multiple videos from multiple shooting devices 10-1 to 10-n, capturing the work status of multiple workers, and stores (accumulates) them in the video storage unit 203. Figure 4 shows an example of a video 501 captured by the shooting device 10 of this embodiment and acquired by the video acquisition unit 201. The video 501 shown in Figure 4 is a video of worker W performing individual packaging work on a workbench 51, captured by a shooting device 10 installed on the ceiling. Specifically, worker W wraps parts in cushioning material using tapes 61 and 63 on the workbench 51 (not shown), packs the parts wrapped in cushioning material into an outer box 71, attaches tape 65 to the lid of the outer box 71, and seals the outer box 71.

[0045] In this embodiment, multiple camera devices 10 are arranged in the warehouse to each photograph a workbench where an employee performs individual packaging work. Therefore, the video acquisition unit 201 acquires images from each camera device 10, as shown in Figure 4, of each employee performing individual packaging work, and stores (accumulates) the acquired images in the video storage unit 203. For this reason, it is assumed that the images stored in the video storage unit 203 include images of each employee sequentially performing individual packaging work on various parts, such as images of employee A performing individual packaging work on part X and then on part Y, or images of employee B performing individual packaging work on part Z and then on part Y.

[0046] The recognition model storage unit 207 stores a recognition model for recognizing the work being performed by an operator from the video stored in the video storage unit 203. The recognition model is, for example, an operator interaction recognition model, and is a trained model generated by a learning device (not shown) learning from training data.

[0047] Training data includes, for example, work videos such as live-action footage of workers performing individual packaging tasks or CG footage of computer graphics models of workers performing individual packaging tasks, to which annotations (labels) have been added. The annotations indicate, for example, whether or not there is interaction between the worker's hands and an object (tool) for each frame of the work video, and if there is interaction, the name of the object involved and the action taken using that object.

[0048] The learning device (not shown) learns from such training data and generates a recognition model that, for each frame of the input video, outputs recognition results indicating what objects (tools) the worker is using and what actions they are performing in the individual packaging process.

[0049] The work recognition unit 209 analyzes multiple videos acquired by the video acquisition unit 201 (stored in the video storage unit 203) to recognize multiple types of work performed by each worker. Specifically, the work recognition unit 209 analyzes multiple videos and recognizes each of the multiple types of work performed by each worker on a work process basis. The work recognition unit 209 also recognizes the work time required for each of the multiple types of work performed by each worker. The work recognition unit 209 stores the work recognition results as work history in the work history storage unit 211.

[0050] In this embodiment, the work recognition unit 209 acquires video from the video storage unit 203 and inputs the acquired video into the recognition model stored in the recognition model storage unit 207. The work recognition unit 209 acquires recognition results from the recognition model for each frame of the video, indicating what objects (tools) the worker is using and what actions they are performing in the individual packaging work. This allows the work performed by the worker in the video to be recognized in terms of work processes and stored as work history in the work history storage unit 211. By performing the above processing on all video stored in the video storage unit 203, the work recognition unit 209 can break down and recognize each of the multiple types of work performed by each worker down to the work process level.

[0051] Figure 5 shows an example of the work history in this embodiment. The work history shown in Figure 5 shows the recognition result of recognizing worker A's work from the video footage of worker A performing the individual packaging work on part X and then on part Y, as described above, specifically the recognition result for the individual packaging work on part X. In addition, in the work history shown in Figure 5, frames with the same work process are displayed together. In the example shown in Figure 5, the work history includes the worker, work type, start date and time, end date and time, frame, work process, and frame video.

[0052] The worker is the worker performing the task recognized by the work recognition unit 209 (in the example shown in Figure 5, "Worker A"). The worker may be recognized by the work recognition unit 209 from the video using a recognition model, or the worker may be set using information such as who will perform which task and which workbench, which the worker entered into the worker terminal 40 before performing the task. In the latter case, the work recognition unit 209 only needs to obtain the information from the worker terminal 40.

[0053] The work type indicates the type of work recognized by the work recognition unit 209 (in the example shown in Figure 5, "Individual packaging of part X"). The work type may be recognized by the work recognition unit 209 from the video using a recognition model, or it may be set using information such as part code information that the worker inputs into the worker terminal 40 before performing each task. Generally, before performing individual packaging work, the worker terminal 40 reads code information such as barcodes or 2D codes attached to the parts to be individually packaged, and after the individual packaging work for the parts is completed, the worker inputs the completion of the work from the worker terminal 40. In the latter case, the work recognition unit 209 only needs to obtain the information from the worker terminal 40.

[0054] The start date and time indicate the date and time when the work process began, the end date and time indicate the date and time when the work process ended, and the frame indicates the frame number from the start to the end of the work process. The work time required for the work may be, for example, the total time obtained by summing the difference between the end date and start date and time for each work process, or it may be the time obtained by dividing the total number of frames required for the work by the frame rate (for example, 60fps).

[0055] The work process shows the work process for individual packaging of part X as recognized by the work recognition unit 209 from the video using the recognition model. Specifically, the work process shows what objects (tools) the worker is using and what actions they are taking during the individual packaging process.

[0056] For example, suppose that during the time period from "5 / 10 12:00:00" to "5 / 10 12:00:05" in the video, worker A is placing cushioning material on the workbench, and an interaction is occurring between worker A's hand and the cushioning material. Furthermore, suppose that the recognition model has learned training data in which frames of video showing the worker placing the cushioning material are annotated with a defined action of placing the cushioning material if the object interacting with the worker's hand is the cushioning material. In this case, the work recognition unit 209, using the recognition model, would recognize the object and action as "placement" for the work process during the above time period in the video.

[0057] For example, suppose that during the time period from "5 / 10 12:00:05" to "5 / 10 12:00:10" in the video, worker A is securing part X, which is placed on cushioning material, with tape α, and interaction is occurring between worker A's hand, tape α, and part X. Furthermore, suppose that the recognition model has learned training data in which, in the video frame in which the worker secures part X, which is placed on cushioning material, with tape α, the object interacting with the worker's hand is tape α and part X, and the action is defined as securing part X with tape α. In this case, the work recognition unit 209, using the recognition model, recognizes the work process during the above time period in the video as object "tape α, part X" and action "securing".

[0058] For example, suppose that during the time frame "5 / 10 12:00:10" to "5 / 10 12:00:20" in the video, worker A wraps part X with cushioning material and secures it with tape β, and interaction occurs between worker A's hand and tape β and cushioning material. Furthermore, suppose that the recognition model has learned training data in which, in the video frame in which the worker wraps part X with cushioning material and secures it with tape β, if the objects interacting with the worker's hand are tape β and cushioning material, the action of wrapping with cushioning material and securing with tape β is defined and an annotation is added. In this case, the work recognition unit 209, using the recognition model, recognizes the work process during the above time frame in the video as objects "tape β, cushioning material" and action "wrapping".

[0059] For example, suppose that during the time period from "5 / 10 12:00:20" to "5 / 10 12:00:30" in the video, worker A is assembling the outer box, and an interaction occurs between worker A's hand and the outer box. Furthermore, suppose that the recognition model has learned training data in which frames of video showing the worker assembling the outer box are annotated with a defined action of assembling the outer box when the object interacting with the worker's hand is the outer box. In this case, the work recognition unit 209, using the recognition model, recognizes the work process during the above time period in the video as the object "outer box" and the action "assembly".

[0060] For example, suppose that during the time period from "5 / 10 12:00:30" to "5 / 10 12:00:35" in the video, worker A is attaching a label to the outer box, and an interaction occurs between worker A's hand and the label. Furthermore, suppose that the recognition model has learned training data in which frames of video showing the worker attaching the label to the outer box are annotated with a defined action of attaching a label when the object interacting with the worker's hand is the label. In this case, the work recognition unit 209, using the recognition model, recognizes the work process during the above time period in the video as the object "label" and the action "attach".

[0061] For example, suppose that during the time period from "5 / 10 12:00:35" to "5 / 10 12:00:40" in the video, worker A is packing the cushioning material that wraps part X into an outer box, and interaction occurs between worker A's hand and the cushioning material and the outer box. Furthermore, suppose that the recognition model has learned training data in which, in the video frame in which the worker is packing the cushioning material that wraps part X into an outer box, if the objects that the worker's hand is interacting with are the cushioning material and the outer box, an annotation is added that defines the action as "packing". In this case, the work recognition unit 209, using the recognition model, recognizes the work process during the above time period in the video as objects "cushioning material, outer box" and action "packing".

[0062] For example, suppose that during the time period from "5 / 10 12:00:40" to "5 / 10 12:00:50" in the video, worker A is attaching tape γ to the lid of the outer box and sealing the box, and that interaction is occurring between worker A's hand, tape γ, and the outer box. Furthermore, suppose that the recognition model has learned training data in which, in the video frame showing tape γ being attached to the lid of the outer box and the outer box being sealed, if the objects interacting with the worker's hand are tape γ and the outer box, the action defined as sealing is added. In this case, the work recognition unit 209, using the recognition model, recognizes the work process during the above time period in the video as objects "tape γ, outer box" and action "sealing".

[0063] Furthermore, prior to the work process of placing the object "cushioning material" and performing the action "placing," worker A reads the code information assigned to part X using worker terminal 40, and after the work process of placing the object "tape γ, outer box" and performing the action "sealing," worker A completes the individual packaging work of part X using worker terminal 40. Therefore, the work recognition unit 209 can recognize that the type of work for each work process performed from "5 / 10 12:00:00" to "5 / 10 12:00:50" is individual packaging of part X.

[0064] A frame image is video data corresponding to the frame number indicated by the frame, and is video data extracted by the work recognition unit 209 from the video used for work recognition.

[0065] The input / output control unit 221 controls the input and output of various types of information to and from the worker terminal 40. Hereinafter, the worker using the worker terminal 40 will be referred to as the first worker. The reception unit 223 included in the input / output control unit 221 receives a request from the worker terminal 40 to output a recommendation screen (improvement suggestion screen) for making recommendations (improvement suggestions) based on instructions from the first worker to the worker terminal 40. When the reception unit 223 receives the request to output the recommendation screen, the output unit 225 included in the input / output control unit 221 outputs the recommendation screen to the worker terminal 40, and the control device 41 of the worker terminal 40 displays the recommendation screen on the display device 46. Note that the display of the recommendation screen by the worker terminal 40 may be contingent on the first worker's login authentication.

[0066] The recommendation screen is a screen used by the identification unit 235 (described later) to set predetermined conditions used to identify one or more types of work that are targets for improvement by the first worker from among multiple types of work recognized by the work recognition unit 209. The recommendation screen is also a screen used by the generation unit 237 (described later) to display suggested information indicating improvement plans for each of the one or more types of work.

[0067] Figure 6 shows an example of the recommendation screen 601 of this embodiment. The example shown in Figure 6 shows the initial screen of the recommendation screen 601 and does not include any suggested information. In the recommendation screen 601 shown in Figure 6, filter conditions (an example of predetermined conditions) can be set on the checkbox 611, and the number of suggested improvements to the work can be set on the input box 621. Examples of filter conditions include sorting by difference, top 10% difference, difference of 20 seconds or more, etc., but are not limited to these.

[0068] The difference refers to the difference between the work time of the first worker and the shortest work time for that work (the work time of the worker who performed the work in the shortest time). If the worker performs the work multiple times, the difference may be the average of the differences. Furthermore, when using the average of the differences, the maximum and minimum values ​​of the differences may be excluded to remove outliers when calculating the average.

[0069] "Sorted by difference" means, for example, that improvement proposals are generated for all types of work recognized by the work recognition unit 209 and displayed in descending order of difference. "Top 10% difference" means, for example, that improvement proposals are generated for work types whose differences are within the top 10% of all types of work recognized by the work recognition unit 209 and displayed in descending order of difference. "Difference of 20s or more" means, for example, that improvement proposals are generated for work types whose work time difference is 20s or more among all types of work recognized by the work recognition unit 209 and displayed in descending order of difference.

[0070] The first worker uses the input device 45 to set filter conditions and the number of items to display on the recommendation screen 601. When the worker selects a setting button (not shown), the worker terminal 40 outputs these set conditions to the information processing device 20. The reception unit 223 receives input of predetermined conditions from the worker terminal 40. Specifically, the reception unit 223 receives the filter conditions (e.g., top 10% by difference) and the number of items to display (e.g., 10) set on the recommendation screen 601 from the worker terminal 40.

[0071] When the management unit 231 receives the filter conditions and the number of items to display from the reception unit 223, it stores them in the condition storage unit 233 and manages these conditions on the condition storage unit 233.

[0072] The identification unit 235 identifies one or more types of work that are subject to improvement by the first worker from among multiple types of work, based on the recognition result of the work recognition unit 209 (work history stored in the work history storage unit 211). Specifically, the identification unit 235 identifies one or more types of work based on the work time included in the recognition result of the work recognition unit 209 (work history stored in the work history storage unit 211). For example, for each type of work, the identification unit 235 calculates the difference between the shortest work time for that work and the work time of the first worker, and identifies one or more types of work that satisfy the predetermined conditions that the difference has been received by the receiving unit 223 (stored in the condition storage unit 233). The identification unit 235 re-identifies one or more types of work each time the input of the predetermined conditions is received (the filter conditions are reset).

[0073] The generation unit 237 generates proposal information that shows improvement plans for each of the one or more types of work identified by the identification unit 235. Specifically, for each of the one or more types of work, the generation unit 237 generates proposal information using the recognition result of the worker who completes the work in the shortest time. For example, for each of the one or more types of work, the generation unit 237 generates a prompt using at least the recognition result of the worker who completes the work in the shortest time, and generates proposal information by inputting the generated prompt into the generation model 30.

[0074] The proposed information is a suggestion to improve at least a part of the work process for each of one or more types of work. In this embodiment, the proposed information is a suggestion to rearrange the order of the work processes for each of one or more types of work, but it is not limited to this.

[0075] The following describes in detail the processing of the identification unit 235 and the generation unit 237 in this embodiment. In this embodiment, the identification unit 235 calculates the difference between the shortest work time and the work time of the first worker for each type of work recognized by the work recognition unit 209. Figure 7 is a diagram showing an example of the work history in this embodiment, and shows the work time of the first worker for individual packaging of part X. Here, we assume that the work time of worker A for individual packaging of part X, as explained in Figure 5, is the shortest work time. In this case, the work time of worker A for individual packaging of part X is 50 seconds (s), as shown in Figure 5, and the work time of the first worker for individual packaging of part X is 80 seconds (s), as shown in Figure 7, and the difference is 30 seconds (s).

[0076] In the example shown in Figure 7, the first worker attaches the label to the outer box before assembling it, and is unsure where to place the label, which takes time. As a result, in the individual packaging of part X by the first worker shown in Figure 7, the order of the work process is reversed compared to the individual packaging of part X by worker A shown in Figure 5, where the object is "outer box" and the action is "assembly," and the work process where the object is "label" and the action is "attached." This results in the work process where the object is "label" and the action is "attached" taking more time.

[0077] The identification unit 235 performs the same processing as described above for other types of work recognized by the work recognition unit 209, and calculates the difference between the shortest work time for each type of work and the work time of the first worker.

[0078] Furthermore, it is assumed that the predetermined conditions received by the reception unit 223 (stored in the condition storage unit 233) are in the top 10% of differences, and that the differences of the first worker in the individual packaging of part X described above are within the top 10% of differences.

[0079] Therefore, the generation unit 237 generates improvement suggestions for each task that falls within the top 10% of differences, including the individual packaging of part X. For example, the generation unit 237 generates an improvement suggestion for the first worker of individual packaging of part X using the work recognition result (work history) of worker A for individual packaging of part X, as explained in Figure 5. For example, the generation unit 237 extracts information on the work process from the work recognition result (work history) of worker A for individual packaging of part X, as explained in Figure 5, and generates a prompt that has instructions to put this information into text, along with information on the objects and actions of the extracted work process, and inputs it into the generation model 30. As a result, the generation unit 237 obtains text (improvement suggestion) from the generation model 30, such as, "For part X, please perform the process in the order of process 1 → process 2 → process 3 → process 4 → process 5 → process 6 → process 7."

[0080] For example, step 1 could be described as, "1. Place the cushioning material on the workbench." Step 2 could be described as, "2. Place part X on the cushioning material and secure it with tape α." Step 3 could be described as, "3. Wrap part X in the cushioning material and secure it with tape β." Step 4 could be described as, "4. Assemble the outer box." Step 5 could be described as, "5. Attach the label to the outer box." Step 6 could be described as, "6. Pack the cushioning material containing part X into the outer box." Step 7 could be described as, "7. Attach tape γ to the lid of the outer box and seal it."

[0081] Alternatively, for example, the generation unit 237 may extract information about the work process from the work recognition result (work history) of the first worker's individual packaging of part X, as explained in Figure 7, and, including the information about the objects and actions of the extracted work process, generate a prompt to suggest improvements to the first worker's work on individual packaging of part X, as explained in Figure 7, using the work recognition result of worker A's individual packaging of part X, as explained in Figure 5, as explained in Figure 7, and input this prompt into the generation model 30. In this case, it is expected that a sentence (improvement suggestion) such as "Instead of assembling the outer box after attaching the label to the outer box, it would be better to assemble the outer box first and then attach the label to the outer box" can be obtained.

[0082] Alternatively, for example, the generation unit 237 may extract the contents of the objects and actions in the work process without using the generation model 30, and generate an improvement suggestion such as, "For part X, please perform the following in order: process 1 (object "cushioning material", action "place") → process 2 (object "tape α, part X", action "fix")... → process 7 (object "tape γ, outer box", action "seal")."

[0083] The generation unit 237 generates improvement suggestions for each task that falls within the top 10% of differences other than the individual packaging of part X, thereby generating suggestion information and creating a recommendation screen that includes the suggestion information.

[0084] Figure 8 shows an example of a recommendation screen 601 that includes the suggestion information 631 of this embodiment. In the example shown in Figure 8, the recommendation screen 601 includes suggestion information 631 for each task that is within the top 10% of differences, including the individual packaging of part X, with items such as priority, part, shortest time, average difference, and improvement suggestion. The item "procedure" included in the improvement suggestion has a link to a frame video of the task with the shortest work time used to generate the improvement suggestion.

[0085] The output unit 225 outputs the suggestion information generated by the generation unit 237 to the first worker. Specifically, the output unit 225 outputs a recommendation screen containing the suggestion information generated by the generation unit 237 to the first worker's worker terminal 40, and the control device 41 of the worker terminal 40 displays, for example, the recommendation screen shown in Figure 8 on the display device 46.

[0086] Next, the operation of the work analysis system of this embodiment will be described.

[0087] Figure 9 is a flowchart showing an example of the work recognition process performed by the information processing device 20 in this embodiment. The work recognition process shown in Figure 9 may be performed sequentially or periodically.

[0088] First, the video acquisition unit 201 acquires multiple videos from multiple shooting devices 10-1 to 10-n, capturing the work status of multiple workers, and stores (accumulates) them in the video storage unit 203 (step S101).

[0089] Next, the work recognition unit 209 acquires video from the video storage unit 203, inputs the acquired video into the recognition model stored in the recognition model storage unit 207, and obtains the recognition result, thereby recognizing the work being performed by the worker shown in the video on a work process basis (step S103).

[0090] Next, the work recognition unit 209 stores the work recognition result as a work history in the work history storage unit 211 (step S105).

[0091] Figure 10 is a sequence diagram showing an example of the condition setting process performed in the work analysis system 1 of this embodiment. Below, the condition setting process shown in Figure 10 will be explained using the setting of filter conditions and the number of items to display as an example.

[0092] First, the reception unit 223 receives a request from the worker terminal 40 to output a recommendation screen (improvement suggestion screen) for making recommendations (improvement suggestions) based on instructions from the first worker to the worker terminal 40 (step S201).

[0093] Next, the output unit 225 outputs the recommendation screen to the worker terminal 40 (step S203), and the control device 41 of the worker terminal 40 displays the recommendation screen on the display device 46 (step S205).

[0094] Next, the first worker uses the input device 45 to set the filter conditions and the number of items to display on the recommendation screen 601 (step S207). When the worker selects a setting button (not shown), the worker terminal 40 outputs these set conditions to the information processing device 20 (step S209).

[0095] Next, the reception unit 223 receives these conditions from the worker terminal 40, and the management unit 231 stores these conditions in the condition storage unit 233 and manages these conditions on the condition storage unit 233 (step S211).

[0096] Figure 11 is a sequence diagram showing an example of the improvement suggestion process performed in the work analysis system 1 of this embodiment. The improvement suggestion process shown in Figure 11 may be performed, for example, following the process shown in Figure 10.

[0097] First, the identification unit 235 calculates the difference between the shortest working time for each type of work and the working time of the first worker, and identifies one or more types of work for which the difference is received by the receiving unit 223 (stored in the condition storage unit 233) and satisfies predetermined conditions (step S301).

[0098] Next, the generation unit 237 generates a prompt using at least the worker recognition result that results in the shortest work time for each of the one or more types of work (step S303), and inputs the generated prompt into the generation model 30 to obtain the suggested information (step S305).

[0099] Next, the generation unit 237 generates a recommendation screen that includes suggested information (step S307).

[0100] Next, the output unit 225 outputs the recommendation screen to the worker terminal 40 (step S309), and the control device 41 of the worker terminal 40 displays the recommendation screen on the display device 46 (step S311).

[0101] In this embodiment, the recommendation screen 601 (see Figures 6 and 8) is used to set predetermined conditions and display suggested information. However, the screen for setting predetermined conditions and the screen for displaying suggested information may be displayed separately. In this case, the screen for setting predetermined conditions may include input boxes for setting other conditions in addition to filters and the number of items to display.

[0102] As described above, the work analysis system of this embodiment analyzes video footage of each worker's work, identifies tasks requiring improvement using the analysis results, and proposes improvements to the identified tasks. This enables the improvement of tasks that were initially unclear (unknown) and required improvement. For example, even if a worker takes longer to complete a task than other workers, they can reduce their work time by understanding the optimal method for that worker.

[0103] (Variation 1) In the above embodiment, the example given was that the proposed information is a suggestion to rearrange the order of the work steps for one or more types of work, but it is not limited to this. The proposed information may be, for example, a suggestion regarding the placement of items used in at least part of the work steps, and a suggestion regarding work actions in at least part of the work steps. For example, if the reason why work takes time is that a tool such as tape is placed far away, the proposed information may suggest placing the tool closer to the worker. Also, for example, if the reason why work takes time is that redundant actions such as applying tape multiple times are performed, the proposed information may suggest improving the redundant action (performing it in one go).

[0104] (program) The programs executed by the information processing device 20 in the above embodiments and modified examples are provided as installable or executable files stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0105] Furthermore, the program executed by the information processing device 20 in the above embodiment and modified example may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device 20 in the above embodiment and modified example may be provided or distributed via a network such as the Internet. Alternatively, the program executed by the information processing device 20 in the above embodiment and modified example may be pre-installed in ROM or the like and provided.

[0106] The program executed in the information processing device 20 of the above embodiment and modified example is configured as a module for realizing each of the above-described parts on a computer. In actual hardware, for example, the CPU reads the learning program from the HDD into RAM and executes it, thereby realizing each of the above-described parts on the computer.

[0107] The above embodiments and modifications are merely examples of how this disclosure may be implemented, and they do not restrict the technical scope of this disclosure. Therefore, this disclosure can be implemented in various ways without departing from its gist or its main features. For example, the above embodiments and modifications may be combined as appropriate on a component basis. Also, for example, some components may be removed from the total components in the above embodiments and modifications.

[0108] This disclosure includes the following aspects:

[0109] (1) A video acquisition unit that acquires multiple videos of the work status of multiple workers, A work recognition unit analyzes the aforementioned multiple videos to recognize multiple types of tasks performed by each worker, Based on the recognition results of the aforementioned work recognition unit, the identification unit identifies one or more types of work from among the multiple types of work that are subject to improvement for the first worker, A generation unit that generates proposal information showing improvement plans for each of the one or more specified types of work, An output unit that outputs the proposed information to the first worker, An information processing device equipped with the following features.

[0110] (2) The work recognition unit further recognizes the time required for each work, The identification unit identifies one or more types of work based on the work time included in the recognition result. The information processing device described in (1) above.

[0111] (3) The specified unit calculates the difference between the shortest working time for each type of work and the working time of the first worker, The one or more types of work mentioned above are types of work where the difference satisfies the predetermined conditions. The information processing device described in (2) above.

[0112] (4) The system further includes a reception unit that receives input of the predetermined conditions from the worker terminal used by the first worker, An information processing device as described in any one of the above (3).

[0113] (5) Each time the specified conditions are input, the specified unit re-specifies the one or more types of work. The information processing device described in (4) above.

[0114] (6) The work recognition unit recognizes each work as a work process unit, The proposed information is a suggestion to improve at least a part of the work process for each of the one or more types of work described above. An information processing device as described in any one of (1) to (5) above.

[0115] (7) The proposed information is at least one of the following: a proposal to rearrange the order of the work processes for each of the one or more types of work; a proposal regarding the arrangement of items used in at least part of the work processes; and a proposal regarding work actions in at least part of the work processes. The information processing device described in (6) above.

[0116] (8) The work recognition unit further recognizes the time required for each work, The generation unit generates the proposed information for each of the one or more types of work using the worker recognition result that results in the shortest work time. An information processing device as described in any one of (1) to (5) above.

[0117] (9) The work recognition unit further recognizes the time required for each work, The generation unit generates a prompt for each of the one or more types of work using at least the worker recognition result that results in the shortest work time, and generates the proposed information by inputting the generated prompt into the generation model. The information processing device described in (8) above.

[0118] (10) A video acquisition step in which the video acquisition unit acquires multiple videos of the work status of multiple workers, The work recognition unit analyzes the multiple videos and recognizes multiple types of work performed by each worker in the work recognition step, The identification unit identifies one or more types of work from among the multiple types of work that are subject to improvement by the first worker, based on the recognition result of the work recognition unit, in the identification step, The generation unit generates proposal information that shows improvement proposals for each of the one or more specified types of work, The output unit performs an output step of outputting the proposed information to the first worker, A method for proposing improvements, including the following.

[0119] (11) A video acquisition step in which multiple videos are taken of the work status of multiple workers, A work recognition step involves analyzing the aforementioned multiple videos to recognize multiple types of tasks performed by each worker, Based on the recognition results of the work recognition unit, the identification step involves identifying one or more types of work from among the multiple types of work that are subject to improvement by the first worker, A generation step that generates proposal information showing improvement proposals for each of the one or more types of work identified above, An output step to output the proposed information to the first worker, A program that causes a computer to execute something. [Explanation of Symbols]

[0120] 1. Work Analysis System 2 Network 10, 10-1 to 10-n Multiple imaging devices 20 Information Processing Devices 30 Generative Models 40, 40-1~40-m Multiple worker terminals 201 Video Acquisition Unit 203 Video Memory Unit 207 Recognition Model Memory Unit 209 Work Recognition Unit 211 Work history storage unit 221 Input / Output Control Unit 223 Reception Department 225 Output section 231 Management Department 233 Condition memory section 235 Specific part 237 Generation part

Claims

1. A video acquisition unit that acquires multiple videos capturing the work status of multiple workers, A work recognition unit analyzes the aforementioned multiple videos to recognize multiple types of tasks performed by each worker, Based on the recognition results of the aforementioned work recognition unit, the identification unit identifies one or more types of work from among the multiple types of work that are subject to improvement for the first worker, A generation unit that generates proposal information showing improvement plans for each of the one or more specified types of work, An output unit that outputs the proposed information to the first worker, An information processing device equipped with the following features.

2. The aforementioned work recognition unit further recognizes the time required for each task, The identification unit identifies one or more types of work based on the work time included in the recognition result. The information processing apparatus according to claim 1.

3. The specified unit calculates, for each type of work, the difference between the shortest work time for that work and the work time of the first worker. The one or more types of work mentioned above are types of work where the difference satisfies the predetermined conditions. The information processing apparatus according to claim 2.

4. The system further includes a reception unit that receives input of the predetermined conditions from the worker terminal used by the first worker, The information processing apparatus according to claim 3.

5. The identifying unit, each time it receives input of the predetermined conditions, re-identifies the one or more types of work. The information processing apparatus according to claim 4.

6. The aforementioned work recognition unit recognizes each work as a work process unit, The proposed information is a suggestion to improve at least a part of the work process for each of the one or more types of work described above. The information processing apparatus according to any one of claims 1 to 5.

7. The proposed information includes at least one of the following: a proposal to rearrange the order of the work processes for each of the one or more types of work; a proposal regarding the arrangement of items used in at least part of the work processes; and a proposal regarding work actions in at least part of the work processes. The information processing apparatus according to claim 6.

8. The aforementioned work recognition unit further recognizes the time required for each task, The generation unit generates the proposed information for each of the one or more types of work using the worker recognition result that results in the shortest work time. The information processing apparatus according to any one of claims 1 to 5.

9. The aforementioned work recognition unit further recognizes the time required for each task, The generation unit generates a prompt for each of the one or more types of work using at least the worker recognition result that results in the shortest work time, and generates the proposed information by inputting the generated prompt into the generation model. The information processing apparatus according to claim 8.

10. The video acquisition unit acquires multiple videos capturing the work status of multiple workers in a video acquisition step, The work recognition unit analyzes the multiple videos and recognizes multiple types of work performed by each worker in the work recognition step, The identification unit identifies one or more types of work from among the multiple types of work that are subject to improvement by the first worker, based on the recognition result of the work recognition unit. The generation unit generates proposal information indicating improvement plans for each of the one or more specified types of work, The output unit performs an output step of outputting the proposed information to the first worker, A method for proposing improvements, including the following.

11. A video acquisition step involves acquiring multiple videos capturing the work status of multiple workers, A work recognition step involves analyzing the aforementioned multiple videos to recognize multiple types of tasks performed by each worker, Based on the recognition results of the work recognition unit, the identification step involves identifying one or more types of work from among the multiple types of work that are subject to improvement by the first worker, A generation step that generates proposal information showing improvement plans for each of the one or more specified types of work, An output step to output the proposed information to the first operator, A program that causes a computer to execute something.

Images