Video data processing system, video data processing method, and program

Abstract

Provided are a video data processing system, a video data processing method, and a program that make it possible to improve the efficiency of analysis of video data. This video data processing system comprises a computation circuit (45) that is capable of accessing one or more pieces of video data (D1) in which are shown one or more tasks that one or more objects are involved in. The computation circuit (45) is configured so as to: detect the state of one or more objects in first video data (D4) which is a portion of the one or more pieces of video data (D1); determine second video data (D5) which is another portion of the one or more pieces of video data (D1) on the basis of the state of the one or more objects in the first video data (D4); output a display screen that includes a reproduction window in which the first video data (D4) is reproduced and a button for reproducing the second video data (D5); and reproduce the second video data (D5) in response to the button being selected.

Description

Video data processing system, video data processing method, and program

[0001] The present disclosure relates to a video data processing system, a video data processing method, and a program, and more particularly to a video data processing system, a video data processing method, and a program for analyzing video data.

[0002] Patent Literature 1 discloses a work improvement support system that supports the improvement of work processes. The work improvement support system disclosed in Patent Literature 1 includes a performance management unit that manages performance data for each work included in the work process, a plan management unit that manages plan data for each work, a work video management unit that manages work video data capturing the status of the work site where each work is performed by a worker, and a work analysis unit that determines whether a work corresponds to a predetermined work that should be improved based on the performance data and plan data related to the work, and if it is determined that the work corresponds to the predetermined work, extracts predetermined work video data related to the work determined to correspond to the predetermined work from the work video data.

[0003] International Publication No. 2019 / 021592

[0004] According to Patent Document 1, it is possible to extract predetermined work video data related to work determined to be a predetermined work that should be improved. In Patent Document 1, the predetermined work video data allows the user to confirm what the predetermined work is like. However, in order to improve the predetermined work, it is necessary to identify the cause of the occurrence of the predetermined work, and it is often necessary to check not only the predetermined work video data but the entire work video data, which is very time-consuming and labor-intensive.

[0005] The present disclosure provides a video data processing system, a video data processing method, and a program that enable more efficient analysis of video data.

[0006] A video data processing system according to one aspect of the present disclosure includes an arithmetic circuit that can access one or more video data depicting one or more tasks involving one or more objects, and the arithmetic circuit is configured to detect the state of one or more objects in first video data that is a part of the one or more video data, determine second video data that is another part of the one or more video data based on the state of the one or more objects in the first video data, output a display screen that includes a playback window for playing the first video data and a button for playing the second video data, and execute playback of the second video data in response to selection of the button.

[0007] A video data processing method according to one aspect of the present disclosure is executed by an arithmetic circuit, detects the state of one or more objects in first video data that is part of one or more video data depicting one or more tasks involving one or more objects, determines second video data that is another part of the video data based on the state of the one or more objects in the first video data, outputs a display screen including a playback window for playing the first video data and a button for playing the second video data, and executes playback of the second video data in response to selection of the button.

[0008] A program according to one aspect of the present disclosure is a program (computer program) for causing an arithmetic circuit to execute the above-described video data processing method.

[0009] Aspects of the present disclosure enable more efficient analysis of video data.

[0010] Schematic diagram of a video data processing system according to an embodiment; Schematic diagram of a facility from which video data to be processed by the video data processing system according to an embodiment is obtained; Block diagram of a processing device of the video data processing system according to an embodiment; Flowchart of an example of processing by an arithmetic circuit of the data processing system according to an embodiment; Explanatory diagram of a first example of an image in video data; Explanatory diagram of a second example of an image in video data; Explanatory diagram of a third example of an image in video data; Explanatory diagram of an example of a state of an object in video data; Explanatory diagram of an example of a selection screen of the data processing system according to an embodiment; Explanatory diagram of a first example of a display screen of the data processing system according to an embodiment; Explanatory diagram of a second example of a display screen of the data processing system according to an embodiment; Explanatory diagram of a third example of a display screen of the data processing system according to an embodiment; Explanatory diagram of a fourth example of a display screen of the data processing system according to an embodiment; Explanatory diagram of a fifth example of a display screen of the data processing system according to an embodiment;

[0011] [1. Embodiments] Hereinafter, embodiments of the present disclosure will be described, with reference to the drawings where appropriate. However, the following embodiments are merely examples for explaining the present disclosure, and are not intended to limit the present disclosure to the following content (e.g., the shape, dimensions, and arrangement of each component). Positional relationships, such as up, down, left, and right, are based on the positional relationships shown in the drawings unless otherwise specified. Each figure described in the following embodiments is a schematic diagram, and the ratios of the size and thickness of each component in each figure do not necessarily reflect the actual dimensional ratios. Furthermore, the dimensional ratios of each element are not limited to the ratios shown in the drawings.

[0012] In the following description, when it is necessary to distinguish between multiple components, prefixes such as "first" and "second" are added to the names of the components. However, when the components can be distinguished from each other by the symbols attached to them, the prefixes such as "first" and "second" may be omitted in consideration of readability of the text.

[0013] In the following description, when it is necessary to distinguish between multiple components, suffixes such as "-1" and "-2" are added to the symbols of the components. However, when it is not necessary to distinguish between multiple components, the suffixes "-1" and "-2" may be omitted to improve readability.

[0014] 1 is a schematic diagram of a video data processing system 1 according to the present embodiment. The video data processing system 1 is used to process video data showing one or more operations involving one or more objects. The video data is obtained, for example, in a workspace 100.

[0015] FIG. 2 is a schematic diagram of a workspace 100. The workspace 100 is a facility (e.g., a factory) where products are actually manufactured. Products are manufactured from raw materials through multiple processes. Products include, but are not limited to, a variety of items such as food, medicine, electrical appliances, jewelry, furniture, and vehicles. Products are not limited to finished products, but may also be parts. An example of a finished product is an automobile, and an example of a part is a core part of the automobile (such as a piston).

[0016] The workspace 100 includes multiple cells 110-1 to 110-N (N is any integer) for manufacturing products using a cellular production system. Each cell 110 may include one or more manufacturing equipment that enables manufacturing using various manufacturing technologies. Examples of manufacturing technologies include additive manufacturing technologies (material extrusion, liquid vapour photopolymerization, material jetting, binder jetting, powder bed fusion, sheet lamination, directed energy deposition, etc.), subtractive manufacturing technologies (cutting, grinding, electrical discharge machining, casting, die casting, pressing, forging, sheet metal processing, etc.), formative manufacturing technologies (injection molding, extrusion molding, etc.), surface treatment technologies (coating, painting, plating, polishing, etc.), heat treatment technologies (sintering, cooling, etc.), joining technologies (ultrasonic bonding, thermal welding, mechanical bonding, adhesives, etc.), and assembly technologies (part assembly, micro-imprinting, impregnation, etc.). The cell 110 may include not only manufacturing equipment but also measurement equipment, inspection equipment, and transport equipment. Examples of the workspace 100 include factories, stores, and buildings (whole buildings, floors). The workspace 100 may be a manufacturing facility that produces products using a line production system rather than a cell production system. The workspace 100 is not limited to a manufacturing facility, but may be any facility where one or more tasks involving one or more objects can be performed. For example, the workspace 100 may include non-residential facilities such as factories, offices, buildings, stores, schools, welfare facilities, or medical facilities (hospitals, etc.), as well as residential facilities such as detached houses, apartment buildings, or individual units in detached houses or apartment buildings. Non-residential facilities include theaters, movie theaters, public halls, amusement parks, complexes, restaurants, department stores, hotels, inns, kindergartens, libraries, museums, art galleries, underground shopping malls, train stations, airports, and the like. Furthermore, the workspace 100 may include not only buildings (structures) but also outdoor facilities such as baseball stadiums, gardens, parking lots, athletic fields, and parks. The workspace 100 may also be a mobile facility such as a vehicle, ship, or airplane.

[0017] In each cell 110, a worker 121 performs work assigned to the cell 110. Multiple workers 121-1 to 121-N perform work in sequence, resulting in a finished product from the raw materials of the product, via one or more work-in-progress items 122-1 to 122-N. As an example, in cell 110-2, worker 121-2 works on work-in-progress 122-2 from cell 110-1 upstream of cell 110-2, completes work-in-progress 122-3, and sends it to cell 110-3 downstream of cell 110-2. As will be described in more detail below, the objects involved in the work in cell 110 are the worker 121, the work-in-progress item 122, and the workbench.

[0018] 1 again, the video data processing system 1 includes a camera system 2, a sensor system 3, a processing device 4, and an information terminal 5.

[0019] The information terminal 5 is used by a user 6. The information terminal 5 can be realized, for example, by a personal computer (desktop computer, laptop computer), a mobile terminal (smartphone, tablet terminal, etc.), etc. The data processing system 1 can present information to the user 6 via the information terminal 5 and, if necessary, request the user 6 to input information.

[0020] The camera system 2 is installed in the workspace 100. The camera system 2 is used to acquire video data D1 showing one or more tasks involving one or more objects. The video data D1 is used to determine the state of the objects. The video data D1 may include chronologically ordered image data, time information, and location information.

[0021] In this embodiment, the camera system 2 includes first cameras 21-1 to 21-N and a second camera 22. The first camera 21 is used to capture a narrower range than the second camera 22. In this embodiment, as shown in FIG. 2, the first camera 21 captures images of work performed by workers 121 on work-in-progress 122 in a cell 110. The first camera 21 can collect video data D1 about the work performed by each worker 121 on the work-in-progress 122. The second camera 22 is used to capture a wider range than the first camera 21. In this embodiment, as shown in FIG. 2, the second camera 22 captures images of work performed by workers 121-1 to 121-N on the work-in-progress 122 in multiple cells 110-1 to 110-N. The second camera 22 can collect video data D1 about the work performed by multiple workers 121-1 to 121-N on the work-in-progress 122. In particular, the first camera 21 is advantageous for grasping the detailed condition of each worker 121 , and the second camera 22 is advantageous for grasping the overall condition of each worker 121 .

[0022] The sensor system 3 is used to acquire sensor data D2 related to one or more objects. The sensor data D2, together with the video data D1, is used to determine the state of the objects. The sensor data D2 may include data on the presence or absence of a worker 121, data on the number of work-in-progress 122, or data on the weight of items placed on a workbench. The data on the presence or absence of a worker 121 is obtained, for example, by a human presence sensor. The data on the number (or amount) of work-in-progress 122 is obtained by a weight sensor or an optical sensor. The data on the weight of items placed on a workbench is obtained by a weight sensor. In this embodiment, the sensor system 3 includes multiple sensors 31-1 to 31-N. The multiple sensors 31-1 to 31-N are, for example, human presence sensors. The multiple sensors 31-1 to 31-N may be, for example, human presence sensors. The multiple sensors 31-1 to 31-N may be arranged, for example, in each of the multiple cells 110-1 to 110-N.

[0023] The processing device 4 is used to process the video data D1. The processing device 4 is connected to the camera system 2, the sensor system 3, and the information terminal 5 so as to be able to communicate with them.

[0024] 3 is a block diagram of the processing device 4. The processing device 4 includes an input device 41, an output device 42, a communication device 43, a storage device 44, and an arithmetic circuit 45. The processing device 4 can be realized by, for example, one or more servers or the like.

[0025] The input device 41 includes one or more human-machine interfaces for inputting information. Examples of the human-machine interface include a keyboard, a pointing device (such as a mouse or a trackball), a touchpad, a position input device for a touch panel display, and the like. The one or more human-machine interfaces of the input device 41 may be built into the processing device 4 or may be externally attached. That is, the input device 41 may include a human-machine interface of the processing device 4 itself and a human-machine interface connected to the processing device 4.

[0026] The output device 42 includes one or more human-machine interfaces for outputting information. Examples of the human-machine interface include a display, a speaker, a touch panel display, and the like. The one or more human-machine interfaces of the output device 42 may be built into the processing device 4 or may be externally attached. That is, the output device 42 may include a human-machine interface of the processing device 4 itself and a human-machine interface connected to the processing device 4. The human-machine interface may be connected to the processing device 4 via a network and located in a remote location.

[0027] The communication device 43 is used for communication through a communication network. The communication device 43 has one or more communication interfaces. The communication device 43 is connectable to a communication network and has the function of communicating through the communication network. The communication device 43 complies with a predetermined communication protocol. The predetermined communication protocol can be selected from various well-known wired and wireless communication standards. In this embodiment, the processing device 4 is communicatively connected to the camera system 2, the sensor system 3, and the information terminal 5 via the communication device 43.

[0028] The storage device 44 includes one or more storages (non-transitory storage media). The storages may be, for example, hard disk drives, optical drives, or solid-state drives (SSDs). The storages may be internal, external, or network-attached storage (NAS).

[0029] The information stored in the storage device 44 includes one or more pieces of video data D1, one or more pieces of sensor data D2, one or more pieces of equipment data D3, one or more pieces of first video data D4, and one or more pieces of second video data D5. Fig. 3 shows a state in which the storage device 44 stores the video data D1, the sensor data D2, the equipment data D3, the first video data D4, and the second video data D5. The video data D1 and the sensor data D2 do not need to be stored in the storage device 44 at all times; they only need to be stored in the storage device 44 when needed by the arithmetic circuit 45.

[0030] The workspace data D3 includes information about the workspace 100. By using the workspace data D3, information about the workspace 100 can be used to analyze the video data D1, improving the accuracy of the analysis. If the workspace 100 is a manufacturing facility, the workspace data D3 includes time information, product information, location information, model information, lot information, facility information, etc.

[0031] The arithmetic circuit 45 is connected to the input device 41, the output device 42, and the communication device 43, and can access the storage device 44. The arithmetic circuit 45 can be realized, for example, by a computer system. The computer system includes one or more processors (microprocessors) and one or more memories. The one or more processors execute programs (stored in one or more memories or the storage device 44) to realize various functions of the processing device 4. The programs may be pre-recorded in the storage device 44, or may be provided via a telecommunications line such as the Internet, or recorded on a non-transitory recording medium such as a memory card.

[0032] The arithmetic circuit 45 executes processing for displaying the first moving image data D4 and the second moving image data D5 based on the moving image data D1.

[0033] Fig. 4 is a flowchart of an example of processing by the arithmetic circuit 45 of the moving image data processing system 1. Fig. 4 shows processing by the arithmetic circuit 45 to identify the first moving image data D4 and the second moving image data D5 from the moving image data D1.

[0034] The arithmetic circuit 45 acquires data (S1). For example, the arithmetic circuit 45 acquires video data D1 from each of the first cameras 21-1 to 21-N and the second camera 22 of the camera system 2. The arithmetic circuit 45 acquires sensor data D2 from each of the multiple sensors 31 of the sensor system 3. The arithmetic circuit 45 acquires workspace data D3. When acquiring the video data D1, the arithmetic circuit 45 may acquire the video data D1 from the camera system 2, or may acquire the video data D1 stored in advance in the storage device 44. When acquiring the sensor data D2, the arithmetic circuit 45 may acquire the sensor data D2 from the sensor system 3, or may acquire the sensor data D2 stored in advance in the storage device 44. When acquiring the workspace data D3, the arithmetic circuit 45 may acquire the workspace data D3 via the input device 41 or the communication device 43, or may acquire the workspace data D3 stored in advance in the storage device 44.

[0035] The arithmetic circuit 45 detects (determines) the state of one or more objects in one or more video data D1 (S2). The arithmetic circuit 45 detects the state of the object based on the video data D1 and, as necessary, the sensor data D2 and the workspace data D3.

[0036] As an example, the arithmetic circuitry 45 may detect an object from one or more pieces of video data D1 and record the movement path of the detected object. The arithmetic circuitry 45 may connect the movement paths of the object detected from the plurality of pieces of video data D1 to form the movement path of the object throughout the entire workspace 100. This makes it possible to determine the location of a specific object in the workspace 100, and to easily determine in which of the video data D1 of the cameras 21 and 22 an object that has disappeared from the video data D1 of the cameras 21 and 22 being focused on during a certain time period is captured.

[0037] As an example, the arithmetic circuit 45 can link the video data D1, the sensor data D2, and the workspace data D3 to one another. Based on the time information and position information of the video data D1 and the workspace data D3, the arithmetic circuit 45 can link the product information, model information, lot information, etc. of the workspace data D3 to the object detected from the video data D1. This enables a more detailed analysis of the object.

[0038] In this embodiment, the one or more objects include at least one of a worker 121, work-in-progress 122 that is a product manufactured by the work, and a workbench. The state of the object includes the state of the worker 121, the number of work-in-progress 122, and the workbench. The number of work-in-progress 122 may include the number of work-in-progress on the upstream side or the number of work-in-progress on the downstream side. The workbench is a table on which a work object that is the target of work by the worker 121 is placed. The work object may be a work-in-progress 122.

[0039] In this embodiment, the arithmetic circuit 45 determines whether the state of the object is abnormal. An abnormal state of the worker 121 may include a state in which the worker 121 has stopped working or is absent. An abnormal state of the work-in-progress 122 may include a state in which the number of work-in-progress 122 on the upstream side is less than a predetermined number (a state in which there is a shortage of work-in-progress 122 on the upstream side) or a state in which the number of work-in-progress 122 on the downstream side is more than a predetermined number (a state in which there is an excess of work-in-progress 122 on the downstream side). An abnormal state of the workbench may include a state in which there is no work object on the workbench or a state in which the work object on the workbench has not moved for a long period of time.

[0040] The arithmetic circuit 45 detects the state of the object based on the video data D1 and, if necessary, the sensor data D2.

[0041] 5 to 7 are explanatory diagrams of first to third example images of the video data D1 at a certain time. In the first example image shown in Fig. 5, the worker 121 has stopped working, and there is a shortage of work-in-progress 122 on the upstream side. In the second example image shown in Fig. 6, the worker 121 has stopped working, and there is an excess of work-in-progress 122 on the downstream side. In the third example image shown in Fig. 7, the worker 121 is absent.

[0042] Whether the state of the object indicates that the worker 121 has stopped working can be determined based on the movements of the worker 121 captured in the video data D1. The state in which the worker 121 has stopped working may include, for example, a state in which the worker 121 has not started working, or a state in which the working time of the worker 121 is longer than a threshold (a state in which the worker 121 has not completed working). As an example, as shown in Figures 5 and 6, an image region R1 of a predetermined part of the worker 121, such as the arm, can be extracted from the video data D1 using image recognition technology, and it can be determined whether the worker 121 is working based on changes in the image region R1, etc.

[0043] Whether the state of the object indicates that the worker 121 is absent can be determined based on whether the worker 121 is captured in the video data D1. As an example, as shown in Fig. 7, the presence or absence of the worker 121 can be detected from the video data D1 using image recognition technology. If the sensor data D2 includes data on the presence or absence of the worker 121, it can be determined whether the state of the object indicates that the worker 121 is absent based on the sensor data D2.

[0044] The number of work-in-progress items 122 can be determined based on the number of work-in-progress items 122 in the video data D1. As an example, as shown in FIGS. 5 and 6 , image areas R21 and R22 of the locations where the work-in-progress items 122 are placed can be extracted from the video data D1, and the number of work-in-progress items 122 can be determined based on the image areas R21 and R22. The image area R21 in FIG. 5 corresponds to the location where the work-in-progress items 122 are placed upstream, and the number of work-in-progress items 122 on the upstream side can be determined based on the image area R21. The image area R22 in FIG. 6 corresponds to the location where the work-in-progress items 122 are placed downstream, and the number of work-in-progress items 122 on the downstream side can be determined based on the image area R22. If the sensor data D2 includes data on the number of work-in-progress items 122, the number of work-in-progress items 122 can be determined based on the sensor data D2.

[0045] FIG. 8 is an explanatory diagram of an example of the state of an object in video data D1. In FIG. 8, in the section from time t1 to t2, as in the first example of the image in FIG. 5, the state of the worker 121 is determined to be a state in which the worker 121 has stopped working (particularly, the worker 121 has not started working), and it is determined that there is a shortage of work-in-progress items 122 on the upstream side. In the section from time t3 to t4, as in the second example of the image in FIG. 6, the state of the worker 121 is determined to be a state in which the worker 121 has stopped working (particularly, the worker 121's working time is longer than the threshold), and it is determined that there is an excess of work-in-progress items 122 on the downstream side. In the section from time t5 to t6, as in the third example of the image in FIG. 7, the state of the worker 121 is determined to be absent. In FIG. 5, the state of the object is determined to be not abnormal in the time intervals t0 to t1, t2 to t3, t4 to t5, and t6 to t7.

[0046] The arithmetic circuit 45 determines the first moving image data D4 (S3). In this embodiment, the arithmetic circuit 45 determines the first moving image data D4 based on the states of one or more objects in one or more moving image data D1. The first moving image data D4 is at least a portion of the one or more moving image data D1. The first moving image data D4 is mainly divided by changes in the states of the objects. In the example of FIG. 8 , the portions of the moving image data D1 corresponding to the sections from time t0 to t1, t1 to t2, t2 to t3, t3 to t4, t4 to t5, and t6 to t7 respectively become the first moving image data D4.

[0047] The arithmetic circuit 45 determines second video data D5 (S4). The second video data D5 is a portion of the one or more video data D1 that is considered preferable to refer to together with the first video data D4. The second video data D5 is determined from the one or more video data D1 based on the state of one or more objects in the first video data D4. In this embodiment, the second video data D5 is video data related to an abnormal state of the object in the first video data D4. In this embodiment, the second video data D5 is not determined for first video data in which the state of the object is not abnormal.

[0048] Examples of the second video data D5 include (1) video data showing an object that is no longer shown in the first video data, (2) video data of a time period before the same work as the first video data, (3) video data of a time period after the same work as the first video data, (4) video data of work upstream of the same time period as the first video data, or (5) video data of work downstream of the same time period as the first video data. Here, the term "same time period" does not mean the same time period in the strict sense, i.e., the same start time and end time, but it is sufficient that the time period of the first video data D4 and the time period of the second video data D5 overlap at least partially.

[0049] A plurality of second video data D5 may be determined for the first video data D1. Priorities may be assigned to the plurality of second video data D5. The second video data D5 with the highest priority among the plurality of second video data D5 may be highlighted to distinguish it from the rest. The priority may be determined, for example, based on the degree of relevance to the first video data D1, past performance, etc. This makes it possible to recommend second video data D5 that is highly relevant to the first video data D4 from the plurality of second video data D5.

[0050] For example, if the first video data D4 corresponds to work by the worker 121-2, then "(1) video data showing an object that is no longer shown in the first video data" is video data showing the worker 121-2 during the same time period as the first video data D4. If the worker 121-2 is not in the cell 110-2, the worker 121-2 may be performing some work in another location in the workspace 100. "(1) video data showing an object that is no longer shown in the first video data" is video data showing the work of the worker 121-2 in such another location.

[0051] For example, if the first video data D4 corresponds to work performed by worker 121-2, then "(2) video data from a time period before the same work as the first video data" corresponds to the same work performed by worker 121-2 in the past, and "(3) video data from a time period after the same work as the first video data" corresponds to the same work performed by worker 121-2 in the future.

[0052] For example, if the first video data D4 corresponds to the work of worker 121-2, then "(4) Video data of upstream work in the same time period as the first video data" corresponds to the work of worker 121-1, and "(5) Video data of downstream work in the same time period as the first video data" corresponds to the work of worker 121-3.

[0053] When the abnormal state is a stoppage of the work of the worker 121, the second video data D5 preferably includes at least one of "(4) video data of upstream work in the same time period as the first video data" or "(5) video data of downstream work in the same time period as the first video data." When the abnormal state is a state in which the number of work-in-progress 122 on the upstream side is less than a predetermined number, the second video data D5 preferably includes "(4) video data of upstream work in the same time period as the first video data." When the abnormal state is a state in which the number of work-in-progress 122 on the downstream side is greater than or equal to a predetermined number, the second video data D5 preferably includes "(5) video data of downstream work in the same time period as the first video data." This is because if an abnormality occurs in the upstream work and the work-in-progress 122 does not arrive from the upstream side, the worker 121 may be unable to perform the work. Furthermore, if an abnormality occurs in the downstream work and the work-in-progress 122 cannot be sent downstream, the worker 121 may be unable to perform the work.

[0054] If the abnormal state is the absence of the worker 121, the second video data D5 preferably includes at least one of “(1) video data showing an object that is no longer shown in the first video data,” “(2) video data taken in a time period before the same work as the first video data,” or “(3) video data taken in a time period after the same work as the first video data.” This is because the absence of the worker 121 may mean that the worker 121 is dealing with some kind of trouble.

[0055] When the abnormal state is a state in which there is no work object on the workbench, it is preferable that the second video data D5 include "(4) video data of upstream work in the same time period as the first video data." This is because when there is no work object on the workbench, there is a possibility that there is an abnormality in the upstream work, and that the work in progress 122 will not arrive from the upstream side.

[0056] If the abnormal state is a state in which the work object on the workbench does not move for a long period of time, it is preferable to include at least one of "(1) video data showing the object that is no longer shown in the first video data," "(2) video data from a time period before the same work as the first video data," "(3) video data from a time period after the same work as the first video data," or "(5) video data of work downstream in the same time period as the first video data." This is because if the work object on the workbench does not move for a long period of time, there is a possibility that the worker 121 has stopped working or is absent.

[0057] In this way, the arithmetic circuit 45 can determine the first moving image data D4 and the second moving image data D5 from the one or more moving image data D1 based on the states of the one or more objects in the one or more moving image data D1. Information on the first moving image data D4 and the second moving image data D5 is stored in the storage device 44. The information on the first moving image data D4 and the second moving image data D5 is information that can identify the first moving image data D4 and the second moving image data D5, and may be information on the first moving image data D4 and the second moving image data D5 themselves, or information on portions of the one or more moving image data D1 that correspond to the first moving image data D4 and the second moving image data D5.

[0058] The arithmetic circuit 45 outputs a selection screen for selecting the first video data D from the one or more video data D1 (S5). For example, the arithmetic circuit 45 transmits data necessary for displaying the selection screen on the information terminal 5 to the information terminal 5 in response to an input from the user 6 to the information terminal 5.

[0059] FIG. 9 is an explanatory diagram of an example of a selection screen. The selection screen G1 shown in FIG. 9 provides a graphical user interface (GUI) that enables playback of video data D1. The selection screen G1 displays seek bars SB1 to SB7 for video data D1 corresponding to cells 1 to 7. Operating the seek bars SB1 to SB7 enables playback of video data D1. In FIG. 9, each seek bar SB1 to SB7 is displayed in a manner that distinguishes between sections in which the object is in an abnormal state and sections in which the object is not in an abnormal state. This allows easy selection of first video data D1 in which the object is in an abnormal state from video data D1. In the seek bars SB1 to SB7, sections P11 to P13, P21, P31, P32, P41, P42, P51, P52, P61, P71, and P72, indicated by hatching with dots, indicate first video data D1 in which the object is in an abnormal state. In the seek bars SB1 to SB7, the sections without hatching indicate the first moving image data D1 in which the state of the object is not abnormal.

[0060] The user 6 can use the information terminal 5 to select any part of the seek bars SB1 to SB7.

[0061] The arithmetic circuit 45 outputs a display screen for displaying the first video data D4 corresponding to the portion selected by the user 6 (S6). For example, the arithmetic circuit 45 transmits to the information terminal 5 data necessary for displaying the selection screen on the information terminal 5 in response to an input by the user 6 to the information terminal 5.

[0062] 10 to 15 are explanatory diagrams showing examples of the display screen.

[0063] FIG. 10 is an explanatory diagram of a first example of a display screen. The display screen G21 shown in FIG. 10 includes a playback window W1 for playing the first video data D4 and buttons L1 and L2 for playing the second video data D5. For example, the display screen G21 is displayed upon selection of section P11 in FIG. 9. The playback window W1 plays the first video data D1 corresponding to section P11. The buttons L1 and L2 are links to the second video data D5 associated with the first video data D1 corresponding to section P11, and realize the function of playing the second video data D5. Section P11 corresponds, for example, to the section from time t1 to time t2 in FIG. 8. Between times t1 and t2, the state of the worker 121 is determined to be that the worker 121 has stopped working, as in the first example of the image in FIG. 5. The second video data D5 played by button L1 is "(4) video data of upstream work performed in the same time period as the first video data," and the second video data D5 played by button L2 is "(5) video data of downstream work performed in the same time period as the first video data." Furthermore, because it is determined that the number of upstream work-in-progress 122 is insufficient, button L1 is displayed more emphasized than button L2. Specifically, the images representing buttons L1 and L2 are the same, but the size of the image representing button L1 is larger than the size of the image representing button L2. This allows user 6 to easily navigate from button L1 to second video data D5 that may be related to the problem in the first video data D1 in order to find a solution to the problem without having to first review the contents of video data D1. This enables more efficient analysis of video data D1.

[0064] FIG. 11 is an explanatory diagram of a second example of a display screen. The display screen G22 shown in FIG. 11 includes a playback window W1 for playing the first video data D4 and buttons L1 and L2 for playing the second video data D5. For example, the display screen G22 is displayed upon selection of section P12 in FIG. 9. The playback window W1 plays the first video data D1 corresponding to section P12. The buttons L1 and L2 are links to the second video data D5 associated with the first video data D1 corresponding to section P12, and realize the function of playing the second video data D5. Section P12 corresponds, for example, to the section from time t3 to time t4 in FIG. 8. Between times t3 and t4, the state of the worker 121 is determined to be that the worker 121 has stopped working, as in the second example of the image in FIG. 6. The second video data D5 played by button L1 is "(4) video data of upstream work performed in the same time period as the first video data," and the second video data D5 played by button L2 is "(5) video data of downstream work performed in the same time period as the first video data." Furthermore, because it is determined that the number of downstream work-in-progress items 122 is excessive, button L2 is displayed more emphasized than button L1. Specifically, the images representing buttons L1 and L2 are the same, but the size of the image representing button L2 is larger than the size of the image representing button L1. This allows user 6 to easily navigate from button L2 to second video data D5 that may be related to the problem in the first video data D1 in order to find a solution to the problem without having to first review the contents of video data D1. This enables more efficient analysis of video data D1.

[0065] FIG. 12 is an explanatory diagram of a third example of a display screen. The display screen G23 shown in FIG. 12 includes a playback window W1 for playing the first video data D4 and a button L3 for playing the second video data D5. For example, the display screen G23 is displayed by selecting section P13 in FIG. 9. The playback window W1 plays the first video data D1 corresponding to section P13. The button L3 is a link to the second video data D5 associated with the first video data D1 corresponding to section P13, and realizes the function of playing the second video data D5. Section P13 corresponds, for example, to the section from time t5 to time t6 in FIG. 8. From time t5 to time t6, the state of the worker 121 is determined to be absent, as in the third example image in FIG. 7. The second video data D5 played by the button L3 is "(1) video data showing an object that is no longer shown in the first video data." This allows the user 6 to easily access the second video data D5, which may be related to the problem, from the button L3 in order to solve the problem in the first video data D1, without having to first check the contents of the video data D1. This makes it possible to analyze the video data D1 more efficiently.

[0066] FIG. 13 is an explanatory diagram of a fourth example of a display screen. The display screen G24 shown in FIG. 13 includes a playback window W1 for playing the first video data D4 and a button window L4 for multiple second video data D5. For example, the display screen G24 is displayed by selecting section P11 in FIG. 9 . The playback window W1 plays the first video data D1 corresponding to section P11. The button window L4 includes multiple buttons L41, L42, L43, and L44 for playing multiple second video data D5 associated with the first video data D1 corresponding to section P11. The buttons L41, L42, L43, and L44 correspond to "(4) video data of an upstream task in the same time slot as the first video data," "(5) video data of a downstream task in the same time slot as the first video data," "(2) video data in a time slot before the same task as the first video data," and "(3) video data in a time slot after the same task as the first video data," respectively. For example, section P11 corresponds to the section from time t1 to time t2 in FIG. 8 . During the section from time t1 to time t2, as in the first example image of FIG. 5 , the status of the worker 121 is determined to be suspended, and the number of upstream work-in-progress items 122 is determined to be insufficient. Therefore, button L41 is displayed more emphasized than the other buttons L42 to L44. Specifically, the color (e.g., red) of a portion of the image representing button L41 is different from the color (e.g., blue) of the other buttons L42 to L44. This allows user 6 to easily navigate from button L41 to second video data D5, which may be related to the problem, in order to find a solution to the problem in first video data D1, without having to first review the contents of video data D1. This allows for more efficient analysis of video data D1.

[0067] FIG. 14 is an explanatory diagram of a fifth example of a display screen. The display screen G25 shown in FIG. 14 includes a playback window W1 for playing the first video data D4 and a button window L4 for the second video data D5. For example, the display screen G25 is displayed by selecting section P12 in FIG. 9 . The playback window W1 plays the first video data D1 corresponding to section P12. The button window L4 includes multiple buttons L41, L42, L43, and L44 for playing multiple pieces of second video data D5 associated with the first video data D1 corresponding to section P12. The buttons L41, L42, L43, and L44 correspond to "(4) Video data of an upstream task in the same time slot as the first video data," "(5) Video data of an downstream task in the same time slot as the first video data," "(2) Video data in a time slot before the same task as the first video data," and "(3) Video data in a time slot after the same task as the first video data," respectively. For example, section P12 corresponds to the section from time t3 to time t4 in FIG. 8 . During the section from time t3 to time t4, as in the second example image of FIG. 6 , the status of the worker 121 is determined to be suspended, and the number of downstream work-in-progress items 122 is determined to be excessive. Therefore, button L42 is displayed more prominently than the other buttons L41, L43, and L44. Specifically, the color (e.g., red) of a portion of the image representing button L42 is different from the color (e.g., blue) of the other buttons L41, L43, and L44. This allows user 6 to easily navigate from button L42 to second video data D5, which may be related to the problem, in order to find a solution to the problem in the first video data D1, without first reviewing the contents of video data D1. This allows for more efficient analysis of video data D1.

[0068] FIG. 15 is an explanatory diagram of a sixth example of a display screen. The display screen G26 shown in FIG. 15 includes a playback window W1 for playing back the first video data D4, a button L3, and a button window L4. For example, the display screen G26 is displayed upon selection of section P13 in FIG. 9 . The playback window W1 plays back the first video data D1 corresponding to section P13. The button L3 corresponds to "(1) video data showing an object that is no longer captured in the first video data." The button window L4 includes multiple buttons L41, L42, L43, and L44 for playing back multiple pieces of second video data D5 associated with the first video data D1 corresponding to section P12. Buttons L41, L42, L43, and L44 correspond to "(4) video data of an upstream task within the same time period as the first video data," "(5) video data of a downstream task within the same time period as the first video data," "(2) video data of a time period before the same task as the first video data," and "(3) video data of a time period after the same task as the first video data," respectively. Section P13 corresponds, for example, to the section from time t5 to time t6 in FIG. 8 . During the section from time t5 to time t6, as in the third example image in FIG. 7 , the status of the worker 121 is determined to be "absent." Therefore, button L3 is displayed more emphasized than the other buttons L41 to L44. Specifically, the color of a portion of the image representing button L3 (e.g., red) is different from the color (e.g., blue) of the other buttons L41 to L44. This allows the user 6 to easily access the second video data D5 that may be related to the problem by clicking the button L3 in order to find a solution to the problem in the first video data D1, without having to first check the contents of the video data D1. This makes it possible to analyze the video data D1 more efficiently.

[0069] The user 6 can use the information terminal 5 to select any button (for example, buttons L1, L2, L3, L41 to L44).

[0070] The arithmetic circuit 45 plays back the second video data D5 corresponding to the button selected by the user 6 (S7). For example, the arithmetic circuit 45 transmits the second video data D5 to the information terminal 5 in response to an input from the user 6 to the information terminal 5.

[0071] Thus, according to the video data processing system 1, when a user 6 selects first video data D4 from video data D1, the first video data D4 is played and a button for playing second video data D5 related to the first video data D4 is displayed.

[0072] Traditionally, efforts to improve production efficiency in assembly processes at manufacturing sites have involved capturing footage of the site with cameras, analyzing the captured video to identify problems, and then formulating and implementing measures to address those issues. However, in large-scale manufacturing sites that operate for long periods of time, the number of cameras required for capture is large, resulting in a huge volume of captured video data, making it difficult to manually analyze and review all of the video. For example, in manufacturing facilities with multiple cells, cameras are installed in each cell, but rather than reviewing all of the captured video data, it is being considered to shorten analysis time by, for example, referring to actual production data from each cell to estimate which cell and at what time period there is likely to be a problem, and then analyzing that portion of the video data.

[0073] When observing video footage of a process to find problematic processes or problematic phenomena, it is often not possible to find the problem by simply observing one video, and it is often necessary to analyze multiple videos by comparing them. For example, to find clues for process improvement, i.e., to find a video of a bottleneck process, the worker observing the video will determine from the content of the observed video whether the problem is likely to be in the process immediately preceding it, or in the process immediately following it, whether the cause is likely to be in that process, or whether the video itself provides any clues, and in some cases will select a video based on the results of this observation. However, there are problems in that it takes time to find problematic videos, and since observation and analysis skills are required, the number of people who can perform the analysis is limited. One method for finding bottlenecks in processes is to observe videos, but this requires repeatedly switching between playing and analyzing multiple videos of the process, which can take a long time to analyze.

[0074] In this way, the video data processing system 1 according to the present embodiment, while playing video data D1 captured on camera of a workspace 100 such as a manufacturing facility, analyzes the status of the object (e.g., worker 121) and process recorded in the video data (first video data D4), determines where in the process a problem is likely to exist, and displays play buttons (buttons L1, L2, L3, L41-L44) for candidate video data (second video data D5) that may lead to solving the problem on the screen of the video being played (playback window W1). Therefore, by observing a certain video and automatically displaying candidate videos to watch next on the screen, the analyst's time and effort in searching for videos can be reduced. This enables more efficient analysis of video data D1.

[0075] [1.2 Effects, etc.] The video data processing system 1 described above includes an arithmetic circuit 45 that can access one or more video data D1 depicting one or more tasks involving one or more objects. The arithmetic circuit 45 is configured to detect the status of one or more objects (workers 121, work-in-progress 122) in the one or more video data D1, determine second video data D5 from the one or more video data D1 based on the status of the one or more objects in first video data D4, which is at least a portion of the one or more video data D1, output display screens G21-G26 including a playback window W1 for playing the first video data D4 and buttons L1, L2, L3, L41-L44 for playing the second video data D5, and execute playback of the second video data D5 in response to a selection of buttons L1, L2, L3, L41-L44. This configuration enables efficient analysis of the video data D1.

[0076] In the above-described video data processing system 1, the one or more objects include at least one of a worker 121 or a work-in-progress 122 that is a product manufactured by the work. This configuration enables efficient analysis of the video data D1.

[0077] In the video data processing system 1 described above, the second video data D5 includes at least one of video data showing an object that is no longer shown in the first video data D4, video data from a time slot before the same work as the first video data D4, video data from a time slot after the same work as the first video data D4, video data of work upstream in the same time slot as the first video data D4, or video data of work downstream in the same time slot as the first video data D4. This configuration enables efficient analysis of the video data D1.

[0078] In the above-described video data processing system 1, when the state of the object in the first video data D4 is abnormal, the second video data D5 is video data related to the abnormal state of the object in the first video data D4. This configuration enables efficient analysis of the video data D1.

[0079] In the video data processing system 1 described above, the one or more objects include a worker 121, and the states of the one or more objects include the state of the worker 121. This configuration enables efficient analysis of the video data D1 related to the work of the worker 121.

[0080] In the video data processing system 1 described above, the state of the worker 121 includes a stoppage of the worker 121's work, and the second video data D5 includes at least one of video data of work upstream in the same time period as the first video data D4 or video data of work downstream in the same time period as the first video data D4. This configuration enables efficient analysis of the video data D1 related to the work of the worker 121.

[0081] In the video data processing system 1 described above, the status of the worker 121 includes the absence of the worker 121, and the second video data D5 includes at least one of video data in which the worker who is no longer present in the first video data D4 is present, video data of a time period before the same work as the first video data D4, or video data of a time period after the same work as the first video data D4. This configuration enables efficient analysis of the video data D1 related to the work of the worker 121.

[0082] In the video data processing system 1 described above, the one or more objects include upstream work-in-progress 122 or downstream work-in-progress 122 in the operation, and the status of the one or more objects includes the number of upstream work-in-progress 122 or the number of downstream work-in-progress 122. This configuration enables efficient analysis of the video data D1 related to the work-in-progress 122.

[0083] In the video data processing system 1 described above, the state of the one or more objects includes a state in which the number of upstream work-in-progress 122 is equal to or less than a predetermined number, and the second video data D5 includes video data of upstream work in the same time period as the first video data D4. This configuration enables efficient analysis of the video data D1 related to the work-in-progress 122.

[0084] In the video data processing system 1 described above, the state of one or more objects includes a state in which the number of downstream work-in-progress 122 is equal to or greater than a predetermined number, and the second video data D5 includes video data of downstream work in the same time period as the first video data D4. This configuration enables efficient analysis of the video data D1 related to the work-in-progress 122.

[0085] In the above-described video data processing system 1, the buttons L1, L2, L3, L41 to L44 are displayed overlapping the playback window W1. This configuration makes it possible to analyze the video data D1 more efficiently.

[0086] In the above-described video data processing system 1, the display screens G21, G22, G24 to G26 each include a plurality of buttons L1, L2, L3, L41 to L44, and whether or not to highlight the plurality of buttons L1, L2, L3, L41 to L44 is determined based on the state of one or more objects in the first video data D4. This configuration enables efficient analysis of the video data D1.

[0087] In the above-described video data processing system 1, the arithmetic circuit 45 detects the state of one or more objects based on at least one of image processing technology and sensor data D2 relating to the one or more objects. This configuration enables improved accuracy in determining the state of the objects.

[0088] The above-described video data processing system 1 executes the following video data processing method using the arithmetic circuit 45. The video data processing method detects the states of one or more objects (workers 121, work-in-progress 122) in one or more video data D1 that show one or more tasks involving the one or more objects (workers 121, work-in-progress 122). The method determines second video data D5 from one or more video data D11 based on the states of the one or more objects (workers 121, work-in-progress 122) in first video data D4, which is at least a part of the one or more video data D1. The system outputs display screens G21-G26 that include a playback window W1 for playing the first video data D4 and buttons L1, L2, L3, L41-L44 for playing the second video data D5. The system then plays the second video data D5 in response to a selection of buttons L1, L2, L3, L41-L44. This configuration enables efficient analysis of video data.

[0089] The above-described video data processing method is realized by the execution of a program (computer program) by the arithmetic circuitry 45. This computer program is a program for causing the arithmetic circuitry 45 to execute the above-described video data processing method. This configuration enables efficient analysis of video data.

[0090] [2. Modifications] The embodiments of the present disclosure are not limited to the above-described embodiments. The above-described embodiments can be modified in various ways depending on the design, etc., as long as the object of the present disclosure can be achieved. Modifications of the above-described embodiments are listed below. The modifications described below can be applied in appropriate combinations.

[0091] In one variation, the workspace 100 is not limited to a facility. The workspace 100 may be a certain area, and may include, for example, a logistics site or a construction site. At a logistics site, for example, the worker may be a truck driver, and the camera system 2 can acquire video data D1 of the driver using a camera 21 installed on the truck.

[0092] In one modification, there are no particular limitations on the number of cameras 21 and 22 included in the camera system 2, their installation locations, etc. The camera system 2 only needs to be set so as to be able to acquire the desired video data D1.

[0093] In one modified example, there are no particular limitations on the number of sensors 31 included in the sensor system 3, installation locations, etc. The sensor system 3 may be set so as to be able to acquire desired sensor data D2.

[0094] In one variation, the second video data D5 is not limited to "(1) video data showing an object that is no longer shown in the first video data," "(2) video data taken in a time period before the same work as the first video data," "(3) video data taken in a time period after the same work as the first video data," "(4) video data of work upstream in the same time period as the first video data," or "(5) video data of work downstream in the same time period as the first video data." Furthermore, if the state of the object in the first video data D4 is abnormal, the second video data D5 is preferably, but is not particularly limited to, video data related to the abnormal state of the object in the first video data D4. The second video data D5 may be any video data that is preferably referenced in relation to the first video data D4.

[0095] In one variant, the computation circuitry 45 may have access to a trained model and may use the trained model to detect the state of the object. For example, the trained model may include one or more trained parameters generated by machine learning using a training dataset that inputs a group of images constituting the video data and outputs the state of one or more objects, and an inference program incorporating the one or more trained parameters. The trained model may also be a generative AI such as a large-scale language model. The computation circuitry 45 may input a group of images constituting the video data D1 to the trained model and obtain the state of one or more objects from the trained model. This allows for improved accuracy in determining the state of the object.

[0096] In this disclosure, terms related to machine learning are defined as follows:

[0097] A "trained model" refers to an "inference program" that incorporates "trained parameters."

[0098] "Trained parameters" refer to parameters (coefficients) obtained as a result of learning using a training dataset. Trained parameters are generated by inputting the training dataset into a training program and mechanically adjusting them for a specific purpose. Although trained parameters are adjusted to suit the purpose of learning, they are simply parameters (numerical information, etc.) on their own, and only function as a trained model when incorporated into an inference program. For example, in the case of deep learning, the main trained parameters are parameters used to weight the links between each node.

[0099] An "inference program" is a program that can output a certain result for an input by applying built-in trained parameters. For example, it is a program that specifies a series of calculation procedures for applying trained parameters acquired as a result of training to an image given as input and outputting a result (authentication or judgment) for that image.

[0100] A "learning dataset," also known as a training dataset, refers to secondary processed data that is generated to facilitate analysis using the target learning method by converting and processing raw data through preprocessing such as removing missing values ​​and outliers, adding separate data such as label information (ground truth data), or a combination of these. A learning dataset may also include data that has been "padded" by applying certain transformations to the raw data.

[0101] "Raw data" refers to data that is primarily acquired by users, vendors, other businesses, research institutions, etc., and that has been converted and processed so that it can be loaded into a database.

[0102] A "learning program" is a program that executes an algorithm to find certain rules from a training dataset and generate a model that expresses those rules. Specifically, this refers to a program that specifies the procedures to be executed by a computer in order to realize learning using the adopted learning method.

[0103] In one modified example, the video data processing system 1 does not necessarily have to include the camera system 2. That is, the video data D1 may be prepared in advance. Similarly, the video data processing system 1 does not necessarily have to include the sensor system 3. That is, the sensor data D2 may be prepared in advance.

[0104] In one modified example, the processing device 4 and the information terminal 5 of the video data processing system 1 may be configured as a single computer system.

[0105] In one variation, the processing device 4 of the video data processing system 1 may be realized by a computer system such as a plurality of servers. It is not necessary for the plurality of functions (components) of the processing device 4 to be concentrated in a single housing, and the components of the processing device 4 may be distributed across a plurality of housings. Furthermore, at least some of the functions of the processing device 4, for example, some of the functions of the arithmetic circuit 45, may be realized by the cloud (cloud computing) or the like.

[0106] [3. Aspects] As is clear from the above-described embodiment and modifications, the present disclosure includes the following aspects.

[0107] [Aspect 1] A video data processing system comprising an arithmetic circuit capable of accessing one or more video data depicting one or more tasks involving one or more objects, wherein the arithmetic circuit is configured to: detect the states of the one or more objects in the one or more video data; determine second video data from the one or more video data based on the states of the one or more objects in first video data that is at least a part of the one or more video data; output a display screen including a playback window for playing the first video data and a button for playing the second video data; and execute playback of the second video data in response to selection of the button.

[0108] [Aspect 2] The video data processing system of aspect 1, wherein the one or more objects include at least one of a worker, a work-in-progress product produced by the work, or a workbench.

[0109] [Aspect 3] The video data processing system of Aspect 1 or 2, wherein the second video data includes at least one of: video data showing an object that is no longer shown in the first video data; video data from a time period before the same work as the first video data; video data from a time period after the same work as the first video data; video data of work upstream in the same time period as the first video data; or video data of work downstream in the same time period as the first video data.

[0110] [Aspect 4] The video data processing system according to any one of Aspects 1 to 3, wherein, when the state of the object in the first video data is an abnormal state, the second video data is video data related to the abnormal state of the object in the first video data.

[0111] [Aspect 5] The video data processing system of Aspect 4, wherein the one or more objects include a worker, and the states of the one or more objects include a state of the worker.

[0112] [Aspect 6] The video data processing system of Aspect 5, wherein the worker status includes a stoppage of the worker's work, and the second video data includes at least one of video data of an upstream work in the same time period as the first video data, or video data of a downstream work in the same time period as the first video data.

[0113] [Aspect 7] The video data processing system of Aspect 5 or 6, wherein the status of the worker includes the absence of the worker, and the second video data includes at least one of video data in which the worker appears, video data taken in a time period before the same work as the first video data, or video data taken in a time period after the same work as the first video data.

[0114] [Aspect 8] The video data processing system of any one of Aspects 4 to 7, wherein the one or more objects include work-in-progress on an upstream side or work-in-progress on a downstream side in the operation, and the status of the one or more objects includes the number of work-in-progress on the upstream side or the number of work-in-progress on the downstream side.

[0115] [Aspect 9] The video data processing system of Aspect 8, wherein the state of the one or more objects includes a state in which the number of work-in-progress items on the upstream side is equal to or less than a predetermined number, and the second video data includes video data of upstream work in the same time period as the first video data.

[0116] [Aspect 10] The video data processing system of Aspect 8 or 9, wherein the state of the one or more objects includes a state in which the number of work-in-progress items on the downstream side is equal to or greater than a predetermined number, and the second video data includes video data of downstream work occurring in the same time period as the first video data.

[0117] [Aspect 11] The video data processing system according to any one of Aspects 1 to 10, wherein the button is displayed superimposed on the playback window.

[0118] [Aspect 12] The video data processing system according to any one of Aspects 1 to 11, wherein the display screen includes a plurality of the buttons, and whether to highlight the plurality of buttons is determined based on a state of the one or more objects in the first video data.

[0119] [Aspect 13] The video data processing system according to any one of Aspects 1 to 12, wherein the arithmetic circuit detects a state of the one or more objects based on at least one of an image processing technique and sensor data relating to the one or more objects.

[0120] [Aspect 14] A video data processing method, executed by an arithmetic circuit, detecting states of one or more objects in first video data, which is part of one or more video data showing one or more tasks involving the one or more objects; determining second video data, which is another part of the video data, based on the states of the one or more objects in the first video data; outputting a display screen including a playback window for playing the first video data and a button for playing the second video data; and executing playback of the second video data in response to selection of the button.

[0121] [Aspect 15] A program for causing the arithmetic circuit to execute the video data processing method of aspect 14.

[0122] Aspects 2 to 13 are optional elements and are not essential. Aspects 2 to 13 can be appropriately combined with Aspect 14.

[0123] The present disclosure is applicable to a data processing system, a data processing method, and a program. Specifically, the present disclosure is applicable to a video data processing system, a video data processing method, and a program (program product) for analyzing video data.

[0124] 1 Video data processing system 45 Arithmetic circuit D1 Video data D2 Sensor data D3 First video data D4 Second video data G21 to G26 Display screen W1 Playback window L1, L2, L3, L41 to L44 Buttons 121 Worker (object) 122 Work in progress (object)

Claims

1. A video data processing system comprising an arithmetic circuit capable of accessing one or more video data depicting one or more operations involving one or more objects, the arithmetic circuit being configured to: detect the state of the one or more objects in first video data that is a part of the one or more video data; determine second video data that is another part of the one or more video data based on the state of the one or more objects in the first video data; output a display screen including a playback window for playing the first video data and a button for playing the second video data; and execute playback of the second video data in response to selection of the button.

2. The video data processing system according to claim 1, wherein the one or more objects include at least one of a worker, a work-in-progress product produced in the operation, or a workbench.

3. The video data processing system of claim 1, wherein the second video data includes at least one of: video data showing an object that is no longer shown in the first video data; video data from a time period before the same work as the first video data; video data from a time period after the same work as the first video data; video data of work upstream in the same time period as the first video data; or video data of work downstream in the same time period as the first video data.

4. The video data processing system of claim 1, wherein, when the state of the object in the first video data is an abnormal state, the second video data is video data related to the abnormal state of the object in the first video data.

5. The video data processing system according to claim 1, wherein the one or more objects include a worker, and the states of the one or more objects include a state of the worker.

6. The video data processing system of claim 5, wherein the worker's status includes a stoppage of the worker's work, and the second video data includes at least one of video data of upstream work occurring in the same time period as the first video data, or video data of downstream work occurring in the same time period as the first video data.

7. The video data processing system of claim 5, wherein the worker's status includes the absence of the worker, and the second video data includes at least one of video data in which the worker is present, video data from a time period before the same work as the first video data, or video data from a time period after the same work as the first video data.

8. The video data processing system of claim 1, wherein the one or more objects include upstream work-in-progress or downstream work-in-progress in the operation, and the status of the one or more objects includes the number of upstream work-in-progress or downstream work-in-progress.

9. The video data processing system of claim 8, wherein the state of the one or more objects includes a state in which the number of work-in-progress items on the upstream side is less than a predetermined number, and the second video data includes video data of upstream work in the same time period as the first video data.

10. The video data processing system of claim 8, wherein the state of the one or more objects includes a state in which the number of work-in-progress items on the downstream side is equal to or greater than a predetermined number, and the second video data includes video data of downstream work occurring in the same time period as the first video data.

11. The video data processing system according to claim 1, wherein the button is displayed superimposed on the playback window.

12. The video data processing system of claim 1, wherein the display screen includes a plurality of the buttons, and whether or not to highlight the plurality of buttons is determined based on the state of the one or more objects in the first video data.

13. The video data processing system according to claim 1, wherein the arithmetic circuit detects the state of the one or more objects based on at least one of image processing technology and sensor data relating to the one or more objects.

14. A video data processing method, executed by an arithmetic circuit, which detects the state of one or more objects in first video data, which is part of one or more video data showing one or more tasks involving the one or more objects; determines second video data, which is another part of the video data, based on the state of the one or more objects in the first video data; outputs a display screen including a playback window for playing the first video data and a button for playing the second video data; and executes playback of the second video data in response to selection of the button.

15. A program for causing the arithmetic circuit to execute the video data processing method of claim 14.

Images