Information processing apparatus, machine tool, information processing method, program

Abstract

To provide a technology capable of more efficiently confirming a situation of a construction site in a posterior manner by utilizing a moving image obtained by imaging a situation of the construction site.SOLUTION: According to an embodiment of this disclosure, an information processing device 40 includes an image feature amount extraction unit 403 for extracting an image feature amount in each still picture in a moving image acquired by an imaging device 10 for imaging a situation of a construction site, and a summary moving image generation unit 406 for extracting a scene representing relatively large change in an original moving image from an analysis result of an original moving image based on the image feature amount to generate a summary moving image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and the like.

Background Art

[0002] Conventionally, a technique for monitoring the state of a construction site using an imaging device has been known (see Patent Documents 1 and 2).

[0003] In Patent Document 1, an imaging device is mounted on a work machine (excavator), and the state around the work machine at the construction site is monitored.

[0004] In Patent Document 2, the state of the construction site is monitored by an imaging device (surveillance camera) installed at the construction site.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, it is conceivable that a video capturing the state of a construction site is recorded, and afterwards, an administrator or the like uses the video to check the situation of the construction site.

[0007] However, it is not realistic for an administrator or the like to check all the contents of the video, as it consumes a great deal of labor and time.

[0008] Therefore, in view of the above problems, an object is to provide a technique that can more efficiently check the situation of a construction site afterwards using a video capturing the state of the construction site.

Means for Solving the Problems

[0009] In order to achieve the above object, in one embodiment of the present disclosure, a video acquired by an imaging device that images the state of a construction site An image feature quantity extraction unit that extracts image feature quantities for each still image in; A scene division unit that divides the video into a plurality of scenes each composed of a series of still images having relatively close image feature quantities based on the image feature quantities for each still image; , The plurality of scenes among from extract a scene representing a relatively large change and, if there is a scene that continues for a relatively long period among the plurality of scenes, extract a part of the scene regardless of whether it is a scene representing a relatively large change; to generate a relatively short summary video A summary video generation unit, and includes; , an information processing apparatus is provided.

[0010] Also, in another embodiment of the present disclosure, an imaging device that images the state around a work machine, and a video acquired by the imaging device An image feature quantity extraction unit that extracts image feature quantities for each still image in; A scene division unit that divides the video into a plurality of scenes each composed of a series of still images having relatively close image feature quantities based on the image feature quantities for each still image; , The plurality of scenes among from extract a scene representing a relatively large change and, if there is a scene that continues for a relatively long period among the plurality of scenes, extract a part of the scene regardless of whether it is a scene representing a relatively large change; to generate a relatively short summary video A summary video generation unit, and has; an information processing apparatus, and a work machine is provided. a work machine is provided.

[0011] Also, in still another embodiment of the present disclosure, an information processing method executed by an information processing apparatus, comprising: a video acquired by an imaging device that images the state of a construction site An image feature quantity extraction step of extracting image feature quantities for each still image in; A scene division step of dividing the video into a plurality of scenes each composed of a series of still images having relatively close image feature quantities based on the image feature quantities for each still image; , The plurality of scenes among from extract a scene representing a relatively large change and, if there is a scene that continues for a relatively long period among the plurality of scenes, extract a part of the scene regardless of whether it is a scene representing a relatively large change; to generate a relatively short summary video A summary video generation step, and includes; , an information processing method is provided.

[0012] Also, in still another embodiment of the present disclosure, to the information processing apparatus, a video acquired by an imaging device that images the state of a construction site An image feature quantity extraction step of extracting image feature quantities for each still image in; A scene division step of dividing the video into a plurality of scenes each composed of a series of still images having relatively close image feature quantities based on the image feature quantities for each still image; and The plurality of scenes extract scenes representing relatively large changes from from among them and, if there is a scene that continues for a relatively long period among the plurality of scenes, extract a part of the scene regardless of whether it is a scene representing a relatively large change; to generate a relatively short summary video Execute a summary video generation step of doing; and cause a program to be provided. [Advantages of the Invention]

[0013] According to the above-described embodiment, the state of the construction site can be more efficiently confirmed retrospectively by using the video obtained by imaging the state of the construction site. [Brief Description of the Drawings]

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0015] Hereinafter, embodiments will be described with reference to the drawings.

[0016] [Outline of Work Log Generation System] With reference to FIG. 1, the outline of the work log generation system 1 according to the present embodiment will be described.

[0017] As shown in FIG. 1, the work log generation system 1 includes an imaging device 10, a control device 20, a storage device 30, an information processing device 40, and a management device 50.

[0018] In the information processing device 40 of the work log generation system 1, the video data of the construction site CS acquired by the imaging device 10 is edited, and a summary version of the video (hereinafter, "summary video") that is relatively shorter than the original video is generated as the work log of the construction site CS. As a result, the administrator of the construction site CS can retrospectively confirm the summary video as the work log through the information processing device 40 (display device 48 described later). In addition, an administrator in charge of managing a plurality of construction sites CS can confirm the summary video uploaded from the information processing device 40 to the management device 50 through the management device 50 (display device).

[0019] The number of imaging devices 10 included in the work log generation system 1 may be one or a plurality. Further, the work log generation system 1 may include imaging devices 10 for each of a plurality of construction sites CS. That is, the work log generation system 1 may include one or a plurality of imaging devices 10 provided for each of a plurality of construction sites CS. Also, when the work log generation system 1 includes imaging devices 10 for each of a plurality of construction sites, the management device 50 may be one or a plurality. For example, a plurality of management devices 50 may each be in charge of a non-overlapping part of a plurality of construction sites CS among the plurality of construction sites CS, and acquire work logs (summary videos) from the information processing devices 40 of that part of the construction sites CS.

[0020] The imaging device 10 is provided at the construction site CS and images the state of the construction site CS.

[0021] The imaging device 10 includes, for example, a surveillance camera installed at a construction site CS. Further, the imaging device 10 may include a camera mounted on a drone for acquiring the construction status of the construction site CS, the topographic shape before the start of construction, and the like. Further, the imaging device 10 may be a camera or a camera-equipped mobile terminal (for example, a smartphone or a tablet terminal) carried by an administrator or the like of the construction site CS. Further, the imaging device 10 may include a camera mounted on a work machine or a work vehicle (hereinafter, "work vehicle, etc.") that performs work at the construction site CS. Work machines include, for example, excavators, bulldozers, wheel loaders, mobile cranes, and the like. Work vehicles include, for example, dump trucks and the like. The camera mounted on the work machine, etc. may be a dedicated camera of the work log generation system 1, or may be a camera for monitoring the situation around the work machine, etc. and notifying the operator of information regarding the situation around the work machine, etc.

[0022] The control device 20 controls the imaging device 10.

[0023] For example, when the imaging device 10 is a surveillance camera installed at the construction site CS, the control device 20 is installed integrally with the surveillance camera or at a location relatively close to the surveillance camera. Further, for example, when the imaging device 10 is a camera mounted on a drone, the control device 20 is mounted on that drone. Further, when the imaging device 10 is a camera or a camera-equipped mobile terminal carried by an administrator or the like of the construction site CS, for example, the control device 20 is built into the housing of the camera or the mobile terminal. Further, when the imaging device 10 is mounted on a work machine or the like, for example, the control device 20 is mounted on that work machine or the like. When the control device 20 is mounted on a work machine or the like, it may be a dedicated control device for the imaging device 10, or may be a control device also used for controlling other devices mounted on the work machine or the like.

[0024] In the memory device 30, video data captured by the imaging device 10 is stored under the control of the control device 20. The memory device 30 may be provided integrally with the control device 20 or in a relatively close location, or may be provided in a location relatively distant from the control device 20.

[0025] For example, when the imaging device 10 is a surveillance camera installed at the construction site CS, the memory device 30 may be provided integrally with the surveillance camera or the control device 20 or in a relatively close location, or may be provided in a location relatively distant from the surveillance camera, such as in the management office of the construction site CS. Also, for example, when the imaging device 10 is a camera mounted on a drone, the memory device 30 may be mounted on the drone or may be provided in a location relatively distant from the drone, such as in the management office of the construction site CS. In the latter case, the drone (control device 20) transmits the video data to the memory device 30 through a predetermined communication line for storage. The predetermined communication line is, for example, a wireless local area network (LAN) at the construction site CS. Also, the predetermined communication line may be a short-range communication line based on wireless communication standards such as WiFi or Bluetooth (registered trademark). Also, for example, when the imaging device 10 is a camera or a camera-equipped mobile terminal carried by the manager of the construction site CS, etc., the memory device 30 may be built into its housing or may be provided in a location relatively distant from the camera or camera-equipped mobile terminal, such as in the management office of the construction site CS. In the latter case, the camera or camera-equipped mobile terminal (control device 20) transmits the video data to the memory device 30 through a predetermined communication line, such as a wireless local area network or a short-range communication line, in the same manner as in the case of the above-mentioned drone for storage. Also, for example, when the imaging device 10 is mounted on a work machine, etc., the memory device 30 may be mounted on the work machine, etc., or may be provided in a location relatively distant from the work machine, etc., such as in the management office of the construction site CS.

[0026] Note that the function of the memory device 30 may be transferred to the internal memory (auxiliary storage device) of the control device 20 or to the internal memory (auxiliary storage device) of the information processing device 40.

[0027] The information processing device 40 analyzes the video data of the construction site CS stored in the storage device 30, and performs editing to extract scenes with relatively high importance from the video data, thereby creating a summary video that is relatively shorter than the original video. As a result, the manager of the construction site CS can view the summary video through the display device 48 described below.

[0028] In addition, the information processing device 40 transmits the summary video to the management device 50. As a result, the managers in charge of multiple construction sites CS can view the summary videos of their respective construction sites CS through the display device of the management device 50.

[0029] For example, the information processing device 40 acquires information regarding the video data recorded in the storage device 30 from the control device 20, and creates a summary video for the unedited video data at a predetermined timing (for example, at night after the end of a day's work at the construction site CS). Also, for example, if the information processing device 40 determines that the user needs to view the summary video at a relatively early timing, it may create the summary video at a timing earlier than the predetermined timing. Specifically, the information processing device 40 may determine that the user needs to view the summary video at a relatively early timing when a predetermined control command is received from the management device 50 through the communication interface 46 described below. Also, the information processing device 40 may determine that the user needs to view the summary video at a relatively early timing when a predetermined input from the user is received through the input device 47 described below. In particular, when the information processing device 40 is mounted on a work machine or the like, the operator can trigger a predetermined input when an event or the like occurs where it is considered better for the manager to view it earlier, so that the summary video can be transmitted to the management device 50 at an earlier timing.

[0030] For example, in the case where the imaging device 10 is a surveillance camera installed at the construction site CS, the information processing device 40 may be provided integrally with or relatively close to the surveillance camera or the control device 20, or may be provided at a location relatively far from the surveillance camera such as the management office of the construction site CS. Also, for example, when the imaging device 10 is a camera mounted on a drone, the information processing device 40 may be mounted on the drone together with the storage device 30, or may be provided at a location relatively far from the drone such as the management office of the construction site CS regardless of the arrangement of the storage device 30. Further, for example, when the imaging device 10 is a camera or a camera-equipped mobile terminal carried by the administrator of the construction site CS or the like, the information processing device 40 may be built into its housing, or may be provided at a location relatively far from the camera or the camera-equipped mobile terminal such as the management office of the construction site CS. Also, for example, when the imaging device 10 is mounted on a work machine or the like, the information processing device 40 may be mounted on the work machine or the like, or may be provided at a location relatively far from the work machine or the like such as the management office of the construction site CS. When set in the management office of the construction site CS or the like, the management device 50 may be, for example, an edge server or a terminal device for managing the construction site CS.

[0031] Note that part or all of the functions of the information processing device 40 may be integrated into the control device 20. Also, part or all of the functions of the information processing device 40 may be provided outside the construction site CS, and may be integrated into, for example, the management device 50.

[0032] The management device 50 is provided outside the construction site CS and performs processing related to the management and operation of the construction site CS.

[0033] For example, the management device 50 is an on-premises server or a cloud server. Also, for example, the management device 50 may be an edge server. Further, the management device 50 may be a terminal device used by a manager in charge of a plurality of construction sites CS. The terminal device may be, for example, a stationary terminal device such as a desktop computer terminal, or a portable terminal device (mobile terminal) such as a smartphone, a tablet terminal, or a laptop computer terminal.

[0034] [Configuration of Work Log Generation System] Next, in addition to FIG. 1, with reference to FIGS. 2 and 3, the configuration of the work log generation system 1 will be described centering on the configuration of the information processing device 40.

[0035] FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 40. FIG. 3 is a diagram showing an example of the functional configuration of the information processing device 40.

[0036] The control device 20 is realized by any hardware, or any combination of hardware and software, etc. For example, the control device 20 is mainly configured by a computer including a CPU (Central Processing Unit), a memory device such as a RAM (Random Access Memory), an auxiliary storage device such as a ROM (Read Only Memory), and an interface device for input / output with the outside.

[0037] The storage device 30 is configured by a non-volatile semiconductor memory, a magnetic disk, etc. For example, the storage device 30 is, for example, an HDD (Hard Disc Drive) or an SSD (Solid State Disc).

[0038] The functions of the information processing apparatus 40 are realized by any combination of arbitrary hardware, or arbitrary software, etc. For example, as shown in FIG. 2, the information processing apparatus 40 includes an external interface 41, an auxiliary storage device 42, a memory device 43, a CPU 44, a high-speed arithmetic unit 45, a communication interface 46, an input device 47, and a display device 48.

[0039] The external interface 41 functions as an interface for reading data from the recording medium 41A and writing data to the recording medium 41A. The recording medium 41A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray Disc) (registered trademark), an SD memory card, a USB (Universal Serial Bus) memory, and the like. Thereby, the information processing apparatus 40 can read various data used in processing through the recording medium 41A and store them in the auxiliary storage device 42, or install a program for realizing various functions.

[0040] Note that the information processing apparatus 40 may acquire various data and programs from an external device (for example, a management device 50).

[0041] The auxiliary storage device 42 stores various installed programs and also stores files, data, etc. necessary for various processes. The auxiliary storage device 42 includes, for example, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), an HDD, an SSD, and the like.

[0042] When an instruction to start a program is given, the memory device 43 reads the program from the auxiliary storage device 42 and stores it. The memory device 43 includes, for example, a DRAM (Dynamic Random Access Memory) and an SRAM (Static Random Access Memory).

[0043] The CPU 44 executes various programs loaded from the auxiliary storage device 42 to the memory device 43, and realizes various functions related to the information processing device 40 according to the programs.

[0044] The high-speed arithmetic unit 45 operates in conjunction with the processing of the CPU 44 and realizes high-speed arithmetic processing at a speed relatively higher than the processing speed of the CPU 44. The high-speed arithmetic unit 45 includes, for example, a GPU (Graphical Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0045] Note that the high-speed arithmetic unit 45 may be omitted.

[0046] The communication interface 46 is used as an interface for communicably connecting to an external device. Thereby, the information processing device 40 is communicably connected to external devices such as, for example, the control device 20, the storage device 30, and the management device 50 through the communication interface 46. Further, the communication interface 46 may have a plurality of types of communication interfaces depending on the communication method with the connected device, etc.

[0047] The input device 47 receives various instruction inputs related to the information processing device 40 from a user such as an operator or a manager of the information processing device 40. The input device 47 includes, for example, operation input devices such as a keyboard, a mouse, a touch panel, and a touch pad. Further, the input device 47 includes, for example, a voice input device such as a microphone that receives voice input from the user. Further, the input device 47 includes, for example, a gesture input device such as a camera that receives gesture input from the user. Further, the input device 47 includes a biometric input device such as a camera that receives biometric input (image information related to the user's fingerprint or iris) from the user.

[0048] The display device 48 displays a GUI (Graphical User Interface) according to a program executed by, for example, the CPU 44. The display device 48 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[0049] The management device 50 has its functions realized by any combination of hardware or software, etc. For example, similar to the information processing device 40, the management device 50 includes an external interface device, an auxiliary storage device, a memory device, a CPU, a high-speed arithmetic device, a communication interface device, an input device, and a display device. Also, similar to the case of the information processing device 40, the high-speed arithmetic device may be omitted.

[0050] As shown in FIG. 3, the information processing device 40 includes, as functional units, an image acquisition unit 401, a storage unit 402, an image feature amount extraction unit 403, a scene segmentation unit 404, a scene evaluation unit 405, a summary video generation unit 406, and a communication unit 407.

[0051] The functions of the image acquisition unit 401, the image feature amount extraction unit 403, the scene segmentation unit 404, the scene evaluation unit 405, the summary video generation unit 406, and the communication unit 407, etc. are realized, for example, by loading a program installed in the auxiliary storage device 42 into the memory device 43 and executing it on the CPU. Also, the functions of the storage unit 402, etc. are realized, for example, by a storage area defined in the auxiliary storage device 42.

[0052] The image acquisition unit 401 accesses the storage device 30 based on information regarding the video data of the imaging device 10 recorded in the storage device 30 acquired from the control device 20, and acquires the video data to be edited.

[0053] The storage unit 402 stores the video data to be edited acquired by the image acquisition unit 401.

[0054] The image feature extraction unit 403 analyzes the video data to be edited in the storage unit 402, and extracts the image features for each still image in the video data.

[0055] The scene segmentation unit 404 divides the image data into a plurality of scenes based on the image features for each series of still images extracted by the image feature extraction unit 403. For example, the scene segmentation unit 404 divides the video data into a plurality of scenes each composed of a series of still images having relatively close image features.

[0056] The scene evaluation unit 405 evaluates the importance representing the degree of necessity to be left as the work log of the construction site CS for each of the plurality of scenes.

[0057] The summary video generation unit 406 extracts some scenes to be left as work logs from among the plurality of scenes based on the evaluation results of the scene evaluation unit 405, that is, the importance for each of the plurality of scenes, and generates the data of the summary video by connecting the extracted scenes in time series. Some scenes included in the summary video may be the entire scene or a part of the scene respectively.

[0058] The summary video generation unit 406 stores the generated summary video data in the storage device 30. Then, the summary video generation unit 406 transmits the generated summary video to the management device 50 through the communication unit 407.

[0059] The communication unit 407 communicates with the management device 50.

[0060] For example, in response to a request from the summary video generation unit 406, the communication unit 407 transmits the summary video in the storage device 30 to the management device 50. Also, for example, the communication unit 407 may transmit the video in the storage device 30 in response to a request received from the management device 50. Further, for example, the communication unit 407 receives a signal requesting generation of a summary video from the management device 50. In this case, the image acquisition unit 401 may capture the video to be edited from the storage device 30 into the storage unit 402 in response to the request for generation of the summary video from the management device 50. Then, the image feature quantity extraction unit 403, the scene segmentation unit 404, the scene evaluation unit 405, and the summary video generation unit 406 may perform a series of processes for generating a summary video in response to the request for generation of the summary video from the management device 50, and generate a summary video.

[0061] [Details of the summary video generation process] Next, with reference to FIGS. 4 to 6, details of the process for generating a summary video by the information processing apparatus 40 will be described.

[0062] FIG. 4 is a diagram schematically showing the process of the image feature quantity extraction unit 403. FIG. 5 is a diagram schematically and conceptually explaining the process of the scene segmentation unit 404. FIG. 6 is a diagram schematically explaining the flow of the analysis process for generating a summary video.

[0063] [Process of the image feature quantity extraction unit] As shown in FIG. 4, the image feature quantity extraction unit 403 includes a convolutional neural network (CNN: Convolutional Neural Network) 403A. The CNN 403A is configured by connecting a plurality of combinations of a convolutional layer that performs a convolutional process and a pooling layer that performs a pooling process with an activation function. The activation function is, for example, ReLU (Rectified Linear Unit).

[0064] Note that the CNN 403A in this example does not include a fully connected layer.

[0065] In this example, a still image of one frame of the video data to be edited is composed of 800 pixels × 600 pixels and RGB (Red Green Red) 3 planes. The CNN403A can output 1024 image feature amounts for each input still image (see FIG. 6A).

[0066] In addition, the image feature extraction unit 403 may extract other types of image feature amounts by arbitrarily applying a known method other than the method using the CNN403A. Also, in this example, the information processing apparatus 40 extracts the image feature amounts for each still image using the CNN403A, but the CNN403A may be used for labeling each still image. In this case, the CNN403A may be connected with a fully connected layer at the end of a combination of a plurality of stages of convolutional layers and a pooling device, and labeling may be performed for each still image by applying machine learning in advance. A plurality of labels related to events at the construction site CS (for example, the excavation work of a shovel) to be included in the summary video or objects (for example, dump trucks for earth and sand transportation, etc.) to be included in the summary video are prepared for the labels prepared in advance. Thereby, the CNN403A can output the probability that the target still image corresponds to each of the plurality of labels.

[0067] <Processing of the scene segmentation unit> In this example, the scene segmentation unit 404 uses the KTS (Kernel Temporal Segmentation) method to divide the video data into a plurality of scenes.

[0068] As shown in FIGS. 5A and 5B, first, the scene segmentation unit 404 generates a feature matrix X in which the target image feature amounts for each frame (still image) of the video are arranged in each row in time series for a plurality of (for example, 1024 in the example of FIG. 4 above) image feature amounts. The number of rows of the feature matrix X corresponds to the number (type number) of image feature amounts, and the number of columns corresponds to the number of frames of the video to be edited.

[0069] In the examples of FIGS. 5A and 5B, a 2-row 16-column feature matrix X in which the image feature amounts A and B of 16-frame video data are arranged in each row in time series is generated.

[0070] Next, as shown in FIG. 5C, the scene division unit 404 calculates a transposed matrix X T and the product of feature matrix X (=X T ·X) to calculate a square matrix whose number of rows and columns corresponds to the number of frames of the video. The scene division unit 404 assumes that the diagonal direction of the square matrix resulting from the calculation is the axis of the time series, and assumes a number of frame division points SDP that divide the scene into a predetermined number of scenes. Then, as shown in Fig. 5D, the scene division unit 404 optimizes the frame division points SDP under predetermined constraints such that the total value of the off-diagonal elements of a number of small square matrices corresponding to a number of scenes, which are defined by adjacent frame division points SDP, is maximized.

[0071] This allows the scene division section 404 to divide the video data to be edited into a plurality of scenes, each of which is made up of a series of still images (frames) having relatively similar image features (see FIG. 6B).

[0072] The scene division unit 404 may divide the video data into a plurality of scenes by arbitrarily applying a known method other than the method of this example. For example, when labeling is performed for each still image using the CNN 403A with an additional fully connected layer, scene division points in a time series may be determined by determining relatively large changes in events occurring in the video or objects shown in the video based on the labeling results.

[0073] <Scene evaluation section processing> The scene evaluation unit 405 evaluates, for each of the multiple scenes divided by the scene division unit 404, the degree to which the scene represents a relatively large change (importance) in the moving image to be edited.

[0074] The scene evaluation unit 405 calculates the importance based on, for example, the difference between the average of the image feature amounts in a certain period before and after in time series for each still image (frame) of the video data and the target still image (frame). At this time, the scene evaluation unit 405 may use the difference as the importance, or may calculate the importance by further performing arithmetic processing on the difference.

[0075] Thereby, the scene evaluation unit 405 can calculate the importance for each frame (still image) in the time series of the video data (see the graph 601 in FIG. 6C).

[0076] Then, the scene evaluation unit 405 can calculate the average value of the importance of a series of frames (still images) for each of the plurality of scenes as the importance for each of the plurality of scenes.

[0077] In addition, the scene evaluation unit 405 may evaluate (calculate) the importance for each of the plurality of scenes by arbitrarily applying a known method other than the method in this example.

[0078] <Processing of the summary video generation unit> The summary video generation unit 406 extracts scenes to be included in the summary video from among the plurality of scenes based on the evaluation result of the scene evaluation unit 405, that is, the importance for each of the plurality of scenes.

[0079] For example, the summary video generation unit 406 extracts scenes to be included in the summary video from among the plurality of scenes based on the following condition (1).

[0080] (1) They are extracted in order from the scenes with high importance among the plurality of scenes.

[0081] Specifically, in addition to the condition (1) described above, the summary video generation unit 406 extracts scenes to be included in the summary video from among the plurality of scenes based on the following condition (2).

[0082] (2) From among a plurality of scenes, scenes are extracted such that the time of the summary video, the ratio with respect to the original video data of the summary video, or the number of scenes included in the summary video becomes equal to or less than a predetermined threshold value.

[0083] As a result, the summary video generation unit 406 can extract, as scenes to be included in the summary video, scenes with relatively high importance based on the conditions (1) and (2) described above (see the graph 602 in FIG. 6C).

[0084] Also, for example, in addition to the conditions (1) and (2) described above, the summary video generation unit 406 extracts scenes to be included in the summary video based on the following conditions (3) and (4).

[0085] (3) When there is a relatively long scene exceeding a predetermined time among a plurality of scenes, regardless of the importance, a part of the scene is extracted.

[0086] (4) Scenes are extracted such that the interval between adjacent scenes in the extracted time series is equal to or less than a predetermined number of frames or a predetermined time.

[0087] The summary video generation unit 406 may, for example, in addition to the scenes extracted in accordance with the condition (1) and the conditions regarding the time and ratio of (2), extract a part of the scene corresponding to the condition (3) within the range corresponding to the conditions regarding the time and ratio of (2). As a result, when there is a relatively long scene, the information processing apparatus 40 can include, as a work log, a part of the scene even if it is a scene with a relatively small change in the video, that is, a scene with a relatively small change in the image feature amount, in the summary video.

[0088] Also, the summary video generation unit 406 may additionally extract scenes so as to satisfy the condition (4) within the range corresponding to the conditions regarding the time and ratio of (2), in addition to the scenes extracted in accordance with the condition (1) and the conditions regarding the time and ratio of (2).

[0089] In this way, the summary video generation unit 406 can generate a summary video by connecting in time series the scenes extracted from the plurality of scenes divided from the video data to be edited by the scene division unit 404.

[0090] In addition, the summary video generation unit 406 may directly generate a summary video from the video data before being divided into a plurality of scenes. For example, by appropriately extracting a scene including a still image (frame) corresponding to a predetermined label in the video data through labeling by the CNN 403A including the fully connected layer described above, a summary video may be generated.

[0091] <Processing of the communication unit> The communication unit 407 transmits the summary video generated by the summary video generation unit 406 and registered in the storage device 30 to the management device 50.

[0092] For example, the communication unit 407 transmits the summary video to the management device 50 in response to a notification of the completion of the summary video by the summary video generation unit 406.

[0093] Also, for example, the communication unit 407 may transmit a specific summary video to the management device 50 in response to a request from the management device 50.

[0094] Also, for example, when the information processing device 40 is mounted on a working machine or the like, the communication unit 407 may transmit only the summary video in which the importance of the scene of the constituent elements of the summary video generated by the summary video generation unit 406 is relatively high to the management device 50. Thereby, the communication volume from the working machine or the like to the management device 50 can be suppressed, the occupancy state of the communication line can be improved, and the communication cost can be suppressed.

[0095] [Specific examples of scenes of constituent elements of the summary video] Next, with reference to FIG. 7, specific examples of scenes of constituent elements of the summary video will be described.

[0096] FIG. 7 is a diagram showing an example of changes in the time series of image feature amounts.

[0097] As shown in FIG. 7, when there is little movement in the people, work machines, etc. at the construction site CS, the change in the image feature amount is relatively small (the time-series part excluding the parts surrounded by the dashed-dotted line and the two-dotted chain line frames in the figure).

[0098] On the other hand, when there is a relatively large movement in the people, work machines, etc. at the construction site CS, a relatively large change occurs in the image feature amount (the time-series part surrounded by the dashed-dotted line and the two-dotted chain line frames in the figure).

[0099] Therefore, as described above, the information processing apparatus 40 extracts a scene in which the image feature amount in the moving image data represents a relatively large change and generates a summary video, so that the summary video can include relatively large movements of people, work machines, etc. at the construction site CS. At this time, the information processing apparatus 40 may include in the summary video a scene in which a relatively large change occurs in the image feature amount over a relatively long period, such as the time-series part surrounded by the dashed-dotted line in the figure (for example, a scene representing a state in which a predetermined work is started and the work continues). Further, the information processing apparatus 40 may include in the summary video a scene in which a relatively large change occurs in the image feature amount over a relatively short period, such as the time-series part surrounded by the two-dotted chain line in the figure (for example, a scene representing a state in which people, work machines, etc. enter the frame and immediately exit the frame).

[0100] Scenes in which the image feature amount in the moving image data represents a relatively large change may include, for example, scenes including the start time of a predetermined work at the construction site.

[0101] Further, scenes in which the image feature amount in the moving image data represents a relatively large change may include, for example, scenes including when work machines, etc. at the construction site CS shift from a stopped state to an operating state. The work machines, etc. include not only the work machines, etc. included in the moving image data but also the work machines, etc. equipped with the imaging device 10. When the work machines, etc. shift from a stopped state to an operating state, for example, it includes when the work machines, etc. shift from a traveling stopped state to a traveling state. Further, when the work machines, etc. shift from a stopped state to an operating state, for example, it includes when a swing-type work machine, etc. shifts from a swing stopped state to a swing state.

[0102] In addition, scenes in which the image feature amounts in the video data represent relatively large changes may include, for example, scenes including when a working machine or the like at the construction site CS shifts from an operating state to a stopped state. The working machine or the like includes, as in the above case, not only the working machine or the like included in the video data, but also the working machine or the like on which the imaging device 10 is mounted. When the working machine or the like shifts from an operating state to a stopped state, for example, it includes when the working machine or the like shifts from a traveling state to a stopped traveling state. In addition, when the working machine or the like shifts from an operating state to a stopped state, for example, it includes when a swing-type working machine or the like shifts from a swinging state to a stopped swinging state.

[0103] In addition, scenes in which the image feature amounts in the video data represent relatively large changes may include, for example, scenes including when people, working machines, or the like appear in the video data.

[0104] In this way, the information processing apparatus 40 can extract, from the video data, a scene including relatively large movements of people, working machines, or the like at the construction site and generate a summary video by analyzing the video data. Therefore, a manager or the like of the construction site CS can efficiently confirm the working situation of the construction site CS centering on relatively large movements of people, working machines, or the like at the construction site CS.

[0105] Note that the information processing apparatus 40 may extract, for example, as described above, in addition to or instead of scenes including relatively large movements, scenes including a specific working situation (for example, excavation work or the like) and generate a summary video.

[0106] [Function] Next, the function of the work log generation system 1 (information processing apparatus 40) according to the present embodiment will be described.

[0107] In the present embodiment, the information processing apparatus 40 analyzes a video acquired by the imaging device 10 that images the state of the construction site CS, extracts a scene representing a relatively large change in the video, and generates a relatively short summary video.

[0108] As a result, the information processing apparatus 40 can more efficiently confirm the working conditions of the construction site CS, centering on relatively large movements of people, work machines, etc. at the construction site CS.

[0109] Further, in the present embodiment, scenes representing relatively large changes in the original video may include scenes including the start of a predetermined operation at the construction site CS.

[0110] As a result, the information processing apparatus 40 can specifically include in the summary video the scenes that administrators and the like want to confirm as the working conditions of the construction site CS.

[0111] Further, in the present embodiment, scenes representing relatively large changes in the original video may include at least one of a scene including when a work machine or work vehicle at the construction site CS transitions from a stopped state to an operating state, and a scene including when a work machine or work vehicle at the construction site CS transitions from an operating state to a stopped state.

[0112] As a result, the information processing apparatus 40 can specifically include in the summary video the scenes that administrators and the like want to confirm as the working conditions of the construction site CS.

[0113] Further, in the present embodiment, scenes representing relatively large changes in the original video may include scenes including when a person, a work machine, or a work vehicle appears in the video.

[0114] As a result, the information processing apparatus 40 can specifically include in the summary video the scenes that administrators and the like want to confirm as the working conditions of the construction site CS.

[0115] Further, in the present embodiment, the information processing apparatus 40 may extract a scene representing a relatively large change in the original video and a part of a scene that is relatively small in change and continues for a relatively long period in the video to generate a summary video.

[0116] As a result, even for a scene with relatively small changes in the original video but continuing for a relatively long period, the information processing apparatus 40 can include a part thereof in the summary video as a work log.

[0117] Also, in the present embodiment, the information processing apparatus 40 may extract a scene representing a relatively large change in the original video and extract scenes with relatively small changes in the video so that the time interval in the time series for each scene included in the summary video in the original video is equal to or less than a predetermined standard, and generate a summary video.

[0118] As a result, the information processing apparatus 40 can extract scenes evenly in the time series from the original video and generate a summary video.

[0119] Also, in the present embodiment, the image feature amount extraction unit 403 may extract the image feature amount for each still image (frame) in the original video. Then, the summary video generation unit 406 may extract a scene representing a relatively large change in the original video from the analysis result of the original video based on the image feature amount and generate a summary video.

[0120] As a result, the information processing apparatus 40 can specifically generate a summary video using the image feature amount for each frame.

[0121] Also, in the present embodiment, the scene division unit 404 may divide the original video into a plurality of scenes each composed of a series of still images having relatively close image feature amounts based on the image feature amount for each still image in the original video. Also, the scene evaluation unit 405 may evaluate the degree (importance) corresponding to a scene representing a relatively large change in the original video for each of the plurality of scenes based on the image feature amount for each still image therein. Then, the summary video generation unit 406 may extract a scene representing a relatively large change in the original video from among the plurality of scenes based on the evaluation result by the scene evaluation unit 405 and generate a summary video.

[0122] As a result, the information processing apparatus 40 can evaluate the importance for each of a plurality of scenes by using the image feature amount for each frame, and specifically, can generate a summary video.

[0123] In addition, in the present embodiment, the summary video generation unit 406 generates a summary video at a predetermined timing after acquiring the original video, and when it is determined that the user needs to view the summary video at a relatively early timing, the summary video may be generated at a timing earlier than the predetermined timing after acquiring the video.

[0124] As a result, the information processing apparatus 40 can generate a summary video at an earlier timing according to the convenience of the user.

[0125] In addition, in the present embodiment, the imaging device 10 may be mounted on the work machine and image the state around the work machine. Then, the information processing apparatus 40 may be mounted on the work machine and extract a scene representing a relatively large change in the video by analyzing the video acquired by the imaging device 10, and generate a relatively short summary video.

[0126] As a result, the work machine can generate a summary video by using the image of the imaging device that images the situation around it.

[0127] In addition, in the present embodiment, the work machine may transmit to the management device 50 a summary video having a relatively high degree (importance) representing a relatively large change in the component scene among the summary videos generated by the information processing apparatus 40.

[0128] As a result, the work machine can selectively transmit to the management device 50 the summary video that the user (such as an administrator in charge of a plurality of construction sites CS) of the management device 50 should view. Therefore, the work machine can improve the occupancy state of the communication line and suppress the communication cost.

[0129] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

Explanation of Signs

[0130] 1 Work log generation system 10 Imaging device 20 Control device 30 Storage device 40 Information processing device 41 External interface 41A Recording medium 42 Auxiliary storage device 43 Memory device 44 CPU 45 High-speed arithmetic unit 46 Communication interface 47 Input device 48 Display device 50 Management device 401 Image acquisition unit 402 Storage unit 403 Image feature extraction unit 403A Convolutional neural network 404 Scene segmentation unit 405 Scene evaluation unit 406 Summary video generation unit 407 Communication unit CS Construction site

Claims

1. An image feature amount extraction unit that extracts image feature amounts for each still image in a video acquired by an imaging device that images the state of a construction site; A scene division unit that divides the video into a plurality of scenes each composed of a series of still images having relatively close image feature amounts based on the image feature amounts for each still image; A summary video generation unit that extracts a scene representing a relatively large change from among the plurality of scenes, and when there is a scene that continues for a relatively long period among the plurality of scenes, extracts a part of that scene regardless of whether it is a scene representing a relatively large change to generate a relatively short summary video. An information processing apparatus.

2. Among the scenes representing relatively large changes in the video, scenes including the start time of a predetermined operation at the construction site are included. The information processing apparatus according to Claim 1.

3. Among the scenes representing relatively large changes in the video, at least one of a scene including when a work machine or work vehicle at the construction site transitions from a stopped state to an operating state and a scene including when a work machine or work vehicle at the construction site transitions from an operating state to a stopped state is included. The information processing apparatus according to Claim 1 or 2.

4. Among the scenes representing relatively large changes in the video, scenes including when a person, work machine, or work vehicle appears in the video are included. The information processing apparatus according to any one of Claims 1 to 3.

5. A summary video is generated by extracting a scene representing a relatively large change in the video and a part of a scene that continues for a relatively long period and has relatively small changes in the video. The information processing apparatus according to any one of Claims 1 to 4.

6. A scene representing a relatively large change in the video is extracted, and a scene with relatively small changes in the video is extracted to generate the summary video so that the time interval in the time series for each scene included in the summary video in the video is below a predetermined standard. The information processing apparatus according to any one of Claims 1 to 5.

7. For each of the plurality of scenes, a scene evaluation unit is provided that evaluates the degree to which it corresponds to a scene representing a relatively large change in the video based on the image feature amounts for each still image in that scene. Based on the evaluation result by the scene evaluation unit, the summary video generation unit extracts a scene representing a relatively large change in the video from among the plurality of scenes, and generates the summary video. The information processing apparatus according to any one of claims 1 to 6.

8. After acquiring the video, at a predetermined timing, the summary video is generated. If it is determined, according to an input from the outside, that the user needs to view the summary video at a relatively early timing, then after acquiring the video, the summary video is generated at a timing earlier than the predetermined timing. The information processing apparatus according to any one of claims 1 to 7.

9. An imaging device that images the state around the work machine, An image feature amount extraction unit that extracts the image feature amount for each still image in the video acquired by the imaging device, A scene division unit that divides the video into a plurality of scenes each composed of a series of still images having relatively close image feature amounts based on the image feature amounts for each still image, An information processing apparatus comprising: a summary video generation unit that extracts a scene representing a relatively large change from among the plurality of scenes, and when there is a scene that continues for a relatively long period among the plurality of scenes, extracts a part of the scene regardless of whether it is a scene representing a relatively large change, and generates a relatively short summary video. A work machine.

10. Transmits to a predetermined external device the summary video in which the degree representing the relatively large change in the scene of the components of the summary video generated by the information processing apparatus is relatively high. The work machine according to claim 9.

11. An information processing method executed by an information processing apparatus, An image feature amount extraction step of extracting the image feature amount for each still image in the video acquired by an imaging device that images the state of a construction site, A scene division step of dividing the video into a plurality of scenes each composed of a series of still images having relatively close image feature amounts based on the image feature amounts for each still image, A summary video generation step of extracting a scene representing a relatively large change from among the plurality of scenes, and when there is a scene that continues for a relatively long period among the plurality of scenes, extracting a part of the scene regardless of whether it is a scene representing a relatively large change, and generating a relatively short summary video. An information processing method.

12. To the information processing apparatus An image feature quantity extraction step of extracting image feature quantities for each still image in a video acquired by an imaging device that images a construction site; A scene division step of dividing the video into a plurality of scenes each composed of a series of the still images having the image feature quantities that are relatively close to each other, based on the image feature quantities for each of the still images; A summary video generation step of extracting a scene representing a relatively large change from among the plurality of scenes, and when there is a scene that continues for a relatively long period among the plurality of scenes, extracting a part of the scene to generate a relatively short summary video regardless of whether the scene represents a relatively large change; and causing the above to be executed. Program.

Images