Image processing method, information processing apparatus, and wearable terminal

Abstract

To provide an image processing method, an information processing apparatus, and a wearable terminal configured to determine images captured when a site worker is moving or images captured when the site worker is not moving, from among video images captured by a wearable terminal carried by the site worker.SOLUTION: An image processing method is executed by an information processing apparatus or a wearable terminal having a control unit. The control unit is configured to: extract an image from a video image captured by a wearable terminal carried by a site worker; detect a change region from regions in peripheral portions in the image; and determine whether the extracted image is an image captured when the site worker is moving or an image captured when the site worker is not moving, based on the change region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing method, an information processing device, and a wearable terminal. [Background technology]

[0002] For example, annotation tools are used in the annotation process of adding information to videos. Analysts, such as annotators, view the videos, determine the content of the footage, and add information such as labels to the images included in the videos.

[0003] BACKGROUND ART Techniques for automatically extracting scenes in which the face of a subject is clearly captured or scenes in which the subject moves significantly from a video are known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-1821 Summary of the Invention [Problem to be solved by the invention]

[0005] Videos captured by wearable devices carried by field workers may contain images captured while the field workers were moving. In annotation work, it is time-consuming to determine from the video captured by the wearable device whether the images were captured while the field workers were moving or not. Note that Patent Document 1 does not mention determining whether the images were captured while the field workers were moving or not from the video captured by the wearable device.

[0006] The present disclosure aims to provide an image processing method, an information processing device, and a wearable device that determine whether images were taken while a field worker was moving or not from videos taken by a wearable device carried by the field worker. [Means for solving the problem]

[0007] A first aspect of the present disclosure is an image processing method executed by an information processing device or a wearable device having a control unit, in which the control unit extracts an image from a video captured by the wearable device carried by a field worker, detects a changed area from a peripheral area within the image, and determines, based on the changed area, whether the extracted image was captured while the field worker was moving or was captured at a time other than when the field worker was moving.

[0008] According to a first aspect of the present disclosure, it is possible to provide an image processing method for determining whether an image was taken while the field worker was moving or not, from a video captured by a wearable device carried by the field worker.

[0009] A second aspect of the present disclosure is an image processing method of the first aspect, wherein the peripheral area within the image is the area of ​​the four corners of the image, the area at the center of the top, bottom, left, and right sides of the image, or the entire peripheral area within the image.

[0010] According to the second aspect of the present disclosure, a changed area can be detected from the four corner areas of an image, the central areas of the top, bottom, left, and right sides of the image, or the entire peripheral area of ​​the image.

[0011] A third aspect of the present disclosure is an image processing method of the first or second aspect, wherein the control unit detects the changed area based on inter-frame differences in a peripheral area within the image extracted from the video at a predetermined time interval.

[0012] According to the third aspect of the present disclosure, a changed area can be detected based on inter-frame differences in peripheral areas within an image.

[0013] A fourth aspect of the present disclosure is an image processing method of the third aspect, wherein the control unit calculates the inter-frame difference for each pixel included in a peripheral area of ​​the image, determines which pixels have changed based on their magnitude relative to a threshold, determines the proportion of changed pixels included in the peripheral area of ​​the image, and detects the changed area according to the proportion of changed pixels.

[0014] According to a fourth aspect of the present disclosure, the inter-frame difference is calculated for each pixel included in the peripheral region of the image, pixels that have changed are determined based on their magnitude relationship with a threshold, the proportion of pixels that have changed included in the peripheral region of the image is determined, and changed regions can be detected based on the proportion of pixels that have changed.

[0015] A fifth aspect of the present disclosure is the image processing method of the fourth aspect, wherein the control unit changes the threshold value according to brightness of the captured image.

[0016] According to a fifth aspect of the present disclosure, pixels that have changed are determined based on their magnitude relationship with a threshold value that is changed according to the brightness of the captured image, the proportion of changed pixels contained in the peripheral area of ​​the image is determined, and the changed area can be detected according to the proportion of changed pixels.

[0017] A sixth aspect of the present disclosure is an image processing method according to any one of the first to fifth aspects, wherein the control unit determines whether the image was taken while the field worker was moving or while the field worker was not moving, based on the number of changed areas or the proportion of changed areas detected from the extracted image.

[0018] According to a sixth aspect of the present disclosure, it is possible to determine whether the image was taken while the field worker was moving or when the field worker was not moving, based on the number of changed areas or the proportion of changed areas detected from the extracted image.

[0019] A seventh aspect of the present disclosure is an image processing method described in any one of the first to sixth aspects, wherein the video is a first-person perspective video captured while the wearable device is worn by the field worker.

[0020] According to a seventh aspect of the present disclosure, an image processing method can be provided that determines images taken while a field worker is moving or not moving from a first-person perspective video taken while the field worker is wearing a wearable device.

[0021] An eighth aspect of the present disclosure is an information processing device having a control unit, wherein the control unit extracts an image from a video captured by a wearable device carried by a field worker, detects a changed area from a peripheral area within the image, and determines, based on the changed area, whether the extracted image was captured while the field worker was moving or was captured at a time other than when the field worker was moving.

[0022] According to the eighth aspect of the present disclosure, it is possible to provide an information processing device that determines images captured while the field worker is moving or not moving from videos captured by a wearable device carried by the field worker.

[0023] A ninth aspect of the present disclosure is a wearable terminal having a control unit and carried by a field worker, wherein the control unit captures a video, extracts an image from the captured video, detects a change area from a peripheral area within the image, and determines based on the change area whether the extracted image was captured while the field worker was moving or was captured at a time other than when the field worker was moving.

[0024] According to the ninth aspect of the present disclosure, it is possible to provide a wearable device that determines images taken while the field worker is moving or not moving from videos taken by the wearable device carried by the field worker. [Brief explanation of the drawings]

[0025] [Figure 1]1 is a configuration diagram of an example of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to the present embodiment. [Figure 3] FIG. 1 is an external view of an example of a wearable terminal according to an embodiment of the present invention. [Figure 4] FIG. 2 is a functional configuration diagram of an example of a wearable terminal according to the present embodiment. [Figure 5] 10A and 10B are explanatory diagrams illustrating an example of a movement determination process according to the embodiment; [Figure 6] FIG. 2 is a sequence diagram showing an example of a processing procedure of the information processing system according to the present embodiment. [Figure 7] 10 is a flowchart illustrating an example of a procedure for a movement determination process according to the present embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a display of an annotation tool that simplifies annotation work performed by an analysis worker. [Figure 9] FIG. 2 is a sequence diagram showing an example of a processing procedure of the information processing system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Next, embodiments of the present disclosure will be described in detail.

[0027] [Embodiment] <System configuration> 1 is a configuration diagram of an example of an information processing system 1 according to this embodiment. The information processing system 1 shown in FIG.

[0028] The wearable device 10 is a device carried by or worn by a field worker performing field work at the field site 2. The wearable device 10 carried by or worn by the field worker can capture video from the field worker's first-person perspective. The video from the field worker's first-person perspective may be video captured in the field worker's line of sight, or may be video captured in front of the field worker.

[0029] For example, the wearable device 10 is attached to any location of the field worker, such as the arm, eyes, ears, or neck, or on the clothing worn by the field worker. In this embodiment, an example of a neck-worn wearable device 10 that is attached to the neck of a field worker and captures video from the field worker's first-person perspective will be described.

[0030] The wearable terminal 10 is communicatively connected to the information processing device 20 via a network 50. The wearable terminal 10 may be communicatively connected to the information processing device 20 via the network 50 by using a communication function of a mobile terminal such as a smartphone of a field worker. The network 50 is, for example, the Internet. The network 50 may also be a LAN (Local Area Network) or a dedicated communication line.

[0031] The wearable terminal 10 has a control unit 12. The control unit 12 is a hardware configuration that executes a program, and is a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The CPU, which is an example of the control unit 12, executes the program, thereby enabling the wearable terminal 10 to perform various processes described below.

[0032] The information processing device 20 is communicably connected to the wearable device 10 and the analysis operator terminal 30 via a network 50. The information processing device 20 receives data of moving images captured by the wearable device 10 (hereinafter simply referred to as moving images).

[0033] As will be described later, the information processing device 20 determines whether an image was taken while the field worker was moving or not moving from the video captured by the wearable terminal 10. For example, the information processing device 20 uses the result of determining whether an image was taken while the field worker was moving or not for processing annotation work, as will be described later.

[0034] The information processing device 20 also has a control unit 22. The control unit 22 is a hardware configuration that executes a program, such as a CPU, an ASIC, or an FPGA. For example, the information processing device 20 can perform various processes described below by having a CPU, which is an example of the control unit 22, execute a program.

[0035] The analysis worker terminal 30 is a device operated by an analysis worker such as an annotator, etc. The analysis worker terminal 30 is equipped with various tools used by the analysis worker, such as annotation tools.

[0036] The analysis worker performs annotation work by operating the analysis worker terminal 30. The analysis worker terminal 30 accepts an annotation work request from the analysis worker and transmits the request to the information processing device 20. The analysis worker terminal 30 receives a response to the request from the information processing device 20 and displays the response to the annotation work.

[0037] The analysis operator terminal 30 is a hardware configuration that executes a program and has a control unit such as a CPU, ASIC, or FPGA. The analysis operator terminal 30 can perform various processes described below by having the CPU, which is an example of a control unit, execute the program.

[0038] The information processing device 20 is, for example, a personal computer (PC) or a workstation. The information processing device 20 may be realized using an application service provider (ASP) or cloud computing. The analysis operator terminal 30 is, for example, a PC, a smartphone, or a tablet terminal.

[0039] 1 is an example. For example, the information processing system 1 may be configured with one or more information processing devices 20. It goes without saying that the information processing system 1 may have various system configuration examples depending on the application and purpose.

[0040] <Device configuration> 1 is realized by, for example, a computer 500 having the hardware configuration shown in Fig. 2. Furthermore, the analysis operator terminal 30 in Fig. 1 may be realized by the computer 500 having the hardware configuration shown in Fig. 2.

[0041] 2 is a diagram showing an example of the hardware configuration of a computer 500 according to this embodiment. The computer 500 includes an input device 501, a display device 502, an external I / F 503, a RAM (Random Access Memory) 504, a ROM (Read Only Memory) 505, a CPU 506, a communication I / F 507, and an HDD (Hard Disk Drive) 508, all of which are interconnected by a bus B. The input device 501 and the display device 502 may be connected and used when necessary.

[0042] The input device 501 includes a touch panel, operation keys, buttons, a keyboard, a mouse, etc. The display device 502 includes a display for displaying a screen, a speaker for outputting sound, etc.

[0043] The communication I / F 507 is an interface that allows the computer 500 to perform data communication via the network 50. The HDD 508 is an example of a non-volatile storage device that stores programs and data. The programs and data include an OS (Operating System), which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use an SSD (Solid State Drive) instead of the HDD 508.

[0044] The external I / F 503 is an interface with an external device. The external device may be a recording medium 503a. The computer 500 reads and writes data from and to the recording medium 503a via the external I / F 503.

[0045] The recording medium 503a is a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), an SD (Secure Digital) memory card, a USB (Universal Serial Bus) memory, or the like.

[0046] The ROM 505 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 505 stores programs and data such as a BIOS (Basic Input Output System), OS settings, and network settings that are executed when the computer 500 starts up. The RAM 504 is an example of a volatile semiconductor memory (storage device) that temporarily retains programs and data.

[0047] The CPU 506 is a computing device that reads programs and data from a storage device such as the ROM 505 or the HDD 508 onto the RAM 504 and executes processing to realize overall control and functions of the computer 500. The CPU 506 is an example of the control unit 22 of the information processing device 20 and the analysis control unit.

[0048] The wearable terminal 10 in Fig. 1, for example, in the case of a neck-worn type wearable terminal 10, is realized by the hardware configuration shown in Fig. 3 and Fig. 4. Fig. 3 is an external view of an example of the wearable terminal 10 according to this embodiment. Fig. 4 is a functional configuration diagram of an example of the wearable terminal 10 according to this embodiment.

[0049] The housing that constitutes the wearable terminal 10 includes a left arm section 110, a right arm section 120, and a main body section 130. The left arm section 110 and the right arm section 120 extend forward from the left end and the right end of the main body section 130, respectively. The wearable terminal 10 has a structure that forms a substantially U-shape as a whole device when viewed from above. For example, when a field worker, who is an example of a wearer, puts on the wearable terminal 10, the main body section 130 comes into contact with the back of the neck, and the left arm section 110 and the right arm section 120 hang down from the side of the neck toward the chest, so that the device is hooked around the neck.

[0050] The left arm 110 and the right arm 120 are each provided with a plurality of sound collection units 141 to 145. The sound collection units 141 to 145 are, for example, microphones. The sound collection units 141 to 145 are arranged mainly for the purpose of acquiring the voices of the wearer and the interlocutor. The sound collection units 141 and 142 are provided on the left arm 110. The sound collection units 143 and 144 are provided on the right arm 120. The left arm 110 and the right arm 120 may be provided with one or more additional sound collection units.

[0051] 3, in addition to the sound collection units 141 and 142, a sound collection unit 145 is provided on the left arm 110. The sound signals acquired by the sound collection units 141 to 145 are transmitted to the control unit 12 provided in the main body 130, where predetermined processing is performed.

[0052] The left arm 110 is further provided with an imaging unit 160. The imaging unit 160 is provided on the distal end surface 112 of the left arm 110. The imaging unit 160 captures video and still images of the front side of the site worker. The video and still images captured by the imaging unit 160 are transmitted to the control unit 12 in the main body 130. The video and still images captured by the imaging unit 160 are stored in, for example, the storage unit 181. The wearable terminal 10 may also transmit the video and still images captured by the imaging unit 160 to, for example, the information processing device 20 for storage.

[0053] The right arm 120 is further provided with a non-contact sensor unit 170. The sensor unit 170 is disposed on the distal end surface 122 of the right arm 120 in order to detect, for example, the movement of the hand of a field worker. The detection information of the sensor unit 170 is used to control the image capturing unit 160, such as activating the image capturing unit 160, starting and stopping image capturing.

[0054] For example, the sensor unit 170 may control the imaging unit 160 by detecting that a hand or the like of a field worker has come close, or may control the imaging unit 160 by detecting a predetermined gesture made by the field worker within the detection range of the sensor unit 170. Note that in this embodiment, the imaging unit 160 is arranged on the distal end surface 112 of the left arm 110, and the sensor unit 170 is arranged on the distal end surface 122 of the right arm 120, but the positions of the imaging unit 160 and the sensor unit 170 may be interchanged.

[0055] The wearable terminal 10 may also use the detection information of the sensor unit 170 to activate at least one of the image capture unit 160, the sound collection units 141 to 145, and the control unit 12. For example, in a state where the sensor unit 170, the sound collection units 141 to 145, and the control unit 12 are always active and the image capture unit 160 is stopped, the wearable terminal 10 may activate the image capture unit 160 when the sensor unit 170 detects a specific gesture. For example, in a state where the sensor unit 170, the sound collection units 141 to 145, and the control unit 12 are always active and the image capture unit 160 is stopped, the wearable terminal 10 may also activate the image capture unit 160 when the sound collection units 141 to 145 detect a specific sound.

[0056] 4, the left arm 110 is provided with a sound collection unit 141, a sound collection unit 142, a sound collection unit 145, an operation unit 150, and an imaging unit 160. The right arm 120 is provided with a sound collection unit 143, a sound collection unit 144, and a sensor unit 170. The main body 130 is provided with a control unit 12, a memory unit 181, a communication unit 182, a proximity sensor 183, a sound emission unit 184, and a battery 190. In addition to the functional configuration shown in FIG. 4, the wearable terminal 10 may be equipped with sensors such as a gyro sensor, an acceleration sensor, a geomagnetic sensor, or a GPS sensor as appropriate.

[0057] Well-known microphones such as a dynamic microphone, a condenser microphone, a MEMS (Micro-Electrical-Mechanical Systems) microphone, etc. can be used as the sound collection units 141 to 145. The sound collection units 141 to 145 convert sound into an electrical signal, which is then converted into digital information by an A / D conversion circuit and transmitted to the control unit 12.

[0058] The operation unit 150 accepts operation inputs from a field worker. The operation unit 150 can be, for example, a known switch circuit or touch panel. The operation unit 150 accepts, from the field worker, operations such as turning the power on or off, and operations necessary to realize the functions of the wearable terminal 10. Information input via the operation unit 150 is sent to the control unit 12.

[0059] The imaging unit 160 captures moving images or still images. For example, a general digital camera may be used as the imaging unit 160. The moving images or still images acquired by the imaging unit 160 are transmitted to the control unit 12. The moving images or still images acquired by the imaging unit 160 are stored in the storage unit 181, and are subjected to processing such as the movement determination processing described below. The moving images or still images acquired by the imaging unit 160 may also be transmitted to the information processing device 20, stored in the information processing device 20, and are subjected to processing such as the movement determination processing described below. Furthermore, since the imaging unit 160 generally consumes a large amount of power, it is preferable that the imaging unit 160 automatically enter a sleep state after a certain period of time has elapsed since imaging stopped.

[0060] The sensor unit 170 is a non-contact detection device for detecting the movement of the worker's fingers, etc. The sensor unit 170 is, for example, a proximity sensor or a gesture sensor. The proximity sensor detects, for example, when the worker's fingers approach within a predetermined range. The proximity sensor may be a known type such as an optical type, an ultrasonic type, a magnetic type, a capacitance type, or a thermal type.

[0061] The gesture sensor detects, for example, the movement and shape of the fingers of a field worker. The gesture sensor is, for example, an optical sensor that irradiates an object with light from an infrared LED and captures changes in the reflected light with a light-receiving element to detect, for example, the movement and shape of the fingers of a field worker. Detection information from the sensor unit 170 is transmitted to the control unit 12 and used for controlling the imaging unit 160, etc.

[0062] The sensor unit 170 generally consumes little power, and is therefore preferably kept running at all times while the power of the wearable terminal 10 is on. Alternatively, the sensor unit 170 may be started when the proximity sensor 183 detects that the wearable terminal 10 is being worn.

[0063] The control unit 12 performs arithmetic processing to control the wearable terminal 10. The control unit 12 can use a processor such as a CPU. The control unit 12 reads out a program stored in the storage unit 181 and executes predetermined arithmetic processing in accordance with the program. The control unit 12 also writes and reads out the results of the arithmetic processing in accordance with the program to and from the storage unit 181.

[0064] The memory unit 181 stores information used for arithmetic processing and the like in the control unit 12, and the results of the arithmetic processing. The storage function of the memory unit 181 can be realized by a non-volatile memory such as an HDD or an SDD. The memory unit 181 may also function as a memory for writing or reading intermediate progress of arithmetic processing by the control unit 12. The memory function of the memory unit 181 can be realized by a volatile memory such as a RAM or a DRAM.

[0065] The communication unit 182 may employ a communication module for wireless communication in accordance with a known mobile communication standard such as 3G (W-CDMA), 4G (LTE / LTE-Advanced), or 5G. The communication unit 182 may employ a communication module for wireless communication in accordance with a wireless LAN system such as Wi-Fi (registered trademark). The communication unit 182 may also employ a communication module for close proximity wireless communication in accordance with a system such as Bluetooth (registered trademark) or NFC.

[0066] The proximity sensor 183 is disposed inside the main body 130 and detects when the neck of the field worker approaches within a predetermined range. The sound emitting unit 184 converts the electrical signal into sound. For example, the sound emitting unit 184 is a general speaker that transmits sound to the field worker by vibrating the air. Alternatively, the sound emitting unit 184 may be a bone conduction speaker that transmits sound to the field worker by vibrating the bones of the field worker.

[0067] The battery 190 supplies power to various electronic components included in the wearable terminal 10. The battery 190 may be a rechargeable battery such as a lithium ion battery, a lithium polymer battery, an alkaline storage battery, a nickel-cadmium battery, a nickel-metal hydride battery, or a lead storage battery.

[0068] <Processing> In this embodiment, a field worker performing field work at a field site 2 carries or wears a wearable device 10 and captures video of the field work. The video of the field work captured by the wearable device 10 is video from the field worker's first-person perspective.

[0069] A video from a first-person perspective of a field worker captured by the wearable device 10 may include a moving scene in which the field worker moves back and forth carrying luggage and tools. For example, a video of an air conditioner repair from a first-person perspective of a field worker may include a moving scene in which the field worker moves back and forth carrying luggage and tools between the indoor and outdoor units of the air conditioner. A moving scene is an image captured by the wearable device 10 while the field worker is moving.

[0070] Annotation work may be performed by an analyst such as an annotator on a first-person perspective video of a field worker captured by the wearable device 10. When annotation work is performed on a first-person perspective video of a field worker captured by the wearable device 10, images other than the moving scenes are often more important than images of the moving scenes. The reason for this is that in annotation work on a video of field work, information such as the work content is often added to images other than the moving scenes, but no information is added to the moving scene images.

[0071] In this embodiment, the following process is provided to determine whether the images included in the video captured by the wearable terminal 10 were captured while the field worker was moving, or whether the images were captured while the field worker was not moving.

[0072] One method for determining whether an image was taken while the field worker was moving or not is to take the difference between frames and make a determination from that difference. However, in the first-person perspective video of the field worker taken by the wearable device 10, the distance to the subject of the video taken while the field worker was working on site is often close, and the accuracy of determining whether the image was taken while the field worker was moving or not may be poor as described below.

[0073] When the video captured by the wearable device 10 from the first-person perspective of the on-site worker is a video of an air conditioner repair, the distance between the on-site worker and the subject of the image is short, and the subject appears large in the image captured by the wearable device 10. Furthermore, when repairing an air conditioner, the on-site worker often faces the subject of the image, and there is a high possibility that the subject will appear in the central area of ​​the image captured by the wearable device 10. Therefore, when the video captured by the wearable device 10 from the first-person perspective of the on-site worker is taken, if the difference between frames is calculated for the entire screen, even a small change in the work is likely to be determined to be a large change (movement).

[0074] Furthermore, if a field worker is moving while carrying luggage or tools, there is a high possibility that the luggage or tools will appear large in the central area of ​​the image captured by the wearable device 10. In this case, even if the difference between frames is taken, there is a high possibility that the change will be determined to be small (no movement). Furthermore, if the field worker stands up or sits down, there is a high possibility that the change will be determined to be large (movement) when the difference between frames is taken.

[0075] Therefore, in this embodiment, the following movement determination process is performed to accurately determine whether the images included in the video captured by the wearable terminal 10 were captured while the field worker was moving or were captured at a time other than when the field worker was moving.

[0076] 5 is an explanatory diagram illustrating an example of the movement determination process according to this embodiment. In the movement determination process according to this embodiment, an image 1000 is extracted from a video taken by the wearable device 10 from a first-person perspective of a field worker, and a changed area is detected from an area 1002 around the periphery of the image 1000.

[0077] The shape of the peripheral region 1002 of the image 1000 in which the changed region is detected may vary. For example, Fig. 5(A) shows an example in which the peripheral region 1002 of the image 1000 is the region at the four corners of the image 1000. Fig. 5(B) shows an example in which the peripheral region 1002 of the image 1000 is the central region of the top, bottom, left, and right sides of the image 1000. Fig. 5(C) shows an example in which the peripheral region 1002 of the image 1000 is the entire peripheral region of the image 1000.

[0078] The peripheral region 1002 of the image 1000 may be set to the central portion of the image 1000, and may be the region excluding the central portion. The central portion of the image 1000 may be set, for example, by dividing the image 1000 into five vertical and four horizontal portions, and the region that does not contact the periphery of the image 1000 may be set to the central portion of the image 1000.

[0079] In the movement determination process according to this embodiment, changed areas are detected from a peripheral area 1002 of the image 1000, and based on the detected changed areas, it is determined whether the image was taken while the field worker was moving or while the field worker was not moving. For example, in the movement determination process according to this embodiment, it is determined whether the image was taken while the field worker was moving or while the field worker was not moving, based on the number of changed areas or the proportion of changed areas detected. The detection of changed areas is performed as described below based on inter-frame differences in the peripheral area 1002 of the image 1000.

[0080] The movement determination process according to this embodiment may be performed by the information processing device 20 or the wearable terminal 10. Below, an example in which the movement determination process according to this embodiment is performed by the information processing device 20 and an example in which the movement determination process according to this embodiment is performed by the wearable terminal 10 will be described.

[0081] 6 is a sequence diagram showing an example of a processing procedure of the information processing system 1 according to this embodiment. FIG. 6 shows an example in which the movement determination processing according to this embodiment is performed by the information processing device 20.

[0082] In step S10, the wearable device 10 carried or worn by the field worker captures a first-person perspective video of the field worker. The first-person perspective video captured by the field worker at the site 2 may include a moving scene in which the field worker moves around the site 2. The first-person perspective video captured by the field worker at the site 2 includes images captured while the field worker is moving or at a time other than when the field worker is moving.

[0083] In step S12, the wearable device 10 transmits the captured first-person perspective video of the field worker to the information processing device 20. The processing of step S12 may be performed after the wearable device 10 has finished capturing the video, or may be performed while the video is being captured by streaming or the like.

[0084] In step S14, the control unit 22 of the information processing device 20 stores the video received from the wearable terminal 10 in a storage device such as the HDD 508. In step S16, the control unit 22 of the information processing device 20 performs, for example, movement determination processing shown in Fig. 7. Fig. 7 is a flowchart showing an example of the procedure of the movement determination processing according to this embodiment.

[0085] In step S30, the control unit 22 extracts images every second from the video of the first-person perspective of the field worker captured by the wearable device 10. The control unit 22 extracts images every second from the video of the first-person perspective of the field worker captured by the wearable device 10 through the processing of step S30. Note that every second is just an example, and any predetermined time interval may be used.

[0086] In step S32, the control unit 22 selects one image to be subjected to movement determination from the images extracted in step S30. The image to be subjected to movement determination selected in step S32 is an unprocessed image that has not been subjected to movement determination processing.

[0087] In step S34, control unit 22 selects an image preceding the image subject to movement determination selected in step S32 (hereinafter referred to as the previous image) and an image following the image subject to movement determination selected in step S32 (hereinafter referred to as the subsequent image). The previous image is the image preceding the image subject to movement determination among the temporally consecutive images extracted in step S30. The subsequent image is the image following the image subject to movement determination among the temporally consecutive images extracted in step S30.

[0088] The control unit 22 cuts out, for example, the peripheral area 1002 shown in Fig. 5 from the image for which movement is to be determined, the previous image, and the subsequent image. Here, an example of the peripheral area 1002 shown in Fig. 5(A) will be described. The control unit 22 cuts out, for example, the four corner areas as shown in Fig. 5(A) from the image for which movement is to be determined, the previous image, and the subsequent image.

[0089] In step S36, control unit 22 determines whether or not processing has been completed for all of the four corner regions cut out in step S34. If processing has not been completed for all of the four corner regions cut out in step S34, in other words, if there are unprocessed areas remaining in the four corner regions cut out in step S34, control unit 22 proceeds to processing in step S38.

[0090] In step S38, the control unit 22 reads out the cut-out image to be subjected to movement determination, the previous image, and the subsequent image for one unprocessed area of ​​the four corner areas cut out in step S34. For example, if the upper left peripheral area 1002 shown in Fig. 5(A) is unprocessed, the control unit 22 reads out the upper left peripheral area 1002 of the image to be subjected to movement determination, the previous image, and the subsequent image.

[0091] In step S40, the control unit 22 grayscales one peripheral area 1002 (for example, the upper left peripheral area) read out in step S38 from the peripheral areas 1002 of the image to be subjected to movement determination, the previous image, and the next image.

[0092] In step S42, the control unit 22 creates a difference image (hereinafter referred to as a first difference image) between the image of the peripheral area 1002 that is the target of movement determination and that has been grayscaled in step S40 and the previous image. The first difference image is an image of the difference in pixel value between the image of the peripheral area 1002 that is the target of movement determination and that has been grayscaled in step S40 and the previous image. The control unit 22 also creates a difference image (hereinafter referred to as a second difference image) between the image of the peripheral area 1002 that is the target of movement determination and that has been grayscaled in step S40 and the subsequent image. The second difference image is an image of the difference in pixel value between the image of the peripheral area 1002 that is the target of movement determination and that has been grayscaled in step S40 and the subsequent image.

[0093] In step S44, the control unit 22 calculates the logical product of the first difference image and the second difference image, and creates a logical product image of the two difference images. For example, the logical product of the first difference image and the second difference image is calculated as follows:

[0094] The logical product of the difference images is calculated for each pixel value at the same position in the first difference image and the second difference image. For example, let us consider a case where the pixel value of the first difference image is decimal "178" and the pixel value of the second difference image is decimal "159."

[0095] The control unit 22 converts the pixel value of the first difference image, expressed in decimal "178", to binary "10110010", and converts the pixel value of the second difference image, expressed in decimal "159", to binary "10011111".

[0096] The control unit 22 calculates the logical product "10010010" by bitwise ANDing the binary numbers "10110010" and "10011111". The control unit 22 converts the calculated logical product "10010010" back into a decimal number to obtain the pixel value of the logical product image.

[0097] In step S46, the control unit 22 performs binarization processing on the logical product image to create a mask image. The binarization processing performed on the logical product image is performed by comparing each pixel value of the logical product image created in step S44 with a threshold value (e.g., 50) used in the binarization processing, and changing pixel values ​​equal to or greater than the threshold value used in the binarization processing to "255" and pixel values ​​smaller than the threshold value used in the binarization processing to "0."

[0098] By performing binarization processing on the logical product image, the mask image can display pixels in white where the difference between the two difference images is equal to or greater than the threshold used in the binarization processing, and pixels in black where the difference between the two difference images is less than the threshold used in the binarization processing. The threshold used in the binarization processing may be changed depending on the brightness of the image captured by the wearable terminal 10. For example, the threshold used in the binarization processing may be changed so that it is larger as the brightness of the image captured by the wearable terminal 10 increases, and smaller as the brightness of the captured image decreases.

[0099] In step S48, the control unit 22 calculates the proportion of white pixels contained in the mask image created in step S46. The proportion of white pixels calculated in step S48 is an example of the proportion of changed pixels. In this embodiment, changed pixels are considered to be white pixels for the purpose of calculation, but changed pixels may also be considered to be black pixels. The processes of steps S36 to S48 are repeated and executed until it is determined that processing has been completed for all of the four corner areas cut out in step S34. In this way, the control unit 22 calculates the proportion of white pixels contained in the mask image created in step S46 for each of the four corner areas cut out in step S34.

[0100] Also, if processing has been completed for all of the four corner areas cut out in step S34, in other words, if there are no unprocessed areas remaining in the four corner areas cut out in step S34, the control unit 22 proceeds to processing in step S50.

[0101] In step S50, the control unit 22 determines whether the image to be subjected to movement determination is in motion or not in motion based on the number or proportion of peripheral areas 1002 where the proportion of white is equal to or greater than the threshold used to determine a changing area. The peripheral areas 1002 where the proportion of white is equal to or greater than the threshold used to determine a changing area are an example of a changing area. The control unit 22 may set the result of determining whether the image to be subjected to movement determination is in motion or not in motion as a label for the image to be subjected to movement determination. The control unit 22 may set a label indicating that the image is in motion for an image to be subjected to movement determination that is determined to be in motion.

[0102] For example, in the case of the peripheral region 1002 in Fig. 5(A), if the peripheral region 1002 of the four corners cut out in step S34 contains a threshold (for example, two) or more changed regions where the ratio of white is equal to or greater than the threshold used to determine the changed region and is used for the final determination of whether the image is moving or not moving, the control unit 22 determines that the image to be subjected to movement determination is not moving. Also, in the case of the peripheral region 1002 in Fig. 5(A), if the peripheral region 1002 of the four corners cut out in step S34 does not contain a threshold (for example, two) or more changed regions where the ratio of white is equal to or greater than the threshold used to determine the changed region and is used for the final determination of whether the image is moving or not moving, the control unit 22 determines that the image to be subjected to movement determination is not moving.

[0103] For example, in the case of the peripheral region 1002 in Fig. 5(A), if the peripheral region 1002 of the four corners cut out in step S34 contains a changing region where the ratio of white is equal to or greater than the threshold used to determine the changing region and is equal to or greater than the threshold (e.g., 50%) used for the final determination of whether the image is moving or not moving, the control unit 22 determines that the image to be subjected to movement determination is not moving. Also, in the case of the peripheral region 1002 in Fig. 5(A), if the peripheral region 1002 of the four corners cut out in step S34 does not contain a changing region where the ratio of white is equal to or greater than the threshold used to determine the changing region and is equal to or greater than the threshold (e.g., 50%) used for the final determination of whether the image is moving or not moving, the control unit 22 determines that the image to be subjected to movement determination is not moving.

[0104] In step S52, the control unit 22 determines whether or not there is an unprocessed image for which movement determination processing has not been performed among the images extracted in step S30. If there is an unprocessed image for which movement determination processing has not been performed among the images extracted in step S30, the control unit 22 returns to the processing of step S32 and continues the processing.

[0105] If there is no unprocessed image that has not been subjected to the movement determination process among the images extracted in step S30, the control unit 22 proceeds to the process of step S54. In step S54, the control unit 22 performs a smoothing process to remove noise from the result of the determination in step S50, and then stores the result of the movement determination process in a storage device such as the HDD 508.

[0106] The smoothing process in step S54 is a process that uses, for example, a moving average of the results of the judgment in step S50, and changes an image judged to be other than moving to an image of moving when a section of consecutive images (frames) judged to be moving contains only one image (frame) judged to be other than moving.

[0107] 6, the analysis worker performs annotation work by operating the analysis worker terminal 30. The analysis worker terminal 30 accepts a request for annotation work from the analysis worker and transmits the request to the information processing device 20. The analysis worker terminal 30 receives a response to the request from the information processing device 20 and displays the response to the annotation work.

[0108] By using the result of the movement determination process in step S16, the information processing device 20 can simplify the annotation work of adding information to the video of the first-person viewpoint of the field worker as follows.

[0109] Figure 8 shows an example of an annotation tool that simplifies the annotation work performed by analysts. Figure 8(A) shows a timeline image in which images included in a video taken from the first-person perspective of a field worker are extracted every minute and the extracted images are displayed as thumbnails every minute.

[0110] By using the results of the movement determination process in step S16, it can be determined whether the images contained in the video taken from the first-person perspective of the field worker shown in Figure 8(A) were taken while moving or while not moving.

[0111] Therefore, the annotation tool according to this embodiment draws black shading on the timeline image shown in Fig. 8(A) during the time periods when scenes of the field worker moving around are filmed, as shown in Fig. 8(B). According to Fig. 8(B), the analyst can easily determine the time periods during which scenes of the field worker moving around are filmed from the first-person viewpoint video of the field worker shown in Fig. 8(A).

[0112] For example, analysts often add information to non-movement scenes (e.g., work scenes) rather than to movement scenes, and there is a need to reduce the effort of checking movement scenes as much as possible. The display of the annotation tool according to this embodiment makes it easy to determine the time period when the movement scenes of field workers are being filmed, thereby reducing the man-hours required for annotation work by analysts.

[0113] In addition, the annotation tool according to this embodiment utilizes a label indicating that the worker is moving, which is set for an image of a movement determination target that has been determined to be moving, making it easy to create a digest video from a first-person perspective video of the worker that removes the time period in which the worker is moving.

[0114] 9 is a sequence diagram showing an example of a processing procedure of the information processing system 1 according to this embodiment. FIG. 9 shows an example in which the movement determination processing according to this embodiment is performed by the wearable terminal 10.

[0115] In step S70, the control unit 12 of the wearable terminal 10 carried or worn by the field worker captures a video from the first-person perspective of the field worker by controlling the imaging unit 160. In step S72, the control unit 12 of the wearable terminal 10 stores the video in a storage device such as the memory unit 181.

[0116] In step S74, the control unit 12 of the wearable device 10 performs, for example, the movement determination process shown in Fig. 7. In step S30, the control unit 12 extracts images every second from the video captured from the first-person perspective of the field worker. The control unit 12 extracts images every second from the video captured from the first-person perspective of the field worker by the process of step S30. Note that every second is just an example, and any predetermined time interval may be used.

[0117] In step S32, control unit 12 selects one image for movement determination from the images extracted in step S30. In step S34, control unit 12 selects an image before the image for movement determination selected in step S32 and an image after the image for movement determination selected in step S32.

[0118] The control unit 12 cuts out, for example, the peripheral area 1002 shown in FIG. 5 from the image to be subjected to movement determination, the previous image, and the subsequent image. In step S36, the control unit 12 determines whether or not processing has been completed for all areas cut out in step S34. If processing has not been completed for all areas cut out in step S34, the control unit 12 proceeds to processing in step S38. In step S38, the control unit 12 reads out the image to be subjected to movement determination, the previous image, and the subsequent image that have been cut out for one unprocessed area cut out in step S34.

[0119] In step S40, the control unit 12 grayscales one peripheral area 1002 read out in step S38 out of the peripheral areas 1002 of the image for which movement is to be determined, the previous image, and the next image.

[0120] In step S42, the control unit 12 creates a first difference image between the image of the peripheral area 1002 that is the target of movement determination and the previous image, which has been grayscaled in step S40. The control unit 22 also creates a second difference image between the image of the peripheral area 1002 that is the target of movement determination and the subsequent image, which has been grayscaled in step S40.

[0121] In step S44, the control unit 12 calculates the logical product of the first difference image and the second difference image, and creates a logical product image of the two difference images. In step S46, the control unit 12 performs binarization processing on the logical product image, and creates a mask image. In step S48, the control unit 12 calculates the proportion of white pixels contained in the mask image created in step S46. The processes of steps S36 to S48 are repeated until it is determined that processing has been completed for all of the four corner areas cut out in step S34.

[0122] If the processing for all the regions extracted in step S34 has been completed, the control unit 12 proceeds to the processing of step S50. In step S50, the control unit 12 determines whether the image to be subjected to movement determination is moving or not moving, based on the number or proportion of peripheral regions 1002 in which the proportion of white is equal to or greater than the threshold used to determine the changing region.

[0123] In step S52, the control unit 12 determines whether or not there is an unprocessed image for which movement determination processing has not been performed among the images extracted in step S30. If there is an unprocessed image for which movement determination processing has not been performed among the images extracted in step S30, the control unit 12 returns to the processing of step S32 and continues the processing.

[0124] If there is no unprocessed image that has not been subjected to the movement determination process among the images extracted in step S30, the control unit 12 proceeds to the process of step S54. In step S54, the control unit 12 performs a smoothing process to remove noise from the result of the determination in step S50, and then stores the result of the movement determination process in a storage device such as the storage unit 181.

[0125] In step S76 of FIG. 9, the control unit 12 transmits to the information processing device 20 the video from the first-person perspective of the site worker captured in step S10 and the result of the movement determination process in step S74.

[0126] In step S78, the control unit 22 of the information processing device 20 stores the video received from the wearable terminal 10 and the result of the movement determination process in step S74 in a storage device such as the HDD 508.

[0127] 9, the analysis worker performs annotation work by operating the analysis worker terminal 30. The analysis worker terminal 30 accepts a request for annotation work from the analysis worker and transmits the request to the information processing device 20. The analysis worker terminal 30 receives a response to the request from the information processing device 20 and displays the response to the annotation work.

[0128] By using the result of the movement determination process in step S74, the information processing device 20 can simplify the annotation work of adding information to the video from the first-person perspective of the field worker, as described above.

[0129] As described above, the information processing system 1 according to this embodiment can provide an image processing method, an information processing device 20, and a wearable terminal 10 that determine images taken while a field worker is moving or not moving from videos taken by the wearable terminal 10 carried or worn by the field worker.

[0130] [Effect] This embodiment is an image processing method executed by an information processing device 20 or a wearable terminal 10 having a control unit 22, in which the control unit 22 extracts an image 1000 from a video captured by the wearable terminal 10 carried by a field worker, detects a changed area from a peripheral area 1002 within the image, and determines, based on the changed area, whether the extracted image was captured while the field worker was moving or was captured at a time other than when the field worker was moving.

[0131] In this embodiment, a change area is detected from a peripheral area 1002 of an image 1000 included in a video captured by a wearable terminal 10 carried by a field worker, and it is determined based on the change area whether the image 1000 was captured while the field worker was moving or whether it was captured while the field worker was not moving.

[0132] As described above, according to this embodiment, it is possible to determine whether an image was taken while the field worker was moving or not from images included in a video captured by the wearable terminal 10 carried by the field worker.

[0133] In this embodiment, the peripheral area 1002 in the image is the area 1002 in the four corners of the image 1000, the area 1002 in the center of the top, bottom, left, and right sides of the image 1000, or the entire peripheral area 1002 in the image 1000.

[0134] According to this embodiment, it is possible to determine whether an image contained in a video captured by a wearable terminal 10 carried by a field worker was captured while the field worker was moving or while not moving by using the areas 1002 at the four corners of the image 1000, the areas 1002 at the center of the top, bottom, left, and right sides of the image 1000, or the entire peripheral area 1002 of the image 1000.

[0135] In this embodiment, the control unit 12 or 22 detects a changed area based on the inter-frame difference of a peripheral area 1002 in an image 1000 extracted from a moving image at predetermined time intervals.

[0136] According to this embodiment, even if, for example, a field worker's luggage or tools are captured prominently in the center of image 1000, it is possible to detect a changed area based on the inter-frame difference in peripheral area 1002 of image 1000, and therefore it is possible to accurately determine whether the image was taken while the field worker was moving or while the field worker was not moving.

[0137] In addition, in this embodiment, the control unit 12 or 22 calculates the inter-frame difference for each pixel included in the peripheral region 1002 within the image 1000, determines which pixels have changed based on their magnitude relationship with a threshold, determines the proportion of changed pixels included in the peripheral region 1002 within the image 1000, and detects the changed region based on the proportion of changed pixels.

[0138] According to this embodiment, the proportion of changed pixels (e.g., white pixels) contained in the peripheral area 1002 of the image 1000 is determined based on the magnitude relationship with the threshold value used in the binarization process, and the changed area can be detected according to the proportion of changed pixels.

[0139] In this embodiment, the control unit 12 or 22 changes the threshold value depending on the brightness of the captured image.

[0140] According to this embodiment, the brighter the image captured by the wearable terminal 10, the larger the threshold value used for the binarization process, and the darker the image captured, the smaller the threshold value used for the binarization process. This makes it possible to accurately determine whether the image captured by the wearable terminal 10 was captured while the field worker was moving or while the field worker was not moving, regardless of the brightness of the image captured by the wearable terminal 10.

[0141] Furthermore, in this embodiment, the control unit 12 or 22 determines whether the image was taken while the field worker was moving or while the field worker was not moving, based on the number of changed areas or the proportion of changed areas detected from the extracted image 1000.

[0142] According to this embodiment, it is possible to accurately determine whether the image was taken while the field worker was moving or when the field worker was not moving, based on the number of changed areas or the proportion of changed areas detected from the extracted image 1000.

[0143] In this embodiment, the video is a first-person perspective video captured while the wearable device 10 is attached to a field worker.

[0144] According to this embodiment, a change area is detected from a peripheral area 1002 within an image 1000 included in a video captured by a wearable terminal 10 worn by a field worker, and based on the change area, it is determined whether the image was captured while the field worker was moving or while the field worker was not moving.

[0145] As described above, according to this embodiment, it is possible to determine whether an image was taken while the field worker was moving or not from images included in a video captured by the wearable terminal 10 worn by the field worker.

[0146] Furthermore, this embodiment is an information processing device 20 having a control unit 22, which extracts an image 1000 from a video captured by a wearable terminal 10 carried by a field worker, detects a changed area from a peripheral area 1002 within the image 1000, and determines, based on the changed area, whether the extracted image 1000 is an image captured while the field worker is moving or an image captured at a time other than when the field worker is moving.

[0147] In this embodiment, a change area is detected from a peripheral area 1002 within an image 1000 contained in a video captured by a wearable terminal 10 carried by a field worker, and based on the change area, it is determined whether the image was captured while the field worker was moving or while the field worker was not moving.

[0148] As described above, according to this embodiment, it is possible to determine whether an image was taken while the field worker was moving or not from images included in a video captured by the wearable terminal 10 carried by the field worker.

[0149] Furthermore, this embodiment is a wearable terminal 10 that has a control unit 12 and is carried by a field worker, and the control unit 12 captures a video, extracts an image 1000 from the captured video, detects a changed area from a peripheral area 1002 within the image 1000, and determines based on the changed area whether the image was captured while the field worker was moving or while the field worker was not moving.

[0150] In this embodiment, a change area is detected from a peripheral area 1002 within an image 1000 included in a video captured by a wearable terminal 10 carried by a field worker, and it is determined based on the change area whether the extracted image 1000 was captured while the field worker was moving or whether it was captured while the field worker was not moving.

[0151] As described above, according to this embodiment, it is possible to determine whether an image was taken while the field worker was moving or not from images included in a video captured by the wearable terminal 10 carried by the field worker.

[0152] Although the present embodiment has been described above, it will be understood that various changes in form and details can be made without departing from the spirit and scope of the claims. [Explanation of symbols]

[0153] 1. Information Processing Systems 2 On-site 10 Wearable devices 12 Control Unit 20 Information processing equipment 22 Control Unit 30 Analyst terminal 50 Network 160 Imaging unit 1000 images 1002 Peripheral area of ​​the image

Claims

1. An image processing method executed by an information processing device or a wearable device having a control unit, The control unit Extracting images from a video captured by the wearable device carried by a field worker; detecting a change area from a peripheral area within the image; It is determined whether the extracted image is an image taken while the field worker is moving or an image taken while the field worker is not moving, based on the changed area. Image processing methods.

2. The peripheral area of ​​the image is the area of ​​the four corners of the image, the area in the center of the top, bottom, left, and right sides of the image, or the entire peripheral area of ​​the image. The image processing method according to claim 1.

3. The control unit The change area is detected based on inter-frame differences in a peripheral area in the image extracted from the moving image at predetermined time intervals.

3. The image processing method according to claim 1.

4. The control unit calculating the inter-frame difference for each pixel included in a peripheral region of the image; determining the pixels that have changed based on a magnitude relationship with a threshold value, and determining the proportion of the pixels that have changed that are included in a peripheral region within the image; The changed area is detected according to the ratio of the changed pixels.

4. The image processing method according to claim 3.

5. The control unit changes the threshold value depending on the brightness of the captured image.

5. The image processing method according to claim 4.

6. The control unit determines whether the image was taken while the field worker was moving or while the field worker was not moving, based on the number of the changed areas or the ratio of the changed areas detected from the extracted image.

3. The image processing method according to claim 1.

7. The video is a first-person perspective video taken while the wearable device is worn by the field worker.

3. The image processing method according to claim 1.

8. An information processing device having a control unit, The control unit Images are extracted from videos taken by wearable devices carried by field workers, detecting a change area from a peripheral area within the image; It is determined whether the extracted image is an image taken while the field worker is moving or an image taken while the field worker is not moving, based on the changed area. Information processing device.

9. A wearable terminal having a control unit and carried by a field worker, The control unit Shoot a video, Extract images from the video you have taken, detecting a change area from a peripheral area within the image; It is determined whether the extracted image is an image taken while the field worker is moving or an image taken while the field worker is not moving, based on the changed area. Wearable device.

Images