Image processing method, information processing apparatus, and wearable terminal

Abstract

Provided are an image processing method, an information processing apparatus, and a wearable terminal for determining, from a video captured by a wearable terminal carried by a field worker, an image captured while the field worker is moving or while the field worker is not moving. A solution is an image processing method executed by an information processing apparatus or a wearable terminal having a control unit. The control unit extracts an image from a video captured by a wearable terminal carried by a field worker, detects a change region from a region of a peripheral portion in the image, and determines, based on the change region, whether the extracted image is an image captured while the field worker is moving or an image captured while the field worker is not moving.

Description

Technical Field

[0001] The present disclosure relates to an image processing method, an information processing apparatus, and a wearable terminal.

Background Art

[0002] For example, in an annotation operation of attaching information to a video, an annotation tool is used. An analyst such as an annotator views the video, determines the content being photographed, and attaches information such as a label to the images included in the video.

[0003] Techniques for automatically extracting scenes in which the face of a subject is clearly photographed or scenes in which the movement of a subject is large from a video have been conventionally known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Videos taken by a wearable terminal carried by a field worker may include images taken while the field worker is moving. In the annotation operation, it has been troublesome to determine whether the images were taken while the field worker was moving or not from the videos taken by the wearable terminal. Note that Patent Document 1 does not describe determining whether the images were taken while the field worker was moving or not from the videos taken by the wearable terminal.

[0006] An object of the present disclosure is to provide an image processing method, an information processing apparatus, and a wearable terminal that determine whether an image was taken while a field worker was moving or not from a video taken by a wearable terminal carried by the field worker.

Means for Solving the Problem

[0007] A first aspect of the present disclosure is an image processing method executed by an information processing apparatus or a wearable terminal having a control unit. The control unit extracts an image from a video captured by the wearable terminal carried by a field worker, detects a change region from a region of a peripheral portion in the image, and determines whether the extracted image is an image captured during the movement of the field worker or an image captured other than during the movement of the field worker based on the change region.

[0008] According to the first aspect of the present disclosure, it is possible to provide an image processing method for determining an image captured during or other than the movement of a field worker from a video captured by a wearable terminal carried by the field worker.

[0009] A second aspect of the present disclosure is the image processing method of the first aspect, wherein the region of the peripheral portion in the image is a region at the four corners in the image, a region at the center of the upper, lower, left, and right sides in the image, or the entire region of the peripheral portion in the image.

[0010] According to the second aspect of the present disclosure, a change region can be detected from a region at the four corners in the image, a region at the center of the upper, lower, left, and right sides in the image, or the entire region of the peripheral portion in the image.

[0011] A third aspect of the present disclosure is the image processing method of the first aspect or the second aspect, wherein the control unit detects the change region based on the inter-frame difference of the region of the peripheral portion in the image extracted from the video at a predetermined time interval.

[0012] According to the third aspect of the present disclosure, a change region can be detected based on the inter-frame difference of the region of the peripheral portion in the image.

[0013] A fourth aspect of the present disclosure is the image processing method according to the third aspect, wherein the control unit calculates the inter-frame difference for each pixel included in the area of the peripheral portion in the image, determines pixels that are changing based on the magnitude relationship with a threshold value, determines the ratio of the changing pixels included in the area of the peripheral portion in the image, and detects the change area according to the ratio of the changing pixels.

[0014] According to the fourth aspect of the present disclosure, the inter-frame difference can be calculated for each pixel included in the area of the peripheral portion in the image, pixels that are changing can be determined based on the magnitude relationship with a threshold value, the ratio of the changing pixels included in the area of the peripheral portion in the image can be determined, and the change area can be detected according to the ratio of the changing pixels.

[0015] A fifth aspect of the present disclosure is the image processing method according to the fourth aspect, wherein the control unit changes the threshold value according to the brightness of the captured image.

[0016] According to the fifth aspect of the present disclosure, pixels that are changing can be determined based on the magnitude relationship with the threshold value changed according to the brightness of the captured image, the ratio of the changing pixels included in the area of the peripheral portion in the image can be determined, and the change area can be detected according to the ratio of the changing pixels.

[0017] A sixth aspect of the present disclosure is the image processing method according to any one of the first to fifth aspects, wherein the control unit determines whether the image is captured during the movement of the on-site worker or the image is captured other than during the movement of the on-site worker based on the number or ratio of the change areas detected from the extracted image.

[0018] According to the sixth aspect of the present disclosure, it can be determined whether the image is captured during the movement of the on-site worker or the image is captured other than during the movement of the on-site worker based on the number or ratio of the change areas detected from the extracted image.

[0019] The seventh aspect of the present disclosure is the image processing method according to any one of the first to sixth aspects, wherein the video is a first-person perspective video taken with the wearable terminal worn on the on-site worker.

[0020] According to the seventh aspect of the present disclosure, it is possible to provide an image processing method for determining an image taken during or outside the movement of the on-site worker from a first-person perspective video taken with the wearable terminal worn on the on-site worker.

[0021] The eighth aspect of the present disclosure is an information processing apparatus having a control unit, wherein the control unit extracts an image from a video taken by a wearable terminal carried by an on-site worker, detects a change region from a region of the peripheral portion in the image, and determines based on the change region whether the extracted image is an image taken during the movement of the on-site worker or an image taken outside the movement of the on-site worker.

[0022] According to the eighth aspect of the present disclosure, it is possible to provide an information processing apparatus for determining an image taken during or outside the movement of the on-site worker from a video taken by a wearable terminal carried by the on-site worker.

[0023] The ninth aspect of the present disclosure is a wearable terminal having a control unit and carried by an on-site worker, wherein the control unit takes a video, extracts an image from the taken video, detects a change region from a region of the peripheral portion in the image, and determines based on the change region whether the extracted image is an image taken during the movement of the on-site worker or an image taken outside the movement of the on-site worker.

[0024] According to the ninth aspect of the present disclosure, it is possible to provide a wearable terminal for determining an image taken during or outside the movement of the on-site worker from a video taken by a wearable terminal carried by the on-site worker.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0026] Next, embodiments of the present disclosure will be described in detail.

[0027] [Embodiment] [System Configuration]< FIG. 1 is a configuration diagram of an example of the information processing system 1 according to the present embodiment. The information processing system 1 shown in FIG. 1 includes a wearable terminal 10, an information processing device 20, and an analysis operator terminal 30.

[0028] The wearable terminal 10 is a device carried or worn by a field worker who performs field work at the site 2. The wearable terminal 10 carried or worn by the field worker can capture a video from the first-person perspective of the field worker. The video from the first-person perspective of the field worker may be a video captured in the line-of-sight direction of the field worker or a video captured on the front side of the field worker.

[0029] For example, the wearable terminal 10 is attached to any location such as the arm, eye area, ear area, neck area of the on-site worker, or the clothes worn by the on-site worker. In this embodiment, an example of a neck-mounted wearable terminal 10 that is attached to the neck area of the on-site worker and captures a video from the first-person perspective of the on-site worker will be described.

[0030] The wearable terminal 10 is communicably connected to the information processing device 20 via the network 50. The wearable terminal 10 may be communicably connected to the information processing device 20 via the network 50 by using the communication function of a mobile terminal such as the on-site worker's smartphone. The network 50 is, for example, the Internet. The network 50 may be a LAN (Local Area Network) or a dedicated communication line, etc.

[0031] The wearable terminal 10 has a control unit 12. The control unit 12 is a hardware configuration that executes a program, and is a CPU (Central Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array), etc. The wearable terminal 10 can execute various processes described later when a CPU, which is an example of the control unit 12, executes a program.

[0032] The information processing device 20 is communicably connected to the wearable terminal 10 and the analysis worker's terminal 30 via the network 50. The information processing device 20 receives video data (hereinafter simply referred to as video) captured by the wearable terminal 10.

[0033] The information processing device 20 determines an image captured while the on-site worker is moving or other than while moving, as will be described later, from the video captured by the wearable terminal 10. For example, the information processing device 20 uses the result of determining an image captured while the on-site worker is moving or other than while moving in the processing of an annotation operation as will be described later.

[0034] The information processing apparatus 20 also includes a control unit 22. The control unit 22 is a hardware configuration that executes a program, such as a CPU, an ASIC, or an FPGA. For example, in the information processing apparatus 20, by executing a program with a CPU as an example of the control unit 22, various processes described later can be executed.

[0035] The terminal 30 for the analysis operator is a device operated by an analysis operator such as an annotator. The terminal 30 for the analysis operator is equipped with various tools used by the analysis operator, such as an annotation tool.

[0036] The analysis operator performs an annotation operation by operating the terminal 30 for the analysis operator. The terminal 30 for the analysis operator receives a request for an annotation operation by the analysis operator and transmits the request to the information processing apparatus 20. The terminal 30 for the analysis operator receives a response to the request from the information processing apparatus 20 and displays the response to the annotation operation.

[0037] The terminal 30 for the analysis operator is a hardware configuration that executes a program and has a control unit such as a CPU, an ASIC, or an FPGA. In the terminal 30 for the analysis operator, by executing a program with a CPU as an example of the control unit, various processes described later can be executed.

[0038] The information processing apparatus 20 is, for example, a PC (Personal Computer) or a workstation. The information processing apparatus 20 may also be realized by using an ASP (Application Service Provider) or cloud computing. The terminal 30 for the analysis operator is a PC, a smartphone, a tablet terminal, or the like.

[0039] The configuration of the information processing system 1 in FIG. 1 is an example. For example, the information processing apparatus 20 may be composed of one or more units. Needless to say, there are various system configuration examples for the configuration of the information processing system 1 according to the application and purpose.

[0040] <Device Configuration> The information processing apparatus 20 in FIG. 1 is realized by, for example, a computer 500 having the hardware configuration shown in FIG. 2. Further, the operator terminal 30 in FIG. 1 may be realized by the computer 500 having the hardware configuration shown in FIG. 2.

[0041] FIG. 2 is a hardware configuration diagram of an example of the computer 500 according to the present embodiment. The computer 500 includes an input device 501, a display device 502, an external I / F 503, a RAM (Random Access Memory) 504, a ROM (Read Only Memory) 505, a CPU 506, a communication I / F 507, and an HDD (Hard Disk Drive) 508, etc., and each is mutually connected by a bus B. Note that the input device 501 and the display device 502 may be connected and used when necessary.

[0042] The input device 501 is a touch panel, operation keys, buttons, keyboard, mouse, etc. used for operations. The display device 502 is composed of a display for displaying a screen, a speaker for outputting sound, etc.

[0043] The communication I / F 507 is an interface for the computer 500 to perform data communication via the network 50. The HDD 508 is an example of a non-volatile storage device that stores programs and data. Programs and data include an OS (Operating System), which is basic software for controlling the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use an SSD (Solid State Drive) instead of the HDD 508.

[0044] The external I / F 503 is an interface with an external device. Examples of the external device include a recording medium 503a. The computer 500 reads from and writes to the recording medium 503a via the external I / F 503.

[0045] The recording medium 503a is a flexible disk, CD (Compact Disc), DVD (Digital Versatile Disc), SD (Secure Digital) memory card, USB (Universal Serial Bus) memory, or the like.

[0046] The ROM 505 is an example of a non-volatile semiconductor memory (storage device) that can hold programs and data even when the power is turned off. The ROM 505 stores programs and data such as the BIOS (Basic Input Output System), OS settings, and network settings that are executed when the computer 500 is started up. The RAM 504 is an example of a volatile semiconductor memory (storage device) that temporarily holds programs and data.

[0047] The CPU 506 is an arithmetic unit that realizes the control and functions of the entire computer 500 by reading programs and data from a storage device such as the ROM 505 or HDD 508 onto the RAM 504 and executing processing. The CPU 506 is an example of the control unit 22 of the information processing device 20 and the control unit of the analysis.

[0048] The wearable terminal 10 in FIG. 1 is realized by the hardware configuration shown in FIGS. 3 and 4 in the case of, for example, a neck-hanging type wearable terminal 10. FIG. 3 is an external view of an example of the wearable terminal 10 according to the present embodiment. FIG. 4 is a functional configuration diagram of an example of the wearable terminal 10 according to the present embodiment.

[0049] The housing constituting the wearable terminal 10 includes a left arm part 110, a right arm part 120, and a main body part 130. The left arm part 110 and the right arm part 120 extend forward from the left end and the right end of the main body part 130, respectively. The wearable terminal 10 has a structure that forms a substantially U shape as a whole when viewed in plan. For example, a field worker who is an example of a wearer, when wearing the wearable terminal 10, contacts the main body part 130 behind the neck and hangs the left arm part 110 and the right arm part 120 from the sides of the neck toward the chest side to hook around the neck.

[0050] The left wrist part 110 and the right wrist part 120 are each provided with a plurality of sound collecting parts 141 to 145. The sound collecting parts 141 to 145 are, for example, microphones. The sound collecting parts 141 to 145 are arranged mainly for the purpose of acquiring the voices of the wearer and the interlocutor. The sound collecting part 141 and the sound collecting part 142 are provided on the left wrist part 110. The sound collecting part 143 and the sound collecting part 144 are provided on the right wrist part 120. The left wrist part 110 and the right wrist part 120 may be additionally provided with one or a plurality of sound collecting parts.

[0051] In the example of FIG. 3, in addition to the sound collecting part 141 and the sound collecting part 142, the sound collecting part 145 is provided on the left wrist part 110. The sound signals acquired by the sound collecting parts 141 to 145 are transmitted to the control part 12 provided in the main body part 130, and predetermined processing is performed.

[0052] The left wrist part 110 is further provided with an imaging part 160. The imaging part 160 is provided on the front end surface 112 of the left wrist part 110. The imaging part 160 captures moving images and still images on the front side of the on-site worker. The moving images and still images captured by the imaging part 160 are transmitted to the control part 12 in the main body part 130. The moving images and still images captured by the imaging part 160 are stored, for example, in the storage part 181. Further, the wearable terminal 10 may transmit the moving images and still images captured by the imaging part 160 to, for example, the information processing device 20 for storage.

[0053] The right wrist part 120 is further provided with a non-contact type sensor part 170. The sensor part 170 is arranged on the front end surface 122 of the right wrist part 120, for example, to detect the movement of the hand of the on-site worker. The detection information of the sensor part 170 is used for the control of the imaging part 160, such as the activation of the imaging part 160, the start of shooting, and the stop of shooting.

[0054] For example, the sensor unit 170 may detect the proximity of the hand of a field worker or the like and control the imaging unit 160, or may detect a predetermined gesture made by the field worker within the detection range of the sensor unit 170 and control the imaging unit 160. In the present embodiment, the imaging unit 160 is arranged on the front end surface 112 of the left arm unit 110, and the sensor unit 170 is arranged on the front end surface 122 of the right arm unit 120. However, the positions of the imaging unit 160 and the sensor unit 170 may be swapped.

[0055] Further, the wearable terminal 10 may use the detection information of the sensor unit 170 to activate at least one of the imaging unit 160, the sound collection units 141 to 145, and the control unit 12. For example, when the sensor unit 170, the sound collection units 141 to 145, and the control unit 12 are always activated and the imaging unit 160 is stopped, the wearable terminal 10 may activate the imaging unit 160 when the sensor unit 170 detects a specific gesture. Also, for example, when the sensor unit 170, the sound collection units 141 to 145, and the control unit 12 are always activated and the imaging unit 160 is stopped, the wearable terminal 10 may activate the imaging unit 160 when the sound collection units 141 to 145 detect a specific sound.

[0056] As shown in FIG. 4, the left arm unit 110 is provided with the sound collection unit 141, the sound collection unit 142, the sound collection unit 145, the operation unit 150, and the imaging unit 160. The right arm unit 120 is provided with the sound collection unit 143, the sound collection unit 144, and the sensor unit 170. The main body unit 130 is provided with the control unit 12, the storage unit 181, the communication unit 182, the proximity sensor 183, the sound playback unit 184, and the battery 190. In addition to the functional configuration shown in FIG. 4, the wearable terminal 10 may be appropriately equipped with sensors such as a gyro sensor, an acceleration sensor, a geomagnetic sensor, or a GPS sensor.

[0057] The sound collection units 141 to 145 can use known microphones such as dynamic microphones, condenser microphones, and MEMS (Micro-Electrical-Mechanical Systems) microphones. The sound collection units 141 to 145 convert sound into an electrical signal, convert the electrical signal into digital information by an A / D conversion circuit, and transmit it to the control unit 12.

[0058] The operation unit 150 receives input of operations by on-site workers. The operation unit 150 can employ, for example, a known switch circuit or a touch panel. The operation unit 150 receives, for example, operations instructing turning on or off of the power supply and operations necessary for realizing the functions of the wearable terminal 10 from on-site workers. Information input via the operation unit 150 is transmitted to the control unit 12.

[0059] The imaging unit 160 captures a moving image or a still image. The imaging unit 160 may employ, for example, a general digital camera. The moving image or still image acquired by the imaging unit 160 is transmitted to the control unit 12. The moving image or still image acquired by the imaging unit 160 is stored in the storage unit 181, and processes such as the movement determination process described later are performed. Further, the moving image or still image acquired by the imaging unit 160 may be transmitted to the information processing device 20, stored in the information processing device 20, and processes such as the movement determination process described later may be performed. Also, since the imaging unit 160 generally consumes a large amount of power, it is preferable that it automatically enters a sleep state after a certain period of time has elapsed after the shooting stops.

[0060] The sensor unit 170 is a non-contact detection device for detecting the movement of the fingers of on-site workers or the like. The sensor unit 170 is, for example, a proximity sensor or a gesture sensor. The proximity sensor detects, for example, that the finger of an on-site worker has approached within a predetermined range. The proximity sensor uses a known one such as an optical type, an ultrasonic type, a magnetic type, a capacitance type, or a temperature-sensitive type.

[0061] The gesture sensor detects, for example, the movements and shapes of the fingers of on-site workers. The gesture sensor is, for example, an optical sensor that irradiates light from an infrared-emitting LED toward an object and captures changes in the reflected light with a light-receiving element to detect, for example, the movements and shapes of the fingers of on-site workers. The detection information by the sensor unit 170 is transmitted to the control unit 12 and used for controlling the imaging unit 160 and the like.

[0062] Since the sensor unit 170 generally has low power consumption, it is preferably always activated while the power of the wearable terminal 10 is ON. Also, the sensor unit 170 may be activated when the proximity sensor 183 detects the wearing of the wearable terminal 10.

[0063] The control unit 12 performs arithmetic processing for controlling the wearable terminal 10. The control unit 12 can utilize a processor such as a CPU. The control unit 12 reads out the program stored in the storage unit 181 and executes predetermined arithmetic processing according to the program. Also, the control unit 12 writes and reads out the results of the arithmetic processing according to the program to and from the storage unit 181.

[0064] The storage unit 181 stores information used in arithmetic processing and the like in the control unit 12 and the results of the arithmetic processing. The storage function of the storage unit 181 can be realized by a non-volatile memory such as an HDD and an SDD, for example. Also, the storage unit 181 may have a function as a memory for writing or reading out the progress of the arithmetic processing by the control unit 12. The memory function of the storage unit 181 can be realized by a volatile memory such as a RAM and a DRAM.

[0065] The communication unit 182 can adopt, for example, a communication module for wireless communication according to a known mobile communication standard such as 3G (W-CDMA), 4G (LTE / LTE-Advanced), or 5G. The communication unit 182 may adopt a communication module for wireless communication according to a wireless LAN method such as Wi-Fi (registered trademark). Also, the communication unit 182 may adopt a communication module for proximity wireless communication according to a method such as Bluetooth (registered trademark) or NFC.

[0066] The proximity sensor 183 is disposed inside the main body 130 and detects that the head of the on-site worker has approached within a predetermined range. The sound emitting unit 184 converts an electrical signal into sound. For example, the sound emitting unit 184 is a general speaker that transmits sound to the on-site worker by air vibration. Further, the sound emitting unit 184 may be a bone conduction speaker that transmits sound to the on-site worker by vibrating the bones of the on-site worker.

[0067] The battery 190 is a battery that supplies power to various electronic components included in the wearable terminal 10. As the battery 190, a rechargeable battery such as a lithium ion battery, a lithium polymer battery, an alkaline storage battery, a nickel cadmium battery, a nickel metal hydride battery, or a lead storage battery is used.

[0068] <Process> In this embodiment, an on-site worker who performs on-site work at site 2 carries or wears the wearable terminal 10 and shoots a video of the on-site work. The video of the on-site work shot by the wearable terminal 10 is a video from the first-person perspective of the on-site worker.

[0069] The video from the first-person perspective of the on-site worker shot by the wearable terminal 10 may include a moving scene where the on-site worker moves back and forth with luggage and tools. For example, in a video of air conditioner repair from the first-person perspective of the on-site worker, there may be a moving scene where the on-site worker moves back and forth between the indoor unit and the outdoor unit of the air conditioner while carrying luggage and tools. The moving scene is an image captured during the movement of the on-site worker among the videos captured by the wearable terminal 10.

[0070] When an operator analyzes a first-person video captured by the wearable terminal 10, the operator may perform annotation work. When performing annotation work on the first-person video captured by the wearable terminal 10, images other than those of the moving scene are often more important than images of the moving scene. This is because, in the annotation work for the video of on-site work, information such as work content is often added to images other than those of the moving scene, and information is not added to images of the moving scene.

[0071] In this embodiment, the following process is provided to determine whether an image included in the video captured by the wearable terminal 10 is an image captured while the on-site worker is moving or an image captured while the on-site worker is not moving.

[0072] As a method for determining whether an image is captured while the on-site worker is moving or an image is captured while the on-site worker is not moving, there is a method of taking the difference between frames and making a determination based on the difference. However, the first-person video captured by the wearable terminal 10 of the on-site worker is often close to the object being photographed during on-site work, and the accuracy of determining whether an image is captured while the on-site worker is moving or an image is captured while the on-site worker is not moving may deteriorate as follows.

[0073] When the first-person video captured by the wearable terminal 10 of the on-site worker is a video of air conditioner repair, the distance between the on-site worker and the object being photographed becomes close, and the object being photographed appears large in the image captured by the wearable terminal 10. In addition, during air conditioner repair, the on-site worker often faces the object being photographed directly, and there is a high possibility that the object being photographed appears in the central area of the image captured by the wearable terminal 10. Therefore, in the case of the first-person video captured by the wearable terminal 10 of the on-site worker, if the difference between frames is taken for the entire screen, even a slight change in work is likely to be determined as a large (moved) change.

[0074] In addition, when a field worker is moving while carrying a load or a tool, there is a high possibility that the load or the tool will be prominently shown in the central area of the image captured by the wearable terminal 10. In this case, even if the difference between frames is calculated, it is highly likely that the change will be determined to be small (not moving). Furthermore, when a field worker stands or sits, if the difference between frames is calculated, it is highly likely that the change will be determined to be large (moving).

[0075] Therefore, in the present embodiment, for the images included in the video captured by the wearable terminal 10, the following movement determination process is performed to accurately determine whether the image was captured during the movement of the field worker or during a period other than the movement of the field worker.

[0076] FIG. 5 is an explanatory diagram of an example for explaining the movement determination process according to the present embodiment. In the movement determination process according to the present embodiment, an image 1000 is extracted from a first-person perspective video of a field worker captured by the wearable terminal 10, and a change area is detected from an area 1002 in the peripheral part of the image 1000.

[0077] The shape of the area 1002 in the peripheral part of the image 1000 for detecting the change area is various. For example, FIG. 5(A) shows an example in which the area 1002 in the peripheral part of the image 1000 is the areas at the four corners of the image 1000. FIG. 5(B) shows an example in which the area 1002 in the peripheral part of the image 1000 is the areas at the centers of the upper, lower, left, and right sides of the image 1000. FIG. 5(C) shows an example in which the area 1002 in the peripheral part of the image 1000 is the entire area in the peripheral part of the image 1000.

[0078] The area 1002 in the peripheral part of the image 1000 may be set by setting the central part of the image 1000 and using the area excluding the central part. The central part of the image 1000 may be set, for example, by dividing the area of the image 1000 such that the image 1000 is divided into five parts vertically and four parts horizontally, and setting, as the central part of the image 1000, the areas that do not touch the outer periphery of the image 1000 among the divided areas.

[0079] In the movement determination process according to this embodiment, a change area is detected from the area 1002 in the peripheral part of the image 1000, and based on the detected change area, it is determined whether the image is taken while the on-site worker is moving or the image is taken when the on-site worker is not moving. For example, in the movement determination process according to this embodiment, based on the number of detected change areas or the ratio of the change areas, it is determined whether the image is taken while the on-site worker is moving or the image is taken when the on-site worker is not moving. The detection of the change area is performed as described below based on the inter-frame difference of the area 1002 in the peripheral part of the image 1000.

[0080] The movement determination process according to this embodiment may be performed by the information processing device 20 or may be performed by the wearable terminal 10. Hereinafter, an example in which the movement determination process according to this embodiment is performed by the information processing device 20 and an example in which the movement determination process according to this embodiment is performed by the wearable terminal 10 will be described.

[0081] FIG. 6 is a sequence diagram showing an example of the processing procedure of the information processing system 1 according to this embodiment. FIG. 6 shows an example in which the movement determination process according to this embodiment is performed by the information processing device 20.

[0082] In step S10, the wearable terminal 10 carried or worn by the on-site worker captures a video from the first-person perspective of the on-site worker. The video from the first-person perspective captured by the on-site worker at the site 2 may include a movement scene in which the on-site worker moves at the site 2. It is assumed that the video from the first-person perspective captured by the on-site worker at the site 2 includes images taken while the on-site worker is moving or images taken when the on-site worker is not moving.

[0083] In step S12, the wearable terminal 10 transmits the captured video from the first-person perspective of the on-site worker to the information processing device 20. The process of step S12 may be performed after the wearable terminal 10 finishes capturing, or may be performed during capturing by streaming or the like.

[0084] In step S14, the control unit 22 of the information processing apparatus 20 stores the video received from the wearable terminal 10 in a storage device such as the HDD 508. In step S16, the control unit 22 of the information processing apparatus 20 performs, for example, the movement determination process shown in FIG. 7. FIG. 7 is a flowchart showing an example of the procedure of the movement determination process according to the present embodiment.

[0085] In step S30, the control unit 22 extracts an image every second from the first-person perspective video of the on-site worker taken by the wearable terminal 10. The control unit 22 extracts an image every second from the first-person perspective video of the on-site worker taken by the wearable terminal 10 by the process of step S30. Note that every second is an example, and any predetermined time interval may be used.

[0086] In step S32, the control unit 22 selects one image to be subjected to movement determination from the images extracted in step S30. The image to be subjected to movement determination selected in step S32 is an unprocessed image for which movement determination processing has not been performed.

[0087] In step S34, the control unit 22 selects the image before the image to be subjected to movement determination selected in step S32 (hereinafter referred to as the previous image) and the image after the image to be subjected to movement determination selected in step S32 (hereinafter referred to as the subsequent image). The previous image is the image one before the image to be subjected to movement determination among the temporally consecutive images extracted in step S30. The subsequent image is the image one after the image to be subjected to movement determination among the temporally consecutive images extracted in step S30.

[0088] The control unit 22 cuts out, for example, the peripheral region 1002 shown in FIG. 5 from the image to be subjected to movement determination, the previous image, and the subsequent image. Here, an example of the peripheral region 1002 shown in FIG. 5(A) will be described. The control unit 22 cuts out the regions at the four corners as shown in FIG. 5(A), for example, from the image to be subjected to movement determination, the previous image, and the subsequent image.

[0089] In step S36, the control unit 22 determines whether the processing for all the regions of the four corner regions cut out in step S34 has been completed. If the processing for all the regions of the four corner regions cut out in step S34 has not been completed, in other words, if there are unprocessed regions remaining in the four corner regions cut out in step S34, the control unit 22 proceeds to the processing of step S38.

[0090] In step S38, the control unit 22 reads out the cut-out image of the movement determination target, the previous image, and the subsequent image for one unprocessed region of the four corner regions cut out in step S34. For example, if the region 1002 of the upper left peripheral part shown in FIG. 5(A) is unprocessed, the control unit 22 reads out the region 1002 of the upper left peripheral part of the image of the movement determination target, the previous image, and the subsequent image.

[0091] In step S40, the control unit 22 grayscales the region 1002 of one peripheral part (for example, the upper left peripheral part) read out in step S38 among the peripheral part regions 1002 of the image of the movement determination target, the previous image, and the subsequent image.

[0092] In step S42, the control unit 22 creates a difference image (hereinafter referred to as the first difference image) between the image of the movement determination target and the previous image of the region 1002 of one peripheral part grayscaled in step S40. The first difference image is an image obtained by imaging the difference between the pixel values of the image of the movement determination target and the previous image of the region 1002 of one peripheral part grayscaled in step S40. Further, the control unit 22 creates a difference image (hereinafter referred to as the second difference image) between the image of the movement determination target and the subsequent image of the region 1002 of one peripheral part grayscaled in step S40. The second difference image is an image obtained by imaging the difference between the pixel values of the image of the movement determination target and the subsequent image of the region 1002 of one peripheral part grayscaled in step S40.

[0093] In step S44, the control unit 22 calculates the logical product of the first difference image and the second difference image, and creates a logical product image of the two difference images. For example, the logical product of the first difference image and the second difference image is calculated as follows.

[0094] The logical product of the difference images is performed for each pixel value at the same position included in the first difference image and the second difference image. For example, the case where the pixel value of the first difference image is decimal "178" and the pixel value of the second difference image is decimal "159" will be described.

[0095] The control unit 22 converts the pixel value of the first difference image represented by decimal "178" into binary "10110010", and converts the pixel value of the second difference image represented by decimal "159" into binary "10011111".

[0096] The control unit 22 obtains the logical product "10010010" by calculating the logical product bit by bit between binary "10110010" and "10011111". The control unit 22 can obtain the pixel value of the logical product image by converting the calculated logical product "10010010" back to decimal.

[0097] In step S46, the control unit 22 performs binarization processing on the logical product image to create a mask image. The binarization processing performed on the logical product image compares with the threshold value (for example, 50) used for binarization processing for each pixel value of the logical product image created in step S44, changes the pixel value equal to or greater than the threshold value used for binarization processing to "255", and changes the pixel value smaller than the threshold value used for binarization processing to "0".

[0098] By performing the binarization process on the logical AND image, the mask image can display pixels with a difference between the two difference images greater than or equal to the threshold used in the binarization process as white, and pixels with a difference between the two difference images smaller than the threshold used in the binarization process as black. The threshold used in the binarization process may be changed according to the brightness of the image captured by the wearable terminal 10. For example, the threshold used in the binarization process may be changed so that it is larger as the brightness of the image captured by the wearable terminal 10 is brighter, and smaller as the brightness of the captured image is darker.

[0099] In step S48, the control unit 22 calculates the ratio of white pixels included in the mask image created in step S46. The ratio of white pixels calculated in step S48 is an example of the ratio of changing pixels. In this embodiment, for calculation purposes, changing pixels are regarded as white pixels, but changing pixels may also be regarded as black pixels. The processes of steps S36 to S48 are repeated until it is determined that the processing for all the regions of the four corner regions cut out in step S34 has been completed and is executed. In this way, the control unit 22 calculates the ratio of white pixels included in the mask image created in step S46 for each of the four corner regions cut out in step S34.

[0100] Also, if the processing for all the regions of the four corner regions cut out in step S34 has been completed, in other words, if there are no unprocessed regions left in the four corner regions cut out in step S34, the control unit 22 proceeds to the processing of step S50.

[0101] In step S50, the control unit 22 determines whether the image to be subjected to movement determination is in motion or not based on the number or ratio of the peripheral regions 1002 where the ratio of white is equal to or greater than the threshold value used for determining the change region. The peripheral region 1002 where the ratio of white is equal to or greater than the threshold value used for determining the change region is an example of the change region. The control unit 22 may set the result of determining whether the image to be subjected to movement determination is in motion or not as the label of the image to be subjected to movement determination. The control unit 22 may set a label indicating that the image to be subjected to movement determination is in motion for the image to be subjected to movement determination that is determined to be in motion.

[0102] For example, in the case of the peripheral region 1002 in FIG. 5(A), when the peripheral regions 1002 at the four corners cut out in step S34 include two or more change regions where the ratio of white is equal to or greater than the threshold value (for example, two) used for the final determination of whether the change region is in motion or not, the control unit 22 determines that the image to be subjected to movement determination is in motion. Also, in the case of the peripheral region 1002 in FIG. 5(A), when the peripheral regions 1002 at the four corners cut out in step S34 do not include two or more change regions where the ratio of white is equal to or greater than the threshold value (for example, two) used for the final determination of whether the change region is in motion or not, the control unit 22 determines that the image to be subjected to movement determination is not in motion.

[0103] For example, in the case of the peripheral region 1002 in FIG. 5(A), when the peripheral regions 1002 at the four corners cut out in step S34 include a change region where the ratio of white is equal to or greater than 50% of the threshold value used for the final determination of whether the change region is in motion or not, the control unit 22 determines that the image to be subjected to movement determination is in motion. Also, in the case of the peripheral region 1002 in FIG. 5(A), when the peripheral regions 1002 at the four corners cut out in step S34 do not include a change region where the ratio of white is equal to or greater than 50% of the threshold value used for the final determination of whether the change region is in motion or not, the control unit 22 determines that the image to be subjected to movement determination is not in motion.

[0104] In step S52, the control unit 22 determines whether there is an unprocessed image in the image extracted in step S30 for which the movement determination process has not been performed. If there is an unprocessed image in the image extracted in step S30 for which the movement determination process has not been performed, the control unit 22 returns to the process of step S32 and continues the process.

[0105] Also, if there is no unprocessed image in the image extracted in step S30 for which the movement determination process has not been performed, the control unit 22 proceeds to the process of step S54. In step S54, the control unit 22 performs a smoothing process to remove noise from the result of the determination in step S50, and then stores the result of the movement determination process in, for example, the storage device of the HDD 508.

[0106] The smoothing process in step S54 uses, for example, the moving average of the result of the determination in step S50, and when there is only one image (frame) determined to be other than during movement included in a portion where the images (frames) determined to be during movement are continuous, it is a process of changing the image determined to be other than during movement to an image during movement.

[0107] In step S18 of FIG. 6, the analysis operator operates the analysis operator terminal 30 to perform an annotation operation. The analysis operator terminal 30 receives a request for an annotation operation by the analysis operator and transmits the request to the information processing apparatus 20. The analysis operator terminal 30 receives a response to the request from the information processing apparatus 20 and displays the response of the annotation operation.

[0108] By using the result of the movement determination process in step S16, the information processing apparatus 20 can simplify the annotation operation of adding information to the video from the first-person perspective of the on-site worker as follows.

[0109] FIG. 8 is a diagram illustrating an example of the display of an annotation tool for simplifying the annotation operation performed by the analysis operator. FIG. 8(A) is an image of a timeline in which images included in the video from the first-person perspective of the on-site worker are extracted every minute, and the extracted images are displayed as thumbnails every minute.

[0110] The images included in the first-person perspective video of the on-site worker shown in Fig. 8(A) can determine whether they are images taken during movement or images taken other than during movement by using the result of the movement determination process in step S16.

[0111] Therefore, the annotation tool according to the present embodiment draws a black screen overlay on the image of the timeline shown in Fig. 8(A) during the time period when the movement scene of the on-site worker is photographed, as shown in Fig. 8(B) for example. According to Fig. 8(B), the analysis worker can easily determine the time period when the movement scene of the on-site worker is photographed from the first-person perspective video of the on-site worker shown in Fig. 8(A).

[0112] For example, the analysis worker often attaches information to scenes other than the movement scene (for example, the work scene) rather than the movement scene, and there is a need to save the trouble of checking the movement scene as much as possible. According to the display of the annotation tool according to the present embodiment, the time period when the movement scene of the on-site worker is photographed can be easily determined, and the man-hours of the annotation work by the analysis worker can be reduced.

[0113] In addition, the annotation tool according to the present embodiment makes it easy to create a digest video in which the time period when the movement scene of the on-site worker is photographed is deleted from the first-person perspective video of the on-site worker by using the label indicating that it is in motion, which is set for the image of the movement determination target determined to be in motion.

[0114] Fig. 9 is a sequence diagram showing an example of the processing procedure of the information processing system 1 according to the present embodiment. Fig. 9 shows an example in which the movement determination process according to the present embodiment is performed by the wearable terminal 10.

[0115] In step S70, the control unit 12 of the wearable terminal 10 carried or worn by the on-site worker controls the imaging unit 160 to capture a video from the first-person perspective of the on-site worker. In step S72, the control unit 12 of the wearable terminal 10 stores the video in a storage device such as the storage unit 181.

[0116] In step S74, the control unit 12 of the wearable terminal 10 performs, for example, the movement determination process shown in FIG. 7. In step S30, the control unit 12 extracts an image every second from the video of the first-person perspective of the captured on-site worker. The control unit 12 extracts an image every second from the video of the first-person perspective of the captured on-site worker by the process of step S30. Note that extracting an image every second is just an example, and any predetermined time interval is acceptable.

[0117] In step S32, the control unit 12 selects one image to be the movement determination target from the images extracted in step S30. In step S34, the control unit 12 selects the image before the image selected as the movement determination target in step S32 and the image after the image selected as the movement determination target in step S32.

[0118] The control unit 12 cuts out, for example, the peripheral region 1002 shown in FIG. 5 from the image to be the movement determination target, the previous image, and the subsequent image. In step S36, the control unit 12 determines whether the processing for all the regions cut out in step S34 has been completed. If the processing for all the regions cut out in step S34 has not been completed, the control unit 12 proceeds to the process of step S38. In step S38, the control unit 12 reads out the image to be the movement determination target, the previous image, and the subsequent image that have been cut out for one unprocessed region cut out in step S34.

[0119] In step S40, the control unit 12 grayscales one peripheral region 1002 read out in step S38 among the peripheral regions 1002 of the image to be the movement determination target, the previous image, and the subsequent image.

[0120] In step S42, the control unit 12 creates a first difference image between the image of the area 1002 of one peripheral part that was grayscaled in step S40 and the previous image for movement determination. Also, the control unit 22 creates a second difference image between the image of the area 1002 of one peripheral part that was grayscaled in step S40 and the subsequent image for movement determination.

[0121] In step S44, the control unit 12 calculates the logical product of the first difference image and the second difference image to create a logical product image of the two difference images. In step S46, the control unit 12 performs binarization processing on the logical product image to create a mask image. In step S48, the control unit 12 calculates the ratio of white pixels included in the mask image created in step S46. The processing of steps S36 to S48 is repeated until it is determined that the processing for all the areas of the four corner areas cut out in step S34 has been completed, and is executed.

[0122] Also, if the processing for all the areas cut out in step S34 has been completed, the control unit 12 proceeds to the processing of step S50. In step S50, the control unit 12 determines whether the image for movement determination is in motion or not based on the number or ratio of the area 1002 of the peripheral part where the ratio of white is equal to or greater than the threshold value used for determining the change area.

[0123] In step S52, the control unit 12 determines whether there is an unprocessed image in the image extracted in step S30 for which movement determination processing has not been performed. If there is an unprocessed image in the image extracted in step S30 for which movement determination processing has not been performed, the control unit 12 returns to the processing of step S32 and continues the processing.

[0124] Also, if there is no unprocessed image in the image extracted in step S30 for which movement determination processing has not been performed, the control unit 12 proceeds to the processing of step S54. In step S54, after performing smoothing processing to remove noise from the result of the determination in step S50, the control unit 12 stores the result of the movement determination processing, for example, in the storage device of the storage unit 181.

[0125] In step S76 of FIG. 9, the control unit 12 transmits the first-person perspective video of the on-site worker taken in step S10 and the result of the movement determination process in step S74 to the information processing device 20.

[0126] In step S78, the control unit 22 of the information processing device 20 stores the video received from the wearable terminal 10 and the result of the movement determination process in step S74 in a storage device such as the HDD 508.

[0127] In step S80 of FIG. 9, the analysis worker performs an annotation operation by operating the analysis worker terminal 30. The analysis worker terminal 30 receives a request for an annotation operation by the analysis worker and transmits the request to the information processing device 20. The analysis worker terminal 30 receives a response to the request from the information processing device 20 and displays a response to the annotation operation.

[0128] By using the result of the movement determination process in step S74, the information processing device 20 can simplify the annotation operation of adding information to the first-person perspective video of the on-site worker as described above.

[0129] As described above, according to the information processing system 1 according to the present embodiment, it is possible to provide an image processing method, an information processing device 20, and a wearable terminal 10 for determining an image taken during or outside the movement of an on-site worker from a video taken by the wearable terminal 10 carried or worn by the on-site worker.

[0130] [Function] The present embodiment is an image processing method executed by the information processing device 20 or the wearable terminal 10 having the control unit 22. The control unit 22 extracts an image 1000 from a video taken by the wearable terminal 10 carried by the on-site worker, detects a change region from a region 1002 of a peripheral portion in the image, and determines whether the extracted image is an image taken during the movement of the on-site worker or an image taken outside the movement of the on-site worker based on the change region.

[0131] In this embodiment, a change region is detected from the region 1002 at the periphery of the image 1000 included in the video captured by the wearable terminal 10 carried by the on-site worker, and it is determined based on the change region whether the image 1000 is an image captured while the on-site worker is moving or an image captured when the on-site worker is not moving.

[0132] Thus, according to this embodiment, it is possible to determine whether the image included in the video captured by the wearable terminal 10 carried by the on-site worker is an image captured while the on-site worker is moving or an image captured when the on-site worker is not moving.

[0133] Also, in this embodiment, the region 1002 at the periphery in the image is the region 1002 at the four corners in the image 1000, the region 1002 at the center of the upper, lower, left, and right sides in the image 1000, or the entire region 1002 at the periphery in the image 1000.

[0134] According to this embodiment, it is possible to determine whether the image included in the video captured by the wearable terminal 10 carried by the on-site worker is an image captured while the on-site worker is moving or an image captured when the on-site worker is not moving, using the region 1002 at the four corners in the image 1000, the region 1002 at the center of the upper, lower, left, and right sides in the image 1000, or the entire region 1002 at the periphery in the image 1000.

[0135] Also, in this embodiment, the control unit 12 or 22 detects a change region based on the inter-frame difference of the region 1002 at the periphery in the image 1000 extracted from the video at a predetermined time interval.

[0136] According to this embodiment, for example, even when a large object such as the luggage or tool of the on-site worker appears in the central part of the image 1000, a change region can be detected based on the inter-frame difference of the region 1002 at the periphery in the image 1000, so it is possible to accurately determine whether the image is an image captured while the on-site worker is moving or an image captured when the on-site worker is not moving.

[0137] Further, in the present embodiment, the control unit 12 or 22 calculates the inter-frame difference for each pixel included in the area 1002 of the peripheral portion in the image 1000, determines the pixels that are changing based on the magnitude relationship with the threshold value, determines the ratio of the changing pixels included in the area 1002 of the peripheral portion in the image 1000, and detects the change area according to the ratio of the changing pixels.

[0138] According to the present embodiment, the ratio of the changing pixels (for example, white pixels) included in the area 1002 of the peripheral portion in the image 1000 is determined based on the magnitude relationship with the threshold value used for the binarization process, and the change area can be detected according to the ratio of the changing pixels.

[0139] Further, in the present embodiment, the control unit 12 or 22 changes the threshold value according to the brightness of the captured image.

[0140] According to the present embodiment, by increasing the threshold value used for the binarization process as the brightness of the image captured by the wearable terminal 10 is brighter, and decreasing the threshold value used for the binarization process as the brightness of the captured image is darker, it is possible to accurately determine whether the image was captured while the on-site worker was moving or the image was captured other than while the on-site worker was moving, regardless of the brightness of the image captured by the wearable terminal 10.

[0141] Further, in the present embodiment, the control unit 12 or 22 determines whether the image was captured while the on-site worker was moving or the image was captured other than while the on-site worker was moving based on the number or ratio of the change areas detected from the extracted image 1000.

[0142] According to the present embodiment, based on the number or ratio of the change areas detected from the extracted image 1000, it is possible to accurately determine whether the image was captured while the on-site worker was moving or the image was captured other than while the on-site worker was moving.

[0143] Further, in the present embodiment, the moving image is a first-person perspective moving image captured with the wearable terminal 10 worn by the on-site worker.

[0144] According to the present embodiment, a change region is detected from a peripheral region 1002 in an image 1000 included in a video captured by the wearable terminal 10 worn by a field worker, and based on the change region, it is determined whether the image was captured while the field worker was moving or while the field worker was not moving.

[0145] Thus, according to the present embodiment, it is possible to determine whether the image was captured while the field worker was moving or while the field worker was not moving from the image included in the video captured by the wearable terminal 10 worn by the field worker.

[0146] Further, the present embodiment is an information processing apparatus 20 having a control unit 22. The control unit 22 extracts an image 1000 from a video captured by the wearable terminal 10 carried by the field worker, detects a change region from a peripheral region 1002 in the image 1000, and based on the change region, determines whether the extracted image 1000 was captured while the field worker was moving or while the field worker was not moving.

[0147] In the present embodiment, a change region is detected from a peripheral region 1002 in an image 1000 included in a video captured by the wearable terminal 10 carried by the field worker, and based on the change region, it is determined whether the image was captured while the field worker was moving or while the field worker was not moving.

[0148] Thus, according to the present embodiment, it is possible to determine whether the image was captured while the field worker was moving or while the field worker was not moving from the image included in the video captured by the wearable terminal 10 carried by the field worker.

[0149] In addition, this embodiment is a wearable terminal 10 that has a control unit 12 and is carried by a field worker. The control unit 12 captures a video, extracts an image 1000 from the captured video, detects a change region from a region 1002 in the peripheral part within the image 1000, and determines whether the image was captured while the field worker was moving or while the field worker was not moving based on the change region.

[0150] In this embodiment, a change region is detected from a region 1002 in the peripheral part within an image 1000 included in a video captured by a wearable terminal 10 carried by a field worker, and it is determined whether the extracted image 1000 was captured while the field worker was moving or while the field worker was not moving based on the change region.

[0151] Thus, according to this embodiment, it is possible to determine whether an image was captured while the field worker was moving or while the field worker was not moving from an image included in a video captured by the wearable terminal 10 carried by the field worker.

[0152] Although the present embodiment has been described above, it will be understood that various changes in form and details are possible without departing from the spirit and scope of the claims.

Explanation of Reference Numerals

[0153] 1 Information Processing System 2 Site 10 Wearable Terminal 12 Control Unit 20 Information Processing Device 22 Control Unit 30 Terminal for Analysis Worker 50 Network 160 Imaging Unit 1000 Image 1002 Region in the Peripheral Part of the Image

Claims

1. An image processing method executed by an information processing apparatus or a wearable terminal having a control unit, wherein the control unit extracts an image from a video captured by the wearable terminal carried by a field worker, detects a change region from a region of a peripheral part in the image, and determines whether the extracted image is an image captured while the field worker is moving or an image captured while the field worker is not moving based on the change region Image processing method.

2. The region of the peripheral part in the image is the region of the four corners in the image, the region at the center of the upper, lower, left, and right sides in the image, or the entire region of the peripheral part in the image The image processing method according to claim 1.

3. The control unit detects the change region based on the inter-frame difference of the region of the peripheral part in the image extracted from the video at a predetermined time interval The image processing method according to claim 1 or 2.

4. The control unit calculates the inter-frame difference for each pixel included in the region of the peripheral part in the image, determines pixels that have changed based on the magnitude relationship with a threshold value, determines the ratio of the changed pixels included in the region of the peripheral part in the image, and detects the change region according to the ratio of the changed pixels The image processing method according to claim 3.

5. The control unit changes the threshold value according to the brightness of the captured image The image processing method according to claim 4.

6. The control unit determines whether the image is an image captured while the field worker is moving or an image captured while the field worker is not moving based on the number or ratio of the change regions detected from the extracted image The image processing method according to claim 1 or 2.

7. The video is a first-person perspective video captured with the wearable terminal worn on the field worker The image processing method according to claim 1 or 2.

8. An information processing apparatus having a control unit, wherein the control unit extracts an image from a video captured by a wearable terminal carried by a field worker, detects a change region from a region of a peripheral part in the image, and determines whether the extracted image is an image captured while the field worker is moving or an image captured while the field worker is not moving based on the change region Information processing apparatus.

9. A wearable terminal having a control unit and carried by a field worker, wherein the control unit captures a video, extracts an image from the captured video, Detect a change region from the region of the peripheral part in the image, Based on the change region, determine whether the extracted image is an image taken during the movement of the on-site worker or an image taken other than during the movement of the on-site worker Wearable terminal.

Images