Wearable device and transmission control method

Abstract

To provide a wearable device and a method for controlling transmission that transmit information effective for a maintenance work while preventing information leakage during the work.SOLUTION: In a system in which a customer engineer connects to a network by radio communication, puts on a smart glass for executing data communication with a support center server, and performs maintenance work of a maintenance target device, a smart glass 100 includes an acquisition unit 151 for acquiring picture data 141 from a camera 101, a masking unit 152 for executing masking processing on the picture data 141 and outputting the picture data on which the mask processing was executed to a transmission control unit 153; and the transmission control unit 153 for transmitting the picture data acquired by the masking unit 152 to a support center server 50.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to wearable devices and the like.

Background Art

[0002] A Customer Engineer (CE) performs maintenance on equipment to be maintained. When a malfunction or the like of the equipment is found, the corresponding parts are removed, parts are replaced, various adjustments inside the equipment are performed, and the equipment is repaired. If the CE does not know how to confirm malfunctions, how to remove parts, how to repair, etc. during maintenance, the CE makes a phone call to the support center to inquire.

[0003] In the method of inquiring by phone to the support center, since the operator at the support center needs to understand the content only from the voice information of the phone call by the CE, the specific malfunction content and which part of the equipment the CE is pointing to cannot be understood by the operator at the support center, and there are cases where the operator cannot give accurate instructions to the CE. When the operator cannot give accurate instructions to the CE, it becomes necessary to carry out all possible part replacements and various adjustments inside the equipment that may have malfunctions, and the maintenance time is prolonged, increasing the burden on the CE.

[0004] In order to reduce the burden on the CE, the CE may use a goggle-type headset having functions such as VR (Virtual Reality) or AR (Augmented Reality) on-site. In the following description, a goggle-type headset having functions such as VR or AR is referred to as a "smart glass". The CE wears the smart glass to perform maintenance and transmits the video during maintenance to the support center using the camera of the smart glass. Since the operator at the support center can accurately grasp the video during maintenance in real time, the operator can give accurate instructions to the CE.

[0005] Here, the locations where the equipment to be maintained is installed are the customer's office, computer room, data center, etc., and there may be customer confidential information in such locations. For example, when a CE wears smart glasses for maintenance, during the maintenance, the CE may accidentally capture an area that has nothing to do with the equipment to be maintained. In this case, if the video of the smart glasses' camera is directly transmitted to the support center, the customer's confidential information will be leaked to the outside.

[0006] To prevent the leakage of confidential information, an ID medium is attached to the equipment to be maintained, and based on the ID medium included in the video captured by the smart glasses' camera, the video area is classified into the area of the equipment to be maintained and the area outside the equipment to be maintained, and the area of the video that is not the equipment to be maintained is masked. There is also a prior art where, when the video captured by the smart glasses' camera contains character information, such character information is masked.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] However, in the above-mentioned prior art, in addition to the customer's confidential information, there may be cases where information that is effective for supporting the maintenance work is also masked.

[0009] For example, when the device to be maintained is a printer, during maintenance, it may be necessary to check the printed materials of the malfunctioning printer. The printed materials contain actual printing defects (such as smudges, bleeding, whiteouts, etc.) and printer setting information, etc. The characters printed on the output printed materials are information necessary for the maintenance work. However, in the above prior art, the information of such printed materials (for example, character information) is masked and cannot be referred to by the operator of the support center, and there may be cases where CE cannot be supported.

[0010] The above problem can also occur in devices other than printers (for example, devices that output character information, etc.).

[0011] In one aspect, an object of the present invention is to provide a wearable device and a transmission control method capable of transmitting information effective for maintenance work while preventing information leakage during work.

Means for Solving the Problems

[0012] In a first aspect, the wearable device executes a masking process on a first area of the video based on the area of the object included in the video captured by the imaging unit, suppresses masking the second area within the first area, and transmits the information of the video on which the masking process has been executed to an external device.

Effects of the Invention

[0013] It is possible to transmit information effective for maintenance work while preventing information leakage during work.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0015] Hereinafter, embodiments of the wearable device and the transmission control method disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment.

Embodiment

[0016] FIG. 1 is a diagram showing an example of a system according to Embodiment 1. As shown in FIG. 1, CE10 wears the smart glasses 100 and performs maintenance work on the device 20 to be maintained. In this embodiment, as an example, the device 20 to be maintained will be described as a "printer".

[0017] The smart glasses 100 are connected to the network 30 by wireless communication and perform data communication with the support center server 50. The smart glasses 100 are an example of a wearable device.

[0018] The support center server 50 is a terminal device operated by an operator of the support center. The support center server 50 displays the video data transmitted from the smart glasses 100, and the operator of the support center refers to such video data to support the maintenance work of CE10. Further, the operator of the support center operates the support center server 50 to communicate with CE10 wearing the smart glasses 100.

[0019] Here, the smart glasses 100 are equipped with a camera that captures the surroundings of the CE10, and transmits the video data captured by the camera to the support center server 50. The video data includes a plurality of image frames (still images) in time series. In this embodiment, the processing executed by the smart glasses 100 will be described as appropriate using the image frames included in the video data.

[0020] When the smart glasses 100 transmit an image frame to the support center server 50, the smart glasses 100 perform a process of masking the area including the customer's confidential information or the area including character information among all areas of the image frame, thereby preventing the leakage of confidential information. In addition, when the smart glasses 100 perform the masking process, based on the area of the device 20 to be maintained, the smart glasses 100 perform control to suppress the masking process for some areas among all areas of the image frame. For example, some areas where the smart glasses 100 suppress the masking process are the areas of the printed matter output from the discharge port of the device 20 to be maintained during the maintenance work.

[0021] The printed matter has actual printing defects (smudges, bleeding, whiteouts, etc.) and printer setting information printed thereon, and the characters printed on the output printed matter are information necessary for the maintenance work. The smart glasses 100 prevent the area of the characters of the printed matter output during the maintenance work from being masked from the target area of the masking process, and enable the video data (image frame) of the printed matter to be transmitted to the support center server 50.

[0022] FIG. 2 is a diagram for explaining the processing of the smart glasses according to this embodiment. The smart glasses 100 scan the area of the image frame Im10 and detect a QR (Quick Response) code (registered trademark) 5. When the identification information of the device 20 to be maintained is included in the information read from the QR code 5, the smart glasses 100 specify a predetermined area based on the position of the QR code 5 in the image frame Im10 as the area 25 of the device 20 to be maintained. Further, the smart glasses 100 specify the area 25a of the discharge port of the device 20 to be maintained based on the area 25 of the device 20 to be maintained.

[0023] When the smart glasses 100 identify that a printed matter has been output from the area 25a during the maintenance work, the smart glasses 100 recognize the area of the identified printed matter. In the following description, the area of the printed matter output from the area 25a during the maintenance work is denoted as "printed matter area 26". The smart glasses 100 follow the printed matter area 26 in each image frame and prevent the printed matter area 26 from being masked.

[0024] FIG. 3 is a diagram showing the result of the masking process compared with the prior art. In FIG. 3, the image frame Im10-1 shows the image frame masked by the smart glasses 100. The image frame Im10-2 shows the image frame masked based on the prior art.

[0025] For example, in the prior art, among the areas of the image frame Im10-2, the areas other than the area 25-2 of the device 20 to be maintained are masked, and the printed matter area 26-2 of the printed matter effective for the maintenance work is also masked.

[0026] On the other hand, the smart glasses 100 mask the areas other than the area 25-1 of the device 20 to be maintained in the area of the image frame Im10-1, while preventing the printed matter area 26-1 from being masked and leaving the information of the printed matter (for example, the character information of the printed matter) as it is. This makes it possible to transmit the video data (image frame) of the characters printed on the output printed matter to the support center server 50.

[0027] Next, a configuration example of the smart glasses 100 according to this embodiment will be described. FIG. 4 is a functional block diagram showing the configuration of the smart glasses according to this embodiment. As shown in FIG. 4, this smart glasses 100 includes a camera 101, a microphone 102, a speaker 103, a display 104, an acceleration sensor 105, an input device 106, a wireless communication unit 110, a storage unit 140, and a control unit 150.

[0028] The camera 101 is a camera that photographs the surroundings of the CE10. For example, it is assumed that the camera 101 is installed on the smart glasses 100 so as to photograph the range in front of the CE10. The camera 101 outputs the photographed video data to the control unit 150. The camera 101 is an example of a photographing unit.

[0029] The microphone 102 is a microphone that collects sounds around the microphone 102. For example, the microphone 102 outputs the voice data obtained by collecting the voice of the CE10 to the control unit 150.

[0030] The speaker 103 is a speaker that outputs the sound of the sound data output from the control unit 150. For example, the sound output from the speaker 103 is the voice from an operator of the support center who operates the support center server 50.

[0031] The display 104 is a display device that displays various information output from the control unit 150.

[0032] The acceleration sensor 105 is a sensor that measures acceleration. The acceleration sensor 105 outputs acceleration information to the control unit 150.

[0033] The input device 106 is an input device used when the CE10 performs various operations on the smart glasses. The input device 106 corresponds to an input button, a touch panel, etc. For example, when the CE10 starts shooting with the camera 101, it operates the input device 106 to input a "shooting start instruction". When the CE10 ends shooting with the camera 101, it operates the input device 106 to input a "shooting end instruction".

[0034] The wireless communication unit 110 is a processing unit that connects to the network 30 by wireless communication and performs data communication with the support center server 50 via the network 30. The wireless communication unit 110 is an example of a wireless communication device. The control unit 150 described later exchanges data with the support center server 50 via the wireless communication unit 110.

[0035] The video data 141 is video data captured by the camera 101. For example, the video data 141 is data including time-series image frames.

[0036] The control unit 150 includes an acquisition unit 151, a masking unit 152, a transmission control unit 153, and a call control unit 154. The control unit 150 can be realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. Also, the control unit 150 can be realized by hard-wired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0037] The acquisition unit 151 acquires the video data 141 from the camera 101. The acquisition unit 151 registers the acquired video data 141 in the storage unit 140. For example, when the acquisition unit 151 receives a shooting start instruction from the input device 106, it causes the camera 101 to start shooting and starts acquiring the video data 141. When the acquisition unit 151 receives a shooting end instruction from the input device 106, it causes the camera 101 to stop shooting and ends the acquisition of the video data 141.

[0038] The masking unit 152 executes a mask process on the video data 141 and outputs the video data on which the mask process has been executed to the transmission control unit 153. For example, the masking unit 152 executes a process of specifying the area of the device 20 to be maintained, a process of specifying the printed matter area 26, and a process of executing masking.

[0039] First, the process of specifying the area of the area to be maintained will be described. The masking unit 152 scans the entire area of the image frame to detect a QR code. For example, the masking unit 152 retains the shape characteristics of the QR code and executes pattern matching or the like to detect the QR code. When the masking unit 152 detects a QR code, it acquires the information of the QR code and determines whether the identification information of the device 20 to be maintained is included in the acquired information. The masking unit 152 may notify the support center server 50 of the information of the QR code and make an inquiry request as to whether the identification information of the device 20 to be maintained is included in the information of the QR code.

[0040] When the identification information of the device 20 to be maintained is included in the information of the QR code, the masking unit 152 specifies a predetermined area based on the position of the QR code in the entire area of the image frame as the area of the device 20 to be maintained. The masking unit 152 specifies the area 25a of the discharge port of the device 20 to be maintained based on the area of the device 20 to be maintained. In the example described with reference to FIG. 2, the masking unit 152 specifies the area 25 of the device 20 to be maintained and the area 25a of the discharge port.

[0041] Next, the process of identifying the printed matter area 26 will be described. The masking unit 152 generates a difference image for the area of the outlet of the time-series image frames. Based on the difference image, the masking unit 152 identifies, as the area of the printed matter (printed matter area 26) output from the device 20 to be maintained, the area that is different between the previous frame and the subsequent frame (dynamic area). Also, as described above, since the video data 141 is the video data captured after the shooting start instruction is input, it can be said that the printed matter area 26 identified by the masking unit 152 is the area of the printed matter output from the device 20 to be maintained during the maintenance work. Note that before the maintenance work is performed, the area of the printed matter output to the device 20 to be maintained is a static area, and thus is not identified as the printed matter area 26.

[0042] When the masking unit 152 identifies the printed matter area 26, it tracks the printed matter area 26 in subsequent image frames. For example, the masking unit 152 maintains the shape characteristics of the printed matter area 26, and repeatedly executes a process of identifying, as the printed matter area 26 in the next image frame, an area having the same shape characteristics as the shape characteristics of the printed matter area 26 with respect to the area of the next image frame.

[0043] Note that the process in which the masking unit 152 identifies and tracks the printed matter area 26 described above is an example. The masking unit 152 may detect the printed matter area 26 from the image frames (video data 141) and perform tracking using existing detection algorithms such as YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), and RCNN (Region Based Convolutional Neural Networks).

[0044] Next, the masking process will be described. The masking unit 152 sets, as a target area candidate for the masking process, an area other than the area of the device 20 to be maintained among all areas of the image frame.

[0045] Further, the masking unit 152 scans the entire area of the image frame to detect the area where characters are described. The masking unit 152 sets the area where characters are described among the entire area of the image frame as a target area candidate for mask processing.

[0046] The masking unit 152 specifies the target area for mask processing by excluding the printed matter area 26 from the target area candidates for mask processing. The masking unit 152 masks the target area for mask processing of the image frame. For example, the masking unit 152 fills the target area with a predetermined pixel value.

[0047] The masking unit 152 repeatedly executes the above processing to perform mask processing on the time-series image frames, and outputs the video data (time-series image frames) on which the mask processing has been performed to the transmission control unit 153.

[0048] The transmission control unit 153 transmits the video data acquired from the masking unit 152 to the support center server 50. When the transmission control unit 153 receives an instruction to end shooting from the input device 106, the transmission control unit 153 stops the process of transmitting the video data to the support center server 50.

[0049] The call control unit 154 is a processing unit that controls the call between the smart glasses 100 and the support center server 50. For example, the call control unit 154 establishes a call line between the smart glasses 100 and the support center server 50, and transmits the voice data of CE10 input from the microphone 102 to the support center server 50. The call control unit 154 outputs the voice data of the operator of the support center transmitted from the support center server 50 to the speaker 103.

[0050] Next, an example of the processing procedure of the smart glasses 100 according to this embodiment will be described. FIG. 5 is a flowchart showing the processing procedure of the smart glasses according to this embodiment. As shown in FIG. 5, when the acquisition unit 151 of the smart glasses 100 receives a shooting start instruction, it starts shooting a video and registers the video data 141 in the storage unit 140 (step S101).

[0051] The masking unit 152 of the smart glasses 100 detects a QR code from the image frame (step S102). The masking unit 152 reads the information of the QR code and determines whether the identification information of the device 20 to be maintained is included (step S103).

[0052] When the masking unit 152 determines that the identification information of the device 20 to be maintained is included in the QR code information (step S104, Yes), it proceeds to step S105. On the other hand, when the masking unit 152 determines that the identification information of the device 20 to be maintained is not included in the QR code information (step S104, No), it returns to step S102.

[0053] The masking unit 152 executes a masking process (step S105). The transmission control unit 153 of the smart glasses 100 transmits the video data subjected to the masking process to the support center server 50 (step S106).

[0054] Subsequently, an example of the processing procedure of the masking process shown in step S105 of FIG. 5 will be described. FIG. 6 is a flowchart showing the processing procedure of the masking process. As shown in FIG. 6, the masking unit 152 of the smart glasses 100 specifies the area of the device 20 to be maintained and the area of the discharge port based on the position of the QR code (step S201).

[0055] The masking unit 152 specifies the printed matter area of the printed matter output from the device to be maintained based on the differential image of the area of the discharge port of the time-series image frames (step S202). The masking unit 152 specifies the area of the character information from the image frame (step S203).

[0056] The masking unit 152 sets, as candidate regions for masking processing, the regions other than the region of the device 20 to be maintained and the region of character information among all the regions of the image frame (step S204). The masking unit 152 sets, as the region for masking processing, the region obtained by excluding the printed matter region from the candidate regions for masking processing (step S205).

[0057] The masking unit 152 fills the target region with a predetermined pixel value (step S206).

[0058] Note that the processing content of the flowchart shown in FIG. 6 is an example. For example, the smart glasses 100 may set, as the target region, the region obtained by excluding the region of the device to be maintained and the printed matter region from all the regions of the image frame, and perform masking processing.

[0059] Next, the effects of the smart glasses 100 according to this embodiment will be described. When transmitting video data to the support center server 50, the smart glasses 100 perform a process of masking the regions including customer confidential information and the regions including character information among all the regions of the image frame of the video data. When performing the masking process, the smart glasses 100 perform control to suppress the masking process for some regions among all the regions of the image frame based on the region of the device 20 to be maintained. For example, some regions to be excluded from the masking target are the regions of printed matter output from the device 20 during maintenance work.

[0060] On the printed matter, actual printing defects (smudges, bleeding, whiteouts, etc.) and printer setting information are printed, and the characters printed on the output printed matter are information necessary for the maintenance work. The smart glasses 100 suppress the masking of the character regions of the printed matter output during the maintenance work from the regions to be masked, and enable the transmission of the video data (image frame) of the characters printed on the printed matter to the support center server 50. That is, it is possible to transmit information effective for the maintenance work while preventing information leakage during the work.

[0061] The smart glasses 100 detect a QR code from an image frame of video data, and based on the position of the QR code, identify the area of the device 20 to be maintained and the area of the discharge port. As a result, the area where the printed matter is output can be narrowed down, and the processing load for identifying the printed matter area can be reduced.

[0062] The smart glasses 100 identify the printed matter area of the printed matter based on the video data captured by the camera during the maintenance work. As a result, during the maintenance work, the printed matter output from the device 20 to be maintained can be targeted, the printed matter area can be set, and the masking process can be suppressed.

[0063] By the way, the processing of the smart glasses 100 described above is an example, and the smart glasses 100 may execute processing other than the above processing. Here, other processing 1 to 3 of the smart glasses 100 will be described.

[0064] The "other processing 1" of the smart glasses 100 will be described. The device 20 to be maintained described in the above embodiment was described assuming a small printer. However, in the case of a large printer, when the entire device to be maintained does not fit in the image frame, or even if it fits, the area of the discharge port becomes small within the image frame. Therefore, it becomes difficult to identify the area of the device to be maintained and the area of the discharge port of the device to be maintained. For this reason, by presetting a QR code near the discharge port of the device to be maintained, it is possible to identify the area of the discharge port even in the case of a large printer.

[0065] FIG. 7 is a diagram for explaining other processing 1 executed by the smart glasses. The image frame Im20 in FIG. 7 is an image frame included in the video data captured by the camera 101 of the smart glasses 100. The masking unit 152 of the smart glasses 100 scans the entire area of the image frame Im20 to detect the position of the QR code 6.

[0066] When the information read from the QR code 6 includes the identification information of the device to be maintained (large printer), the masking unit 152 identifies the area 27 of the discharge port of the device to be maintained based on the position of the QR code 6 in the image frame Im20. In a large printer, since the paper discharge port 28 swings left and right to output printed materials, the area 27 becomes the area of the printed materials.

[0067] For example, the masking unit 152 of the smart glasses 100 suppresses the masking process for the area 27 among all areas of the image frame Im20. That is, the masking unit 152 of the smart glasses 100 performs a masking process on areas other than the area 27 among all areas of the image frame Im20.

[0068] In this way, when the device to be maintained is a large printer, the smart glasses 100 can notify the operator of the support center server 50 of the character information effective for the maintenance work by suppressing the masking process for the area 27 based on the position of the QR code.

[0069] The "Other Process 2" of the smart glasses 100 will be described. In the case of a large printer, there are areas where the printing status can be confirmed even in areas other than the discharge port. Therefore, by presetting a QR code in such areas, it is possible to prevent the areas where the printing status can be confirmed from being masked.

[0070] FIG. 8 is a diagram for explaining Other Process 2 executed by the smart glasses. The device 60 to be maintained shown in FIG. 8 includes a viewing location 60a and a discharge port 60b. The device 60 to be maintained corresponds to a large printer. The viewing location 60a is a location where an operator or worker can check the internal operation and printing status of the device 60 to be maintained. For example, the worker attaches a QR code with information for identifying the viewing location inside the viewing location 60a. Note that printed materials are output from the discharge port 60b.

[0071] The masking unit 152 of the smart glasses 100 scans the image frames of the video data captured by the camera 101 to detect the QR code. When the information read from the QR code 6 includes the information for identifying the peeping location 60a, the smart glasses 100 skip the masking process for the area of the image frame. As a result, the video data captured by the camera 101 of the smart glasses 100 and including the state inside the peeping location 60a can be notified to the operator of the support center server 50.

[0072] "Other processing 3" of the smart glasses 100 will be described. In the above embodiment, the smart glasses 100 track the printed matter area 26 of the printed matter output from the device 20 to be maintained during the maintenance work and explain the case where the masking process for the printed matter area 26 is suppressed, but it is not limited thereto. For example, when the masking unit 152 of the smart glasses 100 detects the printed matter output from the device 20 to be maintained during the maintenance work, it may perform a process of suppressing the masking process for the area 25a of the discharge port of the device 20 to be maintained and skip the process of tracking the printed matter area 26.

[0073] During the maintenance work, the printed matter output from the device 20 to be maintained may continue to exist in the area 25a of the discharge port of the device 20 to be maintained. By skipping the process of tracking the printed matter area 26, the processing load of the smart glasses 100 can be reduced.

Explanation of reference numerals

[0074] 100 Smart glasses 101 Camera 102 Microphone 103 Speaker 104 Display 105 Acceleration sensor 106 Input device 110 Wireless communication unit 140 Storage unit 141 Video data 150 Control unit 151 Acquisition unit 152 Masking section 153 Transmission control section 154 Call control section

Claims

1. A photographing unit that photographs an image of a photographing range; Based on the area of the object included in the image photographed by the photographing unit, among the first areas of the image, a dynamic area is specified, and among the specified dynamic areas, an area with characters described is specified as a second area, mask processing is executed on the first area, and a masking unit that suppresses performing the mask processing on the second area within the first area; A transmission unit that transmits information on the image on which the mask processing has been executed to an external device; A wearable device, characterized by comprising the above.

2. The masking unit of the wearable device according to claim 1, characterized in that it detects a predetermined code from the image and specifies the area of the object based on the area of the predetermined code.

3. The masking unit of the wearable device according to claim 1 further executes a process of specifying, as the second area, an area of a printed matter output from the area of the object based on an image photographed by the photographing unit after the start time of the work.

4. The masking unit of the wearable device according to claim 3, characterized in that it specifies, as the first area, an area other than the area of the object and an area including printed matter information among the areas of the image.

5. The masking unit of the wearable device according to claim 1 further executes a process of detecting a predetermined code from the image and specifying, as the second area, an area of a paper discharge port of a printing machine that is the object based on the area of the predetermined code.

6. Based on the area of the object included in the image photographed by the photographing unit, among the first areas of the image, a dynamic area is specified, and among the specified dynamic areas, an area with characters described is specified as a second area, and mask processing is executed on the first area. Suppress the mask processing for the second region within the first region, Transmit information of the video for which the mask processing has been executed to an external device A transmission control method characterized by causing a computer to execute the processing.

Images