Information processing method, information processing device, and information processing program

Abstract

An image processing device acquires an image captured by a camera, executes mask processing for creating a mask image in which a region outside an image-capturable region in the image is masked, determines whether or not the mask processing is executed, and transmits the mask image if it is determined that the mask processing is executed.

Description

Technical Field

[0001] The present disclosure relates to a technology for masking an area other than a photographable area in an image. Background Art

[0002] For example, Patent Document 1 discloses an image control device that sets a permitted image-capable area in a work area for image sharing with a remote assistant operator, and sends a masked image of the work area that masks the non-image-capable area to a remote terminal for the remote assistant operator. The existing image control device detects a mark contained in the image data, extracts mark information indicating the image-capable area from the detected mark, determines an image area in the image data that is judged to be outside the image-capable area as an image-unshareable area, masks the image-unshareable area, and generates a masked image.

[0003] However, in the above-mentioned existing technologies, confidential information may be leaked, and further improvement is needed.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-150866 Summary of the invention

[0007] The present disclosure is made to solve the above-mentioned problems, and an object of the present disclosure is to provide a technology capable of preventing leakage of confidential information.

[0008] The information processing method involved in the present disclosure is an information processing method in a computer, which obtains an image captured by a camera, performs mask processing, and creates a mask image that masks an area outside a captureable area in the image, determines whether the mask processing has been performed, and sends the mask image if it is determined that the mask processing has been performed.

[0009] According to the present disclosure, it is possible to prevent confidential information from being leaked. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a diagram showing an example of the configuration of the work support system in Embodiment 1 of the present disclosure.

[0011] Figure 2 This is a diagram showing an example of a work space in the first embodiment.

[0012] Figure 3 This is a diagram showing an example of the configuration of the image processing device in the first embodiment.

[0013] Figure 4This is a diagram showing an example of an image captured by a camera in which all the signs are reflected in the image in the first embodiment.

[0014] Figure 5 This means that in this embodiment 1, based on Figure 4 FIG. 1 is a diagram showing an example of a mask image created using the image shown.

[0015] Figure 6 This is a first flowchart for explaining the job assisting process of the image processing device in the first embodiment of the present disclosure.

[0016] Figure 7 This is a second flowchart for explaining the job assisting process of the image processing device in the first embodiment of the present disclosure.

[0017] Figure 8 This is a diagram showing an example of the structure of the image processing device in the second embodiment.

[0018] Fig. 9 This is a first flowchart for explaining the job assisting process of the image processing device in the second embodiment of the present disclosure.

[0019] Fig.10 This is a second flowchart for explaining the job assisting process of the image processing device in the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] (Insights that form the basis of this disclosure)

[0021] In order for workers at the site to efficiently receive work assistance from workers at a remote location, it is effective to share images of the conditions at the site with the workers at the remote location.

[0022] When transmitting images to a remote location, confidential information may be reflected in the images. Therefore, in order to prevent confidential information from being leaked to the outside through image transmission, objects that can be image transmitted or objects that are confidential information are identified and the area corresponding to the confidential information in the image is masked.

[0023] For example, an operator at the scene configures multiple markers to surround an area where images can be transmitted, that is, an area that does not contain confidential information. By identifying multiple markers in the captured image, the area outside the imageable area surrounded by the identified multiple markers is masked, thereby preventing confidential information from being leaked.

[0024] However, the camera is activated at an arbitrary timing by the operator. Therefore, if the transmission of the image is started when the mask processing function for hiding the confidential information is not activated at the timing of the image output from the camera, the image with the confidential information reflected in it may be transmitted to a remote terminal, and the confidential information may be leaked.

[0025] In the above-mentioned prior art, the image data after the mask processing is transmitted to the remote terminal, but it is not determined whether the mask processing is reliably performed, and there is a possibility that confidential information is leaked.

[0026] In order to solve the above problems, the following technology is disclosed.

[0027] (1) An information processing method according to one aspect of the present disclosure is an information processing method in a computer, which obtains an image captured by a camera, performs mask processing to create a mask image that masks an area outside a captureable area in the image, determines whether the mask processing has been performed, and transmits the mask image if it is determined that the mask processing has been performed.

[0028] According to this structure, after performing mask processing to create a mask image that masks an area outside the photographable area in the image, it is determined whether the mask processing has been performed. Then, if it is determined that the mask processing has been performed, the created mask image is sent. Therefore, it is possible to prevent confidential information from being leaked.

[0029] (2) In the information processing method described in (1) above, the mask processing may recognize a plurality of markers included in the image and create a mask image that masks an area outside the imageable area surrounded by the recognized plurality of markers.

[0030] According to this configuration, by arranging a plurality of markers so as to surround an area not containing confidential information, the imageable area in the image can be easily specified, thereby preventing confidential information from being leaked.

[0031] (3) In the information processing method described in (1) or (2) above, it is also possible that, after the mask processing is initially executed, a mark indicating that the mask image can be sent is enabled (ON), and in determining whether the mask processing has been executed, it is determined whether the mark has become enabled, and if it is determined that the mark has become enabled, it is determined that the mask processing has been executed.

[0032] According to this configuration, after the mask process is executed for the second time or later, it is determined whether the flag indicating that the mask image can be sent is enabled, and if it is determined that the flag is enabled, the mask image is sent. Therefore, by determining whether the flag indicating that the mask image can be sent is enabled, it is possible to easily determine whether the mask process has been executed.

[0033] (4) In the information processing method described in (1) or (2) above, it is also possible that, after the mask processing is initially executed, the created mask image is displayed, and then an input of an instruction from an operator indicating whether the mask processing has been normally executed is accepted. Furthermore, when an input of an instruction from the operator indicating that the mask processing has been normally executed is accepted, a mark indicating that the mask image can be sent is enabled (on), and in determining whether the mask processing has been executed, it is determined whether the mark has become enabled, and when it is determined that the mark has become enabled, it is determined that the mask processing has been executed.

[0034] According to this configuration, since the operator can check whether the masking process has been normally performed by observing the displayed mask image, leakage of confidential information can be reliably prevented.

[0035] (5) In the information processing method described in (4) above, further, when receiving an input from the operator indicating that the mask processing has not been performed normally, the mark may be disabled (off), and in determining whether the mask processing has been performed, it is determined whether the mark has become effective, and when it is determined that the mark has become disabled, it is determined that the mask processing has not been performed, and in sending the mask image, when it is determined that the mask processing has not been performed, the mask image is not sent.

[0036] According to this configuration, when an instruction indicating that the masking process is not normally executed is input by the operator, the flag is disabled. Then, after the masking process is executed for the second time or later, if it is determined that the flag is disabled, the mask image is not transmitted.

[0037] Therefore, when the operator determines that the masking process is not normally performed, the flag is disabled, and even if the masking process is performed, the mask image is not transmitted, so that leakage of confidential information can be reliably prevented.

[0038] (6) In the information processing method described in any one of (3) to (5) above, the flag may be disabled when transmission of the mask image is completed.

[0039] According to this configuration, when the transmission of the mask image is completed, the flag can be switched from enabled to disabled.

[0040] In addition, the present disclosure can be implemented not only as an information processing method that performs the above-mentioned characteristic processing, but also as an information processing device having a characteristic structure corresponding to the characteristic processing performed by the information processing method. In addition, it can also be implemented as a computer program that causes a computer to perform the characteristic processing included in such an information processing method. Therefore, in the following other modes, the same effect as the above-mentioned information processing method can be achieved.

[0041] (7) Another embodiment of the present disclosure relates to an information processing device comprising: an acquisition unit that acquires an image captured by a camera; a mask processing execution unit that executes mask processing to create a mask image that masks an area outside a captureable area in the image; a determination unit that determines whether the mask processing has been executed; and a sending unit that sends the mask image when it is determined that the mask processing has been executed.

[0042] (8) Another aspect of the present disclosure relates to an information processing program in which a computer performs the following functions: obtains an image captured by a camera, performs mask processing to create a mask image that masks an area outside a captureable area in the image, determines whether the mask processing has been performed, and transmits the mask image if it is determined that the mask processing has been performed.

[0043] (9) Another embodiment of the present disclosure relates to a non-temporary computer-readable recording medium that records an information processing program, wherein the information processing program enables a computer to perform the following functions: obtain an image captured by a camera, perform mask processing, wherein the mask processing creates a mask image that masks an area outside a captureable area within the image, determine whether the mask processing has been performed, and send the mask image if it is determined that the mask processing has been performed.

[0044] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. In addition, in all embodiments, the various contents can also be combined.

[0045] (Implementation method 1)

[0046] Figure 1 It is a diagram showing an example of the configuration of the work support system in Embodiment 1 of the present disclosure. Figure 1 The illustrated work support system includes a camera 1 , an image processing device 2 , and a remote terminal 3 .

[0047] The camera 1 is, for example, a wearable camera that can be mounted on the head of an operator at a site. The operator is equipped with the camera 1 to perform work. At this time, the operator at the site uses the camera 1 to photograph the auxiliary object device located in the work space at the site, and receives work assistance from a remote operator. The remote operator observes the image transmitted from the site, and performs assistance in operating and repairing the auxiliary object device.

[0048] Figure 2 This is a diagram showing an example of a work space in the first embodiment.

[0049] like Figure 2 As shown, an auxiliary object device 101 is set in a work space 100. An on-site operator uses a camera 1 to photograph the auxiliary object device 101. The image photographed by the camera 1 is sent to a remote terminal 3 operated by an operator at a distance. At this time, there is a possibility that confidential information is reflected in the photographed image. For this reason, in order to prevent the transmission of images containing confidential information, the area outside a specific work area is blocked by image processing (masking). The on-site operator attaches a plurality of marks 111, 112, 113, and 114 so as to surround the work area that does not contain confidential information of the auxiliary object device 101. The plurality of marks 111, 112, 113, and 114 are directly attached so as to surround the work area that can be photographed, for example, by means of an adhesive or a magnet.

[0050] The camera 1 transmits the captured image to the image processing device 2 via short-range wireless communication. The camera 1 transmits the image at a given frame rate to the image processing device 2. For this purpose, a dynamic image is transmitted from the camera 1 to the image processing device 2. In addition, the camera 1 is connected to the image processing device 2 wirelessly, but can also be connected to the image processing device 2 by wire.

[0051] In addition, the camera 1 may be a terminal with a camera such as a smartphone or a tablet computer. In addition, the camera 1 may be a digital camera.

[0052] The image processing device 2 is, for example, a personal computer operated by an on-site worker. The image processing device 2 and the remote terminal 3 are connected to each other so as to be communicable via a network 4. The network 4 is, for example, the Internet.

[0053] Figure 3 This is a diagram showing an example of the configuration of the image processing device 2 in the first embodiment.

[0054] Figure 3 The image processing device 2 shown includes a processor 21 , a memory 22 , and a communication unit 23 .

[0055] The processor 21 is, for example, a CPU (Central Processing Unit). The processor 21 implements an image acquisition unit 211 , a marker recognition unit 212 , a mask processing execution unit 213 , a marker setting unit 214 , a determination unit 215 , and an output unit 216 .

[0056] The memory 22 is a storage device capable of storing various information, such as a RAM (Random Access Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), or a flash memory.

[0057] In addition, the image processing device 2 is equivalent to an example of an information processing device, the image acquisition unit 211 is equivalent to an example of an acquisition unit, the mark recognition unit 212 is equivalent to an example of a recognition unit, the mask processing execution unit 213 is equivalent to an example of a mask processing execution unit, the determination unit 215 is equivalent to an example of a determination unit, and the communication unit 23 is equivalent to an example of a sending unit.

[0058] The communication unit 23 receives the image transmitted from the camera 1. The communication unit 23 outputs the received image to the processor 21. The communication unit 23 receives the moving image transmitted at a given frame rate, and outputs the image to the processor 21 for each frame.

[0059] The image acquisition unit 211 acquires an image captured by the camera 1. The image acquisition unit 211 acquires an image for each frame.

[0060] The marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211. In the first embodiment, four markers 111, 112, 113, and 114 of different types are used. Figure 2 As shown, the four marks 111, 112, 113, and 114 are rectangular and have different patterns.

[0061] The worker at the site affixes the sign 111 to the upper left of the work area 150 that can be photographed, the sign 112 to the lower left of the work area 150 that can be photographed, the sign 113 to the upper right of the work area 150 that can be photographed, and the sign 114 to the lower right of the work area 150 that can be photographed. In addition, there is no particular determination as to where the four signs 111, 112, 113, and 114 are arranged. The four signs 111, 112, 113, and 114 may be arranged at any of the upper left, lower left, upper right, and lower right of the work area 150 that can be photographed.

[0062] The mark recognition unit 212 recognizes the four marks 111, 112, 113, 114 contained in the image by pattern matching. That is, the mark recognition unit 212 recognizes the four marks 111, 112, 113, 114 in the image by comparing the patterns of the multiple marks pre-stored in the memory 22 with the patterns of the four marks 111, 112, 113, 114 contained in the image.

[0063] In addition, the sign recognition unit 212 can recognize not only a sign photographed from the front but also a sign photographed from an oblique direction.

[0064] Furthermore, in the first embodiment, four markers are used, but the present disclosure is not particularly limited thereto, and three markers may be used, or five or more markers may be used.

[0065] The mask processing execution unit 213 executes mask processing to create a mask image that masks an area outside the imageable area in the image. The mask processing recognizes multiple markers included in the image and creates a mask image that masks an area outside the imageable area surrounded by the recognized multiple markers in the image.

[0066] When the marker recognition unit 212 recognizes a plurality of markers 111, 112, 113, and 114, the mask processing execution unit 213 determines a captureable area surrounded by the recognized plurality of markers 111, 112, 113, and 114. That is, when all of the plurality of markers 111, 112, 113, and 114 are recognized, the mask processing execution unit 213 determines a captureable area surrounded by the recognized plurality of markers 111, 112, 113, and 114.

[0067] When the four marks 111, 112, 113, and 114 are identified, the mask processing execution unit 213 determines a photographable area that connects the upper left vertex of the upper left mark 111, the lower left vertex of the lower left mark 112, the upper right vertex of the upper right mark 113, and the lower right vertex of the lower right mark 114 with a straight line.

[0068] In addition, when the four marks 111, 112, 113, and 114 are recognized, the mask processing execution unit 213 can determine the imageable area by connecting the lower right vertex of the upper left mark 111, the upper right vertex of the lower left mark 112, the lower left vertex of the upper right mark 113, and the upper left vertex of the lower right mark 114 with a straight line. In addition, when the four marks 111, 112, 113, and 114 are recognized, the mask processing execution unit 213 can determine the imageable area by connecting the center points of the four marks 111, 112, 113, and 114 with a straight line.

[0069] In addition, when all of the multiple markers are not recognized, the mask processing execution unit 213 does not determine the imageable region. The number of the multiple markers is predetermined, and in the first embodiment, the number of the multiple markers is 4. When all 4 markers are recognized, the mask processing execution unit 213 determines the imageable region, and when only 3 or less markers are recognized, the mask processing execution unit 213 does not determine the imageable region.

[0070] The mask processing execution unit 213 creates a mask image that masks an area outside the determined imageable area in the image. If the imageable area is not determined, the mask processing execution unit 213 may create a mask image that masks the entire image.

[0071] After the masking process is initially executed, the flag setting unit 214 validates the transmission permission flag indicating that the mask image can be transmitted. The memory 22 stores the transmission permission flag. After the masking process is initially executed, the flag setting unit 214 validates the transmission permission flag stored in the memory 22. In addition, when the transmission of the mask image is terminated, the flag setting unit 214 invalidates the transmission permission flag stored in the memory 22.

[0072] The determination unit 215 determines whether the masking process is executed by the masking process execution unit 213. Here, the determination unit 215 determines whether the transmission permission flag is effective. When the determination unit 215 determines that the transmission permission flag is effective, it determines that the masking process is executed. In addition, when the determination unit 215 determines that the transmission permission flag is not effective, that is, when the determination unit 215 determines that the transmission permission flag is invalid, it determines that the masking process is not executed.

[0073] When the determination unit 215 determines that the mask process is executed, the output unit 216 transmits the mask image. When the determination unit 215 determines that the mask process is executed, the output unit 216 transmits the mask image created by the mask process execution unit 213 to the remote terminal 3 via the communication unit 23. The communication unit 23 transmits the mask image to the remote terminal 3. In addition, when the determination unit 215 determines that the mask process is not executed, the output unit 216 does not transmit the mask image.

[0074] Figure 4 1 is a diagram showing an example of an image 121 captured by the camera 1 in the first embodiment in which all the marks are reflected. Figure 4 In order to make the description easier to understand, the symbols are shown larger than their actual size.

[0075] The image 121 includes all of the four markers 111, 112, 113, and 114. The marker recognition unit 212 recognizes the four markers 111, 112, 113, and 114 included in the image 121 acquired by the image acquisition unit 211. When all of the four markers 111, 112, 113, and 114 are recognized, the mask processing execution unit 213 specifies the imageable area 131 surrounded by the recognized four markers 111, 112, 113, and 114.

[0076] Figure 5 This means that in this embodiment 1, based on Figure 4 FIG. 1 is a diagram showing an example of a mask image 141 created from the image 121 shown.

[0077] The mask processing execution unit 213 creates a mask image 141 that masks the area 132 outside the imageable area 131 in the image 121. The mask processing execution unit 213 performs mask processing on the area 132 outside the imageable area 131 in the image 121. The mask processing may be, for example, a process of filling the area 132 with a given color such as black, white, or gray, a process of blurring the area 132, or a process of mosaicking the area 132. Figure 5 In the mask image 141 shown, a photographable image 161 within the photographable region 131 can be visually recognized, and a region 132 outside the photographable region 131 is concealed.

[0078] When the four marks 111 , 112 , 113 , and 114 are recognized for the first time and the masking process is executed for the first time, the flag setting unit 214 validates the transmission permission flag indicating that the mask image can be transmitted.

[0079] In the first embodiment, the image processing device 2 is disposed on-site, but the present disclosure is not particularly limited thereto, and the image processing device 2 may be a server provided on the cloud. In this case, the image processing device 2 and the camera 1 are connected to each other via the network 4 so as to be able to communicate with each other.

[0080] The remote terminal 3 is, for example, a personal computer, a smartphone, or a tablet computer operated by a remote operator. The remote terminal 3 receives the mask image sent by the image processing device 2. The remote terminal 3 displays the received mask image. The image processing device 2 sends the mask image to the remote terminal 3 at a given frame rate. For this purpose, a dynamic image is sent from the image processing device 2 to the remote terminal 3.

[0081] A remote worker can observe the mask image displayed on the remote terminal 3 and assist the work of the on-site worker. In addition, the image processing device 2 and the remote terminal 3 can transmit and receive not only image data but also audio data to and from each other.

[0082] Next, the job assisting process of the image processing device 2 in the first embodiment of the present disclosure will be described.

[0083] Figure 6 is a first flowchart for explaining the operation assisting process of the image processing device 2 in the first embodiment of the present disclosure. Figure 7 This is a second flowchart for explaining the job assisting process of the image processing device 2 in the first embodiment of the present disclosure.

[0084] First, in step S1, the image acquisition unit 211 acquires an image captured by the camera 1. When the camera 1 is powered on or a button provided in the image processing device 2 for starting transmission of a mask image is pressed, an image is acquired from the camera 1.

[0085] Next, in step S2, the mark recognition unit 212 determines whether the initialization flag stored in the memory 22 is valid. Here, when the initialization flag is not valid, that is, the initialization flag is invalid (step S2 "No"), in step S3, the mark recognition unit 212 recognizes a plurality of marks contained in the image acquired by the image acquisition unit 211.

[0086] Next, in step S4, the mask processing execution unit 213 determines whether all the marks are recognized by the mark recognition unit 212. In addition, the number of the plurality of marks used in image recognition is predetermined. The memory 22 stores the number of the plurality of marks in advance. In the first embodiment, the number of the plurality of marks is 4. The mask processing execution unit 213 determines whether the mark recognition unit 212 recognizes 4 marks.

[0087] Here, when it is determined that all the markers are not recognized, that is, when it is determined that at least one of the plurality of markers is not recognized (No in step S4 ), the process returns to step S1 .

[0088] On the other hand, when it is determined that all the markers are recognized (Yes in step S4 ), in step S5 , the mask processing execution unit 213 specifies the imageable region surrounded by the plurality of markers recognized by the marker recognition unit 212 .

[0089] Next, in step S6 , the mask processing execution unit 213 creates a mask image by performing mask processing on the area outside the determined imageable area in the image.

[0090] Next, in step S7 , the flag setting unit 214 validates the initialization flag stored in the memory 22 .

[0091] Next, in step S8, the flag setting unit 214 validates the transmission permission flag stored in the memory 22. After the transmission permission flag is validated, the process proceeds to step S14.

[0092] On the other hand, when it is determined that the initialization flag is valid (Yes in step S2 ), in step S9 , the marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211 .

[0093] Next, in step S10 , the mask processing execution unit 213 determines whether all the markers have been recognized by the marker recognition unit 212 .

[0094] Here, when it is determined that all the flags are not recognized, that is, when it is determined that at least one of the multiple flags is not recognized (step S10 "No"), in step S11, the flag setting unit 214 invalidates the initialization flag stored in the memory 22. After the initialization flag is invalidated, the process returns to step S1.

[0095] On the other hand, when it is determined that all the markers are recognized (Yes in step S10 ), in step S12 , the mask processing execution unit 213 specifies the imageable region surrounded by the plurality of markers recognized by the marker recognition unit 212 .

[0096] Next, in step S13 , the mask processing execution unit 213 creates a mask image by performing mask processing on the area outside the determined imageable area in the image.

[0097] Next, in step S14, the determination unit 215 determines whether the masking process is executed. At this time, the determination unit 215 determines whether the transmission permission flag stored in the memory 22 is valid. If the determination unit 215 determines that the transmission permission flag is valid, it determines that the masking process is executed. If the determination unit 215 determines that the transmission permission flag is not valid, it determines that the masking process is not executed.

[0098] Here, when it is determined that the masking process is not executed, that is, when it is determined that the transmission permission flag is invalid (step S14 “No”), the mask image is not transmitted, and the process proceeds to step S11 .

[0099] On the other hand, when it is determined that the mask processing is executed, that is, when it is determined that the transmission permission flag is effective (step S14 "Yes"), in step S15, the output unit 216 transmits the mask image created by the mask processing execution unit 213 to the remote terminal 3 via the communication unit 23. The remote terminal 3 receives the mask image transmitted by the image processing device 2. The remote terminal 3 displays the received mask image.

[0100] Next, in step S16, the output unit 216 determines whether to terminate the transmission of the mask image. The transmission of the mask image is terminated when the power of the camera 1 is turned off or when a button for terminating the transmission of the mask image provided in the image processing device 2 is pressed.

[0101] Here, when it is determined that the transmission of the mask image is to be terminated (Yes in step S16 ), in step S17 , the flag setting unit 214 invalidates the transmission permission flag stored in the memory 22 .

[0102] Next, in step S18 , the output unit 216 ends the transmission of the mask image.

[0103] On the other hand, when it is determined that the transmission of the mask image is not to be completed (No in step S16 ), the process returns to step S1 .

[0104] In this way, after performing mask processing to create a mask image that masks an area outside the photographable area in the image, it is determined whether the mask processing has been performed. Then, if it is determined that the mask processing has been performed, the created mask image is sent. Therefore, it is possible to prevent confidential information from being leaked.

[0105] (Implementation method 2)

[0106] In the first embodiment, after the masking process is initially executed, the transmission permission flag indicating that the mask image can be transmitted is automatically enabled. In contrast, in the second embodiment, after the masking process is initially executed, the created mask image is displayed, and an input indicating whether the masking process is normally executed by the operator is accepted. Then, when an input indicating that the masking process is normally executed by the operator is accepted, the transmission permission flag indicating that the mask image can be transmitted is enabled.

[0107] Figure 8 2A is a diagram showing an example of the configuration of an image processing device 2A in Embodiment 2. In Embodiment 2, the configuration of the work support system other than the image processing device 2A is the same as that of the work support system in Embodiment 1.

[0108] Figure 8 The image processing device 2A shown includes a processor 21A, a memory 22, a communication unit 23, an input receiving unit 24, and a display unit 25. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0109] The image acquisition unit 211 , the marker recognition unit 212 , the mask processing execution unit 213 , the mark setting unit 214A, the determination unit 215 , and the output unit 216 are implemented by the processor 21A.

[0110] The display unit 25 is, for example, a liquid crystal display device. After the mask processing execution unit 213 first executes the mask processing, the display unit 25 displays the mask image created by the mask processing execution unit 213. The operator visually checks whether the mask processing is normally executed by viewing the mask image displayed on the display unit 25.

[0111] The input receiving unit 24 is, for example, a keyboard, a mouse, or a touch panel. The input receiving unit 24 receives an input from the operator indicating whether the masking process is normally performed. The input receiving unit 24 includes a first button for indicating that the masking process is normally performed, and a second button for indicating that the masking process is not normally performed. The operator confirms the mask image displayed on the display unit 25. Then, the operator presses the first button when it is determined that the masking process is normally performed, and presses the second button when it is determined that the masking process is not normally performed.

[0112] In addition, the first button and the second button can be displayed on the display unit 25. The operator can use a mouse to move the cursor on the screen to the first button or the second button and click it. In addition, when the display unit 25 is a touch panel, the operator can touch either the first button or the second button on the screen.

[0113] When the flag setting unit 214A receives input from the operator indicating that the masking process has been performed normally, it enables the transmission permission flag indicating that the mask image can be transmitted. In addition, when the flag setting unit 214A receives input from the operator indicating that the masking process has not been performed normally, it disables the transmission permission flag.

[0114] The determination unit 215 determines whether the transmission permission flag is effective. If the determination unit 215 determines that the transmission permission flag is effective, it determines that the masking process is executed. If the determination unit 215 determines that the transmission permission flag is invalid, it determines that the masking process is not executed.

[0115] When the determination unit 215 determines that the mask process is not executed, the output unit 216 does not transmit the mask image.

[0116] Next, the job assisting process performed by the image processing apparatus 2A in Embodiment 2 of the present disclosure will be described.

[0117] Fig. 9 is a first flowchart for explaining the job assisting process of the image processing device 2A in the second embodiment of the present disclosure. Fig.10 This is a second flowchart for explaining the job assisting process of the image processing device 2A in the second embodiment of the present disclosure.

[0118] The processing of steps S21 to S27 is due to Figure 6 The processes of step S1 to step S7 shown are the same, and thus description thereof will be omitted.

[0119] Next, in step S28, the display unit 25 displays the mask image created by the mask processing execution unit 213. This allows the operator to be presented with the mask image obtained by performing the mask processing on the acquired image.

[0120] Next, in step S29, the input receiving unit 24 receives input from the operator of either the first button for indicating that the masking process was performed normally or the second button for indicating that the masking process was not performed normally. The operator confirms the displayed mask image and presses the first button when the masking process was performed normally, that is, when the confidential information in the masking image was masked. On the other hand, the operator presses the second button when the masking process was not performed normally, that is, when the confidential information in the masking image was not masked.

[0121] Next, in step S30 , the mark setting unit 214A determines whether or not the input accepting unit 24 has accepted the pressing of the first button.

[0122] Here, when it is determined that the first button is not pressed, that is, when it is determined that the second button is pressed (step S30: No), the process returns to step S21. In addition, at this time, the flag setting unit 214A invalidates the transmission permission flag stored in the memory 22. In addition, when the initial transmission permission flag is invalid, the flag setting unit 214A maintains the transmission permission flag invalid.

[0123] On the other hand, when it is determined that the first button has been pressed (Yes in step S30), in step S31, flag setting unit 214A enables the transmission permission flag stored in memory 22. After the transmission permission flag is enabled, the process proceeds to step S37.

[0124] On the other hand, when it is determined that the initialization flag is valid (Yes in step S22 ), in step S32 , the marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211 .

[0125] In addition, the processing of steps S32 to S41 is Figure 6 as well as Figure 7 The processes in steps S9 to S18 are the same, and thus their description is omitted.

[0126] In this way, since the operator can check whether the masking process has been normally performed by observing the displayed mask image, it is possible to reliably prevent confidential information from being leaked.

[0127] In addition, in the present embodiments 1 and 2, the mask processing execution unit 213 recognizes multiple marks contained in the image and executes mask processing to create a mask image that masks the area outside the photographable area surrounded by the multiple recognized marks in the image, but the present disclosure is not particularly limited to this, and the following first mask processing to third mask processing may also be executed.

[0128] In the first mask processing, the input receiving unit 24 may receive input from the operator for specifying a photographable area in the image when the image is first acquired. In this case, the photographable area may be specified by enclosing the photographable area in the image using the user interface. Then, the mask processing execution unit 213 may store the photographable image in the photographable area received by the input receiving unit 24 as an initial photographable image in the memory 22. Later, the mask processing execution unit 213 may also compare the initial photographable image with the image acquired this time, and determine the area in the image that is consistent with the initial photographable image as the photographable area. At this time, the mask processing execution unit 213 may extract multiple feature points of the initial photographable image, and extract multiple feature points of the image acquired this time, establish correspondence between the multiple feature points of the initial photographable image and the multiple feature points of the image acquired this time, and determine the area in the image that has the same feature points as the multiple feature points of the initial photographable image as the photographable area.

[0129] In addition, after initially storing the initial photographable image in the memory 22, the mask processing execution unit 213 can compare the photographable image in the image one frame ago with the image acquired this time for each frame, and estimate the region in the image that matches the photographable image one frame ago as the photographable region. Then, the mask processing execution unit 213 can perform estimation and determine the photographable region as the photographable region of the acquired image.

[0130] In the second mask processing, similar to the first mask processing, the input receiving unit 24 may receive an input from the operator for specifying a photographable area in the image when the image is first acquired. The camera 1 may further include a first sensor for detecting the position and posture of the camera 1. The first sensor is, for example, an acceleration sensor and a gyro sensor, and detects translational motion (x, y, z) and rotational motion (roll angle, pitch angle, yaw angle) of the camera 1 in three axes. The camera 1 may send the image to the image processing device 2 or 2A, and send the first sensing value representing the position and posture of the camera 1 detected by the first sensor to the image processing device 2 or 2A. The processor 21 or 21A may include a sensing value acquisition unit. The sensing value acquisition unit may acquire the first sensing value representing the position and posture of the camera 1 for each frame. Then, the mask processing execution unit 213 may store the photographable image in the photographable area received by the input receiving unit 24 as an initial photographable image in the memory 22, and store the first sensing value representing the position and posture of the camera 1 as a first initial sensing value in the memory 22.

[0131] Then, the mask processing execution unit 213 can calculate the current position and current posture of the camera 1 relative to the initial position and initial posture of the camera 1 represented by the first initial sensing value, and determine the photographable area based on the calculation result. The mask processing execution unit 213 can estimate the current photographing range of the camera 1 relative to the initial photographable image based on the initial position and initial posture of the camera 1 and the current position and current posture of the camera 1. The mask processing execution unit 213 can determine the area in the image obtained this time that is consistent with the initial photographable image as the photographable area from the current photographing range relative to the initial photographable image.

[0132] That is, the moving direction and moving distance from the initial position and initial posture of the camera 1 are calculated based on the first initial sensing value and the current first sensing value. Then, the current position and current posture of the camera 1 are estimated based on the calculated moving direction and moving distance. The current shooting range of the camera 1 relative to the initial image that can be shot is estimated based on the estimated current position and current posture of the camera 1, and the area in the image obtained this time that is consistent with the initial image that can be shot is determined as the shooting area.

[0133] In addition, the mask processing execution unit 213 may obtain the first sensing value one frame ago and the current first sensing value. The mask processing execution unit 213 may determine the area in the image obtained this time that is consistent with the image that can be captured one frame ago as the captureable area based on the current position and current posture of the camera 1 relative to the position and posture of the camera 1 one frame ago, which are estimated based on the first sensing value one frame ago and the current first sensing value.

[0134] In the third mask processing, similarly to the first mask processing, the input receiving unit 24 may receive an input from the operator for specifying a photographable area in the image when the image is first acquired. The camera 1 may further include a second sensor for detecting the distance from the camera 1 to the object for each pixel in the image. The second sensor is, for example, a depth sensor. The camera 1 may send a second sensing value representing the distance from the camera 1 to the object for each pixel in the image to the image processing device 2 or 2A together with the image. The mask processing execution unit 213 may store the photographable image in the photographable area in the image as an initial photographable image in the memory 22 when the image is first acquired, and store each distance from the camera 1 to the object for each pixel in the initial photographable image in the memory 22 as a second initial sensing value. The mask processing execution unit 213 may acquire the second initial sensing value and the second sensing value representing the distance from the camera 1 to the object for each pixel in the image acquired this time. Then, the mask processing execution unit 213 may determine, as the imageable area, a region where the second sensing values ​​of the plurality of pixels in the image acquired this time are equal to or less than the longest distance among the second initial sensing values.

[0135] The mask processing execution unit 213 may determine the imageable region using the first sensing value or the second sensing value. In addition, the mask processing execution unit 213 may determine the imageable region using both the first sensing value and the second sensing value. In this case, the mask processing execution unit 213 may calculate the logical AND of the imageable region determined using the first sensing value and the imageable region determined using the second sensing value.

[0136] Furthermore, the mask process execution unit 213 may combine any two of the first mask process, the second mask process, and the third mask process, or may combine all three.

[0137] In addition, in each of the above-mentioned embodiments, each component can be composed of dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. In addition, by recording the program in a recording medium and transferring it, or by transferring the program via a network, the program can be implemented by another independent computer system.

[0138] Part or all of the functions of the device involved in the embodiments of the present disclosure are typically implemented as LSI (Large Scale Integration) as an integrated circuit. These can be individually single-chip, or part or all of them can be single-chip. In addition, integrated circuitization is not limited to LSI, and can also be implemented using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connection and settings of the circuit unit inside the LSI.

[0139] Furthermore, a part or all of the functions of the apparatus according to the embodiments of the present disclosure may be realized by executing a program by a processor such as a CPU.

[0140] In addition, all the numbers used in the above description are exemplifications for specifically describing the present disclosure, and the present disclosure is not limited to the exemplified numbers.

[0141] In addition, the order of executing each step shown in the above flowchart is an example for specifically explaining the present disclosure, and other orders may be used within the scope of obtaining the same effect. In addition, part of the above steps may be executed simultaneously (in parallel) with other steps.

[0142] Industrial Applicability

[0143] The technology according to the present disclosure can prevent leakage of confidential information and is therefore useful as a technology for masking areas other than the imageable area in an image.

Claims

1. An information processing method is an information processing method in a computer. The information processing method performs the following processing: Get the image captured by the camera, performing mask processing to create a mask image that masks an area outside the imageable area in the image, determining whether the masking process has been performed, When it is determined that the mask processing has been performed, the mask image is transmitted.

2. The information processing method according to claim 1, wherein: The mask process: identifying a plurality of landmarks contained within the image, A mask image is created that masks an area outside the imageable area that is surrounded by the plurality of recognized markers in the image.

3. The information processing method according to claim 1, wherein: Furthermore, after the mask processing is initially performed, a flag indicating that the mask image can be sent is enabled. In determining whether the masking process has been executed, it is determined whether the flag has become effective, and if it is determined that the flag has become effective, it is determined that the masking process has been executed.

4. The information processing method according to claim 1, wherein: Furthermore, after the mask processing is initially performed, the created mask image is displayed. Furthermore, receiving an input of an instruction from an operator indicating whether the masking process has been performed normally, Furthermore, when receiving an input of an instruction from the operator indicating that the masking process has been normally performed, a flag indicating that the masking image can be transmitted is enabled, In determining whether the masking process has been executed, it is determined whether the flag has become effective, and if it is determined that the flag has become effective, it is determined that the masking process has been executed.

5. The information processing method according to claim 4, wherein: Furthermore, when receiving an input from the operator indicating that the masking process is not normally performed, the mark is invalidated. In determining whether the masking process has been executed, it is determined whether the flag has become effective, and if it is determined that the flag has become ineffective, it is determined that the masking process has not been executed. In the transmission of the mask image, if it is determined that the mask processing has not been performed, the mask image is not transmitted.

6. The information processing method according to any one of claims 3 to 5, wherein: Furthermore, when the transmission of the mask image is completed, the flag is disabled.

7. An information processing device comprising: an acquisition unit that acquires an image captured by a camera; a mask processing execution unit that executes a mask processing to create a mask image that masks an area outside the imageable area in the image; a determination unit that determines whether the masking process has been performed; and A transmitting unit transmits the mask image when it is determined that the mask processing has been performed.

8. An information processing program that causes a computer to function as follows: Get the image captured by the camera, performing mask processing to create a mask image that masks an area outside the imageable area in the image, determining whether the masking process has been performed, When it is determined that the mask processing has been performed, the mask image is transmitted.

Images