Information processing method, information processing device, and information processing program

Abstract

The image processing apparatus acquires an image captured by a camera, recognizes a plurality of markers included in the image, identifies an image-capturing area surrounded by the recognized plurality of markers when the plurality of markers are recognized, and identifies an image-capturing area surrounded by the recognized plurality of markers when the plurality of markers are not recognized. A region in an image acquired this time, which coincides with an image that can be captured in an image-capturing region specified in the past, is identified as an image-capturing region, a mask image is created in which a region outside the image-capturing region in the image is masked, and the mask image is output.

Description

Technical Field

[0001] The present disclosure relates to a technology for masking an area other than a photographable area in an image. Background Art

[0002] For example, Patent Document 1 discloses an image control device that sets a permitted image-capable area in a work area for image sharing with a remote assistant operator, and sends a masked image of the work area that masks the non-image-capable area to a remote terminal for the remote assistant operator. The existing image control device detects a mark contained in the image data, extracts mark information indicating the image-capable area from the detected mark, determines an image area in the image data that is judged to be outside the image-capable area as an image-unshareable area, masks the image-unshareable area, and generates a masked image.

[0003] However, in the above-mentioned existing technology, when multiple markers are not recognized, an image in which the cameraable area surrounded by the multiple markers is masked is output. Therefore, the operator at the scene may not receive assistance from the operator at a distance, and further improvement is needed.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-150866 Summary of the invention

[0007] The present disclosure is made to solve the above-mentioned problem, and an object of the present disclosure is to provide a technology that can output an image masked outside a photographable area surrounded by a plurality of markers even when the plurality of markers are not recognized.

[0008] The information processing method involved in the present disclosure is an information processing method in a computer, and the information processing method performs the following processing: obtaining an image taken by a camera, identifying multiple marks contained in the image, and when the multiple marks are identified, determining a photographic area surrounded by the multiple marks identified; when the multiple marks are not identified, determining an area in the image that is consistent with a photographic image within the photographic area determined in the past and obtained this time as the photographic area, creating a mask image that masks an area outside the photographic area in the image, and outputting the mask image.

[0009] According to the present disclosure, even when a plurality of markers are not recognized, an image in which the area other than the imageable region surrounded by the plurality of markers is masked can be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1It is a diagram showing an example of the configuration of the work support system in Embodiment 1 of the present disclosure.

[0011] Figure 2 This is a diagram showing an example of a work space in the first embodiment.

[0012] Figure 3 This is a diagram showing an example of the configuration of the image processing device in the first embodiment.

[0013] Figure 4 This is a diagram showing an example of an image captured by a camera in which all the signs are reflected in the image in the first embodiment.

[0014] Figure 5 This is a diagram showing an example of an initial photographable image in the first embodiment.

[0015] Figure 6 This means that in this embodiment 1, based on Figure 4 FIG. 1 is a diagram showing an example of a mask image created using the image shown.

[0016] Figure 7 This is a diagram showing an example of an image captured by a camera of the approach assistance object device in the first embodiment.

[0017] Figure 8 This means that in this embodiment 1, based on Figure 7 FIG. 1 is a diagram showing an example of a mask image created using the image shown.

[0018] Fig. 9 This is a diagram showing an example of an image captured by the camera whose imaging range is moved to the right in the first embodiment.

[0019] Fig.10 This means that in this embodiment 1, based on Fig. 9 FIG. 1 is a diagram showing an example of a mask image created using the image shown.

[0020] Fig.11 This is a first flowchart for explaining the job assisting process of the image processing device in the first embodiment of the present disclosure.

[0021] Fig.12 This is a second flowchart for explaining the job assisting process of the image processing device in the first embodiment of the present disclosure.

[0022] Fig.13 This is a diagram showing an example of the structure of the image processing device in the second embodiment.

[0023] Fig.14 This is a first flowchart for explaining the job assisting process of the image processing device in the second embodiment of the present disclosure.

[0024] Fig.15 This is a second flowchart for explaining the job assisting process of the image processing device in the second embodiment of the present disclosure.

[0025] Fig.16 This is a diagram showing an example of the structure of an image processing device in the third embodiment.

[0026] Fig.17 This is a diagram showing an example of an image captured by a camera when all of a plurality of markers are first recognized in the third embodiment.

[0027] Fig.18 This is a diagram showing an example of a mask image created when the camera approaches the auxiliary target device from the initial position in the third embodiment.

[0028] Fig.19 This is a diagram showing an example of a mask image created when the camera approaches the auxiliary target device from the initial position and the imaging range moves to the right in the third embodiment.

[0029] Fig. 20 This is a first flowchart for explaining the job assisting process of the image processing device in the third embodiment of the present disclosure.

[0030] Fig.21 This is a second flowchart for explaining the job assisting process of the image processing device in the third embodiment of the present disclosure.

[0031] Fig. 22 This is a third flowchart for explaining the job assisting process of the image processing device in the third embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] (Insights that form the basis of this disclosure)

[0033] In order for workers at the site to efficiently receive work assistance from workers at a remote location, it is effective to share images of the conditions at the site with the workers at the remote location.

[0034] When transmitting images to a remote location, confidential information may be reflected in the images. Therefore, in order to prevent confidential information from being leaked to the outside through image transmission, objects that can be image transmitted or objects that are confidential information are identified and the area corresponding to the confidential information in the image is masked.

[0035] For example, an operator at the scene configures multiple markers to surround an area where images can be transmitted, that is, an area that does not contain confidential information. By identifying multiple markers in the captured image, the area outside the imageable area surrounded by the identified multiple markers is masked, thereby preventing confidential information from being leaked. However, if part or all of the multiple markers are outside the camera's shooting range due to camera movement, the imageable area cannot be determined, and all pixels of the image will be masked, and the desired image may no longer be transmitted.

[0036] In the above-mentioned prior art, the multiple marks are two-dimensional codes with mark information. Therefore, in the existing image control device, if more than one mark among the multiple marks is located in the field of view of the camera, the mark information indicating the imageable area can be extracted from the detected one or more marks. Then, the existing image control device can determine the image area in the image data that is judged to be outside the imageable area as the non-shareable image area, mask the non-shareable image area, and generate a masked image.

[0037] However, in the prior art, when all the marks are outside the field of view of the camera, it is difficult to determine the imageable area. In addition, when the marks are not QR codes and some of the marks are outside the field of view of the camera, it is difficult to determine the imageable area.

[0038] In order to solve the above problems, the following technology is disclosed.

[0039] (1) An information processing method according to one aspect of the present disclosure is an information processing method in a computer, which obtains an image captured by a camera, identifies a plurality of marks contained in the image, and when the plurality of marks are identified, determines a photographable area surrounded by the plurality of identified marks; when the plurality of marks are not identified, determines an area in the image acquired this time that is consistent with a photographable image within the photographable area determined in the past as the photographable area, creates a mask image that masks an area outside the photographable area in the image, and outputs the mask image.

[0040] According to this structure, when multiple markers included in the image are recognized, the photographable area surrounded by the recognized multiple markers is determined. On the other hand, when multiple markers are not recognized, the area in the image obtained this time that is consistent with the photographable image in the photographable area determined in the past is determined as the photographable area. Then, a mask image is created to mask the area outside the photographable area in the image, and the mask image is output.

[0041] Therefore, even when the plurality of markers are not recognized, a mask image can be outputted that masks the area outside the imageable region surrounded by the plurality of markers. As a result, the operator at the site can receive efficient work assistance from the operator at a remote location.

[0042] (2) In the information processing method described in (1) above, when the imageable area is not determined during the creation of the mask image, the mask image may be created by masking the entirety of the image.

[0043] According to this configuration, when the imageable area is not determined, a mask image is created to mask the entire image, thereby reliably preventing an image including information to be kept confidential from being output.

[0044] (3) In the information processing method described in (1) or (2) above, it is also possible that, in determining the photographable area, when the multiple markers are not recognized, the photographable image within the photographable area determined in the past and the image obtained this time are compared, and the area within the image that is consistent with the photographable image is determined as the photographable area.

[0045] According to this configuration, when a plurality of markers are not recognized, the photographable area in the image acquired this time can be determined by performing matching processing between the photographable image in the photographable area determined in the past and the image acquired this time.

[0046] (4) In the information processing method described in (3) above, further, when the multiple marks are initially recognized, the photographic image within the photographic area within the image is used as an initial photographic image and stored in a memory. In determining the photographic area, when the multiple marks are not recognized, the initial photographic image and the image obtained this time are compared, and the area within the image that is consistent with the initial photographic image is determined as the photographic area.

[0047] According to this configuration, when a plurality of markers are not recognized, the imageable region in the image acquired this time can be determined by performing matching processing between the initial imageable image obtained when the plurality of markers are first recognized and the image acquired this time.

[0048] (5) In the information processing method described in (4) above, further, when the multiple markers are initially recognized, the first sensing value representing the position and posture of the camera is stored in the memory as the first initial sensing value, and further, when the photographable area is not determined, the first initial sensing value and the current first sensing value are obtained, and further, based on the current position and current posture of the camera relative to the initial position and initial posture of the camera estimated based on the first initial sensing value and the current first sensing value, the area within the image that is consistent with the initial photographable image and obtained this time is determined as the photographable area.

[0049] According to this structure, when an area within an image obtained this time is not determined as a photographable area and is consistent with a photographable image within a photographable area determined in the past, the photographable area within the image obtained this time can be determined based on the current position and current posture of the camera relative to the initial position and initial posture of the camera.

[0050] (6) In the information processing method described in (3) above, in determining the photographable area, when the multiple markers are not recognized, the area within the image obtained this time and which is consistent with the photographable image within the photographable area determined one frame ago is determined as the photographable area.

[0051] According to this configuration, when a plurality of markers are not recognized, the photographable area in the image acquired this time can be determined by performing matching processing between the photographable image in the photographable area determined one frame ago and the image acquired this time.

[0052] (7) In the information processing method described in (6) above, the photographic area estimated one frame ago and stored in the memory is compared with the image of the current frame obtained this time, and the area within the image that is consistent with the photographic image is estimated as the photographic area of ​​the current frame. The estimated photographic area of ​​the current frame is stored in the memory, and in determining the photographic area, when the multiple marks are not recognized, the estimated photographic area of ​​the current frame is determined as the photographic area.

[0053] According to this configuration, since the imageable region is estimated for each frame, when a plurality of markers are not recognized, the estimated imageable region of the current frame can be determined as the imageable region in the image acquired this time.

[0054] (8) In the information processing method described in (7) above, it is also possible to, further, obtain a first sensing value representing the position and posture of the camera for each frame, store the obtained first sensing value in the memory, and further, obtain the first sensing value one frame ago and the current first sensing value when the photographable area has not been determined, and further, determine the area within the image that is consistent with the photographable image one frame ago and is obtained this time as the photographable area based on the current position and current posture of the camera relative to the position and posture of the camera one frame ago, which are estimated based on the first sensing value one frame ago and the current first sensing value.

[0055] According to this structure, when an area within the image obtained this time is not determined as a photographable area and is consistent with a photographable image within the photographable area determined in the past, the photographable area within the image obtained this time can be determined based on the current position and current posture of the camera relative to the position and posture of the camera 1 frame ago.

[0056] (9) In the information processing method described in (1) or (2) above, it is also possible that, when the multiple marks are initially recognized, the capable image within the capable image area within the image of the initial frame is used as an initial capable image and stored in a memory, and then the image one frame ago and the image of the current frame are compared, and the change in the photographing range of the image one frame ago and the photographing range of the image of the current frame is calculated for each frame, and then the cumulative value of the change from the initial frame to the current frame is calculated. In determining the capable image area, when the multiple marks are not recognized, the capable image area of ​​the current frame is determined based on the initial capable image area and the cumulative value.

[0057] According to this configuration, the imageable area of ​​the current frame can be determined based on the change in the image capture range of the image from the initial frame to the current frame and the initial imageable image in the image of the initial frame.

[0058] (10) In the information processing method described in (1) or (2) above, further, when the multiple marks are initially recognized, the image within the image within the image that can be photographed is used as an initial image that can be photographed and is stored in a memory. Further, when the multiple marks are initially recognized, each distance from the camera to the object for each pixel in the initial image that can be photographed is stored in the memory as a second initial sensing value. Further, when the imageable area is not determined, the second initial sensing value and a second sensing value representing the distance from the camera to the object for each pixel in the image that is obtained this time are obtained. Further, the area with the second sensing values ​​of the multiple pixels in the image that is obtained this time and below the longest distance among the second initial sensing values ​​is determined as the imageable area.

[0059] According to this configuration, since an object located farther away than an object located in the initial photographable image is masked in the image acquired this time, it is possible to easily prevent an image including information to be kept confidential from being output.

[0060] In addition, the present disclosure can be implemented not only as an information processing method that performs the above-mentioned characteristic processing, but also as an information processing device having a characteristic structure corresponding to the characteristic processing performed by the information processing method. In addition, it can also be implemented as a computer program that causes a computer to perform the characteristic processing included in such an information processing method. Therefore, in the following other modes, the same effect as the above-mentioned information processing method can be achieved.

[0061] (11) The information processing device involved in other embodiments of the present disclosure comprises: an acquisition unit, which acquires an image captured by a camera; a recognition unit, which recognizes multiple markers contained in the image; a determination unit, which determines a photographable area surrounded by the multiple markers when the multiple markers are recognized, and determines, when the multiple markers are not recognized, an area in the image acquired this time that is consistent with a photographable image within the photographable area determined in the past as the photographable area; a creation unit, which creates a mask image that masks an area outside the photographable area in the image; and an output unit, which outputs the mask image.

[0062] (12) The information processing program involved in other embodiments of the present invention enables a computer to function as follows: obtain an image captured by a camera, identify multiple markers contained in the image, and when the multiple markers are identified, determine a photographic area surrounded by the multiple markers identified; when the multiple markers are not identified, determine an area in the image acquired this time that is consistent with a photographic image within the photographic area determined in the past as the photographic area, create a mask image that masks an area outside the photographic area in the image, and output the mask image.

[0063] (13) Other embodiments of the present disclosure involve a non-temporary computer-readable recording medium that records an information processing program, which enables a computer to function as follows: obtain an image captured by a camera, identify multiple markers contained in the image, and when the multiple markers are identified, determine a photographic area surrounded by the multiple markers that are identified; when the multiple markers are not identified, determine an area within the image that is consistent with a photographic image within the photographic area determined in the past and that is acquired this time as the photographic area, create a mask image that masks an area outside the photographic area in the image, and output the mask image.

[0064] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. In addition, in all embodiments, the various contents can also be combined.

[0065] (Implementation method 1)

[0066] Figure 1 It is a diagram showing an example of the configuration of the work support system in Embodiment 1 of the present disclosure. Figure 1 The illustrated work support system includes a camera 1 , an image processing device 2 , and a remote terminal 3 .

[0067] The camera 1 is, for example, a wearable camera that can be mounted on the head of an operator at a site. The operator is equipped with the camera 1 to perform work. At this time, the operator at the site uses the camera 1 to photograph the auxiliary object device located in the work space at the site, and receives work assistance from a remote operator. The remote operator observes the image transmitted from the site, and performs assistance in operating and repairing the auxiliary object device.

[0068] Figure 2This is a diagram showing an example of a work space in the first embodiment.

[0069] like Figure 2 As shown, an auxiliary object device 101 is set in a work space 100. An on-site operator uses a camera 1 to photograph the auxiliary object device 101. The image photographed by the camera 1 is sent to a remote terminal 3 operated by a remote operator. At this time, there is a possibility that confidential information is reflected in the photographed image. For this reason, in order to prevent the transmission of images containing confidential information, the area outside a specific work area is blocked by image processing (masking). The on-site operator attaches a plurality of marks 111, 112, 113, and 114 so as to surround the work area that does not contain confidential information of the auxiliary object device 101. The plurality of marks 111, 112, 113, and 114 are directly attached so as to surround the work area that can be photographed, for example, by means of an adhesive or a magnet.

[0070] The camera 1 transmits the captured image to the image processing device 2 via short-range wireless communication. The camera 1 transmits the image at a given frame rate to the image processing device 2. For this purpose, a dynamic image is transmitted from the camera 1 to the image processing device 2. In addition, the camera 1 is connected to the image processing device 2 wirelessly, but can also be connected to the image processing device 2 by wire.

[0071] In addition, the camera 1 may be a terminal equipped with a camera, such as a smartphone or a tablet computer. In addition, the camera 1 may be a digital camera.

[0072] The image processing device 2 is, for example, a personal computer operated by an on-site worker. The image processing device 2 and the remote terminal 3 are connected to each other so as to be able to communicate with each other via a network 4. The network 4 is, for example, the Internet.

[0073] Figure 3 This is a diagram showing an example of the configuration of the image processing device 2 in the first embodiment.

[0074] Figure 3 The image processing device 2 shown includes a processor 21 , a memory 22 , and a communication unit 23 .

[0075] The processor 21 is, for example, a CPU (Central Processing Unit). The image acquisition unit 211 , the marker recognition unit 212 , the region identification unit 213 , the mask image creation unit 214 , and the output unit 215 are implemented by the processor 21 .

[0076] The memory 22 is a storage device capable of storing various information, such as a RAM (Random Access Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), or a flash memory.

[0077] In addition, the image processing device 2 is equivalent to an example of an information processing device, the image acquisition unit 211 is equivalent to an example of an acquisition unit, the mark recognition unit 212 is equivalent to an example of a recognition unit, the area determination unit 213 is equivalent to an example of a determination unit, the mask image creation unit 214 is equivalent to an example of a creation unit, and the output unit 215 is equivalent to an example of an output unit.

[0078] The communication unit 23 receives the image transmitted from the camera 1. The communication unit 23 outputs the received image to the processor 21. The communication unit 23 receives the moving image transmitted at a given frame rate, and outputs the image to the processor 21 for each frame.

[0079] The image acquisition unit 211 acquires an image captured by the camera 1. The image acquisition unit 211 acquires an image for each frame.

[0080] The marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211. In the first embodiment, four markers 111, 112, 113, and 114 of different types are used. Figure 2 As shown, the four marks 111, 112, 113, and 114 are rectangular and have different patterns.

[0081] The worker at the site affixes the sign 111 to the upper left of the work area 150 that can be photographed, the sign 112 to the lower left of the work area 150 that can be photographed, the sign 113 to the upper right of the work area 150 that can be photographed, and the sign 114 to the lower right of the work area 150 that can be photographed. In addition, there is no particular determination as to where the four signs 111, 112, 113, and 114 are arranged. The four signs 111, 112, 113, and 114 may be arranged at any of the upper left, lower left, upper right, and lower right of the work area 150 that can be photographed.

[0082] The mark recognition unit 212 recognizes the four marks 111, 112, 113, 114 contained in the image by pattern matching. That is, the mark recognition unit 212 recognizes the four marks 111, 112, 113, 114 in the image by comparing the patterns of the multiple marks pre-stored in the memory 22 with the patterns of the four marks 111, 112, 113, 114 contained in the image.

[0083] In addition, the sign recognition unit 212 can recognize not only a sign photographed from the front but also a sign photographed from an oblique direction.

[0084] Furthermore, in the first embodiment, four markers are used, but the present disclosure is not particularly limited thereto, and three markers may be used, or five or more markers may be used.

[0085] When the marker recognition unit 212 recognizes a plurality of markers 111, 112, 113, and 114, the region determination unit 213 determines a photographable region surrounded by the recognized plurality of markers 111, 112, 113, and 114. When the marker recognition unit 212 does not recognize a plurality of markers 111, 112, 113, and 114, the region determination unit 213 determines a region within the image acquired this time that matches a photographable image within the photographable region determined in the past as the photographable region.

[0086] That is, when all of the multiple markers 111, 112, 113, and 114 are recognized, the area determination unit 213 determines the imageable area surrounded by the recognized multiple markers 111, 112, 113, and 114. In addition, when at least one of the multiple markers 111, 112, 113, and 114 is not recognized, the area determination unit 213 determines the area in the image obtained this time that is consistent with the imageable image in the imageable area determined in the past as the imageable area.

[0087] When four marks 111, 112, 113, and 114 are identified, the area determination unit 213 determines a photographable area by connecting the upper left vertex of the upper left mark 111, the lower left vertex of the lower left mark 112, the upper right vertex of the upper right mark 113, and the lower right vertex of the lower right mark 114 with straight lines.

[0088] In addition, when the four marks 111, 112, 113, and 114 are recognized, the area determination unit 213 may determine a photographable area by connecting the lower right vertex of the upper left mark 111, the upper right vertex of the lower left mark 112, the lower left vertex of the upper right mark 113, and the upper left vertex of the lower right mark 114 with a straight line. In addition, when the four marks 111, 112, 113, and 114 are recognized, the area determination unit 213 may determine a photographable area by connecting the center points of the four marks 111, 112, 113, and 114 with a straight line.

[0089] Furthermore, even when one of the four markers is not recognized, the region specifying unit 213 can predict the position of the one unrecognized marker and specify the imageable region surrounded by the four markers.

[0090] Furthermore, when a plurality of markers are not recognized, the region specifying unit 213 compares a previously specified photographable image within the photographable region with the image acquired this time, and specifies a region within the image that matches the photographable image as the photographable region.

[0091] More specifically, when multiple marks are initially recognized, the region determination unit 213 stores the photographable image in the photographable area in the image as the initial photographable image in the memory 22. Then, when multiple marks are not recognized, the region determination unit 213 compares the initial photographable image with the image obtained this time, and determines the area in the image that is consistent with the initial photographable image as the photographable area. At this time, the region determination unit 213 extracts multiple feature points of the initial photographable image, and extracts multiple feature points of the image obtained this time, establishes a correspondence between the multiple feature points of the initial photographable image and the multiple feature points of the image obtained this time, and determines the area in the image that has the same feature points as the multiple feature points of the initial photographable image as the photographable area.

[0092] The mask image generator 214 generates a mask image that masks an area outside the imageable area determined by the image area determination unit 213. If the imageable area is not determined by the area determination unit 213, the mask image generator 214 may generate a mask image that masks the entire image.

[0093] The memory 22 stores an initial captureable image determined when a plurality of markers are first recognized.

[0094] The output unit 215 outputs the mask image created by the mask image creating unit 214. The output unit 215 outputs the mask image to the communication unit 23. The communication unit 23 transmits the mask image to the remote terminal 3.

[0095] Figure 4 1 is a diagram showing an example of an image 121 captured by the camera 1 in the first embodiment so that all the signs are reflected. Figure 5 1 is a diagram showing an example of the initial photographable image 171 in the first embodiment. Figure 4 In order to make the description easier to understand, the symbols are shown larger than their actual size.

[0096] The image 121 includes all of the four markers 111, 112, 113, and 114. The marker recognition unit 212 recognizes the four markers 111, 112, 113, and 114 included in the image 121 acquired by the image acquisition unit 211. When all of the four markers 111, 112, 113, and 114 are recognized, the region determination unit 213 determines the imageable region 131 surrounded by the recognized four markers 111, 112, 113, and 114. In addition, when the four markers 111, 112, 113, and 114 are initially recognized, the region determination unit 213 stores the imageable image 161 in the imageable region 131 in the image 121 as the initial imageable image 171 in the memory 22.

[0097] like Figure 5As shown, the initial photographable image 171 is an image obtained when all of the four markers 111 , 112 , 113 , and 114 are first recognized, and is an image of the photographable region 131 surrounded by the four markers 111 , 112 , 113 , and 114 .

[0098] Figure 6 This means that in this embodiment 1, based on Figure 4 FIG. 1 is a diagram showing an example of a mask image 141 created from the image 121 shown.

[0099] The mask image creation unit 214 creates a mask image 141 that masks the area 132 outside the imageable area 131 in the image 121. The mask image creation unit 214 performs mask processing on the area 132 outside the imageable area 131 in the image 121. The mask processing may be, for example, a process of filling the area 132 with a given color such as black, white, or gray, a process of blurring the area 132, or a process of mosaicking the area 132. Figure 6 In the mask image 141 shown, a photographable image 161 within the photographable region 131 can be visually recognized, and a region 132 outside the photographable region 131 is concealed.

[0100] Figure 7 This is a diagram showing an example of an image 122 captured by the camera 1 of the approach assistance object device 101 in the first embodiment.

[0101] When the camera 1 approaches the auxiliary object device 101, the shooting range of the camera 1 changes to Figure 4 area 151. In this case, the four markers 111, 112, 113, and 114 are no longer included in the image 122. The image 122 does not contain the four markers 111, 112, 113, and 114. For this reason, the marker recognition unit 212 cannot recognize the markers from the image 122 acquired by the image acquisition unit 211. When at least one of the four markers 111, 112, 113, and 114 is not recognized, the area determination unit 213 compares the initial photographable image 171 and the image 122 acquired this time, and determines the area within the image 122 that is consistent with the initial photographable image 171 as the photographable area. Figure 7 In the example, since the camera 1 is close to the auxiliary object device 101, the four marks 111, 112, 113, and 114 are not included in the image 122. However, the image 122 obtained this time is Figure 5 The region 191 of the initial captureable image 171 shown matches. Therefore, the region specifying unit 213 specifies the entire image 122 acquired this time as the captureable region 131 .

[0102] Figure 8 This means that in this embodiment 1, based on Figure 7 FIG. 1 is a diagram showing an example of a mask image 142 created from the image 122 shown.

[0103] The mask image creation unit 214 creates a mask image 142 that masks an area 132 outside the imageable area 131 in the image 122. Figure 8 In the mask image 142 shown, since the image 122 acquired this time is entirely the imageable region 131 , the mask image 142 is entirely visible.

[0104] Fig. 9 This is a diagram showing an example of an image 123 captured by the camera 1 whose imaging range is moved to the right in the first embodiment.

[0105] When the shooting range of camera 1 moves to the right, the shooting range of camera 1 changes to Figure 4 area 152. In this case, two marks 111, 112 out of the four marks 111, 112, 113, 114 are no longer included in the image 122. The image 123 does not contain two marks 111, 112 out of the four marks 111, 112, 113, 114. For this reason, the mark recognition unit 212 recognizes the two marks 113, 114 included in the image 123 acquired by the image acquisition unit 211, and does not recognize the two marks 111, 112 not included in the image 123. In the case where at least one of the four marks 111, 112, 113, 114 is not recognized, the area determination unit 213 compares the initial photographable image 171 with the image 123 acquired this time, and determines the area 181 in the image 123 that is consistent with the initial photographable image 171 as the photographable area 131. Fig. 9 In the above example, since the image capturing range of the camera 1 moves to the right, the two marks 111 and 112 are not included in the image 123. However, the area 181 in the image 123 obtained this time is different from the area 182 in the image 123. Figure 5 The region 192 of the initial photographable image 171 shown matches. Therefore, the region specifying unit 213 specifies the region 181 in the image 123 that matches the initial photographable image 171 as the photographable region 131.

[0106] Fig.10 This means that in this embodiment 1, based on Fig. 9 FIG. 1 is a diagram showing an example of a mask image 143 created from the image 123 shown.

[0107] The mask image creation unit 214 creates a mask image 143 that masks the area 132 outside the imageable area 131 in the image 123. The mask image creation unit 214 performs mask processing on the area 132 outside the imageable area 131 in the image 123. Fig.10In the mask image 143 shown, the image 161 that can be captured in the imageable area 131 becomes visible, and the area 132 outside the imageable area 131 is hidden.

[0108] like Fig.10 As shown, even if a part of the four markers 111 , 112 , 113 , and 114 cannot be recognized, the entire acquired image 123 is not masked, and a mask image 143 that does not mask the imageable area 131 can be created.

[0109] In the first embodiment, the image processing device 2 is installed on-site, but the present disclosure is not particularly limited thereto, and the image processing device 2 may be a server installed on the cloud. In this case, the image processing device 2 and the camera 1 are connected to each other via the network 4 so as to be able to communicate with each other.

[0110] The remote terminal 3 is, for example, a personal computer, a smartphone, or a tablet computer operated by a remote operator. The remote terminal 3 receives the mask image sent by the image processing device 2. The remote terminal 3 displays the received mask image. The image processing device 2 sends the mask image to the remote terminal 3 at a given frame rate. For this purpose, a dynamic image is sent from the image processing device 2 to the remote terminal 3.

[0111] A remote worker can observe the mask image displayed on the remote terminal 3 and assist the work of the on-site worker. In addition, the image processing device 2 and the remote terminal 3 can transmit and receive not only image data but also audio data to and from each other.

[0112] Next, the job assisting process of the image processing device 2 in the first embodiment of the present disclosure will be described.

[0113] Fig.11 is a first flowchart for explaining the operation assisting process of the image processing device 2 in the first embodiment of the present disclosure. Fig.12 This is a second flowchart for explaining the job assisting process of the image processing device 2 in the first embodiment of the present disclosure.

[0114] First, in step S1, the image acquisition unit 211 acquires an image captured by the camera 1. When the camera 1 is powered on or a button provided in the image processing device 2 for starting transmission of a mask image is pressed, an image is acquired from the camera 1.

[0115] Next, in step S2, the flag recognition unit 212 determines whether the initialization flag stored in the memory 22 is valid (on). Here, when it is determined that the initialization flag is not valid, that is, when it is determined that the initialization flag is invalid (off) (step S2 "No"), in step S3, the flag recognition unit 212 recognizes a plurality of flags included in the image acquired by the image acquisition unit 211.

[0116] Next, in step S4, the region determination unit 213 determines whether all the markers are recognized by the marker recognition unit 212. In addition, the number of the plurality of markers used in image recognition is predetermined. The memory 22 stores the number of the plurality of markers in advance. In the first embodiment, the number of the plurality of markers is 4. The region determination unit 213 determines whether the marker recognition unit 212 recognizes 4 markers.

[0117] Here, when it is determined that all the markers are not recognized, that is, when it is determined that at least one of the plurality of markers is not recognized (No in step S4 ), the process returns to step S1 .

[0118] On the other hand, when it is determined that all the markers are recognized (Yes in step S4 ), in step S5 , the region specifying unit 213 specifies the imageable region surrounded by the plurality of markers recognized by the marker recognizing unit 212 .

[0119] Next, in step S6 , the area specifying unit 213 stores the photographable image within the photographable area in the image in the memory 22 as an initial photographable image.

[0120] Next, in step S7, the area identification unit 213 validates the initialization flag stored in the memory 22. After the initialization flag is validated, the process returns to step S1.

[0121] On the other hand, when it is determined that the initialization flag is valid (Yes in step S2 ), in step S8 , the marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211 .

[0122] Next, in step S9 , the area identification unit 213 determines whether all the markers have been recognized by the marker recognition unit 212 .

[0123] Here, when it is determined that all the marks are not recognized, that is, when it is determined that at least one of the multiple marks is not recognized (step S9 "No"), in step S10, the area determination unit 213 determines the photographable area by matching the initial photographable image with the image obtained this time. That is, the area determination unit 213 compares the initial photographable image with the image obtained this time, and determines the area in the image that matches the initial photographable image as the photographable area. In addition, when there is no area in the image that matches the initial photographable image, the area determination unit 213 does not determine the photographable area.

[0124] Next, in step S11, the region specifying unit 213 determines whether the imageable region has been specified. If it is determined that the imageable region has been specified (step S11: Yes), the process proceeds to step S14.

[0125] On the other hand, when it is determined that the imageable area has not been determined (No in step S11), in step S12, the area determination unit 213 invalidates the initialization flag stored in the memory 22. After the initialization flag is invalidated, the process returns to step S1.

[0126] On the other hand, when it is determined that all the markers are recognized (Yes in step S9 ), in step S13 , the region specifying unit 213 specifies the imageable region surrounded by the plurality of markers recognized by the marker recognizing unit 212 .

[0127] Next, in step S14 , the mask image creation unit 214 creates a mask image by performing a masking process on the area outside the imageable area specified by the in-image area specification unit 213 .

[0128] Next, in step S15, the output unit 215 transmits the mask image created by the mask image creating unit 214 to the remote terminal 3 via the communication unit 23. The remote terminal 3 receives the mask image transmitted by the image processing device 2. The remote terminal 3 displays the received mask image.

[0129] Next, in step S16, the output unit 215 determines whether to terminate the transmission of the mask image. When the power of the camera 1 is turned off or a button for terminating the transmission of the mask image provided in the image processing device 2 is pressed, the transmission of the mask image is terminated.

[0130] Here, when it is determined that the transmission of the mask image is to be terminated (Yes in step S16 ), the output unit 215 terminates the transmission of the mask image in step S17 .

[0131] On the other hand, when it is determined that the transmission of the mask image is not to be completed (No in step S16 ), the process returns to step S1 .

[0132] In this way, when multiple markers included in the image are recognized, the photographable area surrounded by the recognized multiple markers is determined. On the other hand, when multiple markers are not recognized, the area in the image obtained this time that is consistent with the photographable image in the photographable area determined in the past is determined as the photographable area. Then, a mask image is created that masks the area outside the photographable area in the image, and the mask image is output.

[0133] Therefore, even when the plurality of markers are not recognized, a mask image can be outputted that masks the area outside the imageable region surrounded by the plurality of markers. As a result, the operator at the site can receive efficient work assistance from the operator at a remote location.

[0134] (Implementation method 2)

[0135] In the above-mentioned embodiment 1, when multiple marks are initially recognized, the photographic image within the photographic area in the image is taken as the initial photographic image and stored in the memory 22. Thereafter, when multiple marks are not recognized from the acquired image, the initial photographic image and the image acquired this time are compared, and the area within the image that is consistent with the initial photographic image is determined as the photographic area.

[0136] In contrast, in the second embodiment, when a plurality of marks are initially recognized, a photographable image in the photographable area in the image is taken as an initial photographable image and stored in the memory 22. Thereafter, the photographable image in the image one frame ago and the image acquired this time are compared for each frame, and the area in the image that matches the photographable image one frame ago is estimated as the photographable area. Then, if a plurality of marks are not recognized from the acquired image, the estimated photographable area is determined as the photographable area of ​​the acquired image.

[0137] Fig.13 2A is a diagram showing an example of the configuration of an image processing device 2A in Embodiment 2. In Embodiment 2, the configuration of the work support system other than the image processing device 2A is the same as that of the work support system in Embodiment 1.

[0138] Fig.13 The image processing device 2A shown includes a processor 21A, a memory 22, and a communication unit 23. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0139] The image acquisition unit 211 , the marker recognition unit 212 , the region identification unit 213A, the mask image creation unit 214 , the output unit 215 , and the region estimation unit 216 are implemented by the processor 21A.

[0140] The region estimation unit 216 compares the photographable image in the photographable region estimated one frame ago and stored in the memory 22 with the image of the current frame acquired this time, and estimates the region in the image that matches the photographable image as the photographable region of the current frame. The region estimation unit 216 estimates the photographable region in the image of the current frame for each frame. The region estimation unit 216 stores the estimated photographable image in the photographable region of the current frame in the memory 22.

[0141] The memory 22 stores a possible photographic image estimated from an image one frame before and a possible photographic image estimated from an image of a current frame.

[0142] If multiple markers are not recognized, the region determination unit 213A determines the region in the image obtained this time that is consistent with the photographable image in the photographable region determined one frame ago as the photographable region. If multiple markers are not recognized, the region determination unit 213A determines the photographable region based on the photographable region of the current frame estimated by the region estimation unit 216.

[0143] Next, the job assisting process performed by the image processing apparatus 2A in Embodiment 2 of the present disclosure will be described.

[0144] Fig.14 is a first flowchart for explaining the job assisting process of the image processing device 2A in the second embodiment of the present disclosure. Fig.15 This is a second flowchart for explaining the job assisting process of the image processing device 2A in the second embodiment of the present disclosure.

[0145] The processing of steps S21 to S26 is due to Fig.11 The processes in steps S1 to S6 shown are the same, and thus description thereof will be omitted.

[0146] Next, in step S27 , the mask image creation unit 214 creates a mask image by performing a masking process on the area outside the imageable area specified by the in-image area specifying unit 213A.

[0147] Next, in step S28, the region identification unit 213A validates the initialization flag stored in the memory 22. After the initialization flag is validated, the process proceeds to step S37.

[0148] On the other hand, when it is determined that the initialization flag is effective (step S22 "Yes"), in step S29, the region determination unit 213A estimates the photographable region by matching the photographable image estimated one frame ago and stored in the memory 22 with the image of the current frame acquired this time. That is, the region determination unit 213A compares the photographable image one frame ago and the image of the current frame acquired this time, and estimates the region in the image of the current frame that is consistent with the photographable image one frame ago as the photographable region. In addition, when there is no region in the image of the current frame that is consistent with the photographable image one frame ago, the region determination unit 213A does not estimate the photographable region.

[0149] In addition, when the current frame is the next frame of the image in which all the marks are initially recognized, the area estimation unit 216 compares the initial photographic image stored in the memory 22 with the image of the current frame acquired this time, and estimates the area in the image of the current frame that matches the initial photographic image as the photographic area of ​​the current frame. Thereafter, the area estimation unit 216 performs matching processing between the photographic image of one frame before and the image of the current frame for each frame, and estimates the area in the image of the current frame that matches the photographic image of one frame before as the photographic area of ​​the current frame.

[0150] When estimating the possible photographic area of ​​the current frame, the area estimation unit 216 stores the possible photographic images within the estimated possible photographic area of ​​the current frame in the memory 22 .

[0151] Next, in step S30 , the marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211 .

[0152] Next, in step S31 , the region identification unit 213A determines whether all the markers have been recognized by the marker recognition unit 212 .

[0153] Here, when it is determined that all the marks are not recognized, that is, when it is determined that at least one of the multiple marks is not recognized (step S31 "No"), in step S32, the area determination unit 213A determines whether the imageable area of ​​the current frame is estimated by the area estimation unit 216. In addition, when the imageable image of the current frame is stored in the memory 22, the area determination unit 213A may determine that the imageable area of ​​the current frame is estimated. When the imageable image of the current frame is not stored in the memory 22, the area determination unit 213A may determine that the imageable area of ​​the current frame is not estimated.

[0154] Here, when it is determined that the photographable area of ​​the current frame has been estimated (step S32: Yes), in step S33, the area determination unit 213A determines the photographable area based on the photographable area of ​​the current frame estimated by the area estimation unit 216. That is, the area determination unit 213A determines the photographable area of ​​the current frame estimated by the area estimation unit 216 as the photographable area of ​​the image acquired this time.

[0155] On the other hand, when it is determined that the imageable area of ​​the current frame has not been estimated (No in step S32), in step S34, the area identification unit 213A invalidates the initialization flag stored in the memory 22. After the initialization flag is invalidated, the process returns to step S21.

[0156] On the other hand, when it is determined that all the markers are recognized (Yes in step S31 ), in step S35 , the region specifying unit 213A specifies the imageable region surrounded by the plurality of markers recognized by the marker recognizing unit 212 .

[0157] The processing of steps S36 to S39 is due to Fig.12 The processes of step S14 to step S17 shown are the same, and thus the description thereof will be omitted.

[0158] As in the first embodiment, in the matching process between the initial photographic image when a plurality of marks are first recognized and the image of the current frame, if the movement amount of the camera 1 from the initial position is large, the matching process may become difficult and the accuracy of determining the photographic area may be reduced.

[0159] In contrast, in the second embodiment, since the matching process between the photographable image in the photographable area one frame before and the image of the current frame is performed for each frame, the accuracy of specifying the photographable area can be improved.

[0160] In addition, in the second embodiment, when a plurality of marks are initially recognized, the region determination unit 213A may store the photographable image within the photographable region within the image of the initial frame as the initial photographable image in the memory 22. In addition, the region determination unit 213A compares the image one frame ago and the image of the current frame, and calculates the amount of change between the photographing range of the image one frame ago and the photographing range of the image of the current frame for each frame. In addition, the amount of change, for example, indicates the moving direction and amount of the photographing range of the current frame relative to the photographing range one frame ago. In addition, the region determination unit 213A may calculate the cumulative value of the amount of change from the initial frame to the current frame. The cumulative value is stored in the memory 22. In the case where a plurality of marks are not recognized, the region determination unit 213A may determine the photographable region of the current frame based on the initial photographable region and the cumulative value.

[0161] In this case, the change amount of the image capture range from the initial frame to the current frame is known from the cumulative value. Therefore, the image captureable area in the image of the current frame can be determined based on the initial image captureable area in the image of the initial frame and the cumulative value.

[0162] (Implementation method 3)

[0163] In implementation mode 3, a first sensing value representing the position and posture of the camera 1 is obtained, and when the photographic area is not determined by matching the initial photographic image with the image obtained this time, or when the photographic area of ​​the current frame is not determined by matching the photographic image of each frame with the image of the current frame obtained this time, the first sensing value obtained is used to determine the photographic area.

[0164] Fig.16 1 is a diagram showing an example of the configuration of the image processing device 2B in the third embodiment. In the third embodiment, the configuration of the work support system other than the camera 1 and the image processing device 2B is the same as that of the work support system in the first embodiment.

[0165] The camera 1 also includes a first sensor for detecting the position and posture of the camera 1. The first sensor is, for example, an acceleration sensor and a gyro sensor, and detects translational motion (x, y, z) and rotational motion (roll angle, pitch angle, yaw angle) of the camera 1 in three axes. The camera 1 sends an image to the image processing device 2, and sends a first sensed value indicating the position and posture of the camera 1 detected by the first sensor to the image processing device 2.

[0166] Fig.16 The image processing device 2B shown includes a processor 21B, a memory 22, and a communication unit 23. In the third embodiment, the same components as those in the first and second embodiments are denoted by the same reference numerals, and description thereof will be omitted.

[0167] The image acquisition unit 211 , the marker recognition unit 212 , the region identification unit 213B, the mask image creation unit 214 , the output unit 215 , the region estimation unit 216 , and the sensing value acquisition unit 217 are implemented by the processor 21B.

[0168] The communication unit 23 receives the image and the first sensing value sent by the camera 1. The communication unit 23 outputs the received image and the first sensing value to the processor 21. The communication unit 23 receives the dynamic image sent at a given frame rate and the first sensing value sent at a given sampling period, and outputs the image for each frame and the first sensing value for each sampling to the processor 21.

[0169] The sensing value acquisition unit 217 acquires the first sensing value representing the position and posture of the camera 1 for each frame. Generally, the sampling period of sensors such as gyro sensors is higher than the frame rate of the image. By acquiring sensing values ​​at a high sampling period, high-frequency changes can also be detected. For this reason, the first sensor acquires the first sensing value at the highest possible sampling period and performs signal processing. For example, the signal processing of the gyro sensor is a process of removing drift noise and a process of integrating the angular velocity to convert it into an angle. Then, the sensing value acquisition unit 217 acquires the first sensing value whose time coincides with or is closest to the image frame.

[0170] If multiple markers are not recognized, the region determination unit 213B determines the region in the image obtained this time that is consistent with the photographable image in the photographable region determined one frame ago as the photographable region. If multiple markers are not recognized, the region determination unit 213B determines the photographable region based on the photographable region of the current frame estimated by the region estimation unit 216.

[0171] When a plurality of marks are initially recognized, the region determination unit 213B stores the photographable image in the photographable region in the image as an initial photographable image in the memory 22, and stores the first sensing value representing the position and posture of the camera 1 as a first initial sensing value in the memory 22. Then, when a plurality of marks are not recognized, the region determination unit 213B compares the initial photographable image with the image obtained this time, and determines the region in the image that matches the initial photographable image as the photographable region.

[0172] Furthermore, when the photographable area is not determined based on the photographable area of ​​the current frame estimated by the area estimating unit 216, the area determining unit 213B acquires the first initial sensing value and the current first sensing value stored in the memory 22. The area determining unit 213B determines the area within the image that is consistent with the initial photographable image and is obtained this time as the photographable area, based on the current position and current posture of the camera 1 estimated based on the first initial sensing value and the current first sensing value relative to the initial position and initial posture of the camera 1.

[0173] The area determination unit 213B calculates the current position and current posture of the camera 1 relative to the initial position and initial posture of the camera 1 represented by the first initial sensing value, and determines the photographable area based on the calculation result. The area determination unit 213B estimates the current photographing range of the camera 1 relative to the initial photographable image based on the initial position and initial posture of the camera 1 and the current position and current posture of the camera 1. The area determination unit 213B determines the area in the image that is consistent with the initial photographable image and obtained this time from the current photographing range relative to the initial photographable image as the photographable area.

[0174] That is, the moving direction and moving distance from the initial position and initial posture of the camera 1 are calculated based on the first initial sensing value and the current first sensing value. Then, the current position and current posture of the camera 1 are estimated based on the calculated moving direction and moving distance. The current shooting range of the camera 1 relative to the initial image that can be shot is estimated based on the estimated current position and current posture of the camera 1, and the area in the image that is consistent with the initial image that can be shot and obtained this time is determined as the imageable area.

[0175] The memory 22 stores an initial photographable image determined when a plurality of markers are first recognized. In addition, the memory 22 stores a first initial sensing value acquired when a plurality of markers are first recognized.

[0176] Fig.17 1 is a diagram showing an example of an image 121 captured by the camera 1 when all of the plurality of markers are first recognized in the third embodiment. Fig.17 In order to make the description easier to understand, the symbols are shown larger than their actual size.

[0177] The image 121 includes all of the four markers 111, 112, 113, and 114. The marker recognition unit 212 recognizes the four markers 111, 112, 113, and 114 included in the image 121 acquired by the image acquisition unit 211. When all of the four markers 111, 112, 113, and 114 are recognized, the region determination unit 213B determines the imageable region 131 surrounded by the recognized four markers 111, 112, 113, and 114. Furthermore, when the four markers 111, 112, 113, and 114 are recognized for the first time, the region determination unit 213B stores the imageable image 161 in the imageable region 131 in the image 121 as an initial imageable image 171 in the memory 22, and stores the first sensing value representing the position and posture of the camera 1 in the memory 22 as a first initial sensing value.

[0178] Fig.18 This is a diagram showing an example of a mask image 144 created when the camera 1 approaches the support object device 101 from the initial position in the third embodiment.

[0179] The initial position and shooting range of camera 1 in the initial posture are Fig.17 When the camera 1 approaches the auxiliary object device 101 from the initial position, the shooting range of the camera 1 captured at the current position and current posture of the camera 1 changes to Fig.17 The area identification unit 213B estimates the current imaging range of the camera 1 that is initially capable of capturing an image, based on the initial position and initial posture of the camera 1 and the current position and current posture of the camera 1 .

[0180] The mask image creation unit 214 creates a mask image 144 that masks the area outside the imageable area 131 in the image acquired this time. Fig.18 In the mask image 144 shown, since the entire image acquired this time is the imageable region 131 , the entire mask image 144 can be visually recognized.

[0181] Fig.19 This is a diagram showing an example of a mask image 145 created when the camera 1 approaches the support object device 101 from the initial position and the imaging range moves to the right in the third embodiment.

[0182] The initial position and shooting range of camera 1 in the initial posture are Fig.17 When the camera 1 approaches the auxiliary object device 101 from the initial position and the shooting range moves to the right, the shooting range of the camera 1 captured at the current position and current posture of the camera 1 changes to Fig.17 The area identification unit 213B estimates the current imaging range of the camera 1 that is initially capable of capturing an image, based on the initial position and initial posture of the camera 1 and the current position and current posture of the camera 1 .

[0183] The mask image creation unit 214 creates a mask image 145 that masks the area 132 outside the imageable area 131 in the image acquired this time. The mask image creation unit 214 performs mask processing on the area 132 outside the imageable area 131 in the image acquired this time. Fig.19 In the mask image 145 shown, the image 161 that can be captured in the imageable area 131 becomes visible, and the area 132 outside the imageable area 131 is hidden.

[0184] Next, the job assisting process of the image processing device 2B in Embodiment 3 of the present disclosure will be described.

[0185] Fig. 20 is a first flowchart for explaining the job assisting process of the image processing device 2B in the third embodiment of the present disclosure. Fig.21 is a second flowchart for explaining the job assisting process of the image processing device 2B in the third embodiment of the present disclosure. Fig. 22 This is a third flowchart for explaining the job assisting process of the image processing device 2B in the third embodiment of the present disclosure.

[0186] The processing of step S41 is due to Fig.14 The processing of step S21 shown is the same, so the description is omitted.

[0187] Next, in step S42 , the sensing value acquisition unit 217 acquires a first sensing value representing the position and posture of the camera 1 . The sensing value acquisition unit 217 acquires the first sensing value measured by the camera 1 .

[0188] The processing of steps S43 to S48 is due to Fig.14 The processes of step S22 to step S27 shown are the same, and thus the description thereof is omitted.

[0189] Next, in step S49 , the region identification unit 213B stores the first sensing value representing the position and posture of the camera 1 when all the markers are first recognized in the memory 22 as the first initial sensing value.

[0190] Next, in step S50, the region identification unit 213B validates the initialization flag stored in the memory 22. After the initialization flag is validated, the process proceeds to step S62.

[0191] The processing of steps S51 to S54 is due to Fig.14 as well as Fig.15 The processes of step S29 to step S32 shown are the same, and thus the description thereof is omitted.

[0192] Here, if it is determined that the imageable area of ​​the current frame has been estimated (step S54: Yes), the process proceeds to step S59. Fig.15 The processing of step S33 shown is the same, so the description is omitted.

[0193] On the other hand, when it is determined that the imageable area of ​​the current frame has not been estimated (No in step S54 ), in step S55 , the area identification unit 213B acquires the first initial sensing value stored in the memory 22 and the current first sensing value acquired by the sensing value acquisition unit 217 .

[0194] Next, in step S56, the area determination unit 213B and the area determination unit 213B determine the area within the image that is consistent with the initial photographable image and obtained this time as the photographable area based on the current position and current posture of the camera 1 relative to the initial position and initial posture of the camera 1 estimated based on the first initial sensing value and the current first sensing value.

[0195] Next, in step S57 , the area identification unit 213B determines whether the imageable area has been identified using the first initial sensing value and the current first sensing value.

[0196] Here, when it is determined that the imageable area has not been determined (No in step S57), in step S58, the area determination unit 213B invalidates the initialization flag stored in the memory 22. After the initialization flag is invalidated, the process returns to step S41.

[0197] On the other hand, when it is determined that the imageable area has been determined (YES in step S57 ), the process proceeds to step S59 .

[0198] On the other hand, when it is determined that all the markers are recognized (Yes in step S53 ), in step S60 , the region specifying unit 213B specifies the imageable region surrounded by the plurality of markers recognized by the marker recognizing unit 212 .

[0199] The processing of steps S61 to S64 is due to Fig.15 The processes of step S36 to step S39 shown are the same, and thus the description thereof is omitted.

[0200] As in Implementation 2, even if the matching process of the photographable image within the photographable area one frame ago and the image of the current frame is performed every frame, when light shines on the auxiliary object device 101 or a shadow is generated, the matching process may become difficult and the photographable area may no longer be estimated.

[0201] On the other hand, in the third embodiment, the imageable region is determined using the first initial sensing value and the current first sensing value, so the accuracy of determining the imageable region can be improved.

[0202] In the third embodiment, when the camera 1 has a zoom function for enlarging or reducing the subject, the first sensed value may include information on the zoom ratio. The zoom function may enlarge or reduce the subject optically or through image processing.

[0203] In addition, in the third embodiment, the first initial sensing value and the current first sensing value are used to determine the photographable area, but the present disclosure is not particularly limited thereto. The area determination unit 213B may obtain the first sensing value one frame ago and the current first sensing value without determining the photographable area based on the photographable area of ​​the current frame estimated by the area estimation unit 216. The area determination unit 213B may determine the area in the image obtained this time that is consistent with the photographable image one frame ago as the photographable area based on the current position and current posture of the camera 1 relative to the position and posture of the camera 1 one frame ago estimated based on the first sensing value one frame ago and the current first sensing value.

[0204] In this case, after the region determination unit 213B stores the first initial sensing value in the memory 22 when all the marks are initially recognized, it obtains the first sensing value representing the position and posture of the camera 1 for each frame, and stores the obtained first sensing value in the memory 22. Thus, the first initial sensing value stored in the memory 22 when all the marks are initially recognized is updated for each frame. The region determination unit 213B determines the region in the image that is consistent with the imageable image one frame ago and is obtained this time as the imageable region, based on the current position and current posture of the camera 1 relative to the position and posture of the camera 1 one frame ago, which are estimated based on the first sensing value one frame ago and the current first sensing value.

[0205] The area determination unit 213B calculates the current position and current posture of the camera 1 relative to the position and posture of the camera 1 represented by the first sensing value one frame ago, and determines the photographable area based on the calculation result. The area determination unit 213B estimates the current photographic range relative to the photographable image one frame ago based on the position and posture of the camera 1 one frame ago and the current position and current posture of the camera 1. The area determination unit 213B determines the area in the image that is consistent with the photographable image one frame ago and obtained this time from the current photographic range relative to the photographable image one frame ago as the photographable area.

[0206] That is, the moving direction and moving distance from the position and posture of the camera 1 one frame ago are calculated based on the first sensing value one frame ago and the current first sensing value. Then, the current position and current posture of the camera 1 are estimated based on the calculated moving direction and moving distance. The current shooting range of the camera 1 relative to the image that can be shot one frame ago is estimated based on the estimated current position and current posture of the camera 1, and the area in the image that is consistent with the image that can be shot one frame ago and obtained this time is determined as the shooting area.

[0207] In the third embodiment, the image processing device 2A in the second embodiment includes the sensing value acquisition unit 217 and the region identification unit 213B, but the present disclosure is not particularly limited thereto, and the image processing device 2 in the first embodiment may include the sensing value acquisition unit 217 and the region identification unit 213B. In this case, Fig.11 After step S6, the area determination unit 213B stores the first sensing value representing the position and posture of the camera 1 when all the marks are initially recognized as the first initial sensing value in the memory 22. In addition, if it is determined that the photographable area has not been determined by the matching process between the initial photographable image and the image obtained this time ( Fig.12 Step S11 “No”), the area determination unit 213B may perform Fig.21 The processing of steps S55 to S59 is shown.

[0208] In addition, the camera 1 may further include a second sensor for detecting the distance from the camera 1 to the object for each pixel in the image. The second sensor is, for example, a depth sensor. The camera 1 may send a second sensed value representing the distance from the camera 1 to the object for each pixel in the image to the image processing device 2 together with the image. When a plurality of markers are initially recognized, the region determination unit 213B may store the photographable image in the photographable region in the image as an initial photographable image in the memory 22, and store each distance from the camera 1 to the object for each pixel in the initial photographable image in the memory 22 as a second initial sensed value. When the photographable region is not determined based on the photographable region of the current frame estimated by the region estimation unit 216, the region determination unit 213B may obtain the second initial sensed value and the second sensed value representing the distance from the camera 1 to the object for each pixel in the image acquired this time. Then. The region determination unit 213B may determine the region in which the second sensed values ​​of the plurality of pixels in the image acquired this time are equal to or less than the longest distance among the second initial sensed values ​​as the photographable region.

[0209] In addition, in the third embodiment, the region determination unit 213B may determine the imageable region using the first sensing value, or may determine the imageable region using the second sensing value. In addition, the region determination unit 213B may determine the imageable region using both the first sensing value and the second sensing value. In this case, the region determination unit 213B may calculate the logical AND of the imageable region determined using the first sensing value and the imageable region determined using the second sensing value.

[0210] In addition, in each of the above-mentioned embodiments, each component can be composed of dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. In addition, the program can also be implemented by another independent computer system by recording the program on a recording medium and transferring it, or by transferring the program via a network.

[0211] Part or all of the functions of the device involved in the embodiments of the present disclosure are typically implemented as LSI (Large Scale Integration) as an integrated circuit. These can be individually single-chip, or part or all of them can be single-chip. In addition, integrated circuitization is not limited to LSI, and can also be implemented by a dedicated circuit or a general-purpose processor. FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connection and setting of the circuit unit inside the LSI can be used.

[0212] Furthermore, a processor such as a CPU may implement a part or all of the functions of the apparatus according to the embodiments of the present disclosure by executing a program.

[0213] In addition, all the numbers used in the above description are exemplifications for specifically describing the present disclosure, and the present disclosure is not limited to the exemplified numbers.

[0214] In addition, the order of executing each step shown in the above flowchart is an example for specifically explaining the present disclosure, and other orders may be used within the scope of obtaining the same effect. In addition, part of the above steps may be executed simultaneously (in parallel) with other steps.

[0215] Industrial Applicability

[0216] The technology involved in the present disclosure can output an image in which the area outside the photographable area surrounded by the plurality of markers is masked even when the plurality of markers are not recognized. Therefore, it is useful as a technology for masking the area outside the photographable area in an image.

Claims

1. An information processing method is an information processing method in a computer. The information processing method performs the following processing: Get the image captured by the camera, identifying a plurality of landmarks contained within the image, When the plurality of marks are recognized, a photographable area surrounded by the recognized plurality of marks is determined, and when the plurality of marks are not recognized, an area within the image obtained this time and consistent with a photographable image within the photographable area determined in the past is determined as the photographable area. creating a mask image for masking an area outside the imageable area in the image, The mask image is output.

2. The information processing method according to claim 1, wherein: In the creation of the mask image, when the imageable area has not been determined, the mask image is created by masking the entirety of the image.

3. The information processing method according to claim 1 or 2, wherein: In determining the photographable area, when the multiple marks are not recognized, the photographable image within the photographable area determined in the past is compared with the image obtained this time, and the area within the image that is consistent with the photographable image is determined as the photographable area.

4. The information processing method according to claim 3, wherein: Furthermore, when the plurality of markers are initially recognized, the photographable image within the photographable area within the image is stored in a memory as an initial photographable image. In determining the photographable area, when the plurality of markers are not recognized, the initial photographable image and the image acquired this time are compared, and an area within the image that matches the initial photographable image is determined as the photographable area.

5. The information processing method according to claim 4, wherein: Furthermore, when the plurality of markers are initially recognized, a first sensing value representing the position and posture of the camera is stored in the memory as a first initial sensing value. Furthermore, when the imageable area is not determined, the first initial sensing value and the current first sensing value are obtained, Furthermore, based on the current position and current posture of the camera relative to the initial position and initial posture of the camera estimated based on the first initial sensing value and the current first sensing value, the area within the image that is consistent with the initial photographable image and obtained this time is determined as the photographable area.

6. The information processing method according to claim 3, wherein: In determining the photographable area, when the plurality of markers are not recognized, the area within the image acquired this time that is consistent with the photographable image within the photographable area determined one frame ago is determined as the photographable area.

7. The information processing method according to claim 6, wherein: Furthermore, the photographable image in the photographable area estimated one frame ago and stored in the memory is compared with the image of the current frame acquired this time, and the area in the image that matches the photographable image is estimated as the photographable area of ​​the current frame. Furthermore, the estimated photographable image within the photographable area of ​​the current frame is stored in the memory, In the determination of the photographable area, when the plurality of markers are not recognized, the estimated photographable area of ​​the current frame is determined as the photographable area.

8. The information processing method according to claim 7, wherein: Furthermore, a first sensing value representing the position and posture of the camera is obtained for each frame, and the obtained first sensing value is stored in the memory. Furthermore, when the imageable area is not determined, the first sensing value one frame ago and the current first sensing value are acquired, Furthermore, based on the current position and current posture of the camera relative to the position and posture of the camera one frame ago, which are inferred based on the first sensing value one frame ago and the current first sensing value, the area within the image obtained this time that is consistent with the photographable image one frame ago is determined as the photographable area.

9. The information processing method according to claim 1 or 2, wherein: Furthermore, when the plurality of markers are initially recognized, the photographable image within the photographable area within the image of the initial frame is taken as an initial photographable image and stored in a memory, Furthermore, the image one frame ago and the image of the current frame are compared, and the amount of change between the shooting range of the image one frame ago and the shooting range of the image of the current frame is calculated for each frame. Furthermore, the cumulative value of the change amount from the initial frame to the current frame is calculated, In the determination of the photographable region, when the plurality of markers are not recognized, the photographable region of the current frame is determined based on the initial photographable region and the cumulative value.

10. The information processing method according to claim 1 or 2, wherein: Furthermore, when the plurality of marks are initially recognized, the photographable image within the photographable area in the image is taken as an initial photographable image and stored in a memory, Furthermore, when the plurality of marks are initially recognized, each distance from the camera to the object for each pixel in the initial photographable image is stored in the memory as a second initial sensing value, Furthermore, when the imageable area is not determined, the second initial sensing value and a second sensing value representing the distance from the camera to the object for each pixel in the image obtained this time are obtained, Furthermore, an area where the second sensing values ​​of a plurality of pixels in the image acquired this time are equal to or less than the longest distance among the second initial sensing values ​​is determined as the imageable area.

11. An information processing device comprising: an acquisition unit that acquires an image captured by a camera; a recognition unit for recognizing a plurality of marks contained in the image; a determination unit that, when the plurality of markers are recognized, determines a photographable area surrounded by the plurality of recognized markers, and, when the plurality of markers are not recognized, determines an area within the image that is consistent with a photographable image within the photographable area determined in the past and is obtained this time as the photographable area; and a creating unit that creates a mask image that masks an area outside the imageable area in the image; and An output unit outputs the mask image.

12. An information processing program that causes a computer to function as follows: Get the image captured by the camera, identifying a plurality of landmarks contained within the image, When the plurality of marks are recognized, a photographable area surrounded by the recognized plurality of marks is determined, and when the plurality of marks are not recognized, an area within the image obtained this time and consistent with a photographable image within the photographable area determined in the past is determined as the photographable area. creating a mask image for masking an area outside the imageable area in the image, The mask image is output.

Images