Information processing method, information processing device, and information processing program

Abstract

The image processing device acquires an image captured by a camera, recognizes a plurality of markers included in the image, and calculates the distance between each of the recognized markers. Creating, on the basis of the calculated distance, a first image-detectable region surrounded by a predetermined number of markers belonging to a first marker group among the plurality of identified markers, and a second image-detectable region surrounded by a predetermined number of markers belonging to a second marker group among the plurality of identified markers; a mask image is created in which a region outside the first image-capturing region and the second image-capturing region in the image is masked, and the mask image is output.

Description

Technical Field

[0001] The present disclosure relates to a technology for masking areas other than a plurality of imageable areas within an image. Background Art

[0002] For example, Patent Document 1 discloses an image control device that sets a permitted image-capable area in a work area for image sharing with a remote assistant operator, and sends a masked image of the work area that masks the non-image-capable area to a remote terminal for the remote assistant operator. The existing image control device detects a mark contained in the image data, extracts mark information indicating the image-capable area from the detected mark, determines an image area in the image data that is judged to be outside the image-capable area as an image-unshareable area, masks the image-unshareable area, and generates a masked image.

[0003] However, the above-mentioned conventional technology does not take into account the setting of a plurality of imageable areas, and it is difficult to output an image in which areas other than the plurality of imageable areas are appropriately masked, and further improvement is required.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-150866 Summary of the invention

[0007] The present disclosure has been made to solve the above-mentioned problem, and an object of the present disclosure is to provide a technology capable of outputting an image in which areas other than the plurality of imageable areas are appropriately masked even when a plurality of imageable areas are set.

[0008] The information processing method disclosed herein is an information processing method in a computer, which obtains an image taken by a camera, identifies multiple marks contained in the image, calculates the distance between each of the multiple marks identified, and determines, based on the calculated distance, a first photographable area surrounded by a given number of marks belonging to a first mark group among the multiple marks identified, and a second photographable area surrounded by a given number of marks belonging to a second mark group among the multiple marks identified, creates a mask image that masks the first photographable area in the image and the area outside the second photographable area, and outputs the mask image.

[0009] According to the present disclosure, even when a plurality of imageable areas are set, an image in which areas other than the plurality of imageable areas are appropriately masked can be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1It is a diagram showing an example of the configuration of a work support system in an embodiment of the present disclosure.

[0011] Figure 2 This is a diagram showing an example of a work space in this embodiment.

[0012] Figure 3 It is a diagram showing an example of the structure of the image processing device in this embodiment.

[0013] Figure 4 This is a diagram showing an example of an image captured by a camera in the present embodiment in which two groups of markers are reflected.

[0014] Figure 5 This means that in this embodiment, based on Figure 4 FIG. 1 is a diagram showing an example of a mask image created using the image shown.

[0015] Figure 6 This is a first flowchart for explaining the job assisting process of the image processing device in the embodiment of the present disclosure.

[0016] Figure 7 This is a second flowchart for explaining the job assisting process of the image processing device in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] (Insights that form the basis of this disclosure)

[0018] In order for workers at the site to efficiently receive work assistance from workers at a remote location, it is effective to share images of the conditions at the site with the workers at the remote location.

[0019] When transmitting images to a remote location, confidential information may be reflected in the images. Therefore, in order to prevent confidential information from being leaked to the outside through image transmission, objects that can be image transmitted or objects that are confidential information are identified and the area corresponding to the confidential information in the image is masked.

[0020] For example, an operator at the scene configures multiple markers to surround an area where images can be transmitted, that is, an area that does not contain confidential information. By identifying multiple markers in the captured image, the area outside the imageable area surrounded by the identified multiple markers is masked, thereby preventing confidential information from being leaked.

[0021] Here, it is desirable to set not only one imageable area but multiple imageable areas on site, and to send images masking the working areas outside the multiple imageable areas to a remote location. For example, when two imageable areas are set, four different types of marks are used as one mark group, and two mark groups are used.

[0022] However, the existing device cannot distinguish which combination of 4 marks is a group of marks by only performing image recognition on 8 marks. For example, even if 4 marks out of 8 marks are recognized by image recognition, there is a possibility that these 4 marks are not the combination expected by the operator.

[0023] This problem can be solved by predetermining the conditions for the types and positions of the four marks belonging to one mark group, and not recognizing combinations other than the combination of the four marks arranged under the determined conditions. However, when an image contains multiple mark groups, it is possible that an unexpected combination of the four marks meets the predetermined conditions, and mask processing cannot be properly performed, resulting in the transmission of an image containing confidential information.

[0024] In the above-mentioned conventional technology, setting of a plurality of imageable areas is not considered, and it is difficult to output an image in which areas other than the plurality of imageable areas are appropriately masked.

[0025] In order to solve the above problems, the following technology is disclosed.

[0026] (1) An information processing method involved in one aspect of the present disclosure is an information processing method in a computer, which obtains an image captured by a camera, identifies multiple marks contained in the image, calculates the distance between each of the multiple marks identified, and determines, based on the calculated distance, a first photographable area surrounded by a given number of marks belonging to a first mark group among the multiple marks identified, and a second photographable area surrounded by a given number of marks belonging to a second mark group among the multiple marks identified, creates a mask image that masks the first photographable area in the image and the area outside the second photographable area, and outputs the mask image.

[0027] According to this structure, when a plurality of marks included in an image are recognized, the distances between the recognized marks are calculated. Then, based on the calculated distances, a first photographable area surrounded by a given number of marks belonging to a first marker group among the recognized marks and a second photographable area surrounded by a given number of marks belonging to a second marker group among the recognized marks are determined. Then, a mask image is created that masks the areas outside the first photographable area and the second photographable area in the image, and the mask image is output.

[0028] Therefore, even when a plurality of imageable areas are set, the plurality of imageable areas are determined respectively, so that an image with the areas outside the plurality of imageable areas appropriately masked can be output. That is, it is possible to prevent the first imageable area and the second imageable area from being mistakenly determined as one imageable area, and to prevent the output of an image containing confidential information located in one mistakenly determined imageable area. As a result, an operator at the site can receive efficient work assistance from an operator at a remote location.

[0029] (2) In the information processing method described in (1) above, it is also possible to determine, in determining the first photographable area and the second photographable area, the first marker group consisting of the given number of markers whose distance is below the threshold, and the second marker group consisting of the given number of markers whose distance is below the threshold.

[0030] According to this structure, when the distance between the given number of marks is below the threshold, it is determined that the given number of marks belongs to the same mark group. Therefore, it is possible to determine the given number of marks belonging to the correct combination of the first mark group and the given number of marks belonging to the correct combination of the second mark group.

[0031] (3) In the information processing method described in (1) or (2) above, it is also possible that, in determining the first photographable area and the second photographable area, when there are multiple marks of the same type, the mark closest to the mark serving as the benchmark may be determined as a mark belonging to the same mark group as the mark serving as the benchmark.

[0032] According to this structure, when there are multiple marks of the same type, the mark closest to the mark to be used as the benchmark is determined as a mark belonging to the same mark group as the mark to be used as the benchmark. Therefore, it is possible to more reliably determine the given number of marks belonging to the correct combination of the first mark group and the given number of marks belonging to the correct combination of the second mark group.

[0033] (4) In the information processing method described in any one of (1) to (3) above, it is also possible to predetermine the positional relationship between the given number of marks belonging to the first mark group and the second mark group, respectively, and then, when the multiple marks are initially identified, determine whether the given number of marks are arranged according to the predetermined positional relationship, and then, when it is determined that the given number of marks are not arranged according to the predetermined positional relationship, output notification information for notifying the operator that the positional relationship between the given number of marks is incorrect.

[0034] For example, a plurality of marks are attached to the working area by the operator. For this reason, it is possible that the marks are attached in an incorrect positional relationship due to the operator's mistake. In this case, it becomes difficult to determine the first photographable area and the second photographable area. However, according to the above-mentioned structure, when it is determined that the given number of marks are not arranged according to a predetermined positional relationship, a notification message is output to notify the operator that the positional relationship between the given number of marks is incorrect. Therefore, the operator can be informed that the given number of marks are arranged in an incorrect positional relationship, and the operator can be urged to re-arrange the given number of marks in a correct positional relationship.

[0035] (5) In the information processing method described in any one of (1) to (3) above, the positional relationship between the given number of marks belonging to the first mark group and the second mark group, respectively, is predetermined, and then, when the plurality of marks are initially identified, it is determined whether the given number of marks are arranged according to the predetermined positional relationship, and then, when it is determined that the given number of marks are not arranged according to the predetermined positional relationship, notification information is output to notify the operator that the marks are not arranged according to the predetermined positional relationship.

[0036] For example, a plurality of marks are attached to the working area by the operator. For this reason, it is possible that the marks are attached in an incorrect positional relationship due to the operator's mistake. In this case, it becomes difficult to determine the first photographable area and the second photographable area. However, according to the above-mentioned structure, when it is determined that a given number of marks are not arranged in accordance with a predetermined positional relationship, notification information for notifying the operator of the marks that are not arranged in accordance with the predetermined positional relationship is output. Therefore, the operator can be informed of the marks arranged in an incorrect positional relationship, and the operator can be urged to re-arrange the marks arranged in an incorrect positional relationship in a correct positional relationship.

[0037] (6) In the information processing method described in (4) above, the operator is further inputted with respect to the number of the plurality of mark groups used and the number of the given number of marks belonging to each mark group, and then, when the plurality of marks are initially recognized, it is determined whether the plurality of marks are recognized in the same number as the total number of marks belonging to the plurality of mark groups inputted, and in determining whether the given number of marks are arranged according to the predetermined positional relationship, when it is determined that the plurality of marks are recognized in the same number as the total number of marks belonging to the plurality of mark groups inputted, it is determined whether the given number of marks are arranged according to the predetermined positional relationship.

[0038] According to this structure, the operator's input of the number of multiple mark groups to be used and the number of a given number of marks belonging to each mark group is accepted, and when multiple marks are initially recognized, it is determined whether the same number of marks as the total number of marks belonging to the input multiple mark groups are recognized.

[0039] Therefore, by determining whether the number of the plurality of markers initially recognized is the same as the number of the plurality of markers actually arranged, it is possible to automatically determine whether all of the plurality of markers actually arranged are recognized.

[0040] (7) In the information processing method described in (4) above, it is also possible that, when the multiple marks are initially recognized, the obtained image is presented to the operator, and then the operator's input is accepted as to whether all the marks belonging to the multiple mark groups are recognized in the image. In the determination of whether the given number of marks are arranged according to the predetermined positional relationship, when the operator inputs that all the marks belonging to the entire mark group are recognized, it is determined whether the given number of marks are arranged according to the predetermined positional relationship.

[0041] According to this configuration, when a plurality of markers are initially recognized, the acquired image is presented to the operator, and the operator's input as to whether all markers belonging to a plurality of marker groups are recognized in the image is accepted.

[0042] Therefore, the operator can confirm whether all of the multiple markers actually arranged are recognized.

[0043] (8) In the information processing method described in any one of (1) to (7) above, the given number of marks belonging to the first mark group may include a first mark arranged at the upper left position in a first working area capable of being photographed within the working space, a second mark arranged at the lower left position, a third mark arranged at the upper right position, and a fourth mark arranged at the lower right position, and the given number of marks belonging to the second mark group may include a fifth mark arranged at the upper left position, a sixth mark arranged at the lower left position, a seventh mark arranged at the upper right position, and an eighth mark arranged at the lower right position in a second working area capable of being photographed within the working space.

[0044] According to this structure, a first photographable area surrounded by a first mark arranged at the upper left position of the first working area, a second mark arranged at the lower left position of the first working area, a third mark arranged at the upper right position of the first working area, and a fourth mark arranged at the lower right position of the first working area, and a second photographable area surrounded by a fifth mark arranged at the upper left position of the second working area, a sixth mark arranged at the lower left position of the second working area, a seventh mark arranged at the upper right position of the second working area, and an eighth mark arranged at the lower right position of the second working area are determined.

[0045] Therefore, it is possible to output an image in which the areas other than the first rectangular imageable area and the second rectangular imageable area are masked.

[0046] (9) In the information processing method described in (8) above, in determining the first photographable area and the second photographable area, when the second mark and the sixth mark are located below the first mark, the second mark which is closer to the first mark and whose distance from the first mark is less than a threshold value is determined to be a mark belonging to the first mark group; when the third mark and the seventh mark are located to the right of the first mark, the third mark which is closer to the first mark and whose distance from the first mark is less than a threshold value is determined to be a mark belonging to the first mark group; when the fourth mark and the eighth mark are located to the right of the second mark, the fourth mark which is closer to the second mark and whose distance from the second mark is less than a threshold value is determined to be a mark belonging to the first mark group; and when the fourth mark and the eighth mark are located below the third mark, the fourth mark which is closer to the third mark and whose distance from the third mark is less than a threshold value is determined to be a mark belonging to the first mark group.

[0047] According to this structure, when the second mark and the sixth mark are below the first mark, the second mark can be determined as a mark belonging to the first mark group; when the third mark and the seventh mark are to the right of the first mark, the third mark can be determined as a mark belonging to the first mark group; when the fourth mark and the eighth mark are to the right of the second mark, the fourth mark can be determined as a mark belonging to the first mark group; when the fourth mark and the eighth mark are below the third mark, the fourth mark can be determined as a mark belonging to the first mark group.

[0048] In addition, the present disclosure can be implemented not only as an information processing method that performs the above-mentioned characteristic processing, but also as an information processing device having a characteristic structure corresponding to the characteristic processing performed by the information processing method. In addition, it can also be implemented as a computer program that causes a computer to perform the characteristic processing included in such an information processing method. Therefore, in the following other modes, the same effect as the above-mentioned information processing method can be achieved.

[0049] (10) The information processing device involved in other embodiments of the present invention comprises: an acquisition unit, which acquires an image captured by a camera; a recognition unit, which recognizes multiple markers contained in the image; a calculation unit, which calculates the distance between each of the multiple markers recognized; a determination unit, which determines, based on the calculated distances, a first photographable area surrounded by a given number of markers belonging to a first marker group among the multiple markers recognized, and a second photographable area surrounded by a given number of markers belonging to a second marker group among the multiple markers recognized; a creation unit, which creates a mask image that masks the first photographable area in the image and an area outside the second photographable area; and an output unit, which outputs the mask image.

[0050] (11) The information processing program involved in other embodiments of the present invention enables a computer to perform the following functions: obtain an image captured by a camera, identify multiple markers contained in the image, calculate the distance between each of the multiple markers identified, and determine, based on the calculated distances, a first photographable area surrounded by a given number of markers belonging to a first marker group among the multiple markers identified, and a second photographable area surrounded by a given number of markers belonging to a second marker group among the multiple markers identified, create a mask image that masks the first photographable area in the image and the area outside the second photographable area, and output the mask image.

[0051] (12) Other embodiments of the present disclosure involve a non-temporary computer-readable recording medium recording an information processing program, wherein the information processing program enables a computer to perform the following functions: obtain an image captured by a camera, identify multiple markers contained in the image, calculate the distance between each of the multiple markers identified, and determine, based on the calculated distances, a first photographable area surrounded by a given number of markers belonging to a first marker group among the multiple markers identified, and a second photographable area surrounded by a given number of markers belonging to a second marker group among the multiple markers identified, create a mask image that masks the first photographable area in the image and the area outside the second photographable area, and output the mask image.

[0052] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. In addition, in all embodiments, the various contents can also be combined.

[0053] (Implementation Method)

[0054] Figure 1 It is a diagram showing an example of the configuration of a work support system in an embodiment of the present disclosure. Figure 1 The illustrated work support system includes a camera 1 , an image processing device 2 , and a remote terminal 3 .

[0055] The camera 1 is, for example, a wearable camera mounted on the head of a worker at the site. The worker wears the camera 1 and performs work. At this time, the worker at the site uses the camera 1 to photograph the auxiliary object device at the work space at the site, and receives work assistance from a remote worker. The remote worker observes the image transmitted from the site and performs assistance in the operation and repair of the auxiliary object device.

[0056] Figure 2 This is a diagram showing an example of a work space in this embodiment.

[0057] like Figure 2 As shown, an auxiliary object device 101 is set in a work space 100. An operator at the site uses a camera 1 to photograph the auxiliary object device 101. The image photographed by the camera 1 is sent to a remote terminal 3 operated by an operator at a distant location. At this time, there is a possibility that confidential information is reflected in the photographed image. For this reason, in order to prevent the transmission of an image containing confidential information, the area outside a specific work area is blocked by image processing (masking). In this embodiment, there is not only one work area that does not contain confidential information and can be photographed, but there are multiple areas. The operator at the site attaches the first mark 111 to the eighth mark 118 so that the first work area 151 and the second work area 152 that do not contain confidential information of the auxiliary object device 101 are respectively surrounded. The first mark 111 to the eighth mark 118 are directly attached, for example, by an adhesive or a magnet, so that the first work area 151 and the second work area 152 that can be photographed are respectively surrounded.

[0058] The camera 1 transmits the captured image to the image processing device 2 via short-range wireless communication. The camera 1 transmits the image at a given frame rate to the image processing device 2. For this purpose, a dynamic image is transmitted from the camera 1 to the image processing device 2. In addition, the camera 1 is connected to the image processing device 2 wirelessly, but can also be connected to the image processing device 2 by wire.

[0059] In addition, the camera 1 may be a terminal equipped with a camera, such as a smartphone or a tablet computer. In addition, the camera 1 may be a digital camera.

[0060] The image processing device 2 is, for example, a personal computer operated by an operator at a site. The image processing device 2 and the remote terminal 3 are connected to each other so as to be communicable via a network 4. The network 4 is, for example, the Internet.

[0061] Figure 3 1 is a diagram showing an example of the configuration of the image processing device 2 in this embodiment.

[0062] Figure 3 The image processing device 2 shown includes a processor 21 , a memory 22 , a communication unit 23 , an input receiving unit 24 , and a display unit 25 .

[0063] The processor 21 is, for example, a CPU (central processing unit). The processor 21 implements an image acquisition unit 211, a marker recognition unit 212, a marker recognition determination unit 213, a marker arrangement determination unit 214, a notification information output unit 215, a distance calculation unit 216, an area determination unit 217, a mask image creation unit 218, and a mask image output unit 219.

[0064] The memory 22 is a storage device capable of storing various information, such as a RAM (Random Access Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), or a flash memory.

[0065] In addition, the image processing device 2 is equivalent to an example of an information processing device, the image acquisition unit 211 is equivalent to an example of an acquisition unit, the mark recognition unit 212 is equivalent to an example of a recognition unit, the distance calculation unit 216 is equivalent to an example of a calculation unit, the area determination unit 217 is equivalent to an example of a determination unit, the mask image creation unit 218 is equivalent to an example of a creation unit, and the mask image output unit 219 is equivalent to an example of an output unit.

[0066] The communication unit 23 receives the image transmitted from the camera 1. The communication unit 23 outputs the received image to the processor 21. The communication unit 23 receives the moving image transmitted at a given frame rate, and outputs the image to the processor 21 for each frame.

[0067] The image acquisition unit 211 acquires an image captured by the camera 1. The image acquisition unit 211 acquires an image for each frame.

[0068] The mark recognition unit 212 recognizes a plurality of marks contained in the image acquired by the image acquisition unit 211. In the present embodiment, two groups of mark groups are used. The first mark group includes a given number of marks of different types. The given number is, for example, 4. The first mark group includes a first mark 111, a second mark 112, a third mark 113, and a fourth mark 114. The second mark group includes a given number of marks of different types. The given number is, for example, 4. The second mark group includes a fifth mark 115, a sixth mark 116, a seventh mark 117, and an eighth mark 118. Figure 2 As shown, the eight first marks 111 to the eighth mark 118 are rectangular, the first mark 111 to the fourth mark 114 have different patterns, and the fifth mark 115 to the eighth mark 118 have different patterns. The first mark 111 and the fifth mark 115 are of the same type (pattern), the second mark 112 and the sixth mark 116 are of the same type (pattern), the third mark 113 and the seventh mark 117 are of the same type (pattern), and the fourth mark 114 and the eighth mark 118 are of the same type (pattern).

[0069] The positional relationship between the given number of markers belonging to the first marker group and the second marker group is predetermined.

[0070] The given number of marks belonging to the first mark group includes the first mark 111 arranged at the upper left position, the second mark 112 arranged at the lower left position, the third mark 113 arranged at the upper right position, and the fourth mark 114 arranged at the lower right position in the first work area 151 that can be photographed in the work space. In addition, the given number of marks belonging to the second mark group includes the fifth mark 115 arranged at the upper left position, the sixth mark 116 arranged at the lower left position, the seventh mark 117 arranged at the upper right position, and the eighth mark 118 arranged at the lower right position in the second work area 152 that can be photographed in the work space.

[0071] The worker at the site affixes the first sign 111 at the upper left of the first work area 151 that can be photographed, the second sign 112 at the lower left of the first work area 151 that can be photographed, the third sign 113 at the upper right of the first work area 151 that can be photographed, and the fourth sign 114 at the lower right of the first work area 151 that can be photographed. In addition, the worker at the site affixes the fifth sign 115 at the upper left of the second work area 152 that can be photographed, the sixth sign 116 at the lower left of the second work area 152 that can be photographed, the seventh sign 117 at the upper right of the second work area 152 that can be photographed, and the eighth sign 118 at the lower right of the second work area 152 that can be photographed.

[0072] Thus, the first mark 111 is arranged at the upper left of the first work area 151 that can be photographed, the second mark 112 is arranged at the lower left of the first work area 151 that can be photographed, the third mark 113 is arranged at the upper right of the first work area 151 that can be photographed, and the fourth mark 114 is arranged at the lower right of the first work area 151 that can be photographed. In addition, the fifth mark 115 is arranged at the upper left of the second work area 152 that can be photographed, the sixth mark 116 is arranged at the lower left of the second work area 152 that can be photographed, the seventh mark 117 is arranged at the upper right of the second work area 152 that can be photographed, and the eighth mark 118 is arranged at the lower right of the second work area 152 that can be photographed.

[0073] The mark recognition unit 212 recognizes the eight first marks 111 to the eighth marks 118 contained in the image by pattern matching. That is, the mark recognition unit 212 recognizes the eight first marks 111 to the eighth marks 118 in the image by comparing the patterns of the plurality of marks pre-stored in the memory 22 with the patterns of the eight first marks 111 to the eighth marks 118 contained in the image.

[0074] In addition, the sign recognition unit 212 can recognize not only a sign photographed from the front but also a sign photographed from an oblique direction.

[0075] In addition, in the present embodiment, two groups of flag groups are used, but the present disclosure is not particularly limited thereto, and three or more groups of flag groups may be used. In addition, in the present embodiment, one group of flag groups includes four flags, but the present disclosure is not particularly limited thereto, and may include three flags, or may include five or more flags.

[0076] The input receiving unit 24 is, for example, a keyboard, a mouse, or a touch panel. The input receiving unit 24 receives input of the number of used marker groups and the number of given number of markers belonging to each marker group by the operator. The memory 22 stores the number of used marker groups and the number of given number of markers belonging to each marker group input by the input receiving unit 24.

[0077] When a plurality of markers are initially recognized, the marker recognition determination unit 213 determines whether a plurality of markers equal in number to the number of all markers belonging to a plurality of marker groups input in advance by the input acceptance unit 24 are recognized.

[0078] In the case where a plurality of marks are initially recognized, the mark configuration determination unit 214 determines whether a given number of marks belonging to the first mark group and the second mark group, respectively, are configured according to a predetermined positional relationship. More specifically, in the case where the mark recognition determination unit 213 determines that a plurality of marks equal in number to the number of all marks belonging to the inputted plurality of mark groups are recognized, the mark configuration determination unit 214 determines whether a given number of marks are configured according to a predetermined positional relationship. For example, a plurality of marks are affixed to the working area by an operator. For this reason, it is possible that the marks are affixed in an incorrect positional relationship due to the operator's mistake. In this case, it becomes difficult to determine the first photographable area and the second photographable area. For example, in the case where the mark that should be configured in the upper left and the mark that should be configured in the lower left are exchanged, the mark configuration determination unit 214 determines that the given number of marks are not configured according to a predetermined positional relationship.

[0079] In the plurality of marks, identification information for identifying each mark is associated, and in each identification information, a predetermined configuration position is associated. For this purpose, if a plurality of marks are identified, the mark configuration determination unit 214 determines the coordinate position on the image of each of the identified plurality of marks, the identification information of each of the identified plurality of marks, and the configuration position associated with the identification information of each of the identified plurality of marks.

[0080] When the identification information of the identified multiple marks is inconsistent with the identification information of the given number of marks belonging to the predetermined multiple mark groups, the mark arrangement determination unit 214 determines that the given number of marks are not arranged according to the predetermined positional relationship. For example, when two sets of mark groups are used, when there are three or more marks with the same identification information, the mark arrangement determination unit 214 determines that the given number of marks are not arranged according to the predetermined positional relationship.

[0081] In addition, when the positional relationship of the arrangement positions of the recognized multiple marks does not match the positional relationship of the predetermined arrangement positions, the mark arrangement determination unit 214 determines that the given number of marks are not arranged at the predetermined positions. For example, when the mark for the upper left is arranged at the lower left, the mark arrangement determination unit 214 determines that the given number of marks are not arranged according to the predetermined positional relationship.

[0082] In addition, when two groups of marker groups are used, two square-shaped areas that can be photographed are determined. To this end, the marker configuration determination unit 214 can create a square-shaped area in which the marker for the upper left, the marker for the lower left, the marker for the upper right, and the marker for the lower right are connected by straight lines, and determine whether at least two square-shaped areas are created. In the case of creating at least two square-shaped areas, the marker configuration determination unit 214 can determine that a given number of markers are configured according to a predetermined positional relationship. In addition, in the case of creating only one square-shaped area, or in the case of not creating a square-shaped area, the marker configuration determination unit 214 can determine that a given number of markers are not configured according to a predetermined positional relationship.

[0083] In addition, the positional relationship of the four marks belonging to one mark group is predetermined. Therefore, when the mark for the upper right does not exist in the right direction of the mark for the upper left, when the mark for the lower left does not exist in the lower direction of the mark for the upper left, or when the mark for the lower right does not exist in the lower right direction of the mark for the upper left, the mark arrangement determination unit 214 can determine that the given number of marks are not arranged according to the predetermined positional relationship.

[0084] When the mark arrangement determination unit 214 determines that the given number of marks are not arranged in accordance with the predetermined positional relationship, the notification information output unit 215 outputs notification information for notifying the operator that the positional relationship between the given number of marks is wrong. The notification information output unit 215 creates notification information for notifying the operator that the positional relationship between the given number of marks is wrong, and outputs the created notification information to the display unit 25.

[0085] The display unit 25 is, for example, a liquid crystal display device. The display unit 25 displays notification information for notifying the operator that the positional relationship between the given number of marks is incorrect. For example, the display unit 25 displays a text such as "The arrangement of the marks is incorrect." The operator confirms the notification information displayed on the display unit 25 and arranges the incorrectly arranged marks in the correct position.

[0086] In addition, when the mark arrangement determination unit 214 determines that the given number of marks are not arranged in accordance with the predetermined positional relationship, the notification information output unit 215 outputs notification information for notifying the operator of the marks that are not arranged in accordance with the predetermined positional relationship. The notification information output unit 215 can determine the marks that are not arranged in accordance with the predetermined positional relationship. Then, the notification information output unit 215 can create notification information for notifying the operator of the marks that are not arranged in accordance with the predetermined positional relationship, and output the created notification information to the display unit 25.

[0087] The display unit 25 is, for example, a liquid crystal display device. The display unit 25 may display notification information for notifying the operator that the mark is not arranged in accordance with the predetermined positional relationship. For example, the display unit 25 may display a text such as "The arrangement of the lower left mark is wrong." The operator confirms the notification information displayed on the display unit 25 and re-arranges the mark arranged in the wrong positional relationship to the correct positional relationship.

[0088] The distance calculation unit 216 calculates the distance between each of the plurality of markers recognized by the marker recognition unit 212 .

[0089] The distance calculation unit 216 calculates, for multiple groups of marks, the distance between the upper left mark and at least one upper right mark located to the right of the upper left mark, the distance between the upper left mark and at least one lower left mark located to the bottom of the upper left mark, the distance between the lower left mark and at least one lower right mark located to the right of the lower left mark, and the distance between the upper right mark and at least one lower right mark located to the bottom of the upper right mark.

[0090] The region determination unit 217 determines, based on the distance calculated by the distance calculation unit 216, a first imageable region surrounded by a given number of marks belonging to a first mark group among the plurality of marks recognized by the mark recognition unit 212, and a second imageable region surrounded by a given number of marks belonging to a second mark group among the plurality of marks recognized by the mark recognition unit 212. The region determination unit 217 determines a first mark group consisting of a given number of marks whose distances calculated by the distance calculation unit 216 are less than a threshold value, and a second mark group consisting of a given number of marks whose distances calculated by the distance calculation unit 216 are less than a threshold value. In addition, when there are a plurality of marks of the same type, the region determination unit 217 determines the mark that is closest to the reference mark as a mark belonging to the same mark group as the reference mark. Then, the region determination unit 217 determines a first imageable region surrounded by a given number of marks belonging to the determined first mark group, and a second imageable region surrounded by a given number of marks belonging to the determined second mark group. In addition, the threshold value may be set in advance by the operator.

[0091] The mask image creation unit 218 creates a mask image that masks the areas outside the first photographable area and the second photographable area in the image. In addition, due to the influence of lighting or the location of the mark outside the photographing range, the mark recognition unit 212 may not be able to recognize the mark. For this reason, when both the first photographable area and the second photographable area are not determined by the area determination unit 217, the mask image creation unit 218 may create a mask image that masks the entire image. In addition, when one of the first photographable area and the second photographable area is determined by the area determination unit 217, the mask image creation unit 218 may create a mask image that masks the outside of one of the first photographable area and the second photographable area.

[0092] The mask image output unit 219 outputs the mask image created by the mask image creating unit 218. The mask image output unit 219 outputs the mask image to the communication unit 23. The communication unit 23 transmits the mask image to the remote terminal 3.

[0093] The remote terminal 3 is, for example, a personal computer, a smartphone, or a tablet computer operated by an operator at a remote location. The remote terminal 3 receives the mask image sent by the image processing device 2. The remote terminal 3 displays the received mask image. The image processing device 2 sends the mask image to the remote terminal 3 at a given frame rate. For this purpose, a dynamic image is sent from the image processing device 2 to the remote terminal 3.

[0094] A remote operator can observe the mask image displayed on the remote terminal 3 and assist the operator at the site in his work. The image processing device 2 and the remote terminal 3 can transmit and receive not only image data but also audio data to and from each other.

[0095] Figure 4 1 is a diagram showing an example of an image 121 captured by the camera 1 in the present embodiment as two groups of markers. Figure 4 In order to make the description easier to understand, the symbols are shown larger than their actual size.

[0096] The image acquisition unit 211 acquires the image 121 captured by the camera 1 . The image 121 includes the first to eighth markers 111 to 118 . The marker recognition unit 212 recognizes the first to eighth markers 111 to 118 included in the image 121 acquired by the image acquisition unit 211 .

[0097] The distance calculation unit 216 calculates the distance between each of the first to eighth markers 111 to 118 recognized by the marker recognition unit 212 .

[0098] To the right of the first mark 111 for the upper left, there are two marks, namely the third mark 113 for the upper right and the seventh mark 117 for the upper right. To the right of the second mark 112 for the lower left, there are two marks, namely the fourth mark 114 for the lower right and the eighth mark 118 for the lower right. To this end, the distance calculation unit 216 calculates the distance 301 between the first mark 111 for upper left and the third mark 113 for upper right located in the right direction of the first mark 111 for upper left, the distance 302 between the first mark 111 for upper left and the seventh mark 117 for upper right located in the right direction of the first mark 111 for upper left, the distance 303 between the first mark 111 for upper left and the second mark 112 for lower left located in the lower direction of the first mark 111 for upper left, the distance 304 between the second mark 112 for lower left and the fourth mark 114 for lower right located in the right direction of the second mark 112 for lower left, the distance 305 between the second mark 112 for lower left and the eighth mark 118 for lower right located in the right direction of the second mark 112 for lower left, and the distance 306 between the third mark 113 for upper right and the fourth mark 114 for lower right located in the lower direction of the third mark 113 for upper right.

[0099] In addition, the distance calculation unit 216 calculates the distance 307 between the 5th mark 115 for upper left and the 7th mark 117 for upper right located in the right direction of the 5th mark 115 for upper left, the distance 308 between the 5th mark 115 for upper left and the 6th mark 116 for lower left located in the lower direction of the 5th mark 115 for upper left, the distance 309 between the 6th mark 116 for lower left and the 8th mark 118 for lower right located in the right direction of the 6th mark 116 for lower left, and the distance 310 between the 7th mark 117 for upper right and the 8th mark 118 for lower right located in the lower direction of the 7th mark 117 for upper right.

[0100] When the first mark group and the second mark group are arranged in the horizontal direction, the distance calculation unit 216 calculates the distance 301 between the first mark 111 and the third mark 113, the distance 302 between the first mark 111 and the seventh mark 117, the distance 303 between the first mark 111 and the second mark 112, the distance 304 between the second mark 112 and the fourth mark 114, the distance 305 between the second mark 112 and the eighth mark 118, the distance 306 between the third mark 113 and the fourth mark 114, the distance 307 between the fifth mark 115 and the seventh mark 117, the distance 308 between the fifth mark 115 and the sixth mark 116, the distance 309 between the sixth mark 116 and the eighth mark 118, and the distance 310 between the seventh mark 117 and the eighth mark 118.

[0101] In addition, when the first mark group and the second mark group are arranged in the vertical direction, the distance calculation unit 216 calculates the distance between the first mark 111 and the third mark 113, the distance between the first mark 111 and the second mark 112, the distance between the first mark 111 and the sixth mark 116, the distance between the second mark 112 and the fourth mark 114, the distance between the third mark 113 and the fourth mark 114, the distance between the third mark 113 and the eighth mark 118, the distance between the fifth mark 115 and the seventh mark 117, the distance between the fifth mark 115 and the sixth mark 116, the distance between the sixth mark 116 and the eighth mark 118, and the distance between the seventh mark 117 and the eighth mark 118.

[0102] The area determination unit 217 determines a first marker group consisting of the first marker 111 to the fourth marker 114 whose distances calculated by the distance calculation unit 216 are less than the threshold, and a second marker group consisting of the fifth marker 115 to the eighth marker 118 whose distances calculated by the distance calculation unit 216 are less than the threshold. Then, the area determination unit 217 determines a first imageable area 131 surrounded by the first marker 111 to the fourth marker 114 belonging to the determined first marker group, and a second imageable area 132 surrounded by the fifth marker 115 to the eighth marker 118 belonging to the determined second marker group.

[0103] Here, to the right of the first mark 111 for the upper left, there are not only the third mark 113 for the upper right, but also the seventh mark 117 for the upper right, and to the right of the second mark 112 for the lower left, there are not only the fourth mark 114 for the lower right, but also the eighth mark 118 for the lower right. Therefore, in the past, it was possible to determine the area 130 surrounded by the first mark 111, the second mark 112, the seventh mark 117, and the eighth mark 118 as the imageable area. When the area 130 is determined as the imageable area, the area containing the confidential information between the first imageable area 131 and the second imageable area 132 is no longer masked, and it is possible to send the masked image containing the confidential information to the remote terminal 3.

[0104] However, in the present embodiment, the region determination unit 217 determines a combination of a given number of marks in which the distances between the respective marks are less than the threshold value. That is, the distance 301 between the first mark 111 and the third mark 113 is less than the threshold value, but the distance 302 between the first mark 111 and the seventh mark 117 is longer than the threshold value. In addition, the distance 304 between the second mark 112 and the fourth mark 114 is less than the threshold value, but the distance 305 between the second mark 112 and the eighth mark 118 is longer than the threshold value. Therefore, the region surrounded by the first mark 111 to the fourth mark 114 is determined as the first imageable region 131, and the region surrounded by the fifth mark 115 to the eighth mark 118 is determined as the second imageable region 132, but the region 130 surrounded by the first mark 111, the second mark 112, the seventh mark 117, and the eighth mark 118 is not determined as the imageable region.

[0105] Furthermore, even if the distance 301 between the first mark 111 and the third mark 113 and the distance 302 between the first mark 111 and the seventh mark 117 are both below the threshold, the distance 301 is shorter than the distance 302, and therefore the third mark 113 is determined to belong to the same mark group as the first mark 111. Furthermore, even if the distance 304 between the second mark 112 and the fourth mark 114 and the distance 305 between the second mark 112 and the eighth mark 118 are both below the threshold, the distance 304 is shorter than the distance 305, and therefore the second mark 112 is determined to belong to the same mark group as the first mark 111.

[0106] When the second mark 112 and the sixth mark 116 are located below the first mark 111, the region determination unit 217 determines the second mark 112, which is closer to the first mark 111 and whose distance from the first mark 111 is less than the threshold, as a mark belonging to the first mark group. Furthermore, when the third mark 113 and the seventh mark 117 are located to the right of the first mark 111, the region determination unit 217 determines the third mark 113, which is closer to the first mark 111 and whose distance from the first mark 111 is less than the threshold, as a mark belonging to the first mark group. Furthermore, when the fourth mark 114 and the eighth mark 118 are located to the right of the second mark 112, the region determination unit 217 determines the fourth mark 114, which is closer to the second mark 112 and whose distance from the second mark 112 is less than the threshold, as a mark belonging to the first mark group. Furthermore, when the fourth mark 114 and the eighth mark 118 are below the third mark 113, the region identification unit 217 determines the fourth mark 114, which is closer to the third mark 113 and whose distance from the third mark 113 is less than the threshold, as a mark belonging to the first mark group.

[0107] In addition, the first mark 111 to the fourth mark 114 are respectively identified as one mark group when they are relatively located at the upper left, lower left, upper right, and lower right. For image processing, since the relative positions are not known, processing for identifying the relative positions is required. For this purpose, the following processing can also be performed.

[0108] First, the region identification unit 217 searches for a top left marker that serves as a reference in the captured image. The coordinate system of the captured image is represented by an x-axis with increasing coordinate values ​​from the origin to the right and a y-axis with increasing coordinate values ​​from the origin to the bottom, with the top left corner of the captured image as the origin.

[0109] If the upper left marker is detected, the region determination unit 217 searches for a lower left marker having a larger y coordinate value than the upper left marker. At this time, if a plurality of lower left markers are detected, the region determination unit 217 determines that the lower left marker, which is closest to the upper left marker and whose distance from the upper left marker is less than a threshold value, belongs to the same marker group as the upper left marker.

[0110] Next, the region identification unit 217 searches for a marker for the upper right having a value greater than the x-coordinate of the marker for the upper left. At this time, when a plurality of markers for the upper right are detected, the region identification unit 217 determines that the marker for the upper right, which is closest to the marker for the upper left and whose distance from the marker for the upper left is less than the threshold, belongs to the same marker group as the marker for the upper left.

[0111] Next, the region identification unit 217 searches for a lower right marker having a value greater than the x coordinate of the lower left marker and a value greater than the y coordinate of the upper right marker. At this time, when a plurality of lower right markers are detected, the region identification unit 217 determines that the lower right marker, which is closest to the lower left marker and the upper right marker, and whose distance from the lower left marker is less than the threshold value and whose distance from the upper right marker is less than the threshold value, belongs to the same marker group as the upper left marker.

[0112] Through the above processing, one marker group consisting of four markers is determined. Once one marker group is determined, the region specifying unit 217 further searches for a marker for the upper left portion of the captured image, and repeats the above processing until all markers for the upper left portion are detected.

[0113] When the region specifying unit 217 specifies the first marker group consisting of the first marker 111 to the fourth marker 114, it specifies the first imageable region 131 by connecting the upper left vertex of the upper left first marker 111, the lower left vertex of the lower left second marker 112, the upper right vertex of the upper right third marker 113, and the lower right vertex of the lower right fourth marker 114 with a straight line. Furthermore, when the region specifying unit 217 specifies the second marker group consisting of the fifth marker 115 to the eighth marker 118, it specifies the second imageable region 132 by connecting the upper left vertex of the upper left fifth marker 115, the lower left vertex of the lower left sixth marker 116, the upper right vertex of the upper right seventh marker 117, and the lower right vertex of the lower right eighth marker 118 with a straight line.

[0114] In addition, when the area determination unit 217 determines the four marks as a group of marks, it can also determine the photographable area by connecting the lower right vertex of the upper left mark, the upper right vertex of the lower left mark, the lower left vertex of the upper right mark, and the upper left vertex of the lower right mark with a straight line. In addition, when the area determination unit 217 determines the four marks as a group of marks, it can also determine the photographable area by connecting the center points of the four marks with a straight line.

[0115] Figure 5 This means that in this embodiment, based on Figure 4 FIG. 1 is a diagram showing an example of a mask image 141 created from the image 121 shown.

[0116] The mask image creation unit 218 creates a mask image 141 that masks the first photographable area 131 in the image 121 and the area 133 outside the second photographable area 132. The mask image creation unit 218 performs mask processing on the first photographable area 131 in the image 121 and the area 133 outside the second photographable area 132. The mask processing may be, for example, a process of filling the area 133 with a given color such as black, white, or gray, a process of blurring the area 133, or a process of mosaicking the area 133. Figure 5 In the mask image 141 shown, the photographic images in the first photographic region 131 and the second photographic region 132 are visually recognized, and the region 133 other than the first photographic region 131 and the second photographic region 132 is concealed.

[0117] In the present embodiment, the image processing device 2 is disposed on-site, but the present disclosure is not particularly limited thereto, and the image processing device 2 may be a server provided on the cloud. In this case, the image processing device 2 and the camera 1 are connected to each other via the network 4 so as to be able to communicate with each other.

[0118] Next, the job assisting process of the image processing device 2 in the embodiment of the present disclosure will be described.

[0119] Figure 6 is a first flowchart for explaining the operation assisting process of the image processing device 2 in the embodiment of the present disclosure. Figure 7 This is a second flowchart for explaining the work assisting process of the image processing device 2 in the embodiment of the present disclosure.

[0120] First, in step S1, the image acquisition unit 211 acquires an image captured by the camera 1. When the camera 1 is powered on or a button provided in the image processing device 2 for starting transmission of a mask image is pressed, an image is acquired from the camera 1.

[0121] Next, in step S2, the mark recognition unit 212 determines whether the initialization flag stored in the memory 22 is valid. Here, when the initialization flag is not valid, that is, the initialization flag is invalid (step S2 "No"), in step S3, the mark recognition unit 212 recognizes a plurality of marks contained in the image acquired by the image acquisition unit 211.

[0122] Next, in step S4, the mark recognition determination unit 213 determines whether a plurality of marks equal in number to the number of all marks belonging to the plurality of mark groups previously input by the input receiving unit 24 are recognized. Here, if it is determined that a plurality of marks equal in number to the number of all marks belonging to the plurality of mark groups previously input are not recognized (step S4 "No"), the process returns to step S1. In the present embodiment, two sets of mark groups are used, and since one set of mark groups is composed of four marks, the mark recognition determination unit 213 determines whether eight marks are recognized.

[0123] On the other hand, when it is determined that a plurality of marks equal in number to the number of all marks belonging to a plurality of mark groups input in advance are recognized (step S4 "Yes"), in step S5, the mark arrangement determination unit 214 determines whether the given number of marks of each mark group is arranged according to a predetermined positional relationship. Here, when it is determined that the given number of marks of each mark group is arranged according to a predetermined positional relationship (step S5 "Yes"), in step S6, the mark arrangement determination unit 214 validates the initialization flag stored in the memory 22. After the initialization flag is validated, the process returns to step S1.

[0124] On the other hand, when it is determined that the given number of marks of each mark group is not arranged according to the predetermined positional relationship (step S5 "No"), in step S7, the notification information output unit 215 outputs notification information for notifying the operator that the positional relationship between the given number of marks is wrong to the display unit 25. The display unit 25 displays the notification information for notifying the operator that the positional relationship between the given number of marks is wrong. After the display information is output, the process returns to step S1.

[0125] On the other hand, when it is determined that the initialization flag is valid (Yes in step S2 ), in step S8 , the marker recognition unit 212 recognizes a plurality of markers included in the image acquired by the image acquisition unit 211 .

[0126] Next, in step S9, the mark recognition determination unit 213 determines whether a plurality of marks equal in number to the number of all marks belonging to the plurality of mark groups input in advance by the input receiving unit 24 are recognized. In addition, the processing of step S9 is the same as the processing of step S4. Here, in the case where it is determined that a plurality of marks equal in number to the number of all marks belonging to the plurality of mark groups input in advance are not recognized (step S9 "No"), in step S10, the mark recognition determination unit 213 invalidates the initialization flag stored in the memory 22. After the initialization flag is invalidated, the processing returns to step S1.

[0127] On the other hand, when it is determined that the same number of markers as the number of all markers belonging to the plurality of marker groups previously input are recognized (step S9 “Yes”), in step S11 , the distance calculation unit 216 calculates the distance between each of the plurality of markers recognized by the marker recognition unit 212 .

[0128] Next, in step S12, the area determination unit 217 determines a first photographable area surrounded by a given number of markers belonging to the first marker group and a second photographable area surrounded by a given number of markers belonging to the second marker group based on the distances between the plurality of markers calculated by the distance calculation unit 216. The area determination unit 217 determines a first photographable area surrounded by four markers belonging to the first marker group among the eight markers recognized by the marker recognition unit 212 and a second photographable area surrounded by four markers belonging to the second marker group among the eight markers recognized by the marker recognition unit 212.

[0129] At this time, the area determination unit 217 determines the first marker group consisting of the first marker 111 to the fourth marker 114 whose distances calculated by the distance calculation unit 216 are less than the threshold value, and the second marker group consisting of the fifth marker 115 to the eighth marker 118 whose distances calculated by the distance calculation unit 216 are less than the threshold value. Then, the area determination unit 217 determines the first imageable area 131 surrounded by the first marker 111 to the fourth marker 114 belonging to the determined first marker group, and the second imageable area 132 surrounded by the fifth marker 115 to the eighth marker 118 belonging to the determined second marker group.

[0130] Next, in step S13 , the mask image creation unit 218 creates a mask image by performing masking processing on the areas outside the first and second imageable areas determined by the in-image area determination unit 217 .

[0131] Next, in step S14, the mask image output unit 219 transmits the mask image created by the mask image creating unit 218 to the remote terminal 3 via the communication unit 23. The remote terminal 3 receives the mask image transmitted from the image processing device 2. The remote terminal 3 displays the received mask image.

[0132] Next, in step S15, the mask image output unit 219 determines whether to terminate the transmission of the mask image. The transmission of the mask image is terminated when the power of the camera 1 is turned off or when a button for terminating the transmission of the mask image provided in the image processing device 2 is pressed.

[0133] Here, when it is determined that the transmission of the mask image is to be terminated (Yes in step S15 ), the mask image output unit 219 terminates the transmission of the mask image in step S16 .

[0134] On the other hand, when it is determined that the transmission of the mask image is not to be completed (No in step S15 ), the process returns to step S1 .

[0135] In this way, when a plurality of marks included in an image are recognized, the distances between the recognized marks are calculated. Then, based on the calculated distances, a first photographable area surrounded by a given number of marks belonging to a first marker group among the recognized marks and a second photographable area surrounded by a given number of marks belonging to a second marker group among the recognized marks are determined. Then, a mask image is created that masks the areas outside the first photographable area and the second photographable area in the image, and the mask image is output.

[0136] Therefore, even when a plurality of imageable areas are set, the plurality of imageable areas are determined respectively, so that an image with the areas outside the plurality of imageable areas appropriately masked can be output. That is, it is possible to prevent the first imageable area and the second imageable area from being mistakenly determined as one imageable area, and to prevent the output of an image containing confidential information located in one mistakenly determined imageable area. As a result, an operator at the site can receive efficient work assistance from an operator at a remote location.

[0137] In addition, in the present embodiment, the input receiving unit 24 receives the input of the number of marker groups used and the number of given number of markers belonging to each marker group by the operator, and when a plurality of markers are initially recognized, the marker recognition determination unit 213 determines whether a plurality of markers are recognized that are the same number as the number of all markers belonging to the plurality of marker groups previously input by the input receiving unit 24, but the present disclosure is not particularly limited to this. Figure 6 After the process of step S3 , when a plurality of markers are initially recognized, the display unit 25 presents the image acquired by the image acquisition unit 211 to the operator.

[0138] At this time, the display unit 25 performs display so that the recognized multiple marks can be identified. For example, the display unit 25 can perform display so that the recognized multiple marks are respectively surrounded by lines of given colors. The input receiving unit 24 can accept the input of whether all marks belonging to multiple mark groups are recognized in the image by the operator. The input receiving unit 24 includes a first button for inputting all marks belonging to multiple mark groups recognized in the image, and a second button for inputting all marks belonging to multiple mark groups not recognized in the image. The operator visually confirms whether all marks belonging to multiple mark groups are recognized in the displayed image. When all marks belonging to multiple mark groups are recognized, the operator presses the first button, and when all marks belonging to multiple mark groups are not recognized, the operator presses the second button.

[0139] In addition, the first button and the second button may be displayed on the display unit 25. The operator may use a mouse to move the cursor on the screen to the first button or the second button and click it. In addition, when the display unit 25 is a touch panel, the operator may touch either the first button or the second button on the screen.

[0140] Then, when the operator inputs that all the marks belonging to the entire mark group are recognized, the mark arrangement determination unit 214 can determine whether the given number of marks are arranged according to a predetermined positional relationship (processing of step S5). On the other hand, when the operator inputs that all the marks belonging to the entire mark group are not recognized, the processing returns to step S1.

[0141] In addition, in each of the above-mentioned embodiments, each component can be composed of dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. In addition, by recording the program on a recording medium and transferring it, or by transferring the program via a network, the program can be implemented by another independent computer system.

[0142] Part or all of the functions of the device involved in the embodiments of the present disclosure are typically implemented as an integrated circuit, i.e., LSI (Large Scale Integration). These can be individually single-chip, or part or all of them can be single-chip. In addition, integrated circuitization is not limited to LSI, and can also be implemented using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, and a reconfigurable processor that can reconfigure the connection and settings of the circuit units inside the LSI.

[0143] Furthermore, a part or all of the functions of the apparatus according to the embodiments of the present disclosure may be realized by executing a program by a processor such as a CPU.

[0144] In addition, all the numbers used in the above description are exemplifications for specifically describing the present disclosure, and the present disclosure is not limited to the exemplified numbers.

[0145] In addition, the order of executing each step shown in the above flowchart is an example for specifically explaining the present disclosure, and other orders may be used within the scope of obtaining the same effect. In addition, part of the above steps may be executed simultaneously (in parallel) with other steps.

[0146] Industrial Applicability

[0147] The technology involved in the present disclosure can output an image in which areas other than the plurality of imageable areas are appropriately masked even when a plurality of imageable areas are set. Therefore, it is useful as a technology for masking areas other than the plurality of imageable areas in an image.

Claims

1. An information processing method is an information processing method in a computer. The information processing method performs the following processing: Get the image captured by the camera, identifying a plurality of landmarks contained within the image, Calculating the distances between the identified multiple markers, Based on the calculated distance, a first photographable area surrounded by a given number of marks belonging to a first marker group among the plurality of recognized marks and a second photographable area surrounded by a given number of marks belonging to a second marker group among the plurality of recognized marks are determined, creating a mask image that masks the first imageable area in the image and an area outside the second imageable area, The mask image is output.

2. The information processing method according to claim 1, wherein: In determining the first photographable area and the second photographable area, the first marker group consisting of the given number of markers whose distance is less than a threshold value and the second marker group consisting of the given number of markers whose distance is less than a threshold value are determined.

3. The information processing method according to claim 1 or 2, wherein: In determining the first photographable area and the second photographable area, when there are a plurality of markers of the same type, the marker closest to the reference marker is determined as a marker belonging to the same marker group as the reference marker.

4. The information processing method according to claim 1 or 2, wherein: predetermining the positional relationship between the given number of marks belonging to the first mark group and the second mark group, respectively; Furthermore, when the plurality of marks are initially recognized, it is determined whether the given number of marks are arranged in accordance with the predetermined positional relationship. Furthermore, when it is determined that the given number of markers are not arranged in accordance with the predetermined positional relationship, notification information for notifying the operator that the positional relationship between the given number of markers is wrong is output.

5. The information processing method according to claim 1 or 2, wherein: predetermining the positional relationship between the given number of marks belonging to the first mark group and the second mark group, respectively; Furthermore, when the plurality of marks are initially recognized, it is determined whether the given number of marks are arranged in accordance with the predetermined positional relationship. Furthermore, when it is determined that the given number of marks are not arranged in accordance with the predetermined positional relationship, notification information for notifying the operator that the marks are not arranged in accordance with the predetermined positional relationship is output.

6. The information processing method according to claim 4, wherein: Furthermore, the operator is accepted to input the number of the plurality of marker groups to be used and the number of the given number of markers belonging to each marker group, Furthermore, when the plurality of marks are initially recognized, it is determined whether the plurality of marks are recognized in the same number as the number of all marks belonging to the plurality of mark groups inputted, In determining whether the given number of marks are arranged according to the predetermined positional relationship, when it is determined that the same number of multiple marks as the number of all marks belonging to the multiple mark groups input are identified, it is determined whether the given number of marks are arranged according to the predetermined positional relationship.

7. The information processing method according to claim 4, wherein: Furthermore, when the plurality of marks are initially recognized, the acquired image is presented to the operator. Furthermore, the operator is inputted whether all the marks belonging to the plurality of mark groups are recognized in the image. In determining whether the given number of marks are arranged according to the predetermined positional relationship, when the operator inputs all the marks identified as belonging to the entire mark group, it is determined whether the given number of marks are arranged according to the predetermined positional relationship.

8. The information processing method according to claim 1 or 2, wherein: The given number of marks belonging to the first mark group includes a first mark arranged at an upper left position in a first work area capable of being photographed within a work space, a second mark arranged at a lower left position, a third mark arranged at an upper right position, and a fourth mark arranged at a lower right position, The given number of marks belonging to the second mark group include a 5th mark arranged at the upper left position in the second work area that can be photographed within the work space, a 6th mark arranged at the lower left position, a 7th mark arranged at the upper right position, and an 8th mark arranged at the lower right position.

9. The information processing method according to claim 8, wherein: In determining the first imageable area and the second imageable area, When the second mark and the sixth mark are below the first mark, the second mark which is closer to the first mark and whose distance from the first mark is less than a threshold value is determined as a mark belonging to the first mark group. When the third mark and the seventh mark are located to the right of the first mark, the third mark which is closer to the first mark and whose distance from the first mark is less than a threshold value is determined as a mark belonging to the first mark group. When the fourth mark and the eighth mark are located to the right of the second mark, the fourth mark which is closer to the second mark and whose distance from the second mark is less than a threshold value is determined to be a mark belonging to the first mark group. When the fourth marker and the eighth marker are located below the third marker, the fourth marker, which is closer to the third marker and whose distance from the third marker is equal to or less than a threshold, is determined to be a marker belonging to the first marker group.

10. An information processing device comprising: an acquisition unit that acquires an image captured by a camera; a recognition unit for recognizing a plurality of marks contained in the image; a calculating unit for calculating the distance between each of the plurality of recognized markers; a determination unit that determines, based on the calculated distance, a first imageable area surrounded by a given number of marks belonging to a first mark group among the plurality of recognized marks, and a second imageable area surrounded by a given number of marks belonging to a second mark group among the plurality of recognized marks; a creating unit that creates a mask image that masks the first imageable area in the image and an area outside the second imageable area; and An output unit outputs the mask image.

11. An information processing program that causes a computer to function as follows: Get the image captured by the camera, identifying a plurality of landmarks contained within the image, Calculating the distances between the identified multiple markers, Based on the calculated distance, a first photographable area surrounded by a given number of marks belonging to a first marker group among the plurality of recognized marks and a second photographable area surrounded by a given number of marks belonging to a second marker group among the plurality of recognized marks are determined, creating a mask image that masks the first imageable area in the image and an area outside the second imageable area, The mask image is output.

Images