Image difference detection system, image difference detection method, and image difference detection program

Abstract

To provide an image difference detection system, an image difference detection method, and an image difference detection program that do not perform unnecessary difference detection for the purpose of preventing human error in a process after work such as maintenance work.SOLUTION: An image difference detection system according to the present invention includes a label 12 attached so as to designate an object for difference detection and having region information specifying a detection region for detecting the difference between images before and after work, a detection region information acquisition unit 53 that recognizes the region information from the captured image of the label, a difference detection unit 56 that obtains an image difference of the detection region indicated by the region information in the captured image of the object before and after the work, and a difference information output unit 57 that outputs the image difference.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an image difference detection system, an image difference detection method, and an image difference detection program.

Background Art

[0002] There is a technology of image difference detection that compares two images captured by a camera and extracts a difference region. Typical application examples of this technology include detecting the occurrence of an abnormal situation by comparing a stored normal image with the current image, and estimating the movement amount of a moving object from the magnitude of the difference between two images. For example, Patent Document 1 describes an image difference detection device characterized by a function of promoting accurate detection of a difference region by correcting the positional deviation between two images.

[0003] Conventionally, captured images by a camera with a fixed position such as a surveillance camera have often been the target of difference detection. In recent years, with the spread of mobile terminals equipped with cameras such as smartphones, it has become easy to capture images at any location, so the utilization scenes of image difference detection are increasing.

[0004] As an example, there is utilization in maintenance and repair work. In this maintenance and repair work, for the purpose of preventing human errors in the process after work, it is confirmed whether the On-Off of switches and the arrangement of parts are different from the state before work. At that time, the operator takes images before and after work with a smartphone or the like carried by the operator, and based on the difference detection results for those images, the work results are confirmed, thereby performing a confirmation work with higher certainty than simple visual confirmation.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the above prior art, there is a problem that it detects both meaningful and meaningless differences without distinction for the purpose of preventing human errors in the confirmation work. Taking the maintenance and repair work of wiring cables as an example, the difference in the connection position of the cables can lead to accidents, so it is a meaningful difference. However, even if there is a difference in the position of the middle part of the cable, in many cases, it is considered a meaningless difference. In addition, it is difficult to completely match the routing before and after the operation even if one is conscious of non-rigid objects such as the middle part of the cable.

[0007] When a large amount of meaningless difference information such as the middle part of the cable is detected, the possibility that the operator will overlook meaningful difference information increases. Therefore, it is desirable that the information presented as a result of the difference detection be strictly selected for meaningful difference information as much as possible.

[0008] An object of the present invention is to provide an image difference detection system that does not perform unnecessary difference detection.

Means for Solving the Problem

[0009] In order to solve the above problems, the image difference detection system of the present invention is provided with region information that is attached so as to specify an object to be detected for difference detection and that specifies a detection region for detecting an image difference before and after the operation, a detection region information acquisition unit that recognizes the region information from a photographed image of the label, a difference detection unit that obtains an image difference of a detection region indicated by the region information in the photographed images before and after the operation of the object, and a difference information output unit that outputs the image difference. When acquiring a captured image of an object after the operation, there is a guidance unit that performs imaging guidance for guiding the imaging position based on the position information of the label extracted by the detection region information acquisition unit from the captured image before the operation. The guidance unit obtains the region number of the detection region of the label in the frame of the image stream of the post-operation image, estimates a projective transformation matrix from the coordinate information of the four corners of the label corresponding to the region number in the captured image before the operation and the coordinate information of the four corners of the label in the image frame, and obtains guidance information from the projective transformation matrix. It was made like this.

[0010] Further, the image difference detection method of the present invention is an image difference detection method for detecting the difference between the photographed images of the object before and after the operation, and includes a step of registering an extraction condition for extracting from the photographed image a label on which region information for specifying a detection region for detecting the image difference before and after the operation is described so as to specify the object to be detected for difference detection; a step of extracting the label from the photographed image of the object and recognizing the region information described on the label; and a step of obtaining the image difference of the detection region indicated by the region information in the photographed images of the object before and after the operation. When capturing a captured image of the object after the operation, it includes the step of displaying guidance information for guiding the imaging position so as to be at the same position as the imaging position of the captured image before the operation. The region number of the detection region of the label in the frame of the image stream of the post-operation image is obtained, a projective transformation matrix is estimated from the coordinate information of the four corners of the label corresponding to the region number in the captured image before the operation and the coordinate information of the four corners of the label in the image frame, and guidance information is obtained from the projective transformation matrix. was made.

Effect of the Invention

[0011] According to the present invention, it is possible to provide an image difference detection system that performs difference detection useful for preventing human errors based on the photographed images before and after the operation.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 8

Figure 9

Figure 10A

Figure 10B

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not to be construed as being limited to the description of the embodiments shown below. It will be readily understood by those skilled in the art that the specific configuration can be changed without departing from the spirit or gist of the present invention.

[0014] In the configuration of the invention described below, the same reference numerals are commonly used among different drawings for the same part or parts having the same or similar functions, and redundant descriptions may be omitted. When there are a plurality of elements having the same or similar functions, they may be described with different subscripts attached to the same reference numeral. However, when it is not necessary to distinguish between the plurality of elements, the subscripts may be omitted in the description.

[0015] Also, the positions, sizes, shapes, ranges, etc. of the respective configurations shown in the drawings and the like may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate the understanding of the invention. For this reason, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings and the like. In this specification, components represented in the singular form include the plural form unless otherwise clearly indicated in the context.

[0016] In the embodiments, as a specific use case of the present invention, the process of confirming the differences before and after work in maintenance and repair work will be described. However, this is for illustrative purposes only and does not limit the use cases.

[0017] First, with reference to FIG. 1, the outline of the image difference detection system according to the embodiment will be described. Specifically, a case where the confirmation of cable replacement work, which is one of the equipment maintenance and repair operations, is performed by the image difference detection system according to the embodiment will be described.

[0018] As shown in FIG. 1, in cable replacement work, in order to cut off the current of the cable to be replaced, the switch is first turned off, and after the cable is replaced, the switch is turned on to return the equipment to its state before the work. Connecting the cable to a different connector or forgetting to return the switch is an incorrect operation in cable replacement work.

[0019] Therefore, in the image difference detection system according to the embodiment, the difference detection device 11 composed of a portable computer such as a smartphone or a tablet takes pictures of the state of the switch, connector, and wiring cable before the work, and after the cable replacement work, takes pictures of the state of the same switch, connector, and wiring cable. Then, the difference detection device 11 detects the difference between the captured images before and after the work, so that the operator can easily confirm the defective cable replacement work. At this time, when instruments indicating the operating state of the equipment are provided, the difference between the captured images of the instruments before and after the work is detected to confirm the coincidence of the meter pointers.

[0020] In the image difference detection system according to the embodiment, for each object such as the switch, connector, wiring cable, and instruments that are the objects of detecting the difference in the captured images, the label 12 is used to set the area to be detected for the difference.

[0021] The label 12 is a planar object such as a magnet sheet or a sticker that an operator carries to the site. However, it is not limited to a magnet sheet or a sticker as long as it has a flat shape and can be arbitrarily written with a string or a mark using a magic marker or a ballpoint pen. Note that the label 12 is not limited to being flat, and it may have a thickness, unevenness, or curvature as long as it does not inhibit the function as the label 12.

[0022] After the operator confirms the structure of the object, the operator determines the area for differential detection based on the captured image and the position where the label 12 can be attached, writes the area information described later by hand on the label 12, and attaches the written label 12 so that it designates the captured area. In the example of FIG. 1, the label 12 is attached close to the object. The area may be defined widely so as to surround a plurality of objects, or may be defined finely for each of the plurality of objects.

[0023] FIG. 2 is a diagram showing an example of the description of area information for specifying a detection area for detecting the image difference of the label 12. The area number 21 is an identification number for specifying a detection area for detecting an image difference. It is identification information for distinguishing each other when there are a plurality of detection areas (in the example of FIG. 2, the area number 21 is "1").

[0024] The area shape 22 is graphic information indicating the outer frame shape (outer shape) of the detection area for detecting the image difference. For example, corresponding to the outer frame shape of the detection area, a square, a rectangle (vertically long / horizontally long), or a circle is described. Note that the outer frame shape of the detection area may be determined so that a plurality of objects to be differentially detected are included in the detection area (in the example of FIG. 2, the area shape 22 is "square").

[0025] The area size 23 is a numerical value indicating the size (width) of the detection area for detecting the image difference specified by the area shape 22. A numerical value may be assigned to the size in the pre-specified image, or it may be a magnification with respect to a predetermined unit area. The setting direction 24 is an arrow indicating the direction located with respect to the label 12 of the detection area for detecting the image difference (in the example of FIG. 2, the area size 23 is "3").

[0026] The difference detection device 11 recognizes and acquires the following area information, that is, the area number 21, the area shape 22, the area size 23, and the setting direction 24 from the photographed image of the label 12 by using the pattern recognition technology of characters and symbols.

[0027] The acquisition of the above area information is performed by extracting the area of the label 12 itself pasted from the photographed image and performing pattern recognition on the extracted area. For this reason, although the details will be described later, it is desirable to make the background color of the label 12 a color scheme different from the working environment of the object or the like and use this as the extraction condition to facilitate the extraction of the image of the label 12 itself from the photographed image.

[0028] Next, a method for designating the detection area for detecting the difference of the object by the label 12 will be described with reference to FIGS. 3A and 3B. The difference detection device 11 performs difference detection processing on the detection area designated by the label 12.

[0029] In FIG. 3A, an arrow of the label 12 designates a detection area with the area number "1", the installation direction being "downward" in contact with the label 12, the area shape being "square", and the area size being "3". It is assumed that the detection area is provided in contact with the label 12 or at a position separated by a predetermined distance.

[0030] In FIG. 3B, an arrow of the label 12 designates a detection area with the area number "1", the installation direction being "rightward" in contact with the label 12, the area shape being "circle", and the area size being "2".

[0031] Next, FIG. 4 shows the difference in the detected difference before and after the operation detected by the difference detection device 11 depending on the presence or absence of the label 12 for designating the detection area. Figure 4 shows the pre-operation captured image, the post-operation captured image, and the detected differences of the cable replacement operation described in Figure 1, for each case of the presence or absence of the label 12 specified by the detection area. Note that for the switch, it has changed from "ON" to "Off" before and after the operation. Among these, since "O" has not changed, it is not displayed in Figure 4, while the changed "N" and "ff" are displayed superimposed.

[0032] The detected differences indicate the ON / OFF state of the switch, the needle positions of the instruments, and the differences in the cable connection destinations, regardless of the presence or absence of the label (the presence or absence of the specified detection area), and can detect incorrect cable connections and forgetting to return the switch.

[0033] However, when there is no label, it has also detected that the routing of the cable is different, and all differences are detected regardless of the meaning for preventing human errors. On the other hand, when there is a label, only the differences in the detection area with a predefined meaning for the operator are detected, and useful difference detection for preventing human errors can be performed.

[0034] The advantage of specifying the detection area by the label 12 is not only that it is convenient for the operator, but also that the detection area can be specified flexibly according to the objects at the site. For example, if only a detection area with a fixed area shape and area size needs to be specified, a marker such as a QR code (registered trademark) with area information embedded in advance can be created and attached to the object.

[0035] However, since the area shape, area size, attachment position, and number of areas of the detection area for detecting appropriate differences vary depending on the object, in order to cope with them, the operator needs to carry a large number of markers. Therefore, the method of preparing the markers in advance has a high preparation cost and a large burden on the operator.

[0036] In the image difference detection system of the embodiment, after an operator confirms an object to be detected for differences, area information is manually entered into the label 12. This high degree of freedom is particularly effective in maintenance and repair work where it is often necessary to define a detection area for detecting differences in an object seen for the first time at the customer's site.

[0037] Furthermore, in the image difference detection system of the embodiment, area information is entered into the label 12 using numerical values that are easy to enter and recognize and simple-shaped figures. This makes recognition easier and can reduce misrecognition of the detection area. Also, by attaching the label 12 near the object, the image difference detection system can improve the positional accuracy of the detection area to be detected for differences.

[0038] Hereinafter, the configuration of the difference detection device 11 of the image difference detection system will be described in detail. The difference detection device 11 is composed of a portable computer such as a tablet or a smartphone including a CPU that performs arithmetic processing, a memory, a camera, a communication unit, an operation unit, a display unit, and a non-volatile storage medium. Then, by the CPU executing a program stored in the non-volatile storage medium, each function of the difference detection device 11 is realized.

[0039] FIG. 5 is a functional block diagram of the difference detection device 11. The difference detection device 11 is composed of a still image capturing unit 51, a label registration unit 52, a label area information recognition unit 53, an area information manual correction unit 54, a difference detection target image registration unit 55, a difference detection unit 56, a result display unit 57, a control unit 58, a label registration DB 59, an area information DB 510, and a captured image DB 511. Hereinafter, each block will be described.

[0040] The following description will be for the case where each functional block is realized by a tablet, but it is not limited to this, and some functional blocks may be configured by a computer on a network connected via the communication unit.

[0041] The still image capturing unit 51 cooperates with the tablet camera to acquire a captured image (still image) of the object for facility maintenance and repair work. The still image capturing unit 51 captures the object to be subjected to differential detection when registering the label 12 that designates the detection area of the object to be subjected to differential detection, and before and after the cable replacement work.

[0042] Specifically, the still image capturing unit 51 notifies the label registration unit 52 of the captured image and stores the captured image in the captured image DB 511.

[0043] The label registration unit 52 registers in the label registration DB 59 the extraction conditions for extracting the label 12 from the captured image. The differential detection device 11 extracts the label 12 from the captured image according to the registered extraction conditions by the label area information recognition unit 53 described later, and recognizes the area information described in the label 12.

[0044] The simplest example of the extraction condition is color information. The background color of the label 12 is set as a gaudy fluorescent color that is unlikely to exist in the target device for maintenance and repair work, and this is used as the extraction condition. Of course, not limited to color information, when the label 12 has some characteristic texture, it is also possible to register the image feature amount thereof.

[0045] The label registration unit 52 automatically extracts the dominant color from the image in which only the label 12 is captured largely, or the operator manually designates the area in the image and registers it in the label registration DB 59 as the extraction condition. The image feature amount of the texture can also be extracted by the same procedure.

[0046] The label area information recognition unit 53 recognizes the area information (area number 21, area shape 22, area size 23, and installation direction 24) of the detection area to be differentially detected described in the label 12, and stores the recognized area information in the area information DB 510. The label area information recognition unit 53 is executed when acquiring a captured image (pre-work image) showing the state of the object before the work.

[0047] Specifically, the label area information recognition unit 53 extracts image information corresponding to the label 12 from the captured image based on the extraction conditions of the label 12 registered in the label registration DB 59. Then, it recognizes the area information described by characters and graphics from the extracted image information.

[0048] For the character and graphic recognition of the area information, a discriminator (such as a neural network) that has learned the image patterns of handwritten characters and marks (graphics) is used. This discriminator outputs, in addition to the recognition type, a quantified recognition confidence level (reliability of the recognition result).

[0049] When the confidence level of the recognized area information is low (below a predetermined threshold), the label area information recognition unit 53 corrects the area information by the area information manual correction unit 54 described later, and then saves the area information in the area information DB 510.

[0050] The area information manual correction unit 54 displays the image information of the label 12 and the recognized area information on the display unit of the tablet, enabling the operator to manually correct the area information recognized by the label area information recognition unit 53 using the operation unit of the tablet.

[0051] FIG. 6 is an example of a correction screen for the area information displayed by the area information manual correction unit 54. The area information manual correction unit 54 displays the captured image (pre-operation image) showing the state of the object before the operation in the display range of the captured image on the display screen of the tablet, and displays the image information corresponding to the label 12 extracted by the label area information recognition unit 53 as an enlarged view of the target label.

[0052] The area information manual correction unit 54 displays, at the bottom of the screen, the recognition results with low confidence levels by the label area information recognition unit 53 for each of the area information (area number, area shape, area size, and installation direction).

[0053] The area information manual correction unit 54 sets the recognition result as the initial value and adds other assumed values to display a list box so that the operator can manually correct the display of the area information to any information. The operator manually corrects the area information that is misrecognized by operating the list box.

[0054] Note that there may be a case where the area of label 12 itself is not detected. In that case, the label area information recognition unit 53 may combine the process of cutting out the image information of label 12 from the captured image (pre-operation image) showing the state of the object before the operation with the manual correction by the above-mentioned area information manual correction unit 54.

[0055] Also, regardless of the confidence level of the recognition result of the area information, the label area information recognition unit 53 may once confirm with the operator whether the recognition result is correct for all the recognition results, and perform manual correction by the area information manual correction unit 54 for the area information whose recognition result is incorrect.

[0056] Returning to FIG. 5, the remaining functional blocks will be described. The differential detection target image registration unit 55 stores the pre-operation image or the post-operation image (captured image showing the state of the object after the operation) acquired by the still image capturing unit 51 in the captured image DB 511. Note that the pre-operation image is also notified to the label area information recognition unit 53, and the area information of the detection area for differential detection described in label 12 is recognized.

[0057] The differential detection unit 56 performs differential detection of the detection area indicated by label 12 between the pre-operation image and the post-operation image stored in the captured image DB 511.

[0058] For the post-operation image, the differential detection unit 56 extracts label 12 based on the extraction conditions of label 12 registered in the label registration DB 59, recognizes the area information by the label area information recognition unit 53, and acquires the detection area for differential detection based on the area information. At this time, when the confidence level of the recognized area information of label 12 is low, manual correction of the area information is performed by the area information manual correction unit 54. For the pre-operation image, the differential detection unit 56 acquires the detection area for differential detection based on the area information stored in the area information DB 510.

[0059] In addition, for each of the extracted labels 12 in the post-operation image, the difference detection unit 56 identifies the corresponding label 12 in the pre-operation image based on the position information of the label 12 in the captured image, obtains the region information of the identified label 12 by referring to the region information DB 510, and may acquire the detection region of the post-operation image based on this region information.

[0060] Then, for each detection region where the region numbers correspond, the difference detection unit 56 performs difference detection by subtracting the pixel value (brightness) at the same position in the detection region of the pre-operation image from each pixel in the detection region of the post-operation image and then taking the absolute value. For the result of the difference value, a threshold for cutting off meaningless difference values such as minute brightness fluctuations is set, and only the difference values exceeding the threshold are notified to the result display unit 57. Note that this threshold may be set for each detection region to be subjected to difference detection.

[0061] In addition, the difference detection unit 56 may perform a process for reducing the positional deviation between the pre-operation and post-operation captured images as a pre-process of the difference detection process. For example, the post-operation captured image may be subjected to projective transformation so that the positions of the respective labels 12 in the post-operation captured image captured by the still image capturing unit 51 coincide with the positions of the respective labels 12 in the pre-operation captured image captured by the still image capturing unit 51, and used as the post-operation captured image.

[0062] The result display unit 57 outputs the difference information, which is the processing result of the difference detection unit 56, to the display unit of the tablet to prompt the operator to confirm.

[0063] As the difference detection device 11, the control unit 58 controls the operations of the above-described respective functional blocks in the registration of the label 12 for designating the detection region of the object to be subjected to difference detection, the capturing of the object before and after the cable replacement operation, and the difference detection process of the captured image of the object.

[0064] In addition, when the pre-operation and post-operation captured images are not stored in the captured image DB 511, the control unit 58 prompts the operator to capture the object, and performs difference detection after the capturing.

[0065] Next, the operation of the difference detection device 11 of the embodiment will be described with reference to the flowcharts of FIGS. 7A, 7B, and 7C. The operation of the difference detection device 11 is roughly classified into pre - preparation processing, processing at the start stage of maintenance and repair work, and processing at the end stage of maintenance and repair work.

[0066] FIG. 7A is a flowchart of the pre - preparation processing of the difference detection device 11. In step S71, the control unit 58 of the difference detection device 11 registers, in the label registration DB59, the extraction conditions for the label 12 to be used by the label registration unit 52. Since this work is possible if the label 12 carried by the operator is determined, this work may be carried out after going to the site or before going to the site.

[0067] Before starting the work, the operator determines how to define the area at which position of the object to be maintained and repaired, describes it on each label 12, and attaches it to each position. Then, the difference detection device 11 photographs the object before the maintenance and repair work. The difference detection device 11 performs the processing at the start stage of the maintenance and repair work shown in FIG. 7B.

[0068] In step S72, the still - image photographing unit 51 acquires a photographed image (pre - work image) of the object before the difference detection, and the difference - detection target image registration unit 55 stores the photographed image in the photographed - image DB511.

[0069] In step S73, the label area information recognition unit 53 extracts the image information of the label 12 from the photographed image (pre - work image) based on the extraction conditions of the label 12 registered in the label registration DB59. Then, it performs character and graphic recognition on the extracted image information to obtain area information.

[0070] In step S74, it is determined whether the confidence level of the area information recognized in step S73 is higher than a predetermined threshold value. If it is higher (Yes in S74), the process proceeds to step S76; if it is not higher (No in S74), the process proceeds to step S75.

[0071] In step S75, as described with reference to FIG. 6, the area information manual correction unit 5 manually corrects the area information recognized by the label area information recognition unit 53.

[0072] In step S76, the label area information recognition unit 53 stores the area information described in the recognized or manually corrected label 12 in the area information DB510. The above processing from steps S73 to S76 is performed for all the labels 12 of the captured image (pre-operation image).

[0073] When the operator finishes the maintenance and repair work, the operator captures an image of the object on which the work has been performed. The difference detection device 11 performs the processing at the end stage of the maintenance and repair work shown in FIG. 7C.

[0074] In step S77, the still image capturing unit 51 acquires a captured image (post-operation image) of the object after the work that is the object of difference detection, and the difference detection target image registration unit 55 stores the captured image in the captured image DB511.

[0075] In step S78, the difference detection unit 56 performs difference detection of the detection areas for each corresponding label 12 of the captured images before and after the work.

[0076] Specifically, for the pre-operation image in the captured image DB511, the difference detection unit 56 extracts, for each area number of the area information, the captured image of the detection area for which difference detection is to be performed from the captured image based on the area information in the area information DB510, and associates the area number of the area information with the captured image of the extracted detection area and stores the result in the captured image DB511.

[0077] Also, for the post-operation image in the captured image DB511, for each of the labels 12 of the post-operation image, the difference detection unit 56 extracts, based on the area information in the area information DB510 of the label 12 of the corresponding pre-operation image, the captured image of the detection area for which difference detection is to be performed from the captured image, and associates the area number of the area information with the captured image of the extracted detection area and stores the result in the captured image DB511.

[0078] Then, the difference detection unit 56 performs difference detection on the captured images of the detection areas of the object before and after the operation, which are associated with the area numbers of the area information in the captured image DB 511.

[0079] In step S79, the result display unit 57 displays the result of the difference detection performed in step S78 on the display unit of the tablet to prompt the operator to confirm.

[0080] According to the difference detection device 11 of the above embodiment, since the difference detection is performed according to the area information of the difference area of the object to be detected for the difference set by the operator using the label 12, it is possible to strictly select the difference information based on the operator's experience and past cases into something meaningful for preventing human errors, and the difference confirmation process before and after the operation can be performed more effectively.

[0081] In the difference detection of the detection areas before and after the operation, since the positional deviation of the captured image is also detected as a difference, in order to prevent the detection of meaningless difference information, it is important to correct one of the images so that the positional deviation is small between the images, or to capture the images before and after the operation from the same spatial position so that no positional deviation occurs.

[0082] In the above difference detection device 11, the captured image after the operation captured by the still image capture unit 51 is subjected to projective transformation so that the positions of the respective labels 12 in the captured image after the operation match the positions of the respective labels 12 in the captured image before the operation, and by using the captured image after the operation, if the object is in a planar shape, it is possible to eliminate the positional deviation.

[0083] However, when the object is a three-dimensional object, if a visual change occurs due to a change in the shooting position, an area that cannot be associated between the captured images may occur, and in some cases, it may not be possible to accurately perform the projective transformation to eliminate the positional deviation.

[0084] In the difference detection device 11 described below, when an operator takes a picture of an object that has undergone maintenance or repair work after the work, the operator is given shooting guidance. As a result, even when the object is a three-dimensional object, it is possible to obtain captured images before and after the work with little positional deviation, preventing the user from being presented with meaningless difference information due to positional deviation, and enabling the difference confirmation process before and after the work to be carried out more effectively.

[0085] FIG. 8 is a diagram showing an example of a scene in which the difference detection device 11 of the embodiment gives shooting guidance when an operator takes a picture of a post-work image. The difference detection device 11 displays on the display unit of the tablet a real-time captured image of the current object and guidance information for guiding the operator to the shooting position of the pre-work image.

[0086] The guidance information is typically an instruction to move the device in the up, down, left, right, front, and back directions, and may be indicated in words as shown in FIG. 8, or may be illustrated by converting it into an arrow. Also, the magnitude of the movement amount may be expressed by an adjective, or converted into the size or color of the arrow.

[0087] The difference detection device 11 notifies the operator that it is the shooting timing when the difference detection device 11 moves to a shooting position where there is little positional deviation between the captured images before and after the work, that is, a position that is spatially sufficiently equal to the position at the time of shooting the pre-work image, thereby prompting the acquisition (shooting) of the post-work image.

[0088] FIG. 9 is a functional block diagram of the difference detection device 11 that displays shooting guidance. The difference detection device 11 is configured by adding the functional blocks of an image stream input unit 91, a guidance information generation / instruction unit 92, a label four-corner coordinate acquisition unit 93, and a label coordinate DB 94 to the difference detection device 11 described in FIG. 5. Since the parts other than the above-described added functional blocks are the same as those in FIG. 5, the description is omitted here.

[0089] The image stream input unit 91 acquires, as an image stream, the image information of the camera of the difference detection device 11 that is facing the object in the scene where the post-operation image is captured, and notifies the guidance information generation / instruction unit 92.

[0090] The guidance information generation / instruction unit 92 generates guidance information on how to spatially move the difference detection device 11 for the image stream of the object notified from the image stream input unit 91, and superimposes and displays the notified image stream and the guidance information. The content of the guidance information generation process will be described later.

[0091] The label four-corner coordinate acquisition unit 93 acquires the four-corner coordinates of the label 12 extracted from the pre-operation image and the image of the frame of the image stream. The four-corner coordinates of the label 12 acquired from the pre-operation image by the label four-corner coordinate acquisition unit 93 are stored in the label coordinate DB 94, and the four-corner coordinates of the label 12 acquired from the image of the frame of the image stream are notified to the guidance information generation / instruction unit 92.

[0092] Next, the operation flow of the difference detection device 11 that displays the shooting guidance will be described. FIG. 10A is a process flow diagram at the start stage of the maintenance / repair work. Steps S72 to S76 are the same as those in FIG. 7B, so the description is omitted.

[0093] In step S101, the label four-corner coordinate acquisition unit 93 of the difference detection device 11 stores the four-corner coordinates of the label acquired from the pre-operation image in the label coordinate DB 94.

[0094] FIG. 10B is a process flow diagram at the end stage of the maintenance / repair work. In step S102, when the operator holds the difference detection device 11 over the object, the image stream input unit 91 of the difference detection device 11 starts acquiring the image stream of the object from the camera.

[0095] In step S103, for each frame of the image stream notified by the image stream input unit 91, the guidance information generation and instruction unit 92 obtains the positional deviation from the four-corner coordinate positions of the label in the pre-operation image and the four-corner coordinate positions of the label in the frame image, generates guidance information for guiding the shooting position, and displays it. The method for detecting the positional deviation will be described later.

[0096] In step S104, the guidance information generation and instruction unit 92 determines whether the positional deviation of the label is the same or less than a predetermined threshold, that is, determines whether the shooting position sufficiently matches the shooting position of the pre-operation image (whether the shooting positions are the same). If they do not match (No in S104), the process returns to step S103. If they sufficiently match (Yes in S104), the operator is prompted to take a shot at that position. The method for determining the match of the positional deviation will be described later.

[0097] Since the processes in steps S77 to S79 are the same as the process of differential detection of the detection area described in FIG. 7C, the description is omitted. Note that in step S77, when it is determined in step S104 that the shooting positions match, the frame image of the image stream may be used as the post-operation image. In this case, instead of prompting the operator to take a shot in step S104, it is notified that the post-operation image has been acquired.

[0098] Next, the method for obtaining the positional deviation from the four-corner coordinate positions of the label in the pre-operation image and the four-corner coordinate positions of the label in the frame image information in step S103 will be described in detail.

[0099] First, among the labels 12 of the pre-operation image saved in step S101 (FIG. 10A), one label 12 to be used for shooting guidance is determined, and the area number p_id_ref of the label 12 and the four-corner coordinates p_corners_ref of the label are obtained. For example, one label 12 closest to the center of the pre-operation image, which is likely to be included in the post-operation image, may be determined, or the operator may be allowed to select it.

[0100] Then, the following process is performed for each frame.

[0101] First, extract label 12 from the frame image, recognize the region information (region number) described in each label 12, and obtain the four-corner coordinates of the label. Then, identify the label 12 that matches the region number p_id_ref from the labels 12 in the frame image, and obtain the four-corner coordinates p_corners_frame of that label.

[0102] These four-corner coordinates p_corners_ref and p_corners_frame are the four-corner coordinates of the same label 12, and there is a relationship between the region indicated by the four-corner coordinates p_corners_ref and the region indicated by p_corners_frame that is connected by the projective transformation matrix H. This projective transformation matrix H is known to have eight degrees of freedom and can be estimated from the correspondence between the four-corner coordinates p_corners_ref and p_corners_frame.

[0103] From this projective transformation matrix H and the internal parameter matrix K of the imaging unit of the device (which includes information such as the image center and focal length), it is known that the relationship between the position of the device when the image of the four-corner coordinates p_corners_frame is acquired and the position of the device when the image of the four-corner coordinates p_corners_ref is acquired can be obtained as the rotation matrix R and the translation vector t in the three-dimensional space. For example, Section 2.2 of "S. Benhimane and E. Malis “Homography-based 2D Visual Tracking and Servoing,” 2007." is detailed.

[0104] The above means that the viewpoint position can be generally estimated from the appearance of label 12. For example, when label 12 is viewed obliquely, the side closer in space appears longer and the side farther away appears shorter. Also, the distance of the viewpoint position with respect to label 12 can be estimated from the size of the area of the region.

[0105] Therefore, based on the rotation matrix R and the translation vector t, it is possible to estimate information on where the difference detection device 11 should be moved in space so that the area composed of the four-corner coordinates p_corners_frame looks like the area composed of the four-corner coordinates p_corners_ref.

[0106] Based on the above, guidance information on how to move the difference detection device 11 is generated to approach the shooting position of the pre-operation image from the current shooting position based on the estimated difference in the position of the difference detection device 11 in the three-dimensional space (rotation R and translation vector t). As described with reference to FIG. 8, the guidance information may present the rotation and translation information as arrows or may present it in converted words.

[0107] Ideally, the guidance information generation and instruction unit 92 generates and instructs guidance information for each frame of the image stream. However, whether this is possible depends on the processing capacity of the difference detection device 11. Therefore, in accordance with the performance of the difference detection device 11, the frames targeted by the guidance information generation and instruction unit 92 may be thinned out for processing.

[0108] The guidance information generation and instruction unit 92 determines that the shooting position of the post-operation image has approached the shooting position of the pre-operation image when the amount of change in the rotation R and the translation vector t obtained for each frame becomes sufficiently small (less than a predetermined threshold).

[0109] According to the difference detection device 11 of the above embodiment, at the time of shooting the post-operation image, guidance of the shooting position is performed so as to be the same as the shooting position of the pre-operation image, thereby suppressing the extraction of meaningless difference information caused by positional deviation and enabling the execution of an effective difference confirmation process before and after the operation.

[0110] The present invention is not limited to the above-described embodiments, and includes various modifications. The above embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.

Explanation of Reference Numerals

[0111] 11 Difference detection device 12 Label 51 Still image capturing unit 52 Label registration unit 53 Label area information recognition unit 54 Area information manual correction unit 55 Difference detection target image registration unit 56 Difference detection unit 57 Result display unit (difference information output unit) 58 Control unit 59 Label registration DB 510 Area information DB 511 Captured image DB 91 Image stream input unit 92 Guidance information generation / instruction unit 93 Label four-corner coordinate acquisition unit 94 Label coordinate DB

Claims

1. A label to which an object to be differentially detected is attached and which describes region information for specifying a detection region for detecting an image difference before and after work; A detection region information acquisition unit that recognizes the region information from a photographed image of the label; A difference detection unit that obtains an image difference of a detection region indicated by the region information in photographed images of the object before and after work; A difference information output unit that outputs the image difference; A guidance unit that performs imaging guidance for guiding an imaging position based on position information of the label extracted by the detection region information acquisition unit from a photographed image before work when acquiring a photographed image of the object after work, The guidance unit, obtains a region number of a detection region of the label in a frame of an image stream of an image after work, estimates a projective transformation matrix from coordinate information of four corners of the label corresponding to the region number in a photographed image before work and coordinate information of four corners of the label in the image frame, obtains guidance information from the projective transformation matrix An image difference detection system characterized by the above.

2. In the image difference detection system according to Claim 1, The detection region information acquisition unit extracts a photographed image of the label from a photographed image of the object based on a predetermined extraction condition An image difference detection system characterized by the above.

3. In the image difference detection system according to Claim 2, The detection region information acquisition unit recognizes the region information including a figure indicating a region shape of the detection region, a number indicating a region size of the detection region inside the figure, an arrow indicating an installation direction where the detection region is located, and a region number of the detection region in the photographed image of the label An image difference detection system characterized by the above.

4. In the image difference detection system according to any one of claims 1 to 3, the label is a planar-shaped object such as a magnet sheet or a sticker having a predetermined background color, the area information is handwritten, An image difference detection system characterized by this.

5. An image difference detection method for detecting the difference between photographed images of an object before and after work, registering an extraction condition for extracting from a photographed image a label on which area information for specifying a detection area for detecting an image difference before and after work is described, which is attached so as to specify an object for difference detection; extracting the label from the photographed image of the object and recognizing the area information described on the label; obtaining an image difference of a detection area indicated by the area information in the photographed images of the object before and after the work; displaying guidance information for guiding the photographing position to the same position as the photographing position of the photographed image before work when photographing the photographed image of the object after work, obtaining the area number of the detection area of the label in the frame of the image stream of the image after work, estimating a projective transformation matrix from the coordinate information of the four corners of the label corresponding to the area number in the photographed image before work and the coordinate information of the four corners of the label in the image frame, obtaining guidance information from the projective transformation matrix An image difference detection method characterized by this.

6. In the image difference detection method according to claim 5, extracting the image information of the label from the photographed image of the object using the background color of the label as an extraction condition, recognizing the area information consisting of a figure indicating the area shape of the detection area, a number indicating the area size of the detection area inside the figure, an arrow indicating the installation direction where the detection area is located, and the area number of the detection area from the image information of the label, Extract a detection area for differential detection from the captured images of the object before and after the operation based on the recognized area information. For each area number, obtain the image difference of the detection area of the object before and after the operation. An image difference detection method characterized by the above.

7. On a computer, Register an extraction condition for extracting from a captured image a label with area information that specifies a detection area for detecting an image difference before and after an operation and is attached to specify an object for differential detection. Extract the label from the captured image of the object and recognize the area information described in the label. Obtain the image difference of the detection area indicated by the area information in the captured images of the object before and after the operation. When capturing a captured image of the object after the operation, display guidance information for guiding the capture position to be the same position as the capture position of the captured image before the operation, including: Obtain the area number of the detection area of the label in the frame of the image stream of the image after the operation. Estimate a projective transformation matrix from the coordinate information of the four corners of the label corresponding to the area number in the captured image before the operation and the coordinate information of the four corners of the label in the image frame. Obtain guidance information from the projective transformation matrix. An image difference detection program for causing the above to be executed.

Images