Position identification device, position identification system, position identification method, and position identification program

Abstract

To provide a position identification device, a position identification system, a position identification method, and a position identification program that can identify a position of a device.SOLUTION: A position identification device includes: a first acquisition unit 311 for acquiring image information related to a first image obtained by capturing a device moving along a plane and device information related to an appearance of the device; a generation unit 312 for generating, from the image information, transformation information when projective transformation is applied to the first image so that a portion of the first image corresponding to a prescribed area of the plane is transformed into a reference portion of the second image; an extraction unit 313 for extracting, in the second image, a candidate area corresponding to a color of the device based on the transformation information and color information included in at least one of the first image and the second image; a second acquisition unit 314 for acquiring association information obtained by associating the reference portion with the prescribed area of the plane; and an identification unit 315 for identifying an image position of the device from within the candidate area based on the device information and the association information.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a positioning device, a positioning system, a positioning method, and a positioning program.

Background Art

[0002] The administrator of a building needs to periodically inspect the walls and floors of the building. This is because if the mortar or tiles that make up the walls etc. fall, it may lead to an accident. An inspector of the walls etc. checks for defects in the mortar or tiles etc. at each position while moving a device for inspection along the walls etc. (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such an inspection operation, it is desirable to identify the position where the inspection is being carried out, that is, the position of the inspection device on the wall etc. By identifying the position of the device, it becomes possible to easily grasp the position of tiles etc. where defects are found, for example.

[0005] The present invention has been made in view of the above problems. Therefore, an object of the present invention is to provide a positioning device, a positioning system, a positioning method, and a positioning program capable of identifying the position of a device.

Means for Solving the Problems

[0006] The above object of the present invention is achieved as follows.

[0007] (1) A first acquisition unit that acquires image information regarding a first image of a device moving along a plane and device information regarding the appearance of the device; a generation unit that generates conversion information for performing projective transformation on the first image from the image information so that a portion corresponding to a predetermined region of the plane in the first image is converted into a reference portion of a second image; an extraction unit that extracts a candidate region corresponding to the color of the device within the second image based on the conversion information and color information included in at least one of the first image and the second image; a second acquisition unit that acquires association information associating the reference portion with the predetermined region of the plane; and an identification unit that identifies the image position of the device from among the candidate regions based on the device information and the association information. A position identification device.

[0008] (2) The position identification device according to (1) above, wherein the extraction unit extracts the candidate region based on the conversion information and color information included in the second image.

[0009] (3) The position identification device according to (1) above, wherein the extraction unit extracts the candidate region based on the conversion information and color information included in the first image.

[0010] (4) The position identification device according to any one of (1) to (3) above, further comprising an output unit that outputs position information regarding the actual position of the device at the time of shooting the first image based on the identified image position.

[0011] (5) The first acquisition unit further acquires state information regarding the state of the plane near the device, and the output unit outputs the state information in association with the position information. The position identification device according to (4) above.

[0012] (6) The position identification device according to (5) above, wherein the state information includes information regarding the presence or absence of defects in the plane near the device.

[0013] (7) The position identification device according to (5) or (6) above, wherein the state information includes information regarding the hitting sound when the plane is hit by the device.

[0014] (8) The position identification device according to any one of (4) to (7) above, wherein the output unit outputs the position information in association with a diagram corresponding to the plane.

[0015] (9) The position identification device according to any one of (1) to (8) above, wherein the device information includes information regarding the color of the device and information regarding at least one of the size and shape of the device.

[0016] (10) The position identification device according to any one of (1) to (9) above, wherein the specifying unit specifies the image position of the device by calculating the image area of the device in the second image based on the device information and the related information.

[0017] (11) The position identification device according to any one of (1) to (10) above, wherein the related information includes information regarding the imaging magnification of the second image.

[0018] (12) The position identification device according to any one of (1) to (11) above, wherein the plane is a part of a structure.

[0019] (13) A position identification system including the position identification device according to any one of (1) to (12) above, an imaging device that photographs the device moving along the plane, and a collection device that collects state information regarding the state of the plane in the vicinity of the device.

[0020] Obtaining image information regarding a first image of a device moving along a plane and device information regarding the appearance of the device; generating conversion information for projective conversion of the first image from the image information such that a portion of the first image corresponding to a predetermined region of the plane is converted into a reference portion of a second image; extracting a candidate region corresponding to the color of the device within the second image based on the conversion information and color information included in at least one of the first image and the second image; obtaining association information associating the reference portion with the predetermined region of the plane; and specifying the image position of the device from among the candidate regions based on the device information and the association information.

[0021] A position specifying program for causing a computer to execute a process including: obtaining image information regarding a first image of a device moving along a plane and device information regarding the appearance of the device; generating conversion information for projective conversion of the first image from the image information such that a portion of the first image corresponding to a predetermined region of the plane is converted into a reference portion of a second image; extracting a candidate region corresponding to the color of the device within the second image based on the conversion information and color information included in at least one of the first image and the second image; obtaining association information associating the reference portion with the predetermined region of the plane; and specifying the image position of the device from among the candidate regions based on the device information and the association information. [Advantages of the Invention]

[0022] According to the position specifying device, position specifying system, position specifying method, and position specifying program of the present invention, a candidate region corresponding to the color of a device is extracted within a second image obtained by projective conversion of a first image, and the image position of the device is specified from among the candidate regions. Therefore, based on this image position, the actual position of the device at the time of shooting the first image can be specified. Thus, it becomes possible to specify the position of the device. [Brief Description of the Drawings]

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16A

Figure 16B

Figure 17A

Figure 17B

Figure 18

Embodiment for Carrying Out the Invention

[0024] Hereinafter, an embodiment of the inspection device, inspection system, position identification method, and position identification program of the present invention will be described with reference to the attached drawings. In the drawings, the same members are denoted by the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may be different from the actual ratios.

[0025] <Embodiment> FIG. 1 shows an example of the configuration of an inspection system 1 according to an embodiment of the present invention. This inspection system 1 inspects the state of a flat surface such as the wall surface or floor surface of a structure. In other words, the flat surface inspected by the inspection system 1 is a part of the structure. The inspection system 1 uses an inspection device (for example, the device 100 in FIG. 2 described later) that moves along the flat surface to inspect the state of the flat surface. Here, the inspection system 1 corresponds to an example of the position identification system of the present invention.

[0026] (Configuration of Device 100) Figure 2 shows an example of the configuration of device 100. Figure 2(a) is a front view of device 100, Figure 2(b) is a rear view of device 100, and Figure 2(c) is a side view of device 100. Device 100 is installed such that its back faces a wall surface, a floor surface, or the like. Device 100 moves along a plane while striking the plane of the wall surface, the floor surface, or the like. Device 100 has, for example, a rectangular planar shape. Device 100 preferably has a vivid red appearance, that is, a high-chroma red appearance. Wall surfaces, floor surfaces, and the like have a low chroma so as not to be irritating to the eyes of passersby and often have a hue closer to blue. Therefore, by having a high-chroma red appearance, the color of device 100 makes it easier to accurately identify the position of device 100.

[0027] Device 100 has, for example, a plurality of striking parts 101, a drive motor (not shown), an exhaust fan 111, and tires 115. The plurality of striking parts 101 are provided on the back of device 100 and strike the plane. Device 100 has, for example, three striking parts 101. Device 100 may have one striking part 101, or may have two, four, or more striking parts 101. Each of these plurality of striking parts 101 includes a support shaft 103 and a striking element 105. The exhaust fan 111 is provided from the front to the back of device 100. The tires 115 are provided on the back of device 100. For example, the tires 115 are provided at four locations on the back of device 100.

[0028] Figure 3 shows an example of the configuration of the striking part 101. The support shaft 103 extends in a predetermined direction. A striking element 105 is rotatably supported at one end of this support shaft 103. The other end of the support shaft 103 is rotatably supported on the back of device 100. Hereinafter, the extending direction of the support shaft 103 may be referred to as the Y direction, the direction intersecting the Y direction may be referred to as the X direction, and the direction intersecting both the X direction and the Y direction may be referred to as the Z direction.

[0029] The cross-sectional (XZ cross-section) shape of the impact roller 105 is, for example, a polygon such as a regular hexagon, and the longitudinal cross-sectional (XY cross-section) shape of the impact roller 105 is, for example, a substantially circular or substantially elliptical shape. That is, the impact roller 105 is a polyhedron such as a hexahedron. The impact roller 105 has, for example, six impact surfaces 105a and corner portions 105b between adjacent impact surfaces 105a. The impact surface 105a contacts the flat surface and strikes the flat surface.

[0030] In the impact section 101, while the support shaft 103 rotates, the impact roller 105 rolls and strikes the flat surface. The inspection system 1 inspects the state of the flat surface based on the impact sound of the impact roller 105 (hereinafter also simply referred to as the impact sound) at each position.

[0031] The exhaust fan 111 rotates by a drive motor and forms an air flow from the back to the front of the device 100. Thereby, the device 100 is adsorbed to the flat surface. The tire 115 rotates in contact with the flat surface. Thereby, the device 100 moves along the flat surface. In this way, by the device 100 having the exhaust fan 111 and the tire 115, the device 100 can be smoothly moved while being stably abutted against the flat surface.

[0032] FIG. 4 shows an example of a method for inspecting the flat surface 40 using the device 100. For example, the inspector 50 suspends and holds the device 100 from, for example, the rooftop of a building and abuts the device 100 against the flat surface 40. The inspector 50 gradually moves from bottom to top while, for example, swinging the device 100 left and right. Thereby, the impact sound of the impact roller 105 can be collected at each position of the flat surface 40. By inspecting the state of the flat surface 40 using the device 100 in this way, even in a place where the inspector 50 cannot reach, the flat surface 40 can be easily inspected. The device 100 may be configured to be drivable and controllable by a remote controller or the like. The device 100 may self-run while in contact with the flat surface 40. Alternatively, the device 100 may be configured by a drone or the like that flies along the flat surface 40. The device 100 may move along the flat surface 40 in a state separated from the flat surface 40.

[0033] The inspection system 1 photographs the device 100 moving along the plane 40 and collects the hitting sound of the transfer roller 105. The inspection system 1 uses this photographed image and the hitting sound to determine the presence or absence of defects at each position on the plane 40.

[0034] (Configuration of the inspection system 1) The inspection system 1 includes, for example, a sound collection device 10, an imaging device 20, and a position identification device 30 (Fig. 1). The sound collection device 10 and the imaging device 20 are each connected to the position identification device 30 via, for example, a network. The sound collection device 10 and the imaging device 20 may be connected by a wired connection or a wireless connection. The sound collection device 10 and the imaging device 20 may be integrated. The sound collection device 10 and the imaging device 20 may not be connected to the position identification device 30. The position identification device 30 may acquire the data of each of the sound collection device 10 and the imaging device 20 via, for example, a recording medium or the like. Here, the sound collection device 10 corresponds to a specific example of the collection device of the present invention.

[0035] The sound collection device 10 includes, for example, a microphone. The sound collection device 10 collects the hitting sound of the transfer roller 105 in association with the sound collection time. The sound collection device 10 includes, for example, a monaural microphone and does not have an active noise cancellation function. A sound absorption material may be attached to the sound collection part of the sound collection device 10. By attaching a sound absorption material to the sound collection part, it is possible to adjust to an arbitrary sound pressure level. Thereby, the occurrence of sound cracking or the like can be suppressed. The sound collection device 10 may have a recording function. The sound collection device 10 may be attached to the device 100. The sound collection device 10 generates sound pressure information regarding the hitting sound when the plane 40 is struck by the device 100. The sound collection device 10 may transmit the sound pressure information to the position identification device 30 via a network.

[0036] The imaging device 20 generates, for example, an image of visible light. The imaging device 20 is fixed at a predetermined position and photographs the device 100 that moves to each position along the plane 40 from this position. The imaging device 20 is preferably fixed, for example, at a position where the entire movement range of the device 100 is included within its imaging range. The imaging device 20 may be arranged in the front direction of the plane 40 or in an oblique direction of the plane 40. The imaging device 20 may photograph the device 100 from below or from above. The imaging device 20 photographs the device 100 over time, for example, in association with the photographing time. The imaging device 20 may generate a moving image or a still image. The imaging device 20 generates image information regarding an image of the device 100 that moves along the plane 40. The imaging device 20 may transmit the image information to the position specifying device 30 via a network.

[0037] The position specifying device 30 is a computer such as a PC (Personal Computer) and a tablet terminal, for example. This position specifying device 30 determines the presence or absence of a defect at each position on the plane 40 based on the sound pressure information generated by the sound collecting device 10 and the image information generated by the imaging device 20. The position specifying device 30 may control the sound collecting device 10 and the imaging device 20.

[0038] FIG. 5 is a block diagram showing an example of the schematic configuration of the position specifying device 30. The position specifying device 30 has, for example, a CPU (Central Processing Unit) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a storage 34, a communication interface 35, a display unit 36, and an operation reception unit 37. Each configuration is communicably connected to each other via a bus 38.

[0039] The CPU 31 performs control of the above-described respective configurations and various arithmetic processes according to programs recorded in the ROM 32 and the storage 34. Specific functions of the CPU 31 will be described later.

[0040] The ROM 32 stores various programs and various data.

[0041] The RAM 33 temporarily stores programs and data as a working area.

[0042] The storage 34 stores various programs including an operating system and various data. For example, device information regarding the appearance of the device 100 is stored in the storage 34. The device information includes, for example, information regarding the color of the device 100 and information regarding at least one of the size and shape of the device 100. Preferably, the device information includes both information regarding the size of the device 100 and information regarding the shape of the device 100. The information regarding the size of the device 100 includes, for example, information regarding the projected area of the device 100 on the plane 40 (for example, the projected area S real ) described later. The information regarding the shape of the device 100 includes, for example, information regarding the aspect ratio of the external shape of the device 100.

[0043] For example, plane information regarding the plane 40 is further stored in the storage 34. The plane information includes, for example, information regarding the size of the plane 40 and information regarding the shape of the plane 40. Drawings such as an elevation view representing the plane 40 may be stored in the storage 34. Alternatively, sound pressure information, image information, inspection results of the plane 40, etc. may be stored in the storage 34. An application for determining the presence or absence of a defect in the plane 40 from the sound pressure information and the image information may be installed in the storage 34. A learned discriminator for determining the presence or absence of a defect in the plane 40 from the sound pressure information and the image information may be stored in the storage 34. A learned model used as a discriminator and teacher data used for machine learning may be stored in the storage 34.

[0044] The communication interface 35 is an interface for communicating with other devices. As the communication interface 35, communication interfaces according to various wired or wireless standards are used. The communication interface 35 is used, for example, when receiving data from the sound collection device 10 and the imaging device 20.

[0045] The display unit 36 is composed of, for example, an LCD (liquid crystal display) or an organic EL display, etc., and displays various information. The display unit 36 may be composed of a viewer software or a printer, etc.

[0046] The operation reception unit 37 is composed of, for example, a pointing device such as a mouse or a keyboard, etc. The operation reception unit 37 may be composed of a touch sensor. The display unit 36 and the operation reception unit 37 may be composed of a touch panel. An inspector or the like may input device information and plane information, etc. to the position identification device 30 via the operation reception unit 37.

[0047] (Functions of the position identification device 30) FIG. 6 is a block diagram showing the functional configuration of the position identification device 30. The position identification device 30 functions as a first acquisition unit 311, a generation unit 312, an extraction unit 313, a second acquisition unit 314, a specification unit 315, and an output unit 316 by the CPU 31 reading a program stored in the storage 34 and executing processing.

[0048] The first acquisition unit 311 acquires image information and device information. As described above, the image information is information regarding an image obtained by photographing the device 100 moving along the plane 40, and the device information is information regarding the appearance of the device. The first acquisition unit 311 acquires image information from the imaging device 20 and acquires device information from the storage 34, for example. The first acquisition unit 311 may acquire image information and device information from the storage 34.

[0049] FIG. 7 shows an example of an image Im1 captured by the imaging device 20 at a predetermined time. The image Im1 is an image of the device 100 existing at a predetermined position on the plane 40. Here, the image Im1 corresponds to a specific example of the first image of the present invention. The first acquisition unit 311 acquires, for example, image information regarding this image Im1. This image information includes, for example, information regarding the imaging time of the image Im1. The first acquisition unit 311 may further acquire, for example, image information regarding images of the device 100 existing at other positions on the plane 40.

[0050] The first acquisition unit 311 further acquires, for example, state information regarding the state of the plane 40 near the device 100. The state information includes, for example, information regarding the presence or absence of defects at each position on the plane 40. Defects on the plane 40 are, for example, peeling, deterioration, swelling, and foreign matter contamination of mortar or tiles. The position identification device 30 determines the presence or absence of defects at each position on the plane 40 based on, for example, the sound pressure information generated by the sound collection device 10. The state information may be, for example, the sound pressure information.

[0051] FIG. 8 shows an example of the sound pressure information generated by the sound collection device 10. The sound pressure information includes, for example, information regarding the time change of the amplitude of the continuous knocking sound. The sound pressure information includes, for example, information regarding the sound collection time. The position identification device 30 estimates a single knocking sound region based on, for example, this sound pressure information, and determines the presence or absence of defects at each position on the plane 40 by performing frequency analysis of this single knocking sound region. The position identification device 30 may determine the presence or absence of defects at each position on the plane 40 using a machine learning model.

[0052] The generation unit 312 generates conversion information from the image information acquired by the first acquisition unit 311. The generation unit 312 generates conversion information, for example, by generating a projected image (the projected image Im2 in FIG. 10 described later) obtained by performing projective transformation on the image Im1.

[0053] FIG. 9 shows a predetermined region of the plane 40 shown in the image Im1, that is, a portion (portion Area1) in the image Im1 corresponding to the predetermined region of the plane 40. FIG. 10 shows a projected image Im2 obtained by performing a projective transformation on the image Im1. The generation unit 312 generates the projected image Im2 such that, for example, the portion Area1 in the image Im1 is converted into a reference portion Area2 of the projected image Im2. Here, the projected image Im2 corresponds to a specific example of the second image of the present invention.

[0054] The predetermined region of the plane 40 is a region photographed by the imaging device 20 and is a region for which a physical size can be determined. The predetermined region of the plane 40 is, for example, determined in advance. The predetermined region of the plane 40 is, for example, a region surrounded by four reference points Rp1 to Rp4 in the plane 40 and has a rectangular planar shape.

[0055] The reference points Rp1, Rp2, Rp3, and Rp4 are, for example, the positions of the upper right end, lower right end, upper left end, and lower left end shown in the image Im1, respectively. It is preferable that the reference points Rp1, Rp2, Rp3, and Rp4 can be easily identified in the image Im1 when the reference points Rp1, Rp2, Rp3, and Rp4 are photographed. The reference points Rp1, Rp2, Rp3, and Rp4 may have, for example, marks such as tapes on the plane 40. The reference points Rp1, Rp2, Rp3, and Rp4 may be intersections of shrinkage joints such as tiles. The number of reference points may be four or more.

[0056] The generation unit 312 performs a projective transformation on the image Im1 to generate a projected image Im2 such that the distances in the depth direction between each of the reference points Rp1, Rp2, Rp3, and Rp4 and the imaging position are equal. That is, the generation unit 312 generates an image (projected image Im2) that is pseudo-captured from the front direction of the plane 40. The generation unit 312 generates the projected image Im2 using, for example, a known method. For example, the generation unit 312 obtains a matrix for rearranging four points (reference points Rp1, Rp2, Rp3, and Rp4) in three dimensions onto a two-dimensional plane, and applies this matrix to each pixel of the image Im1. The conversion information includes, for example, information regarding this matrix. For example, in the projected image Im2, the centroid of the reference portion Area2 is at the center of the projected image Im2, and the width of the reference portion Area2 occupies 80% of the width of the projected image Im2.

[0057] The extraction unit 313 extracts a candidate region within the projected image Im2 based on the conversion information generated by the generation unit 312 and the color information included in at least one of the image Im1 and the projected image Im2. The candidate region is a region corresponding to the color of the device 100 within the projected image Im2, and is, for example, a group of pixels within the projected image Im2 having the same color as the device 100 or a color similar to the device 100. The extraction unit 313 extracts the candidate region using, for example, the color information included in the projected image Im2.

[0058] The color information is represented, for example, by the HSV color space (hue, saturation, value). This color information is represented, for example, by a hue of 0° to 360°, a saturation of 0% to 100%, and a value of 0% to 100%. For example, when the device 100 is imaged by an imaging device without passing through a color filter or the like under general outdoor ambient light, it has a color within the range of a hue of 300° to 60° and a saturation of 40% or more. At this time, the candidate region is, for example, a group of pixels within the projected image Im2 having a hue of 300° to 60° and a saturation of 40% or more.

[0059] FIG. 11 shows an example of a candidate region extracted within the projected image Im2. The white portion in FIG. 11 represents the candidate region. The extraction unit 313 extracts, for example, a plurality of candidate regions.

[0060] The second acquisition unit 314 acquires association information that associates the reference portion Area2 of the projected image Im2 with a predetermined region of the plane 40. The association information includes, for example, information regarding the imaging magnification r of the projected image Im2. For example, the actual distance between the reference point Rp1 and the reference point Rp3 is d real is. When the coordinates of the pixel corresponding to the reference point Rp1 in the projected image Im2 are (x A , y A ), and the coordinates of the pixel corresponding to the reference point Rp3 are (x B , y B ), the distance d image between the reference point Rp1 and the reference point Rp3 in the projected image Im2 can be expressed by the following formula (1).

[0061]

Equation

[0062] At this time, the imaging magnification r of the projected image Im2 can be expressed by the following formula (2). The second acquisition unit 314 may acquire the association information by calculating the imaging magnification r. The second acquisition unit 314 may acquire the association information from the storage 34.

[0063]

Equation

[0064] The specifying unit 315 specifies the image position of the device 100 from among the candidate regions extracted by the extraction unit 313. The image position of the device 100 is the position of the device 100 within the projected image Im2. When specifying this image position, the specifying unit 315 uses the device information acquired by the first acquisition unit 311 and the association information acquired by the second acquisition unit 314. For example, if the device 100 has a projected area S realWhen it has, the image area S of the device 100 shown in the projected image Im2 image can be expressed by the following formula (3) using the imaging magnification r.

[0065]

Equation

[0066] The specific part 315 selects one candidate region by comparing, for example, this image area S image with the area of each candidate region extracted by the extraction part 313, and specifies the image position of the device 100 within the selected candidate region. The area of the device 100 actually shown in the projected image Im2 is assumed to be affected by errors due to various factors in a general outdoor environment, or errors caused by the positional relationship between the device 100 and the imaging device 20 in the inspection work. For example, since the positional relationship between the device 100 and the imaging device 20 changes according to the situation, there may be an error caused by performing projection on images Im1 taken from various directions. Also, there may be an error caused by an area that cannot be extracted by color information due to shadows and reflections of ambient light. Therefore, it is preferable that the specific part 315 selects a candidate region in consideration of such errors. For example, the specific part 315 selects a candidate region having an area of 40% or more and 200% or less of the image area S image .

[0067] FIG. 12 shows an example of one selected candidate region. The white part in FIG. 12 represents one selected candidate region. When there are a plurality of candidate regions having an area of 40% or more and 200% or less of the image area S image , the specific part 315 selects, for example, one candidate region closer to 100% of the image area S image . The specific part 315 specifies, for example, the coordinates (x 100 , y 100 ) of the center of gravity of one selected candidate region as the image position of the device 100. The specific part 315 is the image area S imageWhen there are a plurality of candidate regions having an area of 40% or more and 200% or less, an error signal may be output to the output unit 316.

[0068] The output unit 316 outputs position information based on the image position of the device 100 specified by the specifying unit 315. The position information is information regarding the actual position of the device 100 at the time of shooting the image Im1. The output unit 316 specifies the actual position of the device 100 and outputs position information, for example, using the coordinates of the pixels corresponding to the reference points Rp1, Rp2, Rp3, Rp4 in the projected image Im2 and the pixel positions of the device 100, and the imaging magnification r. The output unit 316 outputs position information, for example, by causing the display unit 36 to display the direction and distance from the reference points Rp1, Rp2, Rp3, Rp4 to the position of the device 100. The output unit 316 may output position information by voice, or may output position information to a printer or the like. The output unit 316 may output state information regarding the state of the plane 40 in the vicinity of the device 100 in association with the position information. Thereby, it becomes easier for the inspector to grasp the presence or absence of defects at each position of the plane 40. The output unit 316 associates the position information and the state information, for example, using the shooting time and the sound collection time.

[0069] FIG. 13 shows an example of the position information and the state information displayed on the display unit 36. Here, the position of the device 100 at a predetermined time is represented by a black circle. FIG. 13 shows that no defect was confirmed on the plane 40 when the device 100 was at a position 1.5 m to the right and 5.0 m upward from the reference point Rp4.

[0070] The output unit 316 may output position information in association with the figure corresponding to the plane 40. The output unit 316 may map the image position of the device 100, for example, on an elevation view or a plan view representing the plane 40. Thereby, since the inspector can confirm the position of the device 100 on the drawing, it becomes easier to grasp the position of the device 100 intuitively.

[0071] (Outline of processing of the position specifying device 30) The process executed in the location identification device 30, that is, the location identification method of the device 100 executed by the location identification device 30, will be described in detail below.

[0072] FIG. 14 is a flowchart showing the procedure of the location identification process executed in the location identification device 30. The process of the location identification device 30 shown in the flowchart of FIG. 14 is stored as a program in the storage 34 of the location identification device 30 and is executed by the CPU 31 controlling each part.

[0073] (Step S101) The location identification device 30 first acquires image information, device information, plane information, and state information. The location identification device 30 acquires, for example, image information from the imaging device 20. The location identification device 30 acquires, for example, device information and plane information from the storage 34. The location identification device 30 acquires, for example, state information from the sound collection device 10.

[0074] (Step S102) The location identification device 30 generates conversion information using the image information and plane information acquired in the process of step S101. The location identification device 30, for example, performs projective transformation on the image Im1 captured by the imaging device 20 to generate a projected image Im2. Thereby, conversion information is generated.

[0075] (Step S103) The location identification device 30 extracts candidate regions within the projected image Im2 using the conversion information generated in the process of step S102 and the color information included in the projected image Im2. The location identification device 30 extracts, for example, an image group having color information in a predetermined HSV space within the projected image Im2.

[0076] (Step S104) The location identification device 30 acquires related information. The location identification device 30 acquires related information, for example, by calculating the imaging magnification r of the projected image Im2.

[0077] (Step S105) The position specifying device 30 specifies the image position of the device 100 from among the candidate regions extracted in the process of step S103. At this time, the position specifying device 30 uses the device information and related information acquired in the processes of steps S101 and S104 respectively. For example, the position specifying device 30 selects one candidate region from among a plurality of candidate regions, and specifies the coordinates of the pixel at the center of gravity of the selected one candidate region as the image position.

[0078] (Step S106) The position specifying device 30 outputs the position information regarding the actual position of the device 100 in association with the state information, and ends the process. The position specifying device 30 specifies the actual position of the device 100 based on the image position of the device 100 specified in the process of step S105.

[0079] (Function and effect of the position specifying device 30 and the inspection system 1) In the position specifying device 30 and the inspection system 1 of the present embodiment, in the projection image Im2 obtained by performing projective transformation on the image Im1, a candidate region corresponding to the color of the device 100 is extracted, and the image position of the device 100 is specified from among this candidate region. Therefore, based on this image position, the actual position of the device 100 at the time of shooting the image Im1 can be specified. Hereinafter, this function and effect will be described in detail.

[0080] When inspecting a flat surface such as a wall surface and a floor surface using an inspection device (for example, the device 100), the inspector may inspect the presence or absence of defects on the flat surface while visually confirming the position of the device. However, on a wall surface, a floor surface, etc., the same pattern often continues, and there are few things that can serve as landmarks. For this reason, it becomes difficult for the inspector to accurately record the position of the device.

[0081] Therefore, it is desirable to identify the position of the device from the captured image without relying on visual inspection. Examples of such position identification methods include the AR (Augmented Reality) marker method and the pattern matching method. However, the AR marker method may significantly reduce the position identification accuracy due to the lighting conditions during inspection, and the pattern matching method may also significantly reduce the position identification accuracy due to the imaging position. The device often inspects a wall surface or the like under external light, and the light during inspection is likely to be unstable. In addition, the location where a wide wall surface or the like can be photographed, that is, the installation location of the imaging device is limited. Therefore, it is difficult to use the AR marker method and the pattern matching method.

[0082] In contrast, the position identification device 30 can identify the actual position of the device 100 without using the AR marker method and the pattern matching method. Therefore, even on the outdoor plane 40 where the light is unstable, it is possible to identify the actual position of the device 100 with high accuracy. In addition, regardless of the imaging position of the image Im1, it is possible to identify the actual position of the device 100.

[0083] Furthermore, the position identification device 30 identifies the image position of the device 100 from the projected image, that is, the candidate region extracted from the projected image Im2. Therefore, the image position of the device 100 can be identified with higher accuracy. This will be described below.

[0084] FIG. 15 shows a region corresponding to the color of the device 100 in the image before projective transformation, that is, the image Im1. Thus, there may be a plurality of regions corresponding to the color of the device 100 in the image Im1. Therefore, it is difficult to identify the image position of the device 100 only by color information.

[0085] If an attempt is made to identify the image position of the device 100 using the size or shape of the device 100 from among the regions extracted from this image Im1, the accuracy may decrease depending on the imaging position of the imaging device 20.

[0086] Figures 16A and 17A show the positional relationship between the device 100 and the imaging device 20, and Figures 16B and 17B show the images Im101 and Im102 taken at each position. The image Im101 is an image taken from the position P1 in the front direction with respect to the device 100. The image Im102 is an image taken from the position P2 shifted from the front direction with respect to the device 100. The size and shape of the device 100 shown in the image Im101 are different from the size and shape of the device 100 shown in the image Im102. Therefore, it is difficult to estimate the image area etc. of the device 100 in the images taken from each of the plurality of directions.

[0087] In contrast, in the present embodiment, the position specifying device 30 specifies the image position of the device 100 from the candidate region extracted within the projected image Im2. In other words, the position specifying device 30 can always specify the image position of the device 100 using an image pseudo-taken from the front direction with respect to the device 100. Therefore, regardless of the imaging position, it is possible to specify the image position of the device 100, and thus the actual position of the device 100, with high accuracy.

[0088] Also, by using the position specifying device 30, it is possible to secure a wide imaging range even when the imaging position is limited. For example, when the space in the front direction of the plane 40 is narrow, if an attempt is made to image the device 100 from the front direction, a sufficient imaging range cannot be secured, and there is a risk that the inspection will take a long time. In contrast, by imaging the plane 40 from an oblique direction, a wide imaging range is secured, making it easier to proceed with the inspection work. In the position specifying device 30, the position of the device 100 can also be specified from the image Im1 in which the device 100 is imaged from such an oblique direction.

[0089] As described above, in the position identification device 30 and the inspection system 1 of the present embodiment, in the projected image Im2 obtained by performing projective transformation on the image Im1, a candidate region corresponding to the color of the device 100 is extracted, and the image position of the device 100 is identified from among these candidate regions. Therefore, based on this image position, the actual position of the device 100 at the time of capturing the image Im1 can be identified.

[0090] Hereinafter, a modification of the inspection system 1 described in the above embodiment will be described. In the following description, in order to avoid duplication of explanation, detailed explanation of configurations similar to those described in the above embodiment will be omitted.

[0091] <Modification> FIG. 18 is a flowchart showing the procedure of the position identification process executed in the position identification device 30 according to the modification. This position identification device 30 extracts a pre-transformation candidate region corresponding to the color of the device 100 from within the image Im1 (step S202 described later), and then extracts a candidate region using the transformation information (step S203 described later). Except for this point, the position identification device 30 according to the modification has the same configuration as the position identification device 30 of the above embodiment and exhibits the same operational effects.

[0092] (Step S201) The position identification device 30 first acquires image information, device information, plane information, and state information in the same manner as in step S101.

[0093] (Step S202) The position identification device 30 extracts a pre-transformation candidate region using the color information included in the image Im1. The pre-transformation candidate region is, for example, a group of images within the image Im1 that correspond to the color of the device 100.

[0094] (Step S203) The position identification device 30 generates transformation information using the image information and plane information acquired in the process of step S201. After that, a candidate region is extracted using this transformation information and the pre-transformation candidate region extracted in the process of step S202.

[0095] (Step S204) The position identifying device 30 acquires the related information in the same manner as in step S104.

[0096] (Step S205) The position identifying apparatus 30 identifies the image position of the device 100 from among the candidate areas extracted in the process of step S203.

[0097] (Step S206) The position identification apparatus 30 outputs the position information regarding the actual position of the device 100 in association with the state information in the same manner as in step S106, and ends the process.

[0098] In the position identifying device 30 according to the modified example, similarly to the embodiment described above, a candidate area is extracted and the image position of the device 100 is identified from within the candidate area. Therefore, based on the image position, the actual position of the device 100 at the time when the image Im1 was captured can be identified.

[0099] <Other Modifications> In the above embodiment, the localization device 30 measures the image area S of the device 100. image However, the position identification device 30 may use other methods to identify the image position of the device 100 from within the candidate area. For example, the position identification device 30 may use other methods such as pattern matching to identify the image position of the device 100 from within the candidate area. Furthermore, the position identification device 30 may identify the image position of the device 100 by appropriately combining edge detection, a method of determining using the aspect ratio of the circumscribing rectangle of the device 100, and the like.

[0100] In the above embodiment, an example in which the position specifying device 30 uses color information represented by the HSV color space has been described. However, the position specifying device 30 may use other color information. For example, the position specifying device 30 may use color information represented by an RGB color space, a Lab color space, an L*a*b* color space, or the like.

[0101] In the above embodiment, an example in which the inspection system 1 inspects the state of the plane 40 using the sound pressure information collected by the sound collection device 10 has been described. However, the inspection system 1 may inspect the state of the plane 40 using other information. For example, the inspection system 1 may inspect the state of the plane 40 using a visible light image, an infrared image, or the like in which the plane 40 is photographed.

[0102] In the above embodiment, an example in which the device 100 has a vivid red appearance has been described. However, the color of the appearance of the device 100 may be other colors. For example, considering that blue and yellow are relatively common as the colors of the tile wall surface to be inspected, green, which has a hue far from blue and yellow, may be adopted as the color of the appearance of the device 100. Further, the color of the appearance of the device 100 may be appropriately changed by applying paint or the like to the device 100, or applying a case, a cover, or the like. Further, considering that the reflectance of the tile is high, a color with an extremely low reflectance may be adopted as the color of the appearance of the device 100. In this case, for example, brightness may be used as the color information, and candidate regions may be extracted based on the difference in brightness.

[0103] Further, the plane 40 on which the device 100 moves may be a plane perpendicular to the ground such as a wall surface, or a plane parallel to the ground such as a floor surface. Alternatively, the plane 40 may be a plane provided in an oblique direction with respect to the ground.

[0104] Further, the position specifying device 30 may be installed at a location away from the device 100, or may be mounted on the device 100 together with the sound collection device 10 or the like. Alternatively, the position specifying device 30 may be provided on the cloud.

[0105] Further, the position identification device 30 described in the above embodiment may be configured by a plurality of devices. For example, a part of the functions of the position identification device 30 described in the above embodiment may be configured on-premises, and other functions may be provided on the cloud. A device that undertakes a part of the functions of the position identification device 30 may be mounted on the device 100, the sound collection device 10, or the imaging device 20.

[0106] Also, the processing units in the flowcharts in the above embodiment are divided according to the main processing contents in order to facilitate the understanding of each process. The present invention is not limited by the way of classifying the processing steps. Each process can be further divided into more processing steps. Also, one processing step may execute more processes. Further, the position identification device 30 may perform the processing in an order different from the order described in the above embodiment and the modified example.

[0107] The means and methods for performing various processes in the system according to the above-described embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The above program may be provided, for example, by a computer-readable recording medium such as a flexible disk and a CD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in a storage unit such as a hard disk. Also, the above program may be provided as a single application software, or may be incorporated into the software of the device as one function of the system.

[0108] As described above, the inspection device, inspection system, inspection method, and inspection program of the present invention have been described in the embodiment and the modified example. However, it goes without saying that those skilled in the art can appropriately add, modify, and omit within the scope of the technical idea of the present invention.

Explanation of Reference Numerals

[0109] 1 inspection system, 10 sound collection device, 20 imaging device, 30 inspection device, 31 CPU, 311 first acquisition unit, 312 generation unit, 313 extraction unit, 314 second acquisition unit, 315 identification unit, 316 output unit, 32 ROM, 33 RAM, 34 storage, 35 communication interface, 36 display unit, 37 operation reception unit, 40 plane, 100 device.

Claims

1. A first acquisition unit that acquires image information regarding a first image of a device moving along a plane and device information regarding the appearance of the device; A generation unit that generates conversion information for performing projective conversion on the first image from the image information so that a portion corresponding to a predetermined region of the plane in the first image is converted into a reference portion of a second image; An extraction unit that extracts a candidate region corresponding to the color of the device in the second image based on the conversion information and color information included in at least one of the first image and the second image; A second acquisition unit that acquires association information associating the reference portion with the predetermined region of the plane; An identification unit that identifies the image position of the device from among the candidate regions based on the device information and the association information A position identification device comprising:

2. The position identification device according to claim 1, wherein the extraction unit extracts the candidate region based on the conversion information and color information included in the second image.

3. The position identification device according to claim 1, wherein the extraction unit extracts the candidate region based on the conversion information and color information included in the first image.

4. The position identification device according to claim 1, further comprising an output unit that outputs position information regarding the actual position of the device at the time of shooting the first image based on the identified image position.

5. The first acquisition unit further acquires state information regarding the state of the plane in the vicinity of the device, The position identification device according to claim 4, wherein the output unit outputs the state information in association with the position information.

6. The position identification device according to claim 5, wherein the state information includes information regarding the presence or absence of a defect in the plane in the vicinity of the device.

7. The position identification device according to claim 5, wherein the state information includes information regarding a hitting sound when the plane is hit by the device.

8. The position identification device according to claim 4, wherein the output unit outputs the position information in association with a diagram corresponding to the plane.

9. The position identification device according to claim 1, wherein the device information includes information regarding the color of the device and information regarding at least one of the size and shape of the device.

10. The position specifying device according to claim 9, wherein the specific part specifies the image position of the device by calculating the image area of the device in the second image based on the device information and the related information.

11. The position specifying device according to claim 1, wherein the related information includes information regarding the imaging magnification of the second image.

12. The position specifying device according to claim 1, wherein the plane is a part of a structure.

13. A position specifying device according to any one of claims 1 to 12, an imaging device that photographs the device moving along the plane, and a collecting device that collects state information regarding the state of the plane in the vicinity of the device comprising a position specifying system.

14. obtaining image information regarding a first image that photographs a device moving along a plane and device information regarding the appearance of the device; generating conversion information for projective conversion of the first image from the image information such that a portion of the first image corresponding to a predetermined region of the plane is converted into a reference portion of a second image; extracting a candidate region corresponding to the color of the device in the second image based on the conversion information and color information included in at least one of the first image and the second image; obtaining related information that associates the reference portion with the predetermined region of the plane; and specifying the image position of the device from among the candidate regions based on the device information and the related information including a position specifying method.

15. obtaining image information regarding a first image that photographs a device moving along a plane and device information regarding the appearance of the device; generating conversion information for projective conversion of the first image from the image information such that a portion of the first image corresponding to a predetermined region of the plane is converted into a reference portion of a second image; extracting a candidate region corresponding to the color of the device in the second image based on the conversion information and color information included in at least one of the first image and the second image; obtaining related information that associates the reference portion with the predetermined region of the plane; and specifying the image position of the device from among the candidate regions based on the device information and the related information including a position specifying program for causing a computer to execute the process.

Images