Information processing method, information processing system, and program

Abstract

To provide an information processing method, an information processing system, and a program for extracting a specific inspection target from an image.SOLUTION: An information processing method causes a computer to execute information processing to extract a straight line corresponding to an elongated inspection target from an image captured by an unmanned mobile body. The computer executes: a process to generate a filtered image in which a line of a specific width is emphasized, from a captured image, by performing spatial filtering based on a first filter having selectability to the line of the specific width corresponding to an inspection target; and a process to extract a straight line corresponding to the inspection target from multiple straight lines in the filtered image, based on Hough transformation on the generated filtered image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing system, and a program. [Background technology]

[0002] In recent years, mobile objects such as drones, unmanned aerial vehicles (UAVs), and unmanned ground vehicles (UGVs) (hereinafter collectively referred to as "mobile objects") have begun to be used in industry. For example, Patent Document 1 discloses a system for photographing and inspecting power lines using an aerial object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-196355 Summary of the Invention [Problem to be solved by the invention]

[0004] One possible method for identifying long inspection targets, such as overhead power lines, in order to track them is to perform edge detection on captured images acquired by an unmanned vehicle to extract the edges of the inspection target and identify its location. However, because the captured images often contain various buildings, roads, and other linear structures in the background, performing edge detection on the captured images can result in the extraction of numerous edges that become noise. This can make it difficult to identify the edges of the target inspection target. In particular, if the edges of linear objects (especially elongated objects) in the background that are not the inspection target are extracted, the unmanned vehicle may end up tracking such objects, losing sight of the inspection target and making it difficult to perform a proper inspection.

[0005] In addition to the edge detection method described above, a method of segmenting and extracting inspection objects in a captured image using deep learning is also possible. However, performing such deep learning segmentation requires a high-spec processing device, which may make it difficult to use this segmentation method in a general-purpose manner. Furthermore, when non-inspection objects of the same type as the inspection object are also captured in the image (for example, when non-inspection power lines are captured in the background of the captured image in addition to the inspection power lines), it is difficult to distinguish between inspection objects and non-inspection objects using segmentation methods.

[0006] The present disclosure has been made in consideration of the above-described background, and one of the objectives of an exemplary embodiment of the present disclosure is to provide an information processing method, an information processing system, and a program capable of extracting a specific inspection object from an image. [Means for solving the problem]

[0007] An information processing method according to one aspect of the present disclosure is an information processing method in which a computer executes information processing to extract a straight line corresponding to a long-sized inspection object from an image captured by an unmanned moving body, generating a filtered image from the captured image in which lines of a specific width corresponding to the object to be inspected are emphasized by spatial filtering based on a first filter having selectivity for the lines of the specific width; The computer executes a process of extracting a straight line corresponding to the object to be inspected from a plurality of straight lines in the filtered image based on line vote count information obtained by performing a Hough transform process on the generated filtered image.

[0008] Because the information processing method has the above-mentioned characteristics, even if the captured image contains linear non-inspection objects such as buildings, roads, or other long objects in addition to the inspection object, it is possible to extract a straight line corresponding to the desired inspection object from the captured image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of one embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of a system configuration of an information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a block diagram showing the hardware configuration of the terminal 1 shown in FIG. [Figure 4] FIG. 4 is a block diagram showing the hardware configuration of the server 2 shown in FIG. [Figure 5] FIG. 5 is a block diagram showing the hardware configuration of the unmanned aerial vehicle 4 shown in FIG. [Figure 6] FIG. 6 is a block diagram illustrating the functions of the terminal 1 and the server 2 shown in FIG. [Figure 7] FIG. 7 is an example of an image acquired from the imaging unit of the unmanned aerial vehicle 4. [Figure 8] FIG. 8 is an example of a filtered image generated from the image of FIG. [Figure 9] FIG. 9 is a diagram for explaining a convolution operation process using a general edge enhancement filter. [Figure 10] FIG. 10 is a schematic diagram showing an example of the specific width emphasis filter (first filter). [Figure 11] FIG. 11 is a diagram for explaining an example of the convolution operation process using the specific width emphasis filter shown in FIG. [Figure 12] FIG. 12 is a schematic diagram showing a modified example of the specific width emphasis filter. [Figure 13] FIG. 13 is a schematic diagram showing an example of the different diameter line extraction filter (second filter). [Figure 14] FIG. 14 is a schematic diagram illustrating a one-side edge filter (third filter). [Figure 15] FIG. 15 is a diagram for explaining the luminance distribution of the inspection object in the test image. [Figure 16]FIG. 16 is a schematic diagram showing an example of the anti-reflection filter (fourth filter). [Figure 17] FIG. 17 is a diagram showing an example of a captured image. [Figure 18] FIG. 18 is a diagram showing a simulated filtered image converted from FIG. [Figure 19] FIG. 19 is a diagram showing a schematic representation of a filtered image converted from FIG. [Figure 20] FIG. 20 is a diagram showing a schematic representation of a filtered image converted from FIG. [Figure 21] FIG. 21 is a diagram showing a schematic example of an image in which the position of the tracking target is visualized. [Figure 22] FIG. 22 is a flowchart showing an example of processing by the information processing system according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The information processing method, information processing system, and program of the present disclosure have the following configuration. [Item 1] An information processing method in which a computer executes information processing to extract a straight line corresponding to a long-sized inspection object from an image captured by an unmanned mobile body, generating a filtered image from the captured image in which lines of a specific width corresponding to the object to be inspected are emphasized by spatial filtering based on a first filter having selectivity for the lines of the specific width; and extracting a line corresponding to the object to be inspected from a plurality of lines in the filtered image based on line vote count information obtained by performing a Hough transform process on the generated filtered image. [Item 2] Item 1. The information processing method according to item 1, wherein in the process of generating the filtered image, the spatial filtering process is performed based on the first filter and a second filter having selectivity for lines of different diameters having a width different from the specific width, and the filtered image in which lines of the specific width are emphasized is generated. [Item 3] Item 1. The information processing method according to item 1, wherein in the process of generating the filtered image, the spatial filtering process is performed based on the first filter and a third filter that responds only to one of the two edges of the line of the specific width, thereby generating the filtered image in which the line of the specific width is emphasized. [Item 4] the third filter includes a filter that reacts only to one of the two edges of the line of the specific width, and a filter that reacts only to the other of the two edges; Item 3. The information processing method according to item 3, wherein the filtered image is generated by the spatial filtering process based on the first filter, the third filter that responds only to one of the two edges, and the third filter that responds only to the other of the two edges. [Item 5] 2. The information processing method according to item 1, wherein in the process of generating the filtered image, the filtered image in which the lines of the specific width are emphasized is generated by the spatial filtering process based on the first filter and a fourth filter having filter coefficients weighted based on the luminance distribution of the object to be inspected in the test image. [Item 6] before performing the spatial filtering process, further comprising a process of generating a composite filter by combining at least one other filter and the first filter; The other filter is a second filter selective for different diameter wires having widths different from the specified width; a third filter that responds to only one of the two edges of the line of the specific width; and a fourth filter having a filter coefficient weighted based on the luminance distribution of the inspection object in the test image; 6. The information processing method according to any one of items 1 to 5, wherein in the process of generating the filtered image, the spatial filtering process is performed by a convolution operation using the composite filter, and the filtered image in which the lines of the specific width are emphasized is generated. [Item 7] 7. The information processing method according to item 6, wherein the composite filter is generated by combining the first filter, the second filter, the third filter, and the fourth filter. [Item 8] The Hough transform process is a weighted Hough transform, 6. The information processing method according to any one of items 1 to 5, wherein a straight line corresponding to the object to be inspected is extracted from a plurality of straight lines in the filtered image based on straight line voting number information obtained by the weighted Hough transform and weighted by intensity values ​​in the filtered image. [Item 9] An information processing system that extracts a straight line corresponding to a long-sized inspection object from an image captured by an unmanned mobile body, a filtering processing unit that generates a filtered image in which lines having a specific width corresponding to the object to be inspected are emphasized from the captured image by spatial filtering processing based on a first filter that has selectivity for the lines having the specific width; and a specific line extraction unit that extracts a line corresponding to the object to be inspected from a plurality of lines in the filtered image based on line vote count information obtained by Hough transform processing of the generated filtered image. [Item 10] A program that causes a computer to execute information processing to extract a straight line corresponding to a long-sized inspection object from an image captured by an unmanned moving body, generating a filtered image from the captured image in which lines of a specific width corresponding to the object to be inspected are emphasized by spatial filtering based on a first filter having selectivity for the lines of the specific width; and extracting a line corresponding to the object to be inspected from a plurality of lines in the filtered image based on line vote count information obtained by Hough transform processing of the generated filtered image.

[0011] <Details of implementation form> An information processing system according to an embodiment of the present disclosure will be described below with reference to the drawings. In the accompanying drawings, identical or similar elements are designated by identical or similar reference symbols and names, and duplicate descriptions of identical or similar elements may be omitted in the description of the embodiment. Note that the contents shown in the drawings are merely examples for explaining the present embodiment and are merely schematic examples for ease of explanation of the present embodiment. The contents of the drawings may be modified or changed within the scope of no technical problem.

[0012] <Outline of this embodiment> The information processing system according to this embodiment is a system for extracting an inspection target from an image captured by an unmanned mobile vehicle (hereinafter referred to as a captured image) and automatically tracking the inspection target using the unmanned mobile vehicle. The inspection target is a long object (e.g., a linear, tubular, columnar, or long plate-shaped object having a longitudinal direction), but is not limited thereto. Examples of the inspection target include overhead lines (power lines), pipes, dividing lines, and road markings (white lines, center lines, etc.). The information processing system according to this embodiment is preferably applied to the inspection of overhead lines or pipes, and more preferably to the inspection of overhead lines. Examples of unmanned mobile vehicles include, but are not limited to, drones, unmanned aerial vehicles, and unmanned ground vehicles. In the case of power line inspection, which will be described later, it is preferable to use an unmanned aerial vehicle such as a drone or unmanned aerial vehicle. In this embodiment, the information processing system will be described in detail using an example of inspecting power lines using an unmanned aerial vehicle.

[0013] As shown in Fig. 1, the information processing system exemplified below extracts straight lines corresponding to a plurality of power lines (particularly multi-conductor power lines, for example, two, three, four, six, etc. power lines separated by span spacers) from an image captured along the plurality of power lines extending side by side, for example, attached to a support (for example, a steel tower). As an example, images of the plurality of power lines may be captured by remotely operating a camera mounted on an unmanned aerial vehicle 4 as shown in Fig. 1, which flies autonomously or remotely, based on instructions from a terminal 1 owned by a user.

[0014] More specifically, the information processing system generates a filtered image by spatial filtering from an image captured by an aircraft. At this time, the spatial filtering is performed based on a specific width enhancement filter that is different from conventionally known edge enhancement filters, thereby generating a filtered image in which lines having a specific width (hereinafter, sometimes referred to as "specific width lines") that correspond to the power line under inspection are emphasized.

[0015] After generating the filtered image, a Hough transform is performed on the filtered image, and specific width lines corresponding to the power line to be inspected are extracted from multiple lines in the filtered image based on the line vote number information obtained by the Hough transform. Because the specific width lines are emphasized in the filtered image, which is the input image for the Hough transform, the lines corresponding to the power line to be inspected can be easily extracted from the line vote number information. A general Hough transform technique can be used for this Hough transform, and it is particularly preferable to perform a weighted Hough transform (WHT) to obtain the line vote number information. In the weighted Hough transform, the line vote number information is weighted based on the intensity value (corresponding to the luminance value (pixel value) after spatial filtering) in the filtered image. In the filtered image, which is the input image for the weighted Hough transform, the specific width lines are displayed with a higher intensity value than other lines corresponding to non-inspection objects such as the background. Therefore, in the line vote number information obtained by the weighted Hough transform, the specific width lines are assigned a higher number of votes than other lines. Therefore, when a weighted Hough transform is performed, lines of a specific width can be extracted more easily than with a normal Hough transform, based on line vote number information weighted based on the intensity value in the filtered image.

[0016] As described above, by performing spatial filtering processing based on a specific width emphasis filter and weighted Hough transform processing, even if a linear non-inspection object such as a building, road, or other long object that is not the inspection object is reflected in the captured image, it is possible to extract a straight line corresponding to the target inspection object from the captured image. Furthermore, the above information processing system makes it possible to extract a straight line corresponding to the inspection object using simpler calculation processing than deep learning, and can use a more inexpensive and general-purpose calculation processing device than the calculation processing device required for segmentation methods using deep learning.

[0017] <System configuration> As shown in FIG. 2, the information processing system of this embodiment includes a terminal 1, a server 2, and an unmanned aerial vehicle 4. The terminal 1, the server 2, and the unmanned aerial vehicle 4 may be connected to each other via a network NW so that they can communicate with each other. Note that the illustrated configuration is an example and is not limited to this. For example, the unmanned aerial vehicle 4 may not be connected to the network NW. In this case, the unmanned aerial vehicle 4 may be operated by a transmitter (a so-called radio) operated by a user, or image data acquired by a camera of the unmanned aerial vehicle 4 may be stored in an auxiliary storage device (e.g., a memory card such as an SD card and / or a USB memory) connected to the unmanned aerial vehicle 4, and the image data may be read out and stored later by the user from the auxiliary storage device to the terminal 1 and / or the server 2. Alternatively, the unmanned aerial vehicle 4 may be connected to the network NW solely for the purpose of either operation or storage of image data.

[0018] <Hardware configuration of terminal 1> 3 is a diagram showing the hardware configuration of the terminal 1 in this embodiment. Note that the configuration shown in the figure is an example, and the terminal 1 may have other configurations.

[0019] The terminal 1 includes at least a processor 10, a memory 11, a storage 12, a transceiver 13, an input / output unit 14, etc., which are electrically connected to each other via a bus 15. The terminal 1 may be a general-purpose computer such as a workstation or a personal computer.

[0020] The processor 10 is a computing device that controls the overall operation of the terminal 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and deployed in the memory 11 to perform various information processing.

[0021] The memory 11 includes a main memory configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary memory configured with a non-volatile storage device such as a flash memory and an HDD (Hard Disc Drive). The memory 11 is used as a work area for the processor 10, and also stores a BIOS (Basic Input / Output System) that is executed when the terminal 1 is started up, various setting information, etc.

[0022] The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12. Furthermore, a storage unit 130 (described later) may be provided in part of the storage area.

[0023] The transceiver 13 is a communication interface that enables the terminal 1 to communicate with an external device (not shown) and the unmanned aerial vehicle 4 via a communication network. The transceiver 13 may further include a short-range communication interface such as Bluetooth (registered trademark) and BLE (Bluetooth Low Energy) and / or a USB (Universal Serial Bus) terminal.

[0024] The input / output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

[0025] A bus 15 is commonly connected to the above elements and transmits, for example, address signals, data signals and various control signals.

[0026] <Server 2> 4 also includes a processor 20, a memory 21, a storage 22, a transceiver 23, an input / output unit 24, etc., which are electrically connected to one another via a bus 25. The server 2 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. The functions of each element of the server 2 can be configured in the same way as the terminal 1 described above, and detailed description of each element of the server 2 will be omitted.

[0027] <Unmanned Aerial Vehicle 4> 5 is a block diagram showing the hardware configuration of the unmanned air vehicle 4. The flight controller 41 may have one or more processors, such as a programmable processor (for example, a central processing unit (CPU)).

[0028] The flight controller 41 may also have or have access to memory 411. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412, such as inertial sensors (acceleration sensors, gyro sensors), GPS sensors, and proximity sensors (e.g., lidar).

[0029] The memory 411 may include, for example, a separable medium such as an SD card and random access memory (RAM) or an external storage device. Data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image and video data captured by a camera or the like may be recorded in an internal memory or an external memory, but this is not limited thereto. The captured still image and video data may be recorded in at least one of the terminal 1 or the server 2 from the camera / sensors 42 or the internal memory via the network NW. The camera 42 is installed on the unmanned aerial vehicle 4 via a gimbal 43.

[0030] The flight controller 41 includes a control module (not shown) configured to control the state of the unmanned air vehicle 4. For example, the control module may have six degrees of freedom (translational motion x, y, and z, and rotational motion θ x , θ y and θ z The control module controls the propulsion mechanism (motor 45, etc.) of the unmanned aerial vehicle 4 via an ESC 44 (Electric Speed ​​Controller) to adjust the spatial arrangement, speed, and / or acceleration of the unmanned aerial vehicle 4. The motor 45, powered by a battery 48, rotates a propeller 46, generating lift for the unmanned aerial vehicle 4. The control module can control one or more of the states of the onboard components and sensors.

[0031] The flight controller 41 can communicate with a transceiver 47 configured to transmit and / or receive data from one or more external devices (e.g., a transceiver 49, a terminal 1, a display device, or other remote control). The transceiver 49 may use any suitable communication means, such as wired or wireless communication.

[0032] For example, the transceiver 47 may utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, and the like.

[0033] The transceiver unit 47 can transmit and / or receive one or more of the following: data acquired by the sensors 42, processing results generated by the flight controller 41, predetermined control data, user commands from a terminal or a remote controller, etc.

[0034] The sensors 42 may include inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), or vision / image sensors (e.g., cameras).

[0035] <Functions of Terminal 1> FIG. 6 is a block diagram illustrating functions implemented in the terminal 1 and the server 2. The terminal 1 includes a communication unit 110, an image acquisition unit 115, a processing unit 120, and a storage unit 130. The processing unit 120 includes a filtering processing unit 121, a specific line extraction unit 122, a filter generation unit 123, a slope determination unit 124, and a tracking target position setting unit 125. The storage unit 130 includes an information / image storage unit 131 and a filter storage unit 132. Note that although the various functional units are illustrated as functional units in the processor 10 of the terminal 1, some or all of the various functional units may be implemented in any of the processor 10 or processor 20 of the terminal 1 or the processor 20 of the server 2, or the flight controller 41 of the unmanned aerial vehicle 4, depending on the capabilities of the processor 10 of the terminal 1 or the processor 20 of the server 2, or the flight controller 41 of the unmanned aerial vehicle 4.

[0036] The communication unit 110 communicates with the server 2 and the unmanned aerial vehicle 4 via the network NW. The communication unit 110 also functions as a reception unit that receives various requests and data from the server 2, the unmanned aerial vehicle 4, etc.

[0037] The image acquisition unit 115 acquires images captured by a digital camera mounted on the unmanned aerial vehicle 4 or a digital camera used by a user from the digital camera, for example, by wireless communication via a communication interface or by wired communication via a USB terminal or the like. The image acquisition unit 115 may be configured to acquire captured images via a storage medium such as a USB memory or SD memory, but a configuration in which the captured images are acquired in real time, particularly via wireless communication from the unmanned aerial vehicle 4 or within the unmanned aerial vehicle 4, is more preferable.

[0038] The captured image acquired by the image acquisition unit 115 may be a moving image or a still image. If the captured image is a moving image, the moving image may be divided into still images for each frame, and the still images may be used in each functional unit of the processing unit 120 described below. Alternatively, still images may be extracted at predetermined intervals from the still images divided into frames, and the extracted still images may be used in each functional unit of the processing unit 120. The captured image captured by the unmanned aerial vehicle 4 or the like may be a color image, a grayscale image, or a black and white image. If the captured image is a color image, the image acquisition unit 115 or a functional unit dedicated to image conversion may perform a process of grayscaling or binarizing the color image.

[0039] The processing unit 120 includes functional units that execute a series of processes for extracting straight lines corresponding to the power line to be inspected (hereinafter referred to as the inspection target line) from the captured image acquired by the image acquisition unit 115 (for example, an image captured along one or more power lines extending alongside one another, or an image obtained by gray-scaling or binarizing the captured image). The inspection target line may be a single line or multiple lines. Furthermore, the captured image may include power lines other than the inspection target line.

[0040] The filtering processing unit 121 performs spatial filtering using a predetermined filter (operator) to generate an image (referred to as a filtered image) in which lines (specific width lines) having a specific width (thickness) corresponding to the inspection target line are emphasized from the captured image. Here, "specific width lines emphasized" means that the specific width lines are displayed with a higher intensity value in the filtered image than the background and non-inspection objects (e.g., power lines, buildings, roads, supports, cross arms, and other structures other than the inspection target line) included in the captured image. The "intensity value" here refers to the pixel value (brightness value) of each pixel, which is the smallest unit constituting the filtered image. For example, pixel values ​​in the filtered image can be expressed in 256 gradations ranging from 0 to 255. In the filtered image, pixels representing the specific width lines have higher pixel values ​​than the background and non-inspection objects. An example of a filtered image generated from the captured image shown in FIG. 7 is the image shown in FIG. 8. In the captured image of Figure 7, the four power lines extending from the upper left to the lower right of the screen are the lines to be inspected, and in Figure 8, these four lines to be inspected are displayed as white lines with higher pixel values ​​than their surroundings.

[0041] Spatial filtering is a technique that has also been used in conventional edge detection processing, which uses edge enhancement filters expressed as square matrices (matrices of n x n size), such as differential filters, Prewitt filters, Sobel filters, Roberts filters, and Laplacian filters, and detects edges such as contours contained in an image through convolution operations using these edge enhancement filters.

[0042] For example, FIG. 9 is a diagram illustrating a convolution operation using a conventional edge enhancement filter. As shown in FIG. 9, spatial filtering using a Prewitt filter having a size of 3×3 involves calculating the output value of a pixel of interest in an output image (processed image) by performing a product-sum operation on the pixel values ​​of the nine pixels (shown in gray in the input image in FIG. 9) and its surrounding pixels, and the pixel values ​​of the nine pixels and the filter coefficients (weights, sometimes called kernel coefficients) corresponding to each pixel. This type of operation, which extracts peripheral information about a pixel of interest and generates a new image, is called a convolution operation. In FIG. 9, a convolution operation is performed on each pixel of the input image, thereby obtaining an output image in which the boundary (i.e., edge) between the pixel value of 50 and the pixel value of 230 in the input image is extracted. Although 540 is output as the product-sum operation result in FIG. 9, the output image is generated by normalizing the product-sum operation result (normalizing to 256 levels from 0 to 255). As shown in Figure 9, in conventional edge detection processing, the contour line of an object is extracted, so when spatial filtering using a conventional edge enhancement filter is performed on one power line, two edges are extracted from the single power line.

[0043] The filtering processing unit 121 of this embodiment performs spatial filtering based on a specific width emphasis filter (first filter) that has selectivity for lines having a specific width corresponding to the line to be inspected (specific width lines), rather than the conventional edge emphasis filter described above. Having "selectivity for lines of a specific width" means that the filter reacts more strongly to lines having a specific width than to other lines of different widths (indicating a value in which the absolute value of the output value for lines of a specific width is greater than that for other lines). Specifically, assuming that the estimated width of the line to be inspected in the captured image is W (unit: pixel), the specific width emphasis filter is a filter that has a calculation area corresponding to this estimated width W.

[0044] The expected width W (specific width) refers to the width in a direction perpendicular to the longitudinal direction of the inspection target line. The width of the inspection target line displayed in the captured image is mainly determined by the distance between the inspection target line and the unmanned aerial vehicle 4. Since it is possible to fly the unmanned aerial vehicle 4 while maintaining a constant distance from the inspection target line, the expected width W can be calculated in advance by test driving the unmanned aerial vehicle 4 and measuring the width (number of pixels) of the inspection target line displayed in the captured image (hereinafter referred to as the test image) obtained during the test run. Based on the expected width W calculated in this way, a specific width emphasis filter 6a as shown below can be designed. In the following description, the length direction of each filter refers to the direction along the longitudinal direction of the inspection target line in the captured image, and the width direction of each filter refers to the direction along the width direction of the inspection target line in the captured image.

[0045] For example, the filter 6a shown in FIG. 10 is an example of a specific width enhancement filter. Unlike conventional edge enhancement filters, the specific width enhancement filter 6a shown in FIG. 10 has a size designed based on the expected width W of the line to be inspected in the captured image. The size of the smallest unit cell constituting the specific width enhancement filter 6a is the same as the size of one pixel, which is the smallest unit of the captured image. The specific width enhancement filter 6a has a central region 61a and two outer regions 62a. The central region 61a has a width corresponding to the expected width W of the line to be inspected in the captured image in the Y-axis direction (width direction) of FIG. 10. Meanwhile, the two outer regions 62a are located outside the central region 61a in the Y-axis direction of FIG. 10, and each have a width (W / 2) that is half the expected width W in the Y-axis direction. In other words, the specific width enhancement filter 6a has a width corresponding to twice the expected width W in the Y-axis direction.

[0046] The length L of the specific width enhancement filter 6a in the X-axis direction (length direction) is not particularly limited as long as it is one pixel or more. For example, the length L may be set so that the specific width enhancement filter 6a is square (i.e., L = 2W). The length L may be set to an even number or an odd number. FIG. 10 illustrates the specific width enhancement filter 6a assuming that the line under inspection extends along the X-axis direction in the captured image (the longitudinal direction of the line under inspection is along the X-axis direction). However, the orientation of the line under inspection in the captured image is not particularly limited. If the captured image is acquired so that the longitudinal direction of the line under inspection is along the Y-axis direction, for example, the matrix of the specific width enhancement filter 6a shown in FIG. 10 may be swapped, and convolution calculation processing may be performed so that the width W of the central region corresponds to the width direction of the line under inspection in the captured image (the direction perpendicular to the longitudinal direction).

[0047] The filter coefficients assigned to each cell of the specific width emphasis filter 6a are designed so that the central region 61a and the outer region 62a have different signs and the sum of the filter coefficients is "0." For example, if the inspection target line is displayed in the captured image with pixel values ​​brighter than the surrounding background, the filter coefficient in the central region 61a may be set with a "positive" sign, and the filter coefficient in the outer region 62a may be set with a "negative" sign, as shown in FIG. 10. Conversely, if the inspection target line is displayed in the captured image with pixel values ​​darker than the surrounding background, the filter coefficient in the central region 61a may be set with a "negative" sign, and the filter coefficient in the outer region 62a may be set with a "positive" sign. In FIG. 10, the weights of the filter coefficients are set to "+1" or "-1" as an example, but weights different from those in FIG. 10 may be set to the filter coefficients as long as the sum of the filter coefficients is "0." The two outer regions 62a can be said to be regions provided to cancel out the filter coefficients of the central region 61a corresponding to the width of the specific width line (i.e., the assumed width W of the line to be inspected) and make the sum of the filter coefficients "0".

[0048] FIG. 11 is a diagram illustrating spatial filtering using the specific width emphasis filter 6a. When using the specific width emphasis filter 6a, a convolution operation is performed on each pixel of the input image, and the resulting output value (product-sum operation result) is normalized to generate an output image (filtered image). In the input image of FIG. 11, white pixels represent the line under inspection, and the direction of the arrow indicates the longitudinal direction of the line under inspection. When a convolution operation is performed on such an input image using the specific width emphasis filter 6a, a filtered image is generated in which the center of the line under inspection (line with a specific width) exhibits higher intensity values ​​than the surrounding area (background, non-inspection object, etc.), as shown in the output image of FIG. 11. In normal edge detection, as described above, edge portions such as the contour line of an object are extracted. However, spatial filtering using the specific width emphasis filter 6a generates a processed image in which the center of the line under inspection (line with a specific width) is emphasized rather than the edges.

[0049] In the above description, the central region 61a has a width (W) corresponding to the assumed width W. However, the "width corresponding to the assumed width W" does not necessarily have to be exactly the same as the assumed width W, but may have some degree of error depending on the value of the assumed width W. For example, the width (assumed width W) of the power line shown in the captured image is assumed to be at least 10 pixels, with an error of ±1 pixel allowed, and the wider the power line, the larger the allowable error. The size of the specific width emphasis filter 6a and the width of each region are integers. From the viewpoint of setting the width of the central region 61a to an even number divisible by 2, if the width W0 (actual measured value) of the line to be inspected measured in the test image is odd, the assumed width W may be set to "W0±1 pixel" and the specific width emphasis filter 6a as shown in FIG. 10 may be designed.

[0050] Furthermore, the specific width emphasis filter 6a is not limited to the example shown in FIG. 10 and may have a configuration such as that shown in FIG. 12. (a) of FIG. 12 is a modified example of the specific width emphasis filter 6a assuming that the expected width W is an odd number. In the specific width emphasis filter 6a of FIG. 12(a), the width of the central region 61a is set to "expected width W (odd number)," and the width of the outer region 62a is set to "(W-1) / 2." In this case, since W × (+1) + ((W-1) / 2) × (-1) does not equal 0, for example, the filter coefficient of one row of each outer region 62a may be set to "-1.5," or an auxiliary row 63 may be inserted as shown in FIG. 12(a) to adjust the sum of the filter coefficients to "0." The total width of the specific width emphasis filter 6a in FIG. 12(a) is 2W + 1. 12(a), the auxiliary row 63 having a filter coefficient of "-0.5" is inserted at the boundary between the central region 61a and the outer region 62a, but the insertion location of the auxiliary row 63 is not necessarily limited. Also, if the expected width W is an odd number, the width of the outer region 62a may be set to "(W+1) / 2" and an auxiliary row 63 having a positive sign may be inserted (in this case, the total width of the specific width emphasis filter 6a is 2W+3).

[0051] FIG. 12(b) shows a modified example of the specific width emphasis filter 6a for when the expected width W is an even number. In the specific width emphasis filter 6a of FIG. 12(b), the width of the outer region 62a is set to "expected width W (even number) / 2," and the width of the central region 61a is set to "W+1." A row with a filter coefficient of "0" is added to the center row in the width direction of the central region 61a. When the total width of the specific width emphasis filter 6a is an even number, the pixel of interest during the convolution operation is shifted by one pixel from the center of the filter. However, as shown in FIG. 12(b), by adding a row with a filter coefficient of 0 to the center of the central region 61a, it is possible to place the pixel of interest at the center of the filter and perform a convolution operation that evenly utilizes information from surrounding pixels.

[0052] The filtering processing unit 121 generates a filtered image in which a specific width line corresponding to the width of the inspection target line is emphasized (especially the center of the specific width line in the width direction) through spatial filtering processing based on the specific width emphasis filter 6a. When multiple power lines of different thicknesses are set as inspection targets, multiple specific width emphasis filters 6a corresponding to each thickness may be prepared. The spatial filtering processing "based on" the specific width emphasis filter 6a does not mean that the convolution operation processing is limited to the specific width emphasis filter 6a alone, but rather that other sub-filters may be used in addition to the specific width emphasis filter 6a. Examples of sub-filters used by the filtering processing unit 121 include a different diameter line extraction filter (second filter), a one-sided edge filter (third filter), and a reflection countermeasure filter (fourth filter). The filtering processing unit 121 preferably performs spatial filtering processing based on one or more filters selected from the different diameter line extraction filter, one-sided edge filter, and reflection countermeasure filter, and the specific width emphasis filter 6a. It is more preferable to perform spatial filtering processing using all four types of filters. By using sub-filters, it is possible to suppress noise from occurring in the filtered image and further improve the accuracy of extracting the inspection target line. Each sub-filter will be described in detail below.

[0053] The different diameter wire extraction filter is used to extract wires with a width W different from the expected width W of the inspection target wire. DThe specific width enhancement filter 6a is a filter that is selective for different diameter lines having a width of 1 / 2. The specific width enhancement filter 6a is selective for specific width lines corresponding to the lines to be inspected, and can enhance the specific width lines through a convolution operation using the specific width enhancement filter 6a. However, the specific width enhancement filter 6a may also react to lines of different thicknesses (e.g., background power lines that are not the target of inspection) that are different from the specific width lines. If such lines of different widths remain as noise in the filtered image, they may cause false detection. Therefore, it is preferable to remove noise due to different diameter lines having widths that are easily misdetected from the filtered image. Therefore, it is preferable to apply a different diameter line extraction filter corresponding to different diameter lines that are easily misdetected to the spatial filtering process. This different diameter line extraction filter can be designed by performing a convolution operation using the specific width enhancement filter 6a on a test image captured during a test drive and identifying line segments with widths that are easily misdetected. Examples of line segments that are easily detected include line segments with a relatively large number of widths and line segments with the most continuity among multiple noises (straight lines other than the lines to be inspected) in the filtered image generated from the test image by the specific width enhancement filter 6a. In other words, the width W different from the expected width W D means a width that is easily erroneously detected, and if there are multiple widths that are easily erroneously detected, multiple different diameter line extraction filters corresponding to the respective widths may be prepared.

[0054] Specifically, the different diameter line extraction filter has the same size as the specific width emphasis filter 6a (i.e., the total width and length L are the same as those of the specific width emphasis filter 6a), but the central region has a width W different from the expected width W. D For example, the filter 6b shown in FIG. 13 is an example of a different diameter line extraction filter. The different diameter line extraction filter 6b in FIG. 13 has a width W D 13 is set to half (W / 2) of the assumed width W. The different diameter line extraction filter 6b in FIG. 13 is the same as the specific width emphasis filter 6a in total width (2W) and length L, but the central region 61b has a width W different from the assumed width W. D In the different diameter line extraction filter 6b, the width W of the central region 61b is set to correspond to (=W / 2). Dis set to be narrower than the width W of the specific width emphasis filter 6a, while the width of the outer region 62a is set to be "(3 / 4)W" wider than the specific width emphasis filter 6a. In the different diameter line extraction filter 6b, the sum of the filter coefficients is also set to "0", so the filter coefficients may be weighted based on the number of cells (number of pixels) in each region. As shown in FIG. 13, the width W that is easily erroneously detected is D is set to half the expected width, the total number of cells in the outer region 62a is three times that of the central region 61b, so the filter coefficient of the outer region 62a is set to "-1 / 3" and the filter coefficient of the central region 61b is set to "+1", which is three times that of the outer region 62a.

[0055] The different diameter line extraction filter 6b shown in FIG. 13 is merely an example, and the width W D may be set appropriately according to the results of the preliminary test, and the filter coefficients may be weighted in a manner other than the above as long as the sum of the filter coefficients is "0". In the convolution operation process of the different diameter line extraction filter 6b on the photographed image, the specific width line hardly reacts, but the set width W D Therefore, by subtracting the output value of the convolution operation by the specific width emphasis filter 6a from the output value of the convolution operation by the specific width emphasis filter 6a, the different diameter lines having the width W are output as a high intensity value (a pixel value higher than that of the specific width line). D This eliminates the need for wires with different diameters, thereby suppressing noise generation.

[0056] For example, suppose that spatial filtering processing based on the specific width emphasis filter 6a and the different diameter line extraction filter 6b is performed on a captured image in which a background power line 71, which is not an object of inspection, is reflected, as shown in Fig. 17. Even if the background power line 71 is displayed as noise in the output image acquired by the convolution operation processing using the specific width emphasis filter 6a, as shown in Fig. 18, by subtracting the output value obtained by performing the convolution operation processing using the different diameter line extraction filter 6b on the captured image in Fig. 17 from the output image in Fig. 18, the intensity value of the background power line 71 (different diameter line) in the filtered image can be weakened and the noise due to the background power line 71 can be removed, as shown in Fig. 19.

[0057] A one-sided edge filter is a filter that responds to only one of the two edges of a specific width line. As mentioned above, power lines included in a captured image typically have two edges per line. While an object having a predetermined area included in an image has at least two contour lines, the specific width enhancement filter 6a may respond to one of the edges (contour lines) of a non-inspection object that does not have the specific width (estimated width W). This response may appear as noise in the filtered image. For example, in the processed images of FIGS. 18 and 19, part of the edge of the building 72, which is a non-inspection object included in the captured image of FIG. 17, is detected as noise. Furthermore, in the convolution operation by the specific width enhancement filter 6a, although a high intensity value is obtained near the center of the specific width line, a response may also occur around the specific width line (output value may occur outside the edge), resulting in noise around the specific width line in the filtered image. To further emphasize the specific width line in the filtered image, it is preferable to suppress such noise. By using a one-sided edge filter as a sub-filter, it is possible to remove noise based on the edge of a part of the object not to be inspected as described above, and noise occurring around a line of a specific width.

[0058] A power line in a captured image has an edge where the background changes to the power line and an edge where the power line changes to the background in a predetermined direction (a direction intersecting the longitudinal direction), and when a luminance distribution (distribution of pixel values) is taken in the predetermined direction, these two edges can be recognized as a rising edge where the luminance value changes from a low value to a high value, and a falling edge where the luminance value changes from a high value to a low value. A one-sided edge filter that responds only to edges on one side means a filter that corresponds to either a rising edge or a falling edge, and one-sided edge filters include rising edge filters and falling edge filters.

[0059] Specifically, the one-sided edge filter has the same size as the specific width emphasis filter 6a (i.e., the total width and length L are the same as those of the specific width emphasis filter 6a), but has only one boundary in the width direction where the sign changes between positive and negative. For example, the filter 6c (6c1 and 6c2) shown in FIG. 14 is an example of a one-sided edge filter. The specific width emphasis filter 6a described above has two outer regions 62a outside the central region 61a in the width direction, and has two boundaries in the width direction where the sign of the filter coefficient changes. On the other hand, as shown in FIG. 14, the one-sided edge filter 6c has an outer region 62c with a different sign from that of the central region 61c only at one end side in the width direction, and is composed of one central region 61c and one outer region 62c. In other words, one of the outer regions 62a in the specific width emphasis filter 6a is replaced by the central region 61c in the one-sided edge filter 6c. One outer region 62a in the one-sided edge filter 6c has the same width (W / 2) as one outer region 62a in the specific width emphasis filter 6a, but the central region 61c in the one-sided edge filter 6c has a width equal to the sum of the width of the central region 61a in the specific width emphasis filter 6a and the width of one outer region 62a (i.e., a width ((3 / 2)W) equivalent to 1.5 times the assumed width W). As with other filters, in the one-sided edge filters 6c1 and 6c2, the filter coefficients of each cell are weighted so that the sum of the filter coefficients becomes "0". Fig. 14 shows an example in which the filter coefficient of the central region 61c, which has three times the number of cells (pixels) of the outer region 62c, is set to "1 / 3", and the filter coefficient of the outer region 62c is set to "-1".

[0060] 14 illustrates a case in which the central region 61c has a positive filter coefficient and the outer region 62c has a negative filter coefficient (i.e., the example illustrates a case in which the line to be inspected has pixel values ​​brighter than the background). In this case, in the one-sided edge filter 6c1 shown on the left side of FIG. 14, the boundary between the central region 61c and the outer region 62c corresponds to the rising edge of the line of the specific width (line to be inspected), and the left one-sided edge filter 6c1 is an example of a rising edge filter. On the other hand, in the one-sided edge filter 6c2 shown on the right side of FIG. 14, the boundary between the central region 61c and the outer region 62c corresponds to the falling edge of the line of the specific width, and the right one-sided edge filter 6c2 is an example of a falling edge filter. When the one-sided edge filter 6c is used as a sub-filter in the spatial filtering process, either the rising edge filter 6c1 or the falling edge filter 6c2 may be used, or it is preferable to use both.

[0061] When a convolution operation using the one-sided edge filter 6c is performed on a captured image, the operation processing result provides output values ​​at positions where noise based on some edges of the object not to be inspected (for example, the edge of a building 72 as shown in FIGS. 18 and 19) and noise on the periphery of a line of a specific width may occur. Therefore, by subtracting the result of the convolution operation using the one-sided edge filter 6c from the operation result based on the specific width emphasis filter 6a as shown in FIGS. 18 and 19 (the output value of the filtered image), the above-mentioned noise can be removed from the filtered image, as shown in the filtered image shown in FIG. 20 (FIG. 20 illustrates an example in which noise based on the edge of the building 72 has been removed). In other words, by using the one-sided edge filter 6c as a sub-filter, the line of a specific width can be further emphasized in the filtered image.

[0062] When subtracting the calculation result by the one-side edge filter 6c from the calculation result based on the specific width emphasis filter 6a, the response (output value) by the one-side edge filter 6c may be entirely subtracted, or a predetermined percentage of the response (output value) by the one-side edge filter 6c may be subtracted. For example, 50% of the output value of the one-side edge filter 6c may be subtracted from the calculation result based on the specific width emphasis filter 6a. The percentage to be subtracted is not particularly limited, and may be determined appropriately depending on the results of a test drive carried out in advance.

[0063] The anti-reflection filter has filter coefficients weighted based on the luminance distribution of the inspection target line acquired from the test image. In a captured image of the inspection target line, a luminance distribution (distribution of pixel values) may occur within the inspection target line due to factors such as the inspection target line reflecting sunlight. For example, FIG. 15 is a diagram illustrating the luminance distribution of the inspection target line in a captured image. (a) of FIG. 15 is a diagram illustrating a simulated example of the inspection target line in a captured image. As shown in the figure, variations in pixel values ​​due to light reflection or the like may occur within the inspection target line. (b) of FIG. 15 is an example of a luminance distribution along the direction of the arrow in (a) of FIG. 15. The horizontal axis of the graph represents the position on the arrow, and the vertical axis represents the luminance value (pixel value) at each position. When the inspection target line in the captured image has such a luminance distribution, depending on the extent of the distribution, the width of the inspection target line in the image may be detected as being different from its expected width W. By adding the response of an anti-reflection filter, which will be described in detail below, to the result of the convolution operation based on the specific width emphasis filter 6a, it is possible to suppress false detection (noise) due to brightness distribution such as light reflection.

[0064] Although the anti-reflection filter has the same overall size (total width and length L) and the size of each region (the width of the central region and the width of the outer regions) as the specific width emphasis filter 6a, the anti-reflection filter has filter coefficients different from those of the specific width emphasis filter 6a. For example, a filter 6d shown in FIG. 16 is an example of an anti-reflection filter. The anti-reflection filter 6d in FIG. 16 has the same size as the specific width emphasis filter 6a, and also has a central region 61d and an outer region 62d that are the same sizes as the central region 61a and the outer region 62a in the specific width emphasis filter 6a. In the anti-reflection filter 6d, weights are set for the filter coefficients according to the luminance distribution of the line under inspection so that the distribution of the filter coefficients in the width direction (Y-axis direction) has a shape similar to the luminance distribution of the line under inspection as shown in FIG. 15(b). When setting the filter coefficients of the anti-reflection filter 6d, for example, an integer corresponding to the luminance distribution of the line to be inspected may be assigned to each cell, and the filter coefficients of each cell may be set by normalizing the coefficients so that the average value of the filter coefficients is "0" (i.e., the sum of the filter coefficients is "0") and the sum of the absolute values ​​of the filter coefficients is "1." The anti-reflection filter 6d in Fig. 16 illustrates an example in which the filter coefficients are set by normalizing the total number N (N = 2W × L) of cells in the filter as the denominator of the filter coefficients so that the average value is 0 and the sum of the absolute values ​​is 1.

[0065] The luminance distribution on the image of the line to be inspected may be obtained by analyzing test images acquired during a test drive. In this case, the test drive may be performed multiple times to acquire multiple test images under different lighting conditions, and then a luminance distribution that occurs with a relatively high frequency may be identified. The anti-reflection filter 6d may then be designed based on the identified luminance distribution. Alternatively, multiple luminance distributions under different lighting conditions may be identified and multiple anti-reflection filters 6d corresponding to each luminance distribution may be prepared. In this case, the filtering processing unit 121 may acquire environmental information, such as weather and lighting conditions, during the inspection (when the image was taken), and then select from the multiple prepared anti-reflection filters 6d (the anti-reflection filters 6d stored in the filter storage unit 132, described later) the anti-reflection filter 6d generated under conditions closest to the conditions during the inspection, and use the selected anti-reflection filter 6d for the spatial filtering process. Alternatively, the user may be allowed to select the anti-reflection filter 6d to be used during the inspection from the multiple anti-reflection filters 6d at their discretion, or multiple anti-reflection filters 6d may be used for the spatial filtering process during the inspection.

[0066] Since the anti-reflection filter 6d has filter coefficients adapted to the reflection of the line under inspection, the result of the convolution operation performed by the anti-reflection filter 6d on the captured image can be added to the result of the convolution operation performed by the specific width emphasis filter 6a, thereby making it possible to further emphasize the specific width line in the filtered image.

[0067] As described above, the filtering processing unit 121 may generate a filtered image by a convolution operation using only the specific width emphasis filter 6a, which is the main filter, or may generate a filtered image by a convolution operation using one or more sub-filters and the specific width emphasis filter 6a (it is preferable to use one or more sub-filters, and it is more preferable to use three types of sub-filters (the different radial line extraction filter 6b, the one-sided edge filter 6c, and the anti-reflection filter 6d)).

[0068] The above description of each sub-filter (6b-6d) illustrates a method in which a captured image is individually convolved with each filter and the results of each computation are summed (subtracted or added together). However, because the law of distribution holds between an input image and multiple filters, multiple filters to be applied to spatial filtering may be combined in advance to generate a composite filter, and the filtering processing unit 121 may use the composite filter to perform a single spatial filtering process (convolution operation) to generate a filtered image. In other words, spatial filtering "based on" the specific width emphasis filter 6a and one or more sub-filters means that a filtered image may be generated from the results of individual convolution operations performed by multiple filters, or may be generated by spatial filtering using a pre-integrated composite filter. From the perspective of increasing computation speed, the latter method using a composite filter is preferred.

[0069] The composite filter is generated by combining the specific width emphasis filter 6a, which is a main filter, with one or more sub-filters, and the filter generation unit 123 shown in FIG. 6 executes the process of generating the composite filter.

[0070] Here, "combining" multiple filters means adding or subtracting filter coefficients for each cell according to the distributive law. As described above, the specific width emphasis filter 6a and the three sub-filters all have the same size, so it is sufficient to add or subtract filter coefficients for each corresponding cell in each filter. For example, when combining four filters, such as coefficient a1 of a specific cell C1 in the specific width emphasis filter 6a, coefficient b1 of a specific cell C1 in the different radial line extraction filter 6b, coefficient c1 of a specific cell C1 in the one-sided edge filter 6c, and coefficient d1 of a specific cell C1 in the anti-reflection filter 6d, the filter generation unit 123 may determine the filter coefficient of the specific cell C1 of the composite filter using the formula "a1-b1-c1+d1." Note that, as described in the description of the one-sided edge filter 6c, when generating a final filtered image by multiplying the output of the sub-filter by a predetermined ratio, the filter generation unit 123 may determine the filter coefficient of the composite filter by multiplying the coefficient of the sub-filter in the above formula by a predetermined ratio to be reflected in the output value.

[0071] In the composite filter generation process by the filter generation unit 123, a filter combination is determined by a user's selection, and the filter generation unit 123 determines the filter coefficients for each cell in accordance with the determination to generate a composite filter. In the composite filter generation process, multiple composite filters may be generated depending on how the sub-filters are combined. For example, a first composite filter combining the specific width emphasis filter 6a and the different diameter line extraction filter 6b, a second composite filter combining the specific width emphasis filter 6a and the one-sided edge filter 6c, a third composite filter combining the specific width emphasis filter 6a and the anti-reflection filter 6d, and a fourth composite filter combining four types of filters may be generated in advance.

[0072] The filtering processing unit 121 may automatically identify a filter to be used in the spatial filtering process from among a plurality of prepared filters (for example, a specific width emphasis filter 6a, one or more sub-filters, and one or more composite filters) according to the shooting conditions, etc., and generate a filtered image. Alternatively, the filtering processing unit 121 may receive a process for selecting a filter to be used in the spatial filtering process from a user, and generate a filtered image by spatial filtering using the filter designated based on the user's selection. The information processing system of this embodiment may have specifications that allow both a mode in which a filter to be used in the spatial filtering process is automatically identified (auto mode) and a mode in which the filter is identified based on a user's selection instruction (manual mode). The information processing system may also have specifications that allow the user to switch between the auto mode and the manual mode.

[0073] The specific straight line extraction unit 122 performs a Hough transform process on the filtered image generated by the filtering processing unit 121, and performs a process of extracting a specific width line corresponding to the line to be inspected from multiple straight lines contained in the filtered image based on the straight line form information obtained by the Hough transform process.

[0074] The Hough transform is a method for detecting straight lines contained in an image by majority vote. When performing a Hough transform on an image after conventional edge detection, for example, the length of a perpendicular line drawn from the origin to a certain line is defined as ρ, and the angle between the perpendicular line and the x-axis is defined as θ. The x- and y-coordinate information of each point constituting an edge in the image is then voted in Hough space (ρθ coordinates) based on the formula "ρ = x cos θ + y sin θ." Line vote count information indicating the number of votes is then obtained. Then, a combination of ρθ representing a line is estimated by majority vote from the line vote count information. This conventional Hough transform, which inputs an edge image, estimates all straight lines contained in the edge image without distinguishing between the line under inspection and other straight lines (straight lines due to the edges of non-inspected objects). Therefore, a process is required to further extract only the line under inspection from the estimated straight lines based on some condition (extraction condition). However, in an edge image (an input image to a Hough transform) generated by conventional edge detection, two edges are extracted from a single line in a captured image, and the edge image may contain numerous straight lines, and the edges of objects other than the object being inspected may be clearly displayed. This makes it difficult to extract only the edge of the inspection target line from the edge image. Even if conditions for extracting an edge corresponding to the inspection target line can be found, complex arithmetic processing is required to extract the inspection target line from the line vote count information for the estimated multiple straight lines. On the other hand, in the filtered image of this embodiment, the center of the inspection target line (a line of a specific width) is emphasized rather than the edges, making it easier to extract the line corresponding to the inspection target line than when a conventional edge image is input and subjected to a Hough transform.

[0075] The specific line extraction unit 122 can employ a general Hough transform technique, but preferably performs a weighted Hough transform as the Hough transform process. The weighted Hough transform process is a technique for generating weighted line vote number information based on the intensity values ​​(the brightness values ​​(pixel values) of each pixel after spatial filtering) in the filtered image. Unlike conventional edge images, the filtered image generated by the filtering processing unit 121 displays specific width lines with higher intensity values ​​than other lines corresponding to the background and non-inspection objects. Therefore, the weighted line vote number information obtained by the weighted Hough transform process assigns a higher number of votes to the specific width lines than to other lines. Therefore, it is possible to more easily extract the specific width lines from multiple lines in the filtered image based on such weighted line vote number information.

[0076] When the lines to be inspected are multiple power lines extending substantially parallel to one another (multiple power lines using a multi-conductor system), the processing unit 120 may include a slope determination unit 124. The slope determination unit 124 executes a process of determining, for example, which of the slopes of the detected multiple lines is the first slope with the most votes based on the number of slope votes. In the determination method based on the number of slope votes, the slope determination unit 124 may determine the first slope with the most votes based on the angle (slope vote number) based on information about the angle (θ) included in line vote number information acquired by a Hough transform process, for example.

[0077] If the inclination determination unit 124 is provided, an extraction condition is set such that there are at least the expected number of lines whose inclinations are within a predetermined range relative to the determined first inclination (since one specific width line is emphasized for each inspection target line in the filtered image, the number of inspection target lines corresponds to the expected number of lines). If the specific line extraction unit 122 determines that this extraction condition is met, it may extract the expected number of lines from among the lines whose inclinations are within a predetermined range relative to the first inclination, in descending order of the number of line votes indicated by the line vote number information. For example, in the filtered image shown in FIG. 18, if the inclination of any of the four lines indicated by the arrows is determined to be the first inclination, there are lines whose inclinations are within a predetermined range relative to the first inclination (other lines extending diagonally), the number of which corresponds to the expected number of lines (four), and therefore the line corresponding to the inspection target line is extracted as shown in FIG. 20. In this way, the inclination determination process of the inclination determination unit 124 may enable the extraction of the line (specific width line) corresponding to the inspection target line with higher accuracy.

[0078] Furthermore, the processing unit 120 may have a functional unit that removes straight lines that are shorter than a predetermined length after the Hough transform processing by the specific line extraction unit 122. As can be seen from the examples of captured images shown in FIGS. 7 and 17, the lines to be inspected in the captured images have a length that reaches from one short side of the image to another short side. Therefore, for example, an extraction condition may be set such that straight lines with a length less than 50% of the length in the X-axis direction of the captured image in FIG. 17 are removed from the filtered image, and the specific line extraction unit 122 may remove straight lines that do not satisfy the extraction condition from the multiple straight lines estimated from the line vote count information, and extract specific width lines that are equal to or longer than a predetermined length. Such processing may enable the straight lines (specific width lines) corresponding to the lines to be inspected to be extracted with higher accuracy.

[0079] The tracking target position setting unit 125 sets a target position to be tracked by the camera (photographing unit) of the unmanned aerial vehicle 4 based on the straight lines extracted by the specific straight line extraction unit 122 (i.e., specific wide lines corresponding to the inspection target lines). As a more specific example, the tracking target position setting unit 125 may calculate an average line at the average position of the multiple straight lines extracted by the specific straight line extraction unit 122 (visualize and add it to the image as necessary), and set the average line as the target position (target straight line) to be tracked. FIG. 21 illustrates an example image in which the average line 8 is visualized and added as a dashed line. Furthermore, the tracking target position setting unit 125 may set one of the specific wide lines extracted by the specific straight line extraction unit 122 as the target position (target straight line) to be tracked.

[0080] After setting the tracking target position as described above, the tracking target position setting unit 125 generates instruction information for controlling at least one of the image capturing unit or the unmanned aerial vehicle 4 itself so that the set target position is included in the image captured by the image capturing unit, and transmits the instruction information to the unmanned aerial vehicle 4 via the communication unit 110. As a more specific example, first, it determines whether a predetermined position (e.g., a center position or an approximate center position) of the inspection target line is included in a predetermined range in the image (particularly, a range near the center of the image and further based on the center position or approximate center position of the image). If it is determined that the target line is not included in the predetermined range, it generates difference information between the predetermined position of the target line and the predetermined range, and generates instruction information based on the difference information for controlling at least one of the state of the image capturing unit (e.g., image capturing direction information, angle of view information, etc.) or the state of the unmanned aerial vehicle itself (e.g., current position information (latitude, longitude, and altitude coordinates), attitude information, rotor motor output information, etc.). The difference information may be any information that allows the amount and direction of deviation from the predetermined range to be grasped, but may be, for example, information regarding the direction and distance (number of pixels) on the image from the center of the image to the predetermined position of the target line. The instruction information may be generated based on, for example, but is not limited to, the difference information, correspondence information between the direction and distance on the image and the control direction and control distance of the imaging unit or unmanned aerial vehicle imaging unit, and at least one of the current state information of the unmanned aerial vehicle. Note that if it is determined that the predetermined position of the inspection target line is included in the predetermined range in the image, the state of the imaging unit or unmanned aerial vehicle is maintained without generating new instruction information.

[0081] 6 stores, in addition to the image acquired by the image acquisition unit 115, a filtered image generated by the filtering processing unit 121 and pixel value information within the filtered image, line vote number information and information on the lines (lines of a specific width) constituting a plurality of power lines generated by the specific line extraction unit 122, tilt vote number information and information on the first tilt generated by the tilt determination unit 124, and an image in which the tracking target position is visualized by the tracking target position setting unit 125. The information and image storage unit 131 may also temporarily store other information and data generated in the processing by each of the functional units 121 to 125 of the processing unit 120.

[0082] The filter storage unit 132 of the storage unit 130 stores information about various filters used for spatial filtering by the filtering processing unit 121, such as the specific width emphasis filter 6a, the different diameter line extraction filter 6b, the one-sided edge filter 6c, the anti-reflection filter 6d, and the composite filter generated by the filter generation unit 123. The information about the filters stored in the filter storage unit 132 includes information for specifying the filters themselves as shown in Figs. 10, 12 to 14, and 16, as well as information about the generation conditions of the filters linked to the identification IDs of the respective filters (for example, a test image used when generating the filter, the expected width W obtained from the test image, the width W of different diameter lines that are likely to be erroneously detected, etc.). D , the luminance distribution of the line under inspection, and the photographing conditions of the test image used when generating the filter).

[0083] <An example of how to extract a straight line> Next, a line extraction method performed by the information processing system according to this embodiment will be described with reference to Fig. 22 etc. Fig. 22 is a flowchart showing the process of performing the line extraction method performed by the information processing system according to this embodiment.

[0084] First, the image acquisition unit 115 of the terminal 1 acquires an image such as an image captured by the camera 42 mounted on the unmanned air vehicle 4 (S101).

[0085] Next, the filtering processing unit 121 of the terminal 1 performs spatial filtering processing on the captured image acquired by the image acquisition unit 115 based on the aforementioned specific width emphasis filter 6a, thereby generating a filtered image in which lines having a specific width (estimated width W) corresponding to the line to be inspected are emphasized from the captured image (S102). In this step S102, it is preferable to perform spatial filtering processing based on one or more sub-filters selected from the different-radius line extraction filter 6b, the one-sided edge filter 6c, and the anti-reflection filter 6d, and the specific width emphasis filter 6a, which is the main filter. It is more preferable to perform spatial filtering processing based on the above three sub-filters and the specific width emphasis filter 6a. In the case of spatial filtering processing using one or more sub-filters, each filter may be individually processed on the captured image, or a convolution operation may be performed using a composite filter that combines multiple filters. The latter method is more preferable.

[0086] Next, the specific line extraction unit 122 of the terminal 1 performs a Hough transform process on the filtered image, and extracts a line (specific width line) corresponding to the object to be inspected from the multiple lines in the filtered image based on the line vote number information acquired by the Hough transform process (S103). It is preferable to perform a weighted Hough transform in the Hough transform process in step S103, and it is preferable to extract a specific width line from the multiple lines in the filtered image based on the line vote number information acquired by the weighted Hough transform and weighted by the intensity value in the filtered image.

[0087] Thus, according to the terminal 1 of this embodiment, even if a linear non-inspection object such as a building, road, or other long object that is not the inspection object is reflected in the captured image, it is possible to extract a straight line corresponding to the target inspection object from the captured image. Furthermore, the above information processing system is capable of extracting a straight line corresponding to the inspection object using a simpler calculation process than deep learning and Hough transform processing based on conventional edge detection technology (technology that extracts two edges from one power line), and can use a more inexpensive and general-purpose calculation processing device than the calculation processing device required for segmentation methods using deep learning, etc.

[0088] The above-described embodiments are merely examples for facilitating understanding of the present disclosure and are not intended to limit the present disclosure. The present disclosure can be modified and improved without departing from the spirit thereof, and it goes without saying that the present disclosure includes equivalents thereof. While the embodiments illustrate a case in which a power line is the inspection target, the information processing system and the like of the present disclosure can also be applied to a system in which piping in a factory is tracked and inspected by an unmanned vehicle, and the extraction target in the present disclosure is not necessarily limited to a power line (preferably, the system is applied to a power line inspection system). [Explanation of symbols]

[0089] 1 device 2 Server 4 Unmanned Aerial Vehicles

Claims

1. An information processing method in which a computer executes information processing to extract a straight line corresponding to a long-sized inspection object from an image captured by an unmanned mobile body, generating a filtered image from the captured image in which lines of a specific width corresponding to the object to be inspected are emphasized by spatial filtering based on a first filter having selectivity for the lines of the specific width; and extracting a line corresponding to the object to be inspected from a plurality of lines in the filtered image based on line vote count information obtained by performing a Hough transform process on the generated filtered image.

2. 2. The information processing method according to claim 1, wherein in the process of generating the filtered image, the spatial filtering process is performed based on the first filter and a second filter having selectivity for lines of different diameters having a width different from the specific width, and the filtered image in which lines of the specific width are emphasized is generated.

3. 2. The information processing method according to claim 1, wherein in the process of generating the filtered image, the spatial filtering process is performed based on the first filter and a third filter that responds only to one of the two edges of the line of the specific width, thereby generating the filtered image in which the line of the specific width is emphasized.

4. the third filter includes a filter that reacts only to one of the two edges of the line of the specific width, and a filter that reacts only to the other of the two edges; The information processing method according to claim 3, wherein the filtered image is generated by the spatial filtering process based on the first filter, the third filter that responds only to one of the two edges, and the third filter that responds only to the other of the two edges.

5. 2. The information processing method according to claim 1, wherein in the process of generating the filtered image, the filtered image in which lines of the specific width are emphasized is generated by the spatial filtering process based on the first filter and a fourth filter having filter coefficients weighted based on the luminance distribution of the object to be inspected in the test image.

6. before performing the spatial filtering process, further comprising a process of generating a composite filter by combining at least one other filter and the first filter; The other filter is a second filter selective for different diameter wires having widths different from the specified width; a third filter that responds to only one of the two edges of the line of the specific width; and a fourth filter having a filter coefficient weighted based on the luminance distribution of the inspection object in the test image; An information processing method according to any one of claims 1 to 5, wherein in the process of generating the filtered image, the spatial filtering process is performed by a convolution operation using the composite filter, and the filtered image in which lines of the specific width are emphasized is generated.

7. The information processing method according to claim 6 , wherein the composite filter is generated by combining the first filter, the second filter, the third filter, and the fourth filter.

8. The Hough transform process is a weighted Hough transform, An information processing method according to any one of claims 1 to 5, further comprising extracting a straight line corresponding to the object to be inspected from a plurality of straight lines in the filtered image based on line voting number information obtained by the weighted Hough transform and weighted by intensity values ​​in the filtered image.

9. An information processing system that extracts a straight line corresponding to a long-sized inspection object from an image captured by an unmanned mobile body, a filtering processing unit that generates a filtered image in which lines having a specific width corresponding to the object to be inspected are emphasized from the captured image by spatial filtering processing based on a first filter that has selectivity for the lines having the specific width; and a specific line extraction unit that extracts a line corresponding to the object to be inspected from a plurality of lines in the filtered image based on line vote count information obtained by Hough transform processing of the generated filtered image.

10. A program that causes a computer to execute information processing to extract a straight line corresponding to a long-sized inspection object from an image captured by an unmanned moving body, generating a filtered image from the captured image in which lines of a specific width corresponding to the object to be inspected are emphasized by spatial filtering based on a first filter having selectivity for the lines of the specific width; and extracting a line corresponding to the object to be inspected from a plurality of lines in the filtered image based on line vote count information obtained by Hough transform processing of the generated filtered image.

Images