Structure identification system, structure identification method, and program

Abstract

To reduce the workload when detecting the same structure multiple times. [Solution] One aspect of the present invention is a structure identification system comprising: an imaging unit mounted on a vehicle; an extraction unit for extracting a structure region including a structure from an image captured by the imaging unit; a position acquisition unit for acquiring position information of the vehicle or imaging unit at the time the image was captured; a identification unit for identifying a structure number that identifies a structure included in the structure region extracted by the extraction unit based on the position information acquired by the position acquisition unit; a grouping unit for classifying a plurality of images captured by the imaging unit into groups; and an output unit for outputting the plurality of images and the structure number identified by the identification unit for each group.

Description

Technical Field

[0001] The present invention relates to a structure identification system, a structure identification method, and a program.

Background Art

[0002] Conventionally, in order to inspect infrastructure facilities such as power grids and communication networks, structures of the infrastructure facilities are imaged by a camera device, and inspection is performed using moving image data. As this type of technology, for example, the technologies described in Patent Documents 1 to 3 are known.

[0003] The same structure identification device of Patent Document 1 is a same structure identification device that extracts a captured image in which the same structure appears from a plurality of captured image groups captured at regular intervals by a plurality of cameras. This same structure identification device estimates a reference image in which the same structure appears in a plurality of captured image groups based on the deviation in the imaging timing of the structure considering a fixed distance and the imaging directions of each of the plurality of cameras, or based on the distance from the imaging positions of each of the plurality of cameras to the structure, and includes a same structure determination unit that determines that the structures appearing in the captured images within a predetermined range from the reference image are the same structure.

[0004] The same structure identification device described in Patent Document 2 is a same structure identification device that extracts a captured image in which the same structure appears from a captured image group captured at regular intervals. This same structure identification device detects a target structure from the captured image group, and determines whether or not the structures appearing in consecutive captured images are the same structure based on the change in the detection position of the structure on the captured image between consecutive captured images. Further, the same structure identification device determines that the structures appearing in consecutive captured images are the same structure when the detection position of the structure moves in the same direction as the moving direction of the scenery of the consecutive captured images.

[0005] The structure identification device described in Patent Document 3 is a structure identification device that detects a target structure from a group of images captured by each of several cameras installed on a vehicle. This structure identification device detects a structure from each of the group of images, extracts images showing the same structure from each of the group of images, determines which images showing the same structure are present among the group of images based on the time and direction of capture, and displays the multiple images showing the same structure side by side for each camera. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 7030732 [Patent Document 2] Patent No. 7065051 [Patent Document 3] Patent No. 7103975 [Overview of the project] [Problems that the invention aims to solve]

[0007] The technologies described in the aforementioned Patent Documents 1-3 extract images of the same structure from a group of images taken at regular intervals by multiple cameras. However, even if images of the same structure can be extracted, having multiple images of the same structure increases the burden on the worker to review them. For example, if the same structure is detected multiple times, the obstacle to the structure may be detected redundantly. Furthermore, when the same structure is detected multiple times, the burden increases even further in order to select the optimal image.

[0008] The present invention has been made in view of the above problems, and aims to provide a structure identification system, a structure identification method, and a program that can reduce the workload when detecting the same structure. [Means for solving the problem]

[0009] (1) One aspect of the present invention is a structure identification system comprising: an imaging unit mounted on a vehicle; an extraction unit for extracting a structure region including a structure from an image captured by the imaging unit; a position acquisition unit for acquiring position information of the vehicle or imaging unit at the time the image was captured; an identification unit for identifying a structure number that identifies a structure included in the structure region extracted by the extraction unit based on the position information acquired by the position acquisition unit; a grouping unit for classifying a plurality of images captured by the imaging unit into groups; and an output unit for outputting the plurality of images and the structure number identified by the identification unit for each group.

[0010] (2) One aspect of the present invention is the above-described structure identification system, wherein the grouping unit may classify a plurality of images included in a predetermined time range into the same group.

[0011] (3) One aspect of the present invention is the above-described structure identification system, wherein the grouping unit may classify a plurality of images included in a predetermined position range into the same group.

[0012] (4) One aspect of the present invention is the above-described structure identification system, wherein the grouping unit may classify a plurality of images into groups of the same structure number identified by the identification unit.

[0013] (5) One aspect of the present invention is a structure identification system, which may include a selection unit that selects the optimal image from a plurality of images classified by group.

[0014] (6) One aspect of the present invention is the structure identification system described above, wherein the selection unit may select an image in which the center of the structure region extracted by the extraction unit is close to a predetermined position in the lateral direction of the image.

[0015] (7) One aspect of the present invention is the structure identification system described above, wherein the selection unit may select an image in which the upper end of the structure region extracted by the extraction unit is included in a predetermined range below the upper end of the image.

[0016] (8) One aspect of the present invention is a method for identifying a structure, comprising the steps of: a computer extracting a structural region including a structure from an image captured by an imaging unit mounted on a vehicle; acquiring location information of the vehicle or imaging unit at the time the image was captured; identifying a structure number that identifies a structure included in the extracted structural region based on the acquired location information; classifying a plurality of images captured by the imaging unit into groups; and outputting the plurality of images and the identified structure number for each group.

[0017] (9) One aspect of the present invention is a program that causes a computer to perform the following steps: extract a structural region including a structure from an image captured by an imaging unit mounted on a vehicle; acquire position information of the vehicle or imaging unit at the time the image was captured; identify a structural number that identifies a structure included in the extracted structural region based on the acquired position information; classify a plurality of images captured by the imaging unit into groups; and output the plurality of images and the identified structural numbers for each group. [Effects of the Invention]

[0018] According to one aspect of the present invention, the workload when detecting the same structure can be reduced. [Brief explanation of the drawing]

[0019] [Figure 1] This is a perspective view showing an example of a vehicle equipped with the structure identification system according to the first embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of the structure identification system 1 in the first embodiment. [Figure 3] FIG. 1 is a block diagram showing an example of the recognition unit 420 according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a screen displayed on the terminal device 300 in the facility grasping service according to the first embodiment. [Figure 5] FIG. 3 is a diagram for explaining time deviation in the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining time synchronization processing in the first embodiment. [Figure 7] FIG. 5 is a diagram showing an example of a display screen for time synchronization processing in the first embodiment. [Figure 8] FIG. 6 is a flowchart showing an example of a processing procedure in time synchronization processing in the first embodiment. [Figure 9] FIG. 7 is a flowchart showing another example of a procedure for obtaining a time difference between moving images in the first embodiment. [Figure 10] FIG. 8 is a diagram for explaining the effect of time synchronization processing in the first embodiment. [Figure 11] FIG. 9 is a diagram showing an example of a process for stabilizing the trajectory of the vehicle 10 (smoothing position information) in the first embodiment. [Figure 12] FIG. 10 is a diagram for explaining a search area setting process for specifying a utility pole number in the first embodiment. [Figure 13] FIG. 11 is a diagram for explaining the relationship between the ratio of the search area in the captured image and the distance Ym from the position of the vehicle 10 to the center of the search area in the first embodiment. [Figure 14] FIG. 12 is a diagram for explaining a process for specifying a utility pole number in the first embodiment. [Figure 15] FIG. 13 is a flowchart for explaining a process for smoothing position information in the first embodiment. [Figure 16] FIG. 14 is a flowchart showing the procedure of a process for specifying a utility pole number in the first embodiment. [Figure 17]This block diagram shows an example of the functional configuration of the structure identification device 200A in the second embodiment. [Figure 18] This is a diagram illustrating an example of grouping in the second embodiment. [Figure 19] This figure illustrates another example of grouping in the second embodiment. [Figure 20] This figure illustrates another example of grouping in the second embodiment. [Figure 21] This figure illustrates an example of the processing of the selection unit in the second embodiment. [Figure 22] This is a diagram illustrating an example of a detection frame in the second embodiment. [Figure 23] This figure illustrates an example of the conditions for selecting an image by the selection unit 270 in the second embodiment. [Figure 24] This flowchart shows an example of the grouping process in the second embodiment. [Figure 25] This flowchart shows an example of the selection process in the second embodiment. [Figure 26] This flowchart shows an example of the selection process in the second embodiment. [Modes for carrying out the invention]

[0020] The following describes a structure identification system, a structure identification method, and a program to which the present invention is applied, with reference to the drawings.

[0021] <First Embodiment> Figure 1 is a perspective view showing an example of a vehicle equipped with a structure identification system according to the first embodiment. The structure identification system 1 of this embodiment comprises an imaging unit 100 and a position acquisition unit 220 mounted on a vehicle 10. The imaging unit 100 is installed, for example, on the upper part of the vehicle 10 to image a range including the direction of travel of the vehicle 10. The pan angle of the imaging unit 100 may be adjusted to image structures in a range diagonally in front of the direction of travel of the vehicle 10. The tilt angle of the imaging unit 100 may also be adjusted to image structures that exist above the ground, such as utility poles.

[0022] In this embodiment, the structures are, for example, utility poles installed at the edge of the road on which the vehicle 10 travels, and equipment installed on the utility poles. The structure identification system 1 can, for example, support a service in which the vehicle 10 travels while taking photographs, identifies the utility pole number of the utility pole included in the acquired image, and transmits the image including the identified utility pole. The structure identification system 1 can also support an equipment identification service in which it uses the acquired image to recognize things like nests being formed on the utility pole, natural objects such as vines being attached to the utility pole, and deterioration of the utility pole and equipment attached to the utility pole, and transmits the recognition results. Furthermore, the structures in this embodiment are not limited to utility poles, but may also be infrastructure equipment related to traffic such as roads and traffic lights, or structures such as building equipment.

[0023] <Functional Configuration of Structure Identification System 1> Figure 2 is a block diagram showing an example of the functional configuration of the structure identification system 1 in the first embodiment. The structure identification system 1 is configured such that, for example, a structure identification device 200 is connected to a terminal device 300, a server device 400, and a utility pole information database device 500 via a communication network NW. The structure identification device 200, the terminal device 300, and the server device 400 are equipped with communication interfaces such as NICs (Network Interface Cards) and wireless communication modules (not shown in Figure 1). The communication network NW includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), and a cellular network.

[0024] The imaging unit 100 includes, for example, a function to generate an image, as well as a timing unit 110. The timing unit 110 generates time information. The imaging unit 100 outputs the image with the added time information to the structure identification device 200.

[0025] The structure identification device 200 is, for example, an information processing device that performs communication processing and various application processing. The structure identification device 200 includes, for example, an extraction unit 210, a position acquisition unit 220, an identification unit 230, a synchronization unit 240, and an output unit 250. The extraction unit 210, the position acquisition unit 220, the identification unit 230, the synchronization unit 240, and the output unit 250 are realized by a processor such as a CPU (Central Processing Unit) executing a program stored in program memory. Furthermore, some or all of these functional units may be realized by hardware such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array), or they may be realized by the cooperation of software and hardware.

[0026] The extraction unit 210 extracts the structural region, including the structure, from the image captured by the imaging unit 100. The position acquisition unit 220 acquires the position information of the vehicle 10 or the imaging unit 100 at the time the image was captured. The position acquisition unit 220 also includes a timing unit 222. The timing unit 222 generates time information. The position acquisition unit 220 acquires position information with the time information added.

[0027] The identification unit 230 identifies a structure number that identifies a structure included in the structure region extracted by the extraction unit 210 based on the location information acquired by the location acquisition unit 220. In the following embodiment, we will describe how to identify a utility pole as a structure and specify the utility pole number as a structure number. The identification unit 230 identifies the utility pole number by referring to the utility pole information database managed by the utility pole information database device 500. The utility pole information database device 500 is an information processing device that associates the location information of a utility pole with the utility pole number and performs management processing such as adding, deleting, and modifying information in the utility pole information database.

[0028] The synchronization unit 240 includes a detection unit 242 that detects a specific sound. The synchronization unit 240 synchronizes the time information measured by the imaging unit 100 and the time information measured by the position acquisition unit 220 with the time when the detection unit 242 detects a specific sound. The output unit 250 outputs the utility pole number and the captured image to the terminal device 300. The output unit 250 may output the utility pole number and the recognition result for the utility pole to the terminal device 300.

[0029] The terminal device 300 is, for example, an information processing device such as a personal computer, smartphone, or tablet terminal that is operated by a user who manages the structure. The terminal device 300 acquires the utility pole number and captured image output by the structure identification device 200 and displays the utility pole number and captured image on a display device or the like. The terminal device 300 may also acquire the utility pole number and the recognition result for the utility pole and display it on a display device or the like.

[0030] The server device 400 is an information processing device that receives requests for recognition of utility poles and provides recognition results. The server device 400 comprises, for example, a coordination unit 410 and a recognition unit 420. The coordination unit 410 and the recognition unit 420 are realized, for example, by a processor such as a CPU executing a program stored in program memory.

[0031] The coordinating unit 410 performs, for example, processing to receive captured images and recognition requests from the structure identification device 200. The coordinating unit 410 may perform predetermined image processing on the captured images acquired from the structure identification device 200 and output the captured images to the recognition unit 420.

[0032] The recognition unit 420 includes, for example, multiple image recognition engines 422A, 422B, ... 422N (where N is a natural number). When referring to multiple image recognition engines collectively, they are simply referred to as image recognition engine 422. Each image recognition engine 422 is a machine learning model (AI model) configured to correspond to the recognition target. The recognition target is, for example, a nest, a natural object, or a utility pole. The image recognition engine 422 takes an captured image as input and outputs information including the presence or absence of the recognition target, the region where the recognition target is present, and the confidence level.

[0033] <Recognition process in the structure identification system 1> Figure 3 is a block diagram showing an example of the recognition unit 420 in the first embodiment. The recognition unit 420 includes, for example, a learning image database 430, a learning processing unit 432, and an image recognition engine 422.

[0034] The training image database 430 is a database that stores, for example, captured images and tag information acquired from a terminal device 300 as training data. The training image database 430 stores, for example, positive example images as training data. A positive example image is, for example, a captured image that includes a nest and a utility pole, to which tag information indicating the nest has been added.

[0035] The learning processing unit 432 outputs captured images from the training data to the image recognition engine 422 and obtains recognition results from the image recognition engine 422. The learning processing unit 432 updates the processing parameters of the image recognition engine 422 so that the image recognition engine 422 outputs tag information attached to the captured images. The processing parameters of the image recognition engine 422 are, for example, filters (also called weights or biases) included in the neural network. The processing parameters are stored in the storage unit 4221.

[0036] The image recognition engine 422 includes, for example, a memory unit 4221 and a processing unit 4222. The memory unit 4221 stores processing parameters as a result of learning. The processing parameters are updated by the learning processing unit 432. When the processing unit 4222 acquires an image, it outputs a recognition result using the recognition model constructed from the processing parameters.

[0037] <Display screen for the equipment monitoring service> Figure 4 shows an example of a screen displayed on the terminal device 300 in the equipment monitoring service of the first embodiment. The terminal device 300, for example, after performing a process to identify structures from captured images using an equipment identification service application, displays a screen as shown in Figure 4. The terminal device 300 displays, for example, the vehicle 10's travel trajectory, a map including the identified utility poles, a list of the identified utility pole numbers, and captured images of the identified utility poles. On the captured images of the identified utility poles, areas where nesting was recognized, areas where natural objects were recognized, and areas where equipment was recognized are superimposed, corresponding to the utility pole numbers. This allows the user of the terminal device 300 to inspect the utility poles with the identified utility pole numbers along the vehicle 10's travel trajectory.

[0038] <Time synchronization process> The following describes the time synchronization process in the structure identification device 200. Figure 5 is a diagram illustrating the time difference in the first embodiment. The imaging unit 100 may be mounted in multiple units on the vehicle 10, and time information may be generated by the timing unit 110 of each imaging unit 100. The vehicle 10 is also equipped with a timing unit 222 of a position acquisition unit 220 (drive recorder). For example, cameras A and C are the timing units 110 of the imaging unit 100, and camera B is the timing unit 222 of the position acquisition unit 220. Therefore, if there is a discrepancy in time information between multiple timing units 110 and 222, a problem is anticipated in that even if the captured images and position information were acquired at the same time, they may be identified as captured images from different locations. In response, the structure identification device 200 detects a specific sound using the detection unit 242, and the synchronization unit 240 synchronizes the time information measured by the imaging unit 100 and the time information measured by the position acquisition unit 220 with the time when the specific sound was detected.

[0039] Figure 6 is a diagram illustrating the time synchronization process in the first embodiment. The detection unit 242 detects audio waveform information indicating volume changes inside the vehicle 10, as shown in the left figure, and normalizes the audio waveform information over several minutes to create audio waveform information with a range of 0 to 1, as shown in the right figure, during time synchronization. The detection unit 242 detects audio waveform information corresponding to, for example, handclaps, at detection times t1 and t2. The synchronization unit 240 synchronizes the time of the imaging unit 100 and the position acquisition unit 220 with the first detection time of the two detected times.

[0040] Figure 7 shows an example of a display screen for time synchronization processing in the first embodiment. The terminal device 300 includes, for example, videos of the front, directly above, directly to the side, behind, and foreground; a button to return to the previous foreground video; a button to advance to the next foreground video; and a button to confirm the time difference between videos. This allows the user to operate the display screen to synchronize the foreground video with the location information corresponding to that foreground video. This allows the structure identification system 1 to synchronize the time information from the timing unit 110, which is added to the image captured by the imaging unit 100, with the time information from the timing unit 222, which is added to the position information of the position acquisition unit 220.

[0041] Figure 8 is a flowchart showing an example of the processing procedure in the time synchronization process in the first embodiment. First, the synchronization unit 240 acquires audio waveform information for a predetermined time from the video image captured by the imaging unit 100 (step S100) and normalizes the audio waveform (step S102). Next, the synchronization unit 240 detects two peaks in the audio waveform within a predetermined period (step S104) and determines whether or not a time synchronization point has been detected (step S106). If the synchronization unit 240 detects a time synchronization timing, it performs time synchronization at the time of the maximum value of the audio waveform (step S106: YES, S108). If the synchronization unit 240 cannot detect a time synchronization timing, it does not perform time synchronization (step S106: NO).

[0042] Figure 9 is a flowchart showing another example of the procedure for obtaining the time difference between video images in the first embodiment. The synchronization unit 240 obtains time synchronization points from the judgment video, which is a moving image used to determine time synchronization (step S200), and obtains time synchronization points from the foreground image (step S202). For example, depending on whether a still image included in the moving image is selected on the display screen of Figure 7, the synchronization unit 240 obtains the time of the selected still image as a time synchronization point. Next, the synchronization unit 240 calculates the time difference between the points in the judgment video and the points in the foreground video as the time difference between the videos (step S204).

[0043] Figure 10 is a diagram illustrating the effect of the time synchronization process in the first embodiment. The structure identification system 1, for example, captures video footage of a utility pole at point 1 using a front camera, video footage of a utility pole at point 2 using a front camera, video footage of a utility pole at point 3 using a rear camera, and video footage of a utility pole at point 4 using a rear camera. In this case, since the time at which the utility pole is captured differs by several centimeters (several seconds) depending on the camera's installation position, the time can be synchronized between the video footage captured by multiple cameras. As a result, the structure identification system 1 can accurately match the utility pole number with the captured image when an object such as a nest or utility pole is discovered.

[0044] <Identification process for utility pole numbers> The following describes the process of identifying utility pole numbers based on location information in the structure identification device 200. Figure 11 shows an example of a process for stabilizing the trajectory of the vehicle 10 (smoothing of position information) in the first embodiment. As described above, even if the recognition unit 420 recognizes a utility pole or nesting site included in the captured image, if the utility pole number cannot be identified, the effort required to identify the utility pole number from the location information increases, leading to higher work costs. In contrast, the structure identification system 1 in this embodiment calculates the trajectory of the vehicle 10 and performs processing to stabilize its true bearing when the location information of the vehicle 10 changes.

[0045] When the vehicle 10 is stopped or moving at a low speed and multiple location information points are detected by the position acquisition unit 220 (left figure), the unit determines a representative position of the multiple location information points (center figure) and acquires the trajectory of the location information based on the determined representative position (right figure). Specifically, if the vehicle 10 stops or moves at a low speed while moving from detection point (t) to detection point (t+a), the location information may fluctuate, for example, every second, as A, B, C, D, E, F, even though the trajectory of the vehicle 10 is moving from detection point (t) to detection point (t+a). In response to this, the position acquisition unit 220 determines a representative point for the positions of A, B, C, D, E, F. At this time, the position acquisition unit 220 plots the average position of each plot of location information A, B, C, D, E, F on a two-dimensional plane and calculates the average position of plots A, B, C, D, E, F. The position acquisition unit 220 then acquires the trajectory of the vehicle 10 as it moves in the order of detection point (t), representative point, and detection point (t+a). This allows the identification unit 230 to identify the utility pole number based on the trajectory of the acquired location information.

[0046] Figure 12 is a diagram illustrating the process of setting a search area for identifying the utility pole number in the first embodiment. The extraction unit 210 sets a search area based on the location information of the vehicle 10, the location information of the structure identification device 200, or the location information of the imaging unit 100, and the imaging range of the imaging unit 100, and extracts structures included in the set search area. At this time, the extraction unit 210 creates a point tilted by XO to the left with respect to the direction of travel from the current position information of the vehicle 10 or the installation position of the imaging unit 100, and sets this point as the center position of the search area. The identification unit 230 sets two diagonals that intersect at the center position. As a result, the extraction unit 210 creates a rectangle, for example, with X (meters) in the vertical direction and Y (meters) in the direction of travel, and sets the created rectangle as the search area for utility poles. The search area may be, for example, about 50 (meters) in the vertical direction and about 100 (meters) in the direction of travel.

[0047] Figure 13 is a diagram illustrating the relationship between the proportion of the search area in the captured image and the distance Ym from the position of the vehicle 10 to the center of the search area in the first embodiment. The extraction unit 210 adjusts the distance from the position information of the vehicle 10 or imaging unit 100 to the center of the search area according to the size of the area recognized as an extracted utility pole. For example, the extraction unit 210 may shorten the distance Ym from the position information of the vehicle 10 or imaging unit 100 to the center of the search area if the size of the area recognized as an extracted utility pole is large. The extraction unit 210 reduces the size of the search area by shortening the distance Ym. Although the physical size of the utility pole captured by the imaging unit 100 is constant, the size of the utility pole region included in the captured image varies depending on the shooting position relative to the utility pole. Therefore, the extraction unit 210 shortens the distance Ym as the size of the utility pole region included in the captured image increases. In other words, the distance Ym is made inversely proportional to the size of the utility pole region.

[0048] The extraction unit 210 may set the distance Ym inversely proportional to the size of the utility pole region included in the captured image, or the square of the distance Ym inversely proportional to the size of the utility pole region included in the captured image. Furthermore, the extraction unit 210 may increase the distance from the position information of the vehicle 10 or the imaging unit 100 to the center of the search region as the size of the region recognized as an extracted utility pole decreases. This makes it possible to capture images of utility poles included in the captured image more clearly and improves the accuracy of identifying utility poles.

[0049] Figure 14 is a diagram illustrating the process of identifying the utility pole number in the first embodiment. The identification unit 230 acquires location information of the utility pole closest to the center position of the search area. At this time, the identification unit 230 calculates the location information of the utility pole closest to the center position of the search area based on the location information acquired by the location acquisition unit 220, the center position of the search area, and the utility pole position relative to the center position. The identification unit 230 compares the calculated location information of the utility pole closest to the center position of the search area with the utility pole information database and identifies the utility pole number corresponding to the location information.

[0050] Figure 15 is a flowchart illustrating the process of smoothing positional information in the embodiment. The position acquisition unit 220 resets the location information retention information for each NMEA (National Marine Electronics Association) file containing GPS-related location information if the file names are not sequential (step S300).

[0051] Next, the position acquisition unit 220 reads the rows (records) contained in the NMEA file and determines whether or not there is previous position information for the position information (row) of interest (step S302). If there is previous position information (step S302: YES), the position acquisition unit 220 divides the elapsed time from the time attached to the previous position information into predetermined intervals and calculates the distance traveled for each predetermined interval (step S304). If the calculated distance traveled indicates that the current position has traveled a predetermined distance or more, the position acquisition unit 220 adds it to the distance traveled (step S306). The position acquisition unit 220 loops through steps S300 to S306 for each NMEA file.

[0052] If there is no previous position information (step S302: NO), the position acquisition unit 220 determines whether the travel distance has exceeded a threshold (step S308). If the travel distance has exceeded a threshold (step S308: YES), the position acquisition unit 220 smooths the position information by averaging the position information included in the travel distance (step S310, Figure 11). The position acquisition unit 220 loops the process from step S304 to step S310 at predetermined intervals, and loops the process from step S300 to step S310 for the number of NMEA files. If there is any position information that has not been smoothed by the above process, the position acquisition unit 220 smooths that position information (step S312).

[0053] Figure 16 is a flowchart showing the procedure for identifying the utility pole number in the first embodiment. The identification unit 230 obtains the location information for the time to be determined and the location information for the time of the previous time to be determined from the NMEA file (step S400). The identification unit 230 determines whether or not the two pieces of location information have been obtained (step S402). If it has not been obtained, it terminates the process of this flowchart (step S402: NO). If it has been obtained, it proceeds to step S404 (step S402: YES).

[0054] In step S404, the identification unit 230 identifies the direction of travel of the vehicle 10. If the direction of travel cannot be determined, it calculates the direction of travel from the current position information and the previous position information. The identification unit 230 calculates the distance Ym of the search area from the area ratio of the detection frame to the captured image (step S406). The detection frame is a rectangular area in the captured image that includes the utility pole. Next, the identification unit 230 sets the center position based on the distance Ym and sets the search area based on the set center position (step S410).

[0055] Next, the identification unit 230 detects utility poles included in the search area, calculates the distance from the center of the search area, and obtains location information for each utility pole. The identification unit 230 refers to the utility pole information database and identifies the utility pole number corresponding to the location information for each utility pole (step S412). Based on the location information in the utility pole information database for the identified utility pole number, the identification unit 230 identifies the utility pole closest to the vehicle 10's current location (step S414).

[0056] As described above, the structure identification system 1 in the first embodiment can stabilize the vehicle 10's trajectory by smoothing its position information when the vehicle 10 is stopped or traveling at a low speed. Furthermore, the structure identification system 1 can set a search area based on the vehicle 10's position information and identify the utility pole numbers of utility poles included in the search area. As a result, the structure identification system 1 can easily identify the utility pole numbers (identification information) for recognized objects such as utility poles, nests, natural objects, and equipment, thereby reducing the cost of inspection work.

[0057] <Second Embodiment> The second embodiment will be described below. <Functional Configuration of Structure Identification System 1> Figure 17 is a block diagram showing an example of the functional configuration of the structure identification device 200A in the second embodiment. Note that the same reference numerals are used for parts identical to those in the above-described embodiment, and detailed explanations are omitted.

[0058] In the structure identification system 1, if the pole number of a utility pole identified by the identification unit 230 is included in multiple captured images, the burden on the user to confirm it becomes significant. In contrast, the structure identification system 1 in the second embodiment groups the captured images. Furthermore, the invention according to claim 1 of the present application in the second embodiment selects the image best suited for inspecting the equipment from the grouped multiple images.

[0059] The structure identification device 200A differs from the structure identification device 200 in the first embodiment in that, in addition to the extraction unit 210, position acquisition unit 220, identification unit 230, synchronization unit 240, and output unit 250, it also includes a grouping unit 260 and a selection unit 270. The grouping unit 260 classifies multiple images captured by the imaging unit 100 into groups. The selection unit 270 selects the optimal image from multiple images classified by group.

[0060] <Grouping methods based on time period> Figure 18 is a diagram illustrating an example of grouping in the second embodiment. The grouping unit 260 may classify multiple images contained within a predetermined time range into the same group. For example, grouping is performed if the time between the Nth detection and the N+1th detection is within a specified time. As shown in Figure 18(a), if the same utility pole number is detected twice in one second, the grouping unit 260 classifies the two captured images into the same group. The selection unit 270 can select the optimal image from the two captured images and output it from the output unit 250. On the other hand, as shown in Figure 18(b), the grouping unit 260 does not classify the two captured images into the same group even if the same utility pole number is detected once in one second and then again one second later. In this case, the output unit 250 outputs one captured image every second. This grouping method does not depend on the location information of utility poles, so it can perform grouping of captured images even when location information is unavailable.

[0061] <Grouping method based on position> Figure 19 is a diagram illustrating another example of grouping in the second embodiment. The grouping unit 260 may classify multiple images included within a predetermined position range into the same group. For example, the grouping unit 260 calculates the distance from the position where a utility pole is detected for the Nth time to the position where a utility pole is detected for the N+1th time, and groups two captured images if they are included within the predetermined position range.

[0062] As shown in Figure 19(a), the grouping unit 260 performs grouping if the predetermined position range is 30 meters and the distance from the Nth pole detection position to the N+1th pole detection position is within 30 meters, because it falls within the predetermined position range. The selection unit 270 can select the optimal image from the two captured images and output it from the output unit 250. As shown in Figure 19(b), the grouping unit 260 has a predetermined position range of 20 meters. If the distance from the Nth detected utility pole to the N+1th detected utility pole exceeds 20 meters, the grouping unit 260 does not perform grouping because the detected pole is outside the predetermined position range. In this case, the output unit 250 outputs an image for each detected utility pole. This grouping method allows for grouping of images acquired when the vehicle 10 is stopped. For example, this grouping method allows for grouping beyond the predetermined time range mentioned above, as long as the vehicle 10 is stopped.

[0063] <Grouping method based on utility pole number> Figure 20 is a diagram illustrating another example of grouping in the second embodiment. The grouping unit 260 may classify multiple images into groups of the same structure number identified by the identification unit 230. For example, if the pole number of a utility pole detected on the Nth time is the same as the pole number of a utility pole detected on the N+1th time, the grouping unit 260 groups the image of the utility pole detected on the Nth time with the image of the utility pole detected on the N+1th time.

[0064] As shown in Figure 20(a), the grouping unit 260 acquires multiple images of the same utility pole and groups the images if the utility pole numbers of the poles included in each image are the same. The selection unit 270 selects the optimal image from the two images and outputs it from the output unit 250. On the other hand, the grouping unit 260 acquires multiple images of different utility poles, as shown in Figure 20(b), and does not perform grouping if the utility pole numbers of the utility poles included in each image are not the same. In this case, the output unit 250 outputs an image for each detected utility pole number. This grouping method allows for the identification and grouping of utility poles one by one. For example, using this method, after identifying utility pole A, vehicle 10 can move and perform grouping when it detects utility pole A again.

[0065] Figure 21 is a diagram illustrating an example of the processing of the selection unit in the second embodiment. The selection unit 270 selects the optimal image from multiple images classified by group. For example, if images are acquired at times t1, t2, t3, and t4, as shown in Figure 21(a), the selection unit 270 will select the image taken at time t3 as the best shot if the four images are grouped together.

[0066] Figure 22 is a diagram illustrating an example of a detection frame in the second embodiment. The detection frame is the image region that includes the utility pole recognized by the recognition unit 420. The detection frame is defined by a determination area at the top of the detection frame and a vertical line. The determination area is, for example, the area that occupies 10% in the vertical (Y) direction. The vertical line is, for example, set at the 20% position from the left edge in the horizontal (X) direction.

[0067] Figure 23 is a diagram illustrating an example of the conditions for selecting an image by the selection unit 270 in the second embodiment. The selection unit 270 selects an image in which the center of the utility pole region, including the utility pole extracted by the extraction unit 210, is close to a predetermined position in the lateral direction of the captured image, as shown in condition (1) of Figure 23. The predetermined position is the set position (X direction position) of the vertical line shown in Figure 22. If the utility pole region includes a vertical line, the selection unit 270 selects an captured image that includes the said utility pole region. As shown in condition (2) of Figure 23, the selection unit 270 selects an image in which the upper end of the utility pole region extracted by the extraction unit 210 is included within a predetermined range below the upper end of the captured image. The predetermined range corresponds to the determination region shown in Figure 22. If the utility pole region is not in contact with the determination region, the selection unit 270 selects an captured image that includes the utility pole region.

[0068] Figure 24 is a flowchart showing an example of grouping processing in the second embodiment. First, the extraction unit 210 detects nesting by using a machine learning model (AI model) determined by the recognition unit 420 (step S500). Next, the grouping unit 260 confirms the grouping method (step S502). The grouping method is set in advance, for example, based on user operations.

[0069] If the grouping method is a period of time, the grouping unit 260 determines whether a predetermined period has elapsed since the previous nest detection and the current nest detection (step S504). If the predetermined period has elapsed since the previous nest detection and the current nest detection (step S504: YES), the grouping unit 260 updates the grouping information indicating the group containing the captured image in which the nest was detected (step S506), and terminates this flowchart. If the next nest is detected, the grouping unit 260 adds the captured image to the grouping information indicating the new group. The grouping unit 260 terminates this flowchart if a predetermined period has not elapsed since the previous nest detection and the current nest detection (step S504: NO). When a nest is detected within the predetermined period, the grouping unit 260 adds the new image to the existing grouping information.

[0070] If the grouping method is location, the grouping unit 260 determines whether a predetermined distance has been moved from the previous nest detection and the current nest detection (step S508). If the grouping unit 260 has moved a predetermined distance from the previous nest detection and the current nest detection (step S508: YES), it updates the grouping information indicating the group containing the captured image in which the nest was detected (step S510) and terminates this flowchart. If a nest is detected again, the grouping unit 260 adds the captured image to the grouping information indicating the new group. The grouping unit 260 terminates this flowchart if it has not moved a predetermined distance from the previous nest detection and the current nest detection (step S508: NO). If a nest is detected again before the grouping unit 260 moves the predetermined distance, the new captured image is added to the existing grouping information.

[0071] If the grouping method is the utility pole number, the grouping unit 260 determines whether the utility pole number has changed in the previous nest detection and the current nest detection (step S512). If the utility pole number has changed in the previous nest detection and the current nest detection (step S512: YES), the grouping unit 260 updates the grouping information indicating the group containing the captured image in which the nest was detected (step S514), and terminates this flowchart. If the next nest is detected, the grouping unit 260 adds the captured image to the grouping information indicating the new group. The grouping unit 260 terminates this flowchart if the pole number has not changed in the previous nest detection and the current nest detection (step S514: NO). If the pole number is the same when a nest is detected again, the grouping unit 260 adds the new image to the existing grouping information.

[0072] Figure 25 is a flowchart showing an example of the selection process in the second embodiment. First, the selection unit 270 detects the captured images included in the grouping information (step S600) and determines the optimal conditions for the vertical line in the utility pole area (step S602). The optimal conditions include: (1) the vertical line is at a certain distance from the left edge of the image captured by the front camera; (2) the vertical line is in the center of the images captured by the side camera and the top camera; and (3) the vertical line is at a certain distance from the left edge of the image captured by the rear camera.

[0073] The selection unit 270 determines whether the utility pole region in the captured image satisfies the optimal conditions (step S604). If the utility pole region does not touch the vertical line in the optimal conditions, the selection unit 270 terminates this flowchart (step S604: NO). If the utility pole region touches the vertical line in the optimal conditions and satisfies the optimal conditions (step S604: NO), the selection unit 270 determines whether the number of facilities is greater than the number of captured images to be grouped (step S606). If the number of facilities is greater than the number of captured images to be grouped (step S606: YES), the selection unit 270 updates the grouping target (step S608); otherwise, the flowchart terminates (step S606: NO).

[0074] Figure 26 is a flowchart showing an example of the selection process in the second embodiment. First, the selection unit 270 detects the captured image included in the grouping information (step S700) and determines whether the utility pole area is in contact with the determination area at the top of the detection frame (step S702). If the utility pole area is in contact with the determination area, the selection unit 270 terminates this flowchart (step S704: NO). If the utility pole area is not in contact with the determination area, the selection unit 270 updates the grouping target to include it in the grouping (step S704: YES, step S706).

[0075] As described above, according to the structure identification system 1 of the second embodiment, multiple images captured by the imaging unit 100 can be classified into groups, and the multiple images and the structure numbers identified by the identification unit 230 can be output for each group. As a result, with the structure identification system 1, it is only necessary to inspect structures for each group, and the workload when the same structure is detected can be reduced.

[0076] Although various embodiments and modifications have been described, these are merely examples and are not limited to these. For example, one embodiment or modification, or a part of one embodiment or modification, may be combined with one or more other embodiments or modifications to realize one aspect of the present invention.

[0077] In this embodiment, programs for executing each process of the structure identification device 200 and the server device 400 may be recorded on a computer-readable recording medium, and the programs recorded on the recording medium may be loaded into a computer system and executed to perform the various processes related to the structure identification device 200 and the server device 400 as described above.

[0078] Furthermore, the term "computer system" as used herein may include hardware such as the operating system and peripheral devices. Additionally, if a WWW system is being used, "computer system" shall also include the homepage provisioning environment (or display environment). Moreover, "computer-readable recording media" refers to storage devices such as flexible disks, magneto-optical disks, ROMs, writable non-volatile memory such as flash memory, portable media such as CD-ROMs, and hard disks built into computer systems.

[0079] Furthermore, "computer-readable recording media" refers to volatile memory (for example, DRAM (Dynamic)) within a computer system that acts as a server or client when a program is transmitted via a network such as the internet or a communication line such as a telephone line. This includes devices that retain programs for a certain period of time, such as Random Access Memory. Furthermore, the above-mentioned program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium.

[0080] Here, the "transmission medium" used to transmit the program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the program described above may be intended to implement only a part of the functions mentioned above. Moreover, it may be a so-called differential file (differential program) that can implement the aforementioned functions in combination with a program already recorded in the computer system.

[0081] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and may include designs that do not depart from the spirit of the invention. [Explanation of symbols]

[0082] 1. Structural Identification System 10 vehicles 200 Structure identification device 100 Imaging Unit 110 Timing section 200 Structure identification device 210 Extraction part 220 Position acquisition part 222 Timing section 230 Specific section 240 Classmates 242 Detection Unit 250 Output section 300 terminal devices 400 Server Devices 410 Liaison Department 420 Recognition part 422 Image Recognition Engine 4221 Storage section 4222 Session Management

Claims

1. The imaging unit mounted on the vehicle, An extraction unit extracts a structural region including a structure from the image captured by the imaging unit, A position acquisition unit that acquires position information of the vehicle or imaging unit at the time the aforementioned image is captured, An identification unit identifies a structure number that identifies a structure included in the structure region extracted by the extraction unit based on the position information acquired by the position acquisition unit, A grouping unit that classifies multiple images captured by the imaging unit into groups, An output unit that outputs the plurality of images and the structure number identified by the identification unit for each group, A structure identification system equipped with [specific features / features].

2. The structure identification system according to claim 1, wherein the grouping unit classifies multiple images included within a predetermined time range into the same group.

3. The structure identification system according to claim 1, wherein the grouping unit classifies a plurality of images included within a predetermined position range into the same group.

4. The structure identification system according to claim 1, wherein the grouping unit classifies a plurality of images into groups of the same structure number identified by the identification unit.

5. A structure identification system according to any one of claims 1 to 4, comprising a selection unit that selects the optimal image from a plurality of images classified according to the aforementioned group.

6. The structure identification system according to claim 5, wherein the selection unit selects an image in which the center of the structure region extracted by the extraction unit is close to a predetermined position in the lateral direction of the image.

7. The structure identification system according to claim 5, wherein the selection unit selects an image in which the upper end of the structure region extracted by the extraction unit is included in a predetermined range below the upper end of the image.

8. Computers The process involves extracting a structural region, including structures, from an image captured by an imaging unit mounted on the vehicle, and A step of acquiring the position information of the vehicle or imaging unit at the time the aforementioned image was captured, The steps include identifying a structure number that identifies a structure included in the structure region extracted based on the acquired location information, The steps include classifying multiple images captured by the imaging unit into groups, The steps include outputting the aforementioned multiple images and identified structure numbers for each group, A method for identifying structures, including the method described above.

9. On the computer, The process involves extracting a structural region, including structures, from an image captured by an imaging unit mounted on the vehicle, and A step of acquiring the position information of the vehicle or imaging unit at the time the aforementioned image was captured, The steps include identifying a structure number that identifies a structure included in the structure region extracted based on the acquired location information, The steps include classifying multiple images captured by the imaging unit into groups, The steps include outputting the aforementioned multiple images and identified structure numbers for each group, A program that executes something.

Images