Structure identification system, structure identification method, and program

Abstract

Easily identify the identity of a structure. [Solution] One aspect of the present invention is a structure identification system comprising an imaging unit mounted on a vehicle, an extraction unit that extracts a structure area including a structure from an image captured by the imaging unit, a position acquisition unit that acquires position information of the vehicle or imaging unit at the time the image was captured, and an identification unit that identifies a structure number that identifies a structure included in the structure area extracted by the extraction unit based on the position information acquired by the position acquisition unit.

Description

[Technical Field]

[0001] The present invention relates to a structure identification system, a structure identification method, and a program. [Background technology]

[0002] Conventionally, in order to inspect infrastructure facilities such as power grids and communication networks, the structures of the infrastructure facilities are photographed by a camera device and the inspection is performed using video image data. As this type of technology, for example, the technologies described in Patent Documents 1 to 3 are known.

[0003] The identical structure identification device in Patent Document 1 is an identical structure identification device that extracts images that show the same structure from a group of multiple captured images taken at a fixed distance by multiple cameras. This identical structure identification device includes an identical structure determination unit that estimates a reference image in which the same structure appears in the group of multiple captured images based on the difference in the timing of capturing the structure, taking into account the fixed distance and the capturing directions of each of the multiple cameras, or the distance from the capturing position of each of the multiple cameras to the structure, and determines that structures that appear in captured images within a predetermined range from the reference image are the same structure.

[0004] The identical structure identification device described in Patent Document 2 is an identical structure identification device that extracts images that show the same structure from a group of photographed images taken at a fixed distance. This identical structure identification device detects a target structure from the group of photographed images and determines whether the structure shown in consecutive photographed images is the same structure based on changes in the detected position of the structure in the photographed images between consecutive photographed images. Furthermore, the identical structure identification device determines that the structure shown in consecutive photographed images is the same structure if the detected position of the structure moves in the same direction as the movement of the scenery in the consecutive photographed images.

[0005] The structure identification device described in Patent Document 3 is a structure identification device that detects a target structure from a plurality of captured image groups captured by each of a plurality of cameras installed in a vehicle. This structure identification device detects a structure from each of the plurality of captured image groups, extracts captured images in which the same structure appears in each of the plurality of captured image groups, determines, based on the shooting time and shooting direction, captured images in which the same structure appears among the plurality of captured image groups, and arranges and displays the plurality of captured images in which the same structure appears for each camera.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] The technologies described in Patent Documents 1-3 above extract captured images in which the same structure appears from a plurality of captured image groups captured at regular intervals by a plurality of cameras. However, even if captured images in which the same structure appears can be extracted, if identification information such as the number of the same structure cannot be specified, the labor for confirming the identification information from the position information when the captured image was taken increases, and there is a problem that the work cost increases.

[0008] The present invention has been made in view of the above problems, and an object thereof is to provide a structure identification system, a structure identification method, and a program that can easily specify identification information of a structure.

Means for Solving the Problems

[0009] (1) One aspect of the present invention is a method for detecting a vehicle's speed by an imaging unit mounted on a vehicle and detecting a speed from an image captured by the imaging unit. structure an extracting unit that extracts the vehicle or the image capturing unit when the captured image is captured; a position acquiring unit that acquires position information of the vehicle or the image capturing unit when the captured image is captured; and structure an identification unit that identifies a structure number that identifies the structure; the extraction unit sets a search area based on position information of the vehicle or the imaging unit and an imaging range including a traveling direction of the vehicle, and extracts the structure included in the set search area. , The distance from the position indicated by the position information of the vehicle or the imaging unit to the center of the search area is adjusted according to the size of the extracted structure. , a structure identification system.

[0010] (2) In one aspect of the present invention, in the structure identification system, when a plurality of pieces of position information are detected while the vehicle is stopped or traveling at a low speed, the position acquisition unit determines a representative position of the plurality of pieces of position information and acquires a trajectory of the position information based on the determined representative position, and the identification unit: The orbit The structure number may be identified based on the

[0013] ( 3 ) One aspect of the present invention is the structure identification system described above, of the image captured by the imaging unit Time information and The location information acquired by the location acquisition unit The sound source may further include a synchronization unit that synchronizes the time information with the time at which the detection unit detects the specific sound.

[0014] ( 4 In one aspect of the present invention, a computer acquires a vehicle image from an image captured by an imaging unit mounted on the vehicle. structure and extracting The computer acquiring position information of the vehicle or the imaging unit at the time the captured image was captured; The computer extracted based on the acquired location information structure identifying a structure number that identifies the Execute , The computer sets a search area based on the position information of the vehicle or the imaging unit and an imaging range including the traveling direction of the vehicle, and extracts the structure included in the set search area. , The computer adjusts the distance from the position indicated by the position information of the vehicle or the imaging unit to the center of the search area according to the size of the extracted structure. , a structure identification method.

[0015] ( 5 One aspect of the present invention is to provide a computer that can acquire a vehicle image from an image captured by an imaging unit mounted on the vehicle. structurea step of acquiring position information of the vehicle or the imaging unit at the time when the captured image was captured; and a step of extracting the extracted position information based on the acquired position information. structure and identifying a structure number that identifies the structure; A search area is set based on the position information of the vehicle or the image capturing unit and an image capturing range including the traveling direction of the vehicle, and the structure included in the set search area is extracted. , The distance from the position indicated by the position information of the vehicle or the imaging unit to the center of the search area is adjusted according to the size of the extracted structure. , program. [Effects of the Invention]

[0016] According to one aspect of the present invention, identification information of a structure can be easily identified. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a perspective view showing an example of a vehicle equipped with a structure identification system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a functional configuration of a structure identification system 1 according to a first embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of a recognition unit 420 according to the first embodiment. [Figure 4] FIG. 3 is a diagram showing an example of a screen displayed on a terminal device 300 in the facility identification service according to the first embodiment. [Figure 5] FIG. 4 is a diagram for explaining a time difference in the first embodiment. [Figure 6] FIG. 3 is a diagram illustrating a time synchronization process according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of a display screen for time synchronization processing in the first embodiment. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure for time synchronization processing according to the first embodiment. [Figure 9] 10 is a flowchart showing another example of the procedure for acquiring the time difference between videos in the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining the effect of the time synchronization process in the first embodiment. [Figure 11]FIG. 3 is a diagram showing an example of processing for stabilizing the trajectory of the vehicle 10 (smoothing the position information) in the first embodiment. [Figure 12] FIG. 4 is a diagram for explaining the process of setting a search area for identifying a utility pole number in the first embodiment. [Figure 13] 10 is a diagram for explaining the relationship between the proportion of the search area in the captured image and the distance Ym from the position of the vehicle 10 to the center of the search area in the first embodiment. FIG. [Figure 14] FIG. 4 is a diagram for explaining a process for identifying a utility pole number in the first embodiment. [Figure 15] 5 is a flowchart illustrating a process of smoothing position information in the first embodiment. [Figure 16] 10 is a flowchart showing the procedure of a process for identifying a utility pole number in the first embodiment. [Figure 17] FIG. 10 is a block diagram showing an example of the functional configuration of a structure identification device 200A according to a second embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of grouping in the second embodiment. [Figure 19] FIG. 10 is a diagram for explaining another example of grouping in the second embodiment. [Figure 20] FIG. 10 is a diagram for explaining another example of grouping in the second embodiment. [Figure 21] FIG. 10 is a diagram illustrating an example of processing by a selection unit in the second embodiment. [Figure 22] FIG. 10 is a diagram illustrating an example of a detection frame according to the second embodiment. [Figure 23] FIG. 10 is a diagram for explaining an example of a condition for selecting a captured image by a selection unit 270 in the second embodiment. [Figure 24] 10 is a flowchart showing an example of grouping processing according to the second embodiment. [Figure 25] 10 is a flowchart showing an example of a selection process according to the second embodiment. [Figure 26] 10 is a flowchart showing an example of a selection process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] A structure identification system, a structure identification method, and a program to which the present invention is applied will be described below with reference to the drawings.

[0019] First Embodiment FIG. 1 is a perspective view showing an example of a vehicle equipped with a structure identification system according to the first embodiment. The structure identification system 1 of the embodiment includes an imaging unit 100 and a position acquisition unit 220 mounted on a vehicle 10. The imaging unit 100 is installed, for example, on top of the vehicle 10 so as to capture an image of a range including the traveling direction of the vehicle 10. The pan angle of the imaging unit 100 may be adjusted, for example, so as to capture an image of structures in a range diagonally forward from the traveling direction of the vehicle 10. The tilt angle of the imaging unit 100 may also be adjusted so as to capture an image of a structure that exists from the ground up, such as a utility pole.

[0020] In the embodiment, the structure is, for example, a utility pole installed at the side of the road on which the vehicle 10 travels, or equipment installed on the utility pole. The structure identification system 1 can support a service in which, for example, the vehicle 10 travels while taking photographs, identifies the utility pole number of the utility pole included in the captured image, and transmits the captured image including the identified utility pole. The structure identification system 1 can also support an equipment identification service that uses the captured image to recognize the presence of nests on utility poles, natural objects such as vines attached to utility poles, and deterioration of utility poles and equipment attached to utility poles, and transmits the recognition results. Furthermore, the structure in the embodiment is not limited to a utility pole, and may be a structure such as transportation infrastructure equipment such as roads and traffic lights, or building equipment.

[0021] <Functional configuration of structure identification system 1> FIG. 2 is a block diagram showing an example of the functional configuration of the structure identification system 1 according to the first embodiment. The structure identification system 1 is configured, for example, by connecting a terminal device 300, a server device 400, and a utility pole information database device 500 to a structure identification device 200 via a communication network NW. The structure identification device 200, the terminal device 300, and the server device 400 are equipped with communication interfaces such as NICs (Network Interface Cards) and wireless communication modules (not shown in FIG. 1). The communication network NW includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a cellular network, etc.

[0022] The imaging unit 100, for example, has a function of generating a captured image, and also includes a clock unit 110. The clock unit 110 generates time information. The imaging unit 100 outputs the captured image with the time information added to the structure identification device 200.

[0023] The structure identification device 200 is an information processing device that performs, for example, communication processing and various application processing. The structure identification device 200 includes, for example, an extraction unit 210, a position acquisition unit 220, an identification unit 230, a synchronization unit 240, and an output unit 250. The extraction unit 210, the position acquisition unit 220, the identification unit 230, the synchronization unit 240, and the output unit 250 are realized by a processor such as a CPU (Central Processing Unit) executing a program stored in a program memory. Furthermore, some or all of these functional units may be realized by hardware such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array), or may be realized by a combination of software and hardware.

[0024] The extraction unit 210 extracts a structure area including a structure from the image captured by the imaging unit 100. The position acquisition unit 220 acquires position information of the vehicle 10 or the imaging unit 100 at the time the captured image was captured. The position acquisition unit 220 also includes a clock unit 222. The clock unit 222 generates time information. The position acquisition unit 220 acquires the position information with the time information added.

[0025] The identification unit 230 identifies a structure number that identifies a structure included in the structure area extracted by the extraction unit 210 based on the position information acquired by the position acquisition unit 220 . In the following embodiment, a description will be given of identifying a utility pole as a structure and specifying the utility pole number as the structure number. The specification unit 230 specifies the utility pole number by referring to the utility pole information database managed by the utility pole information database device 500. The utility pole information database device 500 is an information processing device that associates utility pole position information with utility pole numbers and performs management processes such as adding, deleting, and correcting information in the utility pole information database.

[0026] The synchronization unit 240 includes a detection unit 242 that detects the specific sound. The synchronization unit 240 synchronizes the time information measured by the imaging unit 100 and the time information measured by the position acquisition unit 220 with the time at which the detection unit 242 detects the specific sound. The output unit 250 outputs the utility pole number and the captured image to the terminal device 300. The output unit 250 may output the utility pole number and the recognition result for the utility pole to the terminal device 300.

[0027] The terminal device 300 is, for example, an information processing device such as a personal computer, smartphone, or tablet terminal that is operated by a user who manages a structure. The terminal device 300 acquires the utility pole number and the captured image output by the structure identification device 200, and displays the utility pole number and the captured image on a display device or the like. The terminal device 300 may also acquire the utility pole number and the recognition result for the utility pole and display them on a display device or the like.

[0028] The server device 400 is an information processing device that receives requests to recognize utility poles and provides the recognition results. The server device 400 includes, for example, a collaboration unit 410 and a recognition unit 420. The collaboration unit 410 and the recognition unit 420 are realized by a processor such as a CPU executing a program stored in a program memory.

[0029] The collaboration unit 410 performs, for example, processing to receive a captured image and a recognition request from the structure identification device 200. The collaboration unit 410 may perform predetermined image processing on the captured image acquired from the structure identification device 200 and output the captured image to the recognition unit 420.

[0030] The recognition unit 420 includes, for example, a plurality of image recognition engines 422A, 422B, ..., 422N (N is a natural number). Note that when a plurality of image recognition engines are collectively referred to, they will be simply referred to as image recognition engine 422. Each image recognition engine 422 is a machine learning model (AI model) set corresponding to a recognition target. The recognition target is, for example, a nest, a natural object, or utility pole equipment. The image recognition engine 422 inputs a captured image and outputs information including the presence or absence of the recognition target, the area if the recognition target is present, and the reliability.

[0031] <Recognition processing in structure identification system 1> FIG. 3 is a block diagram showing an example of the recognition unit 420 in the first embodiment. The recognition unit 420 includes, for example, a training image database 430, a training processing unit 432, and an image recognition engine 422.

[0032] The training image database 430 is a database that stores, as training data, captured images and tag information acquired from, for example, the terminal device 300. The training image database 430 stores, as training data, positive example images, for example. A positive example image is, for example, data in which tag information indicating a nest is added to a captured image that includes a nest and a utility pole.

[0033] The learning processing unit 432 outputs the captured images in the learning data to the image recognition engine 422 and obtains a recognition result from the image recognition engine 422. The learning processing unit 432 updates the processing parameters of the image recognition engine 422 so that the image recognition engine 422 outputs tag information added to the captured images. The processing parameters of the image recognition engine 422 are, for example, filters (also called weights or biases) included in the neural network. The processing parameters are stored in the storage unit 4221.

[0034] The image recognition engine 422 includes, for example, a storage unit 4221 and a processing unit 4222. Processing parameters are accumulated as learning results in the storage unit 4221. The processing parameters are updated by the learning processing unit 432. When a captured image is acquired, the processing unit 4222 outputs a recognition result using a recognition model constructed using the processing parameters.

[0035] <Display screen of the facility identification service> FIG. 4 is a diagram showing an example of a screen displayed on the terminal device 300 in the facility identification service according to the first embodiment. For example, the terminal device 300 performs a process of identifying structures from captured images using an application for an equipment identification service, and then displays a screen such as that shown in FIG. 4. The terminal device 300 displays, for example, the travel path of the vehicle 10, a map including the identified utility poles, a list of the pole numbers of the identified utility poles, and captured images of the identified utility poles. On the captured image of the identified utility pole, areas where nesting has been recognized, areas where natural objects have been recognized, and areas where equipment has been recognized are superimposed in association with the utility pole numbers. This allows the user of the terminal device 300 to inspect the utility pole with the utility pole number identified on the travel path of the vehicle 10.

[0036] <Time synchronization processing> The time synchronization process in the structure identification device 200 will be described below. FIG. 5 is a diagram for explaining the time difference in the first embodiment. For example, a plurality of imaging units 100 may be mounted on the vehicle 10, and time information may be generated by the timing unit 110 of each imaging unit 100. The vehicle 10 is also provided with a timing unit 222 of the position acquisition unit 220 (drive recorder). For example, cameras A and C are the timing units 110 of the imaging units 100, and camera B is the timing unit 222 of the position acquisition unit 220. For this reason, if there is a discrepancy in the time information between the plurality of timing units 110, 222, a malfunction may occur in which captured images and position information acquired at the same time are recognized as being captured at different positions. In response to this, the structure identification device 200 detects the specific sound using the detection unit 242, and synchronizes the time information measured by the imaging unit 100 and the time information measured by the position acquisition unit 220 with the time at which the specific sound is detected using the synchronization unit 240.

[0037] FIG. 6 is a diagram illustrating the time synchronization process according to the first embodiment. The detection unit 242 detects audio waveform information that indicates volume changes inside the vehicle 10, as shown in the left diagram, and during time synchronization, normalizes the audio waveform information for several minutes to create audio waveform information with a range of 0 to 1, as shown in the right diagram. The detection unit 242 detects two detection times, t1 and t2, of audio waveform information that corresponds to clapping, for example. The synchronization unit 240 synchronizes the time of the imaging unit 100 and the position acquisition unit 220 with the first of the two detected detection times.

[0038] FIG. 7 is a diagram showing an example of a display screen for the time synchronization process in the first embodiment. Terminal device 300 includes, for example, buttons for displaying videos of the front, directly above, directly to the side, behind, and foreground, a button for reverting the foreground video to the previous video, a button for moving the foreground video to the next video, a button for confirming the time difference between videos, etc. By operating the display screen, the user can synchronize the foreground video to be time-synchronized with the position information corresponding to the foreground video. This allows the structure identification system 1 to synchronize the time information of the timing unit 110 added to the image captured by the imaging unit 100 with the time information of the timing unit 222 added to the position information of the position acquisition unit 220.

[0039] FIG. 8 is a flowchart illustrating an example of a processing procedure of the time synchronization processing according to the first embodiment. First, the synchronization unit 240 acquires audio waveform information for a predetermined time from a video captured by the imaging unit 100 (step S100) and normalizes the audio waveform (step S102). Next, the synchronization unit 240 detects two peaks in the audio waveform within a predetermined period (step S104) and determines whether a time synchronization point has been detected (step S106). If the synchronization unit 240 detects the timing for time synchronization, it performs time synchronization at the time of the maximum value of the audio waveform (steps S106: YES, S108). If the synchronization unit 240 cannot detect the timing for time synchronization, it does not perform time synchronization (step S106: NO).

[0040] FIG. 9 is a flowchart showing another example of the procedure for acquiring the time difference between videos in the first embodiment. Synchronization unit 240 acquires a time synchronization point from a determination video, which is a video for determining time synchronization (step S200), and acquires a time synchronization point from a foreground image (step S202). For example, in response to selecting a still image included in the video on the display screen of Fig. 7, synchronization unit 240 acquires the time of the selected still image as the time synchronization point. Next, synchronization unit 240 calculates the deviation between the point in the determination video and the point in the foreground video as the time deviation between the videos (step S204).

[0041] FIG. 10 is a diagram for explaining the effect of the time synchronization process in the first embodiment. For example, the structure identification system 1 captures a video of a utility pole facility captured by a front camera at point 1, a video of the utility pole facility captured by a front camera at point 2, a video of the utility pole facility captured by a rear camera at point 3, and a video of the utility pole facility captured by a rear camera at point 4. In this case, since the time at which the utility pole is imaged varies by several centimeters (several seconds) depending on the installation location of the camera, it is possible to synchronize the time between the videos captured by the multiple cameras. This allows the structure identification system 1 to accurately match the utility pole number with the captured image when an object such as a nest or utility pole facility is discovered.

[0042] <Identifying utility pole numbers> The process of identifying the utility pole number based on the position information in the structure identification device 200 will be described below. FIG. 11 is a diagram showing an example of processing for stabilizing the trajectory of the vehicle 10 (smoothing the position information) in the first embodiment. As described above, even if the recognition unit 420 recognizes a utility pole or nest, etc., included in the captured image, if the utility pole number cannot be identified, the effort required to identify the utility pole number from the position information increases, resulting in increased work costs. In contrast, the structure identification system 1 in the embodiment performs processing to calculate the trajectory of the vehicle 10 and stabilize the true heading when the position information of the vehicle 10 changes.

[0043] When multiple pieces of position information are detected while the vehicle 10 is stopped or traveling at a low speed (left diagram), the position acquisition unit 220 determines a representative position of the multiple pieces of position information (center diagram) and acquires the trajectory of the position information based on the determined representative position (right diagram). Specifically, if the vehicle 10 stops or travels at a low speed while moving from detection point (t) to detection point (t+a), the position information may fluctuate, for example, every second, such as A, B, C, D, E, and F, even though the trajectory of the vehicle 10 is moving from detection point (t) to detection point (t+a). In response to this, the position acquisition unit 220 determines representative points for the positions A, B, C, D, E, and F. At this time, the position acquisition unit 220 plots the average positions of the plots A, B, C, D, E, and F of the position information in two dimensions and calculates the average position of the plots A, B, C, D, E, and F. Then, the position acquisition unit 220 acquires the trajectory of the vehicle 10 moving from the detection point (t), to the representative point, and back to the detection point (t+a) in this order. This allows the identification unit 230 to identify the utility pole number based on the trajectory of the acquired position information.

[0044] FIG. 12 is a diagram for explaining the process of setting a search area for identifying a utility pole number in the first embodiment. The extraction unit 210 sets a search area based on the position information of the vehicle 10, or the position information of the structure identification device 200, or the position information of the imaging unit 100, and the imaging range of the imaging unit 100, and extracts structures included in the set search area. At this time, the extraction unit 210 creates a point tilted XO to the left in the traveling direction from the current position information of the vehicle 10 or the installation position of the imaging unit 100, and sets this point as the center position of the search area. The identification unit 230 sets two diagonal lines that intersect at the center position. As a result, the extraction unit 210 creates a rectangle, for example, with X (meters) in the vertical direction and Y (meters) in the traveling direction, and sets the created rectangle as the search area for the utility pole. The search area may be, for example, about 50 (meters) in the vertical direction and about 100 (meters) in the traveling direction.

[0045] FIG. 13 is a diagram for explaining the relationship between the proportion of the search area in the captured image and the distance Ym from the position of the vehicle 10 to the center of the search area in the first embodiment. The extraction unit 210 adjusts the distance from the position information of the vehicle 10 or the imaging unit 100 to the center of the search area depending on the size of the area recognized as the extracted utility pole. For example, the extraction unit 210 may shorten the distance Ym from the position information of the vehicle 10 or the imaging unit 100 to the center of the search area as the size of the area recognized as the extracted utility pole increases. The extraction unit 210 reduces the size of the search area by shortening the distance Ym. Although the physical size of the utility pole captured by the imaging unit 100 is constant, the size of the utility pole area included in the captured image varies depending on the capturing position relative to the utility pole. For this reason, the extraction unit 210 shortens the distance Ym as the size of the utility pole area included in the captured image increases. In other words, the distance Ym is made inversely proportional to the size of the utility pole area.

[0046] The extraction unit 210 may make the distance Ym inversely proportional to the size of the utility pole area included in the captured image, or may make the square of the distance Ym inversely proportional to the size of the utility pole area included in the captured image. Furthermore, the extraction unit 210 may increase the distance from the position information of the vehicle 10 or the imaging unit 100 to the center of the search area as the size of the extracted area recognized as a utility pole becomes smaller. This allows the image of the utility pole included in the captured image to be captured more clearly, thereby increasing the accuracy of identifying the utility pole.

[0047] FIG. 14 is a diagram for explaining the process of identifying the utility pole number in the first embodiment. The identification unit 230 acquires position information of the utility pole closest to the center position of the search area. At this time, the identification unit 230 calculates position information of the utility pole closest to the center position of the search area based on the position information acquired by the position acquisition unit 220, the center position of the search area, and the utility pole position relative to the center position. The identification unit 230 collates the calculated position information of the utility pole closest to the center position of the search area with the utility pole information database, and identifies the utility pole number corresponding to the position information.

[0048] FIG. 15 is a flowchart illustrating the process of smoothing position information in the embodiment. The position acquisition unit 220 resets the retained information of the position information for each NMEA (National Marine Electronics Association) file containing GPS-related position information if the file name is not a consecutive number (step S300).

[0049] Next, the position acquisition unit 220 reads a line (record) included in the NMEA file and determines whether there is previous position information for the position information (line) of interest (step S302). If there is previous position information (step S302: YES), the position acquisition unit 220 divides the elapsed time from the time added to the previous position information by a predetermined interval and calculates the travel distance for each predetermined interval (step S304). If the calculated travel distance indicates that the vehicle has traveled a predetermined distance or more, the position acquisition unit 220 adds this to the distance for travel processing (step S306). The position acquisition unit 220 loops the processes from step S300 to step S306 for the number of NMEA files.

[0050] If there is no previous position information (step S302: NO), the position acquisition unit 220 determines whether the traveled distance exceeds a threshold (step S308). If the traveled distance exceeds the threshold (step S308: YES), the position acquisition unit 220 smoothes the position information by averaging the position information included in the traveled distance (step S310, FIG. 11). The position acquisition unit 220 loops the processes from step S304 to step S310 at predetermined intervals, and loops the processes from step S300 to step S310 as many times as the number of NMEA files. If there is position information that has not been smoothed by the above-mentioned process, the position acquisition unit 220 smooths the position information (step S312).

[0051] FIG. 16 is a flowchart showing the procedure of the process for identifying the utility pole number in the first embodiment. The identification unit 230 acquires the location information at the time of the determination target and the location information at the time immediately preceding the time of the determination target from the NMEA file (step S400). The identification unit 230 determines whether or not the two pieces of location information have been acquired (step S402). If not, the process of this flowchart ends (step S402: NO), and if successful, the process proceeds to step S404 (step S402: YES).

[0052] In step S404, the identification unit 230 identifies the traveling direction of the vehicle 10, and if the traveling direction cannot be identified, calculates the traveling direction from the current position information and the immediately previous position information. The identification unit 230 calculates the distance Ym of the search region from the area ratio of the detection frame to the captured image (step S406). The detection frame is a rectangular region in the captured image that includes the utility pole. Next, the identification unit 230 sets a center position based on the distance Ym, and sets the search region based on the set center position (step S410).

[0053] Next, the identification unit 230 detects utility poles included in the search area, calculates the distance from the center position of the search area, and acquires position information for each utility pole. The identification unit 230 refers to the utility pole information database and identifies the utility pole number corresponding to the position information for each utility pole (step S412). The identification unit 230 identifies the utility pole closest to the current position of the vehicle 10 based on the position information in the utility pole information database for the identified utility pole number (step S414).

[0054] As described above, the structure identification system 1 in the first embodiment can smooth the position information when the vehicle 10 is stopped or traveling at a low speed, thereby stabilizing the trajectory of the vehicle 10. Furthermore, the structure identification system 1 can set a search area based on the position information of the vehicle 10 and identify the pole numbers of utility poles included in the search area. As a result, the structure identification system 1 can easily identify the pole numbers (identification information) of recognized objects such as utility poles, nests, natural objects, and equipment, thereby reducing the cost of inspection work.

[0055] <Second embodiment> The second embodiment will be described below. <Functional configuration of structure identification system 1> 17 is a block diagram showing an example of the functional configuration of a structure identification device 200A according to the second embodiment. Note that the same components as those in the above-described embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0056] In the structure identification system 1, if the pole number of a utility pole identified by the identification unit 230 is included in multiple captured images, the burden of checking it on the user increases. In response to this, the structure identification system 1 in the second embodiment groups the captured images. Furthermore, the invention according to claim 1 of the present application in the second embodiment selects an image that is optimal for equipment inspection from the multiple grouped images.

[0057] The structure identification device 200A differs from the structure identification device 200 in the first embodiment in that it includes a grouping unit 260 and a selection unit 270 in addition to an extraction unit 210, a position acquisition unit 220, an identification unit 230, a synchronization unit 240, and an output unit 250. The grouping section 260 classifies a plurality of images captured by the imaging section 100 into groups. The selection unit 270 selects the most suitable image from the plurality of images classified into groups.

[0058] <Grouping method according to period> FIG. 18 is a diagram illustrating an example of grouping in the second embodiment. The grouping unit 260 may classify multiple images included in a predetermined time range into the same group. For example, grouping is performed if the time from the Nth detection to the N+1th detection is within a specified time. As shown in FIG. 18(a), if the same utility pole number is detected twice within one second, the grouping unit 260 classifies the two captured images into the same group. The selection unit 270 can select the most suitable captured image from the two captured images and output it from the output unit 250. 18(b), even if the same utility pole number is detected once in one second and then detected again one second later, the grouping unit 260 does not classify the two captured images into the same group. In this case, the output unit 250 outputs one captured image every second. According to this grouping method, since it does not depend on the position information of the utility poles, it is possible to group the captured images even if the position information cannot be obtained.

[0059] <Location-based grouping method> FIG. 19 is a diagram illustrating another example of grouping in the second embodiment. The grouping unit 260 may classify multiple images that fall within a predetermined position range into the same group. For example, the grouping unit 260 calculates the distance from the position where the utility pole was detected the Nth time to the position where the utility pole was detected the (N+1)th time, and groups two captured images if they fall within the predetermined position range.

[0060] 19(a), if the predetermined position range is 30 meters and the distance from the Nth utility pole detection position to the (N+1)th utility pole detection position is within 30 meters, the grouping unit 260 performs grouping because the predetermined position range exists. The selection unit 270 can select the most suitable captured image from the two captured images and cause the output unit 250 to output the image. 19(b), if the predetermined position range is 20 meters and the distance from the Nth utility pole detection position to the (N+1)th utility pole detection position exceeds 20 meters, the grouping unit 260 does not perform grouping because the detected positions are not within the predetermined position range. In this case, the output unit 250 outputs a captured image for each detected utility pole. According to this grouping method, it is possible to perform grouping on captured images acquired when the vehicle 10 is stopped. For example, according to this grouping method, it is possible to perform grouping beyond the above-mentioned predetermined time range as long as the vehicle 10 is stopped.

[0061] <Grouping method according to utility pole number> FIG. 20 is a diagram illustrating another example of grouping in the second embodiment. The grouping unit 260 may classify a plurality of images into groups of the same structure number identified by the identification unit 230. For example, when the utility pole number of the utility pole detected the Nth time and the utility pole number of the utility pole detected the N+1th time are the same, the grouping unit 260 groups the captured image of the utility pole detected the Nth time and the captured image of the utility pole detected the N+1th time.

[0062] 20(a), the grouping unit 260 acquires multiple captured images of the same utility pole, and performs grouping when the utility pole numbers of the utility poles included in the captured images are the same. The selection unit 270 selects the most suitable captured image from the two captured images, and causes the output unit 250 to output the selected image. On the other hand, the grouping unit 260 does not perform grouping when it acquires multiple captured images of different utility poles as shown in Fig. 20(b) and the utility pole numbers of the utility poles included in the captured images are not the same. In this case, the output unit 250 outputs a captured image for each detected utility pole number. This grouping method allows utility poles to be identified one by one and grouped. For example, after utility pole A is identified, the vehicle 10 moves and utility pole A is detected again, and then the grouping can be performed.

[0063] FIG. 21 is a diagram illustrating an example of processing performed by the selection unit in the second embodiment. The selection unit 270 selects the most suitable image from the plurality of images classified into groups. For example, in the case where captured images are acquired at times t1, t2, t3, and t4 as shown in FIG. 21(a), if four captured images are grouped, the selection unit 270 will adopt the captured image at time t3 as the best shot.

[0064] FIG. 22 is a diagram illustrating an example of a detection frame according to the second embodiment. The detection frame is an image region that includes the utility pole recognized by the recognition unit 420. A determination region and a vertical line are set in the detection frame at the top of the detection frame. The determination region is, for example, an area that occupies 10% in the vertical (Y) direction. The vertical line is, for example, set at a position 20% from the left end in the horizontal (X) direction.

[0065] FIG. 23 is a diagram for explaining an example of a condition for selecting a captured image by the selection unit 270 in the second embodiment. As shown in condition (1) of Fig. 23, the selection unit 270 selects an image in which the center of a utility pole region including a utility pole extracted by the extraction unit 210 is close to a predetermined position in the horizontal direction in the captured image. The predetermined position is the set position (X direction position) of the vertical line shown in Fig. 22. When the utility pole region includes a vertical line, the selection unit 270 selects a captured image including the utility pole region. As shown in condition (2) of Fig. 23, the selection unit 270 selects an image in which the upper end of the utility pole area extracted by the extraction unit 210 is included in a predetermined range downward from the upper end of the captured image. The predetermined range corresponds to the determination area shown in Fig. 22. If the utility pole area does not contact the determination area, the selection unit 270 selects a captured image that includes the utility pole area.

[0066] FIG. 24 is a flowchart showing an example of the grouping process according to the second embodiment. First, the extraction unit 210 detects nesting by the recognition unit 420 using a machine learning model (AI model) (step S500). Next, the grouping unit 260 checks the grouping method (step S502). The grouping method is set in advance based on, for example, a user operation.

[0067] If the grouping method is period, the grouping unit 260 determines whether a predetermined period has passed since the previous nesting detection and the current nesting detection (step S504). If the predetermined period has passed since the previous nesting detection and the current nesting detection (step S504: YES), the grouping unit 260 updates the grouping information indicating the group including the captured image in which nesting was detected (step S506) and ends this flowchart. If nesting is detected next, the grouping unit 260 adds the captured image to the grouping information indicating the new group. If a predetermined period has not elapsed since the previous nest detection and the current nest detection (step S504: NO), the grouping unit 260 ends this flowchart. When a nest is detected within the predetermined period, the grouping unit 260 adds a new captured image to the existing grouping information.

[0068] If the grouping method is position, the grouping unit 260 determines whether a predetermined distance has been moved from the previous nest detection and the current nest detection (step S508). If a predetermined distance has been moved from the previous nest detection and the current nest detection (step S508: YES), the grouping unit 260 updates the grouping information indicating the group including the captured image in which nesting was detected (step S510) and ends this flowchart. If nesting is detected next, the grouping unit 260 adds the captured image to the grouping information indicating the new group. If the predetermined distance has not been moved since the previous nest detection and the current nest detection (step S508: NO), the grouping unit 260 ends this flowchart. When a nest is detected again before the predetermined distance has been moved, the grouping unit 260 adds a new captured image to the existing grouping information.

[0069] If the grouping method is the utility pole number, the grouping unit 260 determines whether the utility pole number has changed between the previous nest detection and the current nest detection (step S512). If the utility pole number has changed between the previous nest detection and the current nest detection (step S512: YES), the grouping unit 260 updates the grouping information indicating the group including the captured image in which nesting was detected (step S514), and ends this flowchart. If nesting is detected next, the grouping unit 260 adds the captured image to the grouping information indicating the new group. If the utility pole number has not changed between the previous nest detection and the current nest detection (step S514: NO), the grouping unit 260 ends this flowchart. If the utility pole number is the same when a nest is detected again, the grouping unit 260 adds a new captured image to the existing grouping information.

[0070] FIG. 25 is a flowchart showing an example of the selection process according to the second embodiment. First, the selection unit 270 detects the captured images included in the grouping information (step S600) and determines the optimum conditions for the vertical line in the utility pole area (step S602). The optimum conditions include (1) a vertical line at a certain percentage from the left edge of the image captured by the front camera, (2) a vertical line at the center of the images captured by the side camera and the camera directly above, and (3) a vertical line at a certain percentage from the left edge of the image captured by the rear camera.

[0071] The selection unit 270 determines whether the utility pole region in the captured image satisfies the optimum conditions (step S604). If the utility pole region does not touch the vertical line in the optimum conditions, the selection unit 270 ends this flowchart (step S604: NO). If the utility pole region touches the vertical line in the optimum conditions and satisfies the optimum conditions (step S604: NO), the selection unit 270 determines whether the number of facilities is greater than the number of facilities in the captured images to be grouped (step S606). If the number of facilities is greater than the number of facilities in the captured images to be grouped (step S606: YES), the selection unit 270 updates the grouping targets (step S608), and if not, ends this flowchart (step S606: NO).

[0072] FIG. 26 is a flowchart showing an example of the selection process according to the second embodiment. First, the selection unit 270 detects the captured image included in the grouping information (step S700), and determines whether the utility pole region is adjacent to the judgment region at the top of the detection frame (step S702). If the utility pole region is adjacent to the judgment region, the selection unit 270 ends this flowchart (step S704: NO), and if the utility pole region is not adjacent to the judgment region, the selection unit 270 updates the grouping target to include the utility pole region in the grouping (step S704: YES, step S706).

[0073] As described above, the structure identification system 1 according to the second embodiment can classify multiple images captured by the imaging unit 100 into groups, and output the multiple images and the structure numbers identified by the identification unit 230 for each group. As a result, the structure identification system 1 can inspect structures by group, reducing the workload when identical structures are detected.

[0074] Although each embodiment and variant has been described, these are merely examples and are not intended to be limiting. For example, one aspect of the present invention may be realized by combining any one of the embodiments or variants, or a part of each embodiment or variant, with one or more other embodiments or one or more other variants.

[0075] The various processes described above relating to the structure identification device 200 and the server device 400 may be performed by recording a program for executing each process of the structure identification device 200 and the server device 400 in this embodiment on a computer-readable recording medium, and then loading and executing the program recorded on the recording medium into a computer system.

[0076] Note that the term "computer system" here may include hardware such as the OS and peripheral devices. Furthermore, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, "computer-readable recording media" refers to storage devices such as flexible disks, magneto-optical disks, ROMs, and writable non-volatile memory such as flash memory, portable media such as CD-ROMs, and hard disks built into computer systems.

[0077] Furthermore, "computer-readable recording medium" refers to the volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. This also includes devices that hold a program for a certain period of time, such as a random access memory (Random Access Memory). The program may also be transmitted from a computer system that stores the program in a storage device to another computer system via a transmission medium or by transmission waves in the transmission medium.

[0078] Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in a computer system.

[0079] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and the present invention also includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0080] 1. Structure Identification System 10 vehicles 200 Structure identification device 100 Imaging unit 110 Timing section 200 Structure identification device 210 Extraction part 220 Position acquisition part 222 Timing section 230 Specific section 240 Synchronization Unit 242 Detector 250 Output section 300 Terminal Equipment 400 Server device 410 Collaboration Department 420 Recognition part 422 Image Recognition Engine 4221 Storage section 4222 Processing section

Claims

1. an imaging unit mounted on the vehicle; an extraction unit that extracts a structure from the captured image captured by the imaging unit; a position acquisition unit that acquires position information of the vehicle or the imaging unit at the time the captured image was captured; an identification unit that identifies a structure number that identifies the structure extracted by the extraction unit based on the location information acquired by the location acquisition unit, The extraction unit setting a search area based on position information of the vehicle or the imaging unit and an imaging range including the traveling direction of the vehicle, and extracting the structure included in the set search area; adjusting the distance from the position indicated by the position information of the vehicle or the imaging unit to the center of the search area according to the size of the extracted structure; Structure Identification System.

2. when a plurality of pieces of position information are detected while the vehicle is stopped or traveling at a low speed, the position acquisition unit determines a representative position of the plurality of pieces of position information, and acquires a trajectory of the position information based on the determined representative position; the identification unit identifies the structure number based on the trajectory. The structure identification system according to claim 1 .

3. a detection unit that detects a specific sound; a synchronization unit that synchronizes time information of the captured image captured by the imaging unit and time information of the position information acquired by the position acquisition unit with the time when the detection unit detects the specific sound; The structure identification system according to claim 1 or 2, comprising:

4. A step in which a computer extracts a structure from an image captured by an imaging unit mounted on a vehicle; a step in which the computer acquires position information of the vehicle or the imaging unit at the time the captured image was captured; The computer executes a step of identifying a structure number that identifies the extracted structure based on the acquired location information; the computer sets a search area based on position information of the vehicle or the image capture unit and an image capture range including the traveling direction of the vehicle, and extracts the structure included in the set search area; the computer adjusts the distance from the position indicated by the position information of the vehicle or the imaging unit to the center of the search area according to the size of the extracted structure. Structure identification method.

5. On the computer, extracting a structure from an image captured by an imaging unit mounted on the vehicle; acquiring position information of the vehicle or the imaging unit at the time the captured image was captured; specifying a structure number that identifies the extracted structure based on the acquired location information; setting a search area based on position information of the vehicle or the imaging unit and an imaging range including the traveling direction of the vehicle, and extracting the structure included in the set search area; adjusting the distance from the position indicated by the position information of the vehicle or the imaging unit to the center of the search area according to the size of the extracted structure; program.

Images