Information processing system, method, and program
The information processing system integrates hammering sound data with structural images by correlating impact positions, addressing the separation of data types in conventional methods and enhancing inspection efficiency.
Patent Information
- Application Number
- JP2023076734
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-20
AI Technical Summary
Existing methods fail to associate and record information obtained from hammering tests with images of inspection targets, requiring separate files for each type of data.
An information processing system that integrates image acquisition with hammering sound information by correlating impact positions on structures with captured images, using a system comprising imaging and sound acquisition units, and a control mechanism for storing this data together.
Enables the association and storage of hammering sound data with corresponding images, facilitating comprehensive inspection by linking visual and auditory data for structural integrity assessment.
Smart Images

Figure 2024161748000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a system for acquiring images and hammering sound information of an inspection target in order to detect abnormalities in the inspection of a structure. [Background technology]
[0002] When inspecting the walls of structures such as bridges, image inspections are carried out using photographed images of the object to be inspected.In addition, inspections using elastic waves (e.g. hammering) are also carried out at the same time to investigate internal deterioration that cannot be detected by image inspections.
[0003] In addition, since the hammering test and the photographing of the test object are separate processes, the information acquired by performing the hammering test is recorded in a file separate from the image of the test object. In Patent Document 1, the information acquired by performing the hammering test is recorded in association with the striking position. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 6506817 Summary of the Invention [Problem to be solved by the invention]
[0005] However, conventionally, information acquired by performing a hammering test has not been recorded in association with an image of the test object.
[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique for correlating and saving information about the striking sound made when an object to be inspected is struck with a photographed image of the object to be inspected. [Means for solving the problem]
[0007] In order to achieve the above object, for example, the information processing system of the present invention has the following configuration: That is, the information processing system of the present invention has a first acquisition means for acquiring a captured image of an object to be inspected, a second acquisition means for acquiring coordinates in the captured image corresponding to information regarding the impact position of the object to be inspected, and a control means for causing the coordinates in the captured image corresponding to the information regarding the impact position to be stored in a storage means in association with the captured image. Effect of the Invention
[0008] According to the present invention, information about the striking sound produced when the test object is struck can be stored in association with a photographed image of the test object. [Brief description of the drawings]
[0009] [Figure 1] System configuration diagram of this embodiment [Diagram 2] Functional block diagram of the present embodiment [Diagram 3] A block diagram showing the configuration of each device of the present embodiment. [Figure 4] FIG. 1 is a diagram for explaining a method for photographing an object to be inspected using the system of this embodiment. [Diagram 5] FIG. 1 is a diagram for explaining an image captured by the system of the present embodiment; [Figure 6] 1 is a flowchart showing the operation of the system according to the present embodiment. [Figure 7] Flowchart showing hitting position setting process [Figure 8] FIG. 1 is a diagram for explaining the coordinates of an image of a test sound acquired by the system of this embodiment. [Figure 9] Flowchart showing tapping processing [Figure 10] A diagram explaining metadata of hammering sound information DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.
[0011] <First embodiment> In the description of this embodiment, "metadata" refers to information related to a file that is stored in the file. It may also be called meta information, additional information, accompanying information, or attached information.
[0012] Fig. 1 is a system configuration diagram according to a first embodiment of the present invention. In Fig. 1, 101 denotes an imaging device, 102 denotes a pan head device incorporating the imaging device 101, 110 denotes a tapping device, and 111 denotes a drone device equipped with the tapping device 110. 103 denotes a computing device that communicates with the imaging device 101 and the pan head device 102, and 104 denotes a tripod on which the pan head device is mounted.
[0013] Additionally, reference numeral 105 denotes a drive unit in the pan direction of the tripod head device, 106 denotes a drive unit in the tilt direction of the tripod head device, and 107 denotes a drive unit in the zoom direction of the tripod head device for controlling the zoom position of the lens of the imaging device 101. Reference numeral 108 denotes a communication cable that connects the imaging device 101 and the tripod head device 102, and 109 denotes a communication cable that connects the computing device 103 and the tripod head device 102. Note that the communication cable 108 may be directly connected between the imaging device 101 and the computing device 103.
[0014] Fig. 2 is a functional block diagram of the imaging system according to the first embodiment of the present invention, in which 201 indicates the functional block diagram of the imaging device 101 in Fig. 1, 202 indicates an imaging unit that captures images, 203 indicates an imaging control unit that controls the entire imaging device 101, and 204 indicates an image recording unit that records the captured images.
[0015] Reference numeral 205 denotes a unit that drives the focus of an optical lens attached to the imaging device 101 , reference numeral 206 denotes an AF distance measuring unit that measures the distance to a subject for an autofocus function, and reference numeral 207 denotes a communication unit that communicates with the camera platform device 102 and the arithmetic device 103 .
[0016] 1. Reference numeral 209 denotes a pan head control unit that controls the pan head device 102, 210 denotes a unit that drives the pan direction drive unit 105, and 211 denotes a unit that drives the tilt direction drive unit 106. Reference numeral 214 denotes a unit that drives the zoom direction drive unit 107 that controls the zoom position of the lens of the imaging device 101. Reference numeral 212 denotes a communication unit for communicating with the imaging device 101 and the arithmetic device 103.
[0017] Reference numeral 213 denotes a functional block diagram of the computing device 103. Reference numeral 214 denotes a camera head control unit that controls the drive unit in the pan / tilt / zoom direction of the camera head device 102, drives the camera head device 102 to a specific pan / tilt angle and zoom position, and acquires the current pan / tilt angle and zoom position. Reference numeral 215 denotes an image storage unit that stores images captured by the imaging device 101 together with additional information. Reference numeral 216 denotes an image processing unit that processes the images stored in the image storage unit 215 to generate a stitched image or a composite image. Reference numeral 217 denotes an imaging control unit that controls imaging of the imaging device 101. Reference numeral 218 denotes a communication unit that communicates with the camera head device 102, the imaging device 101, and the drone device 111.
[0018] Reference numeral 219 denotes a functional block diagram of the drone device 111, 220 denotes a communication unit that communicates with the drone device 111, and 221 denotes an imaging unit that captures images. Reference numeral 222 denotes an input unit that inputs various information to the drone device 111, 223 denotes a striking sound unit that strikes a wall surface, and 224 denotes a sound collection unit that acquires the striking sound generated by the striking sound unit 224.
[0019] Reference numeral 225 denotes a hitting sound data recording unit that records the hitting sound data acquired by the sound collection unit 224 in a photographed image.
[0020] FIG. 3 is an internal configuration diagram of each device in the first embodiment of the present invention. FIG. 3(a) shows the internal configuration of the imaging device 101. Here, 301 is a central processing unit (hereinafter referred to as CPU) that controls the imaging device, and the CPU 301 performs calculations and processing of information and controls each hardware based on a control program to realize each functional configuration and processing described later. 302 is a read-only memory (hereinafter referred to as ROM) that records a control program that defines the operation processing procedure of the CPU 301. The ROM 302 includes a program ROM that records an operating system (OS), which is a system program that controls the imaging device, and a data ROM that records information necessary to operate the system. 303 is a random access memory (hereinafter referred to as RAM), which functions as the main memory of the CPU 301 and as a work memory required for loading execution programs and executing programs. 304 is an imaging module, which is composed of an imaging element that receives subject light incident through an optical lens device, converts it into an imaging signal, and outputs it, and an A / D conversion unit that converts the imaging signal output from the imaging element into image data and outputs it. Furthermore, the imaging module 304 writes the image data output from the A / D conversion unit directly or indirectly to a storage device 311, which will be described later. Reference numeral 305 denotes an AF distance measurement module that measures the distance to a subject via an optical lens device of the imaging device 304 and determines a focus control position, and reference numeral 306 denotes a lens control module that controls the optical lens device. Reference numerals 307 and 308 denote a focus drive module and an iris drive module that drive the focus and the iris, respectively, based on an instruction from the lens control module 306. Reference numeral 309 denotes an input / output I / F module that allows a user to set various shooting settings for the imaging device 101, and reference numeral 310 denotes a communication module that allows the imaging device 101 to exchange various information and control signals with the pan head device 102 and the arithmetic device 103 via the pan head device 102. Reference numeral 311 denotes a recording device that records an image captured by the imaging module 304 using a memory card or the like, and reference numeral 312 denotes an input / output bus (address bus, data bus, and control bus) that connects the above-mentioned modules.
[0021] 3(b) shows the internal configuration of the pan head device 102. Here, the explanation of 313, 314, 315, and 322 will be omitted because their roles are the same as those of 301, 302, 303, and 312 in the imaging device. Reference numeral 316 denotes an imaging device communication module for communicating with the imaging device 101, and 317 and 318 denote a pan drive module and a tilt drive module for driving the pan and tilt of the pan head device 102, respectively. Reference numeral 319 denotes an input / output I / F module for allowing the user to perform various settings for the pan head device 102, and 320 denotes a calculation device communication module for communicating with the calculation device 103. Reference numeral 321 denotes a zoom direction drive unit for controlling the zoom position of the lens of the imaging device 101.
[0022] 3(c) shows the internal configuration of the arithmetic device 103. Here, the roles of 323, 324, 325, 328, and 331 are the same as those of 301, 302, 303, 310, and 312 in the imaging device, so the description will be omitted. In addition, 326 is a display device for presenting various information of the arithmetic device to the user, and 327 is an input device for the user to input various information to the arithmetic device. 329 is a hard disk drive (HDD) that serves as an auxiliary storage device, and 330 is a drone device communication module for communicating with the drone device 111.
[0023] 3(d) shows the internal configuration of the drone device 111. Here, the roles of 333, 334, 335, 337, 338, 340, and 341 are the same as those of 323, 324, 325, 326, 327, 329, and 331 in the computing device, so the description thereof will be omitted.
[0024] Reference numeral 336 denotes an imaging module, which is configured with an A / D conversion unit that receives subject light incident via an optical lens device, converts it into an imaging signal, and outputs it. Reference numeral 339 denotes a communication module for communicating with the arithmetic unit 103.
[0025] The flight control unit 342 controls the rotor driving unit 344 based on a signal from the mixer unit 343, thereby controlling the position and attitude of the drone device 111 and controlling the flight of the drone device 111. Under the control of the flight control unit 342, the rotor driving unit 344 performs drive control of the rotor that causes the drone device 111 to fly.
[0026] The sensor processing unit 345 processes information measured by the sensors to obtain information (sensor information) necessary for flight control of the drone device 111, such as its own position and attitude. Applicable sensors include a gyro sensor, an acceleration sensor, a Global Navigation Satellite System (GNSS), a barometric pressure sensor, an ultrasonic sensor (altitude sensor, distance sensor), a sensor for obtaining ground images, and the like.
[0027] The video transmission unit 342 transmits (sends) the image output from the imaging module 336 to the outside. The mixer unit 343 mixes the sensor information acquired by the sensor processing unit 345 with a signal from the arithmetic device 103 operated by the user or the CPU 323 to control the drone device 111, and outputs the result to the flight control unit 342. As a result, the flight control unit 342 performs flight control according to an operation signal from the arithmetic device in a state in which the flight control of the drone device 111 is performed according to the sensor information acquired by the sensor processing unit 345. The striking device control unit controls the striking device drive unit 347 based on the signal from the arithmetic device 103. The microphone 348 collects the striking sound produced by the striking device drive unit 347 and stores it in the storage device 340.
[0028] FIG. 4 is a diagram for explaining the operation of the imaging system, which is made up of the imaging device 101, the camera platform device 102, and the computing device 103, when photographing the wall surface of a building in the first embodiment of the present invention.
[0029] 4(a) is a simplified aerial view showing how the imaging device 101 mounted on the pan head device 102 captures an image of the first section while continuously panning according to a photography plan created by the computing device 103. Here, 401 denotes the installation position of the imaging device 101 mounted on the pan head device 102, 402 denotes a building, and 403 denotes the wall surface of the building 402 to be inspected. At this time, the installation position 401 of the imaging device should be placed on the perpendicular line to the center of the wall surface 403 to be inspected, but in actual photography sites, it is not rare that the installation position deviates from the center line due to the influence of obstacles such as other buildings.
[0030] Moreover, the shooting direction 404 indicates the center direction of the direction in which the imaging device 101 shoots the first section by panning the camera platform device 102, and 419 indicates the angle of view shot in the shooting direction 404. Moreover, 410 indicates the intersection point where the shooting direction 404 of the first section intersects with the wall surface 403 of the inspection target, and when the imaging device 101 shoots, it applies AF to the wall surface at the intersection point position, which is the center of the angle of view, to adjust the focus. At this time, when focusing at the intersection point 410, the in-focus plane on which the image of the subject can be focused on the smallest circle on the surface of the imaging sensor is the plane shown in 416. However, in reality, even if the distance to the plane of the in-focus plane 416 is out of the range and the size of the circle when the image of the subject is focused is not the smallest, if the size of the circle is within the pixel size of the imaging sensor, the out-of-focus cannot be identified. Such a range of distances in which out-of-focus cannot be identified is called the depth of field, and can be calculated from the pixel size of the imaging sensor, the focal length and aperture value of the optical lens device, and the distance to the subject.
[0031] 4A, the depth of field in front of the focal plane 416 is plane 417, and the depth of field behind it is plane 418, and wall surface 403 within these distance ranges can also be considered to be in focus. Therefore, when photographing wall surface 403 shown in FIG. 4A, it is necessary to appropriately set in advance the installation position 401 of the imaging device and the photographing parameters (focal length and aperture value) of the optical lens device to be used, taking into account the depth of field.
[0032] FIG. 4(b) is a simplified aerial view of the operation of the imaging device 101 mounted on the pan head device 102 when the imaging device 101, after imaging the first section, continuously pans according to the imaging plan created by the computing device 103. Also, 405 to 409 indicate the center direction of the direction in which the imaging device 101 images by panning the pan head device 102, similar to the imaging direction 404 in FIG. 4(a). 420 to 424 indicate the angle of view captured in each imaging direction, similar to the angle of view 419 in FIG. 4(a). Note that in this case, if the wall surface 403 to be inspected continues further to the right when viewed from the front, the angle of view of the wall surface 403 becomes even sharper, and a high-resolution image suitable for inspection cannot be obtained. In such a case, the imaging device 101 is newly installed on the right side, and similar imaging is performed at each of the multiple installation positions as necessary. For the sake of simplicity, it is assumed here that the wall surface 403 to be inspected is photographed six times in the vertical direction of the building, i.e., by driving only in the pan direction, without driving in the tilt direction by the camera platform device 102.
[0033] 4A, 411 to 415 are intersections where the shooting directions 405 to 409 intersect with the inspection target wall surface 403. When the imaging device 101 captures an image, it applies AF to the wall surface at the intersection position, which is the center of the angle of view, to perform focusing.
[0034] 4(a), the focal plane, the range of the front depth of field, and the range of the rear depth of field when autofocus is performed at the intersections with the wall surfaces from intersections 411 to 415 are indicated by a thick solid line, a fine dotted line, and a coarse dotted line, respectively. At this time, it is known that the range of the rear depth of field is wider than the range of the front depth of field.
[0035] At the same time, as shown in FIG. 4(b), it is known that the depth of field becomes wider as the distance from the subject increases. Furthermore, when photographing the wall surface 403 shown in FIG. 4(b), many parts of the photographed images are overlapped as shown in the angles of view 419 to 424 during each photographing. This is to ensure that the subject to be inspected, in this case the wall surface, is covered and to obtain a higher quality image for inspection by using the central part of the photographed image by the optical lens device used. In FIG. 4(b), the edge of the angle of view of the previous frame is set as the center of the next frame, but the overlapping ratio may be different.
[0036] FIGS. 5(a) and (b) are diagrams illustrating the processing of a captured image when photography is performed in the first embodiment of the present invention.
[0037] Here, 501 to 506 in Fig. 5(a) show individual captured images captured by panning the camera platform device. Note that, here, 501 to 506 in Fig. 5(a) are arranged with a vertical offset for ease of understanding, but in reality there is no vertical offset.
[0038] Also, 507, 508, and 509 in FIG. 5(b) show some of the images captured by panning and tilting the camera platform device. FIG. 5(b) shows a collection of images captured by panning and tilting the camera platform device 102. All the images are integrated into a single stitched image using the central portion of the image, and the image becomes as if the wall surface to be inspected was captured in high definition. When inspecting infrastructure facilities, the inspector uses this integrated image to check for minute abnormalities in the wall surface.
[0039] Next, an example of the operation of the entire system shown in Figures 1 and 2 as the first embodiment of the present invention will be described with reference to the flow chart shown in Figure 6. In Figures 6 to 9, steps are indicated as S. For example, step 601 is indicated as S601.
[0040] FIG. 6 is a flow chart for explaining the operation of the entire system shown in FIGS. 1 and 2 as the first embodiment according to the present invention.
[0041] In FIG. 6, S601 is a step of photographing an overall view of an inspection point on a wall surface 403 when inspecting the wall surface 403 that is the inspection target.
[0042] S602 is a step in which the overlap rate between the image of the adjacent photographed location in each photograph is input to the calculation device 103. As described in the explanation of Fig. 4(b), the overlap rate is set in order to reliably cover the subject to be inspected, here the wall surface 403, and to obtain a higher quality image for inspection by utilizing the central portion of the image photographed by the optical lens device to be used.
[0043] S603 is a step in which the arithmetic device 103 stores a divided photographing range based on the entire photographing range of the wall surface 403 acquired in S601 and the overlap rate set in S602. S604 is a step in which the arithmetic device 103 sets a striking position on the wall surface 403 based on the entire map of the inspection point acquired in S601 and the divided photographing range stored in S603.
[0044] FIG. 7 is a flow chart for explaining the operation of setting the hitting position according to the first embodiment of the present invention, and explains the internal processing of S604 in FIG.
[0045] In step S701, the pixel size of an image captured by the imaging device 101 is acquired. At this time, the pixel sizes of the long side and short side are acquired. Here, an image having a pixel size of 6000 on the long side and 4000 on the short side will be described as an example.
[0046] S702 is a step in which the impact position for each image captured by the imaging device 101 is determined.
[0047] 8 shows the impact positions for each image captured by the imaging device 101. In this embodiment, nine points 802 to 810 are determined as the impact positions. Here, as an example, the captured image is divided into four on the long side and four on the short side, and the intersections of these are determined as the impact positions; however, the number of divisions can be set arbitrarily, and the number of divisions may be input by the input device 327.
[0048] Images 501 to 506 are the individual images captured by panning the camera platform, as explained above. Here, the impact position for each image will be explained using the captured image 501 as an example.
[0049] 800 indicates the short side of the photographed image 501, and 801 indicates the long side of the photographed image 501. In this embodiment, the pixel sizes of the short side 800 and the long side 801 are 4000 and 6000, respectively. In this case, the impact position coordinates for each image can be calculated as follows. 802 is (x1500, y3000), 803 is (x3000, y3000), and 804 is (x4500, y3000). 805 is (x1500, y2000), 806 is (x1500, y2000), 807 is (x3000, y2000), 808 is (x1500, y1000), 808 is (x3000, y1000), and 808 is (x4500, y1000).
[0050] The CPU 323 of the calculation device 103 records the impact position for each image captured by the determined imaging device 101 in at least one of the RAM 325, the ROM 324, and the storage device 329. In addition, the recording destination of the impact position for each image captured by the determined imaging device 101 is not limited to the calculation device 103, and the impact position may be recorded in at least one of the imaging device 101, the camera platform device 102, the calculation device 103, and the drone device 111.
[0051] S703 is a step for creating an overall impact map. The impact position in the overall shooting range image after panoramic stitching is determined from the divided shooting ranges stored in S603 and the impact position for each captured image determined in S702. Impact position coordinates are calculated based on the pixel size of the long side and short side for each captured image acquired in S701, the overlap rate set in 602, and the divided shooting ranges set in S603. At this time, the CPU 323 may use the display device 326 and the input device 327 to ask the user whether or not to skip the impact position of the part that overlaps with the adjacent image during panoramic stitching.
[0052] Next, the overlap rate set in S602 is used to perform full shooting to cover the wall surface 403 while overlapping images between shots. S605 is a step in which the pan / tilt angle and zoom position to be instructed to move the camera platform device 102 for each shot are determined based on the divided shooting ranges set in S603.
[0053] In step S606, the calculation device 103 instructs the image capturing device 101 and the camera platform device 102 to capture images based on the divided image capturing ranges set in step S603. After the image capturing device 101 and the camera platform device 102 capture images of the wall surface 403 comprehensively, they transmit the captured images to the calculation device 103. The CPU 323 stores the received captured images in the storage device 329.
[0054] S607 is a step in which the drone device 111 strikes the wall surface 403 based on the striking position determined in S604, and obtains a striking sound.
[0055] FIG. 9 is a flow chart for explaining the operation of the tapping sound processing according to the first embodiment of the present invention, and explains the internal processing of S607 in FIG.
[0056] S901 is a step in which the arithmetic device 103 transmits an instruction to start the hitting sound processing to the drone device 111. At this time, the CPU 323 transmits the instruction to start the hitting sound processing and the entire hitting map created in S703 to the drone device 111 via the communication module 330.
[0057] S902 is a step in which the drone device 111 receives an instruction to start the hitting sound processing from the arithmetic device 103. At this time, the CPU 333 receives the instruction to start the hitting sound processing and the entire impact map created in S703 via the communication module 339, and develops the impact map in the RAM 335.
[0058] S903 is a step in which the drone device 111 moves to the first hitting position in the impact map. The CPU 333 instructs the flight control unit 342 to move to the first hitting position of the impact map. Here, the drone device 111 moves to the impact position coordinates 802 (x1500, y3000) of the captured image 501 described in FIG. 8 as the first hitting position of the impact map.
[0059] S904 is a step of performing a tapping sound on the wall surface 403. The CPU 333 instructs the tapping control unit 346 to perform a tapping sound. The tapping device 110 performs a tapping sound on the wall surface 403, and stores the generated tapping sound in the storage device 340 or the RAM 335. In addition, partial areas of adjacent divided shooting ranges may overlap each other. The control unit 306 judges whether or not to perform a tapping sound on the overlapping area, and instructs the tapping device 110 to perform a tapping sound when it is judged that a tapping sound is to be performed. In addition, the judgment of whether or not to perform a tapping sound may be made by any of the units of the information processing device, not the control unit 306. In addition, the judgment of whether or not to perform a tapping sound may be made based on a partially captured image. For example, crack detection may be performed in the overlapping area of the partially captured image, and it may be judged that a tapping sound is to be performed when the length or width of the detected crack is equal to or greater than a predetermined threshold value. As a result, more inspection data can be obtained for the overlapping area where the inspection is highly important. In addition, the judgment of whether or not to perform a tapping sound may be made by a user and instructed to the control unit 306.
[0060] S905 is a step for judging whether or not all the hitting sounds at the hitting positions determined in the photographed image 501 in the hitting map developed in S902 have been completed. If all the hitting sounds in the photographed image 501 have been completed, the process proceeds to S907. If the hitting sounds in the photographed image 501 have not been completed, the process proceeds to S906.
[0061] In S906, the CPU 333 instructs the flight control unit 342 to move to the next hitting position on the impact map. As an example, the next hitting position after the hitting position 802 in the photographed image 501 is the hitting position 803. Also, if the previous hitting position is the last hitting position of the captured image, it moves to the first hitting position of the next captured image. For example, after the hitting position 810 in the photographed image 501, the drone device 111 moves to the hitting position 802 (x1500, y3000) in the photographed image 502.
[0062] In S907, the drone device 111 transmits the hitting sound data to the calculation device 103. At this time, the CPU 333 instructs the communication module 339 to transmit the hitting sound data stored in the storage device 340 or the RAM 335. The hitting sound data is, for example, time-series data of the amplitude and frequency of each hitting sound at the hitting sound positions 802 to 810 for the hitting sound data captured image 501.
[0063] In step S908, the calculation device 103 receives the hitting sound data transmitted from the drone device 111. The CPU 323 receives the hitting sound data via the communication module 330 and stores it in the RAM 325 or the storage device 329.
[0064] In step S909, the captured image is stored in association with the tapping sound metadata. The CPU 323 records the tapping sound data stored in the RAM 325 or the storage device 329 in an Exif (Exchangeable image file format) area of the captured image stored in the storage device 329.
[0065] In addition, the result of detecting anomalies on the inspection object may be recorded in the Exif area. For example, the type of anomaly (crack, water leakage, peeling, etc.) and the status of the anomaly (crack length, crack width, etc., the position of the anomaly, the degree of the anomaly) may be recorded as the result of detecting anomalies. The anomaly may be detected visually by a user or using a detection model. The detection model may output the result of detecting an anomaly by inputting an image of the inspection object. The detection model may be a trained estimation model trained by machine learning. The machine learning may be performed based on the result of inputting an image into the estimation model and outputting it, and on the teacher data. The estimation model may be a neural network such as a CNN (convolutional neural network), and the neural network may be trained by a backpropagation method.
[0066] The Exif area may also record the time the object was photographed, the location where it was photographed, the user name, the file name, the file type, the update date and time, the image size, the image capacity, the image resolution, the model name of the camera, and the parameters at the time of shooting (aperture, ISO sensitivity, magnification, etc.).
[0067] An example of the recorded hitting sound metadata is shown in Fig. 10. At this time, the CPU 323 may ask the user, using the display device 326 and the input device 327, whether or not to record the hitting position of the part overlapping with the adjacent image during panoramic stitching as metadata.
[0068] The tapping sound metadata has a hierarchical structure with information 1001 as the top information, and there is no particular restriction on the structure. For example, in FIG. 10, there is a three-layer structure of information 1001, 1002, and 1003. Information 1004 is tapping sound data, which is a tapping sound at a tapping position relative to the photographed image 501. For example, it is saved in the form of an audio file such as a WAVE file. Information 1005 is the coordinates of the tapping sound position relative to the photographed image from which the tapping sound data 1004 was acquired. Similarly, tapping sound data obtained by striking a position corresponding to another pixel on the same image is stored under other ID information 1006 and 1007. In addition to this, the date and time of acquisition of the tapping sound data and the tapping position in the stitched image may be recorded in the layer 1003 as the tapping sound metadata.
[0069] Furthermore, when partial areas of adjacent divided photographing ranges overlap each other, the CPU 323 judges whether or not to save the overlapping areas, and when it is judged that the overlapping areas should be saved, it issues an instruction to save the tapping sound data in the overlapping areas in the RAM 325 or the storage device 329. Moreover, the judgment of whether or not to save may be made by any of the components of the information processing device, rather than the CPU 323. Moreover, the judgment of whether or not to save may be made based on the partially photographed image. For example, it may be possible to perform crack detection in the overlapping areas of the partially photographed images, and judge to save the data when the length or width of the detected crack is equal to or greater than a predetermined threshold value. This allows more inspection data to be obtained for overlapping areas where inspection is highly important.
[0070] S910 is a step for judging whether or not all the hitting sounds at all the hitting sound positions in the hitting map developed in S902 have been made. If all the hitting sounds have been made, the process proceeds to S911. If not, the process proceeds to S906.
[0071] S911 is a step in which the drone device 111 transmits a notification of the end of the hitting sound processing to the calculation device 103. At this time, the CPU 333 transmits the notification of the end of the hitting sound processing to the calculation device 103 via the communication module 339. S912 is a step in which the calculation device 103 receives the notification of the end of the hitting sound processing from the drone device 111. At this time, the CPU 323 receives the notification of the end of the hitting sound processing via the communication module 330, and ends the operation of the hitting sound processing.
[0072] S6088 is a step in which a panoramic stitching process is performed on the group of images captured in S606 and having the tapping information metadata recorded thereon to obtain a synthesized panoramic photograph. The CPU 323 performs image processing to synthesize the seams between the images to which the tapping information metadata is added and stored in the storage device 329 to obtain a panoramic photograph, and then the operation is completed.
[0073] As described above, according to this embodiment, the purpose is to capture high-resolution images for inspecting building walls in the inspection of the walls of structures such as bridges. When performing a large amount of continuous photography that divides the image into small parts and covers the entire image, it is possible to add multiple pieces of information, such as hammering test data for investigating internal deterioration linked to the inspection image, to the original image and save it. This makes it possible to manage data linked to the inspection image with a simple system configuration without preparing a separate database, etc. This makes it possible to improve portability and portability when providing inspection results to third parties.
[0074] In the present embodiment, the photographing system is a system composed of a plurality of devices, but it goes without saying that any of the devices may be configured to include the other devices, or may be configured by a single device. In the present embodiment, the arithmetic device 103 is provided separately from the drone device, but the arithmetic device may be mounted on the drone device. In the present embodiment, the camera platform device 102 and the arithmetic device 103 are capable of arbitrarily acquiring and setting the value of the zoom control position, but in the case of a photographing system in which only a zoom drive instruction based on a relative numerical value is possible, the following configuration may be used. That is, the camera platform device 102 or the arithmetic device 103 is configured to grasp the value of the current zoom control position by acquiring and managing all drive information for changing the zoom control position. With such a configuration, it is possible to support a photographing system in which only a zoom drive instruction based on a relative numerical value is possible for the zoom control position. Furthermore, it is possible to support a photographing system in which a zoom drive is realized by adding a zoom drive device to the lens device.
[0075] <Second embodiment> In the first embodiment, the object to be inspected is photographed in parts using the panoramic head device 102, but the present invention is not limited to the above embodiment, and photographing may be performed manually by the user or using another panoramic photographing device.
[0076] In addition, in the first embodiment, the impact was performed using the drone device 111, but the present invention is not limited to the above embodiment, and the impact may be performed manually by a user, or the impact may be performed by mounting an impact device on another device such as an unmanned aerial vehicle.
[0077] <Other embodiments> Although the embodiment has been described above in detail, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium). Specifically, the present invention may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device. Needless to say, the object of the present invention can be achieved by the following. That is, a recording medium (or a storage medium) on which a program code (computer program) of software that realizes the functions of the above-mentioned embodiment is recorded is supplied to the system or apparatus. Needless to say, the storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiment, and the recording medium on which the program code is recorded constitutes the present invention.
[0078] <System 1> A first acquisition means for acquiring an image of an object to be inspected; a second acquiring means for acquiring coordinates in the captured image corresponding to information about the impact position of the test object; a control means for causing a storage means to store the information about the impact position and the corresponding coordinates in the captured image in association with the captured image; An information processing system comprising:
[0079] <System 2> the captured image has an Exif area, The control means stores coordinates of the captured image in an Exif area of the captured image. The information processing system according to System 1,
[0080] <System 3> The first acquisition means acquires the captured image captured for each partial region of the inspection target, the second acquisition means acquires information about the impact position for each of the partial regions; The control means generates image information that associates information about the impact position for each of the partial regions with coordinates of the captured image of the corresponding partial region and the captured image of the corresponding partial region, synthesizes the generated image information for each of the partial regions, and stores the synthesized image information in the storage means. The information processing system according to System 1,
[0081] <System 4> The first acquisition means acquires the captured image by combining images captured for each partial region of the inspection target, The second acquisition means acquires information about the hitting position, The control means causes a storage means to store information about the impact position for each of the partial regions and corresponding coordinates in the captured image in association with the combined captured image. The information processing system according to System 1,
[0082] <System 5> an imaging means for imaging the image for each partial region of the inspection object by moving an imaging region; The first acquisition means acquires the captured image from the imaging means. The information processing system according to System 1,
[0083] <System 6> The information processing system according to System 5, wherein the imaging means is mounted on a camera platform.
[0084] <System 7> a striking means for striking the test object for each partial region by moving a striking position, The second acquisition means acquires information regarding a striking position for each partial area struck by the striking means. 7. The information processing system according to System 5 or 6,
[0085] <System 8> The information processing system described in System 7, wherein the striking means is mounted on an unmanned aerial vehicle.
[0086] <System 9> The information processing system according to System 7, wherein the striking means strikes the test object for each of the partial regions imaged by the imaging means.
[0087] <System 10> the plurality of captured images in which the partial regions are adjacent to each other have overlapping regions in which the captured images are captured overlapping each other, a striking means for striking the overlapping area of the inspection object when it is determined that the overlapping area should be struck based on the captured image; 5. The information processing system according to System 3 or 4.
[0088] <System 11> The information processing system described in system 10 is characterized in that the impact means determines whether or not to impact the overlapping area based on the detection results of detecting a defect in the inspection object based on the captured image.
[0089] <System 12> the plurality of captured images in which the partial regions are adjacent to each other have overlapping regions in which the captured images are captured overlapping each other, A striking means for striking the inspection object in the overlapping area when a user instructs the inspection object to strike in the overlapping area is provided. 5. The information processing system according to System 3 or 4.
[0090] <System 13> the inspection target is a structure, The deformation is a crack. 12. An information processing system according to claim 11,
[0091] <System 14> the plurality of captured images in which the partial regions are adjacent to each other have overlapping regions in which the captured images are captured overlapping each other, When it is determined that the overlapping area is to be stored, the control means causes the storage means to store, for the overlapping area, the coordinates in the captured image corresponding to the information on the impact position in association with the captured image. 5. The information processing system according to System 3 or 4.
[0092] <System 15> The second acquisition means acquires hitting sound information when the test object is struck, The control means causes the hitting sound information to be stored in the storage means in association with the captured image. 7. The information processing system according to any one of Systems 1 to 6,
[0093] <System 16> The information processing system according to system 15, wherein the striking sound information is time series data of the amplitude and frequency of the striking sound when the test object is struck.
[0094] <System 17> The first acquisition means acquires a result of executing a process for detecting an abnormality in the inspection object, The control means causes the storage means to store a result of the execution of the abnormality detection process for the inspection object in association with the captured image. The information processing system according to System 1,
[0095] <Method 1> A first acquisition step of acquiring an image of an object to be inspected; a second acquisition step of acquiring coordinates in the captured image corresponding to information about the impact position of the test object; a control step of storing, in a storage unit, coordinates in the captured image corresponding to the information about the impact position in association with the captured image; 13. An information processing method comprising:
[0096] <Program 1> A computer program for controlling each means of the information processing device according to any one of Systems 1 to 17 by a computer.
[0097] <Configuration 1> A first acquisition means for acquiring an image of an object to be inspected; A second acquisition means for acquiring coordinates in the captured image corresponding to information about the impact position of the test object; a control means for causing a storage means to store the information about the impact position and the corresponding coordinates in the captured image in association with the captured image; 13. An information processing device comprising: [Explanation of symbols]
[0098] 101 Imaging device 102 Panhead device 103 Panhead device 204 Image recording unit 213 Arithmetic equipment 215 Image retention unit 216 Image Processing Unit 217 Imaging control unit 223 Hitting unit 225 Impact sound data recording unit 301,313,323,333 CPU 302,314,324,334 ROM 303,315,325,335 RAM
Claims
1. A first acquisition means for acquiring an image of an object to be inspected; a second acquiring means for acquiring coordinates in the captured image corresponding to information about a striking position of the test object; a control means for causing a storage means to store the information about the impact position and the corresponding coordinates in the captured image in association with the captured image; An information processing system comprising:
2. the captured image has an Exif region; The control means stores coordinates of the captured image in an Exif area of the captured image.
2. The information processing system according to claim 1 .
3. The first acquisition means acquires the captured image captured for each partial region of the inspection target, the second acquisition means acquires information about the impact position for each of the partial regions; The control means generates image information that associates information about the impact position for each of the partial regions with coordinates of the captured image of the corresponding partial region and the captured image of the corresponding partial region, synthesizes the generated image information for each of the partial regions, and stores the synthesized image information in the storage means.
2. The information processing system according to claim 1 .
4. The first acquisition means acquires the captured image by combining images captured for each partial region of the inspection target, The second acquisition means acquires information regarding the hitting position, The control means causes a storage means to store information about the impact position for each of the partial regions and corresponding coordinates in the captured image in association with the combined captured image.
2. The information processing system according to claim 1 .
5. an imaging means for imaging the image for each partial region of the inspection object by moving an imaging region; The first acquisition means acquires the captured image from the imaging means.
2. The information processing system according to claim 1 .
6. 6. The information processing system according to claim 5, wherein the imaging means is mounted on a camera platform.
7. a striking means for striking the test object for each partial region by moving a striking position, The second acquisition means acquires information regarding a striking position for each partial area struck by the striking means.
7. The information processing system according to claim 5 or 6.
8. 8. The information processing system according to claim 7, wherein the striking means is mounted on an unmanned aerial vehicle.
9. 8. The information processing system according to claim 7, wherein the striking means strikes the inspection object for each of the partial regions imaged by the imaging means.
10. the plurality of captured images in which the partial regions are adjacent to each other have overlapping regions that are captured overlapping with each other, a striking means for striking the overlapping area of the inspection object when it is determined that the overlapping area should be struck based on the captured image; 5. The information processing system according to claim 3 or 4.
11. The information processing system according to claim 10, characterized in that the impact means determines whether or not to impact the overlapping area based on a detection result of detecting a defect in the inspection object based on the captured image.
12. the plurality of captured images in which the partial regions are adjacent to each other have overlapping regions that are captured overlapping with each other, A striking means for striking the inspection object in the overlapping area when a user instructs the inspection object to strike in the overlapping area is provided.
5. The information processing system according to claim 3 or 4.
13. the inspection target is a structure, The deformation is a crack.
12. The information processing system according to claim 11.
14. the plurality of captured images in which the partial regions are adjacent to each other have overlapping regions that are captured overlapping with each other, When it is determined that the overlapping area is to be stored, the control means causes the storage means to store, for the overlapping area, the coordinates in the captured image corresponding to the information on the impact position in association with the captured image.
5. The information processing system according to claim 3 or 4.
15. The second acquisition means acquires hitting sound information when the test object is struck, The control means causes the hitting sound information to be stored in the storage means in association with the captured image.
7. The information processing system according to claim 1, wherein the information processing system is a data processing system.
16. 16. The information processing system according to claim 15, wherein the hitting sound information is time-series data on the amplitude and frequency of hitting sounds generated when the test object is hit.
17. The first acquisition means acquires a result of executing a detection process for detecting an abnormality in the inspection object, The control means causes the storage means to store a result of the execution of the abnormality detection process for the inspection object in association with the captured image.
2. The information processing system according to claim 1 .
18. A first acquisition step of acquiring an image of an object to be inspected; a second acquisition step of acquiring coordinates in the captured image corresponding to information about the impact position of the test object; a control step of storing, in a storage unit, coordinates in the captured image corresponding to the information about the impact position in association with the captured image; 13. An information processing method comprising:
19. A computer program for controlling each unit of the information processing system according to any one of claims 1 to 17 by a computer.
20. A first acquisition means for acquiring an image of an object to be inspected; a second acquiring means for acquiring coordinates in the captured image corresponding to information about a striking position of the test object; a control means for causing a storage means to store the information about the impact position and the corresponding coordinates in the captured image in association with the captured image; 13. An information processing device comprising:
Citation Information
Patent Citations
Structure inspection methods
JP6506817B1
