Information processing system, information processing apparatus, method, and program

Abstract

To provide an information processing system, an information processing apparatus, a method, and a program for specifying an area where a desired subject is captured in an image photographed with a moving photographing device.SOLUTION: An information processing system includes: an image composition unit 52 that creates a composite image obtained by connecting partial images in a movement direction which are photographed with a moving photographing device; and an image creation unit 53 that creates an image in which the plurality of composite images are arranged side by side so that the partial images photographed at the same time are aligned with respect to the movement direction.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, a method, and a program for processing an image. [Background technology]

[0002] From the perspective of ensuring traffic safety, it is necessary to properly understand the condition of road earthwork structures such as slopes. However, in order to understand the condition of road earthwork structures, workers have had to go to the site and make an assessment.

[0003] In this regard, Japanese Patent No. 4551990 (Patent Document 1) discloses a configuration in which a vehicle is equipped with a plurality of line cameras and a panoramic image is created by driving the vehicle.

[0004] However, Patent Document 1 does not disclose anything about how to identify an area in a captured image that contains a desired object, and therefore there is a need for a further technique for identifying the area. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in consideration of the problems with the above-mentioned conventional technology, and aims to provide an information processing system, information processing device, method, and program for identifying an area in an image captured while moving that contains a desired subject. [Means for solving the problem]

[0006] That is, according to the present invention, an image synthesis means for generating a synthetic image by stitching together partial images captured while moving in the direction of movement; a generating means for generating an image in which the plurality of composite images are arranged such that partial images captured at the same time are aligned in the direction of movement; An information processing system is provided, including: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an information processing system, an information processing device, a method, and a program for identifying an area in an image captured while moving, in which a desired subject is captured. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the overall configuration of a state inspection system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing an example of how the condition of a slope is inspected using the condition inspection system of this embodiment. [Figure 3] 5A and 5B are diagrams illustrating a composite image obtained by joining partial images captured in the present embodiment. [Figure 4] 3A and 3B are diagrams showing examples of composite images captured by a plurality of image capturing units included in the mobile body system of the present embodiment. [Figure 5] 5A and 5B are diagrams showing examples of combined images in the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating the relationship between the distance to an object to be photographed and resolution. [Figure 7] FIG. 10 is a diagram illustrating the relationship between the distance to an object to be photographed and resolution. [Figure 8] FIG. 2 is a diagram showing an example of the hardware configuration of the data acquisition device according to the present embodiment. [Figure 9] FIG. 2 is a diagram showing an example of the hardware configuration of an evaluation device and a data management device according to the present embodiment. [Figure 10] FIG. 2 is a software block diagram included in the state inspection system of the present embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a screen for performing a process of identifying a slope in the prior art. [Figure 12] FIG. 10 is a diagram showing an example of a screen for performing a process of identifying a slope in the prior art. [Figure 13] 10 is a flowchart showing a process for identifying a slope in the present embodiment. [Figure 14] FIG. 10 is a diagram illustrating a displacement of a composite image. [Figure 15]FIG. 10 is a diagram illustrating a displacement of a composite image. [Figure 16] FIG. 10 is a diagram showing an example of a screen for performing processing to identify a slope in the present embodiment. [Figure 17] FIG. 10 is a diagram showing an example of a report including an image of a slope identified in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described below with reference to embodiments, but the present invention is not limited to the embodiments described below. In the drawings referred to below, the same reference numerals will be used for common elements, and their description will be omitted as appropriate.

[0010] An information processing system, an information processing device, a method, and a program according to this embodiment will be described in detail below with reference to the drawings. In the following description, the X-axis, Y-axis, and Z-axis directions are based on the directions of the arrows shown in the drawings. The X-axis, Y-axis, and Z-axis directions define a three-dimensional space that is orthogonal to one another. The X-axis direction may be interpreted as the movement direction (direction of advancement, direction of travel) of a moving body (mobile device). The Y-axis may be interpreted as the height direction (vertical direction, perpendicular direction) of the slope of the object. The Z-axis may be interpreted as the direction intersecting with the movement direction of the moving body (mobile device) (the depth direction from the moving body (mobile device) toward the slope, the imaging direction). As such, the imaging system and imaging method according to this embodiment use an XYZ Cartesian coordinate system in which the movement direction of the moving body (mobile device) is the X-axis, the direction intersecting the movement direction is the Z-axis, and the direction perpendicular to the X-axis and Z-axis is the Y-axis.

[0011] The imaging system and imaging method of this embodiment divide a slope (object) into multiple regions in the height direction, and when each divided slope region is imaged with multiple cameras (imaging units), the exposure time required to image the slope with an appropriate amount of exposure is ensured. That is, when imaging a slope (object), the exposure time required to image the slope with an appropriate amount of exposure so that the brightness of the captured image is appropriate is ensured. Furthermore, the entire slope is imaged by moving not only in the height direction of the slope (in addition to the height direction) but also in the width direction of the slope while continuously capturing images, and then stitching the captured images together. That is, the slope is also divided into multiple regions in the width direction and imaged.

[0012] In this specification, "imaging" may be read as "photographing." Furthermore, in this specification, the case where the "object" is a slope and the "moving body (moving device)" is a vehicle is described as an example, but the "object" may be something other than a slope, and the "moving body (moving device)" may be something other than a vehicle.

[0013] In this specification, an imaging system is a component that includes multiple imaging units, for example, a first imaging unit (for example, a first camera and its imaging sensor) and a second imaging unit (for example, a second camera and its imaging sensor). In this specification, the first imaging unit (for example, the imaging sensor of the first camera) and the second imaging unit (for example, the imaging sensor of the second camera) do not mean that there are two imaging units (cameras (imaging sensors)), but that there are multiple imaging units (cameras (imaging sensors)). Among the multiple imaging units (cameras (imaging sensors)), the imaging unit (camera (imaging sensor)) with a relatively long imaging distance of the object (slope) is defined as the "first imaging unit (for example, the imaging sensor of the first camera)," and the imaging unit (camera (imaging sensor)) with a relatively short imaging distance of the object (slope) is defined as the "second imaging unit (for example, the imaging sensor of the second camera)." In this specification, for the sake of simplicity, an example of a configuration in which an imaging system has two imaging units is sometimes described; however, this is not intended to limit the embodiment, and the imaging system may have three or more imaging units.

[0014] In this specification, the "imaging reference position" is defined as follows: The imaging reference position refers to a position on the camera side that determines the distance from the camera to the subject (object) during imaging. For example, the imaging surface of the imaging element, which is the imaging sensor of the camera, corresponds to the imaging reference position. For example, the "imaging reference position" may be used to indicate the distance from the imaging surface (imaging reference surface) of the imaging element of the first camera to the first slope imaging area, or the distance from the imaging surface (imaging reference surface) of the imaging element of the second camera to the second slope imaging area. In other words, the installation positions of the cameras (first and second cameras) themselves are not the imaging reference position. In this embodiment, the first and second cameras cannot be installed in the same physical position, so they are installed at different positions in at least one of the horizontal and vertical directions. In reality, even if the imaging reference position of the first camera and the imaging reference position of the second camera are different, they can be corrected later. However, for convenience of explanation (to simplify the explanation), this specification will be described assuming that the imaging reference position of the first camera and the imaging reference position of the second camera are the same. That is, the "imaging reference position (common imaging reference position) of the first imaging unit and the second imaging unit" is defined, and the distance from this imaging reference position (common imaging reference position) to the first and second imaging areas (first and second slope imaging areas) is specified. Furthermore, the first camera and the second camera will be described as being arranged in a line along the moving direction of the moving body or in a line along the height direction of the moving body so that the imaging reference position is common.

[0015] In this specification, the term "object" is defined to include, for example, a slope that is an artificially crafted incline among road wooden structures, and the slope (object) is imaged to obtain an image to be used for inspection. Furthermore, the term "slope imaging area" refers to the area of ​​the slope of the object (object to be inspected) that is imaged for inspection. In this embodiment, the "slope imaging area" is divided and imaged while moving, so the range imaged in one image is defined as the "imaging area." In this way, the terms "slope imaging area" and "imaging area" are defined and used with a clear distinction between them.

[0016] In this specification, "imaging width" refers to the imaging range in the movement direction when the imaging unit (imaging sensor) of a camera captures an imaging area in one image. Also, when the imaging sensor is a line sensor, the "imaging width" is defined as the imaging range in the movement direction captured by one pixel in one image. In this embodiment, a case where the imaging sensor is a line sensor is described as an example, but the present invention is also applicable to cases where the imaging sensor is an area sensor.

[0017] In this specification, the phrase "the second imaging area is closer to the imaging reference position than the first imaging area," or more specifically, "the second imaging area is closer to the imaging reference positions of the first imaging unit and the second imaging unit in a direction intersecting the movement direction than the first imaging area," means that, when the relationship between the distance from the imaging reference position to the first imaging area and the distance from the imaging reference position to the second imaging area is specified, the latter is shorter than the former. In other words, "close / far" from the imaging reference position is a relative expression. Furthermore, in this specification, when expressions such as "the imaging distance of the first imaging area is long / far" or "the imaging distance of the second imaging area is short / near" are simply used, it means that "the second imaging area is closer to the imaging reference positions of the first imaging unit and the second imaging unit in a direction intersecting the movement direction than the first imaging area."

[0018] FIG. 1 shows an example of the overall configuration of a condition inspection system 1 for implementing the imaging system and imaging method of this embodiment. The condition inspection system 1 is an example of an information processing system, and is a system for inspecting the condition of road earthwork structures using various data acquired by a mobile system 60. The mobile system 60 is defined as a mobile body (mobile device) equipped with an imaging system 7 (described later) and a Lidar sensor (a separate component from the imaging system 7). Road earthwork structures are a collective term for structures constructed primarily from ground materials such as soil and rocks for road construction, as well as associated structures. These structures include cut and slope stabilization facilities, embankments, culverts, and similar structures. Furthermore, artificially engineered slopes, such as cut and slope stabilization facilities, embankments, and culverts, and slopes of artificial structures are referred to as slopes. In other words, slopes of road earthwork structures are referred to as slopes. Hereinafter, road earthwork structures may be referred to as "objects," "slope surfaces," or "objects including slope surfaces."

[0019] The condition inspection system 1 is composed of a mobile system 60, an evaluation system 4, a national or local government terminal device 1100, and a contractor terminal device 1200. The mobile system 60 is an example of an imaging system and is composed of a data acquisition device 9 and a mobile body (mobile device) 6, such as a vehicle equipped with the data acquisition device 9. The vehicle may be a vehicle that travels on a road or a vehicle that travels on a railroad. The data acquisition device 9 includes an imaging system 7, an example of a measurement device that measures structures, as well as a distance sensor 8a and a GNSS (Global Navigation Satellite System) sensor 8b. GNSS is a general term for satellite positioning systems such as the Global Positioning System (GPS) or the Quasi-Zenith Satellite System (QZSS).

[0020] The photography system 7 includes the above-mentioned multiple photography units that photograph multiple areas (slope photography areas) divided in the height direction (Y direction) of the slope (object). The multiple photography units have line cameras equipped with line sensors (photography sensors) in which photoelectric conversion elements are arranged in one or more rows. The photography system 7 photographs positions of the slope (object) along a predetermined photography range on the photography surface that is aligned with the traveling direction (X-axis direction) of the mobile object 6. Note that the photography system 7 is not limited to a line camera, and may be a camera equipped with an area sensor (photography sensor) in which photoelectric conversion elements are arranged in a planar manner.

[0021] The distance sensor 8a is a ToF (Time of Flight) sensor that measures the distance to the subject photographed by the photographing system 7. The GNSS sensor 8b is a positioning means that measures a position on Earth by receiving signals transmitted at different times from multiple GNSS satellites and calculating the distance to the satellite from the difference in the time at which each signal was received. The positioning means may be a device dedicated to positioning, or may be a dedicated positioning application installed on a PC (Personal Computer), smartphone, or the like. The distance sensor 8a and the GNSS sensor 8b are examples of sensor devices. The distance sensor 8a is also an example of a three-dimensional sensor.

[0022] The ToF sensor used as the distance sensor 8a measures the distance from the light source to the object by irradiating the object with laser light from the light source and measuring the scattered and reflected light.

[0023] In this embodiment, the distance sensor 8a is a LiDAR (Light Detection and Ranging) sensor. LiDAR is a method of measuring the time of flight of light using pulses, but as another method of ToF sensors, distance may be measured using a phase difference detection method. In the phase difference detection method, a measurement range is irradiated with laser light amplitude-modulated at a fundamental frequency, and the reflected light is received to measure the phase difference between the irradiated light and the reflected light to obtain time, which is then multiplied by the speed of light to calculate distance. Alternatively, the distance sensor 8a may be configured with a stereo camera or the like.

[0024] By using a three-dimensional sensor, the mobile system 60 can obtain three-dimensional information that is difficult to obtain from two-dimensional images, such as the height, inclination angle, or protrusion of a slope.

[0025] The mobile body system 60 may further include an angle sensor 8c. The angle sensor 8c is a gyro sensor or the like for detecting the angle (attitude) or angular velocity (or each acceleration) of the shooting direction of the shooting system 7.

[0026] The evaluation system 4 is configured by an evaluation device 3 and a data management device 5. The evaluation device 3 and the data management device 5 constituting the evaluation system 4 can communicate with a mobile system 60, a terminal device 1100, and a terminal device 1200 via a communication network 100. The communication network 100 is configured by the Internet, a mobile communication network, a local area network (LAN), or the like. Note that the communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). Furthermore, the evaluation device 3 and the data management device 5 may have communication functions using short-range communication technologies such as NFC (Near Field Communication) (registered trademark).

[0027] The data management device 5 is an example of an information processing device, and is a computer such as a PC that manages various data acquired by the data acquisition device 9. The data management device 5 receives various acquired data from the data acquisition device 9 and transfers the received various acquired data to the evaluation device 3 that performs data analysis. Note that the method of transferring the various acquired data from the data management device 5 to the evaluation device 3 may be manual transfer using a USB (Universal Serial Bus) memory or the like.

[0028] The evaluation device 3 is a computer such as a PC that evaluates the condition of the slope based on various acquired data transferred from the data management device 5. A dedicated application program for evaluating the condition of the slope is installed on the evaluation device 3. The evaluation device 3 detects the type or structure of the slope from the captured image data and sensor data, extracts shape data, and performs a detailed analysis by detecting the presence or absence of deformation and the degree of deformation. The evaluation device 3 also generates a report to be submitted to road administrators such as the national government, local government, or contracted business operator using the captured image data, sensor data, evaluation target data, and the results of the detailed analysis. The report data generated by the evaluation device 3 is submitted to the national government or local government via the contracted business operator as electronic data or printed paper. The report generated by the evaluation device 3 is also referred to as an inspection record sheet, inspection sheet, inspection ledger, or report. The evaluation device 3 is not limited to a PC and may also be a smartphone or tablet device. The evaluation system 4 may also be configured such that the evaluation device 3 and the data management device 5 are integrated into a single device or terminal.

[0029] The terminal device 1200 is provided at the entrusted business operator, and the terminal device 1100 is provided at the national or local government. The evaluation device 3, the terminal device 1100, and the terminal device 1200 are examples of communication terminals that can communicate with the data management device 5, and various data managed by the data management device 5 can be viewed.

[0030] 2 is a diagram showing an example of inspecting the condition of a slope using a condition inspection system 1 for implementing the imaging system and imaging method of this embodiment. In the mobile system 60, a mobile unit 6 equipped with a data acquisition device 9 travels along a road while an imaging system 7 captures images of a predetermined range of the slope (the slope of the target object).

[0031] As shown in Figure 2, a cut slope is a slope that has been cut, and a fill slope is a slope that has been filled with soil. Furthermore, the side slopes of a road running along the side of a mountain are called natural slopes. Cut and fill slopes can be made more durable by planting vegetation on their surfaces, allowing them to remain unchanged for decades. However, this is not always the case. When cut, fill, and natural slopes deteriorate due to wind and rain, shallow collapses occur, causing rocks and soil to fall from the surface, or the mountain collapses, resulting in road closures. To prevent this, techniques are used to slow the rate of deterioration of slopes exposed to wind and rain by spraying mortar on the surface (mortar spraying) or by installing and hardening concrete structures. Structures constructed using these techniques are called earthwork structures. Earthwork structures include retaining walls installed between natural slopes and roads, and rockfall protection fences that prevent rocks from falling onto roads. Both of these structures are intended to prevent road closures or personal injury due to the outflow of soil, sand, falling rocks, etc. onto the road.

[0032] In recent years, the deterioration of earthwork structures constructed decades ago has become significant, posing a major challenge for the development of social infrastructure. Therefore, it is important to detect deterioration of earthwork structures early and to inspect and maintain them to ensure their longevity. Conventional inspections of natural slopes and earthwork structures involve investigating rockfalls, collapses, landslides, or debris flows on the slopes and formulating repair plans, which are carried out through visual inspections by experts.

[0033] However, visual inspections by experts have problems with efficiency, such as the inability to inspect all of the large number of earthwork structures throughout Japan in a given period of time, and the inability to inspect embankments at high altitudes or along rivers. In addition, visual inspections are unable to quantitatively grasp the degree of deterioration, such as cracks or peeling that occur on the surface of earthwork structures.

[0034] Therefore, the condition inspection system 1 according to the embodiment acquires photographed image data of the slope of an earthwork structure using the photography system 7, and acquires sensor data including three-dimensional information using a three-dimensional sensor such as a distance sensor 8a. The evaluation system 4 then combines the acquired photographed image data and sensor data to evaluate the condition of the slope, thereby detecting shape data indicating the three-dimensional shape of the slope and detecting abnormalities such as cracks and peeling. This allows the condition inspection system 1 to efficiently perform evaluations that are difficult to inspect visually by humans.

[0035] An image constructed by stitching together images captured in this embodiment will be described below with reference to FIGS. 3 to 5. FIG. 3 is a diagram illustrating a composite image constructed by stitching together partial images captured in this embodiment. In this embodiment, each of a plurality of images captured in accordance with the movement of the vehicle is defined as a partial image. In this embodiment, a plurality of partial images captured by controlling the operation of the camera in accordance with the movement of the vehicle are stitched together and output as a single composite image. That is, the image in this embodiment is constructed as a composite image P by stitching together partial images p1 to pl captured along the direction of movement of the vehicle, as shown in FIG. 3.

[0036] Here, the mobile body system 60 of this embodiment is configured to include a plurality of image capturing units that capture partial images at different heights. Fig. 4 is a diagram showing an example of a composite image captured by the plurality of image capturing units included in the mobile body system 60 of this embodiment.

[0037] 4, the mobile body system 60 includes three image capturing units 7h, 7m, and 7l on the mobile body 6. Here, image capturing unit 7h is configured to capture an image of the upper part of the object, image capturing unit 7m is configured to capture an image of the center part of the object, and image capturing unit 7l is configured to capture an image of the lower part of the object. Therefore, by combining the composite image Ph captured by image capturing unit 7h, the composite image Pm captured by image capturing unit 7m, and the composite image Pl captured by image capturing unit 7l, a single image including the object can be generated.

[0038] The composite images captured by each image capture unit may be images of overlapping areas. For example, the hatched portion at the bottom of composite image Ph and the hatched portion at the top of composite image Pm may be a common capture range. Also, for example, the hatched portion at the bottom of composite image Pm and the hatched portion at the top of composite image Pl may be a common capture range. In this way, by providing overlapping portions in the capture ranges of each image capture unit, it is possible to ensure that there are no missing shots in the combined composite images.

[0039] 4 is an example and does not limit the embodiment. Therefore, the number of imaging units included in the mobile body system 60 can be any number.

[0040] In this embodiment, one combined image can be constructed by combining a plurality of composite images described with reference to Figures 3 and 4. Here, Figure 5 is a diagram showing an example of a combined image in this embodiment.

[0041] The hatched area in Fig. 5 indicates one of the partial images. As indicated by the solid black line in Fig. 5, the partial images are arranged in the X direction to form one composite image P captured by one imaging unit. Then, the composite images captured by each imaging unit are combined in the height direction to form one combined image including the target object. The combined image can be regarded as a so-called panoramic image.

[0042] Next, the resolution of the camera of the imaging unit of this embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are diagrams explaining the relationship between the distance to the object to be imaged and the resolution. Fig. 6 shows an example in which multiple cameras have the same specifications, while Fig. 7 shows an example in which the pixel sizes in the imaging width direction of the imaging sensors of the cameras are different.

[0043] First, FIG. 6 will be described. FIG. 6(a) is a diagram showing the relationship between the imaging width and the imaging interval when the first camera and the second camera have the same specifications. FIG. 6(b) is a diagram showing the imaging width per pixel in an image of a slope (resolution of the captured image). As shown in FIGS. 6(a) and 6(b), the imaging width of the second camera capturing the second slope imaging area, which has a shorter imaging distance, is smaller (shorter) than the imaging width of the first camera capturing the first slope imaging area, which has a longer imaging distance. Also, in FIG. 9A, the first camera and the second camera are shown aligned in the movement direction (X-axis direction) to show that the imaging width differs depending on the imaging distance. However, the first camera and the second camera do not need to be aligned in the movement direction (X-axis direction) and may be aligned in the height direction (Y-axis direction).

[0044] Next, FIG. 7 will be described. FIGS. 7(a) and 7(b) are conceptual diagrams of imaging by the first camera and the second camera of this embodiment. As shown in FIGS. 7(a) and 7(b), the imaging width of the second camera imaging the second slope imaging area with a shorter imaging distance is the same (or substantially the same) as the imaging width of the first camera imaging the first slope imaging area with a longer imaging distance. Also, while FIG. 7(a) shows the first camera and the second camera aligned in the movement direction (X-axis direction), the first camera and the second camera do not need to be aligned in the movement direction (X-axis direction) and may be aligned in the height direction (Y-axis direction). In this way, the pixel size in the imaging width direction of the imaging sensor of the second camera, which images a nearby area, is increased so that the imaging widths of the first camera and the second camera are the same (or substantially the same). Furthermore, by making the imaging widths the same (or substantially the same), the imaging intervals can also be made the same (or substantially the same).

[0045] When the distance to the object is close, the shooting range is narrow, and when the distance is far, the shooting range is wide, so the resolution of the image captured on the closer side is high. Therefore, as shown in Figures 6 and 7, by lowering the resolution of the camera of the shooting unit on the closer side to the object, the difference in resolution between images captured by multiple shooting units at different distances to the object can be reduced, making image analysis easier. In the embodiment described below, for example, by increasing the resolution of shooting unit 7h that captures the upper part of the slope and decreasing the resolution of shooting unit 7l that captures the lower part of the slope, the difference in resolution between the captured images can be reduced.

[0046] Next, the hardware configuration of each device constituting the state inspection system 1 will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram showing an example of the hardware configuration of the data acquisition device 9 of this embodiment, and Fig. 9 is a diagram showing an example of the hardware configuration of the evaluation device 3 and the data management device 5 of this embodiment. Note that components may be added or deleted from the hardware configurations shown in Figs. 8 and 9 as needed.

[0047] ○Hardware configuration of data acquisition device○ 8 is a diagram showing an example of the hardware configuration of the data acquisition device 9 of this embodiment. The data acquisition device 9 includes the image capture device 7 and the sensor device 8 as shown in FIG. 1, as well as a controller 900 that controls the processing or operation of the data acquisition device 9.

[0048] The controller 900 includes an imaging device I / F (Interface) 901, a sensor device I / F 902, a bus line 910, a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HD (Hard Disk) 914, an HDD (Hard Disk Drive) controller 915, a network I / F 916, a DVD-RW (Digital Versatile Disk Rewritable) drive 918, a media I / F 922, an external device connection I / F 923, and a timer 924.

[0049] Of these, the imaging device I / F 901 is an interface for transmitting and receiving various data or information to and from the imaging device 7. The sensor device I / F 902 is an interface for transmitting and receiving various data or information to and from the sensor device 8. The bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 911 shown in FIG. 8.

[0050] Furthermore, the CPU 911 controls the overall operation of the data acquisition device 9. The ROM 912 stores programs used to drive the CPU 911, such as the IPL. The RAM 913 is used as a work area for the CPU 911. The HD 914 stores various data such as programs. The HDD controller 915 controls the reading and writing of various data from and to the HD 914 under the control of the CPU 911. The network I / F 916 is an interface for data communication using the communication network 100.

[0051] The DVD-RW drive 918 controls reading and writing of various data from and to a DVD-RW 917, which is an example of a removable recording medium. Note that the recording medium is not limited to a DVD-RW, and may be a DVD-R, a Blu-ray (registered trademark) Disc, or the like.

[0052] The media I / F 922 controls the reading and writing (storage) of data from and to a recording medium 921 such as a flash memory. The external device connection I / F 923 is an interface for connecting an external device such as an external PC 930 having a display, a reception unit, and a display control unit. The timer 924 is a measurement device with a time measurement function. The timer 924 may be a computer-based software timer. It is preferable that the timer 924 be synchronized with the time of the GNSS sensor 8b. This makes it easy to synchronize the time and associate the positions of each sensor data and captured image data.

[0053] ○Hardware configuration of evaluation device○ 9 is a diagram showing an example of the hardware configuration of the evaluation device 3 of this embodiment. Each hardware component of the evaluation device 3 is indicated by a reference numeral in the 300 series. As shown in FIG. 9, the evaluation device 3 is constructed by a computer, and includes a CPU 301, a ROM 302, a RAM 303, a HD 304, an HDD controller 305, a display 306, an external device connection I / F 308, a network I / F 309, a bus line 310, a keyboard 311, a pointing device 312, a DVD-RW drive 314, and a media I / F 316.

[0054] Of these, the CPU 301 controls the overall operation of the evaluation device 3. The ROM 302 stores programs, such as an IPL, used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data, such as programs. The HDD controller 305 controls the reading and writing of various data from and to the HD 304 under the control of the CPU 301. The display 306 displays various information, such as a cursor, menus, windows, characters, or images. The display 306 is an example of a display unit. The external device connection I / F 308 is an interface for connecting various external devices. In this case, external devices include, for example, a USB memory or a printer. The network I / F 309 is an interface for data communication using the communication network 100. The bus line 310 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 301, shown in FIG. 9 .

[0055] The keyboard 311 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 314 controls reading and writing of various data from a DVD-RW 313, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R or Blu-ray (registered trademark) Disc, etc. The media I / F 316 controls reading and writing (storing) of data from a recording medium 315, such as a flash memory.

[0056] ○Hardware configuration of data management device○ 9 is a diagram showing an example of the hardware configuration of the data management device 5 of this embodiment. Each hardware component of the data management device 5 is indicated by a reference numeral in the 500s in parentheses. As shown in FIG. 9, the data management device 5 is constructed by a computer, and has the same configuration as the evaluation device 3, as shown in FIG. 9, and therefore a description of each hardware component will be omitted. Note that the communication terminals 1100 and 1200 are also constructed by a computer and have the same configuration as the evaluation device 3, but a description of each hardware component will be omitted.

[0057] Each of the above programs may be recorded as an installable or executable file on a computer-readable recording medium and distributed. Examples of recording media include CD-Rs (Compact Disc Recordables), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, SD cards, and USB memory. The recording media may also be provided domestically or internationally as a program product. For example, the evaluation system 4 according to the embodiment realizes the method according to the present invention by executing the program according to the present invention.

[0058] Next, various functional means in this embodiment will be described with reference to Fig. 10. Fig. 10 is a software block diagram included in the state inspection system 1 of this embodiment.

[0059] 10, the data acquisition device 9 includes functional means of a communication unit 91, a photographing unit 92, a photographing time acquisition unit 93, and a movement speed measurement unit 94. The data management device 5 includes functional means of a communication unit 51, an image synthesis unit 52, an image generation unit 53, an image combination unit 54, a report creation unit 55, and a memory unit 56. The evaluation device 3 includes functional means of a communication unit 31, a display unit 32, an operation unit 33, and an object identification unit 34. Each functional means will be described in detail below.

[0060] First, we will explain the functional means of the data acquisition device 9. A communication unit 91 constitutes the communication means of this embodiment and is a means for controlling communication between the data acquisition device 9 and the communication network 100. The communication unit 91 of this embodiment can transmit various data such as a captured image, the time the image was captured, and the moving speed at the time of capture to the data management device 5 via the communication network 100.

[0061] The photographing unit 92 is a means for controlling the operation of the cameras of the photographing system 7 and photographing images. The photographing unit 92 constitutes the photographing means of this embodiment. The photographing unit 92 of this embodiment can photograph at a predetermined timing that matches the travel of the moving body 6, thereby making it possible to photograph a panoramic image that unfolds in the direction of travel of the vehicle, and for example, to photograph an image that makes it easy to grasp the entire slope.

[0062] The photographing time acquisition unit 93 is a means for acquiring the time when the photographing unit 92 photographed an image, and constitutes the acquisition means of this embodiment. The photographing time acquisition unit 93 may acquire the time, for example, from various sensor devices 8, or may acquire the time from a timer 924 built into the data acquisition device 9. The photographing time acquisition unit 93 of this embodiment can acquire the time when the photographing unit 92 photographed a partial image, and associate the partial image with the photographing time.

[0063] The moving speed measuring unit 94 is a means for measuring the moving speed of the moving object 6, and constitutes the measuring means of this embodiment. The moving speed measuring unit 94 can calculate the moving speed using various sensor devices 8, for example.

[0064] Next, the functional means of the data management device 5 will be described. The communication unit 51 constitutes the communication means of this embodiment and is a means for controlling communication between the data management device 5 and the communication network 100. The communication unit 51 of this embodiment can receive various data such as the captured image, the time the image was captured, and the moving speed at the time of capture from the data acquisition device 9. In addition, the communication unit 51 can transmit image data generated by the image generation unit 53 to the evaluation device 3.

[0065] The image synthesis unit 52 is a means for synthesizing an image by stitching together partial images captured by the imaging unit 92. The image synthesis unit 52 constitutes the image synthesis means of this embodiment. As shown in FIG. 3, the image synthesis unit 52 of this embodiment can generate a composite image by stitching together multiple partial images.

[0066] The image generation unit 53 is a means for generating an aligned image in which a plurality of composite images are arranged, and constitutes the generation means of this embodiment. The image generation unit 53 of this embodiment can generate an aligned image configured so that a plurality of composite images output by the image synthesis unit 52 are aligned in appropriate positions. The image data generated by the image generation unit 53 may be stored in the storage unit 56, for example, or may be transmitted to the evaluation device 3 via the communication unit 51.

[0067] The image combining unit 54 is a means for combining images by joining multiple composite images together in the height direction (Y direction). The image combining unit 54 constitutes the image combining means of this embodiment. The image combining unit 54 of this embodiment can combine the composite images by, for example, performing image analysis on the overlapping areas of each composite image (areas indicated by hatching in FIG. 4).

[0068] The record creation unit 55 constitutes the creation means of this embodiment and is a means for creating a record related to a slope based on various data. The record creation unit 55 of this embodiment can create a record including, for example, an ID for identifying the slope, a combined image including the slope, the extension of the slope, and the latitude and longitude of the start and end points of the slope.

[0069] The storage unit 56 is a means for controlling the operation of a storage device such as the HD 504 and storing various types of data, and constitutes the storage means of this embodiment. The storage unit 56 of this embodiment can store, for example, partial image data, composite image data, shooting time data, movement speed data, and aligned image data. The storage unit 56 can also store various types of data in association with each other, and can store, for example, a partial image in association with the time at which the partial image was captured and the movement speed at which the partial image was captured.

[0070] Next, we will explain the functional means of the evaluation device 3. The communication unit 31 constitutes the communication means of this embodiment and is a means for controlling communication between the evaluation device 3 and the communication network 100. The communication unit 31 of this embodiment can receive, for example, an alignment image generated by the image generation unit 53 from the data management device 5.

[0071] The display unit 32 is a means for controlling the operation of the display 306 and displaying various UIs, etc., and constitutes the display means of this embodiment. The display unit 32 of this embodiment can display a screen for performing processing to identify an object using, for example, the aligned images generated by the image generation unit 53.

[0072] The operation unit 33 constitutes the operation means of this embodiment, and is a means for accepting operations from the keyboard 311, the pointing device 312, etc., and performing various operations on the evaluation device 3. The operation unit 33 of this embodiment can perform, for example, an operation for specifying an object in a process for identifying the object.

[0073] The object identification unit 34 constitutes the identification means of this embodiment and is a means for identifying an object based on the aligned images generated by the image generation unit 53. The object identification unit 34 of this embodiment may identify an object by, for example, performing image analysis of the aligned images generated by the image generation unit 53, such as extracting features, or by accepting an operation by a user to identify the object on a screen displaying the aligned images displayed by the display unit 32. Furthermore, the object may be identified by, for example, a user confirming and adjusting the results of image analysis of the aligned images. The slope identified by the object identification unit 34 is transmitted to, for example, the data management device 5 and can be used to create a report or the like.

[0074] The software blocks described above correspond to functional units realized by the CPU of each device executing the program of this embodiment and causing each piece of hardware to function. The functional units shown in each embodiment may be realized entirely by software, or some or all of them may be implemented as hardware that provides equivalent functions.

[0075] Furthermore, all of the above-mentioned functional means do not necessarily have to be included in the configuration shown in Fig. 10. Therefore, for example, the data acquisition device 9 may have functional means corresponding to the image synthesis unit 52, and the evaluation device 3 may have functional means corresponding to the image generation unit 53. Also, for example, in another preferred embodiment, each functional means may be realized by cooperation with multiple devices.

[0076] In the process of identifying a slope, a composite image sufficient for identifying the slope must be displayed appropriately. For example, it is difficult for a user to properly identify the slope on a display screen such as that shown in Figures 11 and 12. Figures 11 and 12 are diagrams showing examples of screens for performing a process of identifying a slope in the prior art.

[0077] For example, as shown in Fig. 11, among the multiple composite images Ph, Pm, and Pl, only one composite image provides information on only a portion of the object (in this embodiment, information on the height direction is insufficient), so the subject cannot be properly identified. Therefore, in such a case, it is difficult to identify the slope.

[0078] Furthermore, as shown in FIG. 12, simply displaying multiple composite images Ph, Pm, and Pl side by side on a single screen can result in misalignment between the composite images, making them difficult to see. For example, in FIG. 12, areas A, B, and C, indicated by dashed rectangles, are misaligned in the left-right direction of the image (i.e., the direction of travel of the vehicle), even though they depict the same object, reducing the visibility of the image and making it difficult to identify the slope. It is also possible to make the image easier to see by aligning and combining multiple composite images into a single image, but combining the entirety of multiple composite images requires combining even areas that do not include the object, which increases the processing load.

[0079] Therefore, in this embodiment, an aligned image is generated and displayed that prevents misalignment when composite images are displayed side by side, which makes it easier to identify slopes and improves user convenience.

[0080] The process of identifying a slope in this embodiment will be described below. Fig. 13 is a flowchart showing the process of identifying a slope in this embodiment. The state inspection system 1 of this embodiment starts the process from step S1000.

[0081] In step S1001, the photographing unit 92 photographs a partial image in accordance with the travel of the vehicle. Note that in step S1001, the photographing time acquisition unit 93 may acquire the photographing time in accordance with the photographing of the partial image, and the travel speed measurement unit 94 may measure the travel speed of the vehicle. The acquired photographing time and the measured travel speed can be associated with the partial image and can be transmitted to the data management device 5 via the communication unit 91, for example.

[0082] In step S1002, the image composition unit 52 stitches together the captured partial images to generate a composite image. The generated composite image may include metadata indicating the time at which the partial images constituting the composite image were captured, the speed at which the partial images were moved, and the like. The image composition unit 52 can generate composite images corresponding to the images captured by the multiple image capture units 92. That is, in step S1002, multiple composite images are generated.

[0083] Next, in step S1003, the image generation unit 53 generates an aligned image in which multiple composite images are arranged to identify the slope. Here, the image generation unit 53 of this embodiment generates the aligned image so that the positions of the multiple composite images are appropriately displayed based on the time at which the partial images that make up the composite image were captured. Note that in step S1003, the aligned image may also be generated so that the positions of the multiple composite images are appropriately displayed based on the vehicle's traveling speed.

[0084] Then, in step S1004, the display unit 32 displays a screen for performing processing to identify an object based on the aligned image generated in step S1003. This allows the user performing processing to identify the slope to view the aligned image in which multiple composite images are appropriately aligned.

[0085] In step S1005, the object identification unit 34 identifies the slope, for example, in response to a user operation. After that, in step S1006, the state inspection system 1 ends the processing.

[0086] By performing the processing described in Fig. 13, it is possible to avoid the difficulty in identifying the object due to an inability to properly grasp the photographed subject, and the increased processing load caused by generating a combined image (of the entire image including areas unnecessary for identifying the object). That is, by performing the processing in Fig. 13, it is possible to obtain a combined image of the object by combining only areas that identify the area in which the object is photographed, without combining images. Therefore, by generating and displaying an aligned image that prevents misalignment when composite images are displayed side by side, as in the processing shown in Fig. 13, it is possible to easily identify the object, thereby improving user convenience.

[0087] Here, correction of misalignment when arranging composite images will be described with reference to FIGS. 14 and 15. FIGS. 14 and 15 are diagrams illustrating misalignment of composite images. As shown in FIG. 14, a mobile body system 60 of this embodiment includes multiple image capturing units 7h, 7m, and 7l on a mobile body 6. However, due to space limitations, the image capturing units may be installed side by side in the direction of travel of the mobile body 6, as shown in FIG. 14(a). Furthermore, even when the image capturing units are arranged vertically, as shown in FIG. 14(b), the angles of the optical axes of the image capturing units do not necessarily match. In such cases, when the mobile body 6 arrives at a certain position, the partial images captured by the multiple image capturing units may not be the same.

[0088] For example, when capturing images of the letters "ABC" as shown in Figures 14(a) and 14(b), ideally, it would be desirable for multiple image capturing units 7h, 7m, and 7l to capture the same letters. However, in reality, the positions of the image capturing units are shifted relative to the traveling direction of the moving object, so as shown in Figure 14(c), image capturing unit 7h may capture a partial image of the letter "A," image capturing unit 7m may capture a partial image of the letter "B," and image capturing unit 7l may capture a partial image of the letter "C." This can cause a shift in the composite image, as described in Figure 12.

[0089] Therefore, the image generating unit 53 of this embodiment aligns the positions of the partial images captured at the same capture time, and generates an aligned image in which a plurality of composite images are arranged (step S1003).

[0090] For example, consider the case where there are composite images taken by photographing unit 7h, photographing unit 7m, and photographing unit 7l, as shown in FIG. 15(a). Here, the area indicated by the dashed rectangle in FIG. 15(a) represents partial images captured at time t. In this case, the partial images captured at time t among the composite image taken by photographing unit 7h, the partial images captured at time t among the composite image taken by photographing unit 7m, and the partial images captured at time t among the composite image taken by photographing unit 7l are aligned to generate an aligned image in which the composite images are arranged. In other words, by offsetting the partial images captured at the same time, an aligned image can be created in which the slope is easy to identify, as shown in FIG. 15(b).

[0091] 14(a), when a plurality of image capturing units are arranged side by side in the traveling direction of the moving object 6, an offset is applied based on the image capturing time, and then a further offset is applied based on the moving speed of the moving object 6 and the distance between each image capturing unit. That is, by offsetting the value obtained by dividing the distance between the image capturing units by the moving speed to the image capturing time, the positions of the partial images can be aligned and each composite image can be arranged, as shown in FIG. 15(b).

[0092] As explained in Fig. 15, by arranging the composite images so that the positions of the composite images in the moving direction of the moving object 6 are aligned with the partial images captured at the same time, the user can easily identify the slope. Here, the operation for identifying the slope will be explained with reference to Fig. 16. Fig. 16 is a diagram showing an example of a screen for performing the process of identifying the slope in this embodiment.

[0093] As shown in Figure 16, the screen displayed in this embodiment for performing the process of identifying the slope includes multiple composite images in which the partial images are aligned and arranged, as shown in Figure 15(b).

[0094] In the screen shown in FIG. 16, a marker indicating the start point of the slope (hereinafter referred to as a start marker Ms) and a marker indicating the end point of the slope (hereinafter referred to as an end marker Me) are superimposed on the composite image. The user can set the start point and end point of the slope by moving the positions of the start marker Ms and the end marker Me. The start marker Ms and the end marker Me can be moved to any position on the image by operating the pointer. The user can identify the slope by placing the start marker Ms and the end marker Me on the boundary between a slope area and a non-slope area in the image. Here, the start marker Ms and the end marker Me can be moved by dragging and dropping with a mouse, swiping on a touch panel, pressing a direction key, or the like. Note that, as shown in FIG. 16, when the slope is identified by a user's operation, the positions of the slope identified by image analysis of the aligned image may be set as the initial positions of the start marker Ms and the end marker Me. This allows the user to identify the slope by adjusting the positions of the start marker Ms and the end marker Me, making it easier to perform the process of identifying the slope.

[0095] The method for identifying the slope is not limited to the method shown in Fig. 16, which uses a start marker Ms and an end marker Me to indicate the boundary between the slope area and the non-slope area in the image. For example, a method may be used in which the slope area in the image is surrounded by a frame so that the boundary between the slope area and the non-slope area in the image is clear. Even in this case, the part of the frame that indicates the boundary between the slope area and the non-slope area in the image is referred to as a marker.

[0096] After placing the start marker Ms and the end marker Me in appropriate positions, the user can identify the slope by pressing the confirm button on the screen of FIG. 16. Here, since the multiple composite images are associated with each other by the time at which the partial images were captured, the slope can be identified by the capture time. When the slope is identified by the user's operation, images are cut out at the start and end positions of the slope in each of the multiple composite images. Note that the cutout of the multiple composite images can be performed based on the capture time, for example. The image of the slope can be identified by cutting out the partial image in which the start marker Ms and the end marker Me are placed and the partial image that was captured at the same time. Then, the cutout composite images are combined in the height direction of the slope to generate a single image that includes the slope area.

[0097] The generated image of the slope is inserted at a predetermined position in a report called a report. An example of the report will now be described with reference to FIG. 17. FIG. 17 is a diagram showing an example of a report including an image of the slope identified in this embodiment. As shown in FIG. 17, the report includes a combined image of the identified slope area, as well as the slope's extension, height, and starting and ending points (latitude and longitude coordinates).

[0098] According to the embodiments of the present invention described above, it is possible to provide an information processing system, an information processing device, a method, and a program for identifying an area in an image captured while moving that contains a desired subject.

[0099] Each function of the above-described embodiments of the present invention can be realized by a device-executable program written in C, C++, C#, Java (registered trademark), etc., and the program of this embodiment can be stored and distributed on a device-readable recording medium such as a hard disk drive, CD-ROM, MO, DVD, flexible disk, EEPROM (registered trademark), EPROM, etc., and can also be transmitted over a network in a format that can be used by other devices.

[0100] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by electronic circuits, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each of the above-described functions.

[0101] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above-described embodiments, and any embodiment that can be conceived by a person skilled in the art is included in the scope of the present invention as long as it exhibits the functions and effects of the present invention. [Explanation of symbols]

[0102] 1...condition inspection system, 3...evaluation device, 4...evaluation system, 5...data management device, 6...mobile body, 7...imaging system (imaging device), 7h, 7m, 7l...imaging unit, 8...sensor device, 8a...distance sensor, 8b...GNSS sensor, 8c...angle sensor, 9...data acquisition device, 31...communication unit, 32...display unit, 33...operation unit, 34...object identification unit, 51...communication unit, 52...image synthesis unit, 53...image generation unit, 54...image combination unit, 55...report creation unit, 56...storage unit, 60...mobile body system, 91...communication unit, 92...imaging unit, 93...imaging time acquisition unit, 94...movement speed measurement unit, 100...communication network, 301...CPU, 302...ROM, 303...RAM, 305...HDD controller, 306...display, 308...external device connection I / F, 309...network I / F, 310...bus line, 311...keyboard, 312...pointing device, 314...DVD-RW drive, 315...recording media, 316...media I / F, 900...controller, 901...imaging device I / F, 902...sensor device I / F, 910...bus line, 911...CPU, 912...ROM, 913...RAM, 915...HDD controller, 916...network I / F, 918...DVD-RW drive, 921...recording media, 922...media I / F, 923...external device connection I / F, 924...timer, 930...external PC, 1100...terminal device, 1200...terminal device [Prior art documents] [Patent documents]

[0103] [Patent Document 1] Patent No. 4551990

Claims

1. an image synthesis means for generating a synthetic image by stitching together partial images captured while moving in the direction of movement; a generating means for generating an image in which the plurality of composite images are arranged such that partial images captured at the same time are aligned in the direction of movement; An information processing system comprising:

2. a display means for displaying the image generated by the generation means; The information processing system of claim 1 , further comprising:

3. a specifying means for specifying the object by displaying, on the display means, markers indicating the start and end points of the boundary between the object area and the non-object area in the composite image; The information processing system of claim 2 further comprising:

4. an image combining means for generating a combined image by joining together in the height direction a plurality of the composite images including the object identified by the identifying means; The information processing system of claim 3 further comprising:

5. A means for creating a record including the combined image. The information processing system of claim 4 further comprising:

6. the generating means generates an image in which partial images captured at the same time are aligned in the direction of movement based on the speed at which the partial images move when they are captured. The information processing system according to claim 1 .

7. an image synthesis means for generating a synthetic image by stitching together partial images captured while moving in the direction of movement; a generating means for generating an image in which the plurality of composite images are arranged such that partial images captured at the same time are aligned in the direction of movement; An information processing device comprising:

8. generating a composite image by stitching together the partial images captured while moving in the direction of movement; generating an image in which the plurality of composite images are arranged such that partial images captured at the same time are aligned in the direction of movement; A method comprising:

9. A program executed by an information processing device, the information processing device an image synthesis means for generating a synthetic image by stitching together partial images captured while moving in the direction of movement; A generating means for generating an image in which the plurality of composite images are arranged so that partial images taken at the same time are aligned in the direction of movement. A program that functions as a

Images