Information processor, information processing method, and information processing program

Abstract

To provide an information processor, an information processing method, and an information processing program that estimate an object that is a difference between a plurality of images.SOLUTION: An information processor includes: an acquisition unit that acquires image information recorded at a plurality of different times; and an estimation unit that estimates, on the basis of a trained model that has learned object images captured at the same location at a plurality of different times and objects that change between the object images, and on each of a plurality of pieces of image information acquired by the acquisition unit, an object that changes in the image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there exists a device that captures images of the Earth's surface using imaging means mounted on multiple flying objects flying on different orbits, and detects changed areas by comparing past images captured in the past with newly captured images (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-167343 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when using a trained model to detect changed areas between a past image and a new image, the estimated results will differ depending on what and how is learned during the learning process to generate the trained model.

[0005] The present disclosure provides an information processing device, an information processing method, and an information processing program that estimate an object that is a difference between a plurality of images. [Means for solving the problem]

[0006] An information processing device of one embodiment includes an acquisition unit that acquires image information recorded at multiple different times, a trained model that has learned about target images captured at the same location at multiple different times and objects that change between the target images, and an estimation unit that estimates objects that change in the images based on each of the multiple image information acquired by the acquisition unit. Effect of the Invention

[0007] The information processing device, information processing method, and information processing program disclosed herein can estimate an object that is a difference between a plurality of images. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing device according to an embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating an information processing device according to an embodiment. [Diagram 3] 1A is a diagram illustrating an example of a mesh, and FIG. 1B is a diagram illustrating meshes A2 and B2 that correspond to each other in position among a plurality of meshes obtained by dividing an image A1 and an image B1. [Figure 4] 1A is a diagram for explaining an example of distortion correction, in which (A) explains the positional relationship of the camera with respect to a comparison target, (B) explains distortion correction of an image captured from the south side of the comparison target, and (C) explains distortion correction of an image captured from the north side of the comparison target. [Diagram 5] 1 is a diagram for explaining an example of a target image used when generating a trained model, in which (A) shows an example of the target image at a certain time, and (B) shows an example of the target image at a time later than the time shown in (A). [Figure 6] 1 is a flowchart illustrating an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment will be described below.

[0010] [Overview of information processing device 100] First, an overview of an information processing device 100 according to an embodiment will be described. FIG. 1 is a diagram illustrating an information processing device 100 according to an embodiment.

[0011] The information processing device 100 may be configured as, for example, an estimation device that estimates an object 400 that has changed between a plurality of images captured at different times. The information processing device 100 is not limited to the device of the above example, and may be configured as various devices. The information processing device 100 may be a computer such as a server, a desktop, a laptop, a tablet, or a smartphone.

[0012] The information processing device 100 acquires a plurality of pieces of image information. Each of the plurality of pieces of image information may be information on an image recorded (for example, captured and generated) at a different time. The image information may also be information on an image captured from an aircraft 200 flying in the sky. The aircraft 200 may be, for example, an artificial satellite, a drone, an airplane, a helicopter, an airship, a balloon, a glider, or the like. That is, the aircraft 200 may be, for example, an artificial satellite, an aircraft, an unmanned aerial vehicle, or the like. The image information may be map image information on a map image. The images recorded at different times may be images recorded at a predetermined interval (first predetermined period). The predetermined period (first predetermined period) may be an interval of a day (predetermined day) such as one day, an interval of a month (predetermined month) such as one month, or an interval of a year (predetermined year) such as one year.

[0013] 1 shows an example in which the information processing device 100 acquires image information (satellite image information) generated by capturing images of the ground by a camera 210 mounted on an aircraft 200 (artificial satellite) via a server 300. The multiple pieces of image information may be image information (two pieces of image information) relating to an image 500 captured at time t1 and an image 500 captured at time t2, or may be three or more pieces of image information captured at different times (times).

[0014] The information processing device 100 acquires a trained model that learns target images 600 (610, 620) (see FIG. 5) (plurality of target images 600) captured at the same point at different times, and objects 640 (e.g., a building 641a with a change and a road 642a with a change) (see FIG. 5) that have changes between the target images 600 (610, 620) (plurality of target images 600). The information processing device 100 may generate the trained model on its own device, or may acquire a trained model generated externally.

[0015] The target image 600 used in generating the trained model may be an image captured from an aircraft 200 flying in the sky. The target image 600 may be a map image. The target image 600 may be captured at different times and recorded at a predetermined interval (second predetermined period). The predetermined period (second predetermined period) may be an interval of a day (predetermined day), such as one day, an interval of a month (predetermined month), such as one month, or an interval of a year (predetermined year). The second predetermined period may be the same period as the above-mentioned first predetermined period, or may be a period different from the first period. The same point may be rephrased as the same area. The same area may be a completely identical area, or may be a substantially identical area. As an example, when there is a first object image 610 (600) captured at a first time period and a second object image 620 (600) captured at a second time period different from the first time period, if there is an area (area on the ground) that overlaps at least partially between a first area recorded in the first object image 610 and a second area recorded in the second object image 620, the object images 600 may be said to be captured at the “same spot.”

[0016] The object 640 may be various objects on the ground, etc. As a specific example, the object 640 may be at least one of a road 642 and a building 641, etc. The object 640 with a change may be a state before and after the change of the object 640. As a specific example, if the object 640 is a road 642, the changed object 640 may be before the construction of the road 642 (642a) (before the change) and after the construction of the road 642 (642a) (after the change). Similarly, as a specific example, if the object 640 is a road 642, the changed object 640 may be before the demolition of the road 642 (or before the route correction work) (before the change) and after the demolition of the road 642 (or after the route correction work) (after the change). Similarly, as a specific example, if the object 640 is a building 641, the changed object 640 may be before the construction of the building 641 (641a) (before the change) and after the construction of the building 641 (641a) (after the change). Similarly, as a specific example, if the object 640 is a building 641, the changed object 640 may be before the demolition of the building 641 (or before the reconstruction of the building 641 (for example, before an extension)) (before the change) and after the demolition of the building 641 (or after the reconstruction of the building 641 (for example, after an extension)) (after the change).

[0017] Based on the trained model and each of the images of the plurality of image information, the information processing device 100 estimates the object 400 that has a change appearing in the image. That is, the information processing device 100 inputs the plurality of image information to the trained model, and estimates the object 400 (for example, the road 421 and the building 411) that has a change between the images based on each of the plurality of image information.

[0018] In the example shown in FIG. 1, compared to the image 500 captured at time t1, an object 400 such as a road 421 and a building 411 has been constructed in the image 500 captured at time t2. The information processing device 100 may estimate the constructed road 421 and the building 411 as an object 400 with changes, and output the estimated object 400 in various forms so that it can be seen (by adding a diagonal line 401, etc., in the example shown in FIG. 1). Note that an example of the output is not limited to the example shown in FIG. 1, and the estimated result may be output in various forms (for example, various file formats and various output contents (interfaces) that are not in a file format, etc.) so that the estimated result can be seen.

[0019] [Details of the information processing device 100] Next, the information processing device 100 according to an embodiment will be described in detail. FIG. 2 is a block diagram illustrating the information processing device 100 according to an embodiment.

[0020] The information processing device 100 includes, for example, a communication unit 121, a storage unit 122, a display unit 123, and a control unit 110. The communication unit 121, the storage unit 122, and the display unit 123 may be an embodiment of an output unit. The control unit 110 includes, for example, an acquisition unit 111, a division unit 112, an adjustment unit 113, a learning unit 114, an estimation unit 115, and an output control unit 116. The control unit 110 may be configured, for example, by an arithmetic processing device of the information processing device 100. The control unit 110 (for example, an arithmetic processing device) may realize the functions of each unit (for example, the acquisition unit 111, the division unit 112, the adjustment unit 113, the learning unit 114, the estimation unit 115, and the output control unit 116) by, for example, appropriately reading and executing various programs stored in the storage unit 122.

[0021] The communication unit 121 is, for example, a communication interface capable of transmitting and receiving various information to and from a device (external device) outside the information processing device 100. The external device may be a server 300, a user terminal (not shown), etc. The user terminal is a terminal used by a user of the information processing device 100. As an example, the user terminal may be a desktop, a laptop, a tablet, a smartphone, etc.

[0022] The storage unit 122 may store, for example, various information and programs. Examples of the storage unit 122 may be a memory, a solid state drive, a hard disk drive, etc. Note that the storage unit 122 may be, for example, a storage area and a server on a cloud.

[0023] The display unit 123 is, for example, a display capable of displaying various characters, symbols, images, and the like.

[0024] The acquisition unit 111 acquires image information recorded at multiple different times. The acquisition unit 111 may acquire multiple pieces of image information from an external device, for example, via the communication unit 121. The external device may be, for example, a server 300 and a camera 210. The server 300 is a device that stores image information. The camera 210 is a device that generates image information, and may be mounted on the flying object 200.

[0025] Each of the multiple pieces of image information may be information on an image recorded (e.g., captured and generated) at a different time. The image information may also be information on an image captured from an aircraft 200 flying in the sky. The aircraft 200 may be, for example, an artificial satellite, a drone, an airplane, a helicopter, an airship, a balloon, a glider, etc. That is, the aircraft 200 may be, for example, an artificial satellite, an aircraft, an unmanned aerial vehicle, etc. The image information may be map image information on a map image. The images recorded at different times may be images recorded at a predetermined interval (first predetermined period). The predetermined period (first predetermined period) may be, for example, an interval of a day (predetermined day) such as one day, an interval of a month (predetermined month) such as one month, or an interval of a year (predetermined year) such as one year.

[0026] Fig. 3 is a diagram for explaining an example of a mesh 510. Fig. 5(A) is a diagram showing an example of a plurality of meshes 510 obtained by dividing an image 500, and Fig. 5(B) is a diagram for explaining meshes A2 and B2 (510) that have a positional correspondence relationship among a plurality of meshes 510 obtained by dividing an image A1 (500) and an image B1 (500).

[0027] 3(A), the dividing unit 112 divides each of the images 500 based on the plurality of pieces of image information acquired by the acquiring unit 111 into a plurality of meshes 510. The dividing unit 112 divides the image 500 into a plurality of meshes 510 (regions). The planar shape of the meshes 510 may be various shapes including a rectangle, a triangle, a polygon, a circle, and the like. The dividing unit 112 may divide each of the plurality of pieces of image information into meshes 510 that have the same (approximately the same) positional range. That is, as shown in an example in FIG. 3(B), when there are two image information (images A1, B1 (500)), the division unit 112 may divide each of the images A1, B1 (500) into a plurality of meshes 510 so that a mesh A2 (each of the plurality of meshes 510) at a certain position when the image A1 is divided into the plurality of meshes 510 and a mesh B2 (each of the plurality of meshes 510) at a certain position when the image B1 is divided into the plurality of meshes 510 have the same (approximately the same and corresponding, etc.) positional range (area) (so that mesh A2 and mesh B2 have the same (approximately the same) corresponding relationship in terms of positional range).

[0028] The dividing unit 112 may be capable of adjusting the granularity of the mesh 510. That is, the dividing unit 112 may be capable of adjusting the planar size of the mesh 510. For example, when a relatively large number of objects (e.g., structures on the ground) (or targets 400) are recorded in an image (at least one of a plurality of images) based on image information, the division unit 112 may relatively reduce the planar size (granularity) of the mesh 510. Also, for example, when a relatively small number of objects (e.g., structures on the ground) (or targets 400) are recorded in an image (at least one of a plurality of images) based on image information, the division unit 112 may relatively increase the planar size (granularity) of the mesh 510. In other words, for example, when a relatively large number of objects (e.g., structures on the ground) (or target objects 400) are recorded in an image based on the image information (at least one of the multiple images), the division unit 112 may reduce the planar size (granularity) of the mesh 510 compared to when a relatively small number of objects (e.g., structures on the ground) (or target objects 400) are recorded in an image based on the image information (at least one of the multiple images).

[0029] Alternatively, the dividing unit 112 may be capable of adjusting the granularity of the mesh 510 depending on whether the change in the object 400 to be estimated is general or detailed (usability), as described below. That is, when estimating a rough change in the object 400, the dividing unit 112 may relatively increase the granularity (planar size) of the mesh 510. Also, when estimating a detailed change in the object 400, the dividing unit 112 may relatively decrease the granularity (planar size) of the mesh 510.

[0030] The adjustment unit 113 may align images based on a plurality of pieces of image information acquired by the acquisition unit 111, that is, adjust to align the position of the object 400 recorded in the images. The adjustment unit 113 may align the positions of the plurality of images by using position information recorded in the image information, for example, or may align the positions of the plurality of images such that the objects 400 recorded in each of the plurality of images overlap.

[0031] The adjustment unit 113 adjusts the color tone and inclination of each of the images based on the plurality of pieces of image information acquired by the acquisition unit 111 so that they become the same.

[0032] The adjustment unit 113 may, for example, compare the color tones (e.g., RGB or CMYK, etc.) of the multiple images and adjust both or one of the multiple images so that the values ​​indicating the color tones of the multiple images are the same (or approximately the same). In this case, the adjustment unit 113 may adjust the color tones of the multiple images, for example, the difference in at least one color tone component of RGB or CMYK, etc. (e.g., at least one of R, G, and B, or at least one of C, M, Y, and K), so that the difference falls within a preset range. When using the example described above, that is, when there are two image information (images A1, B1), the adjustment unit 113 may adjust at least one of the R value of image A1 and the R value of image B1 so that the difference between at least one of the color tone components of image A1 (as an example, the R value) and at least one of the color tone components of image B1 (as an example, the R value) falls within a predetermined range (the same applies to the other color tone components GB and CMYK).

[0033] Furthermore, the adjustment unit 113 may adjust the contrast and brightness of at least one of the images based on the multiple pieces of image information, thereby adjusting the color tones of the images so that they become the same (or substantially the same). The adjustment unit 113 is not limited to the above-mentioned example, and may adjust the color tones of the images to be the same (approximately the same) using various methods.

[0034] In addition, the adjustment unit 113 may compare each of the images based on multiple image information and rotate at least one of the multiple images clockwise or counterclockwise to adjust the inclination of each image in a plane related to rotation to be the same (approximately the same). In this case, the adjustment unit 113 may identify one or more comparison objects, such as buildings, parks, roads, and rivers, that appear in each of the multiple images, compare the positions of each of the identified comparison objects within the images, and rotate the images so that the positions of the comparison objects in each of the multiple images are the same (or approximately the same), thereby adjusting the rotational inclination of each image to be the same (approximately the same).

[0035] Fig. 4 is a diagram for explaining an example of distortion correction. Fig. 4(A) explains the positional relationship of camera 210 with respect to comparison object 440, Fig. 4(B) explains distortion correction of image A1 captured from the south (S) side of comparison object 440, and Fig. 4(C) explains distortion correction of image B1 captured from the north (N) side of comparison object 440.

[0036] In addition, the adjustment unit 113 may compare the angle (imaging angle) between the ground surface and the imaging direction of the camera 210 in each of the images 500 based on multiple image information, and perform distortion correction on at least one of the multiple images 500, thereby adjusting the inclination of each image 500 with respect to the imaging angle to be the same (approximately the same). As shown in the example of Figure 4 (A), consider the case where there are two image information (images A1, B1 (500)), and image A1 is captured by camera 210 (210a) located south of comparison object 440, and image B1 is captured by camera 210 (210b) located north of comparison object 440. In the example shown in Figure 4(A), the adjustment unit 113 expands the north side of image A1 in the width direction (east-west (EW) direction) and shrinks the south side of image A1 in the width direction (east-west direction) to obtain image A11 (500) shown by the dashed line after distortion correction (see Figure 4(B)). Similarly, in the example shown in Figure 4(A), the adjustment unit 113 expands the south side shown in image B1 in the width direction (east-west direction) and shrinks the north side shown in image B1 in the width direction (east-west direction) to obtain image B11 (500) shown by the dashed line after distortion correction (see Figure 4(C)). Thereby, the adjustment unit 113 corrects image distortion according to the imaging angle, and adjusts the inclinations of the images 500 with respect to the imaging angle to be the same (approximately the same) (even if the imaging angle is east-west).

[0037] Fig. 5 is a diagram for explaining an example of a target image 600 used when generating a trained model. Fig. 5(A) shows an example of a target image 600 (first target image 610) at a certain time (first time point), and Fig. 5(B) shows an example of a target image 600 (second target image 620) at a time (second time point) later than the time (first time point) shown in Fig. 5(A).

[0038] The learning unit 114 generates a learned model by learning target images 600 captured at the same location at multiple different times and objects 640 that change between the target images 600 (e.g., a building 641a that changes and a road 642a that changes).

[0039] The target image 600 used in generating the trained model may be an image captured from an aircraft 200 flying in the sky. The aircraft 200 may be, for example, an artificial satellite, a drone, an airplane, a helicopter, an airship, a balloon, a glider, etc., as in the above-mentioned case. That is, the aircraft 200 may be, for example, an artificial satellite, an aircraft, an unmanned aerial vehicle, etc. The target image 600 may be a map image. The multiple different times at which the target image 600 is captured may be images recorded at a predetermined interval (second predetermined period). The predetermined period (second predetermined period) may be an interval of a day (predetermined day), such as one day, an interval of a month (predetermined month), such as one month, or an interval of a year (predetermined year), such as one year. The second predetermined period may be the same period as the above-mentioned first predetermined period, or may be a period different from the first period. The same point may be rephrased as the same area. The same area may be a completely identical area or a substantially identical area in terms of a position (location range). As an example, when there is a first object image 610 captured at a first time period and a second object image 620 captured at a second time period different from the first time period, if there is an area (area on the ground) that overlaps at least partially between a first area recorded in the first object image 610 and a second area recorded in the second object image 620, the object images 600 may be said to be captured at the "same spot."

[0040] The object 640 may be various objects on the ground. As a specific example, the object 640 may be at least one of a road 642 and a building 641. The object 640 (641, 642a) with a change may be a state before and after the change of the object 640. As a specific example, if the object 640 is a road 642, the changed object 640 may be before the construction of the road 642 (642a) (before the change) and after the construction of the road 642 (642a) (after the change). Similarly, as a specific example, if the object 640 is a road 642, the changed object 640 may be before the demolition of the road 642 (or before the route correction work) (before the change) and after the demolition of the road 642 (or after the route correction work) (after the change). Similarly, as a specific example, if the object 640 is a building 641, the changed object 640 may be before the construction of the building 641 (641a) (before the change) and after the construction of the building 641 (641a) (after the change). Similarly, as a specific example, if the object 640 is a building 641, the changed object 640 may be before the demolition of the building 641 (or before the reconstruction of the building 641 (for example, before an extension)) (before the change) and after the demolition of the building 641 (or after the reconstruction of the building 641 (for example, after an extension)) (after the change).

[0041] In the example shown in Fig. 5(A), the solid lines in the target image 600 indicate constructed objects 640 (roads 642 and buildings 641), and the dashed lines indicate unconstructed objects 640 (roads 642 and buildings 641). The area shown by dashed lines in Fig. 5(A) may be, for example, the ground. In the example shown in FIG. 5(B), the object 640 before construction, shown by the dashed line in FIG. 5(A), is shown as having been constructed subsequently (already constructed). The unconstructed building 641 and the unconstructed road 642 shown by dashed lines at the time shown in Figure 5(A) have been constructed (completed) at the subsequent time shown in Figure 5(B), and therefore become a changed building 641a and a changed road 642a, i.e., a changed object 640.

[0042] The learning unit 114 may learn a plurality of target images 600, an object 640 in the target image 600 before a change, and the object 640 after a change, with annotation 650 attached to the side showing the ground (the side closer to the ground). As a specific example, when the target object 640 is a road 642 (or a building 641, etc.), if the ground is recorded in the target image 600 at a point before the construction of the road 642 (or the building 641, etc.) (before the change) and the ground is not recorded in the target image 600 after the construction of the road 642 (or the building 641, etc.) (after the change), the learning unit 114 may attach the annotation 650 to the ground before the construction of the road 642 (or the building 641, etc.). Similarly, as a specific example, in the case where the target object 640 is a road 642 (or a building 641, etc.), if the ground is not recorded in the target image 600 (the road 642 (or the building 641, etc.) is recorded) (before the change) at a point before the demolition of the road 642 (or the building 641, etc.) and the ground is recorded in the target image 600 after the demolition of the road 642 (or the building 641, etc.) (after the change), the learning unit 114 may attach the annotation 650 to the ground after the demolition of the road 642 (or the building 641, etc.). In the example shown in Figure 5 (A), the learning unit 114 may add an annotation 650 shown with diagonal lines to the position of the pre-construction object 640 (641a, 642a) shown with dashed lines, i.e., the position of the ground in the target image 600 (610). In the above-mentioned case, the learning unit 114 has been described as attaching the annotation 650 to the ground, but the present invention is not limited to this example. The annotation 650 may be attached to the side where the object 640 is present, that is, the road 642 and the building 641 after construction, and the road 642 and the building 641 before demolition (the side where the ground is not photographed (the side farther from the ground)).

[0043] In addition, without being limited to the case illustrated in FIG. 5(A), the learning unit 114 may annotate the mesh 510 in which the object 640 with a change (a building 641a with a change and a road 642a with a change) is present. As an example, when the object 640 with a change is present in the meshes A2 and B2 (510) illustrated in FIG. 3(B), the annotation may be added to one of the meshes A2 and B2. That is, the learning unit 114 may generate a learned model by learning a plurality of target images 600 (a plurality of meshes obtained by dividing the target image 600 into a plurality of meshes) including the object 640 with a change, and the object 640 with a change (annotated mesh). Note that the object 640 to be learned (the object to which annotation is added) is not limited to structures such as the building 641 and the road 642, but may be the shoreline of a river, a lake, a coast, or the like.

[0044] As an example, the learning unit 114 may acquire changes in the object 640 (e.g., the building 641 and the road 642, etc.) by using map information in which the object 640, such as the building 641 and the road 642, is recorded, that is, map information of the same or different scales and the same or different purposes (e.g., road maps, narrow street maps, and residential maps (thematic maps), and topographical maps (basic maps) in a chronological order (different published versions), such as before and after in time. The learning unit 114 may acquire changes in the object 640 (e.g., the building 641 and the road 642, etc.) by using attributes of a shape file and a GeoJSON (geospatial data exchange format) file, etc., as a specific example of the map information. The learning unit 114 may acquire changes in the object 640 (for example, the building 641 and the road 642, etc.) via the communication unit 121 or from an external memory (not shown), etc. In this case, the learning unit 114 may acquire a shape file, a GeoJSON file, etc. via the communication unit 121 or from an external memory (not shown), etc. The learning unit 114 may acquire changes in the object 640 (for example, the building 641 and the road 642, etc.) by comparing the shape file, the GeoJSON file, etc. in time series. This allows the learning unit 114 to automatically identify points where the object 640 did not exist (points before construction, points after demolition, points before renovation, etc.). Furthermore, the learning unit 114 acquires a plurality of target images 600 in a time series including a point where the object 640 was not present and the same point where the corresponding object 640 is present, and in a target image among the plurality of target images 600 in which the point where the object 640 was not present is recorded, an annotation 650 is added to the point. The learning unit 114 learns the acquired multiple target images 600 in a time series (including the target images 600 related to the images to which the annotations 650 are attached) to generate a learned model.

[0045] That is, as an example, the learning unit 114 may perform the following processes (1) to (5). (1) The learning unit 114 acquires map information (or object information (such as a shape file and a GeoJSON file)) that includes the same location at multiple different times. (2) The learning unit 114 identifies objects 640 (641a, 642a, etc.) that have changed, based on object information (shape file, GeoJSON file, etc.) related to the objects 640 recorded in the map information. (3) The learning unit 114 records the position of the identified changed object 640 (641a, 642a, etc.), and acquires a target image 600 (a plurality of target images 600) that records the object 640 before and after the change. (4) The learning unit 114 adds an annotation 650 to a position where the object 640 is not present (the area which is the ground shown by the dashed line in FIG. 5(A)) in the object image 600 (610) which records the before and after images of the identified object 640 (641a, 642a, etc.) before and after the change, where the object 640 is not present (the area which is the ground shown by the dashed line in FIG. 5(A)). (5) The learning unit 114 learns the target image 600 (610) with the annotation 650 and the target image 600 (620) without the annotation 650 (the target image 600 recording the image before and after the change in the identified object 640, in which the object 640 is present) to generate a trained model. That is, the learning unit 114 learns the target images 600 (610, 620) captured at the same location at multiple different times and the object 640 (641a, 642a) that has changed between the target images 600 (610, 620) to generate a trained model. This enables the learning unit 114 to automatically perform learning and generate a learned model. In addition, in the above-mentioned (4), the learning unit 114 may add an annotation 650 to the position where the object 640 is located in the object image 600 (610) that records the state before and after the change in the identified object 640 (641a, 642a, etc.) in which the object 640 is located.

[0046] The learning unit 114 may be realized as a function of the control unit 110, or may be realized as a function (learning device) (not shown) of a device (external device) external to the information processing device 100. When the trained model is generated in a learning device external to the information processing device 100, the control unit 110 may acquire the trained model from the external device via the communication unit 121.

[0047] The estimation unit 115 estimates the object 400 (411, 421) having a change appearing in the image 500 based on a trained model that has learned the target images 600 (610, 620) captured at the same point at multiple different times and the object 640 (641a, 642a) having a change between the target images 600 (610, 620) and each of the multiple image information images 500 acquired by the acquisition unit 111. The estimation unit 115 may also estimate the object 400 (411, 421) having a change based on the target image 600, the trained model that has learned the object 640 before the change in each of the target images 600 and the object 640 after the change, which is the object 640 (641a, 642a) with the annotation 650 attached to one of the side where the ground is captured and the side where the ground is not captured, and the image 500 based on the multiple image information. That is, the estimation unit 115 inputs multiple pieces of image information into the trained model trained by the training unit 114 as described above, and estimates objects 400 (e.g., roads 421 and buildings 411) that change between images 500 based on each of the multiple pieces of image information.

[0048] The estimation unit 115 may estimate the object 400 having a change based on the multiple images adjusted by the adjustment unit 113 and the trained model. That is, the multiple images in which the color tone and inclination have been adjusted (made comparable) between the multiple images may be input to the trained model, and the object 400 having a change recorded in the multiple images may be estimated.

[0049] The estimation unit 115 may estimate the object 400 having a change for each mesh 510 based on the meshes 510 whose positions correspond in each image based on a plurality of image information and the learned model. When using the example shown in FIG. 3(B) described above, that is, when there are two pieces of image information (images A1, B1), if a mesh A2 (510) in the image A1 and a mesh B2 (510) in the image B1 have a corresponding relationship in terms of position range, the estimation unit 115 estimates the object 400 having a change that appears in the meshes A2, B2 (510) based on the meshes A2, B2 (510) and the learned model.

[0050] As an example of the above-described embodiment, the information processing device 100 (for example, the acquisition unit 111) acquires image information captured from the sky at multiple different times. As a specific example, the image information may be satellite image information captured by an artificial satellite. The information processing device 100 estimates changes in the road 420, the building 410, and the shoreline (object 400) recorded in the image information (satellite image information) (changes between the object 400 (any point) at a certain time t1 and the same object 400 (same point) at a subsequent time t2), for example, changes between before and after construction of the object 400 (or before and after demolition) (or before and after change) using a trained model. That is, the information processing device 100 (estimation unit 115) estimates the road 421, building 411, and shore line (object 400) that have changed and appear in the image 500 based on a trained model that has trained the target images 600 captured at the same point at multiple different times, the road 642a, the building 641a, and the shore line (object 640) that have changed between the multiple different times in the area (the area where the imaging ranges of the target images 600 overlap) corresponding to the imaging area of ​​each of the target images 600, and each image 500 of the multiple image information acquired by the acquisition unit 111. That is, the information processing device 100 (estimation unit 115) estimates at least one of the object 400 before the change and the object 400 after the change that appear in the image 500.

[0051] The output control unit 116 controls the output unit to output the result estimated by the estimation unit 115, that is, as an example, a mark that identifies the object 400 on the image. The output unit may be, for example, a communication unit 121, a storage unit 122, a display unit 123, etc. That is, the output control unit 116 controls the communication unit 121 to transmit the result estimated by the estimation unit 115 to an external device. The external device may be, for example, the server 300 and a user terminal (not shown). The output control unit 116 controls the storage unit 122 to store the result of estimation by the estimation unit 115, for example. The output control unit 116 controls the display unit 123 to display the result of estimation by the estimation unit 115, for example.

[0052] [Information processing method] Next, an information processing method according to an embodiment will be described. FIG. 6 is a flowchart illustrating an information processing method according to an embodiment.

[0053] In step ST101, the learning unit 114 acquires map information including the same point at a plurality of different times. The learning unit 114 identifies an object 640 that has changed based on object information related to the object 640 (for example, a road, a building, a shoreline, etc.) recorded in the map information. The learning unit 114 records the position of the identified object 640 that has changed, and acquires a target image 600 of the object 640 that has been captured before and after the change. The learning unit 114 adds an annotation 650 to a position where the object 640 does not exist in the target image 600 that records the image of the identified object 640 before and after the change, where the object 640 does not exist. The learning unit 114 learns the target image 600 with the annotation 650 and the target image 600 without the annotation 650 to generate a learned model. That is, the learning unit 114 generates a trained model that has learned target images 600 captured at the same location at multiple different times, and target objects 640 that change between the target images 600. In addition, the learning unit 114 generates a learned model that learns the target image 600, an object 640 before the change in each of the target images 600, and the object 640 after the change, where an annotation 650 is added to the side that shows the ground.

[0054] In step ST102, the acquisition section 111 acquires a plurality of pieces of image information recorded at different times.

[0055] In step ST103, the adjustment unit 113 adjusts the color tones and inclinations of the images based on the multiple pieces of image information acquired in step ST102 so that they become the same.

[0056] In step ST104, the dividing unit 112 divides each of the images based on the multiple pieces of image information acquired in step ST102 (or the image information whose color tone and inclination have been adjusted in step ST103) into multiple meshes 510. The dividing unit 112 may be capable of adjusting the granularity of the meshes 510.

[0057] In step ST105, the estimation unit 115 estimates the object 400 having a change appearing in the image based on the trained model generated in step ST101 and each of the images of the multiple pieces of image information acquired in step ST102. The estimation unit 115 may estimate the object 400 having a change based on the multiple images adjusted in step ST103 and the trained model. The estimation unit 115 may estimate the object 400 that changes for each mesh 510 based on the meshes 510 (each of the multiple meshes 510 divided in step ST104) whose positions correspond in each image based on multiple image information, and based on the learned model.

[0058] After the process of step ST105, the output control unit 116 may control the output unit to output the result of the estimation performed in step ST105. The output unit may be, for example, the communication unit 121, the storage unit 122, the display unit 123, etc.

[0059] [Functions and circuits] Next, the functions and circuits of the above-mentioned information processing device 100 will be described. Each unit of the information processing device 100 may be realized as a function of a computer arithmetic processing device or the like. That is, the acquisition unit 111, division unit 112, adjustment unit 113, learning unit 114, estimation unit 115, and output control unit 116 (control unit 110) of the information processing device 100 may be realized as an acquisition function, division function, adjustment function, learning function, estimation function, and output control function (control function) by a computer arithmetic processing device or the like, respectively. The information processing program can cause a computer to realize each of the above-mentioned functions. The information processing program may be recorded in a non-transitory computer-readable storage medium, such as a memory, a solid-state drive, a hard disk drive, or an optical disk. The storage medium may be rephrased as a non-transitory computer-readable medium that stores the information processing program. As described above, each unit of the information processing device 100 may be realized by an arithmetic processing device of a computer or the like. The arithmetic processing device or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the information processing device 100 may be realized as a circuit constituting the arithmetic processing device or the like. That is, the acquisition unit 111, the division unit 112, the adjustment unit 113, the learning unit 114, the estimation unit 115, and the output control unit 116 (control unit 110) of the information processing device 100 may be realized as an acquisition circuit, a division circuit, an adjustment circuit, a learning circuit, an estimation circuit, and an output control circuit (control circuit) constituting the arithmetic processing device of a computer or the like. The communication unit 121, the storage unit 122, and the display unit 123 (output unit) of the information processing device 100 may be realized as a communication function, a storage function, and a display function (output function) including the functions of an arithmetic processing device, etc. The communication unit 121, the storage unit 122, and the display unit 123 (output unit) of the information processing device 100 may be realized as a communication circuit, a storage circuit, and a display circuit (output circuit) by being configured, for example, by an integrated circuit, etc. The communication unit 121, the storage unit 122, and the display unit 123 (output unit) of the information processing device 100 may be realized as a communication device, a storage device, and a display device (output device) by being configured, for example, by being configured by a plurality of devices.

[0060] The information processing device 100 can combine one or any two or more of the above-mentioned units. In this disclosure, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information."

[0061] [Aspects and Effects of the Present Embodiment] Next, one aspect of this embodiment and the effects of each aspect will be described. Note that each aspect described below is an example at the time of filing, and this embodiment is not limited to the aspects described below. In other words, this embodiment is not limited to each aspect described below, and may be realized by appropriately combining each of the above-mentioned parts. In addition, a lower aspect may be able to cite any of the higher aspects. The effects described below are merely examples, and the effects of each aspect are not limited to those described below. Each aspect may, for example, have at least one of the effects described below.

[0062] (Aspect 1) An information processing device of one embodiment includes an acquisition unit that acquires image information recorded at multiple different times, a trained model that has learned about target images captured at the same location at multiple different times and objects that change between the target images, and an estimation unit that estimates objects that change in the images based on each of the multiple image information acquired by the acquisition unit. This allows the information processing device to estimate objects that are different between multiple images, i.e., objects that have changed among the objects recorded in the images. For example, the information processing device can estimate objects as having changes when there has been construction of an object, expansion of an object, demolition of an object, etc.

[0063] (Aspect 2) In one aspect of the information processing device, the estimation unit may estimate the changed object based on a trained model that has learned a target image, an object before the change in each target image and the object after the change, with annotations attached to either the side showing the ground or the side not showing the ground, and an image based on multiple image information. An information processing device can add annotations to be used for learning to the before and after images of an object that show the ground (closer to the ground), or to the after images of the object that show the ground (farther from the ground), and then use a trained model that has been trained with the annotations added to improve the accuracy (correctness of the estimation) of estimations of objects that have changed.

[0064] (Aspect 3) An information processing device of one embodiment acquires map information including the same location at multiple different times, identifies an object that has changed based on object information regarding the object recorded in the map information, records the location of the identified object that has changed, acquires target images capturing the object before and after the change, and annotates the location where the object is not present or the location where the object is present in the target images recording the location where the object is not present or the location where the object is present before and after the change of the identified object, and learns the annotated target image and the unannotated target image to generate a trained model. The estimation unit may estimate the object that has changed based on the trained model generated by the learning unit and an image based on the multiple image information. This allows the information processing device to automatically generate a trained model. In addition, the information processing device can use the trained model to estimate objects that are differences between multiple images, that is, objects that have changes among objects recorded in the images.

[0065] (Aspect 4) An information processing device of one embodiment may include a division unit that divides each of images based on multiple image information acquired by the acquisition unit into multiple meshes, and the estimation unit may estimate objects that change for each mesh based on meshes whose positions correspond in each of the images based on the multiple image information and a learned model. By estimating the object for each mesh, the information processing device can reduce the processing burden, i.e., reduce costs, compared to estimating the object between multiple images that are not divided into meshes.

[0066] (Aspect 5) In the information processing device of one aspect, the dividing unit may be capable of adjusting the granularity of the mesh. This allows the information processing device to adjust the granularity of the mesh depending on, for example, the condition of the ground surface recorded in the image (e.g., the density of objects such as buildings and roads, etc.). As an example, if the condition of the ground surface recorded in the image (e.g., density, etc.) is relatively dense, the information processing device may make the granularity finer (reduce the mesh plane size) compared to a condition where the ground surface condition (e.g., density, etc.) is relatively sparse. In other words, as an example, if the condition of the ground surface recorded in the image (e.g., density, etc.) is relatively sparse, the information processing device may make the granularity coarser (increase the mesh plane size) compared to a condition where the ground surface condition (e.g., density, etc.) is relatively dense.

[0067] (Aspect 6) An information processing device of one embodiment may include an adjustment unit that adjusts the color tone and tilt of each image based on multiple image information acquired by the acquisition unit so that they are identical, and the estimation unit may estimate a changing object based on the multiple images adjusted by the adjustment unit and a trained model. This makes it easier for the information processing device to compare multiple images using the trained model, thereby improving the accuracy (correctness of estimation) of estimation when estimating objects with changes.

[0068] (Aspect 7) An information processing device of one embodiment includes an acquisition unit that acquires image information captured from the sky at multiple different times, a trained model that has learned about target images captured at the same location at multiple different times and roads, buildings, and shorelines that have changed between the multiple different times within an area corresponding to the imaging area of ​​each of the target images, and an estimation unit that estimates the roads, buildings, and shorelines that have changed in the images based on each of the multiple image information acquired by the acquisition unit. This allows the information processing device to achieve the same effects as those of the above-mentioned aspect.

[0069] (Aspect 8) In one aspect of the information processing method, a computer executes an acquisition step of acquiring image information recorded at multiple different times, and an estimation step of estimating objects that change and are captured in the images based on a trained model that has learned about target images captured at the same location at multiple different times and objects that change between the target images, and each of the images of the multiple image information acquired by the acquisition step. As a result, the information processing method can achieve the same effects as the information processing device according to the above-described aspect.

[0070] (Aspect 9) An information processing program of one embodiment provides a computer with an acquisition function for acquiring image information recorded at multiple different times, a trained model that has learned about target images captured at the same location at multiple different times and objects that change between the target images, and an estimation function that estimates objects that change in the images based on each of the multiple image information acquired by the acquisition function. As a result, the information processing program can achieve the same effects as the information processing device according to the above-described aspect. [Explanation of symbols]

[0071] 100 Information processing device 110 Control section 111 Acquisition Department 112 Division 113 Adjustment section 114 Learning Department 115 Estimation Department 116 Output control section 121 Communications Department 122 Storage section 123 Display section 200 Flying Objects 210 Camera 300 Servers 400 Objects (objects for which changes are estimated) 401 Changing Objects 410 Building 411 Constructed buildings (buildings with changes) 420 Road 421 Constructed roads (Roads with changes) 440 Comparisons 500 images 510 Mesh 600 target images (images to be trained) 610 Target images for the first period 620 Target images for the second period 640 Object (object to be studied) 641 Building 642 Road 650 Annotations

Claims

1. An acquisition unit that acquires image information recorded at multiple different times; An estimation unit that estimates an object that has changed in the image based on a trained model that has learned target images captured at the same location at multiple different times and an object that has changed between the target images, and each image of the multiple image information acquired by the acquisition unit; An information processing device comprising:

2. The estimation unit estimates a changed object based on a trained model that has trained the target image, an object before a change in each of the target images and the object after the change, with annotations added to one of a side showing the ground and a side showing no ground, and an image based on the plurality of image information. The information processing device according to claim 1 .

3. A learning unit that acquires map information including the same point at multiple different times, identifies an object that has changed based on object information related to the object recorded in the map information, records the position of the identified object that has changed, acquires target images capturing the object before and after the change, and adds annotations to the position where the object is not present or the position where the object is present in the target images that record the position where the object is not present or the position where the object is present before and after the change in the identified object, and learns the target image with the annotation and the target image without the annotation to generate a learned model; The estimation unit estimates an object having a change based on a trained model generated by the training unit and an image based on the plurality of pieces of image information. The information processing device according to claim 2 .

4. a division unit that divides each of the images based on the plurality of image information acquired by the acquisition unit into a plurality of meshes, The estimation unit estimates an object having a change for each mesh based on the meshes having corresponding positions in each image based on a plurality of image information and a trained model. The information processing device according to claim 1 .

5. The dividing unit is capable of adjusting the grain size of the mesh. The information processing device according to claim 4.

6. an adjustment unit that adjusts the color tone and inclination of each image based on the plurality of image information acquired by the acquisition unit so that they are uniform; The estimation unit estimates an object having a change based on a plurality of images adjusted by the adjustment unit and a trained model. The information processing device according to claim 1 .

7. An acquisition unit that acquires image information captured from the sky at multiple different times; an estimation unit that estimates roads, buildings, and shorelines that have changed in the images based on target images captured at the same point at multiple different times, a trained model that has learned about roads, buildings, and shorelines that have changed between the multiple different times within an area corresponding to an image capture area of ​​each of the target images, and each image of the multiple image information acquired by the acquisition unit; An information processing device comprising:

8. The computer An acquisition step of acquiring image information recorded at multiple different times; an estimation step of estimating an object that has changed and appears in the image based on a trained model that has learned about target images captured at the same location at multiple different times and objects that have changed between the target images, and each of the multiple pieces of image information acquired by the acquisition step; An information processing method for performing the above.

9. On the computer, An acquisition function for acquiring image information recorded at multiple different times; A trained model that has learned target images captured at the same location at multiple different times and target objects that change between the target images, and an estimation function that estimates a target object that changes in the image based on each image of the multiple image information acquired by the acquisition function; An information processing program that realizes this.

Images