Information processing device, information processing method, and program

Abstract

To provide an information processing device, information processing method, and program, which enable effective annotation by accurately segmenting an area of a target object on an image.SOLUTION: An information processing device 100 (information provision device) is provided, comprising an area detection unit for detecting an area of a target object on an image of a ground surface captured from the sky, an orientation estimation unit for estimating the orientation of the area of the target object, and an image processing unit configured to shape the area of the target object according to the estimated orientation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Techniques related to annotation for detecting a desired object on an image by machine learning are known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] <U+ However, in the conventional technology, the accuracy of segmentation of the region of the object is insufficient, and as a result, it may not be effective to assign the detected object to the image as an annotation.

[0005] One object of the present invention is to provide an information processing apparatus, an information processing method, and a program that can effectively perform annotation by accurately segmenting the region of an object on an image in consideration of such circumstances. <00U+

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing apparatus including a detection unit that detects a region of an object on an image obtained by imaging the ground surface from above, an estimation unit that estimates the orientation of the region, and an image processing unit that shapes the region based on the orientation.

Effects of the Invention

[0007] According to one aspect of the present invention, annotation can be effectively performed by accurately segmenting the region of an object on an image. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of the information processing device 100 according to the embodiment. [Figure 2] This is a flowchart showing the sequence of processes performed by the processing unit 110 according to the embodiment. [Figure 3] This figure illustrates how to detect the area of ​​a pedestrian crossing using the first machine learning model MDL1. [Figure 4] This figure illustrates how to estimate various orientations using the second machine learning model, MDL2. [Figure 5] This is a diagram illustrating how to shape the area of ​​a pedestrian crossing. [Figure 6] This figure shows an example of annotated satellite imagery. [Figure 7] This figure shows an example of the hardware configuration of the information processing device 100 of the embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the information processing apparatus, information processing method, and program of the present invention will be described with reference to the drawings.

[0010] [overview] One aspect of this embodiment of the information processing device detects the region of an object on an image of the ground surface taken from above, and estimates the orientation of the region of the object.

[0011] Images of the Earth's surface taken from above include, for example, satellite images (satellite photographs), aerial images (aerial photographs), and aerial photographs (aerial photographs).

[0012] The objects detected in an image can be arbitrarily determined depending on the type of service or application provided to the user. For example, when providing users with satellite images or aerial images as map data, anything that appears in those images can be an object. More specifically, when navigating a user to a destination using map data, traffic control structures such as pedestrian crossings, traffic lights, school zones, and railway crossings located along the route to the destination may be selected as objects. The following explanation will use "pedestrian crossings" as an example object.

[0013] Furthermore, when map data such as satellite images and aerial images are used for disaster prevention against natural disasters such as landslides and river flooding, objects that could cause such disasters (for example, solar panels installed after forests have been cleared) may be selected as targets.

[0014] The information processing device detects the region of an object (e.g., a pedestrian crossing) in an image and estimates the orientation of the object's region. Based on the estimated orientation, it then reshapes the object's region. By reshaping the object's region in this way, the accuracy of segmentation is improved. As a result, when adding annotations to satellite imagery, such as pedestrian crossings, the annotation process can be performed more effectively.

[0015] [Information Processing Device] Figure 1 is a diagram showing an example of the configuration of an information processing device 100 according to the embodiment. For example, the information processing device 100 is a web server or application server that sends and receives information with a user's terminal device.

[0016] The user's terminal device is a computer device having a communication function and a display function, such as a smartphone, a personal computer, or a tablet terminal. Specifically, the terminal device includes a communication interface for communicating with an external device such as the content providing device 50 via the network NW, a display, and a GNSS (Global Navigation Satellite System) receiver. The network NW includes the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a cellular network, and the like.

[0017] The communication interface includes, for example, a network card such as a NIC (Network Interface Card) and a wireless communication module. The display includes, for example, an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, and the like. A GUI (Graphical User Interface) for receiving various input operations from the user is displayed on these displays. The GNSS receiver measures the position of the terminal device.

[0018] Furthermore, the terminal device includes a processor such as a CPU (Central Processing Unit), and when the CPU executes a UA (User Agent), various contents are displayed on the display.

[0019] When a web browser or an application is launched as a UA on the user's terminal device and a request is sent from the UA, the information processing device 100 provides various contents to the terminal device as a response to the request. The contents include, for example, satellite images with objects such as crosswalks annotated.

[0020] As shown in the figure, the information processing device 100 includes a communication unit 102, a processing unit 110, and a storage unit 130.

[0021] The communication unit 102 is, for example, a communication interface such as a network card for connecting to a network NW.

[0022] The processing unit 110 comprises an acquisition unit 112, a region detection unit 114, an orientation estimation unit 116, an image processing unit 118, a communication control unit 120, and a learning unit 122. These components are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of the components of the processing unit 110 may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware.

[0023] The storage unit 130 is implemented by a storage device such as an HDD (Hard Disc Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 130 stores firmware and application programs executed by the processor. The programs may be pre-stored in the storage unit 130, or they may be stored on a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM and installed in the storage unit 130 when the storage medium is inserted into the drive device.

[0024] The memory unit 130 also stores map data 132, which includes satellite and aerial images, and a training dataset 134. The map data 132 includes various images of the Earth's surface taken from above, such as satellite and aerial images. The training dataset 134 is a dataset (a combination of input data and correct output data) prepared for training the machine learning model described later.

[0025] [Processing flow of information processing equipment] The processing details of the processing unit 110 will be explained below using a flowchart. Figure 2 is a flowchart showing the sequence of processing steps of the processing unit 110 according to this embodiment. The processing steps in this flowchart may be executed repeatedly, for example, at a predetermined interval.

[0026] First, the acquisition unit 112 acquires the satellite image IMG1 (step S100). For example, if the satellite image IMG1 is stored as map data 132 in the storage unit 130, the acquisition unit 112 may read the satellite image IMG1 from the storage unit 130 and acquire it. Alternatively, the acquisition unit 112 may acquire the satellite image IMG1 from an external server (e.g., a data source server) via the communication unit 102. Also, for example, if a non-transient storage medium (e.g., fresh memory) containing the satellite image IMG1 is connected to the drive device of the information processing device 100, the acquisition unit 112 may read the satellite image IMG1 from the storage medium. As described above, the acquisition unit 112 may acquire aerial images or other images instead of satellite images. The following explanation will assume, as an example, that the image to be acquired is the satellite image IMG1.

[0027] Next, the region detection unit 114 detects the region of a pedestrian crossing, which is an example of an object, on the satellite image IMG1 (which may also be an aerial image or aerial image) acquired by the acquisition unit 112 (step S102).

[0028] For example, the region detection unit 114 uses a first machine learning model MDL1, which is one of the segmentation models, to detect the area of ​​a pedestrian crossing on the satellite image IMG1 (performs segmentation).

[0029] The first machine learning model MDL1 may be a segmentation model implemented using a neural network such as a CNN (Convolutional Neural Network(s)).

[0030] The first machine learning model, MDL1, is trained on a first training dataset in which a given satellite image is associated with the region of a ground truth object (e.g., a crosswalk) that should be detected on that satellite image. If the object is a crosswalk, its region will be a quadrilateral with four vertices.

[0031] Figure 3 is a diagram illustrating how to detect the area of ​​a pedestrian crossing using the first machine learning model MDL1. The area detection unit 114 inputs the satellite image IMG1 acquired by the acquisition unit 112 into the first machine learning model MDL1. As described above, the first machine learning model MDL1 is trained on the first training dataset. Therefore, upon receiving the satellite image IMG1 as input, the first machine learning model MDL1 outputs the location and size of the area as a pedestrian crossing area. Furthermore, the first machine learning model MDL1 may output a score (hereinafter referred to as the confidence score) that represents the likelihood of the location and size of the area being reliable.

[0032] Returning to the flowchart explanation, the image processing unit 118 then crops (extracts) the area of ​​the traverse walk from the satellite image IMG1 and generates a cropped image IMG2 from the cropped portion of the satellite image IMG1 (step S104).

[0033] When detecting the area of ​​a pedestrian crossing using the first machine learning model MDL1, the area of ​​the pedestrian crossing is often detected as a quadrilateral, but it may also be detected as a pentagon with 5 vertices or a polygon with more than 5 vertices.

[0034] If the area of ​​the pedestrian crossing is detected as a polygonal area with five or more vertices, the image processing unit 118 may approximate the polygonal area of ​​the pedestrian crossing to a quadrilateral with four vertices (or a pentagon with five vertices) based on the following conditions. (1) First, approximate the area of ​​the crosswalk with a quadrilateral polygon having 4 vertices. (2) If the difference between the area of ​​the approximated quadrilateral polygon and the area of ​​the original crosswalk is less than 25% of the area of ​​the original crosswalk, or if the area of ​​the original crosswalk is less than 128 when the area of ​​one pixel is set to 1, the original crosswalk area will be used as is without approximation. (3) Next, the area of ​​the pedestrian crossing, which was approximated by the quadrilateral polygon, is further approximated by a pentagonal polygon with 5 vertices.

[0035] Next, the image processing unit 118 determines whether the aspect ratio of the cropped image IMG2 is within an acceptable range (step S106). For example, if the cropped image IMG2 input to the second machine learning model MDL2 described later is assumed to be square (when a second training dataset containing square images is used during training), the acceptable range of aspect ratio may be, for example, the same as the aspect ratio of a square (1:1), or allow for an error of a few percent in the aspect ratio of a square.

[0036] If the aspect ratio of the cropped image IMG2 is outside the acceptable range (i.e., the cropped image IMG2 is not square), the image processing unit 118 pads the cropped image IMG2 so that the aspect ratio is within the acceptable range (step S108).

[0037] For example, the image processing unit 118 adds arbitrary pixel values ​​(for example, pixel values ​​of the same color as the road surface) to one or all of the top, bottom, left, or right sides of the matrix representing the cropped image IMG2 as padding, thereby making the shape of the cropped image IMG2 closer to a square.

[0038] On the other hand, if the aspect ratio of the cropped image IMG2 is within an acceptable range (i.e., if the cropped image IMG2 is square), the image processing unit 118 omits the padding process in S118.

[0039] Generally, the shape of a pedestrian crossing is determined according to the road and surrounding traffic conditions, and pedestrian crossings can sometimes be extremely elongated vertically or horizontally. When cropped images IMG2 of pedestrian crossings of various shapes like these are input into the second machine learning model MDL2 (described later), the orientations estimated by the second machine learning model MDL2 (details will be described later) tend to deviate from the orientations assumed during training. For example, if the orientations assumed during training are 0 degrees, 90 degrees, and 180 degrees, angles other than these tend to be estimated by the second machine learning model MDL2.

[0040] In contrast, by adjusting the aspect ratio of the cropped image IMG2 input to the second machine learning model MDL2 (described later) using padding, the accuracy of the output of the second machine learning model MDL2 can be improved. For example, by inputting the cropped image IMG2 with a unified aspect ratio into the second machine learning model MDL2, the second machine learning model MDL2 becomes more likely to output the orientation (0 degrees, 90 degrees, 180 degrees) that was assumed during training.

[0041] Furthermore, if the cropped image IMG2 input to the second machine learning model MDL2 is assumed to be rectangular rather than square (i.e., if a second training dataset containing rectangular images is used during training), the acceptable range of aspect ratios compared in the S106 judgment process may be determined according to the aspect ratio of a rectangle rather than a square.

[0042] Next, the orientation estimation unit 116 estimates the orientation V1 of the pedestrian crossing included in the cropped image IMG2 and the orientation V2 of the white line drawn on the road as the pedestrian crossing (step S110).

[0043] The direction V1 of the crosswalk is the direction in which pedestrians cross the crosswalk, and it is the direction that intersects with the longitudinal direction of the white line (the direction in which the white line extends). The direction V2 of the white line is the longitudinal direction of the white line (the direction in which the white line extends). The direction V1 of the crosswalk is an example of the "first direction," and the direction V2 of the white line is an example of the "second direction."

[0044] For example, the orientation estimation unit 116 uses a second machine learning model MDL2, which is implemented using a neural network such as a CNN (Convolutional Neural Network(s)), to estimate the orientation V1 of the crosswalk and the orientation V2 of the white line on the cropped image IMG2.

[0045] The second machine learning model, MDL2, is trained on a second training dataset in which the orientation of the crosswalk (V1) and the orientation of the white line (V2) are associated with the orientation of the ground truth object region to be estimated on a given cropped image used for training. The cropped image used for training is an image from which the crosswalk region has been cut out from the training satellite image, and is, for example, a square image as described above.

[0046] Figure 4 illustrates the method for estimating various orientations using the second machine learning model MDL2. The orientation estimation unit 116 inputs the cropped image IMG2 generated by the image processing unit 118 to the second machine learning model MDL2. As described above, the second machine learning model MDL2 is trained based on the second training dataset. Therefore, upon receiving the cropped image IMG2 as input, the second machine learning model MDL2 outputs the orientation of the crosswalk V1 and the orientation of the white line V2 as the orientation of the object's region.

[0047] Next, the image processing unit 118 reshapes the area of ​​the pedestrian crossing detected by the area detection unit 114 based on the direction of the pedestrian crossing V1 and the direction of the white line V2 estimated by the direction estimation unit 116 (step S112).

[0048] Figure 5 is a diagram illustrating a method for shaping the area of ​​a pedestrian crossing. In the figure, (a) represents the cropped image IMG2, and R on the cropped image IMG2 represents the area of ​​the pedestrian crossing detected by the area detection unit 114. The cropped image IMG2 is shown as a rectangle for convenience, but it may actually be a square. Even if the cropped image IMG2 is a rectangle, it may be converted to a square by padding as described above.

[0049] Figure (b) indicates that the orientation of the pedestrian crossing V1 and the orientation of the white line V2 are estimated on the cropped image IMG2 of (a). As shown in the figure, the detected pedestrian crossing region R may not completely mask the actual pedestrian crossing region. This is because the segmentation model is model-based using the first machine learning model MDL1. Therefore, the image processing unit 118 reshapes the detected pedestrian crossing region R using the first machine learning model MDL1 so as to completely mask the actual pedestrian crossing region, based on the orientation of the pedestrian crossing V1 and the orientation of the white line V2.

[0050] As shown in (c), the image processing unit 118 first divides the four sides of the rectangle detected as the crosswalk area R into two pairs, with opposite sides forming a pair. Specifically, the image processing unit 118 classifies the four sides of the rectangle detected as the crosswalk area R into two pairs: two sides that are close to the angle of the crosswalk direction V1 (smaller angle difference with V1 and more parallel) and two sides that are close to the angle of the white line direction V2 (smaller angle difference with V2 and more parallel). The two sides that are close to the angle of the crosswalk direction V1 (smaller angle difference with V1 and more parallel) are an example of the "first pair of opposite sides," and the two sides that are close to the angle of the white line direction V2 (smaller angle difference with V2 and more parallel) are an example of the "second pair of opposite sides."

[0051] As shown in (d), the image processing unit 118 then rotates two sides that are close in angle to the direction V1 of the crosswalk (i.e., the angle difference with V1 is small and they are more parallel) so that they approach the direction V1 of the crosswalk. Similarly, the image processing unit 118 rotates two sides that are close in angle to the direction V2 of the white line (i.e., the angle difference with V2 is small and they are more parallel) so that they approach the direction V2 of the white line. In this case, the image processing unit 118 may rotate the sides around the midpoint of each side.

[0052] As shown in (e), the image processing unit 118 defines the region closed by the four sides with corrected angles (i.e., the reshaped region) as the new crosswalk region R#. In this case, if the area of ​​the new crosswalk region R# is 50% or less of the area of ​​the crosswalk region R before reshaping, the image processing unit 118 will not adopt the new crosswalk region R# and will instead adopt the crosswalk region R before reshaping. Furthermore, even if the overlapping portion between the new crosswalk region R# and the crosswalk region R before reshaping is 50% or less of the area of ​​the crosswalk region R before reshaping, the image processing unit 118 may still adopt the crosswalk region R before reshaping instead of the new crosswalk region R#.

[0053] Through this series of processes, the area R of the pedestrian crossing is shaped, and the processing of this flowchart is completed. When the communication control unit 120 receives a request for map data sent from the user's terminal device via the communication unit 102, it may send a satellite image to the user's terminal device via the communication unit 102 as a response to the request, with the shaped area R# of the pedestrian crossing (or the area R of the pedestrian crossing before shaping if the area condition is not met) annotated.

[0054] Figure 6 shows an example of annotated satellite imagery. In the figure, AN represents the annotation, which overlays the reshaped crosswalk area R# onto the crosswalk in the satellite image. By displaying such annotations AN, users can recognize the location and shape of objects such as crosswalks more realistically, and use services such as navigation without feeling any discomfort (without feeling unnatural).

[0055] [Training machine learning models] The training methods for the first machine learning model MDL1 and the second machine learning model MDL2 described above are explained below. The learning unit 122 trains the first machine learning model MDL1 using the first training dataset described above, and trains the second machine learning model MDL2 using the second training dataset described above. When the learning unit 122 trains the second machine learning model MDL2, it sets the loss function as follows. The term "direction" mentioned above can be treated as synonymous with "angle". The numerical values ​​output by the second machine learning model MDL2 as the angles of V1 and V2 are in the range of 0 to 1, and this numerical range of 0 to 1 corresponds to the angle range of 0 degrees to 180 degrees.

[0056] (i) The angle output by the second machine learning model MDL2 (hereinafter referred to as the estimated angle) differs from the correct angle included in the second training dataset by 90 degrees (0.5) or more. (ii) If both the estimated angle and the correct angle are less than 18 degrees (0.1) or 162 degrees (0.9) or greater, (iii) The correct angle is modified to be closer to the estimated angle (to be closer to 0 degrees or 180 degrees), and then the loss function is optimized using BCELoss (Binary Cross Entropy Loss).

[0057] When the angles V1 and V2 output (estimated) by the second machine learning model MDL2 in response to the input cropped image IMG2 are approximately 0 degrees or approximately 180 degrees, the estimated angle and the correct angle may be in opposite directions. For example, if the correct angle is approximately 179.8 degrees (equivalent to approximately 0.999 in the output values ​​of the second machine learning model MDL2), the estimated angle may be approximately 0 degrees (equivalent to approximately 0.0001 in the output values ​​of the second machine learning model MDL2).

[0058] Although nearly 180 degrees and nearly 0 degrees are actually almost the same angle, it is best to consider the estimated angle, which is the output value of the second machine learning model MDL2, as correct even if it is a different value from nearly 0 (e.g., around 0.0001) or nearly 1 (e.g., around 0.999).

[0059] Therefore, the second machine learning model MDL2 is trained based on a loss function that includes the exception conditions (i) to (iii) above. This improves the training efficiency and accuracy of the second machine learning model MDL2.

[0060] According to the embodiments described above, the information processing device 100 detects the area of ​​an object (e.g., a pedestrian crossing) on ​​an image of the ground surface taken from above, such as a satellite image (satellite photograph), an aerial image (aerial photograph), or aerial photograph, and estimates the orientation of the area of ​​the object. The information processing device 100 then shapes the area of ​​the object based on the estimated orientation. By shaping the area of ​​the object (e.g., a pedestrian crossing) in this way, the accuracy of segmentation is improved. As a result, when adding annotations to satellite images or other images of objects such as pedestrian crossings, the annotation can be performed effectively.

[0061] <Hardware Configuration> The information processing device 100 of the above-described embodiment can be realized by a hardware configuration such as that shown in Figure 7. Figure 7 is a diagram showing an example of the hardware configuration of the information processing device 100 of the embodiment.

[0062] The information processing device 100 consists of a NIC 100-1, a CPU 100-2, RAM 100-3, ROM 100-4, a secondary storage device 100-5 such as flash memory or an HDD, and a drive device 100-6, all interconnected by an internal bus or dedicated communication lines. A portable storage medium such as an optical disc is mounted in the drive device 100-6. A program stored in the secondary storage device 100-5 or the portable storage medium mounted in the drive device 100-6 is loaded into RAM 100-3 by a DMA controller (not shown) and executed by CPU 100-2 to realize the processing unit 110. The program referenced by CPU 100-2 may be downloaded from another device via a network NW.

[0063] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0064] 100...Information processing unit, 102...Communication unit, 110...Processing unit, 112...Acquisition unit, 114...Region detection unit, 116...Orientation estimation unit, 118...Image processing unit, 120...Communication control unit, 122...Learning unit

Claims

1. A detection unit that detects the area of ​​an object on an image of the ground surface taken from above, An estimation unit for estimating the orientation of the aforementioned region, An image processing unit that shapes the region based on the orientation, A communication unit that communicates with a terminal device that displays the aforementioned image upon receiving a request from a user, The system includes a communication control unit that transmits an annotation image, which is the image to which the shaped region of the object has been annotated, to the terminal device via the communication unit, The annotation image is displayed on the terminal device as a response to the request. Information processing device.

2. The aforementioned object is a pedestrian crossing, The detection unit detects the area of ​​the pedestrian crossing on the image, The estimation unit estimates a first direction, which is the direction in which a pedestrian crosses the crosswalk, and a second direction, which is the longitudinal direction of the white line drawn on the road as the crosswalk. The image processing unit shapes the area of ​​the pedestrian crossing based on the first direction and the second direction. The information processing apparatus according to claim 1.

3. The area of ​​the aforementioned pedestrian crossing is a quadrilateral area consisting of four sides. The image processing unit reshapes the quadrilateral region such that the first pair of opposite sides of the four sides is parallel to the first direction, and the second pair of opposite sides of the four sides is parallel to the second direction. The information processing apparatus according to claim 2.

4. The detection unit uses a first machine learning model to detect the region of the object on the image. The first machine learning model is a machine learning model that has been trained on a first training dataset in which a training target image is associated with the region of the correct object to be detected on the training target image. The information processing apparatus according to claim 1 or 2.

5. The estimation unit uses a second machine learning model to estimate the orientation of the detected region on a cropped image, which is an image extracted from the image. The second machine learning model is a machine learning model trained on a second training dataset in which the orientation of the object region to be estimated on the cropped image of the training target is associated with the cropped image of the training target, which is an image obtained by cutting out the region of the object from a training target image. The information processing apparatus according to claim 1 or 2.

6. A computer-based information processing method, Detecting the area of ​​an object on an image of the ground surface taken from above, To estimate the orientation of the said region, To shape the region based on the orientation, To communicate with a terminal device that displays the aforementioned image upon receiving a request from a user, This includes transmitting an annotated image, which is the image to which the shaped region of the object has been annotated, to the terminal device via a communication unit. The annotation image is displayed on the terminal device as a response to the request. Information processing methods.

7. A program to be executed by a computer, Detecting the area of ​​an object on an image of the ground surface taken from above, To estimate the orientation of the said region, To shape the region based on the orientation, To communicate with a terminal device that displays the aforementioned image upon receiving a request from a user, This includes transmitting an annotated image, which is the image to which the shaped region of the object has been annotated, to the terminal device via a communication unit. The annotation image is displayed on the terminal device as a response to the request. program.

Images