Learning device, building area detection device, matching system, learning method, and learning program

Abstract

To detect areas of individual buildings by detecting, with high accuracy, segments representing contours of the individual buildings from segments indicating edges detectable from image data by edge detection and using the detected segments.SOLUTION: Input image data being data of a captured image of a building, rectangle position image of rectangular areas each of which is appropriately determined so as to include one of detection object segments included in the input image data, rectangle size data, segment position data indicating positions of both end points of the segment included in each rectangular area, in the rectangular area, and segment type data indicating whether a detection object segment is a first type or a second type are received, and learning processing is performed where learning model data applied to a function approximator is updated in order to approximate the function approximator to such a state that the function approximator outputs the received rectangle position data, rectangle size data, segment position data, and segment type data when the received input image data is given to the function approximator.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning device, a building area detection device, a matching system, a learning method, and a learning program. [Background technology]

[0002] Research is being conducted on applied technologies such as further improving the accuracy of vehicle positions obtained from the Global Positioning System (GPS) by capturing images of cityscapes using cameras mounted on moving vehicles, detecting buildings in the captured images, and matching the detected buildings with 2.5-dimensional map data, etc., or using Augmented Reality (AR) technology to display advertisements on appropriate buildings.

[0003] As a technology for detecting a three-dimensional object contained in an image, for example, there is a technology such as a deep machine learning system disclosed in Patent Document 1 that detects a rectangular parallelepiped contained in an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2019-536154 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in many of the images of building parts included in cityscape images captured by a vehicle-mounted camera, only one or two sides of the building are captured, and furthermore, in some cases, only a portion of a high-rise building is captured. In this way, when the entire building is not visible and does not have a rectangular parallelepiped shape, the technology disclosed in Patent Document 1 has the problem of being unable to detect parts of the building.

[0006] In addition, by performing edge detection on the image, it is possible to detect edges that indicate the outline of the building. However, the edges detected by edge detection include edges other than the outline of the building. Therefore, when the texture of the building, which is indicated by the shape and arrangement of the veranda and windows, is complex, when there are many obstacles other than the building, or when the buildings are adjacent to each other, there is a problem that it is difficult to extract only the edges that indicate the outline of each building.

[0007] The present invention has been made to solve the above-mentioned problems, and its object is to provide a learning device, a building area detection device, a matching system, a learning method, and a learning program that enable detection of line segments indicating the outline of individual buildings with high accuracy from line segments that indicate edges that can be detected by edge detection from image data, and detection of the areas of individual buildings using the detected line segments. [Means for solving the problem]

[0008] In order to solve the above problem, one aspect of the present invention provides a method for detecting lines that are detection target lines, the method comprising: input image data that is data of an image of a building; detecting lines that are selected from lines that show the outline of each building included in the input image data, and that have one end belonging to a lower part of the building that is appropriately determined for each building; and detecting lines that are selected from lines that show the outline of each building included in the input image data, and that have one end belonging to a lower part of the building that is appropriately determined for each building, and detecting lines that are selected from lines that show the outline of each building included in the input image data ... The learning device includes an input unit that inputs size data, line segment position data indicating the positions in each rectangular area of ​​both end points of the line segment contained in each rectangular area, and line segment type data indicating whether the line segment to be detected is the first type or the second type, and a learning processing unit that performs a learning process to update the learning model data applied to the function approximator so as to approach a state in which the function approximator outputs the rectangle position data, rectangle size data, line segment position data, and line segment type data input by the input unit when the input image data input by the input unit is provided to a function approximator.

[0009] Moreover, one aspect of the present invention is a building area detection device that includes a line segment detection unit that detects the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data for the detection target line segments among the line segments included in the image data by providing arbitrary image data to the function approximator to which the trained learning model data generated by the learning device described above is applied, and generates line segment identification data that combines the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data for each of the detected detection target line segments, and a building area detection unit that detects the building area of ​​each individual building included in the image data by selecting line segments that are in a connecting relationship and that form the boundary of the building area based on the line segment identification data generated by the line segment detection unit, and connecting the selected line segments.

[0010] Moreover, one aspect of the present invention includes the building area detection device described above, an imaging device mounted on a moving body and moving, the imaging device generating image data to be processed by the building area detection device, a position detection device detecting a position of the moving body, and a matching device performing matching processing based on the area of ​​the building detected by the building area detection device, the matching device including a map data storage unit that stores map data including position data indicating the position of the building, shape data indicating the shape of the building on the ground plane, and height data indicating the height of the building, and a matching device that performs matching processing for the building area detection device based on the data indicating the position of the moving body detected by the position detection device. This matching system is equipped with a matching processing unit that calculates the direction of travel of the moving body when the imaging device captured the image in order to generate image data of a target, generates map image data to be projected onto the projection surface of the imaging device from the map data stored in the map data storage unit so that the anteroposterior relationship of the building is correct based on data indicating the calculated direction of travel of the moving body, data indicating the position of the moving body when the imaging was performed, and data indicating the angle of view of the imaging device when the imaging was performed, and performs a matching process to overlay the generated map image data with the area of ​​the building detected from the image data by the building area detection device.

[0011] In one aspect of the present invention, a detection target line segment includes input image data which is data of an image of a building, a first type of line segment which constitutes a boundary of a region of the building selected from line segments which indicate the outline of each building included in the input image data and has one end which belongs to a lower part of the building which is appropriately determined for each building, and a second type of line segment which constitutes a boundary of the region of the building and has both ends which do not belong to the lower part, and a rectangular area which is appropriately determined to include one of the detection target line segments, the detection target line segment includes rectangular position data which indicates a position in the input image data of each of the rectangular areas, rectangular size data which indicates a size of each of the rectangular areas, and the detection target line segment includes rectangular position data which indicates a position in the input image data of each of the rectangular areas, the detection target line segment includes ... size data which indicates a size of each of the rectangular areas. The learning method includes an input process for inputting line segment position data indicating the positions in each rectangular area of ​​both end points of the line segment contained in the rectangular area, and line segment type data indicating whether the line segment to be detected is the first type or the second type, and a learning processing process for performing a learning process for updating learning model data applied to the function approximator so as to approach a state in which, when the input image data input by the input process is provided to a function approximator, the function approximator outputs the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data input by the input process.

[0012] In one aspect of the present invention, a computer is provided with input image data which is data of an image of a building, detection target lines being first type of line segments which are selected from line segments which show the outline of each building included in the input image data and which form a boundary of the area of ​​the building, the first type of line segments having one end which belongs to a lower part of the building which is appropriately determined for each building, and a second type of line segments which form the boundary of the area of ​​the building and which have both ends not belonging to the lower part, rectangle position data which indicates a position in the input image data of each of rectangular areas which are appropriately determined to include one of the detection target lines, rectangle size data which indicates a size of each of the rectangular areas, and This is a learning program for executing an input procedure for importing line segment position data indicating the positions in each rectangular area of ​​both end points of the line segment contained in each rectangular area, and line segment type data indicating whether the line segment to be detected is the first type or the second type, and a learning processing procedure for performing a learning process that updates the learning model data applied to the function approximator so as to approach a state in which, when the input image data imported by the input procedure is provided to a function approximator, the function approximator outputs the rectangle position data, rectangle size data, line segment position data, and line segment type data imported by the input procedure. Effect of the Invention

[0013] According to this invention, it is possible to detect with high accuracy line segments indicating the outline of each building from among line segments indicating edges that can be detected by edge detection from image data, and to detect the areas of each building using the detected line segments. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of a learning device according to an embodiment of the present invention; [Diagram 2] 2 is a diagram showing an example of image data provided as training data to the learning device of the embodiment. FIG. [Diagram 3]3 is a diagram showing a process of generating rectangle position data, rectangle size data, line segment position data, and line segment type data from line segments indicating the outline of a building included in the image data of FIG. 2. [Figure 4] 13A to 13C are diagrams showing other examples of image data provided as training data to the learning device of the embodiment and the types of line segments showing the outline of a building contained in the image data. [Diagram 5] 13 is a diagram showing the types of line segments that indicate the outline of a building when a stepped building is included in image data provided as training data to the learning device of the embodiment. FIG. [Figure 6] 4A to 4C are diagrams showing rectangle position data, rectangle size data, and line segment position data given as training data to the learning device of the embodiment. [Figure 7] 2 is a block diagram showing a configuration of a function approximator included in the learning device of the embodiment. FIG. [Figure 8] 4 is a flowchart showing the flow of a learning process performed by the learning device of the embodiment. [Figure 9] 1 is a block diagram showing a configuration of a matching system according to an embodiment of the present invention. [Figure 10] 4 is a diagram showing a data configuration of a line segment identification data table stored in a line segment identification data storage unit of the building area detection device of the embodiment. FIG. [Figure 11] 4 is a diagram showing a data configuration of a building region identification data table stored in a building region identification data storage unit of the building region detection device of the embodiment. FIG. [Figure 12] 10 is a flowchart (part 1) showing a flow of processing by a building area detection unit of the building area detection device of the embodiment. [Figure 13] 13 is a flowchart (part 2) showing the flow of processing by the building area detection unit of the building area detection device of the embodiment. [Figure 14] 11 is a flowchart (part 3) showing the flow of processing by the building region detection unit of the building region detection device according to the embodiment. [Figure 15] 13 is a fourth flowchart showing the flow of processing by the building area detection unit of the building area detection device of the embodiment. [Figure 16] 10A to 10C are diagrams illustrating a process in which the building area detection device detects a building area from image data captured by the imaging device of the embodiment. [Figure 17] FIG. 11 is a diagram (part 1) showing a process of connecting line segments constituting the boundary of a building, which is performed by a building area detection unit of the building area detection device of the embodiment. [Figure 18] FIG. 11 is a diagram (part 2) showing the process of connecting line segments constituting the boundary of a building, which is performed by the building area detection unit of the building area detection device of the embodiment. [Figure 19] FIG. 11 is a diagram (part 3) showing the process of connecting line segments constituting the boundary of a building, which is performed by the building area detection unit of the building area detection device of the embodiment. [Figure 20] 13 is a supplementary explanatory diagram of the process of determining the arrangement order of building regions performed by the building region detection unit of the building region detection device of the embodiment. FIG. [Figure 21] 10 is a flowchart showing a flow of processing by a matching processing unit of the matching device of the embodiment. [Figure 22] 11A to 11C are diagrams illustrating a process of generating map image data on a projection surface of an imaging device from 2.5-dimensional map data by a matching processing unit of the matching device of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] (Learning device configuration) FIG. 1 is a block diagram showing a configuration of a learning device 1 according to an embodiment of the present invention. The learning device 1 includes an input unit 11, a function approximator 12, a learning processing unit 13, and a learning model data storage unit 14. The input unit 11 takes in a plurality of sets of training data prepared in advance, with each set being input image data, rectangle position data, rectangle size data, line segment position data, and line segment type data. The input unit 11 repeatedly selects the taken-in training data one set at a time. The input unit 11 outputs the input image data included in the selected set to the function approximator 12, and outputs the rectangle position data, rectangle size data, line segment position data, and line segment type data included in the set to the learning processing unit 13.

[0016] The input image data is, for example, image data 20 obtained by capturing an image of a street scene including buildings 22, 23, a road 21, and the like shown in Fig. 2. The position of each pixel included in the image data 20 is represented, for example, in an XY coordinate system in which the lower left corner is the origin, an X axis direction in which the X coordinate value increases toward the right, and a Y axis direction in which the Y coordinate value increases toward the top. The image data 20 is, for example, color image data having three channels of RGB.

[0017] 3 to 6, rectangle position data, rectangle size data, line segment position data, and line segment type data other than the input image data included in the training data set will be described. Fig. 3(a) is a diagram showing the process of generating rectangle position data, rectangle size data, line segment position data, and line segment type data from an image of a building 22. Lines indicating edges are detected from the image data 20 by image processing for edge detection. The detected lines may include lines indicating the outline of the building 22 as well as lines indicating the shapes of the balconies and windows of the building 22, so the lines indicating the outline of the building 22 are selected from the lines.

[0018] The lower part 22z of the building 22 is appropriately determined according to the shape of the building 22. Rectangular areas 22a, 22b, 22c, 22d, and 22e, which are rectangular areas, are determined for all line segments excluding the line segments whose both ends belong to the lower part 22z of the building 22 from among the line segments showing the outline of the building 22, that is, the line segments on the bottom side of the building 22. The size of the rectangular areas 22a to 22e is a size that surrounds one line segment. A line segment that is not entirely included in each of the rectangular areas 22a to 22e is not considered to be a target line segment of each of the rectangular areas 22a to 22e. For example, the rectangular area 22a includes the line segment on the left edge of the building 22 and a part of the line segment on the upper edge of the building 22 that is included in the rectangular area 22b. Of these line segments, the line segment whose entire portion is included in the rectangular area 22a is the line segment on the left edge of the building 22. Therefore, the line segments covered by the rectangular area 22a are only the line segments on the left edge of the building 22.

[0019] The line segments covered by each of rectangular regions 22a to 22e are classified as follows. The line segments covered by each of rectangular regions 22a to 22e are classified into line segments that make up the boundary of the area of ​​building 22 and other line segments. In the case of FIG. 3(a), the line segments that make up the boundary of the area of ​​building 22 are rectangular regions 22a, 22b, 22d, and 22e. The line segment included in rectangular region 22c exists within the area of ​​building 22, and therefore is not a line segment that makes up the boundary of the area of ​​building 22.

[0020] The line segments included in rectangular areas 22a, 22b, 22d, and 22e that constitute the boundary of the area of ​​building 22 are classified into line segments whose one end belongs to lower portion 22z and line segments whose both ends do not belong to lower portion 22z. Line segments whose one end belongs to lower portion 22z are line segments included in rectangular areas 22a and 22e, and these line segments are classified as a first type, that is, a "Corner" type line segment that visually indicates the edge of building 22. Line segments whose both ends do not belong to lower portion 22z are line segments included in rectangular areas 22b and 22d, and these line segments are classified as a second type, that is, a "Top" type line segment.

[0021] A line segment that indicates the outline of the building 22 but is not a line segment that constitutes the boundary of the area of ​​the building 22 is classified as a third type, i.e., a "Center" type line segment, if the line segment is located at a corner of the building 22 when viewed from above. A line segment included in the rectangular area 22c is a line segment that is located at a corner of the building 22 when viewed from above, and is therefore classified as a "Center" type line segment.

[0022] As shown in Fig. 3(b), rectangular regions 23a, 23b, 23c, 23d, and 23e are defined for building 23, similar to the case of building 22. In the case of building 23, a part of the region is hidden by building 22, so a lower portion 23z as shown in Fig. 3(b) is defined.

[0023] As a result, the line segments 25a, 25b, 25c, 25d, and 25e that are the subject of each of the rectangular areas 22a to 22e of the building 22, and the line segments 26a, 26b, 26c, 26d, and 26e that are the subject of each of the rectangular areas 23a to 23e of the building 23, are classified into the types shown in Fig. 3(c). The types "Corner", "Top", and "Center" become the line segment type data for each of the above-mentioned line segments.

[0024] In addition, in the image data 20 shown in FIG. 2, the height of building 22 is lower than that of building 23, and the top of building 23 is visible, so line segment 26c in building 23 shown in FIG. 3(c) is classified as a “Center” type.

[0025] If line segment 25e of building 22 and line segment 26c of building 23 shown in FIG. 3(c) are captured in a state where they appear to overlap on the same line, line segment 25e is classified as a "Corner" type, and line segment 26c above and not overlapping with line segment 25e is classified as a "Center" type. However, in reality, it is considered that multiple line segments are rarely captured in a completely overlapping state. Even if line segment 25e and line segment 26c appear to overlap, they are basically not recognized as completely overlapping, but are recognized as separate line segments slightly shifted from each other. As a result, line segment 25e is classified as a "Corner" type, and line segment 26c is classified as a "Center" type.

[0026] In addition, as in image data 20A shown in Fig. 4(a), when building 22 is taller than building 23A, the top of building 23A is behind building 22. In this case, the line segments obtained by edge detection from building 23A are three line segments 26Aa, 26Ab, and 26Ac, as shown in Fig. 4(b). In this case, line segment 26Aa would be classified as a "Center" type like line segment 26c if building 22 did not exist, but because building 22 exists, line segment 26Aa visually indicates the edge of building 23A, and is therefore classified as a "Corner" type.

[0027] In addition, in the line segments showing the outer shape of the building 24 having a staircase shape as shown in FIG. 5, line segment 27f is detected as two line segments when edge detection is performed because line segment 27k exists. However, when multiple line segments exist on a straight line or approximately on a straight line in the vertical direction, the multiple line segments are regarded as one line segment 27f to determine a rectangular area. In addition, in the case of a shape such as building 27, line segment 27k is not a line segment showing the outer shape of building 27, but line segments 27i and 27j are line segments showing the outer shape of building 27. However, line segments 27i and 27j do not correspond to any of the above-mentioned types of "Corner", "Top", and "Center". Therefore, here, the rectangular areas for line segments 27i and 27j are not included in the training data because there is no line segment type data. If line segments of the type of line segments 27i and 27j are also to be targeted, it is necessary to separately determine types other than "Corner", "Top", and "Center".

[0028] Next, the rectangular position data, rectangular size data, and line segment position data will be described with reference to Fig. 6. Fig. 6 is a diagram showing one rectangular area 31, and the rectangular area 31 is one of the rectangular areas defined in the image data 20, such as the above-mentioned rectangular areas 22a to 22e and 23a to 23e. The rectangular area 31 includes one line segment 33 that is the subject of the rectangular area 31. The rectangular position data of the rectangular area 31 is data indicating the position indicated by the reference numeral 32, and the position of the reference numeral 32 is the center of the rectangular area 31, and is represented by the X coordinate value and the Y coordinate value in the XY coordinate system of the image data 20. The rectangular size data of the rectangular area 31 is data indicating the width w and height h of the rectangular area 31, and the values ​​of the width w and height h are values ​​calculated as distances in the XY coordinate system of the image data 20.

[0029] The line segment position data of rectangular area 31 is represented as the relative positions in rectangular area 31 of both end points of line segment 33 included in rectangular area 31. That is, the relative position in rectangular area 31 of one end point 34-1 of line segment 33 is represented as S1 / w and S2 / h. The relative position in rectangular area 31 of the other end point 34-2 of line segment 33 is represented as S3 / w and S4 / h. The length values ​​of S1, S2, S3, and S4 are values ​​calculated as distances in the XY coordinate system of image data 20.

[0030] This makes it possible to generate rectangle position data (x, y), rectangle size data (w, h), line segment position data (S1 / w, S2 / h, S3 / w, S4 / h), and line segment type data which is one of "Corner", "Top", or "Center" for each of the rectangular areas 22a-22e for the building 22 and each of the rectangular areas 23a-23e for the building 23 in the image data 20 shown in Figure 3.

[0031] Similarly, for multiple different input image data other than image data 20, rectangle position data, rectangle size data, line segment position data, and line segment type data are generated, and the input image data, rectangle position data, rectangle size data, line segment position data, and line segment type data are treated as one set of training data, and multiple sets are prepared in advance as training data.

[0032] Returning to Fig. 1, the learning model data storage unit 14 stores learning model data including parameters such as weights and biases applied to the function approximator 12. The function approximator 12 is, for example, a deep neural network, and includes neural networks 15-1, 15-2, and 15-3, and a parameter acquisition unit 16, as shown in Fig. 7. Upon receiving an acquisition instruction signal from the learning processing unit 13, the parameter acquisition unit 16 reads out the learning model data stored in the learning model data storage unit 14, and applies each of the parameters included in the learning model data to the corresponding neural networks 15-1, 15-2, and 15-3.

[0033] Neural network 15-1 has an input layer that takes in input image data, which is three-channel color image data output by input unit 11. The output layer of neural network 15-2 is composed of N×M×3 neurons. The output layer corresponding to the line segment type data of "Corner" is output layer 17a, the output layer corresponding to the line segment type data of "Top" is output layer 17b, and the output layer corresponding to the line segment type data of "Center" is output layer 17c. Each of output layers 17a to 17c has N×M neurons.

[0034] The output layer of the neural network 15-3 is composed of N×M×8 neurons. Here, the output layer corresponding to the X coordinate value "x" of the rectangle position data is output layer 18a, and the output layer corresponding to the Y coordinate value "y" is output layer 18b. The output layer corresponding to the width "w" of the rectangle size data is output layer 18c, and the output layer corresponding to the height "h" is output layer 18d. The output layer corresponding to "S1 / w" of the line segment position data is output layer 18e, the output layer corresponding to "S2 / h" is output layer 18f, the output layer corresponding to "S3 / w" is output layer 18g, and the output layer corresponding to "S4 / h" is output layer 18h. Each of the output layers 18a to 18h has N×M neurons.

[0035] Therefore, the function approximator 12 outputs N×M×11 pieces of data for one input image data. Note that, although N=M=8 is shown as an example in Fig. 7, the values ​​of N and M may be values ​​other than 8, and N and M may be different values. Since the amount of calculation increases as the values ​​of N and M become larger, appropriate values ​​are determined in advance according to the resolution of the input image data and the required detection accuracy.

[0036] As the neural network configuration between the input layer of neural network 15-1 and the output layers 17a to 17c, 18a to 18h of neural networks 15-2 and 15-3, for example, the deep neural network configuration of EfficientDet used for object detection disclosed in Reference 1 below is applied.

[0037] "Reference 1: M. Tan, R. Pang and QV Le, EfficientDet: Scalable and Efficient Object Detection, CVPR, 2020."

[0038] In FIG. 7, each of the output layers 17a to 17c and 18a to 18h has 8×8=64 neurons as an example. The 64 neurons correspond to each divided area obtained by dividing the area of ​​the input image data into 64 areas of equal size. The ideal state of the function approximator 12, which is the target of the learning process performed by the learning processing unit 13, is as follows. For example, assume that there are multiple "Corner" line segments in the input image data. The neurons in the output layer 17a corresponding to the divided areas in which the centers of the rectangular areas corresponding to each of the multiple "Corner" line segments are present among the 64 divided areas of the input image data, output a value of "1.0", and the other neurons output "0.0".

[0039] Also, suppose that the neurons that output "1.0" in the output layer 17a are, for example, neuron 17a-4-3 in the fourth row and third column of the output layer 17a and neuron 17a-6-7 in the sixth row and seventh column, which are filled in black in Fig. 7. In this case, the ideal state of the function approximator 12 is a state in which the neurons in the fourth row and third column of each of the output layers 18a to 18h output the values ​​of "x" and "y" of the rectangle position data, the values ​​of "w" and "h" of the rectangle size data, and the values ​​of "S1 / w", "S2 / h", "S3 / w", and "S4 / h" of the line segment position data of the rectangular area including the line segment of "Corner" whose center is located in the divided area in the fourth row and third column. In addition, the neuron in the 6th row and 7th column of each of output layers 18a to 18h will be in a state where it outputs the values ​​of "x" and "y" in the rectangular position data, the values ​​of "w" and "h" in the rectangular size data, and the values ​​of "S1 / w", "S2 / h", "S3 / w", and "S4 / h" in the line segment position data of the rectangular area containing the "Corner" line segment whose center is in the divided area in the 6th row and 7th column.

[0040] The learning processing unit 13 calculates new learning model data based on the data output from each of the output layers 17a-17c, 18a-18h of the function approximator 12 by receiving the input image data, and the data output from the input unit 11, including the rectangle position data, rectangle size data, line segment position data, and line segment type data, corresponding to the input image data, so as to approach the ideal state of the function approximator 12. The learning processing unit 13 rewrites and updates the learning model data stored in the learning model data storage unit 14 with the calculated new learning model data. The learning processing unit 13 also outputs an acquisition instruction signal to the parameter acquisition unit 16 of the function approximator 12 at the timing of changing the parameters to be applied to the neural networks 15-1, 15-2, and 15-3 of the function approximator 12.

[0041] (Processing by learning device) 8 is a flowchart showing the flow of the learning process by the learning device 1. The learning model data storage unit 14 stores in advance learning model data of initial values ​​initialized with random numbers. When the learning device 1 is started, the parameter acquisition unit 16 of the function approximator 12 reads out the learning model data of initial values ​​stored in the learning model data storage unit 14, and applies each of the parameters included in the read learning model data to the corresponding neural networks 15-1, 15-2, and 15-3. The input unit 11 takes in training data (step Sa1).

[0042] The input unit 11 repeatedly selects one set at a time from the multiple sets included in the imported training data. The input unit 11 outputs the input image data included in the selected set to the function approximator 12. The input unit 11 outputs the rectangle position data, rectangle size data, line segment position data, and line segment type data included in the selected set to the learning processing unit 13. The input layer of the neural network 15-1 of the function approximator 12 imports the input image data output by the input unit 11. The neural network 15-1 performs a calculation according to the internal neuron connection configuration based on the parameters applied to the neural network 15-1 by the parameter acquisition unit 16 and the imported input image data, and outputs the data of the calculation result to the neural networks 15-2 and 15-3.

[0043] The input layer of neural network 15-2 takes in the data of the calculation result output by neural network 15-1, performs calculations according to the internal neuron connection configuration based on the parameters applied to neural network 15-2 by parameter acquisition unit 16 and the data of the calculation result that was taken in, and neurons in each of output layers 17a to 17c output the data of the calculation result. The input layer of neural network 15-3 takes in the data of the calculation result output by neural network 15-1, performs calculations according to the internal neuron connection configuration based on the parameters applied to neural network 15-3 by parameter acquisition unit 16 and the data of the calculation result that was taken in, and neurons in each of output layers 18a to 18h output the data of the calculation result.

[0044] The learning processing unit 13 takes in the rectangle position data, rectangle size data, line segment position data, and line segment type data output by the input unit 11. The learning processing unit 13 calculates new learning model data based on the error between the taken-in rectangle position data, rectangle size data, line segment position data, and line segment type data and the data output by the neurons included in each of the output layers 17a to 17c, 18a to 18h so as to approach the ideal state of the function approximator 12. The learning processing unit 13 may perform online learning to calculate new learning model data based on the error for each set output by the input unit 11, or may perform mini-batch learning to calculate new learning model data based on the error of each set that is the subject of calculation for each of a predetermined number of sets, or may perform batch learning to calculate new learning model data based on the error of each of all sets included in the training data.

[0045] Every time new learning model data is calculated, the learning processing unit 13 rewrites and updates the learning model data stored in the learning model data storage unit 14 with the calculated new learning model data. When the learning processing unit 13 updates the learning model data stored in the learning model data storage unit 14, it outputs an acquisition instruction signal to the parameter acquisition unit 16 of the function approximator 12. When the parameter acquisition unit 16 receives the acquisition instruction signal from the learning processing unit 13, it reads out the learning model data from the learning model data storage unit 14 and applies each of the parameters included in the read learning model data to the corresponding neural networks 15-1, 15-2, and 15-3 (step Sa2).

[0046] The learning processing unit 13 performs a process of rewriting and updating the learning model data stored in the learning model data memory unit 14 with the calculated new learning model data a predetermined number of times, so that the learned learning model data is written and stored in the learning model data memory unit 14 (step Sa3), and the learning device 1 terminates the processing.

[0047] In the above embodiment, the "x" and "y" values ​​of the rectangular position data are values ​​in the XY coordinate system of the image data 20, i.e., values ​​expressed in the absolute coordinate system. Alternatively, the region of the image data 20 may be divided into N x M equally sized divided regions, and the XY coordinates may be determined for each divided region, and the "x" and "y" values ​​of the rectangular position data may be values ​​expressed in a relative coordinate system in which the center position of the rectangular region in the XY coordinate system is the "x" and "y" values ​​of the rectangular position data. However, when values ​​expressed in the relative coordinate system are used, a process of converting absolute coordinate values ​​to relative coordinate values ​​and a process of converting relative coordinate values ​​to absolute coordinate values ​​are performed as necessary.

[0048] The learning device 1 according to the above embodiment includes an input unit 11 and a learning processing unit 13. The input unit 11 takes in image data 20 and information indicating a rectangular area appropriately determined to include one detection target line segment included in the image data 20. The detection target line segment is a line segment of the type "Corner" which is a first type of line segment, and a line segment of the type "Top" which is a second type of line segment. The "Corner" is a line segment (25a, 25b, 25d, 25e, 26a, 26b, 26d, 26e) that constitutes the boundary of the area of ​​the building selected from the line segments (25a to 25e, 26a to 26e) that indicate the outline of each building 22, 23 included in the image data 20, and one end of which belongs to the lower part 22z, 23z of the building that is appropriately determined for each building 22, 23. The "Top" is a line segment that constitutes the boundary of the area of ​​the building 22, 23, and both ends of which do not belong to the lower part. The information indicating the rectangular area is information on each of the rectangular areas (22a, 22b, 22d, 22e, 23a, 32b, 23d, 23e) appropriately determined so as to include one detection target line segment. The information indicating the rectangular area is rectangle position data indicating the position of the rectangular area in the image data 20, rectangle size data indicating the size of the rectangular area, line segment position data indicating the positions of both end points of the line segment included in the rectangular area in the rectangular area, and line segment type data indicating whether the detection target line segment is a first type or a second type. The learning processing unit 13 performs a learning process. The learning process is a process of updating the learning model data applied to the function approximator 12. In the learning process, when the image data 20 taken in by the input unit 11 is given to the function approximator 12, the learning model data is updated so as to approach a state in which the function approximator 12 outputs the rectangle position data, rectangle size data, line segment position data, and line segment type data taken in by the input unit 11. As a result, by finally generating trained learning model data and applying the generated trained learning model data to function approximator 12, it becomes possible to detect with high accuracy the line segments that form the outline of individual buildings and the boundary of the building area from among the line segments that indicate edges that can be detected by edge detection from any image data.

[0049] (Matching system configuration) FIG. 9 is a block diagram showing a configuration of a matching system 200 according to an embodiment of the present invention. The matching system 200 includes a building region detection device 50, a moving body 60, and a matching device 70. The moving body 60 is, for example, a vehicle such as an automobile, and includes an imaging device 61, a position detection device 62, and a communication device 63. The imaging device 61 is, for example, a camera that repeatedly captures images of scenery along the traveling direction of the moving body 60 traveling on a road at regular intervals, and generates image data by capturing the images. The imaging device 61 acquires time data indicating the time from a clocking means such as a clock provided inside each time the imaging device 61 captures an image, and generates an image ID that is identification information that can uniquely identify each image data. The imaging device 61 outputs the acquired time data, the generated image ID, data indicating the angle of view at the time of capturing the image, and the generated image data in association with each other.

[0050] The position detection device 62 is, for example, a GPS terminal device that receives radio waves from a GPS satellite to detect a position, and outputs coordinate data indicating the detected position and time data indicating the detection time in association with each other. The communication device 63 is, for example, a wireless communication terminal device, and transmits image data output by the imaging device 61 and an image ID associated with the image data to the building area detection device 50. The communication device 63 also transmits the image ID output by the imaging device 61 and time data and data indicating the angle of view associated with the image ID to the matching device 70. The communication device 63 also transmits the coordinate data output by the position detection device 62 and time data associated with the coordinate data to the matching device 70.

[0051] The building area detection device 50 includes a receiving unit 51, a trained learning model data storage unit 52, a line segment detection unit 53, a line segment identification data storage unit 54, a semantic segmentation unit 56, a building area detection unit 57, and a building area identification data storage unit 58. In the building area detection device 50, the receiving unit 51 receives image data transmitted by a communication device 63 of a mobile object 60 and an image ID associated with the image data. The trained learning model data storage unit 52 stores in advance trained learning model data written in the learning model data storage unit 14 of the learning device 1.

[0052] The line segment detection unit 53 includes a function approximator having the same configuration as the function approximator 12 included in the learning device 1. Hereinafter, the function approximator included in the line segment detection unit 53 will also be referred to as the function approximator 12 with the reference numeral 12 attached thereto. The line segment detection unit 53 provides the image data received by the receiving unit 51 to the function approximator 12 to which the learned learning model data stored in the learned learning model data storage unit 52 is applied, thereby detecting rectangle position data, rectangle size data, line segment position data, and line segment type data for each detection target line segment of the type "Corner", "Top", or "Center" from the image data. The line segment detection unit 53 generates line segment identification data for each detection target line segment by adding the image ID received by the receiving unit 51 to a combination of the rectangle position data, rectangle size data, line segment position data, and line segment type data for each detection target line segment detected.

[0053] The line segment identification data storage unit 54 stores a line segment identification data table 55 in the data format shown in Fig. 10. The line segment identification data table 55 has the following items: "Image ID", "Line segment type", "Rectangle position", "Rectangle size", and "Line segment position". The image ID, line segment type data, rectangle position data, rectangle size data, and line segment position data contained in each piece of line segment identification data generated by the line segment detection unit 53 are written into each of the items: "Image ID", "Line segment type", "Rectangle position", "Rectangle size", and "Line segment position".

[0054] The semantic segmentation unit 56 includes a deep neural network to which trained learning model data generated in advance by a learning process for detecting a part of a building area included in image data is applied. As the deep neural network, for example, U-Net disclosed in Reference 2 below is applied.

[0055] “Reference 2: O. Ronneberger, P. Fischer and T. Brox, U Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:1505.04597v1, 2015.”

[0056] In the case of adjacent buildings, the deep neural network provided in the semantic segmentation unit 56 does not detect the areas of each individual building, but rather detects the area of ​​the building by treating the adjacent buildings as a single building.

[0057] The building area detection unit 57 selects line segment identification data having the same image ID from the line segment identification data stored in the line segment identification data table 55 of the line segment identification data storage unit 54. The building area detection unit 57 selects line segments that are connected and form the boundary of the building area based on the selected line segment identification data having the same image ID, and detects the area of ​​each building included in the image data corresponding to the image ID by connecting the selected line segments. The building area detection unit 57 also detects an overlapping area between the area of ​​each detected building and the area of ​​the building detected by the semantic segmentation unit 56 that corresponds to the image ID of the area of ​​the building detected by the building area detection unit 57, and sets the detected overlapping area as the final area of ​​the building for each building.

[0058] Furthermore, the building area detection unit 57 generates a building area ID, which is identification information for uniquely identifying the area of ​​the building, for each of the detected building areas of each individual building. Furthermore, the building area detection unit 57 generates building area identification data including building area range data for identifying the range of the area of ​​the building, a building area ID corresponding to the area of ​​the building, and an image ID corresponding to the area of ​​the building. Here, the building area range data is, for example, coordinate data of the XY coordinate system of image data, in which a plurality of coordinate data indicating the range of the area of ​​the building are arranged in the order of connection. Furthermore, the building area detection unit 57 selects building area identification data having the same image ID, and determines the order of arrangement of the building area identification data so that the taller the building area is, based on the height of the building area indicated by the building area range data included in the selected building area identification data, the building area identification data is arranged in the rear, and data indicating the determined order of arrangement is added to the building area identification data.

[0059] The building area identification data storage unit 58 stores a building area identification data table 59 in the data format shown in Fig. 11. The building area identification data table 59 has the items of "Image ID", "Building area ID", "Building area range", and "Arrangement order". The image ID, building area ID, building area range data, and data indicating the arrangement order contained in the building area identification data generated by the building area detection unit 57 are written in each of the items of "Image ID", "Building area ID", "Building area range", and "Arrangement order".

[0060] The matching device 70 includes a matching processing unit 71, a map data storage unit 72, a receiving unit 73, and a history information storage unit 74. The receiving unit 73 receives an image ID transmitted by the communication device 63 of the mobile object 60, time data and data indicating the angle of view associated with the image ID, coordinate data, and time data associated with the coordinate data. The history information storage unit 74 stores the image ID received by the receiving unit 73, the time data and data indicating the angle of view associated with the image ID, in chronological order, and stores the coordinate data and the time data associated with the coordinate data in chronological order. The map data storage unit 72 stores in advance 2.5-dimensional map data of the vicinity where the mobile object 60 travels. Here, the 2.5-dimensional map is, for example, a map in which, when the XY-axis plane is the ground and the Z-axis is the direction toward the sky, there is data indicating the outline of a building on the XY-axis plane, and the outline is elevated to a predetermined height in the Z-axis direction in a constant state, and does not include a texture (pattern) of the building surface. The height of a building is defined as the representative height of the building, which is basically the height corresponding to the highest point.

[0061] Even if there is no information indicating the height of the buildings, it is possible to match buildings in an image with some buildings on a map (buildings that are visually in the foreground in the image). By using information indicating the height of buildings, it becomes possible to match tall buildings behind short buildings, making it possible to match more buildings.

[0062] The matching processing unit 71 is connected to the building area identification data storage unit 58 of the building area detection device 50, and matches the building area indicated by the building area identification data stored in the building area identification data table 59 of the building area identification data storage unit 58 with map image data generated from the 2.5D map data stored in the map data storage unit 72.

[0063] In addition, the matching processing unit 71 and the building region identification data storage unit 58 are connected by a wired electric circuit or the like when the building region detection device 50 and the matching device 70 are located close to each other, and are connected by a wired or wireless communication circuit when the building region detection device 50 and the matching device 70 are located far from each other. In addition, FIG. 9 shows an example in which the building region detection device 50 and the mobile body 60 and the matching device 70 and the mobile body 60 are connected by a wireless communication circuit, but the mobile body 60 may be equipped with either or both of the building region detection device 50 and the matching device 70. In this case, the mobile body 60 and the device equipped in the mobile body 60 may be connected by a wired electric circuit instead of a wireless communication circuit. In addition, when the mobile body 60 is equipped with either the building region detection device 50 or the matching device 70, the matching processing unit 71 and the building region identification data storage unit 58 are connected by a wireless communication circuit.

[0064] (Processing by building area detection device) The processing by the building area detection device 50 will be described with reference to Fig. 12 to Fig. 20. Fig. 12 to Fig. 15 are flowcharts showing the flow of processing by the building area detection unit 57 of the building area detection device 50. In the flowcharts of Fig. 12 to Fig. 15, connection marks in the form of circles containing the letters "A", "B", "C", "D", and "E" indicate that processing is continuing.

[0065] Before the flow chart of FIG. 12 is started, the following process is performed. When the building area detection device 50 is started, the parameter acquisition unit 16 of the function approximator 12 included in the line segment detection unit 53 of the building area detection device 50 reads out the trained learning model data from the trained learning model data storage unit 52, and applies each of the parameters included in the read learning model data to the corresponding neural networks 15-1, 15-2, and 15-3. The reception unit 51 of the building area detection device 50 receives image data transmitted by the communication device 63 of the mobile object 60 and an image ID associated with the image data. The reception unit 51 outputs the received image data and the image ID to the line segment detection unit 53 and the semantic segmentation unit 56. Here, it is assumed that the image data output by the reception unit 51 is, for example, image data 80 shown in FIG. 16(a).

[0066] The line segment detection unit 53 captures the image data 80 output by the receiving unit 51 and the image ID associated with the image data 80, and provides the captured image data 80 to the function approximator 12 to generate line segment identification data for each of the detection target line segments of the types "Corner", "Top", and "Center" from the data output by the function approximator 12. When the line segments corresponding to the line segment identification data generated by the line segment detection unit 53 are displayed on the display screen, for example, line segments 81a to 81e, 82a to 82e, and 83a to 83c for each of the buildings 81, 82, and 83 as shown in FIG. 16(b) are displayed. The line segment detection unit 53 writes and stores the generated line segment identification data in the line segment identification data table 55 of the line segment identification data storage unit 54.

[0067] The semantic segmentation unit 56 captures the image data 80 output by the receiving unit 51 and the image ID associated with the image data 80, and detects the building area included in the image data 80 by providing the captured image data 80 to a function approximator provided inside. The semantic segmentation unit 56 associates the captured image ID with segmentation data indicating the range of the detected building area and outputs the data to the building area detection unit 57. When the segmentation data detected by the semantic segmentation unit 56 is displayed on the display screen, for example, a building area 90 as shown in FIG. 16(d) is displayed.

[0068] The building area detection unit 57 takes in the segmentation data and the image ID output by the semantic segmentation unit 56, and writes and stores the taken-in segmentation data and the image ID in an internal storage area in association with each other. When the line segment identification data table 55 stores all the line segment identification data of the new image data 80, the building area detection unit 57 selects and reads all the line segment identification data of the image ID corresponding to the new image data 80 from the line segment identification data table 55. The building area detection unit 57 calculates coordinate data indicating the positions of both end points of the line segment corresponding to each of the line segment identification data in the coordinate values ​​of the XY coordinate system of the image data 80 based on the rectangle position data, rectangle size data, and line segment position data included in the read line segment identification data. The building area detection unit 57 starts the process of the flowchart shown in FIG. 12.

[0069] The building area detection unit 57 refers to the line type data included in the read line identification data and performs a process of selecting lines that are connected from the lines that make up the boundary of the building area. When the image data includes an entire image of a building, the types of lines that make up the boundary of the building area are arranged in the order of "Corner", at least one "Top", and "Corner" when viewed from the left or right side, and are connected in this order.

[0070] The building area detection unit 57 refers to the line segment type data included in the read line segment identification data, and detects line segment identification data of the type "Corner" from among unprocessed line segments, i.e., line segments that have not been selected as a starting line segment before the processing and have not been selected as a connection target for other lines. The building area detection unit 57 refers to the rectangular position data of the detected line segment identification data, and selects the line segment located at the leftmost position in the XY coordinate system as the starting line segment. Here, it is assumed that the building area detection unit 57 selects line segment 81a of a building 81 included in image data 80 as the starting line segment, as shown in FIG. 17 (step Sb1).

[0071] The building region detection unit 57 selects the end point 81a-2, which is the end point with the larger Y coordinate value, from the two end points 81a-1, 81a-2 of the line segment 81a (step Sb2). The reason why the end point with the larger Y coordinate value is selected here is that when generating training data to be applied to the learning device 1, the position of the end point with the smaller Y coordinate value of the line segment of the type "Corner" belongs to the lower part of the building, and it is assumed that there is no connecting line segment. The building region detection unit 57 determines whether the coordinate of the end point 81a-1 is the coordinate of the edge of the image data 80 (step Sb3). Here, the case where it is the coordinate of the edge of the image data 80 means that the value of the X coordinate is "0" or the maximum value, or the value of the Y coordinate is "0" or the maximum value. When the building area detection unit 57 determines that the coordinates of the end point 81a-1 are the coordinates of the edge of the image data 80 (step Sb3, Yes), the line segments constituting the boundary of the building area are not arranged in the order of "Corner", at least one "Top", and then "Corner". Therefore, as an exceptional process, the process of the flowchart shown in FIG. 13 is performed, as indicated by the connection mark A.

[0072] As shown in FIG. 17, the coordinates of the end point 81a-1 are not the coordinates of the edge of the image data 80. Therefore, the building area detection unit 57 determines that the end point 81a-1 is not the coordinates of the edge of the image data 80 (step Sb3, No), and then determines a neighborhood range 101 of a predetermined size centered on the position of the end point 81a-1 (step Sb4). The building area detection unit 57 refers to the read line segment identification data and determines whether or not an end point of an unprocessed line segment of the type "Top" exists in the determined neighborhood range 101 (step Sb5). When the building area detection unit 57 determines that an end point of an unprocessed line segment of the type "Top" does not exist in the determined neighborhood range 101 (step Sb5, No), in this case, there is no other line segment that constitutes the boundary of the area of ​​the building 81 connected to the start line segment, so the process of identifying the area of ​​the building for the start line segment is not performed, and the process proceeds to step Sb13.

[0073] On the other hand, when the building region detection unit 57 determines that an end point of an unprocessed line segment of the type "Top" exists in the defined vicinity range 101 (step Sb5, Yes), the process proceeds to step Sb6. In the case of the end point 81a-1, the end point 81b-1 of the unprocessed line segment 81b of the type "Top" exists in the defined vicinity range 101. Therefore, in step Sb5, the building region detection unit 57 determines that the end point 81b-1 of the unprocessed line segment 81b of the type "Top" exists in the defined vicinity range 101. Note that, when the building region detection unit 57 detects end points of a plurality of unprocessed line segments of the type "Top" in the vicinity range 101, it selects the end point closest to the center of the vicinity range 101. The building region detection unit 57 selects the line segment 81b as the line segment to be connected (step Sb6).

[0074] The building region detection unit 57 determines a neighborhood range 102 of a predetermined size centered on the position of the other end point 81b-2 of the line segment 81b selected as the connection target (step Sb7). The building region detection unit 57 refers to the read line segment identification data and determines whether or not an end point of an unprocessed line segment of the type "Top" or "Corner" is present in the determined neighborhood range 102 (step Sb8). If the building region detection unit 57 determines that an end point of an unprocessed line segment of the type "Top" or "Corner" is present in the determined neighborhood range 102 (step Sb8, Yes), the process proceeds to step Sb9. In the case of the end point 81b-2, an end point 81d-1 of the line segment 81d of the type "Top" that is not yet processed is present in the neighborhood range 102. Therefore, in step Sb8, the building region detection unit 57 determines that the end point 81d-1 of the unprocessed line segment 81d of the "Top" type is present in the defined vicinity range 102. If the building region detection unit 57 detects multiple end points of unprocessed line segments of the "Top" or "Corner" type in the vicinity range 102, it selects the end point of the "Corner" type with priority, and if it detects multiple unprocessed end points of the "Corner" type, it selects the end point of the "Corner" type that is closest to the center of the vicinity range 101. The building region detection unit 57 selects the line segment 81d as the line segment to be connected (step Sb9).

[0075] On the other hand, if the building area detection unit 57 determines in step Sb8 that there are no unprocessed end points of line segments of type "Top" and "Corner" within the defined vicinity range 102 (step Sb8, No), the building area detection unit 57 determines whether the coordinates of end point 81b-2 are the coordinates of the edge of the image data 80 (step Sb10).

[0076] When the building area detection unit 57 determines that the coordinates of the end point of the processing target are the coordinates of the edge of the image data 80 (step Sb10, Yes), in this case, the line segments constituting the boundary of the building area are not arranged in the order of "Corner", at least one "Top", and "Corner". Therefore, as an exceptional process, the process of the flowchart shown in FIG. 13 is performed as indicated by the connection mark A. On the other hand, when the building area detection unit 57 determines that the coordinates of the end point of the processing target are not the coordinates of the edge of the image data 80 (step Sb10, No), in this case, there is no other line segment connected to the line segment of the connection target, and the area of ​​the building cannot be specified. Therefore, the line segment selected as the connection target in step Sb6 or step Sb9 is released from the connection target, and the process proceeds to step Sb13.

[0077] After the process of step Sb9, the building area detection unit 57 judges whether the type of the line segment 81d is "Corner" or not based on the line segment type data of the line segment identification data of the line segment 81d selected as the connection target (step Sb11). In the case of the line segment 81d, since the type of the line segment is "Top", the building area detection unit 57 judges that the type of the line segment is not "Corner" (step Sb11, No) and proceeds to the process of step Sb7. By performing the processes of step Sb7 and thereafter again, the building area detection unit 57 further selects the line segment 81e shown in FIG. 17 as the connection target. Since the type of the line segment 81e is "Corner", the building area detection unit 57 judges that the type of the line segment 81e is "Corner" in the process of step Sb11 (step Sb11, Yes). As a result, starting from the starting line segment, which is the "Corner" line segment 81a, the building area detection unit 57 selects the two "Top" line segments 81b and 81d and the "Corner" line segment 81e as line segments to be connected.

[0078] The building area detection unit 57 extends the line segments 81a and 81e of "Corner" in the vertical direction to the bottom edge of the image data 80. The building area detection unit 57 connects the line segments 81b and 81d of "Top" and the line segment obtained by extending the line segment 81e of "Corner" in the order of selection to the line segment obtained by extending the line segment 81a, which is the starting line segment. The building area detection unit 57 determines the area surrounded by the connected line segments as the building area corresponding to the building 81. The building area detection unit 57 generates building area range data by arranging the coordinate data of the XY coordinate system surrounding the area of ​​the building in the order of connection. The building area detection unit 57 generates a building area ID for the building 81, generates building area identification data including the image ID of the image data 80, the generated building area ID, and the generated building area range data, and writes and stores the generated building area identification data in an internal storage area (step Sb12).

[0079] The building region detection unit 57 advances the process to step Sb13 and determines whether or not there is any unprocessed line segment identification data of the type "Corner" in the read line segment identification data (step Sb13). As described above, an unprocessed line segment is a line segment that has not been selected as a start line segment and has not been selected as a connection target for another line segment. Note that a line segment that has been selected as a connection target for another line segment and then released from the connection target line segment is included in the unprocessed line segments.

[0080] When the building area detection unit 57 determines that there is line segment identification data of the type "Corner" for an unprocessed line segment (step Sb13, Yes), it performs the process of step Sb1 again. On the other hand, when the building area detection unit 57 determines that there is no line segment identification data of the type "Corner" for an unprocessed line segment (step Sb13, No), it means that the detection of the building area based on the line segment identification data corresponding to the image ID of the image data 80 is completed, and thereafter, as indicated by the connection mark B, the process of the flowchart shown in FIG. 14 is performed.

[0081] (Exception handling when the building area is bordering the edge of the image data) Fig. 13 is a flowchart showing the flow of processing performed after connection mark A in the flowchart of Fig. 12. Building area detection unit 57 determines whether the position of the end point to be processed is the upper edge, either the left or right edge, or the lower edge of image data 80 (step Sc1).

[0082] For example, suppose that the image data 80 includes images of buildings 85 and 86 that are not shown in FIG. 16(a). Suppose that when the line segment identification data of the buildings 85 and 86 are displayed on the display screen, the state shown in FIG. 18(a) is obtained. As shown in FIG. 18(a), the entire image of the building 85 does not fall within the range of the image data 80, and the upper part is not visible. Similarly, the entire image of the building 86 does not fall within the range of the image data 80, and the upper part and the right side are not visible. In this case, it can be assumed that the line segment 85a of the "Corner" and the line segment 85c of the "Top" are connected via another line segment of the "Top" that is not visible. Also, for example, suppose that the image data 80 includes an image of a building 87 that is not shown in FIG. 16(a). Suppose that when the line segment identification data of the building 87 is displayed on the display screen, the state shown in FIG. 18(b) is obtained. As shown in FIG. 18(b), the entire image of the building 87 does not fall within the range of the image data 80, and the upper part is not visible, similar to the building 85. In this case, it can be assumed that the line segment 87a of "Corner" and the line segment 87c of "Corner" are connected via another invisible line segment of "Top". Below, we will explain the processing for the "upper edge" in step Sc1, assuming that the endpoint to be processed is endpoint 85a-2 or endpoint 87a-2.

[0083] When the building area detection unit 57 determines that the position of the end point to be processed is the upper edge of the image data 80 (step Sc1, upper edge), it uses the position of the end point to be processed as a reference position and determines whether there is another end point located to the right of the reference position, which is the upper edge of the image data 80, i.e., the other end point with the maximum coordinate value on the Y axis, and whose line segment type is "Corner" or "Top" (step Sc2). When the building area detection unit 57 determines that such another end point does not exist (step Sc2, No), in this case, there is no other line segment connected to the start line segment or the line segment to be connected, and the area of ​​the building cannot be specified. Therefore, the line segment selected as the connection target in step Sb6 or step Sb9 up to that point is released from the connection target, and the process proceeds to step Sb13 in FIG. 12, as indicated by the connection mark C.

[0084] When the endpoint to be processed is endpoint 85a-2 shown in Fig. 18(a), the other endpoints are endpoint 85c-1 and endpoint 86a-1. When the endpoint to be processed is endpoint 87a-2 shown in Fig. 18(b), the other endpoint is endpoint 87c-1. Endpoint 85b-2 and endpoint 87b-2 are not other endpoints in step Sc2 because their line segment type is "Center."

[0085] Therefore, when the end point to be processed is end point 85a-2 or end point 87a-2, the building area detection unit 57 determines that the other end point as described above exists (step Sc2, Yes), and selects the other end point that has the smallest distance between the reference position, which is the position of the end point to be processed, and the other end point (step Sc3). In the case of end point 87a-2, the other end point is only end point 87c-1, so the building area detection unit 57 selects end point 87c-1. In contrast, in the case of end point 85a-2, end points 85c-1 and end point 86a-1 exist as other end points. Therefore, the building area detection unit 57 selects end point 85c-1, which has the smallest distance.

[0086] The building area detection unit 57 adds a line segment connecting the reference position, which is the position of the end point to be processed, and the selected end point to the line segment to be connected, and selects the line segment of the selected end point as the line segment to be connected (step Sc4). In the case of FIG. 18(a), the building area detection unit 57 adds a line segment 85-add, which is a line segment connecting the end point 85a-2 and the end point 85c-1, to the line segment to be connected, and selects the line segment 85c as the line segment to be connected. In the case of FIG. 18(b), the building area detection unit 57 adds a line segment 87-add, which is a line segment connecting the end point 87a-2 and the end point 87c-1, to the line segment to be connected, and selects the line segment 87c as the line segment to be connected. Thereafter, as indicated by the connection mark D, the process of step Sb11 in FIG. 12 is performed.

[0087] Next, a case where the building area detection unit 57 determines in step Sc1 that the position of the end point to be processed is located on either the left or right edge of the image data 80, or on the lower edge, will be described. For example, it is assumed that the image data 80 includes an image of a building 88 not shown in FIG. 16(a). It is assumed that the line segment identification data of the building 88 is displayed on the display screen in the state shown in FIG. 19. As shown in FIG. 19, the entire image of the building 88 does not fall within the range of the image data 80, and a part of the right side is cut off. In this case, the area surrounded by the line segments 88a, 88b, and 88d can be regarded as the area of ​​the building 88 in the image data 80. Hereinafter, the process in the case of the "left or right edge" in step Sc1 will be described assuming that the end point to be processed is the end point 88d-2.

[0088] Because the position of the endpoint 88d-2 to be processed is on the right edge of the image data 80, the building area detection unit 57 determines that the position of the endpoint 88d-2 to be processed is on either the left or right edge (step Sc1, either the left or right edge). The building area detection unit 57 adds a line segment 88-add extending vertically from the position of the endpoint 88d-2 to the lower edge of the image data 80 to the line segment to be connected (step Sc5).

[0089] The processing of step Sc6 is the same as that of step Sb12, and in the case of Figure 19, the area surrounded by the line segment 88a of "Corner" extended vertically to the bottom edge of image data 80, line segments 88b and 88d of "Top", and line segment 88-add becomes the building area of ​​building 88.

[0090] A case where the building region detection unit 57 determines in step Sc1 that the position of the end point of the processing target is located on the lower edge of the image data 80 is a case where, for example, a line segment such as line segment 88e shown by a dashed line in FIG. 19 exists instead of line segment 88d. In this case, it is only necessary to perform the processing of step Sc5, which is the processing when the position of the end point of the processing target is on either the left or right edge, and the processing of step Sc6, excluding the processing of step Sc5. Therefore, when the building region detection unit 57 determines that the position of the end point of the processing target is on the lower edge of the image data 80 (step Sc1, lower edge), it performs the processing of step Sb12 and step Sc7, which is the same processing as step Sc6.

[0091] (Adding the empty areas on the left and right to the building area) FIG. 14 is a flowchart showing a process flow performed when the building area detection unit 57 judges "No" in the judgment process of step Sb13 in FIG. 12, that is, after the detection of the building area based on the line segment identification data corresponding to the image ID of the image data 80 is completed. In the case of the image data 80 shown in FIG. 16(a), buildings are not present in the areas at both the left and right ends, but depending on the angle of view at the time of shooting, it is assumed that there is image data in which the building area is cut off at the left and right edges. In the process of the flowchart shown in FIG. 12, a building in which the image data does not include a line segment of the type "Corner" at the left end of the building is not targeted for the detection of the building area. Therefore, if an image of a building is present in the left end area of ​​the image data and line segment identification data of the type "Corner" of the line segment at the left end of the building is not present, the building area detection unit 57 cannot detect the building area.

[0092] Furthermore, if the line detection unit 53 is unable to detect line segment 88d among line segments 88a, 88b, 88c, and 88d of the building 88 shown in FIG. 19, the building area detection unit 57 will be unable to detect the building area of ​​the building 88.

[0093] In preparation for the above-mentioned case where the building region cannot be detected, a process is performed in which the left and right empty regions are added to the building region in the regions close to the left and right edges. This makes it possible for the building region detection unit 57 to detect the building region that could not be detected in the process shown in Figures 12 and 13 by detecting an overlapping region with the segmentation data detected by the semantic segmentation unit 56 described later. The flow of the process will be described below with reference to the flowchart shown in Figure 14.

[0094] The building area detection unit 57 refers to the building area range data of the building area identification data stored in the internal memory area, and determines whether or not there is building area identification data with a boundary on the left edge of the image data 80 (step Sd1). If the building area detection unit 57 determines that there is building area identification data with a boundary on the left edge of the image data 80 (step Sd1, Yes), in this case, a building area with a boundary already exists on the left edge of the image data 80, and if the left empty area is added as a building area, the building area with a boundary on the left edge will indicate an incorrect building area. Therefore, the building area detection unit 57 does not add the left empty area as a building area, and proceeds to step Sd4.

[0095] On the other hand, when the building area detection unit 57 determines that there is no building area identification data whose boundary exists on the left edge of the image data 80 (step Sd1, No), it refers to the line segment identification data table 55 and detects the line segment of the "Corner" type that is the shortest distance to the left edge of the image data 80 (step Sd2). Here, the distance between the line segment and the left edge of the image data 80 is the length of the line that is the shortest when extended horizontally from any position on the left side of the image data 80 to any position on the line segment, since the image data 80 has a rectangular shape.

[0096] In the case of image data 80 shown in FIG. 16(a), the building area detection unit 57 detects a line segment 81a shown in FIG. 16(b) as a line segment of the type "Corner" that is the shortest distance from the left edge of the image data 80. The building area detection unit 57 extends the detected line segment 81a to the top and bottom edges of the image data 80, and determines the area between the extended line segment and the left edge of the image data 80 as the building area. The building area detection unit 57 selects the coordinate data of one of the vertices of the coordinate data of the XY coordinate system of the vertices of the building area, and arranges the coordinate data of the other vertices clockwise or counterclockwise from the selected vertex to generate building area range data. When the building area range data is displayed on the display screen, it is displayed as the building area 94 in FIG. 16(c). The building area detection unit 57 generates a building area ID for the building area 94, generates building area identification data including the image ID of the image data 80, the generated building area ID, and the generated building area range data, and writes and stores the generated building area identification data in an internal memory area (step Sd3).

[0097] The building area detection unit 57 refers to the building area range data of the building area identification data stored in the internal storage area, and judges whether or not there is building area identification data with a boundary on the right edge of the image data 80 (step Sd4). If the building area detection unit 57 judges that there is building area identification data with a boundary on the right edge of the image data 80 (step Sd4, Yes), in this case, a building area with a boundary already exists on the right edge of the image data 80, and if the right empty area is added as a building area, the building area with a boundary on the right edge will indicate an incorrect building area. Therefore, the building area detection unit 57 does not add the right empty area as a building area, and then, as indicated by the connection mark E, the process of the flowchart in FIG. 15 is performed.

[0098] On the other hand, if the building area detection unit 57 determines that there is no building area identification data whose boundary exists on the right edge of the image data 80 (step Sd4, No), it refers to the line segment identification data table 55 and detects a line segment of type "Corner" that is the shortest distance to the right edge of the image data 80 (step Sd5).

[0099] In the case of image data 80 shown in FIG. 16(a), the building area detection unit 57 detects line segment 83c shown in FIG. 16(b) as a line segment of the type "Corner" that is the shortest distance from the right edge of the image data 80. The building area detection unit 57 extends the detected line segment 83c to the top and bottom edges of the image data 80, and determines the area between the extended line segment and the right edge of the image data 80 as the building area. The building area detection unit 57 selects the coordinate data of one of the vertices of the coordinate data of the XY coordinate system of the vertices of the building area, and arranges the coordinate data of the other vertices clockwise or counterclockwise from the selected vertex to generate building area range data. When the building area range data is displayed on the display screen, it is displayed as the building area 95 in FIG. 16(c). The building area detection unit 57 generates a building area ID for the building area 95, generates building area identification data including the image ID of the image data 80, the generated building area ID, and the generated building area range data, and writes and stores the generated building area identification data in an internal memory area (step Sd6).Then, the processing is performed according to the flowchart of Figure 15, as indicated by the connection mark E.

[0100] In the flowchart shown in Fig. 14, the order of the left and right may be reversed. That is, the processes of steps Sd4, Sd5, and Sd6 may be performed first, followed by the processes of steps Sd1, Sd2, and Sd3, and if the determination in step Sd4 is "Yes," the process of step Sd1 may be performed, and if the determination in step Sd1 is "Yes," the process indicated by the connection mark E, i.e., the process of the flowchart shown in Fig. 15, may be performed.

[0101] 14, in steps Sd2 and Sd5, the building area detection unit 57 refers to the line segment identification data table 55 and detects a line segment of the type "Corner" that is the shortest distance from the left or right edge of the image data 80. On the other hand, the processing of steps Sd3 and Sd5 may be performed only when there is a line segment of the type "Corner" that is not included in the boundary of the building area indicated by the building area range data of the building area identification data, in other words, only when there is an image of a building that is interrupted by the left and right edges of the image data 80 and there is a line segment of the type "Corner" that is not included in the boundary of the building area detected by the building area detection unit 57.

[0102] (Process to determine the order of placement and detect overlapping areas with segmentation data) For example, assume that the building region detection unit 57 performs a process of arranging building regions 91, 92, and 93 corresponding to buildings 81, 82, and 83, respectively, with the previously detected building region behind them, based on the building region range data of the building region identification data generated from the image data 80 shown in Fig. 16(a). In this case, since the line segment of "Corner" is extended to the bottom edge of the image data 80, a part of the building region 91 of the building 81 is hidden by the building region 92 of the building 82, as shown in Fig. 20. To avoid such a situation, the building region detection unit 57 performs a process of determining the order of arrangement so that the region of the shorter building is in the foreground.

[0103] The building area detection unit 57 detects the largest Y coordinate value from each of the building area range data of the building area identification data stored in the internal storage area. The building area detection unit 57 sets the detected value as the height of the building area indicated by the building area range data from which it was detected (step Se1). The building area detection unit 57 determines the order of arrangement of the building area identification data so that the taller building area is at the back. The building area detection unit 57 adds each of the data indicating the order of arrangement determined for each of the building area identification data to the corresponding building area identification data (step Se2).

[0104] In the case of image data 80, when the building area range data contained in the building area identification data to which data indicating the order of arrangement has been added is displayed on the display screen in the order of arrangement, the building areas 91 to 95 shown in Figure 16(c) will be displayed.

[0105] The building area detection unit 57 reads out from an internal storage area segmentation data associated with the same image ID as the image ID of the building area identification data to which the data indicating the order of arrangement has been added. The building area detection unit 57 detects overlapping areas between the read out segmentation data and each of the building area range data of the building area identification data to which the data indicating the order of arrangement has been added. The building area detection unit 57 regards the detected overlapping areas as new building area range data, and rewrites the building area range data of the building area identification data to the new building area range data. The building area detection unit 57 writes and stores the building area identification data rewritten as the new building area range data in the building area identification data table 59 of the building area identification data storage unit 58 (step Se3), and ends the process.

[0106] When the building area range data included in the building area identification data that has been rewritten with the new building area range data is displayed on the display screen, the building areas 91A, 92A, and 93A shown in FIG. 16(e) are displayed.

[0107] In the above embodiment, the building area detection unit 57 selects as the start line segment the line segment located at the leftmost position in the XY coordinate system among the unprocessed line segments of the type "Corner" as shown in the processing of step Sb1 in Fig. 12, but may select as the start line segment the line segment located at the rightmost position among the unprocessed line segments of the type "Corner". In this case, however, in the processing of step Sc2, the building area detection unit 57 needs to determine whether there exists another end point located to the left of the reference position, which is on the upper edge of the image data 80, i.e., which has the maximum coordinate value on the Y axis, and whose line segment type is "Corner" or "Top".

[0108] Also, the building region detection unit 57 may set any line segment of the "Corner" type that is an unprocessed line segment as the start line segment in the process of step Sb1 in Fig. 12. However, in this case, if the determination process of step Sb3 and step Sb10 is "Yes" and further the determination of "upper edge" is made in step Sc1, the direction to search cannot be specified in the determination process of step Sc2. Therefore, if the determination of "upper edge" is made in step Sc1, the process shown in Figs. 12 and 13 is performed again only on the start line segment to be processed and the line segment selected as the connection target to the start line segment.

[0109] Also, if the image data includes an image of a building in the shape of a cone or pyramid, there will be no line segment of the "Top" type. When detecting the area of ​​a building with such a shape, the process shown in FIG. 12 needs to be changed to a process that also takes into account the order of "Corner", "Corner". In this case, for example, in step Sb5 of FIG. 12, the line segment of "Corner" is also considered as a judgment target in addition to the line segment of "Top", and when the building area detection unit 57 selects the line segment of "Corner" as a line segment to be connected in step Sb6, a process needs to be performed to proceed to the process of step Sb12.

[0110] In the above embodiment, the building area detection unit 57 does not include the line segment of the type "Center" in the building area identification data. However, for example, the line segment identification data of the type "Center" may be added to the building area identification data as follows. For example, an item for writing the line segment data of "Center" is added to the building area identification data table 59 of the building area identification data storage unit 58. In the process of step Se3 in FIG. 15, the building area detection unit 57 writes the building area identification data to the building area identification data table 59, and then selects the building area range data written in the "building area range" item one by one, and detects the line segment of the type "Center" that is associated with the image ID corresponding to the selected building area range data and whose position indicated by the line segment identification data exists within the range indicated by the building area range data. The building area detection unit 57 writes and stores the line segment identification data corresponding to the detected line segment in the item for writing the line segment data of "Center" in the building area identification data table 59 corresponding to the selected building area range data. As a result, when the building area specifying data is displayed on the display screen, the line segment of the "Center" can also be displayed, making it possible to show the exterior shape of the building more accurately.

[0111] In the above embodiment, as shown in FIG. 16(b), the line segment detection unit 53 detects two line segments 81e and 83a of the type "Corner" between the building 81 and the building 83. In contrast, if the building 81 and the building 83 are almost the same height, it is assumed that the line segment detection unit 53 may erroneously detect them as one line segment of "Center" or one line segment of "Corner". If the line segment detection unit 53 detects them as one line segment of "Center", the building area detection unit 57 will detect the areas of the building 81 and the building 83 as the area of ​​one building. However, in this case, the area of ​​the building 81 and the area of ​​the building 83 can be displayed separately by adding information on the line segment of the type "Center" to the building area identification data as described above.

[0112] On the other hand, if the line segment detection unit 53 detects the line segment as one "Corner", the process shown in FIG. 12 cannot detect the building area of ​​the building 83, and even if the process of adding the empty areas on the left and right of FIG. 14 as building areas is performed, the building area of ​​the building 83 cannot be detected. In order to avoid this, for example, the definition of the unprocessed "Corner" line segment in the process of steps Sb1 and Sb13 shown in FIG. 12 is changed to a definition of a "Corner" type line segment that is not selected as a start line segment. This increases the number of processes, but all the "Corner" type line segments are selected as start line segments by the building area detection unit 57 at least once. Therefore, even if the line segment detection unit 53 erroneously detects the line segment between the building 81 and the building 83 as one "Corner" line segment, the building area detection unit 57 can detect the building area of ​​the building 83 by leaving the definition of the unprocessed line segment in steps Sb5 and Sb8 unchanged.

[0113] In this way, when the line segment of the "Corner" is selected as the starting line segment, it may be difficult to identify the line segment of the "Top" that connects to the selected "Corner". For example, as in the case of the building 22 and the building 23A shown in FIG. 4(b), the heights of two buildings that share the line segment of the selected "Corner" as a boundary are different. Specifically, when the area between the building 22 and the building 23A is detected as one "Corner", for example, the longer line segment 25e, it may be difficult to detect the line segment 26Ab of the "Top" of the building 23A behind the line segment 25e by a neighborhood search. As a countermeasure, the vertical area of ​​the search area is set to be wider than the horizontal area. For example, the building area detection unit 57 sets the total length of the line segment of the "Corner" + α as the vertical search area. α may be determined arbitrarily. For example, α is 1 / 4 the length of the total length of the line segment of the "Corner". In this case, the building area detection unit 57 searches from the upper vertex +α of the line segment of "Corner" to the lower vertex. On the other hand, the building area detection unit 57 does not set the search area too wide in the horizontal direction. For example, the search area is set to about 2.5[%] of the horizontal length of the rectangular area corresponding to the line segment of "Corner".

[0114] In addition, there may be cases where the boundaries of three or more buildings are detected as one "Corner." In this case, there is a possibility that multiple Tops are connected to that one "Corner." In this embodiment, such cases are allowed and the area of ​​each building is specified.

[0115] Also, in the flowchart shown in FIG. 13, an exception process was described for the case where the area corresponding to the top of the building is in contact with the edge of the image data. This can also be applied to the case where the areas corresponding to the left and right edges of the building are in contact with the edge of the image data. Specifically, both the left and right ends of the image are defined as "corners." This makes it possible to detect the area of ​​the building when the left and right edges of the building are not within the range of the image data.

[0116] On the other hand, if the image is not cut off and the left and right edges of the building are within the range of the image data, an extra building area will be detected. As a countermeasure against this extra detection, only areas that overlap with the building area detected by semantic segmentation in the subsequent stage are extracted. This makes it possible to finally exclude the extra detected areas.

[0117] In the above embodiment, the function approximator 12 included in the learning device 1 and the line segment detection unit 53 includes three neural networks 15-1, 15-2, and 15-3 as shown in Fig. 7. Alternatively, the function approximator 12 may include one deep neural network having an input layer that takes in input image data, three output layers 17a to 17c corresponding to the above-mentioned line segment type data, and eight output layers 18a to 18h corresponding to rectangle position data, rectangle size data, and line segment position data.

[0118] The learning device 1 and the function approximator 12 included in the line segment detection unit 53 extract features related to the lines of the building in the image data in the upstream neural network 15-1, and input the extracted features to the neural network 15-2 that solves the classification problem (problem of determining the type of line segment) and the neural network 15-3 that solves the regression problem (calculation of the line segment shape).This makes it possible to output the class of the detected number of line segments (either "Corner", "Top", or "Center") and the estimated shape information of the line segments as the final output result.

[0119] In the above embodiment, the configuration of the EfficientDet deep neural network is applied as the configuration of the three neural networks 15-1, 15-2, and 15-3 included in the function approximator 12 included in the learning device 1 and the line segment detection unit 53, but other deep neural network configurations such as FasterRCNN (Region-based convolutional neural network) and RetinaNet may be applied. Furthermore, the function approximator 12 may be realized by other machine learning means than a deep neural network, or may be configured by a function represented by a mathematical expression including parameters.

[0120] In addition, in the above embodiment, the input image data included in the training data of the learning device 1 and the image data captured by the imaging device 61 are color image data having three channels of RGB, but other formats of image data such as four-channel color image data of CMYK, grayscale image data, and binary image data may be applied as long as the image data is capable of detecting lines indicating the edges of a building.

[0121] The building region detection device 50 according to the above embodiment includes a line segment detection unit 53 and a building region detection unit 57. The line segment detection unit 53 detects information on the detection target line segment included in the image data 80 using the trained learning model data, and generates line segment identification data for each of the detected detection target line segments. The trained learning model data is a function approximator 12 to which the trained learning model data generated by the learning device 1 is applied. The line segment detection unit 53 detects information on the detection target line segment by providing any image data 80 to the function approximator 12. The information on the detection target line segment is rectangle position data, rectangle size data, line segment position data, and line segment type data. The identification data is data that combines rectangle position data, rectangle size data, line segment position data, and line segment type data for each of the detected detection target line segments. The building region detection unit 57 detects the building region for each of the individual buildings included in the image data 80 based on the line segment identification data generated by the line segment detection unit 53. Based on the line segment identification data, the building area detection unit 57 selects line segments that are connected and form the boundary of the building area. The building area detection unit 57 detects the building area by connecting the selected line segments. This makes it possible to detect with high accuracy line segments that indicate the outline of each building from line segments that indicate edges that can be detected by edge detection from the image data 80, and to detect the area of ​​each building using the detected line segments.

[0122] (Matching with map data) Next, the processing by the matching device 70 will be described with reference to Fig. 21 and Fig. 22. Fig. 21 is a flowchart showing the flow of processing by the matching device 70. For example, the matching processing unit 71 of the matching device 70 receives an operation by a user and imports any one of the image IDs specified by the user. The matching processing unit 71 detects data indicating the time data and the angle of view associated with the specified image ID from the history information storage unit 74. The matching processing unit 71 reads out the detected time data and data indicating the angle of view from the history information storage unit 74 (step Sf1).

[0123] The matching processing unit 71 detects from the history information storage unit 74 coordinate data associated with the time indicated by the read time data, i.e., time data indicating the time coinciding with the time when the image of the image data corresponding to the specified image ID was captured by the imaging device 61, and sets the detected coordinate data as the reference position coordinate data. The matching processing unit 71 reads out the reference position coordinate data from the history information storage unit 74, and also reads out from the history information storage unit 74 predetermined n pieces of coordinate data that are consecutive in chronological order going back to the past using the reference position coordinate data as a reference. Here, n is an integer value of 2 or more.

[0124] The matching processing unit 71 calculates the position of the imaging device 61 when the image data corresponding to the specified image ID was captured, based on the position indicated by the read reference position coordinate data and the positional relationship between the position detection device 62 in the moving body 60 and the imaging device 61 disposed in the moving body 60. The matching processing unit 71 calculates the traveling direction of the moving body 60 by the following formula (1), based on the read reference position coordinate data and the n pieces of coordinate data (step Sf2).

[0125]

number

[0126] In the formula (1), d is the direction of travel, and s0 is the position indicated by the reference position coordinate data. i The subscript i in the above expression is a number obtained by assigning numbers 1 to n to n pieces of coordinate data in the order of time from the time of the reference position coordinate data. From the viewpoint of the measurement steps of the position detection device 62, i is the coordinate data detected i steps before with respect to the reference position coordinate data. In addition, in formula (1), w(n,i)=n-i+1.

[0127] Based on the data d indicating the direction of travel of the moving body 60, the data indicating the calculated position of the imaging device 61, and the data indicating the angle of view of the imaging device 61 that has been read out, the matching processing unit 71 generates map image data on the projection surface of the imaging device 61 from the 2.5-dimensional map data stored in the map data storage unit 72, in accordance with the anteroposterior relationship of buildings included in the 2.5-dimensional map data (step Sf3).

[0128] Here, a method of generating map image data on the projection plane of the imaging device 61 from the 2.5-dimensional map data according to the front-to-back relationship of buildings included in the 2.5-dimensional map data with reference to FIG. 22 will be described. In FIG. 22, the position indicated by the reference numeral 120 is the position of the imaging device 61, and is hereinafter referred to as the position 120 of the imaging device 61. It is assumed that the imaging device 61 is installed horizontally. The angle formed by the line indicated by the reference numeral 122-1 and the line indicated by the reference numeral 122-2 is the angle of view of the imaging device 61, and the position indicated by the reference numeral 123 is the position of the projection plane of the imaging device 61, and is hereinafter referred to as the projection plane 123. The direction indicated by the arrow of the reference numeral 121 is the traveling direction of the moving body 60, and is hereinafter referred to as the traveling direction 121. The shapes indicated by the reference numerals 131, 132, 133, and 134 are the contours of the buildings included in the 2.5-dimensional map data on the XY plane, and are hereinafter referred to as the buildings 131, 132, 133, and 134.

[0129] For example, assuming that only building 131 exists, if a line of sight to building 131 is drawn with position 120 of imaging device 61 as the center, the line of sight that forms the maximum angle with the line of traveling direction 121 and the line of sight that forms the minimum angle with the line of traveling direction 121 are respectively drawn as dotted lines 131-1 and dotted lines 131-2. Similarly, assuming that only each of buildings 132, 133, and 134 exists, if a line of sight that forms the maximum angle with the line of traveling direction 121 and the line of sight that forms the minimum angle with the line of traveling direction 121 are respectively drawn, the line of sight for building 133 is shown by two dotted lines, and the line of sight for each of buildings 132 and 134 is shown by two alternate long and short dashed lines.

[0130] The 2.5D images between the two lines of sight of each building are projected on a straight line extending to the left and right of the projection surface 123 in the order of the buildings whose center positions are closest to the position 120 of the imaging device 61, that is, building 132, building 134, building 131, and building 133. By cutting out only the portion of the projection surface 123 from the projected images, an image arranged as shown in the balloon diagram in Fig. 22 is obtained, and the data of this image becomes the map image data described above.

[0131] In the balloon diagram of FIG. 22, the projected image of the building 132 is image 142 indicated by a dashed line, the projected image of the building 134 is image 144 indicated by a dashed line, the projected image of the building 131 is image 141 indicated by a dotted line, and the projected image of the building 133 is image 143 indicated by a dotted line. The image 141 of the building 131 is partially blocked by the image 142 of the building 132, and the image 143 of the building 133 is partially blocked by the image 144 of the building 134. Therefore, when the projection plane is referred to from the position 120 of the imaging device 61, the images of the buildings 131 to 134 in a state where the anteroposterior relationship is correct can be seen. This makes it possible to create map image data in a state where the anteroposterior relationship of the buildings 131 to 134 is correct from the 2.5-dimensional map data. Moreover, the generated map image data is an image on the projection plane 123 of the imaging device 61, and the scale of the image is the same as the scale of the image included in the image data captured by the imaging device 61.

[0132] The matching processing unit 71 reads out all building area identification data corresponding to the specified image ID from the building area identification data table 59 in the building area identification data storage unit 58 (step Sf4). The matching processing unit 71 performs a matching process to overlay the building area range data included in the read building area identification data on the generated map image data so that the coordinate positions match (step Sf5). When the image data in which the building area range data is overlaid on the map image data is displayed on the display screen, the images of the areas of the individual buildings are displayed overlaid on the images of the buildings shown in the map image data.

[0133] The matching system 200 according to the above embodiment includes a building area detection device 50, an imaging device 61, a position detection device 62, and a matching device 70. The matching device 70 includes a map data storage unit 72 and a matching processing unit 71. The map data storage unit 72 stores map data. The map data includes position data indicating the positions of the buildings 131 to 134, shape data indicating the shapes of the buildings 131 to 134 on the ground plane, and height data indicating the heights of the buildings 131 to 134. The matching processing unit 71 performs matching processing based on data indicating the position of the moving object 60 detected by the position detection device 62. In the matching processing, the matching processing unit 71 calculates the traveling direction of the moving object 60 when the imaging device 61 captures an image in order to generate image data to be processed by the building area detection device 50. The matching processing unit 71 generates map image data to be projected on the projection surface of the imaging device 61 from the map data stored in the map data storage unit 72. The matching processing unit 71 generates map image data based on data indicating the travel direction of the moving object 60, data indicating the position of the moving object 60 when the image was captured, and data indicating the angle of view of the imaging device 61 when the image was captured, so that the front-to-back relationship of the buildings 131 to 134 is correct. The matching processing unit 71 performs matching processing by superimposing the generated map image data and the building areas of the buildings 131 to 134 detected from the image data by the building area detection device 50. As a result, for example, map image data at the position of the projection plane of the imaging device 61 can be generated from 2.5-dimensional map data with the front-to-back relationship of the buildings 131 to 134 correct, and the buildings 131 to 134 detected by the building area detection device 50 can be superimposed and displayed on the generated map image data. This makes it possible to improve the accuracy of the position of the moving object 60 obtained by the position detection device 62, and to display advertisements at accurate positions when displaying advertisements on the images of the buildings 131 to 134 using AR technology.

[0134] The learning device 1, the building region detection device 50, and the matching device 70 in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0135] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0136] 1...Learning device, 11...Input unit, 12...Function approximator, 13...Learning processing unit, 14...Learning model data storage unit, 15-1, 15-2, 15-3...Neural network, 16...Parameter acquisition unit, 200...Matching system, 50...Building area detection device, 51...Receiving unit, 52...Learned learning model data storage unit, 53...Line detection unit, 54...Line segment identification data storage unit, 55...Line segment identification data table, 56...Semantic segmentation unit, 57...Building area detection unit, 58...Building area identification data storage unit, 59...Building area identification data table, 60...Mobile body, 61...Imaging device, 62...Position detection device, 63...Communication device, 70...Matching device, 71...Matching processing unit, 72...Map data storage unit, 73...Receiving unit, 74...History information storage unit

Claims

1. an input unit which takes in input image data which is data of an image of a building, a first type of line segment which constitutes the boundary of the building area selected from line segments which indicate the outline of each building included in the input image data and has one end which belongs to a lower part of the building which is appropriately determined for each building, and a second type of line segment which constitutes the boundary of the building area and has both ends which do not belong to the lower part, rectangle position data which indicates the position in the input image data of each of rectangular areas which are appropriately determined to include one of the detection target line segments, rectangle size data which indicates the size of each of the rectangular areas, line segment position data which indicates the positions in the rectangular area of ​​both end points of the line segments included in each of the rectangular areas, and line segment type data which indicates whether the detection target line segment is the first type or the second type; a learning processing unit that performs a learning process to update learning model data applied to the function approximator so as to approach a state in which, when the input image data captured by the input unit is provided to a function approximator, the function approximator outputs the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data captured by the input unit; A learning device comprising:

2. adding, as a third type of line segment, to the detection target line segments, a line segment that shows the outline of a building included in the input image data, other than a line segment that constitutes a boundary of the area of ​​the building, and that is located at a corner of the building when the building is viewed from above; The input unit includes: taking in the rectangle position data, the rectangle size data, the line segment position data, line segment type data indicating whether the detection target line segment is of the first type, the second type, or the third type, and the input image data; The learning processing unit: performing a learning process for updating learning model data to be applied to the function approximator so as to approach a state in which, when the input image data captured by the input unit is provided to a function approximator, the function approximator outputs the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data indicating whether the data is of the first type, the second type, or the third type captured by the input unit; The learning device according to claim 1 .

3. a line segment detection unit that detects the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data for a detection target line segment among lines included in the image data by providing arbitrary image data to the function approximator to which trained learning model data generated by the learning device according to claim 1 or 2 is applied, and generates line segment identification data that combines the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data for each of the detected detection target line segments; a building area detection unit that detects the area of ​​each building included in the image data by selecting line segments that are in a connecting relationship and that form the boundary of the building area based on the line segment identification data generated by the line segment detection unit, and connecting the selected line segments; A building area detection device comprising:

4. A semantic segmentation unit that detects a building area from the image data by semantic segmentation, The building area detection device includes: detecting an overlapping area between the building area for each of the detected buildings and the building area detected by the semantic segmentation unit, and determining the detected overlapping area as the building area for each of the buildings; The building area detection device according to claim 3 .

5. The top and bottom of the image data are predetermined, The building area detection unit is selecting, as a start line segment, any one of the line segments corresponding to the line segment identification data detected by the line segment detection unit, the line segment type data of which is the first type; selecting, based on the line segment identification data corresponding to the selected start line segment, as a line segment to be connected, a line segment having one end point within a vicinity range of an upper end point of the selected start line segment and having the line segment type data of the second type; and selecting, as a line segment to be connected, a line segment having one end point within a vicinity range of the other end point of the selected line segment to be connected and having the line segment type data of the first type or the second type; repeating this process until the line segment type data of the line segment to be connected becomes the first type; connecting the selected line segments to be connected to the start line segment in the selected order. The building area detection device according to claim 3 or 4.

6. The building area detection unit is if the position of the upper end point of the start line segment or the position of the other end point of the line segment to be connected is the position of the upper edge of the image data, that position is set as a reference position, and if there is another line segment having one end point at a position in a predetermined direction to the left or right from the reference position and at the position of the upper edge of the image data, and the line segment type data is the first type or the second type, then from among the other line segments, the other line segment having an end point at the upper edge of the image data that is the smallest distance from the reference position is selected, and the section between the reference position and the position of the end point of the other line segment at the upper edge of the image data is added to the line segments to be connected, and the other line segment is selected as the line segment to be connected; The building area detection device according to claim 5 .

7. The image data is rectangular in shape and has a top and bottom that are predetermined, The building area detection unit is determining the length of the longest line among lines extending vertically from any point located on the boundary line of the detected building area toward the lower edge of the image data as the height of the building area, and determining the order of arrangement of the detected building areas so that the taller building area is located at the back; The building region detection device according to any one of claims 3 to 6.

8. The image data is rectangular in shape and has left and right sides predetermined, The building area detection unit is detects, as a building area, an area between a straight line obtained by extending a line segment whose line segment type data is the first type and whose distance to the left edge of the image data is the shortest, and the left edge of the image data, and detects, as a building area, an area between a straight line obtained by extending a line segment whose line segment type data is the first type and whose distance to the right edge of the image data is the shortest, and the right edge of the image data. The building region detection device according to any one of claims 3 to 7.

9. The building region detection device according to any one of claims 3 to 8, an imaging device mounted on a moving body and moving, the imaging device generating image data to be processed by the building area detection device; a position detection device for detecting the position of the moving object; a matching device that performs matching processing based on the area of ​​the building detected by the building area detection device, The matching device includes: a map data storage unit that stores map data including position data indicating a position of a building, shape data indicating a shape of the building on a ground plane, and height data indicating a height of the building; a matching processing unit that calculates a traveling direction of the moving body when the imaging device captured the image in order to generate image data to be processed by the building area detection device, based on data indicating the position of the moving body detected by the position detection device, generates map image data to be projected on a projection surface of the imaging device from the map data stored in the map data storage unit so that the anteroposterior relationship of the building is correct, based on the data indicating the calculated traveling direction of the moving body, data indicating the position of the moving body when the imaging was performed, and data indicating the angle of view of the imaging device when the imaging was performed, and performs a matching processing of superimposing the generated map image data on the area of ​​the building detected from the image data by the building area detection device. A matching system comprising:

10. an input process for inputting input image data which is data of an image of a building, a first type of line segment which constitutes the boundary of the building area selected from line segments which indicate the outline of each building included in the input image data and has one end which belongs to a lower part of the building which is appropriately determined for each building, and a second type of line segment which constitutes the boundary of the building area and has both ends which do not belong to the lower part, and inputting rectangle position data which indicates the position in the input image data of each of rectangular areas which are appropriately determined to include one of the detection target line segments, rectangle size data which indicates the size of each of the rectangular areas, line segment position data which indicates the positions in the rectangular area of ​​both end points of the line segments included in each of the rectangular areas, and line segment type data which indicates whether the detection target line segment is the first type or the second type; a learning process step of performing a learning process of updating learning model data applied to the function approximator so as to approach a state in which, when the input image data taken in the input process is provided to a function approximator, the function approximator outputs the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data taken in the input process; Learning methods including:

11. On the computer, an input step of taking in input image data which is data of an image of a building, a first type of line segment which constitutes a boundary of the building area selected from line segments which indicate the outline of each building included in the input image data and has one end which belongs to a lower part of the building which is appropriately determined for each building, and a second type of line segment which constitutes the boundary of the building area and has both ends which do not belong to the lower part, rectangular position data which indicates a position in the input image data of each of rectangular areas which are appropriately determined to include one of the detection target line segments, rectangular size data which indicates the size of each of the rectangular areas, line segment position data which indicates the positions in the rectangular area of ​​both end points of the line segments included in each of the rectangular areas, and line segment type data which indicates whether the detection target line segment is the first type or the second type; a learning processing procedure for performing a learning process for updating learning model data to be applied to the function approximator so as to approach a state in which, when the input image data taken in the input procedure is provided to a function approximator, the function approximator outputs the rectangle position data, the rectangle size data, the line segment position data, and the line segment type data taken in the input procedure; A learning program to help you implement the above.

Images