Information processing system and method for quantifying landscape

Abstract

To provide an information processing system and a landscape evaluation method which can quantify a landscape of a town on the basis of a landscape image thereof.SOLUTION: An information processing system comprises: a ratio calculation unit which calculates a ratio of an area occupied by an area of an image of a subject in a landscape image that shows a scenery in the extension direction of a road where an imaging point is located for each type of subject; a feature amount extraction unit which extracts a feature amount of the road; and a landscape quantification processing unit which decides an index value of the landscape of a town where the landscape image is captured by using the ratio calculated for each type and the feature amount of the road.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing system and a landscape quantification method.

Background Art

[0002] The landscape of a town is affected by the surrounding buildings and road conditions. A method of quantifying the landscape of a town using landscape images taken on the road is known (Non-Patent Document 1). When the usage form of the road is different, the characteristics of the road may also be different.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, according to the above non-patent document, the analysis of quantifying the landscape of a town based on randomly extracted landscape images may be difficult depending on the conditions. It may be difficult to represent the characteristics of the road using the characteristics of the road part in the landscape image.

[0006] This invention has been made in view of the above circumstances, and aims to provide an information processing system and a landscape evaluation method that enable the quantification of urban landscapes based on landscape images. [Means for solving the problem]

[0007] (1) In order to solve the above problems, one aspect of the present invention is an information processing system comprising: a ratio calculation unit that calculates the ratio of the area occupied by the image region of a subject in a landscape image showing the scenery in the direction of extension of the road where the shooting point is located, for each type of subject; a feature extraction unit that extracts the feature quantities of the road; and a landscape quantification processing unit that determines an index value of the city landscape in which the landscape image was taken, using the ratio calculated for each type and the feature quantities of the road. (2) In the above-described information processing system, the road feature quantity includes at least one of the straightness of the road, the continuity of the road features, and the width of the road. (3) In the above information processing system, the feature extraction unit determines the position of the vanishing point from the foreground road feature quantity, which is the feature quantity of the foreground road portion in the landscape image, and extracts the road feature quantity based on the positional relationship between the foreground road feature quantity and the vanishing point. (4) In the above-described information processing system, the feature extraction unit determines an evaluation target area in the landscape image based on the position of the vanishing point, and assigns a label to landscape images in which the range classified as a moving object exceeds a predetermined value, based on the classification result of the images captured within the determined evaluation target area, to exclude from analysis in subsequent processing. (5) One aspect of the present invention is a landscape quantification method that includes the steps of: calculating the ratio of the area occupied by the image region of a subject in a landscape image showing the scenery in the direction of extension of the road where the shooting point is located, for each type of subject; extracting the feature quantities of the road; and determining an index value of the city landscape in which the landscape image was taken, using the ratio calculated for each type and the feature quantities of the road. [Effects of the Invention]

[0008] According to each aspect of the present invention, it becomes possible to quantify the urban landscape based on images of that landscape. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the configuration of the information processing system 1 according to the embodiment. [Figure 2] This is a diagram showing the configuration of the landscape quantification processing device 3 according to the embodiment. [Figure 3] This is a diagram illustrating the configuration of the data table in the embodiment. [Figure 4] This is a diagram illustrating an example of an image that will be excluded from the analysis of the embodiment. [Figure 5] This figure illustrates an example of an image of the area surrounding the location shown in Figure 4(a) above. [Figure 6] This is a diagram illustrating an example of an image of the area surrounding the station. [Figure 7] This figure illustrates an example of a street route in an area slightly away from the station shown in Figure 6. [Figure 8] This is a flowchart of the quantification process in the embodiment. [Figure 9] This is a diagram illustrating the extraction of road features in the embodiment. [Figure 10] This is a diagram illustrating "images unsuitable for subsequent processing" in the embodiment. [Figure 11] This figure illustrates how to define the range of a partial image in a landscape image according to the embodiment. [Figure 12] This figure illustrates the results of extracting the empty areas and planted areas from the landscape image of the embodiment. [Figure 13] This is a diagram illustrating the identification of the type of subject in a landscape image according to the embodiment. [Figure 14] This is a diagram illustrating the identification of the type of subject in a landscape image according to the embodiment. [Figure 15A] This figure illustrates the results of the landscape quantification process in the embodiment. [Figure 15B]It is a diagram for explaining the result of the landscape quantification processing of the embodiment. [Figure 16A] It is a diagram for explaining the setting of the range of the partial image in the landscape image of the modification example of the embodiment. [Figure 16B] It is a diagram for explaining the setting of the range of the partial image in the landscape image of the modification example of the embodiment. [Figure 17] It is a diagram for explaining the empty area in the landscape image and the tendency of the state of the town. [Figure 18] It is a flowchart of the learning process of the embodiment.

Mode for Carrying Out the Invention

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] (First Embodiment) FIG. 1 is a configuration diagram of an information processing system 1 according to the embodiment. The information processing system 1 includes, for example, a terminal device 2 and a landscape quantification processing device 3. The terminal device 2 and the landscape quantification processing device 3 communicate with each other via, for example, a network NW.

[0012] First, photographer U1 uses terminal device 2, etc., to photograph the area around the target point on the road using the camera of terminal device 2. Terminal device 2 can be either terminal device 2A or terminal device 2B. Terminal device 2A is mounted on a mobile device and has a camera capable of capturing images of the surrounding area of ​​the mobile device. Terminal device 2B is a camera or smartphone, etc., carried by photographer U1 and capable of capturing images of its surrounding area. The image data of the surrounding area generated by this terminal device 2 may be, for example, color image data of surrounding subjects. The images shown below are examples taken using standard to wide-angle lenses, but are not limited to these, and images taken with an appropriate angle of view may be used. For example, an image suitable for processing should include an image showing the view in the direction of extension of the road from the shooting point. Note that photographer U1 may be a specific person or multiple people. Terminal device 2 may be an in-vehicle camera device. In this embodiment, unless otherwise specified, the example described will be one in which terminal device 2A (vehicle-mounted camera device, etc.) is applied to terminal device 2.

[0013] Referring to Figure 2, the landscape quantification processing apparatus 3 according to the embodiment will be described. Figure 2 is a configuration diagram of the landscape quantification processing apparatus 3 according to the embodiment.

[0014] The landscape quantification processing device 3 comprises a control unit 310, a storage unit 320, a communication processing unit 330, a display unit 340, and a learning processing unit 350.

[0015] The communication processing unit 330 receives image data captured by the terminal device 2 by communicating with the terminal device 2. The display unit 340 is a display device that includes a liquid crystal display panel and displays various information such as images.

[0016] The control unit 310 includes, for example, an information acquisition unit 311, a landscape image extraction unit 312, a partial image range generation unit 313, a type identification processing unit 314, a ratio calculation unit 315, a landscape quantification processing unit 316, a regional value estimation unit 317, a display processing unit 318, and a feature quantity extraction unit 319.

[0017] The information acquisition unit 311 receives various data supplied from an external source and stores the received data in the storage unit 320 according to its type. The various data include landscape image data showing the surrounding conditions of the point to be quantified (shooting point), training image data, and setting values ​​for identification processing. For example, the information acquisition unit 311 acquires images obtained by shooting on the road with the camera of each terminal device 2 from each terminal device 2 and adds them to the acquired image data storage unit 321.

[0018] The landscape image extraction unit 312 acquires multiple images captured by the camera of the terminal device 2 from the acquired image data storage unit 321, and selects an image suitable for processing described later from among these multiple images based on predetermined selection criteria. The landscape image extraction unit 312 adds the result to the landscape image data storage unit 322. The image selected here is called a landscape image. The landscape image extraction unit 312 is an example of a road identification unit that distinguishes between pathways and roads within the site based on the feature quantities of the image in the landscape image, and identifies roads extending from the point where the landscape image was taken.

[0019] For example, images suitable for the processing described later include images showing the view in the direction of the road's extension from the shooting location. Images unsuitable for processing include images showing the view in the direction of the road's crossing from the shooting location (Figure 4(a)), and images in which vehicles near the shooting location are relatively large (Figure 4(b)). As predetermined selection criteria for these, it may be specified that images showing the view in the direction of the road's extension from the shooting location are included, but images showing the road's crossing from the shooting location are not, and images in which vehicles near the shooting location are relatively large are not included. As a result, the landscape image extraction unit 312 selects images that include views looking in the direction of the road's extension from the shooting point, but excludes images that include views looking in the direction of the road's crossing from the shooting point. The landscape image extraction unit 312 can also exclude images in which vehicles near the shooting point are relatively large. Based on this selection result, the landscape image extraction unit 312 may extract the selected images as landscape images.

[0020] The partial image range generation unit 313 determines the range of a partial image based on predetermined rules for setting the range of a partial image (Figure 11, etc.) stored in the partial image range data storage unit 323, in association with the vanishing point of the perspective projection display (hereinafter simply referred to as the "vanishing point") determined from the landscape image (landscape image data). For example, the predetermined rules may include defining the range of the partial image as the area within a polygon arranged in association with the position of the vanishing point in the landscape image, and predetermining the size of the polygon and the distance from the vanishing point to the outline of the polygon. The distance from the vanishing point to the outline of the polygon may be set to any distance greater than or equal to 0. In the following explanation, a quadrilateral will be used as an example of a polygon. The partial image range generation unit 313 adds the partial image range data to the landscape image data storage unit 322. The partial image range data may include information such as the upper and lower limits of the range of the partial image and the position in the left and right directions, as information for associating the range of the partial image with a position on the landscape image. Hereafter, an image obtained by extracting a portion of a landscape image using the partial image range data associated with the landscape image data will simply be referred to as a partial image. Furthermore, the partial image range generation unit 313 may extract multiple line segments related to the estimation of the vanishing point from the feature quantities of the image within the landscape image, and determine the vanishing point based on the extracted multiple line segments. For example, two straight lines with different directions on a two-dimensional plane intersect at one point. If each of the above line segments is extended, there is a region where the above intersection points are concentrated. A position representing these intersection points may be used as the vanishing point.

[0021] The type identification processing unit 314 identifies the type of subject shown in a landscape image or partial image that shows the scenery in the direction of extension of the road where the shooting location is located. For example, the type identification processing unit 314 obtains an optimized trained model from the training result storage unit 325 and uses it to identify the type of subject shown in a landscape image or partial image that shows the scenery in the direction of extension of the road where the shooting location is located. The type identification processing unit 314 adds the identification result to the landscape image data storage unit 322.

[0022] The ratio calculation unit 315 calculates the ratio of the area occupied by the subject's image region within the partial image for each type of subject. The ratio calculation unit 315 adds the results to the landscape image data storage unit 322.

[0023] The landscape quantification processing unit 316 uses the ratios calculated for each type of subject to determine an index value for the city landscape where the landscape image was taken. The landscape quantification processing unit 316 adds the result to the landscape image data storage unit 322.

[0024] The display processing unit 318 displays the images that each of the above units is processing, as well as various analysis results, on the display unit 340.

[0025] The feature extraction unit 319 calculates, for example, the straightness of roads in an image and the continuity of those features. The regional value estimation unit 317 estimates the value of the region being evaluated. Details of the feature extraction unit 319 and the regional value estimation unit 317 will be described later.

[0026] The learning processing unit 350 performs learning processing on the analysis models provided by each part of the control unit 310 in order to optimize the processing of each part. For example, the learning processing unit 350 performs learning processing on the type identification processing unit 314 in order to optimize the type identification processing performed by the type identification processing unit 314. The learning processing unit 350 also performs learning processing on the feature extraction unit 319 in order to optimize the feature extraction processing performed by the feature extraction unit 319.

[0027] For example, the learning processing unit 350 performs the learning process of the type identification processing unit 314 based on a supervised machine learning method using training image data acquired via the information acquisition unit 311. The learning processing unit 350 then modifies the type identification processing unit 314 to suit its type identification process and stores the learning results in the storage unit 320.

[0028] The learning processing unit 350 performs the learning process of the feature extraction unit 319 based on a supervised learning method using the learning image data acquired via the information acquisition unit 311. The learning processing unit 350 adjusts the characteristics of the feature extraction unit 319 to suit its feature extraction process and stores the learning results in the storage unit 320.

[0029] The memory unit 320 includes an acquired image data storage unit 321, a landscape image data storage unit 322, a partial image range data storage unit 323, a teacher image data storage unit 324, a learning result storage unit 325, and a set value data storage unit 326.

[0030] For example, the acquired image data storage unit 321 stores data of scenery images acquired from the terminal device 2 (such as the images in Figures 4 to 7 described later), associated with image identification information, image information, location information of the shooting location, information on the direction of shooting, and information on the date and time of shooting.

[0031] The landscape image data storage unit 322 stores landscape image data (such as the images in Figures 4 to 7 described later) selected from landscape image data, associated with image identification information. For example, the landscape image data includes, in addition to image identification information and image data, data for items such as partial image range, identification result image for the analysis unit (each pixel) of the process of identifying the type of subject, area ratio, and landscape index value.

[0032] The partial image range data storage unit 323 stores data that defines the range of a partial image, which corresponds to a specific area within the landscape image.

[0033] The training image data storage unit 324 stores training image data (training image data) used in the training process. The training result storage unit 325 stores data related to the type identification processing unit 314, which has been optimized through the training process, for example. The setting value data storage unit 326 stores setting values ​​such as variables used by each functional unit in its processing.

[0034] Figure 3 is a diagram showing the configuration of the data table of the landscape image data storage unit 322 of the embodiment. The data table of the landscape image data storage unit 322 shown in Figure 3 includes data for items such as image identification information (No.), landscape image data, partial image range data, identification result image data, and the ratio (area ratio) for each type of subject. As shown in the figure, the types of subjects include the sky, plants, buildings, structures, vehicles, etc. In the following explanation, everything except the sky and plants may be grouped together and referred to as "others."

[0035] Figures 4 to 7 illustrate images illustrating the processing of the embodiment.

[0036] Figure 4 is a diagram illustrating an example of an image that is excluded from the analysis of the embodiment. The image shown in Figure 4(a) is an explanatory image taken from a position overlooking a T-junction. There are roads extending straight to the left and right from the intersection. Strictly speaking, this image was not taken on a road that extends relatively long in the direction of the optical axis. This image is rather perpendicular to the direction of extension of the straight road. There is a building in the center of the image, which covers the sky.

[0037] Note that the space in front of the building located at the front of Figure 4(a) shows a portion of the parking lot on the site. The parking lot is an example of a pathway within the site. This space does not represent the road that includes the point where this image was taken extending beyond an intersection. Such images show the usage of the site and lack sufficient information to be used as an image representing the situation in the area.

[0038] The image shown in Figure 4(b) shows that the view ahead is obscured by a vehicle that crossed the path in front of the camera in the direction of the shot. When multiple images are taken, such images may be included.

[0039] Figure 5 is a diagram illustrating an example of an image of the area surrounding the location shown in Figure 4(a) above. The image in Figure 5(a) shows a scene with street trees lining both sides of a road that extends relatively straight into the distance. Most of the buildings facing this road are two to three stories tall. The image in Figure 5(b) shows a road that extends relatively straight into the distance, with street trees on both sides. School grounds face both sides of this road, but the school buildings are hidden by the street trees and cannot be seen. The image shown in Figure 5(c) is a view from a position overlooking a three-way intersection. Facing the intersection, there is a house directly in front of the camera, and the distant scenery can be seen over the roofs of the house. There are street trees on the left side of the road, and no buildings on the property can be seen. On the right side of the road, there is a row of apartment buildings and houses with relatively low-height landscaping.

[0040] Figure 6 is a diagram illustrating an example of an image of the area surrounding the station. The image shown in Figure 6(a) includes a view of a road lined with relatively tall buildings. This road is a relatively wide one-way street. There are sidewalks on both sides of the road, lined with street trees. The image in Figure 6(b) shows the view from a point closer to the station than the shooting location in Figure 6(a). Relatively tall buildings line this road. In front, there are also relatively tall buildings used as the station building. Because this road is narrow, only three lanes of roadway are available, and the sidewalk is also narrow. Therefore, there are no street trees. The image shown in Figure 6(c) includes a view from a road moving away from the aforementioned station. This road has undergone redevelopment, including widening. The area captured in the photograph includes a two-lane roadway in each direction and a central median. An entrance and exit to an underground parking lot are located in the central median.

[0041] Figure 7 is an example of an image of a street in an area slightly away from the station shown in Figure 6. The image shown in Figure 7(a) includes a view of a five-way intersection in the foreground. There are sidewalks on both sides of the road, lined with street trees. This area is a business district. The image in Figure 7(b) shows a view of a road that curves gently to the left. There are sidewalks on both sides of the road, lined with street trees. The surrounding area is a business district. The image shown in Figure 7(c) includes a view of a road curving to the right. There are sidewalks on both sides of the road, lined with plants. Most of the buildings facing the road are two to three stories tall. In the background of this image, power transmission towers and power lines are visible.

[0042] Next, we will explain the main processes performed by the control unit 310, such as "quantification processing" and "learning processing."

[0043] [Quantification process] The quantification process of the embodiment will be described with reference to Figure 8. Figure 8 is a flowchart of the quantification process of the embodiment.

[0044] The feature extraction unit 319 extracts features from each image randomly selected from among multiple images in the acquired image data storage unit 321 (step SA10). This step can be omitted if the images in the acquired image data storage unit 321 are images that can be treated as landscape images.

[0045] The landscape image extraction unit 312 selects a landscape image from among multiple images from which features have been extracted that shows the scenery in the direction of extension of the road where the shooting location is located, according to predetermined selection criteria (step SA12). The landscape image extraction unit 312 adds the selected landscape image to the landscape image data storage unit 322.

[0046] The type identification processing unit 314 identifies the type of subject within a predetermined range (e.g., a partial image) of the landscape image stored in the landscape image data storage unit 322 and divides the area (step SA14). The type identification processing unit 314 adds the result to the landscape image data storage unit 322. The predetermined range of the landscape image is determined based on the data stored in the partial image range data storage unit 323.

[0047] The ratio calculation unit 315 calculates the area ratio for each type of subject within a predetermined range (partial image) of a landscape image stored in the landscape image data storage unit 322, and performs a quantitative processing of the city landscape (step SA16). The results are added to the landscape image data storage unit 322.

[0048] The regional value estimation unit 317 performs landscape value estimation processing using the area ratio of the above-mentioned region stored in the landscape image data storage unit 322 (step SA18).

[0049] By following the above procedure, the urban landscape is quantified based on the resulting landscape images. Below, we will explain the processing of each step of the above procedure with more specific examples.

[0050] (Step SA10: Extraction of features from the image) Referring to Figure 9, the extraction of features within the image related to the processing in step SA10 in Figure 8 above will be explained. Figure 9 is a diagram illustrating the extraction of road features in the embodiment. The four images extracted from Figures 5 through 7 mentioned above are reproduced in Figure 9. Figure 9(a) shows the same image as Figure 5(a) mentioned above. Figure 9(b) shows the same image as Figure 6(b) mentioned above. Figure 9(c) shows the same image as Figure 5(c) mentioned above. Figure 9(d) shows the same image as Figure 6(c) mentioned above. In each of the above images, straight lines are added to indicate the parts from which road features can be extracted from images such as curbs and road markings.

[0051] From each image shown in Figure 9, the road features and vanishing points within each image are extracted. The process involved in this extraction will be explained in detail below.

[0052] (SA10-1: Extraction of road features in the image) This section explains how to extract road features from an image. When information indicating road characteristics is detected in the image, it is desirable to identify that the image contains a road. The road characteristics used for this identification should include, for example, at least one of the following: the straightness of the road, the continuity of the road's characteristics, and the width of the road.

[0053] Urban roads are equipped with sidewalks and roadways, curbs, median strips, and road markings (road lane markings, center lines, lane markings, pedestrian crossing markings, etc.). Even residential roads, as shown in Figure 9(c), are equipped with at least curbs. The boundaries or outlines of the curbs, median strips, and road markings mentioned above include portions that run along the direction of the road's extension. These portions that run along the direction of the road's extension contain information related to the straightness of the road and the continuity of its characteristics.

[0054] The feature extraction unit 319 calculates, for example, the straightness of the road and the continuity of the road's features using the above information, and uses the results as the road's features.

[0055] For example, the feature extraction unit 319 may estimate the direction of road extension from the boundary lines or contours of the curb, median strip, road markings, etc. The feature extraction unit 319 calculates the components of the boundary lines or contours based on those boundary lines or contours. The feature extraction unit 319 derives a regularity based on the position, length, direction, etc., of the calculated components, and derives a feature that indicates the direction of road extension based on this regularity. For example, the feature extraction unit 319 associates the position, length, and direction of the calculated components with the starting position, length, and direction of a vector, respectively. The feature extraction unit 319 may, for example, derive the relationship between the calculated components using vector operations. The same applies hereafter.

[0056] The feature extraction unit 319 may determine that the linearity is high if the range over which the components in the direction of road extension at each point, extracted based on the above-mentioned boundary line or contour, are aligned in the direction of road extension at each point is long. The feature extraction unit 319 may determine that the continuity of the road features is high if the range over which the components in the direction of road extension at each point are aligned is long.

[0057] The feature extraction unit 319 calculates, for example, the width of the road using the information such as the boundary line or outline and the number of lanes. The feature extraction unit 319 may use the calculated width as a feature representing the width of the road.

[0058] Furthermore, the feature extraction unit 319 does not need to accurately identify the location of roads in the image, nor does it need to identify the type of each subject (image) mentioned above. In this case, the feature extraction unit 319 may, for example, obtain the information used for the above processing in a simplified manner by extracting components that extend roughly toward the center of the image in the lower half of the image region. In other words, this can be replaced by extracting components that radiate from the center of the image downwards or diagonally downwards.

[0059] (SA10-2: Extraction of vanishing points in an image) The feature extraction unit 319 determines the position of the vanishing point using, for example, the foreground road feature, which is a feature of the foreground road portion in the image. For example, the feature extraction unit 319 extends line segments along the boundary or contour mentioned above. The feature extraction unit 319 appropriately selects multiple line segments in the image that are directed toward approximately the center of the image, and determines the vanishing point from the intersection of these line segments. Through this process, the feature extraction unit 319 can determine the position of the vanishing point within the image. In each image of Figure 9, a symbol of a cross enclosed in a circle is shown. The position of this symbol indicates the position of the vanishing point. A straight line crossing this symbol horizontally indicates the eye level.

[0060] If the road extends in a straight line, the distance from the far end of the extracted boundary line or contour to the vanishing point will be relatively short. On the other hand, if the road is a dead end, changes direction at an intersection, or curves, the boundary line or contour will become obscured along the way. As a result, the distance from the far end of the boundary line or contour to the vanishing point will be relatively long.

[0061] Based on the trends described above, the straightness of the road can be estimated. The continuity of road features can be detected using the same method as described above for determining the straightness of the road.

[0062] (SA12: Selection of landscape images suitable for subsequent processing) Referring to Figure 10, the selection of a landscape image suitable for subsequent processing by the process in step SA12 described above will be explained. The landscape image is selected based on the results of each process according to the embodiments below.

[0063] Incidentally, some of the comparative examples involve randomly selecting multiple images for subsequent processing. In this comparative example, it is not guaranteed that only images suitable for subsequent processing will be collected. Multiple images collected randomly may include images that are not suitable for subsequent processing. Depending on the extent to which these unsuitable images are included, they may become noise and reduce the accuracy of the subsequent processing. If it is possible to select images suitable for subsequent processing from multiple randomly selected images, the reduction in the accuracy of the subsequent processing can be suppressed.

[0064] Therefore, in this embodiment, images unsuitable for subsequent processing are excluded from processing by focusing on the following two points, thereby selecting and using images suitable for subsequent processing. This will be explained below.

[0065] First, here is an example of an image that is "unsuitable for subsequent processing." Figure 10 is a diagram illustrating "images unsuitable for subsequent processing" in the embodiment.

[0066] The first example of an "image unsuitable for subsequent processing" is an image showing a view across the road from the shooting location. In this image, buildings along the road are centrally visible, and the road extending in the direction of the vanishing point is not included. The image in Figure 10(a) (corresponding to the image in Figure 4(a) mentioned above) is one example. Therefore, images related to this type are influenced more by the characteristics of buildings along the road than by the characteristics of the local landscape. For example, if the distance from the shooting location to the building is even closer, even a two-story building may only show its exterior wall, and the view over the roof may not be included in the image. Also, as in the image in Figure 10(a), there may be on-site pathways around the building, but distinguishing these from general roads has sometimes affected the quantification of the urban landscape.

[0067] The second type of image unsuitable for subsequent processing is one that contains an object extremely close to the shooting location. For example, when acquiring images using an in-vehicle camera, the view may be obstructed by a vehicle stopped in front of the car at a traffic light, or by a vehicle or person crossing in front of the car. The image in Figure 10(b) (corresponding to the image in Figure 4(b) mentioned above) is one such example. In this way, a part of the image may be temporarily covered by an object. Even if only a part of the image is covered, if that area is relatively large, the resulting image will have a different tendency than when there is no object. Thus, even if an image is obtained by capturing a view of the road extending in the direction from the shooting location, if there is an object extremely close to the shooting location, the image may be covered by that object, making it unsuitable for subsequent processing (analysis).

[0068] The landscape image extraction unit 312 identifies images unsuitable for subsequent processing and images suitable for subsequent processing from among multiple images taken on the road, based on predetermined selection criteria, and selects the latter images as landscape images. The following methods may be applied when identifying "images unsuitable for subsequent processing".

[0069] The landscape image extraction unit 312 distinguishes between pathways within the site and general roads by identifying them based on the image features within the landscape image. For example, the image features include at least one of the following: the distance to the subject at the center of the image, the image boundary, continuity, and width. As described above, the vanishing point of the landscape image is determined based on the extracted line segments.

[0070] The landscape image extraction unit 312 distinguishes between pathways within the site and roads based on, for example, an estimated distance from the shooting point to the actual position of the subject in the center of the image. Perspective-based calculations are used to estimate the distance. This makes it possible to distinguish between roads that extend relatively far, as shown in Figure 5(a), roads that bend along the way, as shown in Figure 7(c), and pathways within the site that do not have roads extending in the direction of the optical axis, as shown in Figure 10(a). The landscape image extraction unit 312 may include images containing roads that extend beyond a predetermined value from the shooting point, as shown in Figures 5(a) and 7(c), in the landscape image, and conversely, exclude images containing images of buildings that cover the vanishing point and appear relatively large, as shown in Figure 10(a). Note that this is not limited to buildings; the same applies to walls, hedges, and plantings.

[0071] The landscape image extraction unit 312 distinguishes between on-site pathways and roads based on the positional relationship between the image boundary and the vanishing point, for example, using the image boundary. In the case of on-site pathways where there are no roads extending in the direction of the optical axis, as shown in Figure 10(a), the image boundary, based on curbs, lane markings, etc., near the site boundary, often exists in a direction that crosses the image. In the image of Figure 10(a), lane markings exist in a direction that crosses the image. Thus, when a line or line segment extending beyond a predetermined length in a direction that crosses the image is detected in the image below the vanishing point, there is a possibility that something other than a road exists between that line or line segment and the vanishing point.

[0072] The landscape image extraction unit 312 distinguishes between on-site pathways and roads based on the image continuity and the positional relationship between the image continuity and the vanishing point, for example, using image continuity. In the image of Figure 10(a), a lane marking exists in a direction that crosses the image. In this way, when a line or line segment crossing the image is detected in the region below the vanishing point, it is appropriate to estimate that an on-site pathway exists between that line or line segment and the vanishing point.

[0073] This allows for the separation of roads from pathways within the property.

[0074] In the above explanation, we proposed using road features to select landscape images, but the images also contain buildings facing the roads. It is rare for buildings facing roads to be built with the same design, height, and setback. Therefore, it can sometimes be more difficult to use building features to derive vanishing points than to use curbs or road indicators.

[0075] (SA14: Identification of the type of subject in a landscape image) Referring to Figure 11, the identification of the type of subject in the landscape image related to the processing in step SA14 described above will be explained. Figure 11 is a diagram for explaining the setting of the range of a partial image in the landscape image of the embodiment. As described below, the processing in step SA14 can be divided into, for example, two stages. By performing the following processes on each landscape image, a partial image is set within a predetermined range of the landscape image, and the type of subject within that predetermined range is identified. The following explains each step in order.

[0076] • Setting the range of a partial image in a landscape image. Identifying the type of subject in landscape images.

[0077] (SA14-1: Setting the range of a partial image in a landscape image) In the landscape image shown in Figure 11, for example, a range is defined above the vanishing point within the landscape image as a partial image, based on the vanishing point's position. The range of the partial image is an example of the evaluation target area.

[0078] (SA14-2: Identification of subject types in landscape images) Next, with reference to Figures 12 to 14, the identification (classification) of the types of subjects in the landscape image will be explained. Figures 13 and 14 are diagrams illustrating the identification of the types of subjects in the landscape image according to the embodiment. Figure 12 is a diagram illustrating the results of extracting the empty area SK, the planted area PL, the road area RD, and the building area BL from the landscape image according to the embodiment. Each image in Figure 13 is the same as the image in Figure 10, with the addition of an eye-level indicator and a figure simulating the planting area. Each image in Figure 14 simulates the result of extracting the empty area SK and the planting area PL from the image in Figure 13.

[0079] In recent years, a process has been developed that uses deep learning-based artificial intelligence (pre-trained models) to identify the types of subjects within a single image. One algorithm for this classification process is "semantic segmentation." In such a classification process, a appropriately trained pre-trained model is used to identify the type of subject that corresponds to each pixel in the image.

[0080] For example, the type identification processing unit 314, by performing processing using such a method, identifies each pixel in the image in relation to each category shown in Figure 3 above, and generates an image in which a label corresponding to the category (class) is assigned to each pixel. The images shown in Figures 13 and 14 above represent hypothetical segmentation results to illustrate the process. However, in practice, the same processing should be performed based on the image in Figure 10 to obtain the identification result shown in Figure 12.

[0081] The type identification processing unit 314 calculates the size (area ratio) of each type of region based on the number of pixels for each type, using the labeling results corresponding to the classification (class).

[0082] Furthermore, the algorithm used for the above classification process is not limited to a specific method; any known algorithm with higher object identification capabilities may be applied.

[0083] The type identification processing unit 314 may identify the type of subject in the extracted landscape image using a trained model. This trained model may be implemented as a combination of an encoder network and a decoder network for "semantic segmentation," or it may be constructed using other methods.

[0084] (SA16: Calculation and quantification of area ratios within landscape images) To calculate the area ratio of regions within a landscape image, one of two methods is applied, and based on these results, a quantitative analysis of the urban landscape is performed.

[0085] Method for calculating the first area ratio (CASE 1): In this case, the landscape image to be processed by the landscape quantification processing unit 316 is an image that includes at least an empty area. The landscape quantification processing unit 316 determines an index value for the city landscape where the landscape image was taken, using the proportion of empty space within the proportions calculated for each category.

[0086] Method for calculating the first area ratio (CASE 2): In this case, the landscape image to be processed by the landscape quantification processing unit 316 is an image that includes at least an empty area, and depending on the situation, an area with vegetation. The landscape quantification processing unit 316 determines an index value for the city landscape where the landscape image was taken, using the ratio of empty areas and the ratio of vegetated areas within the ratios calculated for each type.

[0087] In this way, the scope of the evaluation target can be selected in the "quantification process of urban landscapes."

[0088] The landscape quantification processing unit 316 calculates an index value for the city's landscape using predetermined calculation criteria. For example, the following formula (1) may be used as the predetermined calculation criteria.

[0089] (Streetscape) = k1xa1 + k2xa2 + ... + knxan (1)

[0090] In equation (1), a1, a2, ..., an are ratios calculated for each category. k1, k2, ..., kn are coefficients set for each category. The values ​​of the coefficients in equation (1) are set to classify the characteristics of the city's landscape. The values ​​of the coefficients in equation (1) can be determined using statistical methods such as principal component analysis.

[0091] Figure 15A is a diagram illustrating the results of the landscape quantification process in the embodiment. The table shown in Figure 15A is a modified version of the data table for the landscape image data storage unit 322 shown in Figure 3. The table in Figure 15A includes data for items such as image identification information (No.), the ratio of each subject type, and landscape index values. The subject types are categorized into three groups: sky, planted trees, and others. The table in Figure 15A shows the results of calculating landscape index values ​​for several representative locations with distinct urban landscape characteristics. As shown in this table, the subject types include at least the sky and vegetation. The table in Figure 15A is an example of deriving landscape index values ​​for the city where the landscape image was taken, using the ratio of the sky area to the vegetation area from the subject-specific ratios calculated for each subject type.

[0092] In equation (1) above, (k1, a1) is assigned to the coefficient and ratio of the empty area, (k2, a2) to the coefficient and ratio of the planted area, and (k3, a3) to the other areas such as roads, buildings, etc. For example, this table shows "Result 1" when (k1, k2, k3) are set to (1, 0, -0.1), "Result 2" when they are set to (1, 0.5, -0.1), and "Result 3" when they are set to (1, 1, -0.1). "Result 1" corresponds to CASE 1 above. "Result 2" and "Result 3" correspond to CASE 2 above.

[0093] The landscape image used for this explanation is the image shown in Figure 13, mentioned earlier.

[0094] The results shown in Figure 15A(a) are an example of index values ​​calculated using the ratio to the entire landscape image. In this case, even when the proportion of the sky is relatively large, the difference is suppressed and difficult to discern.

[0095] The results shown in Figure 15A(b) are examples of index values ​​calculated using ratios for partial images. In this case, the value shown in "Result 1" is relatively close to the ratio of empty space. The value shown in "Result 3" shows a trend similar to the trend of planted area ratios. Thus, by analyzing partial images, differences in conditions become easier to understand.

[0096] As described above, by setting the coefficient values ​​to values ​​that match the purpose of the analysis, the urban landscape can be quantified in accordance with the purpose of evaluating the urban landscape.

[0097] (SA18: Estimation of landscape value) The regional value estimation unit 317 performs landscape value estimation processing using the area ratio of the above-mentioned region stored in the landscape image data storage unit 322. In this case as well, the range to be evaluated can be selected, similar to the "quantification processing of urban landscape" described above. The regional value estimation unit 317 calculates an index value of the town's landscape using predetermined calculation criteria. For example, the following formula (2) may be used as the predetermined calculation criteria.

[0098] (Cityscape value) = kk1xa1 + kk2xa2 + ... + kknxan (2)

[0099] In equation (2), a1, a2, ..., an are the same as in equation (1), and are the ratios calculated for each type. kk1, kk2, ..., kkn are coefficients set for each type. The values ​​of the coefficients in equation (2) are set to classify them based on the probability of each type of subject being present in the field of view within the urban landscape. The values ​​of the coefficients in equation (2) can be determined by statistical methods such as principal component analysis.

[0100] Furthermore, the regional value estimation unit 317 may use a defined model for estimating the value of the region to be evaluated, and use this model to estimate the value of the region based on landscape images of the region to be evaluated. This model is preferably determined using the relationship between the results of subjective evaluations of landscape images taken with the direction of road extension included in the field of view, and the ratios calculated for each type of subject in the landscape images used for the subjective evaluation. For example, the coefficients kk1, kk2, ..., kkn in equation (2) above may be appropriately adjusted and used as variables to characterize the above model. The results of the subjective evaluation may be the results of a questionnaire given to multiple users who are relevant to the results of the regional value estimation. The questionnaire may include items such as "Is it a town with a good view?", "Is it a town with abundant greenery?", and "Is it a town convenient for shopping?", and kk1, kk2, ..., kkn may be appropriately adjusted according to each item.

[0101] As described above, the scenic value of a town can be quantified using a relatively simple evaluation method based on landscape images of the area being evaluated.

[0102] Refer to Figure 17 to explain the relationship between the sky area in the landscape image and the trends in the state of the city. Figure 17 is a diagram illustrating the relationship between the sky area in a landscape image and the trends in the state of the city. Figure 17 clearly shows the distinction between the sky area (SK) and the area of ​​buildings and vegetation facing the road (EZ). Buildings and vegetation are collectively referred to as "buildings, etc."

[0103] The boundary L between area SK and area EZ is related to the state of the town facing that road. The inclination and position of boundary L in the landscape image change depending on the height and location of buildings, etc.

[0104] For example, among the boundaries L in a landscape image, boundary L1 relating to buildings along the road shows the following trend. The higher the height of buildings and other structures, the steeper the slope of boundary L1 in the boundary landscape image becomes. Furthermore, the wider the road where the landscape image is taken, the further buildings and other structures are positioned from the shooting point in the width of the road. Therefore, assuming the same height for buildings and other structures, the wider the road where the landscape image is taken, the further outward the boundary L1 in the landscape image shifts horizontally, and the gentler the slope of boundary L1 becomes.

[0105] In landscape images, the boundary L2 related to buildings, etc., in the frontal direction shows the following trend. As mentioned above, if there is a building or other structure in the front direction, boundary L2 becomes apparent. If the distance to the building in the front direction is relatively far, a relatively short boundary L2 will exist. In this case, the relatively short boundary L2 has little effect on the analysis results. When the distance to a building in the front direction is relatively short, a relatively long boundary L2 exists. In this case, the closer the distance to the building, and the taller the building, the further the position of the boundary L2 in the landscape image will shift upward in the vertical direction of the landscape image.

[0106] In this way, the relationship between the empty area SK within the landscape image and the area EZ, which includes buildings facing the road, allows us to interpret the urban landscape as depicted in the image.

[0107] According to the information processing system 1 of the above embodiment, the ratio calculation unit 315 calculates the ratio of the area occupied by the image region of the subject within the landscape image, which shows the scenery in the direction of extension of the road where the shooting point is located, for each type of subject. The landscape quantification processing unit 316 uses the ratio calculated for each type to determine an index value of the city landscape where the landscape image was taken. This makes it possible to quantify the city landscape based on the landscape image.

[0108] The landscape image extraction unit 312 may select images from among multiple images taken on the road that include images showing the view in the direction of extension of the road from the shooting point, but exclude images showing the view in the direction of crossing the road from the shooting point, and extract the selected images as landscape images.

[0109] For example, the landscape image extraction unit 312 may determine an index value for the city landscape in which the landscape image was taken by using the ratio of the sky area within the ratio calculated for each subject type, provided that the subject type includes at least the sky. Alternatively, the landscape image extraction unit 312 may determine an index value for the city landscape from which the landscape image was taken, using the ratio of the sky area to the vegetation area within the ratio calculated for each subject type, provided that the subject type includes at least the sky and vegetation. In any of the above cases, the landscape image extraction unit 312 can derive indicator values ​​for the city's landscape, thereby quantifying the city's landscape.

[0110] Furthermore, the landscape quantification processing unit 316 of the information processing system 1 in the above embodiment may determine an index value of the city landscape in which the landscape image was taken, using the area ratio calculated for each type of subject in the landscape image and the road feature quantities including the shooting location.

[0111] Furthermore, the landscape quantification processing unit 316 of the information processing system 1 in the above embodiment distinguishes between pathways and roads within the site based on the feature quantities of the images in the landscape image, identifies roads extending from the point where the landscape image was taken, and uses the identification result to determine whether the landscape image is suitable. This allows for the selection of a landscape image suitable for processing by the ratio calculation unit 315 and the landscape quantification processing unit 316.

[0112] (Modified version of the first embodiment) In the first embodiment, an example was described in which the range of the partial image is positioned above the vanishing point. In this modification, instead, an example is described in which the partial image is positioned within a portion of the range above the vanishing point. This modification is suitable when the direction of the optical axis of the terminal device 2 fluctuates with each image capture. This will be explained below.

[0113] (Regarding variations of "SA14-1: Setting the range of partial images in landscape images") The setting of the partial image range in this modified example will be explained using Figures 15B, 16A, and 16B. Figure 15B is a diagram illustrating the results of the landscape quantification process in the embodiment. Figures 16A and 16B are diagrams illustrating the setting of the partial image range in the landscape image of the modified example of the embodiment.

[0114] First, using Figures 16A and 16B, we will explain an example of a case where fluctuations occur in the orientation of the optical axis of terminal device 2 with each image capture, and then we will explain the analytical results using Figure 15B.

[0115] Figures 16A(a1) to (a3) ​​show three images obtained by taking pictures in the same direction without changing the shooting position. In addition to being taken at different times, these images show that the orientation of the camera's optical axis fluctuates vertically, resulting in different captured ranges. For example, the positions of the traffic lights in the images are clearly different. Also, because the images were taken at different times, the positions and number of vehicles in the images are different as well. Although the area of ​​each image in Figure 16A(a1) to (a3) ​​is the same, it can be seen that at least the area of ​​the empty region differs from one another. Thus, when using the area ratio to the entire image as an evaluation metric, fluctuations in the orientation of the optical axis affect the results of each evaluation metric.

[0116] Therefore, Figures 16A(b1) to (b3) show the three images in Figure 16A(a1) to (a3) ​​above, with their positions adjusted within Figure 11 to align the vanishing points (eye level). In this way, by adjusting the position of the images, the apparent height of the traffic lights can be made uniform.

[0117] Comparing the images in Figure 16A(b1) to (b3), it can be seen that the area above eye level differs from one another. Thus, when the situation supporting the terminal device 2 changes, the range included in the acquired image also changes.

[0118] Therefore, in this modified example, a portion of the landscape image is extracted as a partial image and used for subsequent processing.

[0119] The settings for partial images should primarily be configured to meet the following conditions. - Ensure that the images to be processed are consistent in size, pixel configuration, etc. • Absorbing fluctuations in the optical axis that occur during imaging.

[0120] For example, depending on the degree of influence of fluctuations during imaging, one of the following two cases should be applied. In particular, "Case 1" below is suitable when the influence of fluctuations in imaging conditions is less than that of "Case 2". Case 1 is suitable when the camera support is relatively robust, while Case 2 is suitable when the direction of the optical axis fluctuates, such as when the camera is supported by hand. However, the above examples of application are just examples and are not limited to these.

[0121] Case 1: For example, as shown in Figures 16A(b1) to (b3), the range of the partial image is set to the area above the vanishing point in the landscape image. In this case, the width of the partial image will be equal to the width of the landscape image. The height of the partial image will be the range from the top edge of the landscape image to the height of the vanishing point. The outer frame of the partial image and the vanishing point will be in a positional relationship.

[0122] Case 2: For example, as shown in Figures 16B(c1) to (c3), the area of ​​the partial image is placed within the range above the vanishing point in the landscape image. In this case, the width of the partial image should be equal to, for example, the width of the landscape image, or narrower than the width of the landscape image. The height of the partial image is defined so that it is shorter than the distance from the top edge of the landscape image to the height of the vanishing point. For example, it is good practice to position the range (outer frame) of the partial image so that the bottom edge of the outer frame of the partial image coincides with the vanishing point (eye level).

[0123] I will explain the two cases mentioned above together. A common feature of landscape images is that the area below the vanishing point is excluded from evaluation. This area is below eye level. This area mainly includes roads, roadside structures, vehicles, and people. For example, even if something that is higher than the ground but lower than eye level is in the field of view, it is less likely to cause a sense of oppression than if something that is higher than eye level were in the field of view.

[0124] On the other hand, if the direction of the camera's optical axis changes upward or downward, the position of the vanishing point in the landscape image changes. In particular, depending on the direction of the optical axis, the area above and below the vanishing point in the landscape image changes accordingly. For example, if the optical axis is pointed slightly downward (Figure 16A(a1), (b1), Figure 16B(c1)), the area of ​​the sky above the vanishing point decreases, and the area of ​​the road below the vanishing point increases. Conversely, if the optical axis is pointed slightly upward (Figure 16A(a3), (b3), Figure 16B(c3)), the area of ​​the sky above the vanishing point increases, and the area of ​​the road below the vanishing point decreases.

[0125] As shown in Figures 16A(a1) to (a3), in the comparative example where partial images are not used in the analysis, the ratio of the area of ​​each divided region to the total area of ​​the landscape image is derived. In this case, the area of ​​the landscape image does not change, but the area of ​​the region above the vanishing point changes. Therefore, it can be seen that the accuracy of the value indicated by the area ratio in this case is low.

[0126] In the first case described above, we derive the ratio of the area of ​​each type of region located above the vanishing point to the area of ​​the area above the vanishing point in the landscape image. In this case, the area of ​​the landscape image remains unchanged, as in the comparative example, but this is not used in the analysis. By using the area of ​​the region above the vanishing point instead of the area of ​​the landscape image, we can align the trend of change in the area of ​​the region above the vanishing point with the trend of change in the area of ​​each type of region. For example, if the optical axis points downward, the area of ​​the sky above the vanishing point decreases, and the areas of each part included above the vanishing point may decrease accordingly. As described above, even if the trends of increase and decrease can be aligned, the rates of change related to increase and decrease will not be aligned. Therefore, it is difficult to completely eliminate the above effects, but it is possible to mitigate them.

[0127] In the second case described above, the partial image lies within the range above the vanishing point in the landscape image. We derive the ratio of the area of ​​each type of region located within the partial image to the area of ​​this partial image.

[0128] For example, in this second case, the range of the partial image is set so that the upper edge of the landscape image is excluded from the range of the partial image. By setting the range of the partial image in this way, even if there are vertical fluctuations in the optical axis, as long as the fluctuations are small enough that the range set as the partial image does not move outside the landscape image, the effect of the vertical fluctuations in the optical axis becomes less significant. Thus, in the second case, the accuracy of the value indicated by the area ratio is higher compared to the first case.

[0129] In the above case, the area defined in the partial image of this second case corresponds to the vicinity of the center of a person's field of vision when they look in the direction of the road's extension in a natural posture. Therefore, it can be said that how this area of ​​the partial image is classified affects the impression of the scenery from that point.

[0130] Referring to Figure 15B, the results of the quantification process when the method for setting the range of a partial image is changed will be explained. The table shown in Figure 15B contains the same items as in Figure 15A mentioned above. The landscape images used in this explanation are the images from Figure 16A(b1) to (b3) mentioned above.

[0131] The results shown in Figure 15B(a) are examples of index values ​​calculated using ratios to the entire landscape image. In this case, as the proportion of the sky increases, the proportions of other elements tend to decrease accordingly. As a result, none of the results from "Result 1" to "Result 3" can absorb the effects of the differences between the images in Figures 16B(b1) to (b3), resulting in differences in each value.

[0132] The results shown in Figure 15B(b) are an example of index values ​​calculated using the ratio of a partial image set above the vanishing point. In this case, even if the ratio of the empty space increases, changes are seen in the other ratios, but not as much as in the results shown in Figure 15B(a) above. As a result, among "Result 1" to "Result 3", the result of "Result 3" in particular can be said to mitigate the effect of the differences in the images in Figures 16B(b1) to (b3).

[0133] Although we will omit the explanation of the analytical verification results of the method shown in Figures 16B(c1) to (c3) above, it is clear without further explanation that the values ​​obtained from the analysis using partial images of the landscape image in Figure 16A(b1) above correspond to those obtained from the analysis.

[0134] This section explains the differences from the technology found in typical cameras (comparative example). In camera technology, there is a technique called "image stabilization" that uses image processing methods to correct the effects of vibrational fluctuations in the optical axis caused by so-called "camera shake," thereby suppressing image movement within the image due to camera shake. This technique uses the correlation between multiple images sampled at regular intervals to mitigate the effects of the aforementioned vibrational fluctuations. However, if the component of the fluctuation is not due to vibrational fluctuations in the optical axis caused by so-called "camera shake," then even using such "image stabilization" techniques, the effects of that fluctuation cannot be suppressed.

[0135] For example, when a person holds a camera with their hand while taking a picture, the optical axis tends to point in the direction of the person's interest. This fluctuation is different from the vibrational movement known as "camera shake," so even when shooting with terminal device 2 that has the so-called "image stabilization" function enabled, this fluctuation cannot be completely suppressed. According to this embodiment, it is possible to suppress fluctuations caused by biases in the direction of the optical axis contained in a single still image of a landscape, based on that landscape image. This is different from general "image stabilization" technology.

[0136] The application of this technology is not limited to cases where a person holds the terminal device 2B (camera) by hand while taking pictures. For example, changes in the load of the vehicle's cargo or the number of occupants can cause changes in the load distributed between the front and rear wheels of the vehicle. In such cases, the front of the vehicle may lift slightly or sink slightly, and the vehicle's posture (horizontalness) may change. By applying the above processing to images acquired by the camera of terminal device 2A mounted on such a vehicle, the effects of changes in the vehicle's posture can be suppressed.

[0137] (Second embodiment) In the following embodiment, we will explain an example in which the control unit 310 uses a deep learning method, focusing on the differences from the first embodiment.

[0138] First, we will describe an example in which the landscape image extraction unit 312 of the control unit 310 extracts a landscape image using a deep learning method. The landscape image extraction unit 312 is configured to include, for example, a pre-trained multiphase artificial neural network (DNN). When extracting landscape images, the landscape image extraction unit 312 uses a appropriately trained DNN as a pre-trained model. By using this pre-trained model, the landscape image extraction unit 312 is configured to extract landscape images suitable for subsequent processing.

[0139] During the training of this DNN, each part within the control unit 310 is configured as follows.

[0140] The information acquisition unit 311 (training data acquisition unit) acquires training landscape images as training data and adds them to the acquired image data storage unit 321. The training landscape images are assigned image data and labels indicating that the images are suitable for processing. The training landscape images may include images of landscapes that are not suitable for processing, and in this case, labels indicating that the images are not suitable for processing are assigned.

[0141] As a result, the information acquisition unit 311 acquires training data, to be used as training landscape images, in which labels are assigned to the training landscape images, to be used as training landscape images for selecting from multiple images taken on the road that include images showing the view in the direction of extension of the road from the shooting point, but exclude images showing the view in the direction of crossing the road from the shooting point.

[0142] The learning processing unit 350 uses a training landscape image containing images taken at multiple shooting locations to train the recognition characteristics of the model included in the landscape image extraction unit 312. A general method may be used for training the model.

[0143] The landscape image extraction unit 312 extracts landscape images using the trained model that has been trained as described above. Subsequent processing may follow the example of the first embodiment.

[0144] [Learning Process] The learning process of the embodiment will be described with reference to Figure 18. Figure 18 is a flowchart of the learning process in the embodiment. The information acquisition unit 311 acquires the above-mentioned learning landscape image as training data (step SA22) and adds it to the training image data storage unit 324. Note that the setting value data storage unit 326 may already contain adjustment setting values ​​(parameters), or the information acquisition unit 311 may acquire them and add them to the setting value data storage unit 326.

[0145] The learning processing unit 350 uses a learning landscape image set containing images taken at multiple shooting locations and adjustment settings to perform a learning process (step SA24) to train the recognition characteristics of the model included in the landscape image extraction unit 312, and adds the learning results to the learning result storage unit 325. This model is then trained to extract road features from the image and identify images suitable for subsequent processing.

[0146] According to the above embodiment, the landscape image extraction unit 312 achieves the same effect as the first embodiment by using its trained model.

[0147] (Third embodiment) This section describes an example in which the type identification processing unit 314 of the control unit 310 uses a deep learning method to identify the type of subject in a landscape image. The type identification processing unit 314 includes a multi-layer encoder network and decoder network configured to apply, for example, "semantic segmentation." The type identification processing unit 314 uses a model that has been appropriately trained.

[0148] The information acquisition unit 311 (training data acquisition unit) acquires training landscape images as training data and adds them to the acquired image data storage unit 321. For example, these training landscape images are labeled images in which the image region is divided according to the type of subject (sky, plants, etc.). Furthermore, similar to the second embodiment, these learning landscape images may be labeled to identify images showing the view in the direction of road extension from the shooting point and images showing the view in the direction of road crossing from the shooting point, from among multiple images taken on the road, or they may consist of images showing the view in the direction of road extension from the shooting point, from among multiple images taken on the road.

[0149] The learning processing unit 350 uses the training landscape images to which labeled images have been assigned to train the classification characteristics of the model included in the classification classification processing unit 314. A general method may be used to train the model.

[0150] The type identification processing unit 314 identifies the type of image of a subject in the landscape image using the trained model that was trained as described above. For processes other than those described above, the examples of the first and second embodiments may be applied.

[0151] According to the above embodiment, the type identification processing unit 314 achieves the same effect as in the first embodiment by using its trained model.

[0152] (Fourth embodiment) This section describes an example of how the regional value estimation unit 317 of the control unit 310 estimates regional value using a deep learning method. The regional value estimation unit 317 includes, for example, a DNN. The regional value estimation unit 317 estimates regional value using a appropriately trained DNN as a model.

[0153] The information acquisition unit 311 (training data acquisition unit) acquires training landscape images as training data, to which the results of users' subjective evaluations of sample landscape images (responses to questionnaires, or the aggregated results thereof, etc.) are assigned as labels, and adds them to the acquired image data storage unit 321. For example, for subjective evaluations, multiple items (questions) such as "Is it a city with a good view?", "Is it a city with abundant greenery?", and "Is it a city convenient for shopping?" are prepared, and responses to sample landscape images are collected in the form of a questionnaire. For example, a 5-point scale from 1 to 5, ranging from negative to positive, is provided as the answer. Users evaluate each sample landscape image and respond with a 5-point scale value.

[0154] Furthermore, similar to the second embodiment, these learning landscape images may be labeled to identify images showing the view in the direction of road extension from the shooting point and images showing the view in the direction of road crossing from the shooting point, from among multiple images taken on the road, or they may consist of images showing the view in the direction of road extension from the shooting point, from among multiple images taken on the road.

[0155] The learning processing unit 350 uses the training landscape images, each with a label, to train the discriminant characteristics of the model included in the regional value estimation unit 317. A general method may be used to train the model.

[0156] The regional value estimation unit 317 uses the trained model, which has been trained as described above, to identify the type of landscape image. For processes other than those described above, the examples of the first to third embodiments may be applied.

[0157] According to the above embodiment, the regional value estimation unit 317 achieves the same effect as the first embodiment by using its trained model.

[0158] Furthermore, the description of the regional value estimation unit 317 in this embodiment may be applied to the landscape quantification processing unit 316 to derive indicator values ​​for the town that are appropriate for the purpose of evaluation.

[0159] According to at least the above embodiment, the information processing system 1 includes a ratio calculation unit 315 that calculates the ratio of the area occupied by the image region of a subject in a landscape image showing the scenery in the direction of extension of the road where the shooting point is located, for each type of subject, and a landscape quantification processing unit 316 that uses the ratio calculated for each type to determine an index value of the city landscape where the landscape image was taken, thereby enabling the quantification of the city landscape based on the landscape image.

[0160] Furthermore, the information processing system 1 includes a ratio calculation unit 315 that calculates the ratio of the area occupied by the image region of each subject within a landscape image showing the scenery in the direction of extension of the road where the shooting point is located, for each type of subject; a feature extraction unit 319 that extracts the feature quantities of the road; and a landscape quantification processing unit 316 that uses the ratio calculated for each type and the feature quantities of the road to determine an index value of the city landscape where the landscape image was taken. In this way, the city landscape can be quantified based on the landscape image.

[0161] Furthermore, the information processing system 1 includes a feature extraction unit 319 that extracts feature quantities of images within a landscape image, a landscape image extraction unit 312 (road identification unit) that distinguishes between pathways and roads within the site based on the feature quantities of images within the landscape image and identifies roads extending from the point where the landscape image was taken, a ratio calculation unit 315 that calculates the ratio of the area occupied by the image region of each subject within a landscape image showing the scenery in the direction of extension of the road where the shooting point is located, for each type of subject, and a landscape quantification processing unit 316 that uses the ratio calculated for each type to determine an index value of the city landscape where the landscape image was taken. In this way, the city landscape can be quantified based on the landscape image.

[0162] Furthermore, the information processing system 1 includes a ratio calculation unit 315 that calculates the ratio of the area occupied by the image regions of different types of subjects within a partial image extracted from a landscape image captured by the camera of the terminal device 2 according to predetermined rules, and a landscape quantification processing unit 316 that uses the ratios calculated for each type to determine an index value of the city landscape where the landscape image was taken. In this way, the city landscape can be quantified based on the landscape image.

[0163] Alternatively, the program for implementing Information Processing System 1 may be recorded on a computer-readable recording medium, and each processing operation may be performed by loading the program recorded on this recording medium into a computer system and executing it. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer system" also includes WWW systems equipped with a homepage provisioning environment (or display environment). "Computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into computer systems. Additionally, "computer-readable recording medium" includes volatile memory (RAM) within computer systems that act as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retain programs for a certain period of time.

[0164] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium, or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network or a communication line. Also, the above program may be for the purpose of realizing only a part of the functions described above. Furthermore, it may be a so-called differential file (differential program) that can realize the aforementioned functions in combination with a program already recorded in the computer system.

[0165] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0166] 1…Information processing system, 2, 2A, 2B…Terminal device, 3…Landscape quantification processing device, 310…Control unit, 311…Information acquisition unit, 312…Landscape image extraction unit, 313…Partial image range generation unit, 314…Type identification processing unit, 315…Ratio calculation unit, 316…Landscape quantification processing unit, 317…Regional value estimation unit, 318…Display processing unit, 319…Feature extraction unit, 320…Storage unit, 321…Acquired image data storage unit, 322…Landscape image data storage unit, 323…Partial image range data storage unit, 324…Training image data storage unit, 325…Learning result storage unit, 326…Setting value data storage unit, 350…Learning processing unit

Claims

1. A ratio calculation unit calculates the ratio of the area occupied by the subject's image within a landscape image showing the scenery in the direction of extension of the road where the shooting location is located, for each type of subject. A feature extraction unit for extracting the feature quantities of the aforementioned road, A landscape quantification processing unit determines an index value of the city landscape where the landscape image was taken, using the ratio calculated for each of the aforementioned types, a predetermined position determined to extract the road from the landscape image in the direction of the road's extension, and the feature quantities of the road at least below the predetermined position. Equipped with, The road feature quantities include information indicating the direction of extension of the road. The feature extraction unit, The vanishing point is determined from the foreground road feature quantity, which is the feature quantity of the foreground road portion in the landscape image. Based on the positional relationship between the foreground road feature quantity and the vanishing point, the road feature quantity is extracted when a line or line segment extending beyond a predetermined length in the direction crossing the image is detected in the image below the vanishing point. Information processing system.

2. The road features include information regarding the continuity of the road features. The information processing system according to claim 1.

3. The feature extraction unit, In the aforementioned landscape image, an evaluation target area is predetermined based on the position of the vanishing point, and based on the classification results of the images captured within the predetermined evaluation target area, a label is assigned to landscape images in which the range classified as a moving object covers the vanishing point and exceeds a predetermined value, indicating that it should be excluded from analysis in subsequent processing. The information processing system according to claim 1.

4. The process involves calculating the ratio of the area occupied by the subject's image within a landscape image showing the scenery in the direction of the road's extension from the shooting location, for each type of subject. The steps include extracting the characteristic features of the aforementioned road, A step of determining an index value of the city landscape where the landscape image was taken, using the ratio calculated for each of the above categories, a predetermined position determined to extract the road from the landscape image in the direction of the road's extension, and the feature quantities of the road at least below the predetermined position. Includes, The road feature quantities include information indicating the direction of extension of the road. The vanishing point is determined from the foreground road feature quantity, which is the feature quantity of the foreground road portion in the landscape image. Based on the positional relationship between the foreground road feature quantity and the vanishing point, the road feature quantity is extracted when a line or line segment extending beyond a predetermined length in the direction crossing the image is detected in the image below the vanishing point. A method for quantifying landscape.

Images