Judgment system, judgment method, and program

Abstract

A determination system according to the present disclosure comprises: an acquisition means that acquires a captured image of a road; a recognition means that recognizes the area of the road and the area of a fixed object in the image by image recognition; a depth estimation means that estimates the depth of each portion of the image by depth estimation; an identification means that identifies a determination space in the image on the basis of the position of the area of the road in the image; a detection means that detects a target object area representing the area of the fixed object in the determination space in the image on the basis of the position of the area of the fixed object in the image and said depth; and a determination means that determines whether the fixed object exists in the determination space on the basis of at least one of the position of the target object area in the image and said depth.

Description

Determination system, determination method, and recording medium

[0001] The present disclosure relates to a determination system and the like.

[0002] Road administrators regularly inspect road conditions and take action if any abnormalities are found. To improve the efficiency of road condition inspections, cameras are installed on vehicles to collect images of roads. In addition, systems are known that perform image recognition on the captured images to detect abnormalities such as road surface deterioration.

[0003] Patent Document 1 discloses a device that uses images to check whether a utility pole ahead of a train is outside the construction gauge of the track.

[0004] Japanese Patent Application Laid-Open No. 2015-116916

[0005] Road administrators monitor whether trees and other objects along the roads are encroaching on the road's construction gauge. If it were possible to determine whether an object is encroaching on the road's construction gauge from images captured by a camera, the convenience of the road administrator's management would be greatly improved.

[0006] However, simply analyzing the image in two dimensions does not reveal the depth position of the object, making it difficult to accurately determine whether the object is encroaching on the building gauge.

[0007] One object of the present disclosure is to provide a determination system etc. that enables more accurate determination of objects that pose a risk of disrupting traffic.

[0008] A determination system in one aspect of the present disclosure includes an acquisition means for acquiring an image of a road, a recognition means for recognizing a road area and a fixed object area in the image through image recognition, a depth estimation means for estimating the depth of each part of the image through depth estimation, an identification means for identifying a determination target space in the image based on the position of the road area in the image, a detection means for detecting a target object area in the image representing the fixed object area in the determination target space based on the position and depth of the fixed object area in the image, and a determination means for determining whether the fixed object is in the determination target space based on at least one of the position or the depth of the target object area in the image.

[0009] A determination method in one aspect of the present disclosure acquires an image of a road, recognizes the road area and fixed object area in the image using image recognition, estimates the depth of each part of the image using depth estimation, identifies a determination target space in the image based on the position of the road area in the image, detects a target object area in the image representing the fixed object area in the determination target space based on the position and depth of the fixed object area in the image, and determines whether the fixed object is in the determination target space based on at least one of the position or the depth of the target object area in the image.

[0010] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring an image of a road, recognizing a road area and a fixed object area in the image by image recognition, estimating the depth of each part of the image by depth estimation, identifying a determination target space in the image based on the position of the road area in the image, detecting a target object area in the image that represents the fixed object area in the determination target space based on the position and the depth of the fixed object area in the image, and determining whether the fixed object is in the determination target space based on at least one of the position or the depth of the target object area in the image. The program may be stored in a computer-readable non-transitory recording medium.

[0011] One example of the effect of the present disclosure is that it enables more accurate determination of objects that pose a risk of disrupting traffic.

[0012] FIG. 1 is a configuration diagram showing an example of a device used in a determination system according to an embodiment. FIG. 2 is a block diagram showing an example of the configuration of a determination device according to an embodiment. FIG. 3 is a diagram showing an example of an image of a road. FIG. 4 is a diagram showing an example of a result of recognizing a fixed object from an image. FIG. 5 is a diagram showing an example of a result of recognizing a driving lane from an image. FIG. 6 is a diagram showing an example of a result of estimating depth from an image. FIG. 7 is an explanatory diagram of a method for specifying a determination target space. FIG. 8 is a diagram showing an example of a result of detecting a target object region. FIG. 9 is a diagram showing an example of an estimation result of a depth value of a target object region. FIG. 10 is a diagram showing another example of an estimation result of a depth value of a target object region. FIG. 11 is a table showing example data of fixed objects stored in a database. FIG. 12 is a diagram showing an example of a display screen for determination results. FIG. 13 is a flowchart showing an example of the operation of a determination system according to an embodiment. FIG. 14 is a block diagram showing an example of the hardware configuration of a computer.

[0013] To prevent traffic accidents, road administrators monitor whether fixed objects on or beside the road encroach on spaces where the presence of fixed objects is not permitted. Fixed objects are objects that do not move from a specific location on or beside the road. Spaces on roads where the presence of fixed objects is not permitted are spaces where the presence of fixed objects poses a risk of disrupting traffic. Such spaces may be the construction limit above the road as stipulated by law. However, spaces on roads where the presence of fixed objects is not permitted may also be set at a predetermined distance above or to the side of the road from the construction limit.

[0014] One example of a fixed object that road administrators manage is a tree. Trees whose branches and leaves have grown to the point that they encroach on the building limit must be pruned immediately. Also, trees whose trunks have leaned and encroached on the building limit must be removed promptly. If road administrators can identify trees before they reach the building limit, they can efficiently identify trees that need pruning and develop preventive maintenance plans.

[0015] However, the objects to be managed are not limited to trees. Fixed objects such as signs, billboards, convex mirrors, traffic lights, guardrails, and buildings may also be managed. An example of an embodiment of the determination system 1 that uses an image to determine whether a fixed object has invaded a space on a road where the presence of the fixed object is not permitted will be described below.

[0016] An example of devices used in a determination system 1 according to one embodiment will be described with reference to Fig. 1. The determination system 1 includes a determination device 100, a camera 10, an administrator terminal 20, a storage 40, and a database 50. The devices in the determination system 1 are connected to each other so as to be able to communicate with each other via a communication network 30, for example, via a wired or wireless connection. Note that the determination device 100 does not need to communicate with the camera 10 and the administrator terminal 20. Therefore, the determination device 100 only needs to be connected to the camera 10 and the administrator terminal 20 as necessary.

[0017] The camera 10 is mounted on a moving body 11. The camera 10 captures images of the road and the environment around the road. The camera 10 is realized, for example, by a drive recorder mounted on a vehicle. The drive recorder continuously captures images of at least one of the front and rear of the vehicle while the vehicle is traveling on the road. However, the types of the camera 10 and the moving body 11 are not limited to this, and the camera 10 can be mounted on various types of moving bodies 11. For example, the camera 10 may be mounted on other moving bodies such as a bicycle or a drone. The camera 10 may be carried by a person traveling on the road. Furthermore, the images captured by the camera 10 are not limited to moving images, but may also be still images. The camera 10 is not limited to a camera that continuously captures images.

[0018] The driving data including the images captured by the camera 10 is stored in the storage 40. The camera 10 transmits the driving data including the images to the storage 40 or the determination device 100, for example.

[0019] The travel data further includes location information of the location where the image was captured. The location information is acquired using, for example, the Global Navigation Satellite System (GNSS) or the Global Positioning System (GPS). The location information is expressed, for example, by latitude and longitude or a location on a map. The travel data may also include the date and time when the image was captured.

[0020] The travel data may include information about the route the mobile object 11 was traveling on at the time the image was captured. The route is identified by tracking the location information of the mobile object 11. The route the mobile object 11 was traveling on is represented, for example, by the route name and whether the mobile object 11 was traveling on an inbound or outbound lane of the route.

[0021] The administrator terminal 20 presents information to the administrator. The type of the administrator terminal 20 is not particularly limited, and may be a smartphone, a tablet terminal, a PC (Personal Computer), etc. The administrator terminal 20 accesses the database 50, for example, and displays the information stored in the database 50.

[0022] The storage 40 and the database 50 may be provided as needed. The storage 40 stores driving data including images captured by the camera 10.

[0023] The database 50 stores the geographical location of each fixed object. The geographical location may be represented by, for example, latitude and longitude. The geographical location may also be represented by the location of an area obtained by dividing a road into predetermined sections or by dividing a map into a mesh of a predetermined size, to which the fixed object belongs. The road or map may be divided, for example, at intervals of 10 meters. The database 50 may further include a determination result as to whether the fixed object intrudes into a space on the road where the presence of the fixed object is not permitted. The database 50 may also include information identifying the route on which the fixed object is located, information on the type of the fixed object, or an image of the fixed object and the date and time the fixed object was photographed. Storing information identifying the route makes it possible to identify whether the fixed object is located on an inbound or outbound lane of the road. The type of the fixed object indicates, for example, whether it is a tree or a sign. The type of the fixed object may also be stored as the type of sign. Examples of the format of the database 50 include a relational database (RDB), NoSQL, a spreadsheet file, and the like. However, the format of the database 50 is not limited to this, and any format can be used.

[0024] An example of the configuration of a determination device 100 according to an embodiment will be described with reference to Fig. 2. The determination device 100 includes an acquisition unit 101, a recognition unit 102, a depth estimation unit 103, an identification unit 104, a detection unit 105, and a determination unit 106.

[0025] The acquisition unit 101 acquires an image of a road. For example, the acquisition unit 101 acquires the image by referring to the driving data in the storage 40. Alternatively, the acquisition unit 101 may acquire an image from the camera 10. In this way, the acquisition unit 101 can acquire an image of the front or rear of the road on which the mobile object 11 equipped with the camera 10 is traveling. The acquisition unit 101 may also acquire location information of the point where the acquired image was taken.

[0026] Fig. 3 is a diagram showing an example of an image acquired by the acquisition unit 101. The image in Fig. 3 is an image captured by the camera 10 while the mobile object 11 is traveling in the left lane. The image in Fig. 3 captures a tree growing on the left side of the road.

[0027] The recognition unit 102 recognizes, through image recognition, road areas and fixed object areas in the image acquired by the acquisition unit 101. Fixed objects to be recognized by the recognition unit 102 can be appropriately selected. In the following description, an example will be described in which trees are mainly set as the recognition objects.

[0028] The recognition unit 102 may recognize fixed objects using a model trained by machine learning to recognize fixed objects. For example, the model is trained to input an image of a fixed object and output a correct label of the fixed object attached to the input image. The recognition unit 102 may use a model trained to recognize various types of fixed objects. When an image is input, the model outputs a result in which an object shown in the image is recognized as a fixed object area. The recognition unit 102 may determine, for each pixel, whether or not the pixel represents a fixed object.

[0029] The recognition unit 102 recognizes roads using a technique similar to that for recognizing fixed objects. The recognition unit 102 may recognize roads using the same model for recognizing fixed objects that has been trained to recognize roads and fixed objects. Alternatively, the recognition unit 102 may recognize roads using a model that has been trained to recognize roads but is different from the model for recognizing fixed objects.

[0030] Fig. 4 is a diagram showing an example of the results of recognizing fixed objects and roads from the image of Fig. 3. In the example shown in Fig. 4, it is assumed that the fixed objects to be recognized are trees. In Fig. 4, areas recognized as trees are shown in gray, and areas recognized as roads are shown with wavy lines. In the example of Fig. 4, both the areas of the branches and leaves and the areas of the trunk of the tree are recognized as fixed objects.

[0031] The recognition unit 102 may recognize the area of ​​the driving lane in which the mobile object 11 is traveling. The recognition unit 102 may recognize the driving lane using a model that is trained to recognize the driving lane and that is different from the model that recognizes fixed objects or roads. Alternatively, the recognition unit 102 may recognize the driving lane using the same model that recognizes fixed objects or roads. For example, the recognition unit 102 may recognize dividing lines such as white lines on the road and recognize the area of ​​the driving lane based on the dividing lines. The recognition unit 102 may recognize lanes other than the driving lane and sidewalks in addition to the area of ​​the driving lane.

[0032] Fig. 5 is a diagram showing an example of the result of recognizing the driving lane from the image of Fig. 3. In Fig. 5, in addition to the driving lane, lanes other than the driving lane are also recognized. In the example shown in Fig. 5, the driving lane and the opposite lane are displayed using different patterns.

[0033] The depth estimation unit 103 estimates the depth of each part of the image acquired by the acquisition unit 101 through depth estimation. Depth estimation is a technique for estimating depth from a single image of an object. Depth indicates how far an object is from the camera 10. In depth estimation, depth is estimated based on the position, color, shape, etc. of the object in the image. For example, the depth estimation unit 103 inputs the image into a model trained using images in which the depth values ​​of each part are known, and obtains an output of estimated depth. The closer the estimated distance between the object and the camera 10, the larger the depth value, and the farther the distance, the smaller the depth value. The depth estimated by the depth estimation unit 103 is expressed, for example, by a depth value for each pixel.

[0034] Fig. 6 is a diagram showing an example of the results of depth estimation from the image in Fig. 3. In the example shown in Fig. 6, parts estimated to be farther from the camera 10 are shown in darker colors, and parts estimated to be closer to the camera 10 are shown in lighter colors.

[0035] The identification unit 104 identifies a determination target space in the image based on the position of the road area in the image. The determination target space is a space in which it is possible to determine whether a fixed object has intruded into the determination target space, thereby determining whether the fixed object has intruded into the building gauge or is closer to the building gauge than a predetermined standard. When a road continues in the depth direction of the image, the space of the building gauge on the road also continues in the depth direction. The identification unit 104 identifies, for example, a determination target space that continues in the depth direction and has the same width and height as the building gauge and has a predetermined depth. The identification unit 104 may particularly identify a determination target space that is the building gauge on a driving lane in which the mobile object 11 is traveling. Note that the height of the determination target space to be set is not limited to the height of the building gauge and can be set as appropriate, such as a position higher than the building gauge. The depth of the determination target space can also be set as appropriate, for example, within a range of 1 to 10 meters from the camera 10. As an example, a space with a depth of approximately 3 meters can be set starting from a point approximately 3 meters from the camera 10.

[0036] An example of a method for identifying the determination target space in an image will be described using FIG. 7 . One method for identifying the determination target space is to identify a position in the image that is set in advance based on the shooting range of the camera 10 as the position of the determination target space. In identifying the determination target space, for example, the identification unit 104 identifies the positions of both ends of the driving lane in the image. In one example, the identification unit 104 identifies the positions of both ends of the driving lane at two locations with different depths. In FIG. 7 , the identification unit 104 identifies the positions of the side XY and the side WZ in the image. This identifies the range of the road surface WXYZ having a predetermined width and depth. Then, the identification unit 104 refers to the previous settings and identifies the position of the surface PSRQ at a predetermined height from the road surface WXYZ.

[0037] An example of a process for presetting a position in an image of the determination target space will be described. In the presetting, for example, the size of an object with a known size that appears in the image is used. First, an object with a known height is photographed at a position from the camera 10 corresponding to the side WZ in FIG. 7 . Then, using the actual height of the object and its height in the image as a reference, a position in the image corresponding to the height position of the determination target space at the distance of the side WZ is set. As a specific example, assume that an object with a height of 1.5 meters placed at a distance corresponding to the side WZ is photographed in order to pre-set a position in the image corresponding to a height of 4.5 meters from the road surface. If the height of the 1.5-meter-high object in the image is 100 pixels, then a position 300 pixels above the road surface at the distance of the side WZ is pre-set as the height position of the determination target space. In a similar manner, a position in the image corresponding to a height of 4.5 meters from the road surface is set at a position on the road surface corresponding to the distance from the camera 10 to the side XY in FIG. 7 .

[0038] According to the method of identifying the position of the target space based on a preset height position in an image, the process of identifying the target space is faster than the method of identifying the position for each image, which will be described below. Furthermore, since the height position is fixed, the position of the target space is constant for each image. However, if the mounting position or orientation of the camera 10 is changed, the position of the target space must be set using the height position measurement results each time the change is made.

[0039] Another method for identifying the determination target space in an image is to identify the position of the determination target space for each image. The identification unit 104 identifies the height position of the determination target space using, for example, the result of recognizing the driving lane and data on the actual width of the driving lane. A predetermined value, such as 3.5 meters, may be used as the data on the actual width of the driving lane. Furthermore, the data on the actual width of the driving lane may be registered in advance for each route. The identification unit 104 may acquire the route width data using location information of the point where the image was captured. The identification unit 104 then identifies the height position of the determination target space at a position where the ratio between the actual width of the driving lane and the actual height of the determination target space is equal to the ratio between the number of pixels in the width of the driving lane and the number of pixels from the road surface to a position above the road surface. In FIG. 7 , the number of pixels corresponding to the height of the determination target space is calculated based on the number of pixels of the side WZ. The position of the side PQ, which is shifted upward from the side WZ by the calculated number of pixels, is then identified as the position at the predetermined height.

[0040] In one example, the positions of both ends of the driving lane and the height position above them are identified at two locations at different depths, and therefore the positions of the road surface WXYZ and the plane PQRS parallel to the road surface in Fig. 7 are identified in the image. The identification unit 104 identifies the quadrilateral columnar space between the plane WXYZ and the plane PSRQ above it as the determination target space.

[0041] According to the method of identifying the position of the determination target space for each image, the identification unit 104 can identify the determination target space even if the installation position or orientation of the camera 10 is changed. However, slight differences in the image recognition results may cause the height position of the determination target space to shift.

[0042] The identification unit 104 may identify multiple determination target spaces of different heights in the image. The identification unit 104 may identify multiple determination target spaces, including determination target spaces at heights lower than the building limit height. For example, the identification unit 104 may identify three determination target spaces whose heights are 5 meters, 4.5 meters, and 4 meters from the roadway, respectively. Similarly, the identification unit 104 may identify multiple determination target spaces of different widths. The horizontal position of the determination target space in the image is identified in the same manner as the method for identifying the vertical position. By identifying multiple determination target spaces, the determination unit 106, which will be described later, can determine which of the multiple determination target spaces a fixed object has entered.

[0043] The method by which the specification unit 104 specifies the determination target space is not limited to the example described above.

[0044] The identification unit 104 may further identify a determination target space on a lane other than the driving lane, such as an oncoming lane. Similarly, the identification unit 104 may identify a determination target space on a sidewalk. The construction limit on a sidewalk is set to 2.5 meters above the sidewalk. Therefore, a space 2.5 meters above the sidewalk or a space a predetermined distance above or to the side of the sidewalk may be set as the determination target space on the sidewalk.

[0045] The height of the determination target space identified by the identification unit 104 may be determined depending on whether the road type is a small road, an ordinary road, or an important logistics road. The determination target space may also be determined based on whether the road is a road on which large vehicles pass. For roads on which large vehicles such as trucks pass, the determination target space is determined at a higher position from the road compared to other roads. The determination target space may also be determined based on traffic volume. Roads with heavy traffic volume are subject to a large impact from trees obstructing safe passage, so it is desirable to remove risks early. Therefore, the determination target space is set to include a position farther from the road compared to other roads.

[0046] The detection unit 105 detects a target object region in the image that represents the region of the fixed object in the determination target space, based on the position in the image of the fixed object recognized by the recognition unit 102 and the depth estimated by the depth estimation unit 103. The target object region is a region of the fixed object that is the target for which the determination unit 106, which will be described later, determines whether or not the fixed object has entered the determination target space.

[0047] In detecting the target object region, the detection unit 105 detects, for example, a region of a fixed object whose position in the image overlaps with a position in the determination target space. Then, the detection unit 105 detects, as the target object region, a region of the detected fixed object whose depth is included in the depth range of the determination target space. For example, in FIG. 7 , the detection unit 105 detects a tree reflected inside the rectangle PQZW. Then, the detection unit 105 references the depth of the detected tree region and detects, as the target object region, a region that is included in the depth range of the road surface WXYZ.

[0048] The detection unit 105 may detect a region of a fixed object in the determination target space, and then detect a region whose depth overlaps with the position in the determination target space as the target object region.

[0049] When the identification unit 104 identifies a plurality of determination target spaces at different heights, the detection unit 105 detects a target object region representing a region of a fixed object in each determination target space. For example, the detection unit 105 detects a target object region included in the region of the upper boundary of a determination target space at a height of 4.5 meters, and further detects a target object region included in the region of the upper boundary of a determination target space at a height of 4 meters.

[0050] In detecting the target object region, the detection unit 105 may detect a region of a fixed object whose position in the image is in a region that represents the upper boundary of the determination target space. By narrowing the detection range of the target object region to the region of the upper boundary of the determination target space, the detection unit 105 can detect a region that is suitable for determining a fixed object that intrudes into the determination target space from above, such as tree branches and leaves.

[0051] For example, in Fig. 7, the detection unit 105 detects trees that are reflected inside the quadrangle PQRS. Then, from within the detected tree region, the detection unit 105 detects a target object region whose depth overlaps with the depth range of the determination target space. Fig. 8 is a diagram showing an example of the result of detecting the target object region. In Fig. 8, the target object region is surrounded by a dotted line.

[0052] The detection unit 105 may detect a fixed object that appears in a range that represents the boundary of a side of the space to be determined. In Fig. 7, the detection unit 105 may detect a tree that appears inside the quadrangle PSXW. This allows the detection unit 105 to detect a fixed object that enters the space from the side.

[0053] The detection unit 105 may detect, as separate target object regions, regions of fixed objects that are not continuous in the image among regions of a single determination target space. For example, if a region of a tree invading the determination target space from above and a region of a tree invading the determination target space from the left are not continuous in the image, the detection unit 105 detects these regions separately.

[0054] The determination unit 106 determines whether a fixed object is present in the determination target space based on at least one of the position and depth of the target object region in the image.

[0055] The determination unit 106 may calculate the position of the geometric center in the image of the target object region as the position of the target object region in the image. The position of the geometric center can be used as a point representing the position of the target object region. In the following description, the geometric center is also referred to as the center of gravity. In FIG. 8 , the position of the calculated center of gravity is represented by a black dot. The greater the degree to which the fixed object intrudes from the top to the bottom of the space, the lower the position of the center of gravity of the target object region in the image. Therefore, the determination unit 106 determines that the fixed object is in the determination target space when the vertical position of the calculated center of gravity in the image is lower than a predetermined threshold. For example, the position of the center of gravity of the upper boundary region of the determination target space may be used as the threshold.

[0056] Note that, instead of the center of gravity of the target object region, determination unit 106 may use, as the position of the target object region in the image, the average of the coordinates of the tip position of the target object region in the direction in which the target object region extends from above or to the side of the determination target space. For example, determination unit 106 may use the average of the coordinates of the lower outline of the target object region.

[0057] The determination unit 106 calculates statistics of depth values ​​of pixels in the target object region as the depth of the target object region. The statistics of depth values ​​is, for example, the average of depth values ​​of the target object region. The determination unit 106 obtains the average depth value by dividing the depth value of each pixel in the target object region by the number of pixels in the target object region. It is expected that the average depth value of a tree or the like increases as the tree invades the target space to a greater extent. Therefore, the determination unit 106 determines that a fixed object is present in the target space when the calculated average depth value is greater than a predetermined threshold. For example, the average depth value of the road surface region in the target space may be used as the threshold.

[0058] 9 and 10 schematically show examples of depth value estimation results for the target object region. FIGS. 9 and 10 show depth values ​​of a plane obtained by cutting the target space in the depth direction at a position passing through the center of gravity calculated by the determination unit 106. Points A, A', B, and B' represent points on side PS, side QR, side WZ, and side XY in FIG. 7, respectively. The detection unit 105 detects, as the target object region, a fixed object region whose depth is smaller than points A and B and larger than points A' and B'. The bold-lined region in FIGS. 9 and 10 represents the detected target object region. It is assumed that trees exist above the bold lines. Note that, in reality, the state of the target object region beyond the camera 10 cannot be confirmed from the image. In the example of FIG. 9, almost no fixed objects intrude into the target space. Therefore, the average depth value of the target object region is smaller than when a fixed object intrudes into the target space. 9, in the example of Fig. 10, the fixed object intrudes further into the determination target space on the side closer to the camera 10. In this case, the average depth value of the target object region is larger than in Fig. 9.

[0059] When making a determination based on a combination of the position of the target object region in the image and the statistics of the depth values ​​of the target object region, the determination unit 106 uses a first criterion related to the position and a second criterion related to the depth values. For example, if the position of the center of gravity of the target object region is lower than the first criterion and the average depth value is greater than the second criterion, the determination unit 106 determines that a fixed object has invaded the target space. It is assumed that there is a certain correlation between the depth and center of gravity of a tree that has invaded the building limit. Therefore, the value of one of the first and second criteria may be determined by a function using the value of the other criterion as an explanatory variable. Furthermore, the relationship between the first and second criteria for determining that a fixed object has invaded the target space may be set using a table.

[0060] When the detection unit 105 detects target object regions in multiple determination target spaces with different heights or widths, the determination unit 106 may determine the intrusion degree of the fixed object based on which of the multiple determination target spaces the fixed object intrudes into. The intrusion degree represents the extent to which the fixed object intrudes into the building gauge or is likely to intrude into it. The intrusion degree is expressed in multiple levels, such as large, medium, and small. In one example, when a tree intrudes into a determination target space that is 4.5 meters tall, the determination unit 106 determines the intrusion degree to be "medium." When a tree intrudes into a determination target space that is 4 meters tall, the determination unit 106 determines the intrusion degree to be "large."

[0061] An example of processing in which the determination unit 106 determines whether a fixed object is present in a plurality of determination target spaces will be described. Assume that the detection unit 105 detects a target object region in each of three determination target spaces, e.g., at heights of 5 meters, 4.5 meters, and 4 meters. The determination unit 106 determines whether a fixed object has invaded the determination target space for the target object region in the determination target space at a height of 5 meters. If it is determined that a fixed object has invaded the determination target space, the determination unit 106 determines whether a fixed object has invaded the determination target space for the target object region in the next lowest determination target space at a height of 4.5 meters. If it is determined that a fixed object has not invaded the determination target space, the determination unit 106 determines that the intrusion degree is "small," and does not need to perform a determination for the next lowest determination target space at 4 meters.

[0062] Note that the determination unit 106 may determine the intrusion degree of a fixed object into the determination target space even when only one determination target space is specified. For example, the determination unit 106 determines the intrusion degree based on the degree of lowness of the center of gravity of the target object region or the degree of average magnitude of the depth value of the target object region.

[0063] When the detection unit 105 detects multiple target object regions from one determination target space, the determination unit 106 may determine whether a fixed object is present in the determination target space for each target object region. For example, the determination unit 106 calculates the center of gravity and the average depth value of each target object region. Then, the determination unit 106 determines whether a fixed object in each target object region has entered the determination target space. By distinguishing between discontinuous regions in the image and calculating the average center of gravity and depth value, values ​​more suitable for determination can be calculated.

[0064] The determination unit 106 may register data regarding the fixed object determined to be in the determination target space in the database 50. Alternatively, the determination unit 106 may register data regarding the fixed object in the database 50 regardless of the determination result. The determination unit 106 may further register the determination result regarding the intrusion degree.

[0065] FIG. 11 is a table showing an example of data on fixed objects stored in the database 50. In the example of FIG. 11, the stored data includes a data ID, the geographical position of the fixed object, an image of the fixed object, and an intrusion degree. The determination unit 106 may refer to the travel data and register, as the geographical position of the fixed object, location information of a point where an image in which the fixed object is recognized was taken. The determination unit 106 may also refer to the travel data and register the date and time when the fixed object was photographed. The determination unit 106 may also refer to the travel data and register the route on which the fixed object exists.

[0066] The administrator terminal 20 accesses the database 50 and displays the stored data of fixed objects. The administrator terminal 20 may display the data in the form of a ledger in which one fixed object is represented by one line, as shown in FIG. 11 . The administrator terminal 20 may display a screen that shows the positions of fixed objects on a map based on the stored data. For example, the administrator terminal 20 may display the determination result by the determination unit 106 on a map.

[0067] An example of a display screen of the determination result displayed by the administrator terminal 20 will be described using FIG. 12. The point of the star-shaped icon in FIG. 12 indicates the location of the tree registered in the database 50. In other words, the location of the tree invading the unacceptable space is shown on the map. Depending on the degree of intrusion, an icon with a different size, color, or shape may be displayed. Note that the location of the tree may be represented by a different icon depending on the priority of the response. The priority of the response is determined based on the degree of intrusion and traffic volume. The determination unit 106 determines that the higher the degree of intrusion and the higher the traffic volume, the higher the priority of the point.

[0068] In Figure 12, one icon is selected. On the right side of the map, more detailed information about the tree is displayed, such as the data ID, the extent to which the tree encroaches on the unacceptable space, the latitude and longitude of the tree, and an image of the tree. In the image, the area of ​​the tree that has been determined to encroach on the building gauge is indicated by a dotted line. In this way, the determination unit 106 may register in the database 50 an image showing the area of ​​the tree that has been determined to encroach on the building gauge.

[0069] The determination device 100 may have a function of controlling the display on the administrator terminal 20. The determination device 100 receives a display instruction from the administrator terminal 20, and causes the administrator terminal 20 to display a screen such as that shown in Fig. 11 or 12 for the specified location. In this case, the database 50 may be provided as needed.

[0070] Furthermore, the determination unit 106 may output data of the fixed objects determined to be in the determination target space in CSV (Comma Separated Values) format to the administrator terminal 20. The output data may include, for example, a data ID, a geographical position of the fixed object, an image of the fixed object, or an intrusion degree.

[0071] An example of the operation of the determination system 1 will be described using the flowchart in Fig. 13. The determination system 1 may start the process in Fig. 13 when an image is collected in the storage 40 or at a predetermined timing such as once a month. The determination system 1 may execute the following process for each point in a predetermined range.

[0072] The acquisition unit 101 acquires an image of a road (step S11). The recognition unit 102 recognizes the area of ​​the road and the area of ​​fixed objects in the image acquired by the acquisition unit 101 through image recognition (step S12). In step S12, the recognition unit 102 may recognize the area of ​​the lane in which the mobile object 11 is traveling through image recognition. The order of recognition is not particularly limited.

[0073] The depth estimation unit 103 estimates the depth of each part of the image acquired by the acquisition unit 101 through depth estimation (step S13). Note that step S13 may be executed after step S11 or before step S12.

[0074] The identification unit 104 identifies a determination target space in the image based on the position in the image of the road area recognized by the recognition unit 102 (step S14). In step S14, the identification unit 104 may identify a determination target space in the image based in particular on the position in the image of a driving lane area. Note that step S14 may be executed after step S12, or may be executed before step S13.

[0075] The detection unit 105 detects a target object area in the image representing the area of ​​the fixed object in the space to be determined based on the position in the image of the fixed object recognized by the recognition unit 102 and the depth estimated by the depth estimation unit 103 (step S15).

[0076] The determination unit 106 determines whether a fixed object is present in the determination target space based on at least one of the position and depth in the image of the target object region detected by the detection unit 105 (step S16).

[0077] 13. When the specification unit 104 specifies a plurality of determination target spaces, steps S14 to S16 may be repeated for each determination target space in descending order of height. When the specification is completed for the lowest determination target space, or when it is determined that no fixed object exists in the determination target space, the determination device 100 ends the repetition.

[0078] When a road continues in the depth direction of the image, trees also spread in the depth direction of the image. Therefore, it can be difficult to determine whether a tree is encroaching on the building limit. In addition, it can be difficult to recognize the depth position of other structures such as signs from the image, making it difficult to determine at what point they encroach on the building limit.

[0079] According to one embodiment, the identification unit 104 identifies a determination target space having a predetermined depth in the image based on the position of the road area in the image. Then, for the target object area detected by the detection unit 105, the determination unit 106 determines whether a fixed object is present in the determination target space. Therefore, according to one embodiment, it is possible to more accurately determine whether an object is invading the space on the road. This enables more accurate determination of objects that pose a risk of disrupting traffic.

[0080] According to one embodiment, whether a fixed object is present in the determination target space is determined from an image captured using a monocular camera of a drive recorder. Therefore, a laser scanner or a stereo camera does not need to be used. Therefore, it is possible to determine whether a fixed object is intruding into the space on the road using a simple device.

[0081] A laser scanner or a stereo camera may be mounted on the moving object 11. The detection unit 105 may detect the target object region by combining the depth estimation result with the depth measured by the laser scanner or the stereo camera.

[0082] Furthermore, when a laser scanner or a stereo camera is installed, the determination device 100 may include, as necessary, a depth estimation unit 103. For example, the detection unit 105 may detect the target object region using a depth measured from the detection result of the laser scanner or the stereo camera instead of the depth estimated by the depth estimation unit 103.

[0083] The camera 10 has been described as being mounted on the moving body 11. However, the camera 10 used in the determination system 1 may also be a camera fixed near a road.

[0084] [Hardware Configuration] In the above-described embodiment, each component of the determination device 100 represents a functional block. Some or all of the components of the determination device 100 may be realized by any combination of a computer 500 and a program. The determination device 100 may also be realized by combining multiple separate pieces of hardware.

[0085] Fig. 14 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 14, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input / output interface 511, and a bus 512.

[0086] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0087] The program 504 includes instructions for implementing each function of the determination device 100. The program 504 is stored in advance in the ROM 502, the RAM 503, or the storage device 505. The processor 501 implements each function of the determination device 100 by executing the instructions included in the program 504. The RAM 503 may also store data to be processed in each function of the determination device 100.

[0088] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or a keyboard, and receives information input from road administrators, etc. The output device 510 is, for example, a display, and outputs (displays) information to road administrators, etc. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0089] It should be noted that the hardware configuration shown in FIG. 14 is an example, and other components may be added, or some components may not be included.

[0090] There are various variations in the method of realizing the determination device 100. For example, the determination device 100 may be realized by any combination of a computer and a program, each of which is different for each component. Furthermore, multiple components included in the determination device 100 may be realized by any combination of a single computer and a program.

[0091] Furthermore, at least a part of the determination device 100 may be provided in a software as a service (SaaS) format. That is, at least a part of the functions for realizing the determination device 100 may be executed by software executed via a network.

[0092] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0093] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0094] [Supplementary Note 1] A determination system comprising: an acquisition means for acquiring an image of a road; a recognition means for recognizing a road area and a fixed object area in the image by image recognition; a depth estimation means for estimating the depth of each part of the image by depth estimation; an identification means for identifying a determination target space in the image based on the position of the road area in the image; a detection means for detecting a target object area in the image representing the fixed object area in the determination target space based on the position and depth of the fixed object area in the image; and a determination means for determining whether the fixed object is in the determination target space based on at least one of the position or the depth of the target object area in the image.

[0095] [Supplementary Note 2] The determination system according to Supplementary Note 1, wherein the detection means detects, as the target object region, a region of the fixed object whose position in the image overlaps with a position in the determination target space and whose depth is included in the depth range of the determination target space.

[0096] [Supplementary Note 3] The determination system according to Supplementary Note 2, wherein the detection means detects, as the target object region, a region of the fixed object whose position in the image is in a region that represents an upper boundary of the determination target space.

[0097] [Supplementary Note 4] The determination system according to any one of Supplementary Notes 1 to 3, wherein the determination means determines whether the fixed object is present in the determination target space by using a position of a geometric center of the target object region in the image.

[0098] [Supplementary Note 5] The determination system according to Supplementary Note 4, wherein the determination means determines that the fixed object is in the determination target space when the vertical position of the geometric center in the image is lower than a predetermined threshold value related to height.

[0099] [Supplementary Note 6] The determination system according to any one of Supplementary Notes 1 to 5, wherein the determination means determines whether the fixed object is present in the determination target space by using an average of depth values ​​of the target object region.

[0100] [Supplementary Note 7] The determination system according to Supplementary Note 6, wherein the determination means determines that the fixed object is present in the determination target space when the average of the depth values ​​is greater than a predetermined threshold value related to depth.

[0101] [Supplementary Note 8] The determination system described in any one of Supplementary Notes 1 to 7, wherein the identification means identifies a plurality of determination target spaces of different heights in the image, and the determination means determines the intrusion degree of the fixed object based on which of the plurality of determination target spaces the fixed object has intruded into.

[0102] [Supplementary Note 9] The determination system according to any one of Supplementary Notes 1 to 8, wherein the specifying means further specifies the determination target space in the image based on an actual width of the road.

[0103] [Supplementary Note 10] The determination system according to any one of Supplementary Notes 1 to 8, wherein the specifying means specifies a height position of the determination target space in the image based on a position at which an object of a predetermined height appears in the image.

[0104] [Supplementary Note 11] The determination system described in Supplementary Note 9, wherein the specifying means specifies the height position of the determination target space in the image based on data on the actual width of the lane in which the moving body equipped with the camera that captured the image is traveling, using the widths of both ends of the lane in the image as a reference.

[0105] [Supplementary Note 12] A determination method comprising: acquiring an image of a road; recognizing a road area and a fixed object area in the image by image recognition; estimating the depth of each part of the image by depth estimation; identifying a determination target space in the image based on the position of the road area in the image; detecting a target object area in the image that represents the fixed object area in the determination target space based on the position and depth of the fixed object area in the image; and determining whether the fixed object is in the determination target space based on at least one of the position or the depth of the target object area in the image.

[0106] [Supplementary Note 13] The determination method according to Supplementary Note 12, wherein a region of the fixed object whose position in the image overlaps with a position in the determination target space and whose depth is included in the depth range of the determination target space is detected as the target object region.

[0107] [Supplementary Note 14] The determination method according to Supplementary Note 13, wherein a region of the fixed object whose position in the image is in a region representing an upper boundary of the determination target space is detected as the target object region.

[0108] [Supplementary Note 15] The determination method according to any one of Supplementary Notes 12 to 14, wherein it is determined whether the fixed object is in the determination target space using the position of the geometric center of the target object region in the image.

[0109] [Supplementary Note 16] The determination method according to claim 15, wherein the fixed object is determined to be in the determination target space when the vertical position of the geometric center in the image is lower than a predetermined threshold value related to height.

[0110] [Supplementary Note 17] The determination method according to any one of Supplementary Notes 12 to 16, wherein an average of depth values ​​of the target object region is used to determine whether the fixed object is present in the determination target space.

[0111] [Supplementary Note 18] The determination method according to Supplementary Note 17, wherein it is determined that the fixed object is present in the determination target space when the average of the depth values ​​is greater than a predetermined depth threshold.

[0112] [Supplementary Note 19] The determination method according to any one of Supplementary Notes 12 to 18, wherein a plurality of determination target spaces having different heights are identified in the image, and the intrusion degree of the fixed object is determined based on which of the plurality of determination target spaces the fixed object has intruded into.

[0113] [Supplementary Note 20] The determination method according to any one of Supplementary Notes 12 to 19, further comprising identifying the determination target space in the image based on an actual width of the road.

[0114] [Supplementary Note 21] The determination method according to any one of Supplementary Notes 12 to 19, wherein a height position of the determination target space is identified in the image based on a position at which an object of a predetermined height appears in the image.

[0115] [Supplementary Note 22] The determination method according to Supplementary Note 20, wherein the height position of the determination target space is determined in the image based on data on the actual width of the lane in which the moving body equipped with the camera that captured the image is traveling, using the widths of both ends of the lane in the image as a reference.

[0116] [Supplementary Note 23] A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquiring an image of a road; recognizing a road area and a fixed object area in the image by image recognition; estimating the depth of each part of the image by depth estimation; identifying a determination target space in the image based on the position of the road area in the image; detecting a target object area in the image that represents the fixed object area in the determination target space based on the position and depth of the fixed object area in the image; and determining whether the fixed object is in the determination target space based on at least one of the position or the depth of the target object area in the image.

[0117] [Supplementary Note 24] A recording medium that non-temporarily records the program described in Supplementary Note 23, wherein a region of the fixed object whose position in the image overlaps with a position in the determination target space and whose depth is included in the depth range of the determination target space is detected as the target object region.

[0118] [Supplementary Note 25] A recording medium non-temporarily recording the program according to Supplementary Note 24, wherein a region of the fixed object whose position in the image is in a region representing an upper boundary of the determination target space is detected as the target object region.

[0119] [Supplementary Note 26] A recording medium non-temporarily recording the program according to any one of Supplementary Notes 23 to 25, wherein the program determines whether the fixed object is in the determination target space using the position of the geometric center in the image of the target object region.

[0120] [Supplementary Note 27] A recording medium for non-temporarily recording the program according to claim 26, wherein the fixed object is determined to be in the determination target space when the vertical position of the geometric center in the image is lower than a predetermined threshold value related to height.

[0121] [Supplementary Note 28] A recording medium non-temporarily recording the program according to any one of Supplementary Notes 23 to 27, wherein an average of the depth values ​​of the target object region is used to determine whether the fixed object is present in the determination target space.

[0122] [Supplementary Note 29] A recording medium non-temporarily recording the program according to Supplementary Note 28, wherein the program determines that the fixed object is in the determination target space when the average of the depth values ​​is greater than a predetermined threshold value related to depth.

[0123] [Supplementary Note 30] A recording medium non-temporarily recording a program according to any one of Supplementary Notes 23 to 29, which identifies a plurality of determination target spaces of different heights in the image, and determines the intrusion degree of the fixed object based on which of the plurality of determination target spaces the fixed object has intruded into.

[0124] [Supplementary Note 31] A recording medium non-temporarily recording the program according to any one of Supplementary Notes 23 to 30, further comprising identifying the determination target space in the image based on an actual width of the road.

[0125] [Supplementary Note 32] A recording medium non-temporarily recording the program according to any one of Supplementary Notes 23 to 30, wherein a height position of the determination target space is identified in the image based on a position at which an object of a predetermined height is reflected in the image.

[0126] [Supplementary Note 33] A recording medium that non-temporarily records the program described in Supplementary Note 31, in which a height position of the determination target space is identified in the image based on data on the actual width of the driving lane in which the mobile body equipped with the camera that captured the image is traveling, using the widths of both ends of the driving lane in the image as a reference.

[0127] REFERENCE SIGNS LIST 1 Determination system 100 Determination device 101 Acquisition unit 102 Recognition unit 103 Depth estimation unit 104 Identification unit 105 Detection unit 106 Determination unit 10 Camera 11 Mobile object 20 Administrator terminal 30 Communication network 40 Storage 50 Database

Claims

1. A means of acquiring images of roads, A recognition means that recognizes the road area and the fixed object area in the image using image recognition, Depth estimation means for estimating the depth of each part of the image by depth estimation, A means for identifying a space to be determined in the image based on the position of the road area in the image, A detection means for detecting a target object region in the image that represents the region of the fixed object in the determination target space, based on the position and depth of the region of the fixed object in the image, A determination means for determining whether the fixed object is in the determination target space based on at least one of the position in the image of the target object region or the depth. A judgment system equipped with the following features.

2. The detection means detects the region of the fixed object as the target object region, where the position in the image overlaps with the position in the target space and the depth falls within the depth range of the target space. The determination system according to claim 1.

3. The detection means detects the region of the fixed object whose position in the image is in the region representing the upper boundary of the space to be determined as the target object region. The determination system according to claim 2.

4. The determination means determines whether the fixed object is in the target space using the position of the geometric center in the image of the target object region. The determination system according to claim 1.

5. The determination means determines that the fixed object is in the target space if the vertical position of the geometric center in the image is lower than a predetermined threshold for height. The determination system according to claim 4.

6. The determination means determines whether the fixed object is in the target space by using the average of the depth values ​​of the target object region. The determination system according to any one of claims 1 to 5.

7. The determination means determines that the fixed object is in the target space if the average of the depth values ​​is greater than a predetermined threshold for depth. The determination system according to claim 6.

8. The aforementioned identification means identifies a plurality of determination target spaces with different heights in the image, The determination means determines the degree of intrusion of the fixed object based on which of the plurality of spaces to be determined the fixed object has intruded into. The determination system according to any one of claims 1 to 5.

9. We obtained images of the road, Image recognition is used to distinguish between the road area and the fixed object area in the image. Depth estimation is used to estimate the depth of each part of the image. Based on the location of the road area in the aforementioned image, the space to be determined is identified in the aforementioned image. Based on the position and depth of the area of ​​the fixed object in the image, a target object area representing the area of ​​the fixed object in the target space is detected in the image. Based on the position of the target object region in the image or the depth, it is determined whether the fixed object is in the target space. Judgment method.

10. We obtained images of the road, Image recognition is used to distinguish between the road area and the fixed object area in the image. Depth estimation is used to estimate the depth of each part of the image. Based on the location of the road area in the aforementioned image, the space to be determined is identified in the aforementioned image. Based on the position and depth of the area of ​​the fixed object in the image, a target object area representing the area of ​​the fixed object in the target space is detected in the image. Based on the position of the target object region in the image or the depth, it is determined whether the fixed object is in the target space. A program that instructs a computer to perform a process.