Labeled diagnostic system, labeled diagnostic method, and labeled diagnostic program

Abstract

Provided is a sign diagnosis system comprising an acquisition unit, an identification unit, an estimation unit, a selection unit, and an output unit. In this invention, the acquisition unit acquires time-series images of a road taken by an image-taking device installed on a vehicle. The identification unit identifies the sizes of signs appearing in the time-series images. The estimation unit estimates, on the basis of the size of a sign at a first time point of a series identified by the identification unit, the size of the sign at a second time point later than the first time point in the time series. The selection unit selects an image to use for confirmation of a state in which the sign is covered by an object, on the basis of the size of the sign at the second time point having been identified and the size of the sign at the second time point having been estimated. The output unit outputs the image selected by the selection unit.

Description

[Technical Field]

[0001] The present invention relates to a sign diagnostic system and the like. [Background technology]

[0002] In order for pedestrians to travel safely on roads, signs installed on roads must be kept visible to pedestrians. However, signs installed on roads may be obscured by trees, for example, and may become invisible to pedestrians. For this reason, road administrators, for example, monitor whether signs installed on roads under their management are visible. If the monitoring results show that a sign is obscured by trees, the road administrators, for example, prune the trees to ensure that pedestrians can see the sign. Furthermore, road administrators may use image recognition technology to determine whether signs are visible based on images taken by vehicles traveling on roads under their management.

[0003] In the sign recognition method of Patent Document 1, an image of a sign is synthesized based on two images of signs taken at different times, and the sign recognition method of Patent Document 1 then recognizes the sign based on the synthesized image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-140828 Summary of the Invention [Problem to be solved by the invention]

[0005] With the technology described in Patent Document 1, it is not always possible to select an image suitable for checking the sign's covering state, so it may be difficult to check the sign's covering state with an object.

[0006] In order to solve the above-mentioned problems, the present disclosure aims to provide a sign diagnosis system etc. that can easily check the state of coverage of a sign by an object. [Means for solving the problem]

[0007] In order to solve the above problems, the sign diagnosis system of the present disclosure includes an acquisition means for acquiring time-series images of a road taken by an imaging device mounted on a vehicle, an identification means for identifying the size of signs shown in the time-series images, an estimation means for estimating the size of the sign at a second time point in the time series that is later than the first time point based on the size of the sign at a first time point in the time series identified by the identification means, a selection means for selecting an image to be used to check the state of coverage of the sign by an object based on the size of the sign at the second time point identified by the identification means and the size of the sign at the second time point estimated by the estimation means, and an output of the image selected by the selection means.

[0008] The sign diagnosis method disclosed herein acquires a time series of images of a road taken by an imaging device mounted on a vehicle, identifies the size of signs shown in the time series images, estimates the size of the sign at a second time point in the time series that is later than the first time point based on the identified size of the sign at a first time point in the time series, selects an image to be used to confirm the state of coverage of the sign by an object based on the identified size of the sign at the second time point and the estimated size of the sign at the second time point, and outputs the selected image.

[0009] The recording medium of the present disclosure non-temporarily records a sign diagnosis program that causes a computer to execute the following processes: acquiring time-series images of a road taken by an imaging device mounted on a vehicle; identifying the size of signs shown in the time-series images; estimating the size of the sign at a second time point in the time series that is later than the first time point based on the identified size of the sign at a first time point in the time series; selecting images to be used to check the state of coverage of the sign by an object based on the identified size of the sign at the second time point and the estimated size of the sign at the second time point; and outputting the selected images. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to easily check the state of coverage of a marker by an object. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram schematically illustrating an example of capturing an image of a road as a vehicle travels; [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a sign diagnosis system according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram illustrating an example of an image of a road according to an embodiment of the present disclosure. [Figure 5] FIG. 2 is a diagram illustrating an example of an image of a road according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating an example of a graph showing the relationship between the timing at which an image is captured and the size of a sign according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram illustrating an example of an image of a road according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram illustrating an example of an image of a road according to an embodiment of the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating an example of a graph showing the relationship between the timing at which an image is captured and the size of a sign according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of a display screen according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating an example of a display screen according to an embodiment of the present disclosure. [Figure 12] FIG. 1 is a diagram illustrating an example of an operation flow of a sign diagnosis system according to an embodiment of the present disclosure. [Figure 13] FIG. 1 is a diagram illustrating an example of a hardware configuration of a sign diagnosis system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present disclosure will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating an example of the configuration of a sign management system. The sign management system includes a sign diagnosis system 10, an on-board device 20, and a terminal device 30. The sign diagnosis system 10 is connected to the on-board device 20 via a network, for example. Data input / output between the sign diagnosis system 10 and the on-board device 20 may be performed via a recording medium. For example, data input / output between the sign diagnosis system 10 and the on-board device 20 may be performed via a recording medium using a non-volatile semiconductor memory element. The sign diagnosis system 10 is also connected to the terminal device 30 via the network. There may be multiple on-board devices 20 and multiple terminal devices 30. The number of on-board devices 20 and the number of terminal devices 30 are set as appropriate.

[0013] A sign management system, for example, is a system that manages whether signs installed on roads are visible to pedestrians. The sign management system, for example, monitors the coverage of signs by objects. A sign being covered by an object means, for example, that an object exists between a pedestrian and the sign, causing the sign to be covered by the object as seen by the pedestrian. In other words, an object covering a sign means, for example, that an object exists between a pedestrian and the sign, causing the pedestrian to be unable to see the display surface of the sign. The coverage of a sign by an object also includes, for example, whether or not the sign is covered by an object and the degree of coverage. The sign management system, for example, monitors the coverage of signs by objects based on continuously captured images of the road. The continuously captured images of the road are, for example, video images of the road captured by a camera mounted on a vehicle traveling on the road.

[0014] A sign is an object installed to provide information to pedestrians on a road. Signs include road signs and signs other than road signs. Road signs are display boards installed to provide pedestrians with information necessary for traveling along a road. Road signs, for example, provide pedestrians with information regarding guidance, warnings, regulations, and instructions. Road signs, for example, provide pedestrians with information using text or symbols specified by law. The information provided by road signs is not limited to the above. Signs other than road signs may include, for example, display devices installed to provide information necessary for traveling along a road, such as traffic lights and convex mirrors. Signs other than road signs may also include, for example, billboards, three-dimensional objects, and displays installed for advertising or other purposes. An object covering a sign may be, for example, a tree. Trees may also include grass and bamboo. An object covering a sign may be a flagpole installed on the side of the road. An object covering a sign is not limited to the above.

[0015] The sign diagnosis system 10 determines the state of sign coverage by an object based on, for example, images captured by a camera mounted on a vehicle. The on-board device 20 is, for example, a device that captures images using a camera that the sign diagnosis system 10 uses to determine the sign coverage state. The terminal device 30 is, for example, a terminal used by a road administrator to operate the sign diagnosis system 10 and check the sign coverage state. The road administrator is, for example, an entity that manages roads to ensure safe passage. The road administrator, for example, monitors and maintains road-related facilities. The road administrator, for example, refers to the results of the determination to determine whether or not work is needed to remove the object covering the sign.

[0016] FIG. 2 is a diagram schematically illustrating an example of monitoring the coverage state of road signs based on images captured by a camera mounted on a vehicle. In the example of FIG. 2, a vehicle is traveling on a road. Also, in the example of FIG. 2, signs indicating speed limits are installed on the road. In the example of FIG. 2, an on-board device 20 is mounted on the vehicle. The on-board device 20, for example, uses a camera to capture an image of the road that shows signs installed on the road. The on-board device 20 measures information related to the speed of the vehicle. The information related to the speed of the vehicle is, for example, the vehicle's speed. The information related to the speed of the vehicle may include the acceleration of the vehicle. The on-board device 20 outputs the captured image and the information related to the speed to, for example, the sign diagnosis system 10. For example, a drive recorder is used as the on-board device 20. The on-board device 20 may be something other than a drive recorder.

[0017] The sign diagnosis system 10 determines the sign coverage state by an object based on an image acquired from the in-vehicle device 20. The sign diagnosis system 10 then outputs the result of the determination of the sign coverage state to, for example, the terminal device 30. The terminal device 30 outputs the result of the determination of the sign coverage state to, for example, a display device. The road administrator, for example, refers to the result of the determination of the sign coverage state and determines whether or not the object covering the sign needs to be removed. If the object is a tree, removal of the object may involve, for example, pruning the tree branches, cutting the tree trunk, or transplanting the tree.

[0018] The sign diagnosis system 10 estimates the size of a sign at a second time point, which is later in the time series than the first time point, based on the size of the sign at a first time point in a series of consecutively captured images of the same sign. When capturing images of a forward direction relative to the vehicle's traveling direction, the image at the second time point is captured at a location closer to the sign than the image at the first time point. That is, the sign diagnosis system 10 estimates the size of the sign as it would be if it were captured at a location closer to the sign, based on the size of the sign captured in an image captured at a location farther from the sign. The sign diagnosis system 10 then selects an image to use to confirm the sign's coverage status based on the size of the sign identified from the image captured at the second time point and the size of the sign at the second time point estimated from the image captured at the first time point. The size of the sign captured in the image is the apparent size of the sign in the image. The size of the sign is expressed, for example, using the number of pixels in the image. The size of the sign identified from the image refers to the size of the portion of the image that is recognized as a sign. For example, when a sign is covered by an object, the size of the sign identified from the image is the size of the portion of the sign visible in the image. In other words, the size of the portion of the area where the sign actually exists that is covered by the object is not included in the size of the sign identified from the image. Therefore, when a sign is covered by an object, the size of the sign identified from the image will be smaller than the size of the sign when it is not covered.

[0019] If a sign is not obscured by an object at both the first and second time points, the size of the sign identified from the image captured at the second time point will match the size of the sign at the second time point estimated from the image captured at the first time point. On the other hand, if a sign is obscured by an object at the first time point, only the unobscured portion of the sign is captured in the image, and the size of the sign identified from the image will be smaller than the sign's actual size in the image. The actual size of the sign in the image is the size of the sign in the image when it is not obscured. Furthermore, when photographing from a moving vehicle, the relative positional relationship between the photographing device and the object changes as the vehicle moves forward. Even if a sign is obscured by an object at the first time point, when the sign is far from the sign, it may be possible to photograph the entire sign at a point closer to the sign without being affected by the object. In such cases, the size of the sign at the second time point estimated from the image captured at the first time point will be smaller than the actual size at the second time point. Therefore, for example, if the identification value of the size of a sign identified in an image taken at a second time point is greater than the estimated value of the size of the sign estimated from an image taken at a first time point, it can be determined that the sign was covered by an object at the corresponding point at the first time point.

[0020] The confirmation of the sign's coverage by an object must be based on whether a pedestrian on the road can recognize the information indicated by the sign at an appropriate position when looking at the sign. However, when the confirmation of the sign's coverage by an object is based on continuously captured images of a road, some of the images may not be suitable for confirming the sign's coverage by an object. For example, if a sign is covered by a tree, the sign may not be visible at a point where pedestrians need to take action based on the sign's information, but may be visible near the sign's installation location. For example, a "Stop" sign must be visible from a distance where the vehicle can safely brake. In such a case, confirming the coverage by an image captured near the sign may result in an inappropriate determination of the sign's coverage. Meanwhile, the sign diagnosis system 10 selects images to use for confirming the sign's coverage by an object based on the size of the sign identified from an image captured at a second time point and the size of the sign at the second time point estimated from an image captured at a first time point. By selecting images in this manner, the sign diagnosis system 10 can select an image taken at a point near the sign where the sign is obscured by an object and part of the sign is not visible, rather than an image taken at a point near the sign where the entire sign is visible.

[0021] Here, the configuration of the sign diagnosis system 10 will be described. Fig. 3 is a diagram showing an example of the configuration of the sign diagnosis system 10. The sign diagnosis system 10 basically includes an acquisition unit 11, a recognition unit 12, an estimation unit 13, a selection unit 14, and an output unit 16. The sign diagnosis system 10 also includes, for example, a calculation unit 15 and a storage unit 17.

[0022] The acquisition unit 11 acquires time-series images of a road captured by a camera mounted on a vehicle. The acquisition unit 11 acquires, for example, time-series images of a road and information related to the speed of a vehicle equipped with a camera that captured the images. The acquisition unit 11 acquires, for example, time-series images of a road from an in-vehicle device 20. The time-series images of a road are captured by, for example, a camera mounted on a vehicle. The time-series images of a road are, for example, videos captured by a drive recorder. The acquisition unit 11 also acquires, for example, information related to the speed of a vehicle at the time the road images were captured from the in-vehicle device 20 of the vehicle that captured the road images. The acquisition unit 11 may acquire, from the in-vehicle device 20 of the vehicle that captured the road images, the time-series images of a road and information related to the vehicle speed via a recording medium. For example, a recording medium including a non-volatile semiconductor memory element is used as the recording medium. The recording medium is not limited to the above. The acquisition unit 11 may also acquire time-series images of roads and information related to vehicle speeds from a server connected to the network.

[0023] The time-series images of a road are time-series images showing signs installed on the road. The time-series images of a road are images of a road continuously photographed with signs showing. That is, the time-series images of a road are multiple frames showing the same sign photographed from different positions on the road. The information about the vehicle speed is, for example, the vehicle speed. The information about the vehicle speed may be the vehicle speed and acceleration. The information about the vehicle speed is not limited to the above. The acquisition unit 11 may also acquire the images of the road and the information about the vehicle speed via a server connected via a network. The acquisition unit 11 may further acquire information about the vehicle's position from the in-vehicle device 20. The acquisition unit 11 may also acquire information about the vehicle's position over time from the in-vehicle device 20. The information about the vehicle's position over time acquired by the acquisition unit 11 is used, for example, to calculate the vehicle's speed.

[0024] The imaging device mounted on the vehicle captures an image of the road so that signs installed on the road are captured in the image. Signs installed on the road are signs visible to pedestrians on the road. Signs installed on the road are, for example, signs installed on the shoulder of the road, above the driving lanes, and in the center divider. Signs installed on the road may also include signs installed outside the road premises. Furthermore, a pedestrian on the road is, for example, a driver of a vehicle traveling on the road. A pedestrian on the road may be, for example, a passenger other than the driver of a vehicle traveling on the road. A pedestrian on the road may also be a pedestrian. A sign visible to pedestrians on the road may be a sign identified by an imaging device or a sensor for a driving assistance system.

[0025] The identification unit 12 identifies the size of signs that appear in the time-series images. The identification unit 12 identifies the signs that appear in the images, for example, using an image recognition model. The image recognition model, for example, identifies the signs that appear in the images and the shapes of the signs. The identification unit 12 identifies the size of the parts of the images that it has identified as signs. The identification unit 12 does not identify parts of the images where an object appears in front of the sign, even if the area contains a sign, as the size of the sign. The size of the sign is, for example, the area of ​​the sign that appears in the image. The size of the sign is, for example, the area of ​​the sign that appears in the image. The size of the sign is expressed, for example, by the number of pixels in the part of the image where the sign appears.

[0026] The image recognition model is, for example, a machine learning model using a neural network. The image recognition model is generated by learning the relationship between an image of a sign and the display content and shape of the sign. The image recognition model is generated, for example, in a system external to the sign diagnosis system 10.

[0027] When multiple signs appear simultaneously in a single frame of image, the identification unit 12 distinguishes and identifies each sign based on, for example, at least one of the position, shape, and design of the sign in the image. Furthermore, when a sign appears in each of multiple time-series frame images, the identification unit 12 may identify each sign as a different sign if a sign disappears from the image for a set number of consecutive frames before the next sign appears in the image. For example, when multiple frames in which a sign appears are followed by one frame in which no sign appears, and then multiple frames in which a sign appears, the identification unit 12 identifies the signs appearing in the previous and following frames as the same sign. The set value for the number of frames in which no sign appears when considering the signs appearing in the previous and following frames as the same sign is set, for example, based on the vehicle speed and the frame rate of the time-series images of the road. Furthermore, when multiple signs appear in the time-series images, the identification unit 12 may identify each sign based on vehicle position information acquired by the acquisition unit 11. The information on the position of each sign is set by, for example, a road administrator. The identification unit 12 may identify each sign based on the information on the vehicle position acquired by the acquisition unit 11, the vehicle speed, and the vehicle acceleration.

[0028] The estimation unit 13 estimates the size of a sign at a second time point in the time series that is later than the first time point, based on information about the size and speed of the sign at a first time point in the time series identified by the identification unit 12. The estimation unit 13, for example, estimates the distance between the position of the vehicle at the first time point and the position of the vehicle at the second time point. The estimation unit 13 estimates the distance between the position of the vehicle at the first time point and the position of the vehicle at the second time point, for example, using the speed of a vehicle equipped with an imaging device that captured the image. Then, the estimation unit 13 estimates the size of the sign at the second time point based on the size of the sign at the first time point and the distance between the positions of the vehicle at the first and second time points.

[0029] The estimation unit 13 estimates the size of a sign at a second time point based on, for example, the shape and size of a sign identified by the identification unit 12 at a first time point. For example, if the shape of a sign identified by the identification unit 12 is a circle, the estimation unit 13 assumes that the shape is a circle with the identified size and estimates the size of the sign at a second time point. The estimation unit 13 estimates the distance between the vehicle's positions at the first and second time points based on, for example, the vehicle's speed. Then, the estimation unit 13 estimates the size of the sign at a second time point based on the size of the sign at the first time point and the distance between the vehicle's positions at the first and second time points. The relationship between the distance between the image capture points and the amount of change in sign size is set based on, for example, the characteristics of the image capture device and the number of pixels in the image. The estimation unit 13 estimates the size of the sign at a second time point based on information about the speed of the vehicle equipped with the image capture device. The estimation unit 13 may also calculate the speed of the vehicle used to estimate the size of the sign at the second time point from the time-series positions of the vehicle on which the image capturing device is mounted.

[0030] When the image is captured by a camera that captures the area ahead of the vehicle in the vehicle's traveling direction, the estimation unit 13 estimates, for example, the time when the image with the smallest sign size among images in which the sign size is equal to or larger than a standard was captured as the first time point. That is, the estimation unit 13 estimates, for example, the time when the image with the smallest sign size among images in which the sign size is equal to or larger than a standard was captured at a position farthest from the sign as the first time point. Furthermore, "forward" in the vehicle's traveling direction means, for example, the area ahead as seen by the driver sitting in the vehicle's seat.

[0031] When the image is captured by a camera that captures the rear of the vehicle in the opposite direction to the vehicle's traveling direction, the estimation unit 13 estimates, as the first time point, the time when the image with the largest sign size was captured among images in which the sign size is equal to or larger than a standard. That is, the estimation unit 13 estimates, as the first time point, the time when the image with the largest sign size was captured at a position farthest from the sign among images in which the sign size is equal to or larger than a standard. Furthermore, "rear" with respect to the traveling direction of the vehicle means, for example, the back side as seen from the driver sitting in the seat of the vehicle.

[0032] The selection unit 14 selects an image to be used to check the state of coverage of the sign by the object, based on the size of the sign at the second time point identified by the identification unit 12 and the size of the sign at the second time point estimated by the estimation unit 13. The selection unit 14 selects an image to be used to check the state of coverage of the sign by the object, for example, by comparing the size of the sign at the second time point identified by the identification unit 12 with the size of the sign at the second time point estimated by the estimation unit 13.

[0033] The selection unit 14 selects an image to be used to check the sign coverage state by an object, for example, based on which is larger: the size of the sign identified by the identification unit 12 at the second time point; or the size of the sign estimated by the estimation unit 13 at the second time point. When the difference between the size of the sign identified by the identification unit 12 and the size of the sign estimated by the estimation unit 13 increases over time in the time-series images, the selection unit 14 selects, as the image to be used to check the sign coverage state, an image in which the position of the sign is farthest from the position of the vehicle at the time of shooting, from among the images in which the size of the sign meets a criterion. When the difference between the size of the sign identified by the identification unit 12 at the second time point and the size of the sign estimated by the estimation unit 13 increases over time, the selection unit 14 selects, as the image to be used to check the sign coverage state, an image in which the position of the sign is farthest from the position of the vehicle at the time of shooting, from among the images in which the size of the sign meets a criterion. For example, the selection unit 14 selects, as an image to be used for checking the sign coverage status, an image in which the position of the vehicle at the time of shooting is far from the position of the sign, from among images in which the size of the sign is equal to or larger than a standard. In other words, the selection unit 14 selects, as an image to be used for checking the sign coverage status, an image that was shot at a point where the size of the sign satisfies the appropriate size for visibility and where a distance from the sign sufficient for a passerby to take action is secured.

[0034] When the photographing device is photographing the area ahead in the direction of travel of the vehicle, if the size of the sign at the second time point identified by the identification unit 12 is larger than the size of the sign at the second time point estimated by the estimation unit 13, the selection unit 14 selects the image at the first time point as the image to be used to confirm the state of coverage of the sign by the object.

[0035] 4 and 5 show examples of images captured by a camera that captures the area ahead in the direction of travel of a vehicle. In the example images of FIGS. 4 and 5, a road is captured in the center of the image. In addition, in the example images of FIGS. 4 and 5, a sign is captured on the left side of the road. In addition, in the example images of FIGS. 4 and 5, a tree is captured on the left side of the sign.

[0036] The example image in FIG. 4 is the image in the Nth frame (N is a positive integer) of consecutively captured images. The example image in FIG. 5 is the image in the N+1th frame of consecutively captured images. That is, the example image in FIG. 5 is the image of the frame next to the example image in FIG. 4 in the time series. The example image in FIG. 4 corresponds to, for example, an image captured at a first time point. The example image in FIG. 5 corresponds to an image captured at a second time point.

[0037] In the example image of FIG. 4, a portion of the sign is obscured by a tree. On the other hand, in the example image of FIG. 5, the tree and the sign do not overlap. In addition, in the example image of FIG. 5, the dotted circle inside the sign indicates the size of the sign at the time the image of FIG. 5 was captured, estimated from the size of the sign in the example image of FIG. 4. Here, the size of the sign in the example image of FIG. 4 is the size of the portion of the image that is not obscured by a tree. In the example image of FIG. 4, because the sign is obscured by a tree, the size of the sign identified by the identification unit 12 is smaller than the actual size of the sign in the image. Therefore, in the example image of FIG. 5, the size of the sign identified by the identification unit 12 is larger than the size estimated from the size of the sign at the first time point. Therefore, in a state such as the example display screens of FIGS. 4 and 5, the selection unit 14 determines that the sign was obscured by a tree in the Nth frame image. That is, in a state such as the example display screens of FIGS. 4 and 5, the selection unit 14 selects the image captured at the first time point as the image to be used to confirm the sign's coverage state.

[0038] FIG. 6 is a diagram illustrating an example of a graph showing the relationship between the timing at which an image is captured and the size of a sign when an image is captured by an imaging device that captures an image ahead in the direction of vehicle travel. The example of the graph in FIG. 6 is an example of a graph comparing the size of a sign in frames N+1 and onward, estimated from the size of the sign in the Nth frame, with the size of a sign identified from the image. In the example of the graph in FIG. 6, the "estimated value" indicated by the dashed line is the size of a sign in frames N+1 and onward, estimated from the size of the sign in the Nth frame. In addition, in the example of the graph in FIG. 6, the "actual value" indicated by the solid line is the size of a sign identified from the image. The size of a sign identified from the image is the size of the portion of the sign that appears in the image. The "frame number" on the horizontal axis of the example of the graph in FIG. 6 indicates the timing at which the image was captured. In the example of the graph in FIG. 6, the "frame number" indicates the order of the frames in chronological order. The "number of sign pixels" on the vertical axis of the example of the graph in FIG. 6 indicates the number of pixels in the portion of the image that is recognized as a sign. That is, the "number of sign pixels" is a value that indicates the size of a sign using the number of pixels in the part of the image where the sign appears. Therefore, the vertical axis of the example graph in Figure 6 indicates the size of the sign appearing in the image.

[0039] In the example graph of Fig. 6, the sizes of signs in the N+1, N+2, and N+3 frames are estimated based on the size of the sign identified from the Nth frame image. In the example graph of Fig. 6, the difference between the identified value and the estimated value increases in later images in the time series. In the example graph of Fig. 6, the overlapping portion between the tree and the sign becomes smaller in later frames. For this reason, in the example graph of Fig. 6, the selection unit 14 selects the Nth frame as an image suitable for checking the coverage state.

[0040] When the photographing device is photographing the rear of the vehicle in the direction of travel, if the size of the sign at the second time point identified by the identification unit 12 is smaller than the size of the sign at the second time point estimated by the estimation unit 13, the selection unit 14 selects the image at the second time point as the image to be used to confirm the state of coverage of the sign by the object.

[0041] 7 and 8 show examples of images captured by a camera that captures images behind the vehicle's traveling direction. In the example images of FIGS. 7 and 8, the vehicle is traveling in the opposite direction to the direction in which the camera is capturing the images. In the example images of FIGS. 7 and 8, a road is captured in the center of the image. In addition, in the example images of FIGS. 7 and 8, a sign is captured on the left side of the road. In addition, in the example images of FIGS. 7 and 8, a tree is captured on the left side of the sign.

[0042] The example image in Fig. 7 is an image in frame N of images captured continuously. The example image in Fig. 8 is an image in frame N+1 of images captured continuously, i.e., the example image in Fig. 8 is the image of the frame next to the example image in Fig. 7 in the time series. The example image in Fig. 7 corresponds to an image captured at a first time point, for example. The example image in Fig. 8 corresponds to an image captured at a second time point.

[0043] In the example image of FIG. 7, the trees and the sign do not overlap. On the other hand, in the example image of FIG. 8, the sign is partially obscured by the trees. In addition, in the example image of FIG. 8, the dotted circle around the sign indicates the size of the sign at the time the example image of FIG. 8 was captured, estimated from the size of the sign in the example image of FIG. 7. In the example image of FIG. 7, the sign is not obscured by trees, so the estimated size of the circle matches the outer diameter of the sign in the image. However, the size of the sign identified by the identification unit 12 is smaller than the actual size of the sign in the image due to obscuration by trees. Therefore, in the example image of FIG. 8, the size of the sign identified by the identification unit 12 is smaller than the size estimated from the size of the sign at the first time point. Therefore, in the state of the example display screens of FIGS. 7 and 8, the selection unit 14 selects the image of the Nth frame as the image to be used to confirm whether the sign is obscured by an object. That is, in the state of the example display screens of FIGS. 7 and 8, the selection unit 14 selects the image captured at the second time point as the image to be used for checking the state of coverage of the sign by the object.

[0044] FIG. 9 is a diagram illustrating an example of a graph showing the relationship between the timing at which an image is captured and the size of a sign when an image is captured by an imaging device that captures an image ahead in the direction of vehicle travel. The example of the graph in FIG. 9 is an example of a graph comparing the size of a sign in frames N+1 and onward, estimated from the size of the sign in the Nth frame, with the size of a sign identified from the image. In the example of the graph in FIG. 9, the "estimated value" indicated by the dashed line is the size of a sign in frames N+1 and onward, estimated from the size of the sign in the Nth frame. In addition, in the example of the graph in FIG. 9, the "actual value" indicated by the solid line is the size of a sign identified from the image. The size of a sign identified from the image is the size of the portion of the sign that appears in the image. The "frame number" on the horizontal axis of the example of the graph in FIG. 9 indicates the timing at which the image was captured. In the example of the graph in FIG. 9, the "frame number" indicates the order of the frames in chronological order. The "number of sign pixels" on the vertical axis of the example of the graph in FIG. 9 indicates the number of pixels in the portion of the image that is recognized as a sign. That is, the "number of sign pixels" is a value that indicates, for example, the size of a sign using the number of pixels in the portion of the image where the sign appears. Therefore, the vertical axis of the example graph in FIG. 9 indicates the size of the sign. The "number of sign pixels" is a value that indicates, for example, the size of a sign using the number of pixels in the portion of the image where the sign appears.

[0045] In the example graph of FIG. 9, the sizes of signs in the N+1, N+2, and N+3 frames are estimated based on the size of the sign identified from the Nth frame image. In the example graph of FIG. 9, the difference between the value identified from the image and the estimated value increases in later chronological images. In the example graph of FIG. 9, the overlapping portion between the tree and the sign increases in later frames. For this reason, in the example graph of FIG. 9, the selection unit 14 determines that frames N+1 and later are suitable images for checking the coverage state. For example, the selection unit 14 selects, from among frames N+1 and later, an image in which the size of the sign is equal to or larger than the standard and has the smallest size as an image suitable for checking the coverage state.

[0046] If the difference between the size of the sign identified by the identification unit 12 at the second time point and the size of the sign estimated by the estimation unit 13 at the second time point is less than a criterion, the selection unit 14 selects either the image at the first time point or the image at the second time point as the image to be used to confirm the covering state of the sign. The criterion is set in advance. For example, the criterion is a criterion that defines a range within which the size of the sign identified by the identification unit 12 at the second time point and the size of the sign estimated by the estimation unit 13 at the second time point can be considered to be the same. The criterion is set, for example, based on an error that may occur when estimating the size of the sign at the second time point from the size of the sign identified from the image at the first time point and information related to the vehicle speed.

[0047] The selection unit 14 selects, for example, an image taken at a location farther from the sign from the image at the first time point and the image at the second time point as an image to be used to confirm the sign's covering state. When a sign is not covered by an object, the size of the sign at the second time point estimated based on the size of the sign identified at the first time point matches the size of the sign identified from the image captured at the second time point. Therefore, when the difference between the size of the sign at the second time point identified by the identification unit 12 and the size of the sign at the second time point estimated by the estimation unit 13 is smaller than a reference value, it can be determined that the sign is not covered by an object. In other words, when the difference between the size of the sign at the second time point identified by the identification unit 12 and the size of the sign at the second time point estimated by the estimation unit 13 is smaller than a reference value, the image selected by the selection unit 14 is an image showing a sign that is not covered by an object. When a sign is not covered by an object, if there are multiple corresponding frames, the selection unit 14 selects, for example, an image taken at a position farthest from the sign among images in which the size of the sign is equal to or larger than a standard, as the image to be used to check the covering state of the sign.

[0048] The calculation unit 15 calculates the coverage rate of the marker by the object, for example, based on an image selected as an image used to check the coverage state of the marker. The coverage rate of the marker is the ratio of the size of the portion covered by the object to the size of the entire marker. The calculation unit 15 calculates the size of the marker when it is not covered, for example, based on the dimensions of the marker. The calculation unit 15 calculates the size of the marker by calculating the dimensions of the marker based on the shape of the marker identified by the identification unit 12, for example. For example, when the identification unit 12 identifies a portion of the periphery of a circle, the calculation unit 15 calculates the diameter of the circle by regarding the identified portion as a circle whose periphery includes the circle. The calculation unit 15 then calculates the size of the circle based on the diameter of the circle. The calculation unit 15 calculates the coverage rate of the marker by the object based on the calculated size of the marker and the size of the marker identified by the identification unit 12. For example, the calculation unit 15 calculates the coverage rate of the marker by the object by dividing the size of the marker identified by the identification unit 12 by the size of the marker when it is not covered by the object.

[0049] The output unit 16 outputs an image used to check the state of coverage of a sign by an object. The output unit 16 outputs the image used to check the state of coverage of a sign by an object to, for example, the terminal device 30. The output unit 16 may also output the image used to check the state of coverage of a sign by an object to a display device (not shown) connected to the sign diagnosis system 10. The output unit 16 outputs, for example, an image used to check the state of coverage of a sign by an object for each sign. When a sign has been identified, the output unit 16 may output the sign's identifier or location information and the image used to check the state of coverage of a sign by an object. The output unit 16 may also output the image used to check the state of coverage of a sign by an object by superimposing it on a position on the map corresponding to the installation location of the sign. The output unit 16 may also output a thumbnail of the image used to check the state of coverage of a sign by an object at a position on the map corresponding to the installation location of the sign. Then, for example, when the thumbnail is clicked by a mouse operation, the output unit 16 outputs the image used to check the state of coverage of a sign by an object at the original resolution. Furthermore, the output unit 16 may output an image used to check the coverage state of a marker whose coverage rate with an object is equal to or greater than a standard.

[0050] The output unit 16 may output the coverage rate of the sign by the object in addition to the image used to check the coverage state of the sign by the object. Furthermore, the output unit 16 outputs the image used to check the coverage state of the sign by the object and the coverage rate of the sign by the object to, for example, the terminal device 30.

[0051] The output unit 16 may output an image of a sign installed at each of a plurality of locations to be used for checking the state of coverage of the sign by an object. The output unit 16 may also output an image of a sign installed at each of a plurality of locations and the coverage rate of the sign in each image. When the selection unit 14 selects an image in which the sign is not covered by an object as the image to be used for checking the state of coverage of the sign by an object, the output unit 16 may output the image in which the sign is not covered by an object as the image to be used for checking the state of coverage of the sign by an object.

[0052] FIG. 10 shows an example of a display screen that displays an image suitable for checking the coverage rate of signs by trees. In the example of the display screen in FIG. 10, an image of a sign installed at point S on road A and an image of a sign installed at point B are displayed. In the example of the display screen in FIG. 10, each of the signs shown in the image is covered by trees. In the example of the display screen in FIG. 10, the coverage rate of signs at point S is displayed as 25. In addition, in the example of the display screen in FIG. 10, the coverage rate of signs at point B is displayed as 38. By referring to a display screen such as the example in FIG. 10, a road administrator can, for example, determine which signs require tree pruning.

[0053] FIG. 11 shows an example of a display screen displaying images of the same sign at multiple points in time. The example of the display screen in FIG. 11 displays images of a sign at point S on road A that were taken at three different points in time. The example of the display screen in FIG. 11 displays three images: image O, image P, and image R. In the example of the display screen in FIG. 11, the images were taken in the order of image O, image P, and image R. That is, in the time series, image O was taken at the earliest time, and image R was taken at the latest time. In the example of the display screen in FIG. 11, the coverage rate of image O is displayed as 28. In the example of the display screen in FIG. 11, the coverage rate of image P is displayed as 25. Furthermore, in the example of the display screen in FIG. 11, the coverage rate of image R is displayed as 0. That is, in the example of the display screen in FIG. 11, the sign is not covered by trees in image R.

[0054] In the example display screen of FIG. 11 , the image selected by the selection unit 14 to be used for checking the state of coverage of a sign by an object is surrounded by a dotted frame. The image selected by the selection unit 14 to be used for checking the state of coverage of a sign by an object is, for example, the image that is most suitable for checking the state of coverage of a sign by an object. By referring to the display screen of the example of FIG. 11 , a road administrator can, for example, determine signs that require pruning of surrounding trees while checking the validity of the image selection by the sign diagnosis system 10. Furthermore, the example display screen of FIG. 11 may be displayed when an image is selected in the example display screen of FIG. 10 . For example, in the example display screen of FIG. 10 , when the image of point S is clicked by an operator using the mouse, the display screen shown in the example of FIG. 11 may be displayed.

[0055] The memory unit 17 stores, for example, data related to the process of selecting images to be used to check the state of coverage of a sign by an object. The memory unit 17 stores, for example, images of a road continuously photographed. The memory unit 17 stores, for example, data related to the speed of a vehicle. The memory unit 17 stores, for example, an image recognition model. The image recognition model may be stored in a storage means other than the memory unit 17. The memory unit 17 may also store the results of the selection of images to be used to check the state of coverage of a sign by an object. The memory unit 17 stores, for example, images to be used to check the state of coverage of a sign by an object selected by the selection unit 14. The memory unit 17 stores, for example, the sign coverage rate calculated by the calculation unit 15.

[0056] The in-vehicle device 20 includes, for example, a camera that captures road images. The camera of the in-vehicle device 20 captures, for example, images of the road including signs. The camera continuously captures, for example, images of the road while the vehicle is traveling. The in-vehicle device 20 includes, for example, a camera that captures images of the area in front of the vehicle. The camera of the in-vehicle device 20 may capture images of the area behind the vehicle. The in-vehicle device 20 may include a sensor that measures the speed of the vehicle. The in-vehicle device 20 includes, for example, a sensor that measures the speed and acceleration of the vehicle. The in-vehicle device 20 may also acquire data related to the speed of the vehicle from a vehicle control system. The in-vehicle device 20 outputs the captured images of the road and data related to the speed of the vehicle to the sign diagnosis system 10, for example, via a network. When data is input / output to / from the sign diagnosis system 10 via a recording medium, the in-vehicle device 20 includes, for example, a slot for inserting a removable recording medium.

[0057] The in-vehicle device 20 may add information about the location where the image was taken to the captured image. The in-vehicle device 20 may identify the location of the vehicle when the image was taken, for example, by using a Global Navigation Satellite System (GNSS). The in-vehicle device 20 may identify the location of the vehicle based on a beacon that includes location information. The in-vehicle device 20 may identify the location where the image was taken based on map information and the distance traveled from the location where the vehicle's location was identified. The in-vehicle device 20 may output the captured image to, for example, the sign diagnosis system 10. The in-vehicle device 20 may be, for example, a drive recorder. The in-vehicle device 20 is not limited to a drive recorder.

[0058] The terminal device 30, for example, acquires an image used to check the state of coverage of the marker by the object. The terminal device 30 acquires the image used to check the state of coverage of the marker by the object, for example, from the sign diagnosis system 10. Then, the terminal device 30 outputs the image used to check the state of coverage of the marker by the object, for example, to a display device not shown. The terminal device 30 acquires, for example, the image used to check the state of coverage of the marker by the object, as well as the coverage rate of the marker by the object in the image. Then, the terminal device 30 outputs, for example, the image used to check the state of coverage of the marker by the body and the coverage rate of the marker by the object, to a display device not shown.

[0059] For example, a personal computer, a tablet computer, or a smartphone can be used as the terminal device 30. The terminal device 30 is not limited to the above examples.

[0060] The following describes the operation of the sign diagnosis system 10 when selecting an image to be used to check whether a sign is covered by an object. Fig. 12 shows an example of the operation flow when the sign diagnosis system 10 selects an image to be used to check whether a sign is covered by an object.

[0061] The acquisition unit 11 acquires time-series images of a road and information relating to the speed of a vehicle equipped with an image capturing device that captured the images (step S11). The acquisition unit 11 acquires time-series images of a road and information relating to the speed of the vehicle from, for example, the in-vehicle device 20.

[0062] After acquiring the time-series images of the road and information about the vehicle speed, the identification unit 12 identifies the size of the signs appearing in the time-series images (step S12). The identification unit 12 uses, for example, an image recognition model to identify the size of the signs appearing in each frame included in the time-series images.

[0063] Once the size of the sign has been identified, the estimation unit 13 estimates the size of the sign at a second point in time later on the timeline than the first point in time, based on the size of the sign at the first point in time on the timeline identified by the identification unit 12 and information about the speed (step S13).

[0064] Once the size of the sign at the second time point has been estimated, the selection unit 14 selects an image to be used to confirm the state of coverage of the sign by the object, based on the size of the sign at the second time point identified by the identification unit 12 and the size of the sign at the second time point estimated by the estimation unit 13 (step S14).

[0065] When the image to be used for checking the covering state of the sign by the object is selected, the output unit 16 outputs the image to be used for checking the covering state of the sign by the object (step S15). The output unit 16 outputs the image to be used for checking the covering state of the sign by the object to, for example, the terminal device 30.

[0066] When the images used to check the state of coverage of the sign by the object are output, if the image selection process has been completed for all images acquired by the acquisition unit 11 (Yes in step S16), the sign diagnosis system 10 ends the process of selecting images used to check the state of coverage of the sign by the object.

[0067] In step S16, when an image to be used to check the state of coverage of the sign by the object is output, if there are any images acquired by the acquisition unit 11 for which the image selection process has not been completed (No in step S16), the process returns to step S12, and the recognition unit 12 performs the process of identifying the signs appearing in the images for which recognition has not been completed.

[0068] The sign diagnosis system 10 identifies the size of a sign in an image captured at a first time point among images captured continuously of a road. The sign diagnosis system 10 then estimates the size of a sign at a second time point, which is later than the first time point, based on the size of the sign at the first time point and information related to the speed of a vehicle equipped with an image capture device that captured the image. The sign diagnosis system 10 also identifies the size of a sign in an image captured at the second time point. The sign diagnosis system 10 then selects an image to use to confirm whether the sign is covered by an object at the second time point based on the estimated size of the sign and the size of the identified sign. For example, if the sign image is captured in the direction of travel of the vehicle, the sign diagnosis system 10 determines that the sign was covered by an object at the first time point if the size of the identified sign is larger than the estimated size of the sign. If a sign is covered by an object at the first time point, the area of ​​the sign identified by image recognition will be smaller than the actual area of ​​the sign, and the estimated size of the sign at the second time point will also be smaller. Therefore, if the identified size of the sign is larger than the estimated size of the sign, the sign diagnosis system 10 selects the sign shown in the image at the first time point as the image to be used for confirming whether the sign is covered by an object. By selecting the image to be used for confirming whether the sign is covered by an object in this way, the sign diagnosis system 10 can select an image suitable for confirming whether the sign is covered by an object. Furthermore, for example, a road administrator can easily confirm whether the sign is covered by an object by referring to the image suitable for confirming whether the sign is covered by an object selected by the sign diagnosis system 10. As a result, the sign diagnosis system 10 can easily confirm whether the sign is covered by an object.

[0069] Furthermore, by determining, among multiple images showing the same sign, for example, the image with the smallest sign size that is equal to or larger than a standard, as the image suitable for checking the coverage status, it is possible to select an image of a sign at a point where pedestrians on the road can recognize the content of the sign and take action according to the content of the sign. This makes it possible to check the coverage status of a sign by an object based on a more appropriate image.

[0070] The processes in the sign diagnosis system 10 may be distributed and executed among multiple information processing devices connected via a network. For example, the process of the identification unit 12 and the processes of the estimation unit 13, selection unit 14, and calculation unit 15 may be executed in different information processing devices. It can be appropriately determined which of the multiple information processing devices executes each process in the sign diagnosis system 10.

[0071] Each process in the sign diagnosis system 10 can be realized by executing a computer program on a computer. Fig. 13 shows an example of the configuration of a computer 100 that executes a computer program that performs each process in the sign diagnosis system 10. The computer 100 includes a CPU (Central Processing Unit) 101, a memory 102, a storage device 103, an input / output I / F (Interface) 104, and a communication I / F 105.

[0072] The CPU 101 reads and executes computer programs for performing each process from the storage device 103. The CPU 101 may be configured by a combination of multiple CPUs. Furthermore, the CPU 101 may be configured by a combination of a CPU and another type of processor. For example, the CPU 101 may be configured by a combination of a CPU and a graphics processing unit (GPU). The memory 102 is configured by a dynamic random access memory (DRAM) or the like, and temporarily stores computer programs executed by the CPU 101 and data being processed. The storage device 103 stores computer programs executed by the CPU 101. The storage device 103 is configured by, for example, a non-volatile semiconductor storage device. Other storage devices such as a hard disk drive may also be used for the storage device 103. The input / output I / F 104 is an interface that receives input from an operator and outputs display data, etc. The communication I / F 105 is an interface that transmits and receives data between the in-vehicle device 20, the terminal device 30, and other information processing devices. Furthermore, the terminal device 30 may have a configuration similar to that of the computer 100.

[0073] The computer program used to execute each process can also be stored and distributed on a computer-readable recording medium that non-temporarily stores data. Examples of recording media that can be used include magnetic tapes for recording data and magnetic disks such as hard disks. Optical disks such as CD-ROMs (Compact Disc Read Only Memory) can also be used as recording media. Non-volatile semiconductor storage devices can also be used as recording media.

[0074] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0075] [Appendix 1] an acquisition means for acquiring time-series images of a road taken by an imaging device mounted on a vehicle; an identification means for identifying the size of a sign shown in the time-series images; an estimation means for estimating a size of a sign at a second time point on the time series that is later than the first time point on the time series, based on the size of the sign at the first time point on the time series identified by the identification means; a selection means for selecting an image to be used for checking a state of coverage of the marker by an object based on the size of the marker at the second time point identified by the identification means and the size of the marker at the second time point estimated by the estimation means; an output means for outputting the image selected by the selection means; A sign diagnostic system comprising:

[0076] [Appendix 2] The acquisition means further acquires information regarding a speed of the vehicle; the estimation means estimates the size of the sign at the second time point based on information about the size of the sign and the speed of the vehicle at the first time point; 2. The tagged diagnostic system of claim 1.

[0077] [Appendix 3] the selecting means selects an image to be used for checking a state in which the marker is covered by an object, based on which is larger: the size of the marker identified by the identifying means at the second time point; or the size of the marker estimated by the estimating means at the second time point. 3. The tagged diagnostic system of claim 1 or 2.

[0078] [Appendix 4] When the photographing device photographs the area ahead in the traveling direction of the vehicle, the selecting means selects the image at the first time point as the image to be used for checking the covering state of the sign when the size of the sign at the second time point identified by the identifying means is larger than the size of the sign at the second time point estimated by the estimating means. 4. The tagged diagnostic system of claim 3.

[0079] [Appendix 5] When the photographing device photographs the rear of the vehicle in the traveling direction, the selecting means, when the size of the sign identified by the identifying means at the second time point is smaller than the size of the sign estimated by the estimating means at the second time point, selects the image at the second time point as the image to be used for confirming the covering state of the sign. 4. The tagged diagnostic system of claim 3.

[0080] [Appendix 6] When the difference between the size of the sign identified by the identification means and the size of the sign estimated by the estimation means increases over time in the time series of images, the selection means selects, as an image to be used for checking the covering state of the sign, an image in which the position of the sign is farthest from the position of the vehicle at the time of photographing, from among the images in which the size of the sign meets a criterion. 4. The tagging diagnostic system of any one of appendices 1 to 3.

[0081] [Appendix 7] The image processing apparatus further includes a calculation unit that calculates a coverage rate of the marker by the object based on an image selected as an image to be used for checking the coverage state of the marker, The output means further outputs the estimated coverage. 7. The tagging diagnostic system of any one of appendices 1 to 6.

[0082] [Appendix 8] the selecting means, when a difference between the size of the sign at the second time point identified by the identifying means and the size of the sign at the second time point estimated by the estimating means is less than a predetermined standard, selects either the image at the first time point or the image at the second time point as an image to be used for confirming the covering state of the sign. 8. The tagged diagnostic system of any one of appendices 1 to 7.

[0083] [Appendix 9] the object is a tree, the selection means selects an image to be used for checking the state of coverage of the sign by the tree. 9. The tagging diagnostic system of any one of appendices 1 to 8.

[0084] [Appendix 10] the output means outputs the image selected by the selection means and images taken before and after the selected image in time series. 10. The tagged diagnostic system of any one of appendices 1 to 9.

[0085] [Appendix 11] A time series of images of the road are acquired using a photographing device mounted on a vehicle, Identifying the size of a sign appearing in the time series of images; estimating the size of the sign at a second time point on the time series that is later than the first time point on the time series based on the identified size of the sign at a first time point on the time series; selecting an image to be used for confirming a state in which the marker is covered by an object based on the identified size of the marker at the second time point and the estimated size of the marker at the second time point; outputting the selected image; Sign diagnostic methods.

[0086] [Appendix 12] A process of acquiring time-series images of a road taken by an imaging device mounted on a vehicle; A process of identifying the size of a sign shown in the time series of images; a process of estimating a size of a sign at a second time point on the time series that is later than the first time point on the time series, based on the identified size of the sign at the first time point on the time series; a process of selecting an image to be used for confirming a state in which the marker is covered by an object, based on the identified size of the marker at the second time point and the estimated size of the marker at the second time point; outputting the selected image; A recording medium on which a sign diagnosis program is recorded that causes a computer to execute the above.

[0087] The present disclosure has been described above using the above-described embodiments as examples. However, the present disclosure is not limited to the above-described embodiments. That is, the present disclosure can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present disclosure. [Explanation of symbols]

[0088] 10. Sign diagnostic system 11 Acquisition Department 12 Identification unit 13 Estimation part 14 Selection section 15 Calculation section 16 Output section 17 Memory section 20 Onboard equipment 30 Terminal Equipment 100 computers 101 CPU 102 memory 103 Storage device 104 Input / Output Interface 105 Communication I / F

Claims

1. an acquisition means for acquiring time-series images of a road taken by an imaging device mounted on a vehicle; an identification means for identifying the size of a sign shown in the time-series images; an estimation means for estimating a size of a sign at a second time point on the time series that is later than the first time point on the time series, based on the size of the sign at the first time point on the time series identified by the identification means; a selection means for selecting an image to be used for checking a state in which the marker is covered by an object, based on the size of the marker at the second time point identified by the identification means and the size of the marker at the second time point estimated by the estimation means; an output means for outputting the image selected by the selection means; A sign diagnostic system comprising:

2. The acquisition means further acquires information regarding a speed of the vehicle; the estimation means estimates the size of the sign at the second time point based on information about the size of the sign and the speed of the vehicle at the first time point; The marker diagnostic system of claim 1 .

3. the selecting means selects an image to be used for checking a state in which the marker is covered by an object, based on which of the size of the marker at the second time point identified by the identifying means and the size of the marker at the second time point estimated by the estimating means is larger. The marker diagnostic system according to claim 1 or 2.

4. When the photographing device photographs the area ahead in the traveling direction of the vehicle, the selecting means, when the size of the sign identified by the identifying means at the second time point is larger than the size of the sign estimated by the estimating means at the second time point, selects the image at the first time point as the image to be used for confirming the covering state of the sign. The sign diagnostic system of claim 3 .

5. When the photographing device photographs the rear of the vehicle in the traveling direction, the selecting means, when the size of the sign identified by the identifying means at the second time point is smaller than the size of the sign estimated by the estimating means at the second time point, selects the image at the second time point as the image to be used for confirming the covering state of the sign. The sign diagnostic system of claim 3 .

6. When the difference between the size of the sign identified by the identification means and the size of the sign estimated by the estimation means increases over time in the time series of images, the selection means selects, as an image to be used for checking the covering state of the sign, an image in which the position of the sign is farthest from the position of the vehicle at the time of photographing, from among the images in which the size of the sign meets a criterion. The marker diagnostic system according to claim 1 or 2.

7. The image processing apparatus further includes a calculation unit that calculates a coverage rate of the marker by the object based on an image selected as an image to be used for checking the coverage state of the marker, The output means further outputs the estimated coverage. The marker diagnostic system according to claim 1 or 2.

8. the selecting means, when a difference between the size of the marker identified by the identifying means at the second time point and the size of the marker estimated by the estimating means at the second time point is less than a predetermined standard, selects either the image at the first time point or the image at the second time point as an image to be used for confirming the covering state of the marker. The marker diagnostic system according to claim 1 or 2.

9. A time series of images of the road are acquired using a photographing device mounted on a vehicle, Identifying the size of a sign appearing in the time series of images; estimating a size of the sign at a second time point on the time series that is later than the first time point on the time series based on the identified size of the sign at a first time point on the time series; selecting an image to be used for confirming a state in which the marker is covered by an object based on the identified size of the marker at the second time point and the estimated size of the marker at the second time point; outputting the selected image; Sign diagnostic methods.

10. A process of acquiring time-series images of a road taken by an imaging device mounted on a vehicle; A process of identifying the size of a sign shown in the time series of images; a process of estimating a size of a sign at a second time point on the time series that is later than the first time point on the time series, based on the identified size of the sign at the first time point on the time series; a process of selecting an image to be used for confirming a state in which the marker is covered by an object, based on the identified size of the marker at the second time point and the estimated size of the marker at the second time point; outputting the selected image; A sign diagnostic program that causes a computer to execute the above.

Images