Road surface abnormality detection device, method and program

Abstract

To provide a road surface abnormality detection device capable of detecting an abnormality of a road surface, without learning of an abnormality of the road surface.SOLUTION: A road surface abnormality detection device comprises: data acquisition means; prediction means; abnormality detecting means. The data acquisition means acquires road surface observation data which is acquired by observing the road surface along time series. The prediction means uses a learned model which is generated by machine learning in advance, for predicting road surface data at first timing, from multiple pieces of road surface observation data at multiple pieces of timing before the first timing. The abnormality detecting means detects the abnormality of the road surface, when a difference between the road surface observation data at the first timing and the road surface predicted at the first timing is equal to or more than a predetermined threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a road surface abnormality detection device, a road surface abnormality detection method, and a program. [Background technology]

[0002] Patent Document 1 discloses an estimation device that aims to accurately estimate a label that indicates the state of data. The estimation device according to Patent Document 1 inputs time series data into a trained first model for calculating the likelihood of a label for each of a plurality of types of labels, and estimates the likelihood of the label at each time for each label. The estimation device according to Patent Document 1 inputs the estimated likelihood of the label at each time for each label into a trained second model for outputting one of the labels based on the change in the likelihood for each label, and estimates one of the labels corresponding to the likelihood for each label at each time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 100893 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology according to Patent Document 1 can only estimate the road surface condition defined by the label. In other words, the technology according to Patent Document 1 can only estimate the road surface abnormality defined by the label. Therefore, the technology according to Patent Document 1 can only detect the road surface abnormality assumed at the time of learning. In other words, the technology according to Patent Document 1 may not be able to detect the road surface abnormality that was not assumed at the time of learning. Therefore, it is desired to detect the road surface abnormality without learning the road surface abnormality.

[0005] The object of the present disclosure has been made to solve such problems, and is to provide a road surface abnormality detection device, a road surface abnormality detection method, and a program that are capable of detecting road surface abnormalities without learning about the road surface abnormalities. [Means for solving the problem]

[0006] The road surface abnormality detection device disclosed herein includes a data acquisition means for acquiring road surface observation data obtained by observing the road surface in a time series, a prediction means for predicting road surface data at a first timing from a plurality of road surface observation data at a plurality of timings prior to the first timing using a trained model generated in advance by machine learning, and an abnormality detection means for detecting an abnormality in the road surface when a difference between the road surface observation data at the first timing and the predicted road surface data at the first timing is equal to or greater than a predetermined threshold.

[0007] The road surface abnormality detection method disclosed herein acquires road surface observation data obtained by observing the road surface in a time series, and uses a trained model generated in advance by machine learning to predict road surface data at a first timing from multiple road surface observation data at multiple timings prior to the first timing, and detects an abnormality in the road surface if the difference between the road surface observation data at the first timing and the predicted road surface data at the first timing is greater than or equal to a predetermined threshold.

[0008] The program disclosed herein enables a computer to perform the following functions: acquire road surface observation data obtained by observing the road surface over time; predict road surface data at a first timing from multiple road surface observation data at multiple timings prior to the first timing using a trained model generated in advance by machine learning; and detect an abnormality in the road surface when a difference between the road surface observation data at the first timing and the predicted road surface data at the first timing is greater than or equal to a predetermined threshold value. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a road surface abnormality detection device, a road surface abnormality detection method, and a program that are capable of detecting road surface abnormalities without learning about the road surface abnormalities. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration of a road surface abnormality detection device according to the present disclosure. [Diagram 2] 5 is a flowchart showing a road surface abnormality detection method executed by the road surface abnormality detection device according to the present disclosure. [Diagram 3] 1 is a diagram illustrating an example of a configuration of a road surface abnormality detection system according to the present disclosure. [Figure 4] 1 is a diagram illustrating an example of a hardware configuration of a road surface abnormality detection device according to the present disclosure. [Diagram 5] 1 is a diagram illustrating an example of a functional configuration of a road surface abnormality detection device according to the present disclosure. [Figure 6] FIG. 2 is a diagram for explaining road surface observation data according to the present disclosure. [Figure 7] 10A and 10B are diagrams for explaining a road surface abnormality detection process according to the present disclosure. [Figure 8] 3 is a diagram illustrating an example of road surface abnormality information output by a road surface abnormality detection device according to the present disclosure. FIG. [Figure 9] 3 is a diagram illustrating an example of road surface abnormality information output by a road surface abnormality detection device according to the present disclosure. FIG. [Figure 10] 11 is a diagram for explaining a case where the learning process according to the present disclosure is performed in an operational stage. FIG. [Figure 11] 11 is a diagram for explaining a case where the learning process according to the present disclosure is performed in an operational stage. FIG. [Figure 12] 11 is a diagram for explaining a case where the learning process according to the present disclosure is performed in an operational stage. FIG. [Figure 13] 4 is a flowchart illustrating an example of a process executed by the road surface abnormality detection device according to the present disclosure. [Figure 14]1 is a diagram illustrating an example of a functional configuration of a road surface abnormality detection device according to the present disclosure. [Figure 15] FIG. 2 is a diagram for explaining processing using low-dimensional vector data according to the present disclosure. [Figure 16] FIG. 2 is a diagram for explaining processing using low-dimensional vector data according to the present disclosure. [Figure 17] 4 is a flowchart illustrating an example of a process executed by the road surface abnormality detection device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described with reference to the drawings. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In addition, in each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary. In addition, although the following description will be made using the drawings, the drawings used in the description of a certain embodiment do not apply only to that embodiment. Each drawing may be applied to all embodiments.

[0012] (Embodiment 1) FIG 1 is a diagram showing the configuration of a road surface abnormality detection device 1 according to the present disclosure. The road surface abnormality detection device 1 has a data acquisition unit 12, a prediction unit 14, and an abnormality detection unit 16. The data acquisition unit 12 functions as data acquisition means. The prediction unit 14 functions as prediction means. The abnormality detection unit 16 functions as abnormality detection means. The road surface abnormality detection device 1 detects abnormalities on the road surface using these components.

[0013] The road surface abnormality detection device 1 may be realized by a computer. The road surface abnormality detection device 1 may be realized by, for example, cloud computing. The road surface abnormality detection device 1 may also be realized by, for example, a plurality of computers connected to each other so as to be able to communicate with each other. The components of the road surface abnormality detection device 1 may also be distributed and realized on a plurality of computers. That is, the computer that realizes the data acquisition unit 12, the computer that realizes the prediction unit 14, and the computer that realizes the abnormality detection unit 16 may be physically separate. The functions of each of the plurality of components may also be realized by a plurality of computers.

[0014] In the present disclosure, a "road surface" is a surface or passageway that a user or a moving object can pass through, outdoors or indoors. The road surface is, for example, the surface of an outdoor or indoor road, but is not limited thereto. A user passing through the road surface may walk or run on the road surface, or may pass through the road surface using an assistive device such as a wheelchair. The moving object passing through the road surface may be, for example, a moving object that runs autonomously, such as a robot or an Unmanned Ground Vehicle (UGV), or may be, for example, a moving object such as a vehicle that moves by the operation of a user, such as a car or a bicycle.

[0015] FIG. 2 is a flowchart showing a road surface abnormality detection method executed by the road surface abnormality detection device 1 according to the present disclosure. The data acquisition unit 12 acquires road surface observation data (step S12). Specifically, the data acquisition unit 12 acquires road surface observation data obtained by observing the road surface in a time series. In the present disclosure, "road surface observation data" is data obtained by observing the road surface. The road surface observation data may be acquired by an observation device moving along the road surface. In this case, the data acquisition unit 12 may acquire road surface observation data obtained by observing the road surface in a time series by an observation device moving along the road surface. The type of road surface observation data may differ depending on the type of observation device described later. Here, the observation device may move together with a user or a moving object passing along the road surface. Alternatively, the observation device may move along the road surface independently of the user or the moving object.

[0016] In addition, in the present disclosure, an "observation device" is a device that observes the state of a road surface. The observation device may be, for example, an imaging device such as a camera. The observation device may also be a detection device such as a sensor. The observation device may include, for example, a three-dimensional camera such as an RGB camera or a depth camera, a three-dimensional laser scanner, or a LiDAR (Light Detection and Ranging). The observation device may also include a device that measures the inclination of the road surface, such as an IMU (Inertial Measurement Unit).

[0017] Road surface observation data is data obtained by observing a road surface. Furthermore, the road surface observation data is time series data because it is obtained by observing the road surface in a time series. In other words, the road surface observation data can be acquired over time. For example, the road surface observation data can be acquired at each observation cycle. Furthermore, for example, the road surface observation data can be acquired at each sampling cycle. The type of road surface observation data can vary depending on the type of observation device.

[0018] The prediction unit 14 predicts the road surface data (step S14). Specifically, the prediction unit 14 predicts the road surface data at the first timing from a plurality of road surface observation data at a plurality of timings before the first timing, using a trained model generated in advance by machine learning. That is, the prediction unit 14 predicts the road surface data at the first timing by inputting a plurality of road surface observation data at a plurality of timings before the first timing to the trained model. In other words, the prediction unit 14 generates the road surface data at the first timing by inputting a plurality of road surface observation data at a plurality of timings before the first timing to the trained model. Note that hereinafter, the predicted road surface data may be referred to as "road surface prediction data".

[0019] The trained model is generated by learning its weights (coefficients) using a machine learning algorithm such as a recurrent neural network (RNN) or a long short term memory (LSTM). The trained model can be generated by performing machine learning so that a plurality of road surface observation data at each of a plurality of timings is input and road surface data at a timing next to the last timing of the plurality of timings is output. The term "trained model generated in advance by machine learning" means a trained model that has been trained to such an extent that the prediction unit 14 can make a prediction at a stage prior to prediction. Therefore, it should be noted that the expression "trained model generated in advance by machine learning" does not exclude the trained model (the weights of the trained model) being updated at the inference stage.

[0020] The abnormality detection unit 16 detects an abnormality on the road surface (step S16). Specifically, the abnormality detection unit 16 detects an abnormality on the road surface when a difference between road surface observation data at a first timing and predicted road surface data at the first timing is equal to or greater than a predetermined threshold. That is, the abnormality detection unit 16 detects an abnormality on the road surface by comparing the road surface observation data at the first timing with the road surface prediction data at the first timing. Here, the difference between the road surface observation data and the road surface prediction data may be calculated using, for example, a loss function, but is not limited to this. The loss function may be, for example, a minimum squared error (MSE) or a binary cross entropy (BCE), etc.

[0021] Furthermore, in the present disclosure, as described above, a "road surface abnormality" corresponds to a phenomenon represented in the road surface observation data such that the difference between the road surface observation data and the road surface prediction data is equal to or greater than a threshold. The road surface abnormality may be a phenomenon that impedes the passage of a subject traveling on the road surface. The road surface abnormality may be, for example, the presence of a step, a depression, an obstacle, or the like on the road surface. Furthermore, a threshold may be set such that when there is an abnormality in the road surface, the difference between the road surface observation data and the road surface prediction data is equal to or greater than the threshold.

[0022] Furthermore, the threshold used to detect an abnormality on the road surface may be set according to the attributes of the road surface. Here, the "attributes of the road surface" may include, for example, the environment of the road surface at the timing when the abnormality detection is performed. In other words, the "attributes of the road surface" may include, for example, the environment of the road surface when the road surface observation data is obtained. In this case, the threshold used to detect an abnormality on the road surface may be set according to the environment of the road surface. Note that the "environment of the road surface" may include, for example, the time period and the weather. Furthermore, the "attributes of the road surface" may include the attributes of an entity (user or moving object) passing through the road surface. In this case, the threshold used to detect an abnormality on the road surface may be set according to the attributes of the entity passing through the road surface. Note that the "attributes of the entity passing through the road surface" may include, for example, the age of the user. Furthermore, the "attributes of the entity passing through the road surface" may include, for example, the type of moving object. The type of moving object may correspond to, for example, a robot, a UGV, a car, or a bicycle.

[0023] Furthermore, when road surface observation data is obtained by an observation device moving along the road surface, the "road surface attributes" may include, for example, the usage environment of the observation device. In this case, the threshold value used to detect abnormalities in the road surface may be set according to the usage environment of the observation device. The "usage environment" may include, for example, the attributes of a user or a moving object moving together with the observation device, the surrounding conditions in which the observation device is used, or the time period in which the observation device is used.

[0024] As described above, the road surface abnormality detection device 1 according to the present disclosure predicts road surface data at a first timing from a plurality of road surface observation data at a plurality of timings prior to the first timing using a trained model. In addition, the road surface abnormality detection device 1 according to the present disclosure detects an abnormality on the road surface when the difference between the road surface observation data at the first timing and the predicted road surface data at the first timing is equal to or greater than a predetermined threshold. The road surface abnormality detection device 1 according to the present disclosure is configured as described above, so that it is not necessary to define an abnormal road surface and an abnormal road surface, i.e., a normal road surface, by using labels when generating a trained model used to predict road surface data. Therefore, the road surface abnormality detection device 1 according to the present disclosure can detect an abnormality on the road surface without learning the abnormality on the road surface. This makes it possible to detect the presence of an obstacle to traffic on the road surface.

[0025] In addition, when abnormal road surfaces and normal road surfaces are defined by labels as in the technology of the above-mentioned patent document, the number of patterns of abnormal road surfaces and normal road surfaces may be enormous. Therefore, when it is necessary to define abnormal road surfaces and normal road surfaces by labels, it is extremely difficult to obtain all patterns. Therefore, there is a risk that learning for a trained model cannot be performed appropriately. Furthermore, even if all patterns of abnormal road surfaces and normal road surfaces can be obtained, it is not easy to perform learning using all the patterns.

[0026] In contrast, in the road surface abnormality detection device 1 according to the present disclosure, abnormal road surfaces and normal road surfaces are not defined by labels. The road surface abnormality detection device 1 according to the present disclosure predicts road surface data using a trained model in which abnormal road surfaces and normal road surfaces are not defined by labels, and detects road surface abnormalities by comparing the obtained road surface prediction data with road surface observation data. Therefore, road surface abnormalities can be detected without learning about road surface abnormalities. Therefore, the road surface abnormality detection device 1 according to the present disclosure can easily and appropriately train the trained model used to predict road surface data.

[0027] It is possible to detect road surface abnormalities without learning about the road surface abnormalities by the road surface abnormality detection method executed by the road surface abnormality detection device 1. It is also possible to detect road surface abnormalities without learning about the road surface abnormalities by the program executing the road surface abnormality detection method.

[0028] (Embodiment 2) Next, the second embodiment will be described with reference to the drawings. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In addition, in each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary.

[0029] 3 is a diagram illustrating an example of a configuration of a road surface abnormality detection system 20 according to the present disclosure. The road surface abnormality detection system 20 includes one or more observation devices 30, one or more terminal devices 40, one or more mobile objects 50, and a road surface abnormality detection device 100. The observation devices 30, the terminal devices 40, the mobile objects 50, and the road surface abnormality detection device 100 each have a function as, for example, a computer.

[0030] The observation device 30, the terminal device 40, the mobile body 50, and the road surface abnormality detection device 100 are communicatively connected via a network. The network may be, for example, a wireless network or a wired network. That is, the road surface abnormality detection device 100 is connected to the observation device 30, the terminal device 40, and the mobile body 50 wirelessly or wired. The wireless network may be, for example, a network using a communication line standard such as LTE (Long Term Evolution), or may be a network used in a specific area such as WiFi (registered trademark) or local 5G. The wired network may be, for example, a LAN (Local Area Network) or optical fiber.

[0031] As described above, the observation device 30 is a device that observes the state of the road surface. The observation device 30 may be, for example, an imaging device such as a camera that captures an image of the road surface. The observation device 30 may also be a detection device such as a sensor that detects the state of the road surface. The observation device 30 may be, for example, a three-dimensional camera such as an RGB camera or a depth camera, a three-dimensional laser scanner, or a LiDAR. The observation device 30 may also be an IMU. The observation device 30 may move along the road surface. For example, the observation device 30 may move along the road surface together with a subject passing along the road surface.

[0032] The observation device 30 also generates road surface observation results indicating the results of observing the road surface state. The observation device 30 transmits the road surface observation results to the road surface abnormality detection device 100. The observation device 30 may also generate road surface observation data indicating the results of observing the road surface state. In this case, the observation device 30 may transmit the road surface observation data to the road surface abnormality detection device 100. The observation device 30 may also have a positioning function such as a Global Navigation Satellite System (GNSS). The observation device 30 may also generate road surface observation results or road surface observation data including position information of the observed road surface.

[0033] For example, when the observation device 30 is an imaging device, the observation device 30 may generate image data as road surface observation data. The image data may be data showing a still image or data showing a moving image. The image data may be RGB image data, depth image data, or RGBD image data. For example, when the observation device 30 is a three-dimensional laser scanner or LiDAR, the observation device 30 may generate three-dimensional point cloud data as road surface observation data. For example, when the observation device 30 is an IMU, the observation device 30 may generate IMU data showing the measurement results by the IMU as road surface observation data.

[0034] The observation device 30 may also generate road surface observation data that changes over time. For example, when the observation device 30 moves along the road surface while observing the road surface, the observation device 30 may generate road surface observation data that changes from moment to moment as the observation device 30 moves. When the observation device 30 moves along the road surface together with an entity passing along the road surface, the observation device 30 may generate road surface observation data obtained by observing the road surface in the vicinity of the location where the entity passing along the road surface is present. In this case, the observation device 30 may generate road surface observation data that changes from moment to moment as the entity passing along the road surface moves.

[0035] The terminal device 40 is a terminal that can be carried by a user. For example, the terminal device 40 can be carried by a user who is a subject passing through a road surface. The terminal device 40 presents necessary information to the user. The terminal device 40 may be, for example, a smartphone, a mobile phone, a tablet terminal, a smart watch, or a wearable device.

[0036] As described above, the moving body 50 is a device that travels on a road surface. The moving body 50 may travel autonomously. The moving body 50 may also move by a user's operation. The moving body 50 may be, for example, a robot, a UGV, a car, or a bicycle.

[0037] The road surface abnormality detection device 100 corresponds to the road surface abnormality detection device 1 shown in FIG. 1. The road surface abnormality detection device 100 may be, for example, a server. The road surface abnormality detection device 100 may be, for example, an information terminal. The road surface abnormality detection device 100 may be realized by, for example, cloud computing. The road surface abnormality detection device 100 may be realized by, for example, a plurality of computers connected to each other so as to be able to communicate with each other. Each component of the road surface abnormality detection device 100 described below may be realized by a plurality of computers.

[0038] Any of a plurality of devices constituting the road surface abnormality detection system 20 may be physically integrated. For example, the observation device 30 and the terminal device 40 may be physically integrated. The observation device 30 and the road surface abnormality detection device 100 may be physically integrated. The terminal device 40 and the road surface abnormality detection device 100 may be physically integrated. The observation device 30 and the mobile body 50 may be physically integrated. The mobile body 50 and the road surface abnormality detection device 100 may be physically integrated. The observation device 30, the terminal device 40, and the road surface abnormality detection device 100 may be physically integrated. The observation device 30, the mobile body 50, and the road surface abnormality detection device 100 may be physically integrated.

[0039] Fig. 4 is a diagram illustrating a hardware configuration of the road surface abnormality detection device 100 according to the present disclosure. As shown in Fig. 4, the road surface abnormality detection device 100 has, as main hardware components, a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 (IF; Interface). The control unit 102, the storage unit 104, the communication unit 106, and the interface unit 108 are connected to each other via a data bus or the like. Note that the observation device 30, the terminal device 40, and the mobile object 50 described above may also have the hardware configuration of the road surface abnormality detection device 100 shown in Fig. 4.

[0040] The control unit 102 is a processor such as a CPU (Central Processing Unit). The control unit 102 has a function as a calculation device that performs control processing and calculation processing. The control unit 102 may have multiple processors. The storage unit 104 is a storage device such as a memory or a hard disk. The storage unit 104 is a read only memory (ROM) or a random access memory (RAM). The storage unit 104 has a function for storing a control program and a calculation program executed by the control unit 102. That is, the storage unit 104, which is a memory, stores one or more instructions. The storage unit 104 also has a function for temporarily storing processing data and the like. The storage unit 104 may include a database. The storage unit 104 may also have multiple memories.

[0041] The communication unit 106 performs processing necessary for the road surface abnormality detection device 100 to communicate with other devices via a network. The communication unit 106 may include a communication port, a router, a firewall, and the like. The interface unit 108 (IF; Interface) is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, a touch panel, or a mouse, and an output device such as a display or a speaker. The interface unit 108 may be configured such that the input device and the output device are integrated, such as a touch screen or a touch panel. The interface unit 108 accepts an operation of inputting data by a user such as an operator or a worker, and outputs information to the user. The interface unit 108 may output information regarding abnormalities on the road surface. The interface unit 108 may also output information indicating a trained model.

[0042] 5 is a diagram illustrating a functional configuration of the road surface abnormality detection device 100 according to the present disclosure. The road surface abnormality detection device 100 has, as components, a data acquisition unit 120, a data storage unit 122, a learning unit 130, and a model storage unit 132. The road surface abnormality detection device 100 also has, as components, a prediction unit 140, a difference calculation unit 150, an abnormality detection unit 160, an abnormality information generation unit 162, a threshold setting unit 170, and an output unit 180. The road surface abnormality detection device 100 detects abnormalities on the road surface using these components. The road surface abnormality detection device 100 also performs processing for outputting information on abnormalities on the road surface using these components.

[0043] As described above, the road surface abnormality detection device 100 does not have to be configured as one physical device. In this case, each of the above-mentioned components may be realized by a plurality of physically separate devices. This also applies to other embodiments described later.

[0044] The data acquisition unit 120 functions as a data acquisition means. The data storage unit 122 functions as a data storage means. The learning unit 130 functions as a learning means. The model storage unit 132 functions as a model storage means. The prediction unit 140 functions as a prediction means. The difference calculation unit 150 functions as a difference calculation means. The anomaly detection unit 160 functions as an anomaly detection means. The anomaly information generation unit 162 functions as an anomaly information generation means. The threshold setting unit 170 functions as a threshold setting means. The output unit 180 functions as an output means.

[0045] Each of the above-mentioned components can be realized, for example, by executing a program under the control of the control unit 102. More specifically, each of the components can be realized by the control unit 102 executing a program (instruction) stored in the storage unit 104. Also, each of the components may be realized by recording a necessary program in an arbitrary non-volatile recording medium and installing it as necessary. Also, each of the components may be realized not only by software by a program, but also by a combination of hardware, firmware, and software. Also, each of the components may be realized by using an integrated circuit that can be programmed by a user, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, a program composed of each of the above-mentioned components may be realized by using this integrated circuit. The same applies to other embodiments described later.

[0046] The data acquisition unit 120 corresponds to the data acquisition unit 12 described above. The data acquisition unit 120 acquires road surface observation data obtained by observing the road surface in time series. The data acquisition unit 120 may be realized by the communication unit 106. The data acquisition unit 120 may receive the road surface observation data generated by the observation device 30 from the observation device 30. Furthermore, the data acquisition unit 120 may acquire the road surface observation data by the observation device 30 moving along the road surface.

[0047] The data acquisition unit 120 may generate road surface observation data using the road surface observation results received from the observation device 30. For example, the data acquisition unit 120 may generate a normal map (normal data) indicating the normal vector of each minute plane of the road surface as road surface observation data using the depth image data or the three-dimensional point cloud data received from the observation device 30. The normal map may indicate the inclination of the road surface with respect to the vertical direction at each position. Here, the normal map may represent the shape of the road surface. Furthermore, road surface abnormalities that may impede passage often appear as changes in the shape of the road surface. Therefore, by generating a normal map as road surface observation data, it becomes possible to more appropriately perform road surface abnormality detection processing.

[0048] Moreover, the data acquisition unit 120 acquires road surface observation data, which is time-series data, over time. For example, the data acquisition unit 120 may acquire road surface observation data every time the observation device 30 generates a road surface observation result or road surface observation data. For example, the data acquisition unit 120 may acquire road surface observation data every predetermined period. For example, the data acquisition unit 120 may acquire road surface observation data every observation period of the observation device 30. For example, the data acquisition unit 120 may acquire road surface observation data every sampling period. For example, when the observation device 30 is an imaging device, the data acquisition unit 120 may acquire road surface observation data based on FPS (Flames Per Second; frame rate).

[0049] The data storage unit 122 stores the road surface observation data acquired by the data acquisition unit 120. The data storage unit 122 may be realized by the memory unit 104. The data storage unit 122 stores a plurality of road surface observation data along a time series. Here, the collection of a plurality of road surface observation data along a time series may be data that changes over time, such as video image data. In other words, the road surface observation data changes along a time series. Therefore, it can be said that each of the plurality of road surface observation data along a time series is frame data that indicates the state of the road surface at the corresponding time. That is, the data storage unit 122 stores a plurality of frame data. In this case, the data acquisition unit 120 may acquire road surface observation data that is frame data obtained at a certain timing. That is, the data acquisition unit 120 may acquire frame data at a certain timing for each predetermined period. In other words, the data acquisition unit 120 may acquire frame data corresponding to frames that constitute a collection of a plurality of road surface observation data that changes along a time series.

[0050] The learning unit 130 generates a trained model used in the prediction unit 140 described later. The learning unit 130 generates the trained model by a machine learning algorithm that uses time-series data as an input. The learning unit 130 generates the trained model by a machine learning algorithm such as RNN or LSTM. Specifically, the learning unit 130 learns weights in the trained model by the machine learning algorithm.

[0051] The learning unit 130 learns the weights of the learned model so as to input a plurality of road surface observation data, which are time-series data, and output road surface prediction data. Specifically, the learning unit 130 performs machine learning so as to input a plurality of road surface observation data at each of a plurality of timings and output road surface data at a timing next to the last timing of the plurality of timings. Note that the "road surface data at the next timing" corresponds to the road surface prediction data at the "next timing". More specifically, the learning unit 130 learns the weights of the learned model so as to input the values ​​(component values) of each element of a vector (input vector) representing each of the plurality of road surface observation data, and output each component value of a vector (output vector) representing the road surface prediction data. Note that, when the road surface observation data is image data, the input vector may have the value (pixel value) of each pixel of the road surface observation data as a component value. Similarly, when the road surface prediction data is image data, the output vector may have the value (pixel value) of each pixel of the road surface prediction data as a component value. In this case, the learning unit 130 may learn the weights of the trained model so as to input a vector whose components are the pixel values ​​of each pixel of the road surface observation data, and output a vector whose components are the pixel values ​​of each pixel of the road surface prediction data. In this manner, the learning unit 130 generates a trained model. Note that, as described later, the learning unit 130 may generate (update) the trained model by online learning at the inference stage. Also, the learning unit 130 may update the weights of the trained model at the inference stage. Also, the learning unit 130 may not need to update the weights of the trained model at the inference stage.

[0052] The model storage unit 132 stores data indicating the trained model generated by the learning unit 130. The model storage unit 132 may be realized by the storage unit 104. The model storage unit 132 may store data indicating the weights of the trained model trained by the learning unit 130.

[0053] FIG. 6 is a diagram for explaining road surface observation data according to the present disclosure. FIG. 6 shows road surface observation data 60, which is a plurality of frame data that are time series data. In FIG. 6, the horizontal axis is time τ. Time τ corresponds to the timing at which a road surface observation result corresponding to the road surface observation data 60, which is frame data, is obtained. In other words, time τ corresponds to the timing at which the observation device 30 observes the road surface. FIG. 6 shows examples of road surface observation data 60, which is frame data obtained at times #1 to #5, and road surface observation data 60, which is frame data obtained at times #t-5 to #t. In addition, when the road surface observation data shown in FIG. 6 is used in the learning process, it is assumed that all of the road surface observation data shown in FIG. 6 is data obtained by observing a normal road surface.

[0054] The learning unit 130 learns a trained model so that road surface observation data 60, which is N pieces of frame data obtained in a time series, is input and road surface prediction data at a time τ following the time τ of the last road surface observation data of the N pieces of road surface observation data 60 is output. In other words, the learning unit 130 learns a trained model so that road surface observation data 60 of a section including N pieces of frame data is input and frame data at a time τ following the last time τ of the section is predicted. Note that, in the example of FIG. 6, N=3, but N is not limited to 3. In addition, multiple road surface observation data included in one section may be road surface observation data acquired consecutively.

[0055] In the example of FIG. 6, section #4 includes road surface observation data 60 which is frame data from time #1 to time #3. Section #5 includes road surface observation data 60 which is frame data from time #2 to time #4. Section #6 includes road surface observation data 60 which is frame data from time #3 to time #5. Section #M-2 includes road surface observation data 60 which is frame data from time #t-5 to time #t-3. Section #M-1 includes road surface observation data 60 which is frame data from time #t-4 to time #t-2. Section #M includes road surface observation data 60 which is frame data from time #t-3 to time #t-1. Note that in the example of FIG. 6, there are two overlapping pieces of road surface observation data 60 between adjacent sections, but the overlapping pieces of road surface observation data 60 between adjacent sections is not limited to two. 6, the difference between the road surface observation data 60 in adjacent sections is one, but the difference between the road surface observation data 60 in adjacent sections does not have to be one. For example, if section #M includes road surface observation data 60 that is frame data from time #t-3 to time #t-1, section #M-1 may include road surface observation data 60 that is frame data from time #t-5 to time #t-3.

[0056] A case will be described in which the learning unit 130 performs machine learning using the road surface observation data 60 illustrated in Fig. 6. The learning unit 130 adjusts the weights of the trained model so as to predict road surface data at time #4 following the last time #3 of the multiple times #1 to #3 in the section #4, using the road surface observation data 60 of the section #4 as input. That is, the learning unit 130 adjusts the weights of the trained model so as to predict road surface data at time #4, using the road surface observation data 60 of the multiple frames of data in the section #4 as input, using the road surface observation data 60 of time #4 as correct answer data.

[0057] The learning unit 130 takes the road surface observation data 60 of section #5 as input and adjusts the weights of the trained model so as to predict road surface data at time #5 following the last time #4 of the multiple times #2 to #4 of section #5. That is, the learning unit 130 takes the road surface observation data 60, which is multiple frames of data for section #5, as input, and adjusts the weights of the trained model so as to predict road surface data at time #5 using the road surface observation data 60 at time #5 as correct answer data.

[0058] The learning unit 130 takes as input the road surface observation data 60 for section #6 and adjusts the weights of the trained model so as to predict road surface data at time #6 following the last time #5 of the multiple times #3 to #5 for section #6. That is, the learning unit 130 takes as input the road surface observation data 60 which is multiple frames of data for section #6, and adjusts the weights of the trained model so as to predict road surface data at time #6 using the road surface observation data 60 for time #6 as correct answer data.

[0059] The learning unit 130 takes the road surface observation data 60 of the section #M-2 as input and adjusts the weights of the trained model so as to predict road surface data at time #t-2 following the last time #t-3 of the multiple times #t-5 to #t-3 of the section #M-2. That is, the learning unit 130 takes the road surface observation data 60, which is multiple frames of data of the section #M-2, as input, and adjusts the weights of the trained model so as to predict road surface data at time #t-2 using the road surface observation data 60 at time #t-2 as correct answer data.

[0060] The learning unit 130 takes the road surface observation data 60 of the section #M-1 as input and adjusts the weights of the trained model so as to predict road surface data at time #t-1 following the last time #t-2 of the multiple times #t-4 to #t-2 of the section #M-1. That is, the learning unit 130 takes the road surface observation data 60, which is multiple frames of data of the section #M-1, as input, and adjusts the weights of the trained model so as to predict road surface data at time #t-1 using the road surface observation data 60 at time #t-1 as correct answer data.

[0061] The learning unit 130 receives as input the road surface observation data 60 for section #M and adjusts the weights of the trained model so as to predict road surface data at time #t following the last time #t-1 of the multiple times #t-3 to #t-1 for section #M. That is, the learning unit 130 receives as input the road surface observation data 60 which is multiple frames of data for section #M, and adjusts the weights of the trained model so as to predict road surface data at time #t using the road surface observation data 60 at time #t as correct answer data.

[0062] The learning unit 130 may perform the above-mentioned learning process in the learning stage, and may not perform the learning process in the operation stage (inference stage). The "operation stage" refers to a stage in which prediction processing and road surface anomaly detection processing are performed using a trained model. In this case, the road surface anomaly detection device 100 may not have the learning unit 130. That is, the learning unit 130 may be provided in a device separate from the road surface anomaly detection device 100. In this case, the weights of the trained model generated by the learning unit 130 in the learning stage may be fixed in the operation stage.

[0063] The learning unit 130 may also perform the learning process during the operation phase. The learning unit 130 may perform sequential learning or online learning. That is, the learning unit 130 may perform machine learning every time road surface observation data is acquired, and update the weight of the trained model. In other words, when road surface observation data is acquired, the learning unit 130 may perform machine learning using the acquired road surface observation data. At this time, the learning unit 130 may perform the learning process using the acquired road surface observation data and N pieces of road surface observation data acquired before the road surface observation data was acquired. That is, the learning unit 130 may perform the learning process using the acquired road surface observation data as correct answer data and N pieces of road surface observation data acquired before the road surface observation data was acquired as input data.

[0064] Furthermore, when performing the learning process in the operation stage, the learning unit 130 may perform machine learning using road surface observation data in which no abnormality was detected in the abnormality detection process in the abnormality detection unit 160 described later. That is, the learning unit 130 may perform machine learning using road surface observation data in which the difference from the predicted road surface data is less than a threshold value, and update the weight of the learned model. This makes it possible to perform fine tuning on the learned model. Note that the learning unit 130 may perform machine learning using road surface observation data in which no abnormality was detected, for each case in which no abnormality was detected in the abnormality detection process. At this time, the learning unit 130 may perform the learning process using the road surface observation data in which no abnormality was detected and N pieces of road surface observation data in which no abnormality was detected before the road surface observation data was acquired. That is, the learning unit 130 may perform the learning process using the road surface observation data in which no abnormality was detected as correct answer data and the N pieces of road surface observation data in which no abnormality was detected before the road surface observation data was acquired as input data.

[0065] The prediction unit 140 corresponds to the prediction unit 14 described above. The prediction unit 140 predicts road surface data using a trained model stored in the model storage unit 132. Specifically, like the prediction unit 14, the prediction unit 140 predicts road surface data at a first timing from a plurality of road surface observation data at a plurality of timings prior to the first timing using the trained model. In other words, the prediction unit 140 predicts road surface data at a first timing by inputting a plurality of road surface observation data at a plurality of timings prior to the first timing into the trained model. In further other words, the prediction unit 140 generates road surface data at a first timing by inputting a plurality of road surface observation data at a plurality of timings prior to the first timing into the trained model.

[0066] A case will be described in which the prediction unit 140 performs prediction processing using the road surface observation data 60 illustrated in FIG. 6. The prediction unit 140 predicts road surface data at time #4 following the last time #3 of the section #4 using the road surface observation data 60 of the section #4. In this case, "time #4" corresponds to the above-mentioned "first timing". Specifically, the prediction unit 140 generates road surface prediction data at time #4 using the trained model and the road surface observation data 60 of the section #4. More specifically, the prediction unit 140 inputs the road surface observation data 60 of the section #4 into the trained model and acquires the road surface data at time #4 output from the trained model, thereby generating road surface prediction data at time #4.

[0067] The prediction unit 140 uses the road surface observation data 60 of the section #5 to predict road surface data at time #5 following the last time #4 of the section #5. In this case, "time #5" corresponds to the above-mentioned "first timing." Specifically, the prediction unit 140 generates road surface prediction data at time #5 using the trained model and the road surface observation data 60 of the section #5. More specifically, the prediction unit 140 inputs the road surface observation data 60 of the section #5 into the trained model and acquires the road surface data at time #5 output from the trained model, thereby generating road surface prediction data at time #5.

[0068] The prediction unit 140 uses the road surface observation data 60 of the section #M-2 to predict road surface data at time #t-2 following the last time #t-3 of the section #M-2. In this case, "time #t-2" corresponds to the above-mentioned "first timing". Specifically, the prediction unit 140 uses the trained model and the road surface observation data 60 of the section #M-2 to generate road surface prediction data at time #t-2. More specifically, the prediction unit 140 inputs the road surface observation data 60 of the section #M-2 into the trained model and acquires the road surface data at time #t-2 output from the trained model, thereby generating road surface prediction data at time #t-2.

[0069] The prediction unit 140 uses the road surface observation data 60 of the section #M-1 to predict road surface data at time #t-1 following the last time #t-2 of the section #M-1. In this case, "time #t-1" corresponds to the above-mentioned "first timing". Specifically, the prediction unit 140 generates road surface prediction data at time #t-1 using the trained model and the road surface observation data 60 of the section #M-1. More specifically, the prediction unit 140 inputs the road surface observation data 60 of the section #M-1 into the trained model and acquires the road surface data at time #t-1 output from the trained model, thereby generating road surface prediction data at time #t-1.

[0070] The prediction unit 140 uses the road surface observation data 60 of the section #M to predict road surface data at time #t following the last time #t-1 of the section #M. In this case, "time #t" corresponds to the above-mentioned "first timing." Specifically, the prediction unit 140 generates road surface prediction data at time #t using the trained model and the road surface observation data 60 of the section #M. More specifically, the prediction unit 140 inputs the road surface observation data 60 of the section #M to the trained model, and acquires the road surface data at time #t output from the trained model, thereby generating road surface prediction data at time #t.

[0071] The difference calculation unit 150 calculates the difference between the road surface observation data 60 at the first timing and the road surface prediction data at the first timing. That is, the difference calculation unit 150 compares the road surface observation data 60 and the road surface prediction data at each timing and calculates the difference between the road surface observation data 60 and the road surface prediction data at each timing. The difference calculation unit 150 may calculate the difference using, for example, a loss function such as least square error or binary cross entropy, but is not limited to this. The difference calculation unit 150 may input the component values ​​of the vector representing the road surface observation data 60 and the component values ​​of the vector representing the road surface prediction data to a loss function and calculate the value output from the loss function as the difference. In addition, when the road surface observation data is image data, the difference calculation unit 150 may calculate the difference between the road surface observation data and the road surface prediction data by using a loss function to calculate the difference (deviation) between pixel values ​​of corresponding pixels in the road surface observation data and the road surface prediction data.

[0072] The abnormality detection unit 160 corresponds to the abnormality detection unit 16 described above. The abnormality detection unit 160 detects an abnormality in the road surface when the difference calculated by the difference calculation unit 150 is equal to or greater than a predetermined threshold. That is, the abnormality detection unit 160 detects an abnormality in the road surface at the first timing when the difference between the road surface observation data at the first timing and the road surface prediction data at the first timing is equal to or greater than a predetermined threshold. Specifically, the abnormality detection unit 160 determines whether or not the difference between the road surface observation data at the first timing and the road surface prediction data at the first timing is equal to or greater than a predetermined threshold. More specifically, the abnormality detection unit 160 determines whether or not the output value obtained by inputting the road surface observation data at the first timing and the road surface prediction data at the first timing to a loss function is equal to or greater than a predetermined threshold. The abnormality detection unit 160 detects an abnormality in the road surface at the first timing when the difference is equal to or greater than a threshold. On the other hand, if the difference is not equal to or greater than the threshold, the abnormality detection unit 160 determines that the road surface is normal at the first timing. The threshold may be set by the threshold setting unit 170, which will be described later.

[0073] When a road surface abnormality is detected, the anomaly information generating unit 162 generates road surface abnormality information which is information indicating the road surface abnormality. For example, the anomaly information generating unit 162 may generate road surface abnormality information indicating that a road surface abnormality has been detected. Furthermore, the anomaly information generating unit 162 may generate road surface abnormality information including, for example, location information where road surface observation data in which a road surface abnormality has been detected was observed. Furthermore, the anomaly information generating unit 162 may generate road surface abnormality information in which, for example, a symbol indicating the location where the abnormality was detected is displayed on a map showing the road surface.

[0074] Furthermore, the anomaly information generating unit 162 may generate road surface anomaly information indicating an abnormal location on the road surface, for example. The anomaly information generating unit 162 may generate road surface anomaly information indicating an abnormal location in road surface observation data, for example. If the road surface observation data is image data, the road surface anomaly information may be information indicating an abnormal location in the image data. Note that the abnormal location may correspond to a pixel among the pixels of the road surface observation data where the difference in pixel values ​​between corresponding pixels in the road surface observation data and the road surface prediction data is equal to or greater than a threshold value.

[0075] The anomaly information generating unit 162 may also generate road surface anomaly information indicating the type of the detected anomaly. Specifically, the anomaly information generating unit 162 may determine the type of the anomaly using a trained model trained to input data (image data, etc.) indicating the detected anomaly location and output the type of the anomaly at the anomaly location. For example, the anomaly information generating unit 162 may determine the type of the anomaly using a trained model trained to input image data showing an image of the detected anomaly location (a part of the road surface observation data) and output the type of the object shown in the image. In this case, the type of the anomaly may be determined, for example, by a machine learning algorithm that performs general image recognition. Note that in this case, the type of the anomaly is determined after an anomaly is detected on the road surface. Therefore, compared to the technology of Patent Document 1 that defines all road surface conditions by labels and estimates road surface anomalies, the type of the anomaly can be determined more accurately and robustly.

[0076] FIG. 7 is a diagram for explaining the road surface abnormality detection process according to the present disclosure. FIG. 7 shows a case where the road surface abnormality detection process is performed at time #t. The prediction unit 140 generates road surface prediction data at time #t using road surface observation data of section #M, which is a predetermined number N of road surface observation data before time #t. The difference calculation unit 150 compares the road surface prediction data at time #t with the road surface observation data at time #t to calculate the difference. Then, the abnormality detection unit 160 generates a comparison result according to the difference. If the difference is equal to or greater than the threshold, the abnormality detection unit 160 generates a comparison result indicating that an abnormality in the road surface has been detected. On the other hand, if the difference is not equal to or greater than the threshold, the abnormality detection unit 160 generates a comparison result indicating that the road surface is normal.

[0077] The threshold setting unit 170 sets a threshold used by the difference calculation unit 150. The threshold setting unit 170 sets the threshold so that the difference between the road surface observation data and the road surface prediction data is equal to or greater than the threshold when a phenomenon that may hinder the passage of a subject traveling on the road surface exists on the road surface. Here, the threshold setting unit 170 may set the threshold according to the attributes of the road surface as described above. Then, the abnormality detection unit 160 may detect an abnormality on the road surface using the threshold set according to the attributes of the road surface. With this configuration, it is possible to appropriately determine whether or not the road surface will hinder passage according to the attributes of the road surface.

[0078] When the attributes of the road surface are input to the interface unit 108, the threshold setting unit 170 may set a threshold according to the attributes of the road surface that have been input. The threshold setting unit 170 may set a smaller threshold as the attributes of the road surface make it more difficult for a subject traveling on the road surface to pass. For example, the threshold setting unit 170 may set a smaller threshold as the attributes of the road surface indicate a situation in which a subject traveling on the road surface is more likely to fall. This makes it possible to prevent a subject traveling on the road surface from falling.

[0079] The threshold setting unit 170 may set the threshold according to, for example, the age of a user walking on a road surface. When the user's age is input to the interface unit 108, the threshold setting unit 170 may set the threshold according to the inputted user's age. For example, when the user walking on a road surface is older, the user is more likely to fall compared to a younger user due to a decline in the user's motor skills, even if there is a slight abnormality on the road surface. Therefore, the threshold setting unit 170 may set the threshold to a smaller value when the user walking on the road surface is older. This makes it possible to prevent a user who is prone to falling from falling.

[0080] Furthermore, the threshold setting unit 170 may set the threshold according to, for example, the type of the moving body 50 traveling on the road surface. When the type of the moving body 50 is input to the interface unit 108, the threshold setting unit 170 may set the threshold according to the input type of the moving body 50. For example, when the moving body 50 traveling on the road surface is of a type that is prone to tipping over, the threshold setting unit 170 may set the threshold to a small value. This makes it possible to prevent the moving body 50 that is prone to tipping over from tipping over.

[0081] The threshold setting unit 170 may set the threshold according to the environment of the road surface. For example, the threshold setting unit 170 may set the threshold according to the time period for which the abnormality detection process is performed. When the time period is input to the interface unit 108, the threshold setting unit 170 may set the threshold according to the input time period. For example, in the night time period, the visibility of the road surface is worse than in the daytime, so that even if there is a slight abnormality on the road surface, the subject passing on the road surface is more likely to fall. Therefore, the threshold setting unit 170 may set the threshold to a small value in the case of a dark time period. This makes it possible to prevent the subject passing on the road surface from falling even in the dark time period.

[0082] Furthermore, the threshold setting unit 170 may set the threshold according to, for example, the weather of the target where the abnormality detection process is performed. When the weather is input to the interface unit 108, the threshold setting unit 170 may set the threshold according to the input weather. For example, in a rainy environment, the visibility of the road surface and the condition of the road surface are worse than in a sunny environment, so that even if there is a slight abnormality in the road surface, the subject walking on the road surface is more likely to fall. Therefore, the threshold setting unit 170 may set the threshold to a small value in the case of rainy weather. This makes it possible to prevent a subject walking on the road surface from falling even in rainy weather.

[0083] When an abnormality in the road surface is detected, the output unit 180 performs processing to output road surface abnormality information indicating that an abnormality has been detected in the road surface. The output unit 180 may perform processing to cause the terminal device 40 to output the road surface abnormality information. In this case, the output unit 180 may perform processing to transmit the road surface abnormality information and a command to output the road surface abnormality information to the terminal device 40. In this case, the output unit 180 may be realized by the communication unit 106. Furthermore, the output unit 180 may perform processing to cause the interface unit 108 to output the road surface abnormality information.

[0084] The output unit 180 may perform processing to output road surface abnormality information corresponding to a warning indicating that an abnormality has been detected on the road surface. For example, the output unit 180 may perform processing to display a message indicating the warning. Also, for example, the output unit 180 may perform processing to output a sound indicating the warning. Also, for example, the output unit 180 may perform processing to cause the terminal device 40 to emit light indicating the warning. Also, for example, the output unit 180 may perform processing to cause the terminal device 40 to vibrate indicating the warning.

[0085] The output unit 180 may also perform processing to output road surface anomaly information indicating an abnormal portion of the road surface. In this case, the output unit 180 may perform processing to output road surface anomaly information that is an image that emphasizes a portion detected as an anomaly in the road surface observation data. For example, the output unit 180 may perform processing to output an image in which a portion detected as an anomaly in the road surface observation data is surrounded by a rectangle. The output unit 180 may also perform processing to output road surface anomaly information that is an image in which a symbol indicating a location where an anomaly has been detected is displayed on a map showing the road surface. In other words, the output unit 180 may perform processing to display a symbol indicating a location where an anomaly has been detected on the road surface on a map showing the road surface.

[0086] Furthermore, when road surface anomaly information indicating the type of the detected anomaly is generated, the output unit 180 may perform processing to output the type of the detected anomaly. For example, the output unit 180 may perform processing to display the type of anomaly in the vicinity of a location where an anomaly has been detected in the road surface observation data. Furthermore, for example, the output unit 180 may perform processing to display the type of anomaly detected at a location near a symbol indicating the location where the anomaly was detected on a map showing the road surface. This allows the user to easily recognize the type of anomaly.

[0087] 8 and 9 are diagrams illustrating an example of road surface abnormality information output by the road surface abnormality detection device 100 according to the present disclosure. FIG. 8 illustrates an example of road surface abnormality information that is an image that highlights a portion detected as abnormal in road surface observation data. FIG. 8 shows an image Im1 in which a road surface 702Im is displayed. An abnormal portion 712Im such as a depression exists on the road surface 702Im. In the example of FIG. 8, an emphasis mark 722Im that surrounds the abnormal portion 712Im is superimposed and drawn. This allows the user to easily recognize that an abnormal portion exists on the road surface.

[0088] Fig. 9 illustrates an example of road surface abnormality information, which is an image in which a symbol indicating a location where an abnormality has been detected is displayed on a map showing the road surface. Fig. 9 shows an image Im2 in which a map 704Im showing the road surface is displayed. A symbol 714Im indicating a location where an abnormality has been detected is superimposed on the map 704Im. This allows the user to easily recognize where on the road surface the abnormality exists.

[0089] 10 to 12 are diagrams for explaining a case where the learning process according to the present disclosure is performed in the operation stage. FIG. 10 is a diagram for explaining a case where the learning process time is longer than the frame processing time. The "learning process time" is the time required for the learning process for one frame of road observation data. In other words, the learning process time is the time required for performing the learning process using one frame of road observation data and updating the weights. The "frame processing time" may be the time from acquiring a certain frame of data to acquiring the next frame of data. The frame processing time includes the time required for frame processing for one frame of data (road observation data). The frame processing may be a series of processes from acquiring the frame data to performing an abnormality detection process on the frame data.

[0090] In Figs. 10 to 12, I k indicates frame processing for frame data acquired at time #k. Here, the frame processing includes storage processing, prediction processing, difference calculation processing, and abnormality detection processing, which will be described later. Note that the frame processing time is defined as T I In addition, the learning process time is T l Let us assume that.

[0091] Also, T o indicates the storage processing time. The storage processing time is the time required for the storage processing, which is the processing from when frame data, which is road surface observation data, is acquired until the frame data is stored in the data storage unit 122. eindicates a prediction processing time. The prediction processing time is the time required for the prediction processing executed by the prediction unit 140. In other words, the prediction processing time is the time required until the road surface prediction data at the time #k is generated. d indicates the difference calculation processing time. The difference calculation processing time is the time required for the difference calculation processing executed by the difference calculation unit 150. In other words, the difference calculation processing time is the time required until the difference between the road surface prediction data at time #k and the road surface observation data at time #k is calculated. In addition, T a indicates the abnormality detection processing time. The abnormality detection processing time is the time required for the abnormality detection process executed by the abnormality detection unit 160 for the road surface observation data at time #k. Note that T e , T d , T a are, for example, 1 / 30 sec or less, but are not limited to this.

[0092] Here, in the frame processing, the storage process, the prediction process, the difference calculation process, and the abnormality detection process are executed sequentially. I ≧T o +T e +T d +T a In the example of FIG. 10, the learning process time is longer than the frame process time, so T I <T l In this case, as shown in Fig. 10, the learning process is executed in parallel with the frame process. In this case, while the learning process is being executed for a certain frame data, the prediction process is executed in the frame process using the weights updated in the learning process preceding that learning process.

[0093] In the example of Figure 10, frame processing I t-4 , Frame Processing I t-3 , Frame Processing I t-2 , Frame Processing I t-1 , and Frame Processing I t are executed consecutively. Frame Processing I t-4 In the frame processing I, frame data is acquired at time #t-4.t-4 Frame processing time T I In the frame processing I, the storage process, the prediction process, the difference calculation process, and the abnormality detection process are sequentially executed. On the other hand, when the frame data is acquired at the time #t-4 and stored in the data storage unit 122, the frame processing I t-4 In parallel with this, learning processing begins for the frame data acquired at time #t-4.

[0094] Here, the learning process for the frame data acquired at time #t-4 is frame processing I t-4 Frame processing time T I Therefore, before the learning process for the frame data acquired at time #t-4 is completed, the frame processing I t-4 Next frame processing I t-3 That is, before the learning process for the frame data acquired at time #t-4 is completed, the frame data is acquired at time #t-3 and stored in the data storage unit 122. Therefore, the frame processing I t-3 Now, frame processing I t-4 Frame processing (prediction processing) is performed using the weights used in frame processing I. t-3 When this is completed, the next frame is processed. t-2 That is, at time #t-2, frame data is acquired and stored in data storage unit 122.

[0095] On the other hand, the learning process for the frame data acquired at time #t-4 is frame processing I t-2 Therefore, the frame processing I t-2 In the frame processing I, frame processing (prediction processing) is performed using weights updated by learning processing for frame data acquired at time #t-4. Also, when frame data is acquired at time #t-2 and stored in the data storage unit 122, frame processing I t-2 In parallel with this, learning processing for the frame data acquired at time #t-2 begins.

[0096] Here, the learning process for the frame data acquired at time #t-2 is frame processing I t-2 Frame processing time T I Therefore, before the learning process for the frame data acquired at time #t-2 is completed, the frame processing I t-2 Next frame processing I t-1 That is, before the learning process for the frame data acquired at time #t-2 is completed, the frame data is acquired at time #t-1 and stored in the data storage unit 122. Therefore, the frame processing I t-1 Now, frame processing I t-2 Frame processing (prediction processing) is performed using the weights used in frame processing I. t-1 When this is completed, the next frame processing I t That is, at time #t, frame data is acquired and stored in data storage unit 122.

[0097] On the other hand, the learning process for the frame data acquired at time #t-2 is frame processing I t Therefore, the frame processing I t In the frame processing I, frame processing (prediction processing) is performed using weights updated by learning processing for frame data acquired at time #t-2. Also, when frame data is acquired at time #t and stored in the data storage unit 122, frame processing I t In parallel with this, learning processing is started for the frame data acquired at time #t. Thereafter, frame processing and learning processing are executed in parallel in the same manner.

[0098] FIG. 11 is a diagram for explaining a case where the learning processing time is shorter than the frame processing time. In the example of FIG. 11, since the learning processing time is shorter than the frame processing time, T I >T lOn the other hand, in the example of FIG. 11, if the learning process is included in the frame process, the time required for the frame process will exceed the frame processing time. I <T o +T e +T d +T a +T l In this case, the learning process is executed in parallel with the frame process, as shown in Fig. 11. In this case, while the learning process is being executed for a certain frame data, the prediction process is executed in the frame process using the weights updated in the learning process preceding that learning process.

[0099] In the example of Figure 11, frame processing I t-4 , Frame Processing I t-3 , Frame Processing I t-2 , Frame Processing I t-1 , and Frame Processing I t are executed consecutively. Frame Processing I t-4 In the frame processing I, frame data is acquired at time #t-4. t-4 Frame processing time T I In the frame processing I, the storage process, the prediction process, the difference calculation process, and the abnormality detection process are sequentially executed. On the other hand, when the frame data is acquired at the time #t-4 and stored in the data storage unit 122, the frame processing I t-4 In parallel with this, learning processing begins for the frame data acquired at time #t-4.

[0100] And frame processing I t-4 When this is completed, the next frame processing I t-3 That is, at time #t-3, frame data is acquired and stored in the data storage unit 122. Here, the learning process for the frame data acquired at time #t-4 is frame processing I. t-3 Therefore, the frame processing I t-3In the frame processing I, frame processing (prediction processing) is performed using weights updated by learning processing for frame data acquired at time #t-4. Also, when frame data is acquired at time #t-3 and stored in the data storage unit 122, frame processing I t-3 In parallel with this, learning processing for the frame data acquired at time #t-3 begins.

[0101] And frame processing I t-3 When this is completed, the next frame processing I t-2 That is, at time #t-2, frame data is acquired and stored in the data storage unit 122. Here, the learning process for the frame data acquired at time #t-3 is frame processing I. t-2 Therefore, the frame processing I t-2 In the frame processing I, frame processing (prediction processing) is executed using weights updated by learning processing for frame data acquired at time #t-3. Also, when frame data is acquired at time #t-2 and stored in the data storage unit 122, frame processing I t-2 In parallel with this, the learning process starts. Thereafter, the frame process and the learning process are executed in parallel in the same manner.

[0102] FIG. 12 is a diagram for explaining a case where the learning process time is shorter than the frame process time, and further, the learning process can be executed sequentially with the frame process. In the example of FIG. 12, since the learning process time is shorter than the frame process time, T I >T l Furthermore, in the example of FIG. 12, even if the learning process is executed as part of the frame process, the time required for the frame process does not exceed the frame process time. I ≧T o +T e +T d +T a +T lIn this case, as shown in Fig. 12, the learning process is executed sequentially with the frame process. In this case, while the learning process is being executed for a certain frame data, the prediction process is executed in the frame process using the weights updated in the learning process preceding that learning process.

[0103] In the example of Figure 12, frame processing I t-2 , Frame Processing I t-1 , and Frame Processing I t are executed consecutively. Frame Processing I t-2 In the frame processing I, frame data is acquired at time #t-2. t-2 Frame processing time T I In the frame processing I, a storage process, a prediction process, a difference calculation process, an anomaly detection process, and a learning process are sequentially executed. That is, frame processing for the frame data acquired at time #t-2 and learning processing for the frame data acquired at time #t-2 are sequentially executed. Note that the learning process executed at this time is frame processing I. t-2 Alternatively, the abnormality detection process may be performed using road surface observation data in which no abnormality was detected in the abnormality detection process.

[0104] And frame processing I t-2 When this is completed, the next frame processing I t-1 That is, frame processing I t-1 In the frame processing I, frame data is acquired at time #t-1. t-1 Frame processing time T I In the frame processing I, a storage process, a prediction process, a difference calculation process, an anomaly detection process, and a learning process are sequentially executed. That is, frame processing for the frame data acquired at time #t-1 and learning processing for the frame data acquired at time #t-1 are sequentially executed. Note that the learning process executed at this time is frame processing I. t-1 Alternatively, the abnormality detection process may be performed using road surface observation data in which no abnormality was detected in the abnormality detection process.

[0105] And frame processing I tWhen this is completed, the next frame processing I t That is, frame processing I t In the frame processing I, frame data is acquired at time #t. t Frame processing time T I In the frame processing I, a storage process, a prediction process, a difference calculation process, an anomaly detection process, and a learning process are sequentially executed. That is, frame processing for frame data acquired at time #t and learning processing for frame data acquired at time #t are sequentially executed. Note that the learning process executed at this time is frame processing I. t Alternatively, the abnormality detection process may be performed using road surface observation data in which no abnormality was detected in the abnormality detection process.

[0106] As illustrated in Figs. 10 to 12, the road surface abnormality detection device 100 according to the present disclosure may be configured to execute frame processing and learning processing each time frame data (road surface observation data) is acquired. With such a configuration, the road surface abnormality detection device 100 according to the present disclosure can perform anomaly detection processing in real time by applying the latest weights to the acquired frame data (road surface observation data) while performing online learning processing. As a result, when the user travels along a road surface together with the observation device 30, the road surface abnormality detection device 100 can instantly provide the user with information indicating whether or not an abnormality has been detected on the road surface observed by the observation device 30. This makes it possible to improve convenience for the user.

[0107] As described above, in the road surface abnormality detection device 100 according to the present disclosure, the learning unit 130 may perform machine learning each time road surface observation data is acquired, and update the weights of the learned model. Then, the prediction unit 140 may predict the road surface data using the updated weights. With this configuration, the weights are updated so as to match the state of the road surface at the timing of observation by the observation device 30. Therefore, it is possible to match the learned model to the state of the road surface observed by the observation device 30. As a result, it is possible to perform the prediction process using the learned model that is matched to the state of the road surface observed by the observation device 30. Therefore, it is possible to perform the abnormality detection process more appropriately.

[0108] As described above, in the road surface abnormality detection device 100 according to the present disclosure, the learning unit 130 may perform machine learning using road surface observation data whose difference from the predicted road surface data is less than a threshold value, and update the weights of the trained model. With such a configuration, it is possible to improve the prediction accuracy of the trained model for the road surface observed by the observation device 30. Therefore, it is possible to perform the abnormality detection process more appropriately.

[0109] In addition, in the road surface abnormality detection device 100 according to the present disclosure, the data acquisition unit 120 may acquire road surface observation data by an observation device 30 that moves along the road surface. With such a configuration, it is possible to acquire road surface observation data that changes from moment to moment along the road surface. Here, the fact that the observation device 30 or an entity traveling on the road surface with the observation device 30 was able to pass through the road surface may mean that there is a high possibility that the road surface is free of abnormalities. Therefore, the road surface observation data obtained by observing the passable road surface may be regarded as data corresponding to a normal road surface without abnormalities. Therefore, by performing a learning process using such road surface observation data, the predicted road surface data is more likely to indicate a normal road surface without abnormalities. Therefore, it is possible to more appropriately generate road surface prediction data indicating a normal road surface without defining road surfaces with abnormalities and normal road surfaces by labels. Then, it is possible to determine that road surface observation data with a large difference from the road surface prediction data is abnormal. Therefore, it is possible to more appropriately perform the abnormality detection process.

[0110] In addition, in a case where a user is walking along a road surface with the observation device 30, it is assumed that a step, an obstacle, or the like suddenly appears while the user is walking along a flat road surface. In this case, the road surface abnormality detection device 100 acquires road surface observation data in which the obstacle is observed. Then, the road surface abnormality detection device 100 determines that the difference between the road surface observation data and road surface data predicted using the road surface observation data observed before the obstacle appears is equal to or greater than a threshold. Therefore, the road surface abnormality detection device 100 can detect the presence of an obstacle on the road surface, that is, an abnormality on the road surface. Furthermore, in such a case, the road surface abnormality detection device 100 presents road surface abnormality information representing a warning to the user. Therefore, it is possible to prevent the user from tripping over an obstacle.

[0111] Fig. 13 is a flowchart showing an example of processing executed by road surface abnormality detection device 100 according to the present disclosure. Note that Fig. 13 shows a road surface abnormality detection method executed by road surface abnormality detection device 100 according to the present disclosure. On the other hand, when each component shown in Fig. 5 is realized by cloud computing or a plurality of computers, Fig. 13 is executed by road surface abnormality detection device 100 realized by cloud computing or a plurality of computers. Note that the processing described in Fig. 13 is merely one example. The same applies to other flowcharts.

[0112] The road surface abnormality detection device 100 sets the timing τ of abnormality detection to time #t (step S100). Then, the road surface abnormality detection device 100 acquires road surface observation data at time #t (step S102). Also, the road surface abnormality detection device 100 stores the road surface observation data at time #t (step S106). Specifically, the data acquisition unit 120 acquires road surface observation data that is one frame of data at time #t, as described above. Also, the data storage unit 122 stores road surface observation data that is one frame of data at time #t, as described above. Note that as the time #t is updated, that is, as time passes, the data storage unit 122 stores road surface observation data that is a plurality of frames of data in a time series.

[0113] The road surface abnormality detection device 100 predicts road surface data at time #t using road surface observation data up to time #t-1 (step S108). Specifically, as described above, the prediction unit 140 generates road surface prediction data at time #t by inputting the road surface observation data, which is a plurality of frames of data up to time #t-1, into the trained model.

[0114] The road surface abnormality detection device 100 calculates the difference between the road surface observation data at time #t and the road surface prediction data at time #t (step S110). Specifically, as described above, the difference calculation unit 150 calculates the difference between the road surface observation data, which is frame data at time #t, and the road surface prediction data, which is frame data at time #t, by using, for example, a loss function.

[0115] The road surface abnormality detection device 100 determines whether the difference is equal to or greater than the threshold value Th1 (step S112). Specifically, as described above, the abnormality detection unit 160 determines whether the difference between the road surface observation data, which is frame data at time #t, and the road surface prediction data, which is frame data at time #t, obtained by using, for example, a Loss function, is equal to or greater than the threshold value Th1. Note that the threshold value Th1 is a threshold value set by the threshold setting unit 170.

[0116] If the difference is equal to or greater than threshold value Th1 (YES in S112), road surface abnormality detection device 100 detects an abnormality on the road surface (step S120). Specifically, abnormality detection unit 160 detects an abnormality on the road surface in the road surface observation data at time #t, as described above. Then, road surface abnormality detection device 100 outputs road surface abnormality information (step S122). Specifically, abnormality information generation unit 162 generates road surface abnormality information indicating an abnormality on the road surface, as described above. Then, output unit 180 performs processing for outputting the road surface abnormality information, as described above. Then, the processing flow proceeds to S140.

[0117] On the other hand, if the difference is not equal to or greater than the threshold Th1 (NO in S112), the road surface abnormality detection device 100 determines that the road surface is normal (step S130). Specifically, the abnormality detection unit 160 determines that the road surface is normal in the road surface observation data at time #t, as described above. Then, the road surface abnormality detection device 100 performs a learning process (step S132). Specifically, the learning unit 130 performs a learning process using the road surface observation data determined to be a normal road surface at time #t, as described above, and updates the weights of the trained model. Then, the process flow proceeds to S140. Note that, as described above, the learning unit 130 may perform a learning process in parallel with the above process, using all the acquired road surface observation data. Also, as described above, the learning unit 130 may perform a learning process sequentially to the above process, using all the acquired road surface observation data.

[0118] The road surface abnormality detection device 100 updates the abnormality detection timing τ (step S140). Specifically, the road surface abnormality detection device 100 increments the abnormality detection timing τ by one, and sets #t to #t+1. Then, the process flow returns to S102, and the processes of S102 to S140 are repeated. In this manner, the abnormality detection process is performed on each of the road surface observation data acquired in chronological order.

[0119] (Embodiment 3) Next, the third embodiment will be described with reference to the drawings. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In addition, in each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary. Note that the road surface abnormality detection system 20 according to the third embodiment is substantially similar to that shown in FIG. 3, and therefore explanations are omitted. Also, the hardware configuration of the road surface abnormality detection device according to the third embodiment is substantially similar to that shown in FIG. 4, and therefore explanations are omitted.

[0120] 14 is a diagram illustrating a functional configuration of the road surface abnormality detection device 100 according to the present disclosure. As in the second embodiment, the road surface abnormality detection device 100 has, as its components, a data acquisition unit 120, a data storage unit 122, a learning unit 130, and a model storage unit 132. The road surface abnormality detection device 100 also has, as its components, a prediction unit 140, a difference calculation unit 150, an abnormality detection unit 160, an abnormality information generation unit 162, a threshold setting unit 170, and an output unit 180. Furthermore, in the third embodiment, the road surface abnormality detection device 100 has a conversion unit 220. The conversion unit 220 functions as a conversion means.

[0121] The conversion unit 220 converts the dimension of the vector representing the road observation data into a lower dimension. Specifically, the conversion unit 220 converts the road observation data acquired by the data acquisition unit 120 into low-dimensional vector data. In other words, the conversion unit 220 converts the road observation data into low-dimensional vector data having a dimension lower than the dimension of the vector representing the road observation data. In further other words, the conversion unit 220 generates low-dimensional vector data in which the dimension of the vector representing the road observation data is converted into a lower dimension. Here, the low-dimensional vector data can be said to be road observation data with a reduced dimension. Therefore, the conversion unit 220 converts the road observation data into road observation data that is low-dimensional vector data having a dimension lower than the dimension of the vector representing the road observation data.

[0122] The conversion unit 220 may be realized by, for example, a VAE (Variational Autoencoder) that has been trained in advance. Specifically, the conversion unit 220 may be realized by an encoder among the encoders and decoders that constitute the VAE. In this case, the conversion unit 220 may acquire low-dimensional vector data that is output by inputting road observation data to the encoder of the VAE. That is, the conversion unit 220 may generate low-dimensional vector data in which the original road observation data is abstracted by compressing the dimensions of the road observation data using the VAE. The conversion unit 220 uses the VAE to calculate a normal distribution of the input data, and probabilistically samples components (feature values) from the normal distribution to extract components (feature values) of the low-dimensional vector data. Note that the VAE may be trained in advance using a general dataset such as a public dataset according to the type of input data (road observation data). Alternatively, the VAE may be trained in advance using a dataset that has been uniquely collected according to the type of input data (road observation data). Also, the VAE may be trained in advance so that a dataset is input and the dataset is output. Furthermore, the conversion unit 220 may be realized by an autoencoder other than the VAE. Furthermore, the low-dimensional vector data may correspond to latent variables in which the dimensions of the input data are reduced by an encoder.

[0123] The data storage unit 122 may store low-dimensional vector data obtained by converting the road surface observation data by the conversion unit 220. In other words, the data storage unit 122 may store the road surface observation data converted to low dimensions by the conversion unit 220. In general, the amount of low-dimensional vector data is smaller than the amount of road surface observation data, so resources can be reduced.

[0124] The learning unit 130 may perform machine learning to generate a trained model by using low-dimensional vector data obtained by converting road surface observation data at each of a plurality of timings as input and outputting low-dimensional vector data corresponding to road surface prediction data. That is, the learning unit 130 may perform machine learning to generate a trained model by using converted road surface observation data, which is low-dimensional vector data, at each of a plurality of timings as input and outputting road surface prediction data, which is low-dimensional vector data. The learning unit 130 may perform machine learning to input road surface observation data, which is low-dimensional vector data, at each of a plurality of timings, and output road surface data corresponding to the low-dimensional vector data at a timing next to the plurality of timings. Note that the "triggering time next to the plurality of timings" refers to the timing next to the last timing of the plurality of timings. The learning unit 130 may perform machine learning to input converted road surface observation data corresponding to the plurality of road surface observation data at each of a plurality of timings and output low-dimensional road surface data at a timing next to the plurality of timings. Note that the "low-dimensional road surface data" refers to road surface data corresponding to the dimension of the low-dimensional vector data.

[0125] The prediction unit 140 may predict low-dimensional road surface data corresponding to the number of dimensions of the low-dimensional vector data at the first timing from the converted road surface observation data corresponding to the multiple road surface observation data at multiple timings before the first timing. That is, the prediction unit 140 may predict road surface data of a dimension corresponding to the dimension of the road surface observation data converted to low dimension at the first timing from the road surface observation data converted to low dimension at multiple timings before the first timing. In other words, the prediction unit 140 may generate low-dimensional vector data corresponding to the road surface prediction data at the first timing from the low-dimensional vector data corresponding to the road surface observation data at multiple timings before the first timing. In addition, the "first timing" may correspond to, for example, time #t in FIG. 15 and FIG. 16 described later.

[0126] The difference calculation unit 150 and the abnormality detection unit 160 may perform the abnormality detection process by comparing the converted road surface observation data at the first timing with the predicted low-dimensional road surface data at the first timing. Specifically, the difference calculation unit 150 may calculate the difference between the converted road surface observation data at the first timing and the predicted low-dimensional road surface data at the first timing. That is, the difference calculation unit 150 may calculate the difference between the road surface observation data converted to low dimensions at the first timing and the predicted low-dimensional road surface data at the first timing. In other words, the difference calculation unit 150 may calculate the difference between the low-dimensional vector data corresponding to the road surface observation data at the first timing and the low-dimensional vector data corresponding to the road surface prediction data at the first timing.

[0127] In this case, the abnormality detection unit 160 may detect an abnormality in the road surface when a difference between the converted road surface observation data at the first timing and the predicted low-dimensional road surface data at the first timing is equal to or greater than a threshold. That is, the abnormality detection unit 160 may detect an abnormality in the road surface when a difference between the road surface observation data converted to low dimensions at the first timing and the predicted low-dimensional road surface data at the first timing is equal to or greater than a threshold. In other words, the abnormality detection unit 160 may detect an abnormality in the road surface when a difference between low-dimensional vector data corresponding to the road surface observation data at the first timing and low-dimensional vector data corresponding to the road surface prediction data at the first timing is equal to or greater than a threshold.

[0128] Alternatively, the difference calculation unit 150 and the anomaly detection unit 160 may perform an anomaly detection process by comparing the road surface observation data before conversion at the first timing with the road surface prediction data reconstructed from the low-dimensional road surface prediction data at the first timing to the dimension before conversion. Specifically, the conversion unit 220 may convert the road surface prediction data, which is low-dimensional vector data output from the prediction unit 140, into road surface prediction data of the dimension before conversion by, for example, a decoder of a VAE. The difference calculation unit 150 may calculate the difference between the road surface prediction data of the original dimension reconstructed from the low-dimensional vector data and the road surface observation data before conversion at the first timing. Then, the anomaly detection unit 160 may detect an anomaly when the difference between the road surface prediction data of the original dimension reconstructed from the low-dimensional vector data and the road surface observation data before conversion at the first timing is equal to or greater than a threshold value.

[0129] 15 and 16 are diagrams for explaining processing using low-dimensional vector data according to the present disclosure. FIG. 15 and FIG. 16 show a case where abnormality detection processing of the road surface at time #t is performed. FIG. 15 is a diagram for explaining a case where road surface prediction data, which is low-dimensional vector data, is compared with road surface observation data, which is low-dimensional vector data, in the abnormality detection unit 160. The conversion unit 220 converts road surface observation data of section #M, which is a predetermined number N of road surface observation data before time #t, into road surface observation data, which is low-dimensional vector data. That is, the conversion unit 220 converts the road surface observation data of section #M to obtain low-dimensional vector data of section #M. In addition, the conversion unit 220 converts the road surface observation data at time #t into road surface observation data, which is low-dimensional vector data. That is, the conversion unit 220 converts the road surface observation data at time #t to obtain low-dimensional vector data at time #t.

[0130] Moreover, the prediction unit 140 uses the low-dimensional vector data of the section #M to generate low-dimensional vector data corresponding to the road surface prediction data at time #t. That is, the prediction unit 140 uses the low-dimensional vector data of the section #M to generate predicted low-dimensional vector data at time #t. The difference calculation unit 150 compares the predicted low-dimensional vector data at time #t with low-dimensional vector data corresponding to the road surface observation data at time #t to calculate the difference. The abnormality detection unit 160 generates a comparison result according to the difference. If the difference is equal to or greater than the threshold, the abnormality detection unit 160 generates a comparison result indicating that an abnormality in the road surface has been detected. On the other hand, if the difference is not equal to or greater than the threshold, the abnormality detection unit 160 generates a comparison result indicating that the road surface is normal.

[0131] 16 is a diagram for explaining a case where the anomaly detection unit 160 compares low-dimensional vector data in which predicted low-dimensional vector data is reconstructed to the original dimension with unconverted road surface observation data. The conversion unit 220 converts the road surface observation data of section #M, which is a predetermined number N of road surface observation data prior to time #t, into road surface observation data that is low-dimensional vector data, for example, using a VAE encoder. In other words, the conversion unit 220 converts the road surface observation data of section #M to obtain low-dimensional vector data of section #M.

[0132] The prediction unit 140 uses the low-dimensional vector data of the section #M to generate low-dimensional vector data corresponding to the road surface prediction data at time #t. That is, the prediction unit 140 uses the low-dimensional vector data of the section #M to generate predicted low-dimensional vector data at time #t. Then, the conversion unit 220 converts the low-dimensional vector data corresponding to the road surface prediction data at time #t into road surface prediction data reconstructed to the original dimension, for example, using a VAE decoder. That is, the conversion unit 220 reconstructs the predicted low-dimensional vector data at time #t into road surface prediction data of the original dimension.

[0133] The difference calculation unit 150 compares the road surface prediction data reconstructed into the original dimensions at time #t with the road surface observation data at time #t to calculate the difference. Then, the abnormality detection unit 160 generates a comparison result according to the difference. If the difference is equal to or greater than the threshold, the abnormality detection unit 160 generates a comparison result indicating that an abnormality in the road surface has been detected. On the other hand, if the difference is not equal to or greater than the threshold, the abnormality detection unit 160 generates a comparison result indicating that the road surface is normal.

[0134] As described above, in the road surface abnormality detection device 100 according to the present disclosure, the conversion unit 220 may convert road surface observation data into low-dimensional vector data having a dimension lower than that of a vector representing the road surface observation data. With this configuration, even if the road surface observation data as input data is high-dimensional data such as image data or a normal map, the road surface abnormality detection device 100 can process low-dimensional vector data.

[0135] Specifically, when high-dimensional data is used as input data for a machine learning algorithm such as LSTM, the number of parameters of the trained model may increase. This may have adverse effects such as a decrease in prediction accuracy or difficulty in converging learning. In contrast, by using low-dimensional vector data as input data for a machine learning algorithm, the number of parameters of the trained model can be reduced. This makes it possible to improve prediction accuracy. In addition, it is possible to make it easier for learning to converge.

[0136] As described above, the conversion unit 220 extracts feature quantities sampled from the normal distribution of road surface observation data, and therefore can extract feature quantities that appropriately capture the characteristics of the road surface observation data. Therefore, the parameters of the trained model can be reduced, and the learning can be easily converged. Furthermore, even if the road surface observation data used in the learning stage and the road surface observation data input in the operation stage are significantly different, feature quantities can be robustly extracted from the input road surface observation data. In other words, the robustness of machine learning can be improved. Therefore, the prediction accuracy can be improved.

[0137] Also, as illustrated in FIG. 15, by comparing low-dimensional vector data with each other, the difference calculation unit 150 can appropriately calculate the difference. That is, when calculating the difference by a loss function or the like, if the number of components of the vector data to be compared is large, noise may easily occur. In contrast, the low-dimensional vector data converted by the conversion unit 220 is data in which the dimensions are reduced while capturing the characteristics of the road surface observation data before conversion. Therefore, even when calculating the difference of low-dimensional vector data, a difference according to the magnitude of the difference between the road surface observation data before conversion and the road surface prediction data whose dimensions have not been reduced can be calculated. Therefore, by calculating the difference of low-dimensional vector data, it is possible to appropriately calculate the difference while suppressing the occurrence of noise.

[0138] Furthermore, when it is desired to display abnormal parts of the road surface as a comparison result, the difference calculation unit 150 may calculate the difference between road surface prediction data in which predicted low-dimensional vector data is reconstructed to the original dimension, and unconverted road surface observation data, as exemplified in Fig. 16. This makes it possible to generate road surface abnormality information that highlights abnormal parts, as exemplified in Fig. 8, in the same manner as in the second embodiment. Therefore, it is possible to visualize abnormal parts of the road surface while realizing stable learning and improved prediction accuracy.

[0139] Fig. 17 is a flowchart showing an example of processing executed by the road surface abnormality detection device 100 according to the present disclosure. Fig. 17 shows a road surface abnormality detection method executed by the road surface abnormality detection device 100 according to the present disclosure. As in S100 of Fig. 13, the road surface abnormality detection device 100 sets the timing τ of abnormality detection to time #t (step S200). Then, as in S102 of Fig. 13, the road surface abnormality detection device 100 acquires road surface observation data at time #t (step S202). Specifically, the data acquisition unit 120 acquires the road surface observation data, which is one frame of data at time #t, as described above.

[0140] As described above, the road surface abnormality detection device 100 converts the road surface observation data into low-dimensional vector data (step S204). Specifically, the conversion unit 220 converts the road surface observation data at time #t into road surface observation data that is low-dimensional vector data with reduced dimensions. Then, the road surface abnormality detection device 100 stores the converted road surface observation data at time #t (step S206). That is, the data storage unit 122 stores the road surface observation data that is low-dimensional vector data at time #t. Note that as the time #t is updated as described below, that is, as time passes, the data storage unit 122 stores the road surface observation data that is low-dimensional vector data that is a plurality of frames of data in a time series.

[0141] The road surface abnormality detection device 100 predicts low-dimensional road surface data at time #t using the converted road surface observation data up to time #t-1 (step S208). Specifically, as described above, the prediction unit 140 generates low-dimensional vector data corresponding to the road surface prediction data at time #t by inputting the low-dimensional vector data into which the road surface observation data up to time #t-1 has been converted into the trained model.

[0142] The road surface abnormality detection device 100 calculates the difference between the road surface observation data at time #t and the road surface prediction data at time #t (step S210). Specifically, the difference calculation unit 150 may calculate the difference between the low-dimensional vector data corresponding to the road surface prediction data at time #t and the low-dimensional vector data corresponding to the road surface observation data at time #t, as described above with reference to Fig. 15. Alternatively, the difference calculation unit 150 may calculate the difference between the road surface prediction data in which the predicted low-dimensional vector data at time #t is reconstructed to the original dimension, and the unconverted road surface observation data at time #t, as described above with reference to Fig. 16.

[0143] As in S112 of FIG. 13, the road surface abnormality detection device 100 judges whether the difference is equal to or greater than the threshold Th1 (step S212). Specifically, as described above, the abnormality detection unit 160 may judge whether the difference between the low-dimensional vector data corresponding to the road surface prediction data at time #t and the low-dimensional vector data corresponding to the road surface observation data at time #t is equal to or greater than the threshold Th1. Alternatively, as described above, the abnormality detection unit 160 may judge whether the difference between the road surface prediction data in which the predicted low-dimensional vector data at time #t is reconstructed to the original dimension and the unconverted road surface observation data at time #t is equal to or greater than the threshold Th1. Note that the threshold Th1 used when comparing low-dimensional vector data may be different from the threshold Th1 used when comparing data of the original dimension.

[0144] If the difference is equal to or greater than threshold value Th1 (YES in S212), similar to S120 in Fig. 13, road surface abnormality detection device 100 detects an abnormality on the road surface (step S220). That is, abnormality detection unit 160 detects an abnormality on the road surface from the road surface observation data at time #t. Then, similar to S222 in Fig. 13, road surface abnormality detection device 100 outputs road surface abnormality information (step S222). Then, the processing flow proceeds to S240.

[0145] On the other hand, if the difference is not equal to or greater than the threshold value Th1 (NO in S212), similar to S130 in FIG. 13, the road surface abnormality detection device 100 determines that the road surface is normal (step S230). That is, the abnormality detection unit 160 determines that the road surface is normal for the road surface observation data at time #t. Then, similar to S132 in FIG. 13, the road surface abnormality detection device 100 performs a learning process (step S232). Note that, as described above, the learning unit 130 may perform the learning process in parallel with the above process, using all the acquired road surface observation data. Also, as described above, the learning unit 130 may perform the learning process sequentially to the above process, using all the acquired road surface observation data.

[0146] 13, the road surface abnormality detection device 100 updates the timing τ of abnormality detection (step S240). Then, the process flow returns to S202, and the processes of S202 to S240 are repeated. In this manner, the abnormality detection process is performed on the road surface observation data acquired in chronological order.

[0147] (Modification) Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be appropriately combined with other embodiments.

[0148] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but to one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create an embodiment not explicitly shown or described, for example. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0149] The above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiments when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disk (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0150] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) A data acquisition means for acquiring road surface observation data obtained by observing a road surface in a time series; A prediction means for predicting road surface data at a first time point from a plurality of road surface observation data at a plurality of times points prior to the first time point by using a trained model generated in advance by machine learning; an abnormality detection means for detecting an abnormality in the road surface when a difference between road surface observation data at the first timing and predicted road surface data at the first timing is equal to or greater than a predetermined threshold; A road surface abnormality detection device having the above structure. (Appendix 2) the data acquisition means acquires the road surface observation data by an observation device that moves along the road surface while observing the road surface. 2. A road surface abnormality detection device as described in appendix 1. (Appendix 3) an output means for performing processing to output information indicating that an abnormality has been detected on the road surface when an abnormality has been detected on the road surface; 2. The road surface abnormality detection device according to claim 1, further comprising: (Appendix 4) the output means performs processing so as to display, on a map showing the road surface, a symbol indicating a location on the road surface where the abnormality is detected. 4. A road surface abnormality detection device as described in appendix 3. (Appendix 5) A conversion means for converting the road surface observation data into road surface observation data that is low-dimensional vector data having a dimension lower than that of a vector representing the road surface observation data; and the prediction means predicts low-dimensional road surface data corresponding to a dimension of the low-dimensional vector data at the first time point from converted road surface observation data corresponding to the plurality of road surface observation data at the plurality of time points prior to the first time point; 2. A road surface abnormality detection device as described in appendix 1. (Appendix 6) the abnormality detection means detects an abnormality in the road surface when a difference between the converted road surface observation data at the first timing and the predicted low-dimensional road surface data at the first timing is equal to or greater than the threshold value. 6. A road surface abnormality detection device as described in appendix 5. (Appendix 7) a learning means for performing machine learning to generate the trained model by inputting a plurality of road surface observation data at each of a plurality of timings and outputting road surface data at a timing next to the last timing of the plurality of timings; 2. The road surface abnormality detection device according to claim 1, further comprising: (Appendix 8) the learning means performs the machine learning using the road surface observation data whose difference from the predicted road surface data is less than the threshold value, and updates the weights of the trained model. 8. A road surface abnormality detection device according to claim 7. (Appendix 9) The learning means performs the machine learning each time the road surface observation data is acquired, and updates weights of the trained model; The prediction means predicts the road surface data by using the updated weights. 8. A road surface abnormality detection device according to claim 7. (Appendix 10) the abnormality detection means detects an abnormality in the road surface by using the threshold value set in accordance with an attribute of the road surface. 2. A road surface abnormality detection device as described in appendix 1. (Appendix 11) The output means performs processing to output the type of the detected abnormality. 4. A road surface abnormality detection device as described in appendix 3. (Appendix 12) Obtain road surface observation data obtained by observing the road surface over time, predicting road surface data at a first time point from a plurality of pieces of road surface observation data at a plurality of times points prior to the first time point using a trained model generated in advance by machine learning; detecting an abnormality in the road surface when a difference between the road surface observation data at the first timing and the predicted road surface data at the first timing is equal to or greater than a predetermined threshold; Road surface anomaly detection method. (Appendix 13) acquiring the road surface observation data by an observation device that moves along the road surface while observing the road surface; 13. The road surface abnormality detection method according to claim 12. (Appendix 14) performing processing to output information indicating that an abnormality has been detected on the road surface when an abnormality has been detected on the road surface; 13. The road surface abnormality detection method according to claim 12. (Appendix 15) performing processing so as to display, on a map showing the road surface, a symbol indicating a location on the road surface where the abnormality is detected; 15. The road surface abnormality detection method according to claim 14. (Appendix 16) converting the road surface observation data into road surface observation data that is low-dimensional vector data having a dimension lower than that of a vector representing the road surface observation data; predicting low-dimensional road surface data corresponding to a dimension of the low-dimensional vector data at the first time point from converted road surface observation data corresponding to the plurality of road surface observation data at the plurality of times points prior to the first time point; 13. The road surface abnormality detection method according to claim 12. (Appendix 17) detecting an abnormality in the road surface when a difference between the converted road surface observation data at the first timing and the predicted low-dimensional road surface data at the first timing is equal to or greater than the threshold value; 17. The road surface abnormality detection method according to claim 16. (Appendix 18) generating the trained model by performing machine learning so as to input a plurality of road surface observation data at each of a plurality of timings and output road surface data at a timing next to the last timing of the plurality of timings; 13. The road surface abnormality detection method according to claim 12. (Appendix 19) performing the machine learning using the road surface observation data whose difference with the predicted road surface data is less than the threshold value, and updating weights of the trained model; 19. The road surface abnormality detection method according to claim 18. (Appendix 20) performing the machine learning each time the road surface observation data is acquired, and updating weights of the trained model; predicting the road surface data using the updated weights; 19. The road surface abnormality detection method according to claim 18. (Appendix 21) detecting an abnormality in the road surface by using the threshold value set in accordance with an attribute of the road surface; 13. The road surface abnormality detection method according to claim 12. (Appendix 22) Process to output the type of abnormality detected 15. The road surface abnormality detection method according to claim 14. (Appendix 23) A function to acquire road surface observation data obtained by observing the road surface over time, A function of predicting road surface data at a first timing from a plurality of road surface observation data at a plurality of timings prior to the first timing, using a trained model generated in advance by machine learning; a function of detecting an abnormality in the road surface when a difference between road surface observation data at the first timing and predicted road surface data at the first timing is equal to or greater than a predetermined threshold; A program to make the above happen on a computer. (Appendix 24) a function of acquiring the road surface observation data by an observation device moving along the road surface while observing the road surface; The program described in Appendix 23, which causes the computer to realize the above. (Appendix 25) a function of performing processing to output information indicating that an abnormality has been detected on the road surface when an abnormality is detected on the road surface; The program described in Appendix 23, which causes the computer to realize the above. (Appendix 26) a function of performing processing so as to display, on a map showing the road surface, a symbol indicating a location on the road surface where the abnormality is detected; The program described in Appendix 25, which causes the computer to realize the above. (Appendix 27) A function of converting the road surface observation data into road surface observation data that is low-dimensional vector data having a dimension lower than that of a vector representing the road surface observation data; A function of predicting low-dimensional road surface data corresponding to a dimension of the low-dimensional vector data at the first timing from converted road surface observation data corresponding to a plurality of road surface observation data at a plurality of timings prior to the first timing; The program described in Appendix 23, which causes the computer to realize the above. (Appendix 28) a function of detecting an abnormality in the road surface when a difference between the converted road surface observation data at the first timing and the predicted low-dimensional road surface data at the first timing is equal to or greater than the threshold; The program according to claim 27, which causes the computer to realize the above. (Appendix 29) A function of generating the trained model by performing machine learning so as to input a plurality of road surface observation data at each of a plurality of timings and output road surface data at a timing next to the last timing of the plurality of timings; The program described in Appendix 23, which causes the computer to realize the above. (Appendix 30) A function of performing the machine learning using the road surface observation data whose difference from the predicted road surface data is less than the threshold value, and updating the weights of the trained model; The program described in Appendix 29, which causes the computer to realize the above. (Appendix 31) A function of performing the machine learning each time the road surface observation data is acquired and updating weights of the trained model; A function of predicting the road surface data using the updated weights; The program described in Appendix 29, which causes the computer to realize the above. (Appendix 32) a function of detecting an abnormality in the road surface by using the threshold value set in accordance with an attribute of the road surface; The program described in Appendix 23, which causes the computer to realize the above. (Appendix 33) A function that performs processing to output the type of abnormality detected; The program described in Appendix 25, which causes the computer to realize the above. [Explanation of symbols]

[0151] 1 Road surface abnormality detection device 12 Data Acquisition Section 14 Prediction Department 16 Abnormality detection section 20 Road surface abnormality detection system 30 Observation Equipment 40 Terminal Equipment 50 Mobile 60 Road surface observation data 100 Road surface abnormality detection device 102 Control section 104 Storage section 106 Communications Department 108 Interface section 120 Data Acquisition Section 122 Data storage unit 130 Learning Department 132 Model Storage Unit 140 Prediction Department 150 Difference calculation part 160 Anomaly detection unit 162 Abnormality information generation unit 170 Threshold setting unit 180 Output section 220 Conversion unit

Claims

1. A data acquisition means for acquiring road surface observation data obtained by observing a road surface in a time series; A prediction means for predicting road surface data at a first time point from a plurality of road surface observation data at a plurality of times points prior to the first time point by using a trained model generated in advance by machine learning; an abnormality detection means for detecting an abnormality in the road surface when a difference between road surface observation data at the first timing and predicted road surface data at the first timing is equal to or greater than a predetermined threshold; A road surface abnormality detection device having the above structure.

2. the data acquisition means acquires the road surface observation data by an observation device that moves along the road surface while observing the road surface. The road surface abnormality detection device according to claim 1 .

3. an output means for performing processing to output information indicating that an abnormality has been detected on the road surface when an abnormality has been detected on the road surface; The road surface abnormality detection device according to claim 1 , further comprising:

4. the output means performs processing so as to display, on a map showing the road surface, a symbol indicating a location on the road surface where the abnormality is detected. The road surface abnormality detection device according to claim 3.

5. A conversion means for converting the road surface observation data into road surface observation data that is low-dimensional vector data having a dimension lower than that of a vector representing the road surface observation data; and the prediction means predicts low-dimensional road surface data corresponding to a dimension of the low-dimensional vector data at the first time point from converted road surface observation data corresponding to a plurality of road surface observation data at a plurality of times points prior to the first time point; The road surface abnormality detection device according to claim 1 .

6. the abnormality detection means detects an abnormality in the road surface when a difference between the converted road surface observation data at the first timing and the predicted low-dimensional road surface data at the first timing is equal to or greater than the threshold value. The road surface abnormality detection device according to claim 5.

7. a learning means for performing machine learning to generate the trained model by inputting a plurality of road surface observation data at each of a plurality of timings and outputting road surface data at a timing next to the last timing of the plurality of timings; The road surface abnormality detection device according to claim 1 , further comprising:

8. the learning means performs the machine learning using the road surface observation data whose difference from the predicted road surface data is less than the threshold value, and updates the weights of the trained model. The road surface abnormality detection device according to claim 7.

9. Obtain road surface observation data obtained by observing the road surface over time, predicting road surface data at a first time point from a plurality of pieces of road surface observation data at a plurality of times points prior to the first time point using a trained model generated in advance by machine learning; detecting an abnormality in the road surface when a difference between the road surface observation data at the first timing and the predicted road surface data at the first timing is equal to or greater than a predetermined threshold; Road surface anomaly detection method.

10. A function to acquire road surface observation data obtained by observing the road surface over time, A function of predicting road surface data at a first timing from a plurality of road surface observation data at a plurality of timings prior to the first timing, using a trained model generated in advance by machine learning; a function of detecting an abnormality in a road surface when a difference between road surface observation data at the first timing and predicted road surface data at the first timing is equal to or greater than a predetermined threshold; A program to make the above happen on a computer.

Images