Classification device, classification method, and program

Abstract

The classification device (2000) acquires waterfall data (10) indicating the amplitude of vibration at each time point and each detection location of a vibration sensor (30) arranged along a target object (40), performs semantic segmentation on the waterfall data (10), generates class data indicating a normal class or an abnormal class for each element of the waterfall data (10), and classifies the detection locations into monitored locations and non-monitored locations. The normal class is assigned to elements for which the detection location is presumed to be a monitored location. The abnormal class is assigned to elements for which the detection location is presumed to be a non-monitored location. A monitored location is a detection location arranged along the target object (40). A non-monitored location is a detection location not arranged along the target object (40).

Description

Technical Field

[0001] The present disclosure relates generally to a classification device, a classification method, and a non-transitory computer-readable storage medium.

Background Art

[0002] There is a technique for monitoring an object such as a road using a vibration sensor. Patent Document 1 discloses a technique for acquiring waterfall data indicating the amplitude of vibrations detected by a vibration sensor for each of a plurality of positions and each of a plurality of time points using a distributed acoustic sensing (DAS) system as the vibration sensor.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 does not teach the case where some locations of the vibration sensor are not arranged along the monitoring object. An object of the present disclosure is to provide a new technique for processing data obtained from a vibration sensor installed for monitoring an object.

Means for Solving the Problems

[0005] The present disclosure provides a classification device including at least one memory configured to store instructions and at least one processor. At least one processor is configured to execute instructions to obtain waterfall data indicating the amplitude of vibration at each time point and each detection location of vibration sensors arranged along a target object, perform semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, the normal class is assigned to the elements for which it is inferred that the detection location is a monitored location, the abnormal class is assigned to the elements for which it is inferred that the detection location is a non-monitored location, the monitored location is the detection location arranged along the target object, the non-monitored location is the detection location not arranged along the target object, and classify the detection locations into the monitored locations and the non-monitored locations based on the class data.

[0006] The present disclosure further provides a classification method executed by a computer. The classification method includes obtaining waterfall data indicating the amplitude of vibration at each time point and each detection location of vibration sensors arranged along a target object, performing semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, the normal class is assigned to the elements for which it is predicted that the detection location is a monitored location, the abnormal class is assigned to the elements for which it is inferred that the detection location is a non-monitored location, the monitored location is the detection location arranged along the target object, the non-monitored location is the detection location not arranged along the target object, and classifying the detection locations into the monitored locations and the non-monitored locations based on the class data.

[0007] The present disclosure further provides a non-transitory computer-readable storage medium storing a program. The program causes a computer to acquire waterfall data indicating the amplitude of vibration at each time point and each detection location of vibration sensors arranged along a target object, perform semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the elements for which the detection location is predicted to be a monitored location, the abnormal class being assigned to the elements for which the detection location is inferred to be a non-monitored location, the monitored location being the detection location arranged along the target object, the non-monitored location being the detection location not arranged along the target object, and classify the detection locations into the monitored locations and the non-monitored locations based on the class data.

Effect of the Invention

[0008] According to the present disclosure, a novel technique for processing data obtained from vibration sensors installed for monitoring a target object is provided.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0010] Embodiments according to the present disclosure will be described below with reference to the drawings. The same reference numerals are assigned to the same elements throughout the drawings, and redundant descriptions will be omitted as necessary. Further, a predetermined piece of information (for example, a predetermined value or a predetermined threshold) is stored in advance in a storage device accessible by a computer that uses the information, unless otherwise specified.

[0011] Embodiment 1 <Overview> FIG. 1 shows an overview of the classification device 2000 of Embodiment 1. Note that the overview shown in FIG. 1 shows an example of the operation of the classification device 2000 for easy understanding, and does not limit or narrow the range of operations that the classification device 2000 can perform.

[0012] The classification device 2000 is configured to process the waterfall data 10 indicating the amplitude of the vibration detected by the vibration sensor 30 for each of two or more locations in the vibration sensor 30 and for each of two or more time points. In some embodiments, the waterfall data 10 may be two or more detection data 20 in time series. The detection data 20 is generated by the vibration sensor 30 and indicates the amplitude of the vibration detected at each of two or more locations of the vibration sensor 30 at a certain time point.

[0013] The vibration sensor 30 is installed (arranged) along a monitoring target such as a road. Hereinafter, the monitoring target by the vibration sensor 30 is referred to as "target object 40".

[0014] An example of the vibration sensor 30 is a DAS system including a DAS device and an optical fiber cable. When a DAS system is adopted as the vibration sensor 30, the optical fiber cable is arranged along the target object 40 and attached to the DAS device. The DAS device is configured to transmit a laser pulse via the optical fiber cable and receive the reflection of the transmitted laser pulse.

[0015] Vibration generated at a specific location of the target object 40 affects the laser pulse traveling through that location of the target object 40 at that time. Therefore, the DAS device can measure the amplitude of the vibration generated at that location of the target object 40 by analyzing the reflection of the transmitted laser pulse. Accordingly, the DAS device can generate detection data 20 indicating the amplitudes of the vibrations detected at each of two or more locations of the optical fiber cable. Hereinafter, the location where the amplitude of the vibration is detected in the vibration sensor 30 (for example, the optical fiber cable) is referred to as the "detection location".

[0016] In some implementations, the waterfall data 10 may be formed as matrix data denoted as W. The rows of the matrix data W may represent time points, and the columns may represent the detection locations of the vibration sensor 30. In this case, the element at the i-th row and j-th column of the waterfall data 10 (i.e., W[i][j]) represents the amplitude of the vibration detected at the j-th detection location of the vibration sensor 30 at the i-th time point.

[0017] Furthermore, the sequence of elements in the i-th row of the waterfall data 10 (i.e., {W[i][0], W[i][1],..., W[i][M]}, where M is the total number of detection locations indicated by the waterfall data 10) represents the detection data 20 generated at the i-th time point. On the other hand, the sequence of elements in the j-th column of the waterfall data 10 (i.e., {W[0][j], W[0][j],..., W[N][j]}, where N is the total number of time points indicated by the waterfall data 10) represents the time series of the amplitudes of the vibrations detected at the j-th detection location.

[0018] The waterfall data 10 formed as matrix data can be processed as image data called a "waterfall image" hereinafter. In this case, W[i][j] corresponds to the value of the pixel (i,j) of the waterfall image. Assume that the amplitude of the vibration detected by the vibration sensor 30 is quantized and normalized in the range from 0 to 255. In this case, the waterfall data 10 can be formed as a grayscale image. However, the waterfall data 10 is not necessarily processed as image data.

[0019] The vibration sensor 30 may include one or more detection locations that are not arranged along the target object 40 (in other words, not located at an appropriate position for monitoring the vibration of the target object 40). For example, as shown in FIG. 1, the vibration sensor 30 may have some additional segments 32. The amplitudes of the vibrations detected at the detection locations within these additional segments 32 do not accurately indicate the amplitude of the vibration of the target object 40. Hereinafter, the detection location arranged along the target object 40 is called a "monitoring location", and the detection location not arranged along the target object 40 (for example, the detection location included in the additional segment 32) is called a "non-monitoring location".

[0020] Taking into account the existence of non-monitored locations, the classification device 2000 is configured to detect the monitored locations of the vibration sensor 30 using the waterfall data 10. Specifically, the classification device 2000 acquires the waterfall data 10 and performs semantic segmentation on the waterfall data 10. Semantic segmentation is a technique for analyzing a set of two or more data and classifying them into two or more classes. By semantic segmentation, one of two or more classes (in other words, types) is assigned to each element of the waterfall data 10.

[0021] The classes can include "NORMAL" and "ABNORMAL". "NORMAL" is assigned to the elements of the waterfall data 10 that are presumed to indicate the amplitude of the vibration detected at the monitored locations of the vibration sensor 30. On the other hand, "ABNORMAL" is assigned to the elements of the waterfall data 10 that are presumed to indicate the amplitude of the vibration detected at the non-monitored locations of the vibration sensor 30.

[0022] As a result of the semantic segmentation of the waterfall data 10, the classification device 2000 generates class data indicating the class of each element of the waterfall data 10. The class data can be formed in the same way as the waterfall data 10. Assume that the waterfall data 10 is formed as an NxM matrix denoted as W. In this case, the class data can be formed as an NxM matrix denoted as C (where C[i][j] indicates the class assigned to W[i][j]).

[0023] Based on the class data, the classification device 2000 determines which detection locations are monitoring locations. That is, the classification device 2000 classifies the detection locations into monitoring locations and non-monitoring locations based on the class data. When the waterfall data 10 is matrix data representing the detection locations as columns, it can be rephrased as "the classification device 2000 determines, based on the class data, which columns of the waterfall data 10 indicate the amplitudes of the vibrations detected at the monitoring locations." That is, the classification device 2000 classifies the columns of the waterfall data 10 into columns corresponding to the monitoring locations and columns corresponding to the non-monitoring locations based on the class data. Some other examples of classification devices that determine the monitoring locations and non-monitoring locations, or parts of the monitoring locations and non-monitoring locations, can be analysis techniques based on statistical measures of the waterfall data.

[0024] <Example of the effect> According to the classification device 2000 of Embodiment 1, by semantic segmentation, the elements of the waterfall data 10 are classified into a normal class and an abnormal class, and based on the classification results of the elements of the waterfall data 10, the detection locations of the vibration sensor 30 are classified into monitoring locations and non-monitoring locations. This is a novel method for processing the waterfall data 10 obtained from the vibration sensor that monitors the object.

[0025] As will be described in detail later, classifying the detection locations into monitoring locations and non-monitoring locations is useful in various ways. For example, the classification device 2000 can remove the influence of non-monitoring locations where the amplitude of the vibration of the target object 40 cannot be accurately measured from the waterfall data 10 by removing the region of the non-monitoring locations from the waterfall data 10.

[0026] Hereinafter, the classification device 2000 will be described in more detail.

[0027] <Example of the functional configuration> FIG. 2 is a block diagram showing an example of the functional configuration of the classification device 2000 according to Embodiment 1. The classification device 2000 includes an acquisition unit 2020, a segmentation unit 2040, and a classification unit 2060. The acquisition unit 2020 acquires the waterfall data 10. The segmentation unit 2040 performs semantic segmentation on the waterfall data 10 to generate class data indicating one of two or more classes for each element of the waterfall data 10. The classification unit 2060 classifies the detection locations of the vibration sensor 30 into monitored locations and non-monitored locations based on the class data.

[0028] <Example of Hardware Configuration> The classification device 2000 can be implemented by one or more computers. Each of the one or more computers may be a dedicated computer manufactured to implement the classification device 2000, or a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0029] The classification device 2000 can be realized by installing an application on a computer. The application is implemented by a program that causes the computer to function as the classification device 2000. In other words, the program is an implementation form of the functional units of the classification device 2000 illustrated in FIG. 2.

[0030] FIG. 3 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes the classification device 2000 according to Embodiment 1. In FIG. 3, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120.

[0031] Bus 1020 is a data transmission channel for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DAP), or a Field-Programmable Gate Array (FPGA). The memory 1060 is a main memory element such as a Random Access Memory (RAM) or a Read Only Memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a Solid State Drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a Local Area Network (LAN) or a Wide Area Network (WAN).

[0032] The hardware configuration of the computer 1000 is not limited to that shown in FIG. 3. For example, as described above, the classification device 2000 may be realized as a combination of a plurality of computers. In this case, those computers may be connected to each other via a network.

[0033] <Flow of processing> FIG. 4 is a flowchart showing an exemplary flow of processing by the classification device 2000 according to Embodiment 1. The acquisition unit 2020 acquires the waterfall data 10 (S102). The segmentation unit 2040 performs semantic segmentation on the waterfall data 10 to generate class data (S104). The classification unit 2060 classifies the detected locations into monitored locations and non-monitored locations (S106).

[0034] <Obtaining Waterfall Data 10: S102> The obtaining unit 2020 obtains the waterfall data 10 (S102). As described above, the waterfall data 10 represents the time series of the detection data 20. In some embodiments, the obtaining unit 2020 can obtain the time series of these detection data 20 as the waterfall data 10 by obtaining two or more detection data 20 at different times. In other words, the obtaining unit 2020 converts the two or more obtained detection data 20 into the waterfall data 10.

[0035] There are various methods for obtaining the detection data 20. In some embodiments, the vibration sensor 30 stores the detection data 20 in a storage device accessible to the classification device 2000. In this case, the obtaining unit 2020 can access the storage device to obtain the detection data 20. In other embodiments, the vibration sensor 30 transmits the detection data 20 to the classification device 2000. In this case, the obtaining unit 2020 can obtain the detection data 20 by receiving the detection data 20 transmitted by the vibration sensor 30. Note that the obtaining unit 2020 may obtain two or more detection data 20 one by one or simultaneously.

[0036] The conversion of two or more detection data 20 into the waterfall data 10 may be performed in advance by another computer. In this case, the obtaining unit 2020 can obtain the waterfall data 10 at once.

[0037] <Semantic Segmentation: S104> The segmentation unit 2040 performs semantic segmentation on the waterfall data 10 (S104). As described above, the classification device 2000 can handle two classes called "normal" and "abnormal". In this case, as a result of the semantic segmentation, one of these two classes is assigned to each element of the waterfall data 10.

[0038] There are various ways to perform semantic segmentation on the waterfall data 10. In some embodiments, a machine learning-based model called a "segmentation model" is used to perform semantic segmentation on the waterfall data 10. The segmentation model can be configured to take the waterfall data 10 as input, analyze the waterfall data 10 to determine the class of each element, and output class data. The analysis of the waterfall data 10 can include extracting features from the waterfall data 10 and upsampling the extracted features to the same size as the input data to generate class data.

[0039] The segmentation model can be implemented as one of various types of machine learning-based models such as a neural network. Some examples of neural networks suitable for implementing the segmentation model are U-net, region based convolution neural network (R-CNN), Fast R-CNN, and Faster R-CNN.

[0040] The segmentation model is trained prior to the operation phase (in other words, the test phase) of the classification device 2000. Hereinafter, the computer that trains the segmentation model is referred to as a "training device". The training device may be the classification device 2000 or another device.

[0041] To train the segmentation model, the training device uses a training data set that includes two or more pieces of training data. The training data can be formed as a combination of training input data and ground truth data. The training input data represents the waterfall data, and the ground truth data represents the class data corresponding to the training input data. Specifically, the ground truth data is class data, and each of its elements indicates the class to which the corresponding element of the training input data should be assigned.

[0042] There are various well-known techniques for training a machine learning-based model using a training dataset, and any one of these techniques can be applied to the training device to train the segmentation model. For example, the training device inputs training input data into the segmentation model and obtains class data from the segmentation model. Then, the training device updates the trainable parameters (e.g., the weights and biases of the edges of the neural network) of the segmentation model based on the loss representing the magnitude of the difference between the ground truth data and the class data output from the segmentation model. The training device trains the segmentation model by repeatedly updating the segmentation model using a plurality of training data in the training dataset.

[0043] Note that the size of the data that the segmentation model can process at one time may be smaller than the size of the waterfall data 10. In this case, the segmentation unit 2040 divides the waterfall data 10 into two or more partial data (referred to as "patches") having the same size as the input size of the segmentation model. Then, the segmentation unit 2040 inputs the patches into the segmentation model to obtain the class data of each patch. The segmentation unit 2040 can obtain the class data of the entire waterfall data 10 by concatenating the class data of each patch.

[0044] In some embodiments, the segmentation unit 2040 can further take into account one or more measurement conditions in order to perform semantic segmentation on the waterfall data 10. For example, the measurement conditions may include the period during which the vibration sensor 30 performs measurements to generate the waterfall data 10. In another example, the measurement conditions may include one or more weather conditions such as the weather type (e.g., sunny, cloudy, or rainy), temperature, or humidity during the measurement by the vibration sensor 30. In another example, the measurement conditions may include parameters related to the vibration sensor 30 such as the sensitivity of the vibration sensor 30.

[0045] When one or more measurement conditions are used for semantic segmentation, the segmentation model can be further configured to obtain one or more measurement conditions as inputs. Further, the segmentation model can be further configured to extract features from each of the waterfall data 10 and the measurement conditions, calculate combined features by combining the extracted features, and upsample the combined features to the same size as the input data to generate class data.

[0046] The segmentation model needs to be trained using not only the waterfall data but also the measurement conditions. Therefore, the training input data further includes the measurement conditions in addition to the waterfall data so that the training device can train the segmentation model using the measurement conditions.

[0047] To use the measurement conditions for semantic segmentation, the acquisition unit 2020 acquires the measurement conditions. For example, the acquisition unit 2020 may acquire the measurement conditions from a storage device that stores the measurement conditions in advance and is accessible to the classification device 2000. In another example, the measurement conditions may be transmitted from another computer to the classification device 2000, and the acquisition unit 2020 may receive the measurement conditions.

[0048] <Classification of Detection Location: S106> Based on the class data, the classification unit 2060 classifies the detection location into a monitored location and a non-monitored location (S106). Conceptually, the more the elements of the waterfall data 10 corresponding to the detection location are classified into the "normal" class, the higher the probability that the detection location is a monitored location.

[0049] Since the waterfall data 10 includes two or more elements for each detection location, both the "normal" class and the "abnormal" class can be assigned to the elements of the waterfall data 10 corresponding to the same detection location. To address this situation, the classification unit 2060 may determine whether a detection location is a monitoring location based on the number of elements of the waterfall data 10 corresponding to the detection locations to which the "normal" class is assigned for each detection location. Specifically, the classification unit 2060 may use the class data to identify the number of elements of the waterfall data 10 corresponding to the detection locations to which the "normal" class is assigned for each detection location, and determine whether the identified number is equal to or greater than a predetermined threshold value.

[0050] If the identified number is equal to or greater than the threshold value, the classification unit 2060 determines that the detection location is a monitoring location. On the other hand, if the identified number is less than the threshold value, the classification unit 2060 determines that it is a non-monitoring location.

[0051] Assume that the waterfall data 10 is NxM matrix data where the columns represent detection locations and the rows represent time points. Further, the above-described threshold value is set to T. In this case, the classification unit 2060 identifies the number of elements (denoted as C[j]) to which the "normal" class is assigned in column j (1 <= j <= M) of the waterfall data 10 for each column j. If C[j] >= T (that is, in column j, the "normal" class is assigned to T or more elements), it is determined that the detection location corresponding to column j is a monitoring location. If C[j] < T (that is, in column j, the "normal" class is assigned to less than T elements), it is determined that the detection location corresponding to column j is a non-monitoring location.

[0052] In other embodiments, instead of the number of elements in column j of the waterfall data 10 to which the "normal" class is assigned, the ratio of the elements in column j of the waterfall data 10 to which the "normal" class is assigned (denoted as P[j]) may be used to detect the monitoring location. P[j] can be defined as "P[j]=C[j] / N". In this case, the classification unit 2060 may compare P[j] with a predetermined threshold for each column j, and determine whether the detection location corresponding to column j is a monitoring location or a non-monitoring location.

[0053] In other embodiments, the classification unit 2060 may consider the total number of monitoring locations in the vibration sensor 30. If the length of the target object 40 is known in advance, the total number of monitoring locations in the vibration sensor 30 can be estimated, and this total number is denoted as Ns. Assume that the length of the target object 40 is L [m] and the interval between monitoring locations is defined in advance as a [m]. In this case, the total number of monitoring locations can be estimated as Ns = L / a.

[0054] Further, the classification unit 2060 may sort the detection locations in descending order of the likelihood of being a monitoring location considering the total number of monitoring locations, and determine the detection locations from the 1st to (L / a)th as monitoring locations. The remaining detection locations are determined to be non-monitoring locations. The likelihood that a detection location is a monitoring location can be represented by the number of elements of the waterfall data 10 corresponding to the detection location to which the "normal" class is assigned, and can be represented by C[j] in the case exemplified above, for example.

[0055] The classification unit 2060 can generate information called "detection location information" indicating the result of the classification of the detection locations. Specifically, the detection location information can indicate two lists called "monitoring location list" and "non-monitoring location list". The monitoring location list indicates the identifiers of the detection locations classified as monitoring locations. On the other hand, the non-monitoring location list indicates the identifiers of the detection locations classified as non-monitoring locations.

[0056] <Example of using detection location information> The detection location information can be used in various ways. For example, the classification device 2000 may use the detection location information to remove the influence of non-monitored locations from the waterfall data 10. Specifically, the classification device 2000 can obtain time-series data representing the amplitude of vibration at each monitored location, that is, the amplitude of vibration at each point of the target object 40, by removing the elements of the waterfall data 10 corresponding to non-monitored locations. Hereinafter, this time-series data is referred to as "monitoring data".

[0057] FIG. 5 shows a method of generating monitoring data based on the waterfall data 10. In this example, the waterfall data 10 is matrix data in which columns represent detection locations and rows represent time points. The classification device 2000 executes steps S102 to S106 to classify the detection locations into monitored locations and non-monitored locations. In FIG. 5, the columns of non-monitored locations are shaded in a diagonal stripe pattern.

[0058] The classification device 2000 removes the columns of non-monitored locations from the waterfall data 10 and concatenates the non-removed columns into a single matrix data. This matrix data is treated as the monitoring data 50.

[0059] The classification device 2000 can identify one or more positions of the target object 40 using the detection location information. Specifically, the classification device 2000 can identify the interval between monitored locations by dividing the length of the target object 40 by the number of monitored locations. When the interval between monitored locations is identified as I [m], the position of the target object 40 corresponding to the k-th monitored location can be identified as being at I*k [m] from the start point of the target object 40. By applying the result of position identification to the monitoring data 50, the classification device 2000 can correct the monitoring data 50 to show the time series of the amplitude of vibration at each position of the target object 40 corresponding to the monitored locations.

[0060] The detection location information may be used not only for the current waterfall data 10 but also for the waterfall data 10 to be acquired in the future. Thereby, the classification device 2000 can avoid frequently performing the classification of the detection location, and thereby reduce the computer resources used by the classification device 2000.

[0061] FIG. 6 is a flowchart showing an exemplary flow of processing executed by the classification device 2000 that uses detection location information in future processing.

[0062] The classification device 2000 acquires the waterfall data 10 generated by the vibration sensor 30 (S202), and determines whether the detection location information is stored in the storage device (S204).

[0063] If it is determined that the detection location information is not stored in the storage device (S204: NO), the classification device 2000 performs semantic segmentation on the waterfall data 10 (S206), classifies the detection location (S208). Next, the classification device 2000 generates detection location information and stores it in the storage device (S210). The classification device 2000 generates the monitoring data 50 from the waterfall data 10 based on the detection location information (S212).

[0064] In step S204, if it is determined that the detection location information is stored (S204: YES), the classification device 2000 acquires the detection location information from the storage device (S214), and generates the monitoring data 50 from the waterfall data 10 based on the detection location information (S212).

[0065] Here, in order to regenerate the detection location information at some point, an expiration date can be set for the detection location information. In this case, the classification device 2000 also determines whether the detection location information in the storage device is valid based on the expiration date. Then, the classification device 2000 generates the monitoring data 50 using the detection location information only when valid detection location information is stored in the storage device. Otherwise, the classification device 2000 executes steps S206 to S210 to generate new detection location information.

[0066] <Output of Results> The classification device 2000 may be configured to output one or more pieces of information (collectively referred to as "output information") related to the result of the classification of the detection location. The output information may include the detection location information, the monitoring data 50, or both.

[0067] Here, there are various methods for outputting the output information. In some implementation modes, the output information may be stored in the storage device, displayed on the display device, or transmitted to another computer (such as the user's PC or smartphone of the classification device 2000).

[0068] Embodiment 2 <Overview> FIG. 7 shows an overview of the classification device 2000 according to Embodiment 2. The overview shown in FIG. 7 shows an example of the operation of the classification device 2000 according to Embodiment 2 in order to make it easier to understand the classification device 2000 according to Embodiment 2. Note that it does not limit or narrow the range of operations that the classification device 2000 according to Embodiment 2 can perform.

[0069] In this embodiment, it is assumed that one or more moving objects 70 that vibrate the target object 40 are on the target object 40. For example, when the target object 40 is a road, the moving object 70 may be a vehicle (e.g., a car or a motorcycle) traveling on the road. Further, it is assumed that the waterfall data 10 is formed as matrix data in which columns represent detection locations and rows represent time points (or vice versa). Unless otherwise specified, the columns of the waterfall data 10 represent detection locations and the rows represent time points.

[0070] Under the above assumptions, the classification device 2000 detects the trajectory (e.g., time series of positions) of one or more moving objects 70 from the waterfall data 10, and uses the detected trajectory to correct the detection location information (i.e., the result of classification of the detection locations executed by the classification unit 2060). That is, some of the detection locations classified as monitored locations by the classification unit 2060 may be reclassified as non-monitored locations, some of the detection locations classified as non-monitored locations by the classification unit 2060 may be reclassified as monitored locations, or both. Hereinafter, the trajectory of the moving object 70 will be referred to as an "object trajectory".

[0071] It is considered that the closer the detection location is to the position of the moving object 70, the larger the amplitude of the vibration detected at that detection location. Accordingly, the object trajectory can be detected based on the amplitude of the vibration indicated by the waterfall data 10.

[0072] FIG. 8 shows the object trajectory detected from the waterfall data 10. In this example, the waterfall data 10 is formed as a waterfall image 60 in which the X-axis represents detection locations and the Y-axis represents time points. The waterfall image 60 is shown as a grayscale image in which the larger the amplitude of the vibration corresponding to the pixel, the larger the pixel value. For the sake of illustration, darker colors are drawn with denser and larger dots. The object trajectory 80 is drawn as a white line superimposed on the waterfall image 60.

[0073] As shown in FIG. 8, the object trajectory 80 can be discontinuous (i.e., broken) due to the existence of non-monitored locations as shown in FIG. 8. When the detected locations are not correctly classified into monitored locations and non-monitored locations, if the region corresponding to the non-monitored location is removed from the waterfall data 10, the object trajectory 80 becomes discontinuous.

[0074] FIG. 9 shows the object trajectory 80 when the detected locations are not correctly classified. The waterfall image 60 includes an abnormal section 90 that is a region of one or more continuous non-monitored sections. In the case shown in FIG. 9, when the classification device 2000 removes the abnormal section 90 from the waterfall image 60 to generate the monitoring data 50, the object trajectory 80 becomes discontinuous.

[0075] On the other hand, when the detected locations are correctly classified into monitored locations and non-monitored locations, if the region corresponding to the non-monitored location is removed from the waterfall data 10, the object trajectory 80 becomes continuous. FIG. 10 shows the object trajectory 80 when the detected locations are correctly classified. In the case shown in FIG. 10, when the classification device 2000 removes the abnormal section 90 from the waterfall image 60 to generate the monitoring data 50, the object trajectory 80 becomes continuous within the monitoring data 50.

[0076] In consideration of the above, the classification device 2000 of Embodiment 2 corrects the detected location information based on the object trajectory 80 detected from the waterfall data 10. Specifically, the start point, end point, or both of the abnormal section 90 are corrected so that the object trajectory 80 is substantially continuous before and after the abnormal section 90 (in other words, when the abnormal section 90 is removed from the waterfall data 10, the object trajectory 80 becomes substantially continuous).

[0077] <Example of operational effect> According to the classification device 2000 of Embodiment 2, the detected location information is corrected based on the object trajectory 80, which is the trajectory of the moving object 70 moving on the target object 40. As a result, errors in the detected location information due to misclassification can be reduced, and the detected location information can be made more accurate.

[0078] Hereinafter, the classification device 2000 will be described in more detail.

[0079] <Example of functional configuration> FIG. 11 is a block diagram showing an example of the functional configuration of the classification device 2000 according to Embodiment 2. Similar to the classification device 2000 of Embodiment 1, the classification device 2000 of Embodiment 2 includes an acquisition unit 2020, a segmentation unit 2040, and a classification unit 2060. Further, the classification device 2000 of Embodiment 2 further includes a detection unit 2080 and a correction unit 2100. The detection unit 2080 detects one or more object trajectories 80 from the waterfall data 10. The correction unit 2100 corrects the detection location information based on the object trajectory 80.

[0080] <Example of hardware configuration> The classification device 2000 of Embodiment 2 may be implemented in the same manner as the method by which the classification device 2000 of Embodiment 1 is realized. For example, the classification device 2000 of Embodiment 2 is realized by the computer 1000 shown in FIG. 3. However, the storage device 1080 of Embodiment 2 includes a program for implementing the functions of the classification device 2000 of Embodiment 2.

[0081] <Flow of processing> FIG. 12 is a flowchart showing an exemplary flow of processing executed by the classification device 2000 of Embodiment 2. The classification device 2000 of Embodiment 2 can execute steps S102 to S106 in the same manner as in Embodiment 1. After the execution of step S106, the detection unit 2080 detects the object trajectory 80 from the waterfall data 10 (S302), and corrects the detection location information based on the object trajectory 80 (S304).

[0082] <Detection of object trajectory 80: S302> The detection unit 2080 detects one or more object trajectories 80 from the waterfall data 10 (S302). Methods for detecting the trajectory of the moving object 70 from the time-series data indicating the amplitudes of vibrations at two or more positions are well known, and one of these methods can be applied to the detection unit 2080. For example, the detection unit 2080 can detect one or more positions (i.e., detection locations) at which the moving object 70 is presumed to be located in each of the detection data 20 in the waterfall data 10. Specifically, the detection data 20 may have one or more maximum points, and the detection location corresponding to the maximum point is presumed to be the position of the moving object 70. Then, the object trajectory 80 is detected by connecting the detected positions of the moving object 70 over time.

[0083] <Correction of Detection Location Information: S304> The correction unit 2100 corrects the detection location information generated by the classification unit 2060 (S304). To that end, the classification device 2000 identifies one or more abnormal sections 90 from the waterfall data 10 based on the detection location information. Next, the correction unit 2100 corrects the detection location information by correcting the start point, end point, or both of the abnormal section 90 so that the object trajectory 80 is substantially continuous between the abnormal sections 90 for each abnormal section 90.

[0084] When there is a single object trajectory 80 crossing the abnormal section 90, the correction unit 2100 may correct the detection location information by specifying the target width Wt based on the object trajectory 80 and changing the width of the abnormal section 90 to Wt. Changing the width of the abnormal section 90 includes reclassifying the detection locations near the boundary of the abnormal section 90.

[0085] Assume that the start point and end point of the abnormal section 90 are the detection locations Ss and Se, respectively. Further, the target width Wt of the abnormal section 90 is smaller than its current width by 6. In this case, the correction unit 2100 can shift the start point by +3 and the end point by -3. To that end, the detection locations Ss, Ss + 1, Se + 2, Se, Se - 1, Se - 2 are reclassified as monitoring locations.

[0086] The target width Wt of the abnormal section can be specified by finding the end point of the object trajectory 80 using techniques such as line detection, kink detection, or vehicle tracking algorithms, but is not limited thereto. The width Wt indicates the length or distance of the non-monitored section. When Wt is correctly estimated, after removing this abnormal section, the object trajectory is continued as shown in FIG. 10. If the object trajectory is not continuous, the end point coordinates of the vehicle trajectory are corrected and the width Wt is estimated again.

[0087] When there are two or more object trajectories 80 crossing the same abnormal section 90, the correction unit 2100 determines the target width Wt of the abnormal section 90 based on the object trajectories 80 crossing the abnormal section 90, and may shift the start point and the end point of the abnormal section 90 by the same distance from each other to change the width of the abnormal section 90 to Wt. Specifically, for each object trajectory 80 crossing the same abnormal section 90, the correction unit 2100 specifies a candidate width Wc of the abnormal section 90, and calculates the statistical value (for example, average value) of the candidate width Wc as Wt. The candidate width Wc corresponding to the object trajectory OT1 can be specified in the same manner as the method for specifying the target width Wt when there is no object trajectory 80 other than OT1 crossing the abnormal section 90 described above.

[0088] When determining the target width Wt of the abnormal section 90, the correction unit 2100 may exclude one or more outliers (referred to as "outlier trajectories") from the object trajectory 80 for which the candidate width Wc has been calculated. Assume that there are four object trajectories OT1, OT2, OT3, and OT4 crossing the abnormal section A1. Further, the object trajectory OT2 is determined to be an outlier trajectory. In this case, the correction unit 2100 calculates the candidate widths Wc1, Wc3, and Wc4 of OT1, OT3, and OT4, respectively. Since OT2 is determined to be an outlier, the candidate width Wc is not calculated for OT2. Next, the correction unit 2100 calculates the statistical value SV1 of Wc1, Wc2, and Wc3 as the target width Wt of the abnormal section A1, and corrects the start point and the end point of A1 included in the detection location information so that the width of A1 is changed to Wt (= SV1).

[0089] To determine whether the object trajectory 80 is an outlier trajectory, the correction unit 2100 can calculate the irregularity of the object trajectory 80. The correction unit 2100 determines whether the irregularity of the object trajectory 80 is less than a predetermined threshold. When the correction unit 2100 determines that the irregularity of the object trajectory 80 is less than the predetermined threshold, it calculates a candidate width of the abnormal section based on the object trajectory 80. On the other hand, when the correction unit 2100 determines that the irregularity of the object trajectory 80 is equal to or greater than the predetermined threshold, it does not calculate the candidate width of the abnormal section based on the object trajectory 80.

[0090] In some embodiments, the degree of linearity (in other words, uniformity) of the object trajectory 80 can be used to represent the irregularity of the object trajectory 80. Specifically, the lower the degree of linearity of the object trajectory 80, the higher the irregularity of the object trajectory 80 is determined to be. There are well-known methods for measuring the degree of linearity of a curve, and one of these methods can be applied to the correction unit 2100 to calculate the degree of linearity of the object trajectory 80.

[0091] In other embodiments, the irregularity may be measured based on the direction, speed change, overall movement behavior calculated using the object trajectory 80, or two or more of them. Specifically, for driving behaviors such as low speed, overspeed, or sudden speed changes, the object trajectory is tracked for the above-described measurements. An irregular trajectory shows the measured values of these behaviors as outliers compared to adjacent vehicle trajectories in this measurement interval.

[0092] <Use of Results> In some embodiments, the classification device 2000 generates the monitoring data 50 from the waterfall data 10 using the corrected detection location information. Then, the monitoring data 50 and the object trajectory 80 detected from the waterfall data 10 can be used in a traffic flow monitoring application. The traffic flow monitoring application can calculate traffic flow characteristics such as vehicle speed and vehicle count. These characteristics can be used to monitor the traffic flow.

[0093] The program can be stored using various types of non - transitory computer readable media and provided to a computer. Non - transitory computer readable media include various types of tangible storage media. Examples of non - transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto - optical recording media (e.g., magneto - optical disks), CD - ROM, CD - R, CD - R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM). Also, the program may be provided to a computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer readable media can supply the program to a computer via wired communication channels such as electric wires and optical fibers, or wireless communication channels.

[0094] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the above - described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present invention.

[0095] All or part of the above - described embodiments, although not limited, may be described as the following appendices. <Appendix> (Appendix 1) At least one memory configured to store instructions, execute the instructions to acquire waterfall data indicating the amplitude of vibration at each time point and each detection location of a vibration sensor arranged along an object of interest, Perform semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data. The normal class is assigned to the element for which it is inferred that the detection location is a monitored location, and the abnormal class is assigned to the element for which it is inferred that the detection location is a non-monitored location. The monitored location is the detection location arranged along the target object, and the non-monitored location is the detection location not arranged along the target object. Classify the detection locations into the monitored locations and the non-monitored locations based on the class data. A classification device comprising at least one processor configured as described above. (Appendix 2) The classification of the detection locations includes, for each detection location, Calculating the number of elements of the waterfall data corresponding to the detection location and to which the normal class is assigned. Determining whether the detection location is a monitored location or a non-monitored location based on the calculated number. The classification device according to Appendix 1, including performing the above. (Appendix 3) The at least one processor is Generating detection location information indicating whether the detection location is a monitored location or a non-monitored location for each detection location. Detecting one or more trajectories of a moving object, which is an object moving on the target object, from the waterfall data. The classification device according to Appendix 1 or 2, configured to correct the detection location information based on the detected trajectories. (Appendix 4) The correction of the detection location information includes, for each abnormal section, which is a region of one or more consecutive non-monitored locations in the waterfall data, For each trajectory crossing the abnormal section, specifying a candidate width of the abnormal section based on the trajectory. calculating a statistical value of the calculated candidate width as a target width of the abnormal section; correcting the width of the abnormal section to the target width, and executing the classification device according to appendix 3. (Appendix 5) Identifying the candidate width of the abnormal section of the trajectory includes calculating the irregularity of the trajectory; when the irregularity of the trajectory is less than a predetermined threshold, calculating the candidate width of the abnormal section based on the trajectory, and the classification device according to appendix 4. (Appendix 6) The irregularity of the trajectory is specified based on the degree of linearity of the trajectory, and the classification device according to appendix 5. (Appendix 7) A classification method calculated by a computer, including acquiring waterfall data indicating the amplitude of vibration at each time point and each detection location of a vibration sensor arranged along a target object; performing semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element for which the detection location is predicted to be a monitoring location, the abnormal class being assigned to the element for which the detection location is inferred to be a non-monitoring location, the monitoring location being the detection location arranged along the target object, and the non-monitoring location being the detection location not arranged along the target object; classifying the detection locations into the monitoring locations and the non-monitoring locations based on the class data; A classification method. (Appendix 8) Classifying the detection locations includes, for each detection location, calculating the number of elements of the waterfall data corresponding to the detection location and assigned the normal class; determining whether the detection location is a monitoring location or a non-monitoring location based on the calculated number. The classification method according to appended note 7, including executing (Appended note 9) For each detection location, generating detection location information indicating whether the detection location is the monitoring location or the non-monitoring location; Detecting one or more trajectories of a moving object, which is an object moving on the target object, from the waterfall data; The classification method according to appended note 7 or 8, further including correcting the detection location information based on the detected trajectory. (Appended note 10) The correction of the detection location information is performed for each of the abnormal intervals, which are regions of one or more consecutive non-monitoring locations in the waterfall data. The correction of the detection location information is performed for each of the abnormal intervals, which are regions of one or more consecutive non-monitoring locations in the waterfall data. For each trajectory crossing the abnormal interval, specifying a candidate width of the abnormal interval based on the trajectory; Calculating a statistical value of the calculated candidate width as a target width of the abnormal interval; Correcting the width of the abnormal interval to the target width; The classification method according to appended note 9, including executing (Appended note 11) Specifying the candidate width of the abnormal interval of the trajectory includes: Calculating the irregularity of the trajectory; When the irregularity of the trajectory is less than a predetermined threshold, calculating the candidate width of the abnormal interval based on the trajectory. The classification method according to appended note 10, including (Appended note 12) The classification method according to appended note 11, wherein the irregularity of the trajectory is specified based on the degree of linearity of the trajectory. (Appended note 13) Causing a computer to Obtain waterfall data indicating the amplitude of vibration at each time point and each detection location of a vibration sensor arranged along a target object. Perform semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data. The normal class is assigned to the element for which it is predicted that the detection location is a monitoring location, and the abnormal class is assigned to the element for which it is inferred that the detection location is a non-monitoring location. The monitoring location is the detection location arranged along the target object, and the non-monitoring location is the detection location not arranged along the target object. Classify the detection locations into the monitoring locations and the non-monitoring locations based on the class data. A non-transitory computer-readable storage medium storing a program for causing the above to be executed. (Appendix 14) Classifying the detection locations includes, for each detection location, calculating the number of elements of the waterfall data corresponding to the detection location and to which the normal class is assigned, determining whether the detection location is a monitoring location or a non-monitoring location based on the calculated number, The storage medium according to Appendix 13, including executing the above. (Appendix 15) The program causes the computer to generate detection location information indicating whether each detection location is a monitoring location or a non-monitoring location, detect one or more trajectories of a moving object, which is an object moving on the target object, from the waterfall data, correct the detection location information based on the detected trajectories, The storage medium according to Appendix 13 or 14, further including causing the above to be executed. (Appendix 16) The correction of the detection location information is for each of abnormal intervals, which are regions of one or more consecutive non-monitoring locations in the waterfall data, For each of the trajectories crossing the abnormal section, specifying a candidate width of the abnormal section based on the trajectory; calculating a statistical value of the calculated candidate width as a target width of the abnormal section; correcting the width of the abnormal section to the target width; A storage medium according to Appendix 15, including executing the above. (Appendix 17) Specifying the candidate width of the abnormal section of the trajectory includes: calculating the irregularity of the trajectory; when the irregularity of the trajectory is less than a predetermined threshold, calculating the candidate width of the abnormal section based on the trajectory; A storage medium according to Appendix 16, including the above. (Appendix 18) The irregularity of the trajectory is determined based on the degree of linearity of the trajectory. A storage medium according to Appendix 17.

Explanation of symbols

[0096] 10 Waterfall data 20 Detection data 30 Vibration sensor 40 Target object 50 Monitoring data 60 Waterfall image 70 Moving object 80 Object trajectory 90 Abnormal section 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Classification device 2020 Acquisition unit 2040 Segmentation 2060 Classification unit 2080 Detection unit 2100 Amendment section

Claims

1. at least one memory configured to store commands; executing the commands to acquire waterfall data indicating the amplitude of vibration at each time point and each detection location of vibration sensors arranged along the target object; performing semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, wherein the normal class is assigned to the elements for which the detection location is presumed to be a monitored location, the abnormal class is assigned to the elements for which the detection location is presumed to be a non-monitored location, the monitored location is the detection location arranged along the target object, and the non-monitored location is the detection location not arranged along the target object; classifying the detection locations into the monitored locations and the non-monitored locations based on the class data; at least one processor configured as such, and a classification device comprising the same.

2. The classification of the detection locations includes, for each detection location, calculating the number of elements of the waterfall data corresponding to the detection location and assigned the normal class; determining whether the detection location is a monitored location or a non-monitored location based on the calculated number; The classification device according to claim 1, including performing the above.

3. The at least one processor generates detection location information indicating whether each detection location is a monitored location or a non-monitored location; detects one or more trajectories of a moving object, which is an object moving on the target object, from the waterfall data; The classification device according to claim 1 or 2, configured to correct the detection location information based on the detected trajectories.

4. The correction of the detection location information includes, for each abnormal section, which is a region of one or more consecutive non-monitored locations in the waterfall data, for each trajectory crossing the abnormal section, specifying a candidate width of the abnormal section based on the trajectory; calculating a statistical value of the calculated candidate widths as a target width of the abnormal section; The classification device according to claim 3, including performing the correction of the width of the abnormal section to the target width.

5. Specifying the candidate width of the abnormal section of the trajectory is Calculating the irregularity of the trajectory; When the irregularity of the trajectory is less than a predetermined threshold, calculating the candidate width of the abnormal section based on the trajectory, the classification device according to claim 4.

6. The classification device according to claim 5, wherein the irregularity of the trajectory is specified based on the degree of linearity of the trajectory.

7. A classification method calculated by a computer, Obtaining waterfall data indicating the amplitude of vibration at each time point and each detection location of a vibration sensor arranged along an object to be inspected; Performing semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element for which it is predicted that the detection location is a monitoring location, the abnormal class being assigned to the element for which it is inferred that the detection location is a non-monitoring location, the monitoring location being the detection location arranged along the object to be inspected, and the non-monitoring location being the detection location not arranged along the object to be inspected; Classifying the detection locations into the monitoring locations and the non-monitoring locations based on the class data; A classification method comprising:

8. Classifying the detection locations includes, for each detection location, Calculating the number of elements of the waterfall data corresponding to the detection location and to which the normal class is assigned; Determining whether the detection location is a monitoring location or a non-monitoring location based on the calculated number, the classification method according to claim 7.

9. For each detection location, generating detection location information indicating whether the detection location is a monitoring location or a non-monitoring location; Detecting one or more trajectories of a moving object, which is an object moving on the object to be inspected, from the waterfall data; Further comprising correcting the detection location information based on the detected trajectories, the classification method according to claim 7 or 8.

10. The correction of the detection location information is for each of the abnormal sections, which are regions of one or more consecutive non-monitoring locations in the waterfall data, ​ The correction of the detection location information is performed for each of the abnormal sections that are regions of one or more consecutive non-monitored locations in the waterfall data, for each of the trajectories crossing the abnormal section, specifying a candidate width of the abnormal section based on the trajectory, calculating a statistical value of the calculated candidate width as a target width of the abnormal section, correcting the width of the abnormal section to the target width, The classification method according to claim 9, comprising executing.

11. Specifying the candidate width of the abnormal section of the trajectory includes calculating the irregularity of the trajectory, when the irregularity of the trajectory is less than a predetermined threshold, calculating the candidate width of the abnormal section based on the trajectory, The classification method according to claim 10, comprising.

12. The classification method according to claim 11, wherein the irregularity of the trajectory is specified based on the degree of linearity of the trajectory.

13. Causing a computer to acquire waterfall data indicating the amplitude of vibration at each time point and each detection location of vibration sensors arranged along the target object, performing semantic segmentation on the waterfall data to generate class data indicating a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element for which the detection location is predicted to be a monitored location, the abnormal class being assigned to the element for which the detection location is inferred to be a non-monitored location, the monitored location being the detection location arranged along the target object, and the non-monitored location being the detection location not arranged along the target object, classifying the detection location into the monitored location and the non-monitored location based on the class data, A non-transitory computer-readable storage medium storing a program for causing execution.

14. Classifying the detection location includes, for each detection location, calculating the number of elements of the waterfall data corresponding to the detection location and to which the normal class is assigned, determining whether the detection location is a monitored location or a non-monitored location based on the calculated number, The storage medium according to claim 13, comprising executing.

15. The program causes the computer to For each detection location, generating detection location information indicating whether the detection location is the monitoring location or the non-monitoring location; detecting one or more trajectories of a moving object that is an object moving on the target object from the waterfall data; correcting the detection location information based on the detected trajectories; The storage medium according to claim 13 or 14, further causing the above to be executed.

16. The correction of the detection location information includes, for each abnormal section that is a region of one or more consecutive non-monitoring locations in the waterfall data, for each trajectory crossing the abnormal section, specifying a candidate width of the abnormal section based on the trajectory; calculating a statistical value of the calculated candidate width as a target width of the abnormal section; correcting the width of the abnormal section to the target width; The storage medium according to claim 15, including executing the above.

17. Specifying the candidate width of the abnormal section of the trajectory includes calculating the irregularity of the trajectory; when the irregularity of the trajectory is less than a predetermined threshold, calculating the candidate width of the abnormal section based on the trajectory; The storage medium according to claim 16, including the above.

18. The storage medium according to claim 17, wherein the irregularity of the trajectory is determined based on the degree of linearity of the trajectory.

Images