Method for detecting lidar signal abnormality based on RSU and apparatus therefor

Abstract

To provide a roadside base station (RSU) - based lidar signal abnormality detection device and method capable of determining whether a detected change is a natural phenomenon or an abnormal behavior due to a spoofing attack.SOLUTION: The method includes, when a change is detected based on voxel data obtained by voxelizing a point cloud frame measured by at least one of a plurality of lidars measuring a first area, masking voxels of a first location in which the change is detected, and inputting vectors for the masked voxels and remaining voxels other than the masked voxels to a neural network to obtain predicted point information of the masked voxels, based on the predicted point information of the masked voxels and the voxelized data of each of the lidars, it is determined whether the change detected at the first position is caused by the abnormal action.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for detecting an abnormality in a LiDAR signal based on an RSU (roadside unit), and more particularly, to a method for detecting an abnormality based on location using multiple RSUs. [Background technology]

[0002] Various methods for detecting LiDAR spoofing attacks have been published. For example, the single point cloud frame-based method detects occlusion patterns that are different from normal 3D objects when a spoofing attack is performed on a single LiDAR image and a 3D object is generated.

[0003] Although various conventional multi-object tracking algorithms track objects, it is not possible to determine whether a suddenly detected or suddenly disappeared object is hidden by a surrounding object, whether a hidden object has reappeared, or whether abnormal behavior due to a spoofing attack has been detected. Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention provides an RSU-based LIDAR signal anomaly detection device and method that, when a variation is detected by a multi-object tracking algorithm, can determine whether the variation is a natural phenomenon or an abnormal behavior caused by a spoofing attack.

[0005] The present invention seeks to provide an apparatus and method for determining the causes of such variations using multiple lidars measuring the same area. [Means for solving the problem]

[0006] According to one aspect of the present invention, an RSU-based LIDAR signal anomaly detection device includes a memory that stores at least one instruction, a processor that executes the at least one instruction, and a neural network. The processor acquires voxel data by voxelizing a current point cloud frame measured from each of a number of LIDARs corresponding to each of a number of roadside base stations and measuring a first area, forms a union of the voxel data by voxelizing the point cloud frames measured from each of the multiple LIDARs, and generates combined voxel data. If a fluctuation is detected based on the voxel data by voxelizing the point cloud frames measured by at least one of the multiple LIDARs, the processor masks voxels at a first location where the fluctuation is detected based on the combined voxel data, and inputs vectors for the masked voxels and remaining voxels other than the masked voxels into the neural network to acquire predicted point information for the masked voxels. The processor determines whether the fluctuation detected at the first location is due to abnormal activity based on the predicted point information of the masked voxels and the voxelized data of each LIDAR.

[0007] In this case, when masking the voxels at the first position, the voxels at the first position of each combined voxel data generated for a time point prior to the time point at which the variation is detected are masked, and the neural network may further receive vectors for the masked voxels and the remaining voxels for each of the previous time points.

[0008] In this case, the combined voxel data at the time when the variation is detected and the previous time point is data in which voxels have the same size and position for the same measurement area at the same time point, and the vector for the voxels at the same position at the same time point can be generated by concatenating vectors for the voxels at the same position of voxel data in which point cloud frames measured from each of the multiple lidars are voxelized.

[0009] In this case, the neural network includes an encoder, decoders the number of which is equal to the number of lidars, and prediction heads corresponding to the decoders, and inputs the vector to the encoder to obtain an encoded value, inputs the encoded value, a first mask token for the masked voxels, and a second mask token for voxels where no points exist at the time when the variation is detected and all previous times to decoders the number of which is equal to the number of lidars, respectively, to obtain reconstructed data for the first mask token and the second mask token, and inputs the reconstructed data to a prediction head to obtain the predicted point information.

[0010] In this case, the fluctuations can be detected by a multi-object tracking algorithm.

[0011] In this case, the predicted point information may include at least one of the number of points in the voxel predicted for the position of the masked voxel, the coordinates of the points, and information on whether the voxel is a masked voxel.

[0012] In this case, the prediction heads may include a first prediction head that predicts the coordinates of a point within a voxel, a second prediction head that predicts the number of points within the voxel, and a third prediction head that predicts whether the voxel at the current position to be predicted is the masked voxel or a voxel where no point exists.

[0013] In this case, the processor calculates a chamfer distance loss value using point coordinates in a voxel for each time point predicted by the first prediction head and point coordinates in a voxel at a corresponding position of voxelized data for each time point of a rider matched with the first prediction head, determines whether a maximum value among the chamfer distance loss values ​​calculated for each time point and for the plurality of riders is greater than a predetermined first critical value, calculates a loss value for the number of points using the number of points in a voxel for each time point predicted by the second prediction head and the number of points in a voxel at a corresponding position of voxelized data for each time point of a rider matched with the second prediction head, and calculates a loss value for the number of points calculated for each time point and for the plurality of riders. and determining whether a maximum value among the cross-entropy loss values ​​calculated for each time point and for each of the plurality of riders is greater than a predetermined third critical value; calculating a cross-entropy loss value using a value indicating whether the voxel predicted by the third prediction head is a masked voxel and a value indicating whether a voxel at a corresponding position in the voxelized data of the rider matched with the third prediction head is a masked voxel; and determining whether a maximum value among the cross-entropy loss values ​​calculated for each time point and for each of the plurality of riders is greater than a predetermined third critical value; and determining whether at least one of the first critical value, the second critical value, and the third critical value is smaller as a result of the determination, the variation detected at the first position may be determined to be due to an abnormal activity.

[0014] In this case, the neural network may acquire voxel data obtained by voxelizing point cloud frames measured at each time point from each of a number of lidars corresponding to each of the plurality of roadside base stations and measuring the first area, form a union of the voxel data obtained by voxelizing the point cloud frames measured at each of the plurality of lidars, generate combined voxel data, mask a predetermined number of voxels at random positions in the combined voxel data for each time point, and mask voxels at the same positions for each time point, input vectors for the predetermined number of masked voxels for each time point and the remaining voxels other than the masked voxels to the neural network to acquire predicted point information of the masked voxels, and be trained so that a value calculated using a loss function between the predicted point information of the masked voxels and voxel information at the same positions satisfies a predetermined condition.

[0015] In this case, the neural network may be trained to minimize the sum of weighted values ​​of chamfer distance loss, loss for the number of points, and cross-entropy loss calculated using voxelized data measured for each time point and each lidar with respect to the predicted point information of the masked voxels.

[0016] According to one aspect of the present invention, an RSU-based LIDAR signal anomaly detection method may include: acquiring voxel data by voxelizing a current point cloud frame measured by each of a number of LIDARs corresponding to a number of roadside base stations and measuring a first area; forming a union of the voxel data by voxelizing the point cloud frames measured by each of the multiple LIDARs and generating combined voxel data; if a fluctuation is detected based on the voxel data by voxelizing the point cloud frames measured by at least one of the multiple LIDARs, masking a voxel at a first location where the fluctuation is detected based on the combined voxel data, and inputting vectors for the masked voxels and remaining voxels other than the masked voxels into a neural network to acquire predicted point information for the masked voxels; and determining whether the fluctuation detected at the first location is due to abnormal activity based on the predicted point information of the masked voxels and the voxelized data of each LIDAR.

[0017] In this case, the step of acquiring the predicted point information may include a step of masking the voxels at the first position of each of the combined voxel data generated for the time point at which the variation is detected and the time point before the time point at which the variation is detected, when masking the voxels at the first position; and a step of inputting vectors for the masked voxels and the remaining voxels for each time point into the neural network.

[0018] In this case, the combined voxel data at the time when the variation is detected and the previous time point may be data in which voxels have the same size and position for the same measurement area at the same time point, and the step of obtaining the predicted point information may further include the step of generating vectors for the remaining voxels, wherein the generating step may include the step of generating vectors for each voxel included in the voxel data for each lidar at the same time point; and if voxels for multiple lidars exist at one voxel position in the combined voxel data, the step of concatenating vectors of voxels for the multiple lidars present at the one voxel position and replacing the concatenated vector with the vector for the one voxel position.

[0019] In this case, the method may further include detecting the variation using a multi-object tracking algorithm.

[0020] In this case, the predicted point information includes point coordinates in a voxel for each time point, the number of points, and information on whether the voxel is masked. The step of determining whether the fluctuation detected at the first position is due to an abnormal activity includes the steps of calculating a chamfer distance loss value using the point coordinates in a voxel predicted for each time point and for the multiple riders and the point coordinates in a voxel at a corresponding position of the voxelized data, and determining whether the maximum value of the chamfer distance loss values ​​calculated for each time point and for the multiple riders is greater than a predetermined first critical value; calculating a loss value for the number of points using the number of points in a voxel predicted for each time point and for the multiple riders and the number of points in a voxel at a corresponding position of the voxelized data, and determining whether the maximum value of the chamfer distance loss values ​​calculated for each time point and for the multiple riders is greater than a predetermined first critical value; The method may include determining whether a maximum value among loss values ​​for the number of points calculated for each point and for the multiple riders is greater than a predetermined second critical value; calculating a cross-entropy loss value using a value indicating whether a voxel at a corresponding position of the voxelized data is a masked voxel predicted for each time point and for each of the multiple riders, and determining whether a maximum value among the cross-entropy loss values ​​calculated for each time point and for each of the multiple riders is greater than a predetermined third critical value; and determining that the fluctuation detected at the first position is due to abnormal activity when it is determined that at least one of the first critical value, the second critical value, and the third critical value is smaller as a result of the determination.

[0021] According to one aspect of the present invention, there is provided an RSU-based LIDAR signal anomaly detection training method, which includes the steps of: acquiring voxel data from point cloud frames measured from a plurality of LIDARs corresponding to a plurality of roadside base stations and measuring a first area; forming a union of the voxel data from the point cloud frames measured from the plurality of LIDARs; and generating combined voxel data; masking a predetermined number of voxels at random positions in the combined voxel data; and inputting vectors for the predetermined number of masked voxels and the remaining voxels other than the masked voxels into a neural network to acquire predicted point information of the masked voxels, wherein the neural network can be trained so that a value calculated using a loss function between the predicted point information of the masked voxels and voxel information at the same position satisfies a predetermined condition.

[0022] In this case, the step of acquiring voxelized data may be a step of acquiring voxelized data of point cloud frames measured from each of the plurality of lidars for each time point, and the step of masking voxels may be a step of masking voxels at the same position for each time point, and vectors for the predetermined number of masked voxels for each time point and the remaining voxels other than the masked voxels may be input to the neural network.

[0023] In this case, the combined voxel data is data in which voxels have the same size and position for the same measurement area at the same time point, and the step of obtaining the predicted point information may further include a step of generating vectors for the remaining voxels, and the generating step may include a step of generating vectors for each voxel included in the voxel data for each lidar at the same time point; and if voxels for multiple lidars exist at one voxel position in the combined voxel data, a step of concatenating vectors of voxels for the multiple lidars existing at the one voxel position and replacing the concatenated vector with a vector for the one voxel position.

[0024] In this case, the predicted point information may include point coordinates in a voxel for each time point, the number of points, and information on whether the voxel is a masked voxel.

[0025] In this case, after the step of obtaining the predicted point information, the method may further include: calculating a chamfer distance loss value using point coordinates in the predicted voxel for each time point and for each of the multiple riders and point coordinates in the voxel at a corresponding position in the voxelized data; calculating a loss value for the number of points using the number of points in the predicted voxel for each time point and for each of the multiple riders and the number of points in the voxel at a corresponding position in the voxelized data; calculating a cross-entropy loss value using a value indicating whether a voxel is a masked voxel predicted for each time point and for each of the multiple riders and a value indicating whether a voxel at a corresponding position in the voxelized data is a masked voxel; and calculating a sum of weighted values ​​of the chamfer distance loss value, the loss value for the number of points, and the cross-entropy loss value, wherein the neural network may be trained to minimize the sum of the weighted values. [Effects of the Invention]

[0026] According to the present invention, it is possible to provide an RSU-based LIDAR signal anomaly detection device and method that, when a variation is detected by a multi-object tracking algorithm, can determine whether the variation is a natural phenomenon or an abnormal behavior caused by a spoofing attack.

[0027] According to the present invention, when an attack is made on a specific LIDAR, it is possible to determine whether the cause of the change is due to abnormal behavior by using the correlation between the point clouds measured by other LIDARs measuring the same area and the point cloud of the specific LIDAR that has changed due to the attack. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a diagram illustrating an RSU-based LIDAR abnormal signal detection system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating a process of generating input data for training a neural network according to an embodiment of the present invention. FIG. [Figure 3] 1 is a diagram illustrating voxel data for point cloud frames generated in time order according to an embodiment of the present invention; [Figure 4] 1 is a flowchart illustrating an RSU-based LIDAR signal detection learning method according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating the structure of a neural network according to an embodiment of the present invention. [Figure 6] 10 is a diagram illustrating a process of calculating a predicted head loss value according to an embodiment of the present invention; [Figure 7] 1 is a diagram illustrating voxel data for point cloud frames generated in time order according to an embodiment of the present invention; [Figure 8] 1 is a flowchart illustrating an RSU-based LIDAR signal detection and inference method according to an embodiment of the present invention. [Figure 9]10A and 10B are diagrams illustrating a process for determining whether a variation in inference time is due to abnormal behavior according to an embodiment of the present invention. [Figure 10] FIG. 10 is a conceptual diagram illustrating an example of a generalized RSU-based LIDAR signal anomaly detection device or computing system capable of performing at least some of the processes of FIGS. 1 to 9. DETAILED DESCRIPTION OF THE INVENTION

[0029] Although the present invention can be modified in various ways and can have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, it should be understood that this is not intended to limit the present invention to the specific embodiments, and that the present invention includes all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the present invention.

[0030] Terms such as "first," "second," etc. may be used to describe various components, but the components should not be limited by these terms. These terms are used only to distinguish one component from another. For example, a first component may be designated a "second component," and similarly, a second component may be designated a "first component," without departing from the scope of the present invention. The term "and / or" includes a combination of multiple related listed items or any of multiple related listed items.

[0031] In the embodiments of the present application, "at least one of A and B" means "at least one of A or B" or "at least one of a combination of one or more of A and B." Also, in the embodiments of the present application, "one or more of A and B" means "one or more of A or B" or "one or more of a combination of one or more of A and B."

[0032] When a component is said to be "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but that there may be other components in between. Conversely, when a component is said to be "directly coupled" or "directly connected" to another component, it should be understood that there are no other components in between.

[0033] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly dictates otherwise. In this application, the terms "comprise" or "have" and the like are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or possible addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted to have a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined in this invention.

[0035] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In order to facilitate overall understanding of the present invention, the same reference numerals will be used for the same components in the drawings, and duplicate descriptions of the same components will be omitted.

[0036] FIG. 1 is a diagram illustrating an RSU-based LIDAR abnormal signal detection system according to an embodiment of the present invention.

[0037] The RSU-based lidar abnormal signal detection system may include multiple roadside base stations (RSUs) 110, 120, 130 and a detection device 200.

[0038] A LiDAR may be installed in each of the roadside base stations 110, 120, and 130. The LiDAR can measure point clouds in real time, and the measured point cloud frames or voxel data obtained by voxelizing the point cloud frames can be provided to the detection device 200 in real time.

[0039] The roadside base stations 110, 120, and 130 may be fixedly installed at intersections, for example. In this case, the coverage areas 310 of the lidars 111 and 121 installed at the roadside base stations 110 and 120, respectively, may overlap, as shown in Figure 1. The coverage areas are also referred to as "the same area" in the present invention.

[0040] Data collected by moving vehicles is highly variable due to various external factors such as the vehicle's position, speed, direction, etc. On the other hand, roadside base stations 110, 120, and 130 are installed in fixed locations and are less exposed to environmental changes, so the stability and consistency of the collected data can be guaranteed.

[0041] While moving vehicles may not follow a fixed route, making it difficult to collect data over a long period of time in the same area, roadside base stations in fixed locations can monitor the same area over a long period of time, allowing for more accurate detection of abnormalities by observing changes in a specific area over time.

[0042] Roadside base stations are connected to fixed infrastructure and are advantageous in processing and storing large amounts of data in real time. Fixed-location roadside base stations can more accurately detect location-based abnormal phenomena that may occur in specific locations (e.g., illegal parking, walking in pedestrian-restricted zones, etc.).

[0043] Therefore, the present invention allows for the strategic placement and use of multiple roadside base stations, which allows for monitoring intersections and complex road structures from various angles and fields of view, and the monitoring data can be integrated for more accurate anomaly detection.

[0044] Lidars 111, 121, 131 installed at roadside base stations 110, 120, 130 can be used to monitor general traffic conditions.

[0045] The point cloud frame measured by the LIDAR 111 may be voxelized and used as voxel data. In this case, various multi-object tracking algorithms may be used to track objects in the voxel data measured and acquired by the LIDAR. That is, a vehicle, pedestrian, etc. may be detected for each poxel of the voxel data using the multi-object tracking algorithm, and the detected object may be tracked. In this case, a bounding box may be generated for each detected object. The bounding box for each detected object may be generated for each point cloud frame measured and generated by the LIDAR for tracking.

[0046] For example, if a vehicle that has caught up with another vehicle enters the lidar sensing range due to a lane change, etc., the multi-object tracking algorithm may determine that a new vehicle has suddenly been detected. For example, if a vehicle that has been detected / tracked moves behind another vehicle or is obscured by another vehicle, the multi-object tracking algorithm may determine that the tracked vehicle has suddenly disappeared.

[0047] In addition, in a situation where a LIDAR spoofing attack is carried out, the signals of the LIDARs 111, 121, and 131 installed at the roadside base stations 110, 120, and 130 (i.e., voxel data of point cloud frames measured from the LIDARs) may have the following characteristics.

[0048] For example, a vehicle may suddenly be detected in a space where there was no vehicle on the voxel data, or a vehicle may suddenly disappear in a space where there was a vehicle on the voxel data.

[0049] Therefore, since the disappearance and appearance of vehicles occurs in the same way under both normal and spoofing attack conditions, it is necessary to distinguish the cause of the fluctuation. However, if a sophisticated spoofing attack can make vehicles appear to disappear and appear so that they cannot be distinguished from normal conditions, it is practically impossible to determine the cause of the fluctuation.

[0050] From the perspective of a spoofing attacker, when attacking a single roadside base station (e.g., 110), it is relatively easy to perform a sophisticated spoofing attack because only the lidar signal of the roadside base station 110 needs to be considered.

[0051] Meanwhile, as shown in Fig. 1, if there is an overlapping area 310 between the area measured by the LIDAR 111 of the first roadside base station 110 and the area measured by the LIDAR 121 of the second roadside base station 120, multiple LIDAR signals for the overlapping area 310 can be simultaneously used to detect fluctuations / abnormalities. In this case, for a spoofing attack to be successful, sophisticated spoofing attacks against the roadside base stations must be simultaneously performed, increasing the difficulty of a successful attack.

[0052] The detection device 200, which is the entity that determines whether the detected fluctuation according to the present invention is due to abnormal activity (i.e., a spoofing attack), may be each of the multiple roadside base stations, a roadside base station that acts as a master among the multiple roadside base stations, or an independent local server that manages the multiple roadside base stations.

[0053] When the subject is a roadside base station that acts as a master among a plurality of roadside base stations, the remaining roadside base stations can function as slaves.

[0054] In the following description, it is assumed that the detection device 200 is an independent local server that manages multiple roadside base stations.

[0055] FIG. 2 is a diagram illustrating a process of generating input data for training a neural network according to an embodiment of the present invention.

[0056] The following description will be made with reference to both FIG. 1 and FIG.

[0057] Point cloud frames 410, 420 measured from multiple lidars 111, 121 corresponding to multiple roadside base stations 110, 120 and measuring the same area 310 are unioned in the same space, and the union space is divided into voxel units to generate combined voxel data 400.

[0058] Specifically, a union of the areas measured by each of the multiple LIDARs 111 and 121 can be formed, and the union area can be divided into voxel units to generate combined voxel data. The point clouds measured by each LIDAR can be located in the combined voxel data space. In this case, a feature vector for each voxel position of each LIDAR can be calculated.

[0059] In this case, for example, the detection device 200 can acquire point cloud frames 410, 420 from multiple roadside base stations 110, 120, and then form a union to generate the combined voxel data 400.

[0060] Alternatively, the detection device 200 may acquire voxel data 412, 422 by voxelizing point cloud frames 410, 420 measured from the multiple LIDARs 111, 121. Then, the detection device 200 may form a union of the voxel data obtained by voxelizing the point cloud frames measured from the multiple LIDARs 111, 121, and generate combined voxel data 440. In this case, the voxel data 412, 422 acquired by the detection device 200 may be formed according to a predetermined position and size for each voxel of the combined voxel data.

[0061] In this case, the voxel data may be generated after a pre-processing step has been performed to remove (points relative to) fixed backgrounds such as road surfaces, building walls, etc. from the point cloud frame.

[0062] For example, the combined voxel data 400 may be data combined in one space such that the voxels included in the same region 430 measured at the same time point have the same size and position.

[0063] In this case, there may be eight voxels 430 at the same position at the same time in FIG.

[0064] At this time, the vectors for each of the eight voxels can be generated by concatenating the vectors for the voxels at the same positions of the voxelized voxel data 412 and 422 with each other.

[0065] That is, since a voxel (j=7) of the voxel data 412 and a voxel (j=7) of the voxel data 422 are co-located voxels, a vector generated by concatenating the vector of the voxel of the voxel data 412 (dynamic voxel feature embedding) z(k, j, t) (where k is the rider number, j is the voxel, and t is the time point) with the vector of the voxel of the voxel data 422 may be the vector for the co-located voxel (j=7) of the combined voxel data 440. The vector generated by concatenation is also referred to as a combined feature vector. Specifically, after the vectors at the co-located positions are concatenated, a final input vector can be generated through a linear layer W.

[0066] For example, the final input vector JPEG2026016283000002.jpg21151

number

[0067] In this case, the vector for the voxel can be generated as a dynamic voxel feature embedding if a point exists in the voxel (non-empty voxel), and if no point measured by a particular lidar exists (empty voxel), the dynamic voxel feature embedding can be a 0 vector.

[0068] According to one embodiment of the present invention, by combining and learning point clouds measured by different lidars over the same area, a neural network can learn the association between point cloud frames measured by different lidars.

[0069] This means that if an attack is made on a specific lidar, the correlation between the point cloud measured by the remaining lidars and the point cloud changed by the attack will be broken, making it possible to detect any abnormalities caused by this.

[0070] FIG. 3 is a diagram illustrating voxel data for point cloud frames generated in time order according to an embodiment of the present invention.

[0071] The following description will be made with reference to both FIGS.

[0072] In a first embodiment of the present invention, multiple roadside base stations can be used to determine whether variations detected in point cloud frames or voxel data for a single time point are due to abnormal activity.

[0073] Alternatively, in a second embodiment of the present invention, multiple roadside base stations can be used to determine whether variations detected in point cloud frames or voxel data are due to abnormal activity for multiple time points rather than just one time point.

[0074] For example, assume there are two roadside base stations measuring the same area (or coverage area). Voxel data 510 can be acquired for multiple time points from the LIDAR 111 of the first roadside base station RSU 1110. Voxel data 520 can be acquired for multiple time points from the LIDAR 121 of the second roadside base station RSU 2120.

[0075] Since each roadside base station is fixed and does not move, voxel data acquired over time can be voxelized so that the positions and sizes of each voxel are the same for the same region. For example, voxel data 510 and voxel data 520 can have the size of combined voxel data generated by forming a union of point cloud frames in the same space and dividing the union space into voxel units.

[0076] For example, the multiple time points t for learning may be a predetermined number T of time points. The predetermined number of time points may be a predetermined number of consecutive time points. For example, if T=10, t=0 to 9.

[0077] In this case, the sets 510 and 520 (440 in FIG. 2) containing voxels at the same position can be called a set of voxels at the same position or combined voxel data.

[0078] For the voxel set at the same position, a set in which at least one point exists at at least one time point for at least one rider at the measurement time point (t=0 to T-1) can be referred to as a non-empty voxel set. In this case, in a non-empty voxel set, a point may not exist within a voxel at some observation time points for some riders. Since predictions are made for each rider in the neural network described below, voxels observed as not having the point for some riders must be predicted / determined as empty voxels.

[0079] In other embodiments, non-empty voxel sets can also be determined for whole voxels (eg, 412 and 422 in FIG. 2) that contain co-located voxel sets (eg, 430 in FIG. 2).

[0080] 3, a predetermined number (e.g., three) of voxels in a randomly selected position from a non-empty voxel set may be masked, and all voxels corresponding to all measurement intervals (e.g., t=0 to 9) may be masked. That is, voxels at the same position may be masked for each time point.

[0081] FIG. 4 is a flowchart illustrating an RSU-based LIDAR signal detection learning method according to an embodiment of the present invention.

[0082] The following steps S610 to S630 may be preprocessing steps for generating inputs for the neural network included in the detection device 200 of FIG.

[0083] The following description will be given with reference to both FIGS.

[0084] In step S610, voxel data 412, 422, ... can be obtained by voxelizing point cloud frames measured from a number of lidars 111, 121, ... corresponding to a number of roadside base stations 110, 120, ... and measuring a first region 430. For example, the voxel data may be generated by each of the multiple roadside base stations 110, 120, or may be generated by the detection device 200 using point cloud frames received from the multiple roadside base stations.

[0085] In step S620, the point cloud frames measured from each of the multiple lidars 111, 121, ... are unioned to generate combined voxel data. At this time, vectors for each voxel of the combined voxel data can be calculated. A vector for a voxel at the same time and position in the combined voxel data can be generated by concatenating vectors for voxels at the same position in the voxel data where the point cloud frames measured from each of the multiple lidars are voxelized.

[0086] In step S630, a predetermined number (e.g., three) of voxels 511, 512, 513, 521, 522, and 523 at random positions in the combined voxel data may be masked. In this case, preferably, voxels included in a non-empty voxel set in the combined voxel data may be masked.

[0087] In step S640, vectors for the predetermined number of masked voxels 511, 512, 513, 521, 522, and 523 and the remaining voxels other than the masked voxels 511, 512, 513, 521, 522, and 523 may be input to a neural network to obtain predicted point information for the masked voxels. In this case, the remaining voxels may be voxels of a non-empty voxel set.

[0088] In this case, the neural network may be trained so that a value calculated using a loss function between predicted point information of the masked voxel and voxel information at the same position satisfies a predetermined condition. That is, the step of training the neural network to predict voxels that are invisible due to masking using only the remaining voxels may be a step of training the neural network so that correlation between actual voxel information at corresponding positions of all voxels is minimized. In this case, the predetermined condition may be a condition that minimizes the correlation. This will be further described below with reference to FIG. 6.

[0089] Although steps S610 to S640 have been described based on voxel data of a point cloud frame at a single time point, steps S610 to S640 can also be performed based on voxel data acquired at multiple time points, as in the method described above with reference to FIG. 3.

[0090] 5 and 6 will be explained below based on voxel data acquired at multiple time points.

[0091] FIG. 5 is a diagram illustrating the structure of a neural network according to an embodiment of the present invention.

[0092] The detection device 200 may include a Transformer Neural Network. In another embodiment, when one of a plurality of roadside base stations acts as a detection device, the one roadside base station may include a Transformer Neural Network.

[0093] The neural network may include an encoder 210, lidar-specific decoders 220, 230, 240, and lidar-specific prediction heads 221, 231, 241.

[0094] FIG. 6 is a diagram illustrating a process of calculating a predicted head loss value according to an embodiment of the present invention.

[0095] The following description will be made with reference to both FIGS.

[0096] In step S640, vectors for the remaining voxels other than the masked voxels 511, 512, 513 or 521, 522, 523 for each time point may be input to the encoder 210.

[0097] JPEG2026016283000004.jpg49151

[0098] The vectors for the remaining voxels excluding masked voxels and voxels that have no points in the voxel data of all riders at all time points, i.e., empty voxels, can also be input to the encoder 210.

[0099] For example, with reference to FIG. 3, if there are no points in the voxels 514, 524 at the same location in all of the voxel data 412 and voxel data 422 and at all of the time points, then that voxel may also be excluded from the input to the encoder 210.

[0100] The encoder 210 may encode input vectors arranged in order of voxel position and time point (ie, in spatial order and chronological order) and output the encoded values.

[0101] In this case, a positional embedding vector may be further input to the encoder 210 and the decoders 220, 230, and 240. For example, assume that there are 100 voxels per point cloud frame and 10 frames occur sequentially. In this case, the positional embedding vector to be input to the transformer neural network may have index values ​​corresponding to 0 to 99, 100 to 199, ..., 900 to 999 sequentially for the voxels per frame.

[0102] In this case, the voxel positions of the voxels excluded from the input of the encoder for each time point may be left empty, and encoded values ​​may be sequentially placed for each time point / position.

[0103] The vacated site may be filled with a first mask token of a voxel masked at the voxel position at the time instant, and may be filled with a second mask token of an empty voxel at the voxel position at the time instant.

[0104] In this case, the first mask token and the second mask token may be the same.

[0105] Referring to FIG. 3, the voxel data 510 of the first roadside base station RSU1 and the voxel data 520 of the second roadside base station RSU2 may have all their voxels arranged at the same position.

[0106] Therefore, since masked voxel 511 and masked voxel 521 are voxels at the same position at the same time t=0, one first mask token can be generated, and in Figure 5, a total of 30 first mask tokens 710, 720 (three per time point, a total of 10 time points) can be placed at the positions of the voxels at different time points.

[0107] Since empty voxel 514 or 515 and empty voxel 524 or 525 are voxels at the same position at the same time point t=0, one second mask token can be generated. If there are two empty voxels at all time points, a total of 20 second mask tokens 730 and 740 can be placed at the voxel positions for each time point. In this case, the number of input vectors to the encoder can be 70 (=120-30-20).

[0108] Each decoder 220, 230, 240 receives the encoded value, a first mask token for the masked voxel, and a second mask token for voxels where no points exist in the voxel data for all time points and all lidars, and can generate reconstruction data for the first mask token and the second mask token.

[0109] The prediction heads 221, 231, and 241 corresponding to the decoders 220, 230, and 240 are input with the reconstruction data and can generate predicted point information.

[0110] For example, there may be three prediction heads corresponding to the LIDAR-specific or point cloud frame-specific decoders 220, 230, 240. The three prediction heads may be a first prediction head that predicts the coordinates of a point in a voxel, a second prediction head that predicts the number of points in the voxel, and a third prediction head that predicts whether the voxel at the current position to be predicted is the masked voxel or a voxel where no points exist.

[0111] The prediction heads corresponding to the first decoder 220 may be a first prediction head 221, a second prediction head 222, and a third prediction head 223.

[0112] The prediction heads corresponding to the second decoder 230 may be a first prediction head 231, a second prediction head 232, and a third prediction head 233.

[0113] For example, although FIG. 6 illustrates that three types of loss values ​​are calculated for each decoder, in reality, loss values ​​may be calculated for each mask token input to the decoder per prediction head. For example, if there are four time points for two masking voxels and one empty voxel, 12 mask tokens are input to the decoder, and each prediction head may output 12 values. A loss value may be calculated for each of the 12 output values. For example, since there are the same number of first prediction heads as there are riders, if there are two riders, the total number of outputs from the first prediction head may be 24.

[0114] The chamfer distance loss LOSS1 can be calculated using the point coordinates in the voxels predicted for each time point and for each of the multiple riders output from each first prediction head 221, 231 and the point coordinates in the voxels at the corresponding positions of the voxelized data (ground truth).

[0115] For example, for a non-empty voxel j measured at observation time t by the kth lidar among the mask tokens, fixed Q three-dimensional coordinates can be output (i.e., a total of 3Q coordinates). In this case, each output can have a value between -1 and 1 because it is output using a tanh function. The ground truth coordinates have an unlimited number of point coordinates, and all corresponding coordinates can be normalized so that the minimum and maximum values ​​for each dimension of the corresponding voxel correspond to -1 and 1, respectively, based on the center coordinate of the corresponding voxel j at time t.

[0116] Equation 2 for the chamfer distance loss value CD is:

number

[0117] In this case, j means a voxel and i means a point.

[0118] In this case, Pt means the normalized point coordinate within the voxel at time t.

[0119] JPEG2026016283000006.jpg24151

[0120] V(j, k, t) denotes the set of indices of all points within non-empty voxel j at the kth lidar and time t.

[0121] JPEG2026016283000007.jpg28151

[0122] The loss value for the number of points (Number of points loss) LOSS2 can be calculated using the number of points in the voxels predicted for each time point and for each of the multiple lidars output from each second prediction head 222, 232 and the number of points (ground truth) in the voxels (non-empty voxels) at the corresponding positions of the voxelized data.

[0123] Equation 3 for calculating the loss value for the number of points is as follows:

number

[0124] Here, N(j, k, t) denotes the number of points in voxel j at the kth lidar and time t.

[0125] JPEG2026016283000009.jpg28151

[0126] The cross-entropy loss LOSS3 can be calculated using the value indicating whether the voxel is a masked voxel predicted for each time point and for each of the multiple lidars output from each of the third prediction heads 223 and 233 and the value indicating whether the voxel at the corresponding position of the voxelized data is a masked voxel (ground truth).

[0127] For example, for all mask tokens input to the decoder, if voxel j measured by the kth lidar at time t is a non-empty voxel, it is assigned the label a(j, k, t) = 1, and if it is an empty voxel, it is assigned the label a(j, k, t) = 0, and the cross-entropy loss value can be calculated.

[0128] Equation 4 for calculating the cross-entropy loss value CE is as follows:

number

[0129] In this case, a may be a ground truth that can be confirmed by observation, and f may be the probability of a non-empty voxel output by the decoder.

[0130] Then, a final loss value (Total Loss) can be calculated by multiplying the chamfer distance loss value, the point count loss value, and the cross-entropy loss value by weighting factors and then adding them up. The neural network can be trained to minimize the final loss value. Using the final loss value, the values ​​for the three loss types can be applied to the prediction head so that they have similar values.

[0131] The equation for calculating the sum of the weighting values ​​is:

number

[0132] JPEG2026016283000012.jpg22151

[0133] JPEG2026016283000013.jpg32151

[0134] JPEG2026016283000014.jpg33151

[0135] JPEG2026016283000015.jpg19151

[0136] T means the time index number of the whole point cloud.

[0137] K stands for the total number of riders.

[0138] FIG. 7 is a diagram illustrating voxel data for point cloud frames generated in time order according to an embodiment of the present invention.

[0139] FIG. 7 is for explaining an inference process for determining whether or not a current fluctuation is due to an abnormal behavior using, for example, a neural network that has completed learning according to FIGS.

[0140] FIG. 8 is a flowchart illustrating an RSU-based LIDAR signal detection and inference method according to an embodiment of the present invention.

[0141] The following description will be made with reference to FIGS. 1, 2, 7 and 8.

[0142] In step S910, voxel data can be obtained by voxelizing the current point cloud frame measured from each of the multiple lidars 111, 121 corresponding to each of the multiple roadside base stations 110, 120 and measuring the first area 430. For example, the voxel data can be generated by each of the multiple roadside base stations 110, 120, or can be generated by the detection device 200 using the point cloud frames received from the multiple roadside base stations.

[0143] In step S920, the point cloud frames measured from each of the multiple lidars 111, 121 may be unioned to generate combined voxel data. At this time, a vector for each voxel of the combined voxel data may be calculated. At this time, a vector for a voxel at the same time and position in the combined voxel data may be generated by concatenating vectors for voxels at the same position in the voxel data where the point cloud frames measured from each of the multiple lidars are voxelized.

[0144] In step S930, if a fluctuation is detected based on voxel data in which the point cloud frame measured by at least one of the multiple lidars is voxelized, voxels 811, 821 at the first position where the fluctuation is detected are masked based on the combined voxel data, and vectors for the masked voxels and the remaining voxels other than the masked voxels are input into a neural network to obtain predicted point information for the masked voxels.

[0145] In this case, the voxel at the first position where the variation is detected may be the most suspicious voxel (e.g., voxels 811 and 821) among all the voxels where the variation is detected, or may be multiple voxels (e.g., voxels 811, 821, voxels 812 and 822) among all the voxels where the variation is detected.

[0146] In this case, the voxels at the first position may be voxels at positions included in the voxel data of each of the multiple lidars (ie, voxels in the same region).

[0147] In this case, the change can be detected in the process of sequentially tracking the movement of an object using a multi-object tracking algorithm. In this case, the change means that an observed object has disappeared or that an unobserved object has appeared. In this case, the position of the change can be determined using the multi-object tracking algorithm when the change is detected. Therefore, the voxel at the first position, which is the determined position of the change, can be masked.

[0148] In this case, according to an embodiment, step S930 may include, when masking the voxels at the first position, masking the voxels at the first position of each combined voxel data generated for the time point at which the variation is detected and for the time point before the time point at which the variation is detected, generating vectors for the remaining voxels 813, 814, 815, ..., and inputting the vectors for the masked voxels and the remaining voxels for each time point into the neural network.

[0149] In this case, the remaining voxels may be voxels excluding the voxel at the first position and voxels for which no points exist in the combined voxel data.

[0150] In this case, the step of generating vectors for the remaining voxels may include generating vectors for each voxel included in the voxel data for each rider at the same time point, and, if voxels for multiple riders exist at one voxel position in the combined voxel data, concatenating vectors of voxels for the multiple riders existing at the one voxel position and replacing the concatenated vector with a vector for the one voxel position.

[0151] For example, vectors can be generated for voxels 813, 814, and 815 of the first voxel data 810. Vectors can be generated for voxels 823, 824, and 825 of the second voxel data 820. For example, the position of voxel 813 and the position of voxel 823 may be the same. The vectors of voxel 813 and voxel 823, which exist at the same voxel position, can be concatenated with each other and replaced with the vector for the same voxel position.

[0152] For example, the neural network for inference can be the same as in FIG.

[0153] For example, if there is one masking voxel and no voxels have any points across the entire lidar for each time point, the number of inputs to the encoder can be (number of voxels in voxel data - 1) x number of time points. For example, if there are 20 voxels and 10 total time points, the number of input vectors can be 190.

[0154] In this case, the number of all time points that affect the number of inputs to the neural network encoder may be the same as the number of all time points used in the neural network learning process. For example, if the neural network is learned using voxel data for each lidar acquired at 10 time points (T=10), then during inference, voxel data for each lidar acquired at the current time point at which a change is detected and at nine consecutive time points from the current time point can be used.

[0155] In step S940, it can be determined whether the fluctuation detected at the first position is due to abnormal activity based on the predicted point information of the masked voxels and the voxelized data of each lidar.

[0156] FIG. 9 is a diagram illustrating a process for determining whether a variation in inference time is due to abnormal behavior according to an embodiment of the present invention.

[0157] As explained in the training process of Figure 6, the predicted values ​​from the prediction head corresponding to each decoder can be compared with the ground truth to calculate the loss values ​​LOSS1, LOSS2, and LOSS3.

[0158] First, a chamfering distance loss value is calculated using point coordinates within a voxel predicted for each time point and for each of the multiple riders and point coordinates within a voxel at a corresponding position of the voxelized data (see FIG. 3), and it can be determined whether the maximum value of the chamfering distance loss values ​​calculated for each time point and for each of the multiple riders is greater than a preset first critical value.

[0159] For example, referring to FIG. 1, if a specific chamfer distance loss value of the first decoder 220 is the largest among the chamfer distance loss values ​​LOSS1 (e.g., 12) of the first decoder 220 corresponding to the LIDAR 111 installed in the first roadside base station 110 and the chamfer distance loss values ​​LOSS1 (e.g., 12) of the second decoder 230 corresponding to the LIDAR 121 installed in the second roadside base station 120, the specific chamfer distance loss value of the first decoder 220 can be compared with the first critical value.

[0160] A loss value for the number of points is calculated using the number of points in the voxel predicted for each time point and for each of the multiple riders and the number of points in the voxel at the corresponding position of the voxelized data, and it can be determined whether the maximum value of the loss values ​​for the number of points calculated for each time point and for each of the multiple riders is greater than a preset second critical value.

[0161] For example, referring to FIG. 1, if the loss value for a specific number of points of the second decoder 230 is the largest among the loss value LOSS2 (e.g., 12) for the number of points of the first decoder 220 corresponding to the LIDAR 111 installed in the first roadside base station 110 and the loss value LOSS2 (e.g., 12) for the number of points of the second decoder 230 corresponding to the LIDAR 121 installed in the second roadside base station 120, the loss value for the specific number of points of the second decoder 230 can be compared with the second critical value.

[0162] A cross-entropy loss value is calculated using a value indicating whether a voxel is a masked voxel predicted for each time point and for each of the multiple riders and a value indicating whether a voxel at a corresponding position of the voxelized data is a masked voxel, and it can be determined whether the maximum value of the cross-entropy loss values ​​calculated for each time point and for each of the multiple riders is greater than a preset third critical value.

[0163] For example, referring to FIG. 1, if a specific cross-entropy loss value of the first decoder 220 is the largest among the cross-entropy loss values ​​LOSS3 (e.g., 12) of the first decoder 220 corresponding to the LIDAR 111 installed in the first roadside base station 110 and the cross-entropy loss values ​​LOSS3 (e.g., 12) of the second decoder 230 corresponding to the LIDAR 121 installed in the second roadside base station 120, the specific cross-entropy loss value of the first decoder 220 can be compared with the third threshold value.

[0164] If the determination result indicates that at least one of the first critical value, the second critical value, and the third critical value is small, it can be determined that the fluctuation detected at the first position is due to abnormal behavior.

[0165] The equation for determining whether the fluctuation detected at the first location is due to abnormal activity is:

number

[0166] JPEG2026016283000017.jpg35151

[0167] JPEG2026016283000018.jpg30151

[0168] JPEG2026016283000019.jpg19151

[0169] For example, in the embodiment of FIG. 9, the maximum values ​​of the point number loss and cross entropy loss are smaller than the second and third critical values, respectively, while the maximum value of the chamfer distance loss is larger than the first critical value, so it can be determined that the cause of the fluctuation detected at the current time point is abnormal behavior.

[0170] In another embodiment, when a change is detected based on voxel data obtained by voxelizing point cloud frames measured by at least one of a number of lidars corresponding to each of a number of roadside base stations, the process of masking the voxel at the first position where the change is detected is as follows.

[0171] Voxel data obtained by voxelizing point cloud frames at past times from the at least one or more lidars measuring an area including the voxel at the first location, and voxel data obtained by voxelizing point cloud frames at the current time and past times from each of lidars other than the at least one or more lidars measuring an area including the voxel at the first location, can be obtained.

[0172] After forming a union of the voxel data of all lidars measuring the area containing the voxel at the first location at the current time and past time, combined voxel data can be generated.

[0173] All voxels at the first position at the current time point and past time points can be masked based on the combined voxel data.

[0174] In this case, the number of neural network decoders (and three prediction heads corresponding to the decoders) can be selected and used to be the same as the number of lidars measuring the area including the voxel at the first position.

[0175] FIG. 10 is a conceptual diagram illustrating an example of a generalized RSU-based LIDAR signal anomaly detection device or computing system capable of performing at least some of the processes of FIGS. 1 to 9.

[0176] At least some of the processes of the RSU-based LIDAR signal anomaly detection method and the RSU-based LIDAR signal anomaly detection learning method according to an embodiment of the present invention may be performed by the computing system 1000 of FIG. 10 .

[0177] Referring to FIG. 10, a computing system 1000 according to one embodiment of the present invention may include a processor 1100, a memory 1200, a communication interface 1300, a storage device 1400, an input interface 1500, an output interface 1600, and a bus 1700.

[0178] A computing system 1000 according to an embodiment of the present invention may include at least one processor 1100 and a memory 1200 that stores instructions for instructing the at least one processor 1100 to perform at least one step. At least some steps of a method according to an embodiment of the present invention may be performed by the at least one processor 1100 loading and executing instructions from the memory 1200.

[0179] The processor 1100 may represent a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed.

[0180] Each of the memory 1200 and the storage device 1400 may be composed of at least one of a volatile storage medium and a non-volatile storage medium, for example, the memory 1200 may be composed of at least one of a read only memory (ROM) and a random access memory (RAM).

[0181] The computing system 1000 may also include a communication interface 1300 for communicating over a wireless network.

[0182] The computing system 1000 may further include a storage device 1400, an input interface 1500, an output interface 1600, and the like.

[0183] Furthermore, the components included in the computing system 1000 are connected by a bus 1700 to enable communication.

[0184] The computing system 1000 of the present invention may be, for example, a communications-enabled desktop computer, a laptop computer, a notebook computer, a smartphone, a tablet PC, a mobile phone, a smart watch, smart glass, an e-book reader, a portable multimedia player (PMP), a portable game console, a navigation device, a digital camera, a digital multimedia broadcasting (DMB) player, a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, a personal digital assistant (PDA), or the like.

[0185] The operations of the methods according to the embodiments of the present invention may be embodied as a computer-readable program or code stored in a computer-readable recording medium. The computer-readable recording medium may include any type of recording device that stores information readable by a computer system. The computer-readable recording medium may also be distributed among computer systems connected via a network, allowing the computer-readable program or code to be stored and executed in a distributed manner.

[0186] The computer-readable recording medium may also include a hardware device specially configured to store and execute program instructions, such as a ROM, RAM, flash memory, etc. The program instructions may include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0187] Some aspects of the present invention have been described in terms of devices, but they can also be described in terms of corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in terms of methods can also be described in terms of corresponding blocks or items or features of corresponding devices. Some or all of the method steps can be performed by (or using) hardware devices, such as, for example, microprocessors, programmable computers, or electronic circuits. In some embodiments, at least one or more of the most important method steps can be performed by such devices.

[0188] In embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, a field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. In general, it is preferred that the methods be performed by some hardware device.

[0189] Although the present invention has been described above with reference to preferred embodiments, it will be understood by those skilled in the art that various modifications and variations of the present invention can be made without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims

1. a memory for storing at least one instruction; a processor for executing said at least one instruction; and neural networks, The processor: Acquire voxel data in which a current point cloud frame measured by each of a number of lidars corresponding to each of a number of roadside base stations and measuring a first region is voxelized; forming a union of voxel data from the point cloud frames measured from each of the multiple lidars, and then generating combined voxel data; When a variation is detected based on voxel data obtained by voxelizing point cloud frames measured by at least one of the multiple lidars, Based on the combined voxel data, mask a voxel at a first position where the variation is detected, and input vectors for the masked voxel and the remaining voxels other than the masked voxel into the neural network to obtain predicted point information for the masked voxel; and an RSU-based LIDAR signal anomaly detection device that determines whether fluctuations detected at the first location are due to abnormal activity based on the predicted point information of the masked voxels and the voxelized data of each LIDAR.

2. When masking the voxels at the first position, mask the voxels at the first position in each combined voxel data generated for a time point before the time point at which the variation is detected; The RSU-based LIDAR signal anomaly detection device of claim 1 , wherein the neural network further receives vectors for the masked voxels and the remaining voxels for each of the previous time points.

3. At the time the variation is detected and at the previous time, The combined voxel data is data in which voxels have the same size and position for the same measurement region at the same time point, 3. The RSU-based LIDAR signal anomaly detection device of claim 2, wherein the vectors for the voxels at the same position at the same time are generated by concatenating vectors for the voxels at the same position of voxel data obtained by voxelizing point cloud frames measured from each of the multiple LIDARs.

4. The neural network includes an encoder, a number of decoders equal to the number of lidars, and prediction heads corresponding to the decoders, respectively; inputting the vector into the encoder to obtain an encoded value; inputting the encoded value, a first mask token for the masked voxel, and a second mask token for a voxel where no point exists at the time point when the variation is detected and at all the previous time points into decoders the same number as the LIDAR, to respectively obtain reconstruction data for the first mask token and the second mask token; The RSU-based LIDAR signal anomaly detection device of claim 2 , wherein the reconstruction data is input to a prediction head to obtain the predicted point information.

5. The RSU-based LIDAR signal anomaly detection device of claim 1 , wherein the fluctuations are detected by a multi-object tracking algorithm.

6. 2. The RSU-based LIDAR signal anomaly detection device of claim 1, wherein the predicted point information includes at least one of the number of points in the voxel predicted for the position of the masked voxel, coordinates of the points, and information on whether the voxel is a masked voxel.

7. 5. The RSU-based LIDAR signal anomaly detection device of claim 4, wherein the prediction heads include a first prediction head that predicts coordinates of a point within a voxel, a second prediction head that predicts the number of points within the voxel, and a third prediction head that predicts whether the voxel at the current position to be predicted is the masked voxel or a voxel where no point exists.

8. The processor: Calculating a chamfering distance loss value using point coordinates in a voxel for each time point predicted by the first prediction head and point coordinates in a voxel at a corresponding position of voxelized data for each time point of a rider matched with the first prediction head, and determining whether a maximum value of the chamfering distance loss values ​​calculated for each time point and for the plurality of riders is greater than a predetermined first critical value; calculating a loss value for the number of points using the number of points in a voxel for each time point predicted by the second prediction head and the number of points in a voxel at a corresponding position of the voxelized data for each time point of a rider matched with the second prediction head; and determining whether the maximum value of the loss values ​​for the number of points calculated for each time point and for each of the plurality of riders is greater than a second predetermined critical value; Calculating a cross-entropy loss value using a value indicating whether the voxel predicted by the third prediction head is a masked voxel and a value indicating whether a voxel at a corresponding position of the voxelized data of the rider matched with the third prediction head is a masked voxel, and determining whether a maximum value among the cross-entropy loss values ​​calculated for each time point and for each of the plurality of riders is greater than a third predetermined critical value; 8. The RSU-based LIDAR signal anomaly detection device of claim 7, wherein when the determination result indicates that at least one of the first critical value, the second critical value, and the third critical value is small, it is determined that the fluctuation detected at the first position is due to an abnormal activity.

9. The neural network Obtaining voxel data in which point cloud frames measured at each time point from a plurality of lidars corresponding to each of the plurality of roadside base stations and measuring the first area are voxelized; forming a union of voxel data from the point cloud frames measured from each of the multiple lidars, and then generating combined voxel data; Masking a predetermined number of voxels at random positions in the combined voxel data for each time point, and masking voxels at the same positions for each time point; inputting vectors for the predetermined number of masked voxels for each time point and the remaining voxels other than the masked voxels into the neural network to obtain predicted point information for the masked voxels; 2. The RSU-based LIDAR signal anomaly detection device of claim 1, wherein a value calculated using a loss function between the predicted point information of the masked voxel and the voxel information at the same position is learned to satisfy a predetermined condition.

10. 8. The RSU-based LIDAR signal anomaly detection device of claim 7, wherein the neural network is trained to minimize a final loss value obtained by multiplying a chamfer distance loss, a loss for the number of points, and a cross-entropy loss calculated using voxelized data measured for each time point and for each LIDAR with a weighting value and then adding them up for the predicted point information of the masked voxels.

11. acquiring voxel data from a current point cloud frame measured by each of a plurality of lidars corresponding to each of a plurality of roadside base stations and measuring a first area; generating a union of voxel data obtained by voxelizing point cloud frames measured from the respective multiple lidars; When a variation is detected based on voxel data obtained by voxelizing point cloud frames measured by at least one of the multiple lidars, masking a voxel at a first position where the variation is detected based on the combined voxel data, and inputting vectors for the masked voxel and the remaining voxels other than the masked voxel into a neural network to obtain predicted point information for the masked voxel; determining whether the fluctuation detected at the first location is due to abnormal activity based on the predicted point information of the masked voxels and the voxelized data of each of the lidars.

12. The step of acquiring predicted point information includes: When masking the voxels at the first location, masking the voxels at the first location in each of the combined voxel data generated for the time at which the variation was detected and for the time before the time at which the variation was detected; and 12. The method of claim 11, further comprising inputting vectors for the masked voxels and the remaining voxels for each time point into the neural network.

13. At the time the variation is detected and at the previous time, The combined voxel data is data in which voxels have the same size and position for the same measurement region at the same time point, The step of acquiring predicted point information includes: generating vectors for the remaining voxels; The generating step includes: generating vectors for each voxel included in the voxel data for each lidar at the same time point; 13. The RSU-based LIDAR signal anomaly detection method of claim 12, further comprising: when voxels for multiple LIDARs exist at one voxel position in the combined voxel data, concatenating vectors of voxels for the multiple LIDARs present at the one voxel position and replacing the concatenated vector with the vector for the one voxel position.

14. 12. The method of claim 11, further comprising detecting the variations using a multi-object tracking algorithm.

15. The predicted point information includes point coordinates in a voxel for each time point, the number of points, and information on whether the voxel is a masked voxel, determining whether the fluctuation detected at the first location is due to abnormal activity includes: calculating chamfering distance loss values ​​using point coordinates in voxels predicted for each time point and for each of the plurality of riders and point coordinates in voxels at corresponding positions of the voxelized data, and determining whether a maximum value among the chamfering distance loss values ​​calculated for each time point and for each of the plurality of riders is greater than a predetermined first critical value; calculating a loss value for the number of points using the number of points in a voxel predicted for each time point and for each of the multiple riders and the number of points in a voxel at a corresponding position of the voxelized data, and determining whether the maximum value of the loss values ​​for the number of points calculated for each time point and for each of the multiple riders is greater than a predetermined second critical value; calculating a cross-entropy loss value using a value indicating whether a voxel at a corresponding position of the voxelized data is a masked voxel and a value indicating whether a voxel at a corresponding position of the voxelized data is a masked voxel for each time point and for each of the plurality of riders; and determining whether a maximum value of the cross-entropy loss values ​​calculated for each time point and for each of the plurality of riders is greater than a third predetermined critical value; 12. The method of claim 11, further comprising: determining that the fluctuation detected at the first location is due to an abnormal activity when it is determined that at least one of the first critical value, the second critical value, and the third critical value is small.

16. acquiring voxel data by voxelizing point cloud frames measured by a plurality of lidars corresponding to a plurality of roadside base stations and measuring a first area; generating a union of voxel data obtained by voxelizing point cloud frames measured from the respective multiple lidars; masking a predetermined number of randomly positioned voxels in the combined voxel data; and inputting vectors for the predetermined number of masked voxels and the remaining voxels other than the masked voxels into a neural network to obtain predicted point information for the masked voxels; The neural network is trained so that a value calculated using a loss function between predicted point information of the masked voxel and voxel information at the same position satisfies a predetermined condition.

17. The step of acquiring voxelized data includes acquiring voxelized data of point cloud frames measured from each of the plurality of lidars at different time points, The step of masking the voxels includes masking voxels at the same position for each time point, 17. The RSU-based LIDAR signal anomaly detection learning method of claim 16, wherein vectors for the predetermined number of masked voxels for each time point and the remaining voxels other than the masked voxels are input to the neural network.

18. The combined voxel data is data in which voxels have the same size and position for the same measurement region at the same time point, The step of acquiring predicted point information includes: generating vectors for the remaining voxels; The generating step includes: generating vectors for each voxel included in the voxel data for each lidar at the same time point; 17. The RSU-based LIDAR signal anomaly detection learning method of claim 16, further comprising: when voxels for multiple LIDARs exist at one voxel position in the combined voxel data, concatenating vectors of voxels for the multiple LIDARs present at the one voxel position and replacing the concatenated vector with the vector for the one voxel position.

19. 17. The RSU-based LIDAR signal anomaly detection learning method of claim 16, wherein the predicted point information includes point coordinates within a voxel for each time point, the number of points, and information regarding whether the voxel is a masked voxel.

20. After the step of obtaining the predicted point information, Calculating a chamfer distance loss value using point coordinates in a voxel predicted for each time point and for each of the plurality of lidars and point coordinates in a voxel at a corresponding position of the voxelized data; calculating a loss value for the number of points using the number of points in a voxel predicted for each time point and for each of the plurality of lidars and the number of points in a voxel at a corresponding position of the voxelized data; Calculating a cross-entropy loss value using a value indicating whether a voxel is a masked voxel predicted for each time point and for each of the plurality of lidars and a value indicating whether a voxel at a corresponding position of the voxelized data is a masked voxel; and The method further includes calculating a final loss value by multiplying the chamfer distance loss value, the loss value for the number of points, and the cross-entropy loss value by weighting values ​​and then summing the weighted values; 20. The RSU-based LIDAR signal anomaly detection training method of claim 19, wherein the neural network is trained to minimize the final loss value.

Images