RSU-based signal anomaly detection method and apparatus therefor

Abstract

This invention provides an RSU-based signal anomaly detection device and method that uses a multi-object tracking algorithm to determine whether fluctuations are natural phenomena or the result of a spoofing attack. [Solution] Combined voxel data is generated by combining voxel data from lidar and radar installed at a roadside base station and 3D feature vectors from multiple cameras installed at the roadside base station, with voxels at corresponding locations. Based on the combined voxel data, the voxel at the first location where the fluctuation is detected is masked. The vectors for the masked voxel and the remaining voxels other than the masked voxel are input to a neural network to obtain predicted point information for the masked voxel. Based on the predicted point information for the masked voxel and the lidar voxel data, it is determined whether the fluctuation detected at the first location is due to abnormal activity.

Description

[Technical Field]

[0001] This invention relates to a method for detecting RSU (roadside unit) base signal anomalies and an apparatus for the same purpose, and in particular aims to provide a method for detecting position base anomalies using lidar, radar, and cameras. [Background technology]

[0002] Various methods for detecting LiDAR spoofing attacks have been published. For example, a single point cloud frame-based method detects the occlusion pattern that occurs when a spoofing attack is performed on a single LiDAR image and generates a 3D object, which differs from that of a normal 3D object.

[0003] Another conventional method for detecting Rider spoofing attacks is to combine MotionNet and 3D object detection using a time-domain consistency detection method for 3D object levels of point cloud frames to detect spoofing attacks.

[0004] While various conventional multi-object tracking algorithms are used to track objects, there is a problem in that it is impossible to determine whether an object that is suddenly detected or suddenly disappears was blocked by surrounding objects, or whether it is an abnormal act detected due to a spoofing attack. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The present invention aims to provide an RSU-based signal anomaly detection device and method that can determine whether a fluctuation detected through a multiple object tracking algorithm is a natural phenomenon or an abnormal act caused by a spoofing attack.

[0006] The present invention aims to provide an RSU-based signal anomaly detection device and method that can detect point cloud distortion caused by lidar spoofing by learning the correlation between lidar, radar, and camera data. [Means for solving the problem]

[0007] An RSU-based signal anomaly detection device provided in one aspect of the present invention includes a memory for storing at least one instruction; a processor for executing the at least one instruction; and a neural network, wherein the processor acquires first voxel data, which is a voxelized version of a point cloud frame at the present time measured from a lidar installed at a roadside base station; and if a variation is detected based on the first voxel data, the first voxel data, second voxel data, which is a voxelized version of a radar frame measured from a radar installed at the roadside base station, and an image frame captured by a camera installed at the roadside base station. Combined voxel data is generated by combining the first feature vectors extracted from the frame with the voxels at their corresponding positions. Based on the combined voxel data, the voxel at the first position where the variation is detected is masked. Vectors for the masked voxel and the remaining voxels other than the masked voxel are input to the neural network to obtain predicted point information for the masked voxel. Based on the predicted point information for the masked voxel and the first voxel data of the lidar, it is possible to determine whether the variation detected at the first position is due to abnormal behavior.

[0008] At this time, the roadside base station is equipped with multiple cameras, including the aforementioned camera, and the first feature vector may be extracted from image frames captured by each of the multiple cameras.

[0009] At this time, when the processor generates the combined voxel data, it generates a lidar voxel feature embedding vector for each voxel included in the first voxel data, generates a radar voxel feature embedding vector for each voxel included in the region of the second voxel data that overlaps with the first voxel data, and connects the lidar voxel feature embedding vector, the radar voxel feature embedding vector, and the first feature vector for voxels at the same location to generate the combined voxel data.

[0010] At this time, the processor can generate a two-dimensional image feature vector from the image frame, generate a probability value of the distance value for each pixel of the image frame, and multiply the probability value of the distance value for each pixel by the two-dimensional image feature vector of each pixel to generate the first feature vector corresponding to each voxel of the first voxel data.

[0011] At this time, when the processor masks the voxel at the first position, it masks the voxel at the first position in each combined voxel data generated for a time point prior to the time when the variation was detected, and the neural network may further receive vectors for the masked voxel and the remaining voxels for each of the earlier time points.

[0012] At this time, the neural network includes an encoder, a decoder, and a prediction head. The processor inputs vectors for voxels that are empty voxels where there are no points at all times including the time when the change is detected and the previous time and the remaining voxels other than the masked voxels into the encoder to obtain encoded values, inputs the encoded values, a first mask token for the masked voxels, and a second mask token for the masked empty voxels obtained by masking the empty voxels into the decoder to obtain reconstructed data for the first mask token and the second mask token respectively, and inputs the reconstructed data into the prediction head to obtain the predicted point information.

[0013] At this time, the predicted point information can include point coordinates in the voxel by time, the number of points in the voxel by time, and a value for whether it is a masked empty voxel. The processor calculates a loss value by using the point coordinates in the voxel by time, the number of points in the voxel by time, and the value for whether it is a masked empty voxel corresponding to the predicted point information and the point coordinates in the voxel by time, the number of points in the voxel by time, and the value for whether it is a masked empty voxel of the first voxel data of the lidar corresponding respectively, compares the loss value with the corresponding threshold value, and if it is determined that at least one of the loss values is greater than the corresponding threshold value, it can be determined that the change detected at the first position is due to an abnormal behavior.

[0014] At this time, the neural network obtains first training voxel data, which is obtained by voxelizing point cloud frames measured at each time point by a lidar installed at the roadside base station; second training voxel data, which is obtained by voxelizing radar frames measured at each time point by a radar installed at the roadside base station; first training feature vectors, which are extracted from image frames captured by a camera installed at the roadside base station; and combined voxel data, which is obtained by combining the time-specific first training voxel data, the time-specific second training voxel data, and the time-specific first training feature vectors at each time point using voxels at corresponding positions. The process involves generating a dataset and masking a predetermined number of voxels at random positions using the time-series combined voxel data, while masking the same voxels at each time point. The neural network is then input with vectors representing the predetermined number of masked voxels and the remaining voxels other than the masked voxels to obtain predicted point information for the masked voxels. The network may be trained to satisfy predetermined conditions by using a loss function between the predicted point information for the masked voxels and the voxel information at the same position in the first training voxel data.

[0015] The RSU-based signal anomaly detection method provided by one aspect of the present invention includes the steps of: obtaining first voxel data obtained by voxelizing a point cloud frame at the current time measured by a lidar installed in a roadside base station; when a variation is detected based on the first voxel data, combining the first voxel data, second voxel data obtained by voxelizing a radar frame measured by a radar installed in the roadside base station, and a first feature vector extracted from an image frame captured by a camera installed in the roadside base station with voxels at corresponding positions to generate combined voxel data; based on the combined voxel data, masking the voxel at the first position where the variation is detected, and inputting vectors for the masked voxel and the remaining voxels other than the masked voxel into the neural network to obtain predicted point information for the masked voxel; and determining whether the variation detected at the first position is due to an abnormal behavior based on the predicted point information for the masked voxel and the first voxel data of the lidar.

[0016] At this time, the step of generating the combined voxel data may include the steps of: generating a two-dimensional image feature vector from image frames captured by a plurality of cameras installed in the roadside base station including the camera; generating a probability value of a distance value for each pixel of the image frame; and multiplying the two-dimensional image feature vector of each pixel by the probability value of the distance value for each pixel to generate the first feature vector corresponding to each voxel of the first voxel data.

[0017] In this case, the step of generating the combined voxel data may include: generating a lidar voxel feature embedding vector for each voxel included in the first voxel data; generating a radar voxel feature embedding vector for each voxel included in the region of the second voxel data that overlaps with the first voxel data; and generating the combined voxel data by concatenating the lidar voxel feature embedding vector, the radar voxel feature embedding vector, and the first feature vector for voxels at the same location.

[0018] At this time, the step of acquiring the predicted point information may include the step of masking the voxel at the first position when masking the voxel at the first position, masking the voxel at the first position in each combined voxel data generated for the time at which the variation was detected and for time prior to the time at which the variation was detected; and inputting vectors for the time-specific masked voxels and the remaining voxels into the neural network.

[0019] In this case, the neural network includes an encoder, a decoder, and a prediction head, and the steps of acquiring the predicted point information may further include: inputting vectors for empty voxels (voxels that do not have points at the time the variation is detected and all prior time points) and the remaining voxels other than the masked voxels into the encoder to acquire an encoded value; inputting the encoded value, a first mask token for the masked voxel, and a second mask token for the masked empty voxel that masks the empty voxel into the decoder to acquire reconstructed data for the first mask token and the second mask token, respectively; and inputting the reconstructed data into the prediction head to acquire the predicted point information.

[0020] At this time, the predicted point information includes the point coordinates within the time-series voxel, the number of points within the time-series voxel, and a value indicating whether or not it is a masked empty voxel. The step of determining whether the fluctuation detected at the first position is due to abnormal activity may include: calculating a loss value using the point coordinates within the time-series voxel, the number of points within the time-series voxel, and the value indicating whether or not it is a masked empty voxel, and the corresponding point coordinates within the time-series voxel, the number of points within the time-series voxel, and the value indicating whether or not it is a masked empty voxel from the first voxel data of the Lidar; comparing the loss value with a corresponding critical value; and determining that the fluctuation detected at the first position is due to abnormal activity if at least one of the loss values ​​is determined to be greater than the corresponding critical value.

[0021] A RSU-based signal anomaly detection learning method provided in one aspect of the present invention includes the steps of: acquiring first voxel data obtained by voxelizing point cloud frames measured from a lidar installed at a roadside base station; acquiring second voxel data obtained by voxelizing radar frames measured from a radar installed at the roadside base station; acquiring a first feature vector extracted from an image frame captured by a camera installed at the roadside base station; and combining voxel data obtained by joining the first voxel data, the second voxel data, and the first feature vector at corresponding positions. The process includes a generation step; a step of masking a predetermined number of voxels at random positions in the combined voxel data; and a step of inputting the predetermined number of masked voxels and vectors for the remaining voxels other than the masked voxels into the neural network to obtain predicted point information for the masked voxels, wherein the neural network can be trained to satisfy predetermined conditions by using a loss function between the predicted point information for the masked voxels and the voxel information at the same position in the first voxel data.

[0022] In this case, the step of acquiring the first feature vector may include the steps of: generating a two-dimensional image feature vector from image frames captured by each of the multiple cameras installed at the roadside base station, including the camera; generating probability values ​​of distance values ​​for each pixel of the image frame; and generating the first feature vector corresponding to each voxel of the first voxel data by multiplying the probability values ​​of distance values ​​for each pixel by the two-dimensional image feature vector of each pixel.

[0023] In this case, the step of generating the combined voxel data may include: generating a lidar voxel feature embedding vector for each voxel included in the first voxel data; generating a radar voxel feature embedding vector for each voxel included in the region of the second voxel data that overlaps with the first voxel data; and generating the combined voxel data by concatenating the lidar voxel feature embedding vector, the radar voxel feature embedding vector, and the first feature vector for voxels at the same location.

[0024] At this time, the steps of acquiring the first voxel data, acquiring the second voxel data, and acquiring the first feature vector are each steps performed at multiple time points, and the step of masking the voxels is a step of masking the voxels at the same position at each time point, and the predetermined number of masked voxels and vectors for the remaining voxels other than the masked voxels at each time point can be input to the neural network.

[0025] In this case, the neural network includes an encoder, a decoder, and a prediction head, and the steps of acquiring the predicted point information may further include: inputting vectors for empty voxels (voxels that do not have points at the time the variation is detected and all prior time points) and the remaining voxels other than the masked voxels into the encoder to acquire an encoded value; inputting the encoded value, a first mask token for the masked voxel, and a second mask token for the masked empty voxel that masks the empty voxel into the decoder to acquire reconstructed data for the first mask token and the second mask token, respectively; and inputting the reconstructed data into the prediction head to acquire the predicted point information.

[0026] At this time, after the step of acquiring the predicted point information, the neural network may be trained to minimize the final loss value, further including the steps of: calculating a chamfer distance loss value using the point coordinates in the voxel predicted for each time point and the point coordinates in the voxel at the corresponding position in the first voxel data; calculating a loss value for the number of points using the number of points in the voxel predicted for each time point and the number of points in the voxel at the corresponding position in the first voxel data; calculating a cross-entropy loss value using the value for whether or not it is a masked empty voxel predicted for each time point and the value for whether or not the voxel at the corresponding position in the first voxel data is a masked empty voxel; and calculating a final loss value by multiplying the chamfer distance loss value, the loss value for the number of points, and the cross-entropy loss value by a weighted value and adding them together. [Effects of the Invention]

[0027] According to the present invention, when a fluctuation is detected through a multiple object tracking algorithm, it is possible to provide an RSU base signal anomaly detection device and method that can determine whether the fluctuation is a natural phenomenon or an abnormal act caused by a spoofing attack.

[0028] According to the present invention, an RSU-based signal anomaly detection device and method can be provided that can detect point cloud distortion caused by lidar spoofing by learning the correlation between lidar, radar, and camera data. [Brief explanation of the drawing]

[0029] [Figure 1] This is a diagram illustrating the configuration of a roadside base station infrastructure anomaly signal detection device according to one embodiment of the present invention. [Figure 2] This diagram shows the sensing range of a lidar and radar and the shooting range of a camera according to one embodiment of the present invention. [Figure 3] This shows voxelized data of a lidar point cloud frame according to one embodiment of the present invention. [Figure 4] This shows voxel data obtained by voxelizing the radar frame of a radar according to one embodiment of the present invention. [Figure 5] This diagram illustrates the process of generating feature vectors and height value probability distributions from a camera image frame according to one embodiment of the present invention. [Figure 6] This diagram illustrates the process of calculating the camera feature vector for each voxel according to one embodiment of the present invention. [Figure 7] This diagram illustrates the process of combining voxel feature vectors for lidar, radar, and camera according to one embodiment of the present invention. [Figure 8] This is a diagram illustrating voxel data generated in chronological order according to one embodiment of the present invention. [Figure 9] This is a flowchart illustrating an RSU-based lidar signal detection and learning method according to one embodiment of the present invention. [Figure 10] This is a diagram illustrating the structure of a third neural network according to one embodiment of the present invention. [Figure 11] This is a diagram illustrating voxel data for point cloud frames generated in chronological order according to one embodiment of the present invention. [Figure 12] This is a flowchart illustrating an RSU-based lidar signal detection and inference method according to one embodiment of the present invention. [Figure 13] This is a conceptual diagram illustrating an example of a generalized RSU-based signal anomaly detection device or computing system capable of performing at least part of the processes shown in Figures 1 to 12. [Modes for carrying out the invention]

[0030] While the present invention can be modified in various ways and has many embodiments, specific embodiments will be illustrated and described in detail in the drawings. However, this should not be understood as limiting the present invention to specific embodiments, but rather as including all modifications, equivalents, or substitutes that fall within the spirit and technical scope of the present invention.

[0031] The terms "first," "second," etc., may be used to describe a variety of components, but the components should not be limited by such terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The terms "and / or" include combinations of multiple related described items or any of the multiple related described items.

[0032] In the embodiments of this application, "at least one of A and B" may mean "at least one of A or B" or "at least one of one or more combinations of A and B". Also, in the embodiments of this application, "one or more of A and B" may mean "one or more of A or B" or "one or more of one or more combinations of A and B".

[0033] When it is stated that one component is "linked" or "connected" to another component, it should be understood that it may be directly linked or connected to the other component, but there may also be other components in between. On the other hand, when it is stated that one component is "directly linked" or "directly connected" to another component, it should be understood that there are no other components in between.

[0034] The terminology used in this application is used solely to describe specific embodiments and is not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “includes” or “having” are intended to specify the existence of features, figures, stages, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the possibility of the existence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0036] The following describes preferred embodiments of the present invention in more detail with reference to the attached drawings. In describing the present invention, the same reference numerals are used for the same components in the drawings to facilitate overall understanding, and redundant descriptions of the same components are omitted.

[0037] Figure 1 is a diagram illustrating the configuration of a roadside base station infrastructure anomaly signal detection device according to one embodiment of the present invention.

[0038] The roadside base station (RSU) 100 can be permanently installed, for example, at an intersection.

[0039] The roadside base station 100 has the following characteristics:

[0040] Since the roadside base station 100 is installed in a fixed location, it is less exposed to environmental changes. Therefore, the stability and consistency of the data collected from the sensing device installed at the roadside base station 100 can be guaranteed. On the other hand, data collected from a moving vehicle may be highly volatile due to various external factors such as the vehicle's position, speed, and direction.

[0041] Because the same area can be monitored over a long period of time through fixed roadside base stations 100, it is possible to observe changes in a specific area over time and detect abnormal phenomena more accurately. On the other hand, since moving vehicles do not always repeat a fixed route, it may be difficult to collect long-term data from the same area.

[0042] Since roadside base stations 100 are connected to fixed infrastructure, they may be advantageous for processing and storing large amounts of data in real time.

[0043] The fixed roadside base station 100 can more accurately detect location-based anomalies that may occur at specific locations (e.g., illegal parking, walking in pedestrian-restricted areas, etc.).

[0044] Therefore, the present invention aims to provide an anomaly detection device and method utilizing roadside base stations.

[0045] A roadside base station (RSU) 100 may be equipped with a LiDAR 110, a radar 120, one or more cameras 130, a first neural network 140, a second neural network 150, and a third neural network 160.

[0046] For example, a roadside base station 100 may be equipped with a first camera 131, a second camera 132, a third camera 133, and a fourth camera 134 that capture images in the east, west, north, and south directions centered on the base station 100. In this case, multiple cameras can be integrated and referred to as a multi-view camera.

[0047] Lidar 110, radar 120, and one or more cameras 130 installed at roadside base stations 100 can be used to monitor general traffic conditions.

[0048] LiDAR has advantages such as excellent 3D environment mapping capabilities, precise distance measurement, and relatively robustness against weather conditions. On the other hand, it has disadvantages such as high costs, large data processing requirements, and the absence of hue information. LiDAR can be subjected to spoofing attacks through methods such as manipulated signal injection, laser light source jamming, and reflector use.

[0049] Radar has advantages such as a wide detection range, operation even in extreme weather conditions, and excellent high-speed tracking. On the other hand, radar has disadvantages such as low resolution, limited object identification capabilities, and the possibility of interference. In the case of radar, spoofing attacks can be carried out using methods such as replay attacks, signal jamming, and false target generation.

[0050] Multi-view cameras have the advantages of providing high-resolution images, hue and texture information, and being cost-effective. On the other hand, multi-view cameras have the disadvantage of degrading performance in low light or direct sunlight and being affected by weather conditions. Multi-view cameras can also be subjected to spoofing attacks through methods such as projection attacks, manipulated image / video transmission, and the use of strong light sources.

[0051] For an attacker, attempting a spoofing attack targeting only one specific sensor can disrupt the normal operation of a traffic monitoring system. Since the strengths and weaknesses of each sensor are clearly defined by weather conditions, it may be advantageous to fuse information obtained from various sensors rather than using a single sensor.

[0052] The point cloud frames measured by LiDAR 111 can be voxelized and used as voxel data. At this time, various multi-object tracking algorithms can be used for object tracking with the voxel data measured and acquired by LiDAR. That is, by using a multi-object tracking algorithm, vehicles, pedestrians, etc., can be detected in each voxel of the voxel data, and the detected object can be tracked. At this time, bounding boxes for the detected object can be generated. The bounding boxes for the detected object can be generated for each point cloud frame measured and generated by LiDAR for tracking.

[0053] For example, if a vehicle that was following another vehicle enters the lidar sensing range due to a lane change or other reason, the multi-object tracking algorithm can determine that a new vehicle has suddenly been detected. For example, if a detected / tracked vehicle moves behind another vehicle or is obstructed by another vehicle, the multi-object tracking algorithm can determine that the tracked vehicle has suddenly disappeared.

[0054] On the other hand, in a situation where a lidar spoofing attack is being carried out, the signal from the lidar 110 installed at the roadside base station 100 (i.e., the voxel data of the point cloud frame measured from the lidar) may have the following characteristics:

[0055] For example, a vehicle might suddenly be detected in a space in the voxel data where there was no vehicle before. Or, a vehicle might suddenly disappear in a space in the voxel data where there was a vehicle before.

[0056] Therefore, since the disappearance and reappearance of vehicles occur identically in both normal and spoofing attack situations, it should be possible to distinguish the cause of the fluctuations. However, if a sophisticated spoofing attack can make vehicles appear to disappear and reappear in a way that makes it impossible to distinguish between normal and spoofing situations, then it may be practically impossible to determine the cause of the fluctuations.

[0057] When a spoofing attacker attacks a single roadside base station 100, they only need to consider the LiDAR signal of the roadside base station 100, which makes a sophisticated spoofing attack relatively easy.

[0058] While it is possible to detect the presence or absence of an attack by monitoring the abnormal operation of each sensor, this invention aims to provide a method for detecting the presence or absence of anomalies overall by fusing related sensor information.

[0059] The first neural network 140 is a camera neural network that can receive image input from camera 130 and generate pixel-specific feature vectors and probability distributions for distance. The first neural network 140, being a camera neural network, may be a CNN-type neural network.

[0060] The second neural network 150 is a neural network for generating the final vector, and can generate the final vector by concatenating the vectors generated using the lidar 110, radar 120, and camera 130.

[0061] The third neural network 160 is a transformer neural network that can be used to determine whether the detected fluctuations are due to abnormal behavior by utilizing the final vector.

[0062] Figure 2 shows the sensing range of the lidar and radar and the shooting range of the camera according to one embodiment of the present invention.

[0063] The following explanation will refer to both Figure 1 and Figure 2.

[0064] The lidar 110 and radar 120 can sense a 360-degree range. In this case, the coverage area 210 of the lidar 110 may be smaller than the coverage area 220 of the radar 130.

[0065] The sensing data from LiDAR 110 can be three-dimensional information. On the other hand, the sensing data from Radar 120 can be two-dimensional information lacking height information.

[0066] The LiDAR 110 can scan traffic conditions around an intersection in all directions to generate and store point cloud frames in chronological order. It can be assumed that the point cloud is collected for each point in the form (x, y, z, intensity, t), where x, y, and z are 3D coordinate values ​​and t may represent the measured time.

[0067] The height at which the radar 120 is installed may be fixed. The radar 120 can scan the traffic situation around the intersection in all directions and generate radar frames in chronological order. Unlike the lidar 110, it can be assumed that the radar 120 does not have height information (z-axis). Since the radar 120 can measure speed using the Doppler effect, it can be assumed that detected objects are collected in the form of (x, y, rcs, vx, vy, t). In this case, rcs means radar cross section, and vx and vy may mean speed in the x and y directions, respectively.

[0068] The shooting areas 231-234 of cameras 131-134 may be limited by the field of view of cameras 131-134. Therefore, by installing multiple cameras facing various directions around the roadside base station 100, it is possible to generate multi-view frames of traffic conditions in chronological order.

[0069] Spoofing camera images may be relatively difficult. It would be nearly impossible to implement a method that distorts optical information directed towards the camera lens for the purpose of actually carrying out an attack. Manipulating the camera image by hacking the system internally would be extremely difficult. Furthermore, if there is overlap in the areas illuminated by two cameras, attacks on those cameras must be carried out simultaneously.

[0070] When carrying out a LiDAR spoofing attack, the correlation between multiview image frames captured by a camera and point cloud frames measured by LiDAR may not be maintained.

[0071] Therefore, by utilizing deep learning to learn the correlation between multi-view image frames, radar frames, and point cloud frames, it is possible to detect point cloud distortion caused by RID spoofing.

[0072] Figure 3 shows voxel data obtained by voxelizing a lidar point cloud frame according to one embodiment of the present invention.

[0073] The point cloud frame 310, obtained by measurement with LiDAR 110, can be voxed to generate voxel data 320. At this time, the voxel data may have been generated after a preprocessing step has been performed on the point cloud frame to remove points against a fixed background, such as a road surface or a building wall.

[0074] In this case, for each voxel in the voxel data 320, voxels containing one or more points may be referred to as non-empty voxels (e.g., 321), and voxels without points may be referred to as empty voxels (e.g., 322).

[0075] The vector for each voxel is the dynamic voxel feature embedding vector (h) if a point exists in the voxel (i.e., if it is a non-empty voxel). j It can be generated as j (where j is the voxel index). Also, if no points are measured by lidar 110 (i.e., it is an empty voxel), the corresponding dynamic voxel feature embedding vector may be a 0 vector.

[0076] At this time, the dynamic voxel feature embedding vector (h j To calculate the result, operations can be performed on the point (i) of each voxel. These operations can be performed by sequentially passing through layers (Linear, ReLU, Concat) of a dynamic voxel feature encoder.

[0077] Once the operation on the points is performed, the feature vectors of all points within the same voxel are pooled to form a single dynamic voxel feature embedding vector (h j It can generate ).

[0078] Figure 4 shows voxel data obtained by voxing the radar frame of a radar according to one embodiment of the present invention.

[0079] A two-dimensional radar frame can be generated using a virtual point cloud 410.

[0080] Since the virtual point cloud 410 lacks height (z-axis) information, it can be gridded relative to the 2D plane 420.

[0081] In this case, since the lidar has multiple voxels on the same plane that have different heights, radar information without height information can be generated in voxel data 430 by copying the same information to the corresponding voxel. That is, the first row and the second row of voxel data 430 can have the same values ​​for the corresponding columns.

[0082] The information measured by radar is used to create a radar voxel feature embedding vector (r) for each non-empty voxel (j). j It can be converted to ). In this case, since the radar does not have height information, the same feature vector can be defined for all non-empty voxels that are on the same plane and differ only in height.

[0083] The radar voxel feature embedding vector is the same as the lidar dynamic voxel feature embedding vector (h jThis can be calculated using a method similar to that of the lidar dynamic voxel feature embedding vector. That is, unlike the lidar dynamic voxel feature embedding vector, the radar voxel feature embedding vector generates a feature vector for voxel points on a 2D plane that have no height information, and the corresponding feature vector can be copied and used in the same planar coordinates, differing only in height. This calculation can be performed by sequentially passing through the layers (Linear, ReLU, Concat) of the dynamic voxel feature encoder.

[0084] Once the operation on the points is performed, the feature vectors of all points within the same voxel are pooled to form a single dynamic voxel feature embedding vector (r j It can generate ).

[0085] Furthermore, in the case of radar, if there are no detection objects (or points) at all within any non-empty voxel j, the corresponding feature vector can be set to a 0 vector (i.e., r j =0).

[0086] Although Figure 4 shows that the size of the radar voxel data 430 is different from the size of the lidar voxel data 320 in Figure 3, the radar voxel data 430 can be formed to be the same size as the lidar voxel data 320, based on the region that overlaps with the coverage area of ​​the lidar.

[0087] Figure 5 is a diagram illustrating the process of generating feature vectors and height value probability distributions from a camera image frame according to one embodiment of the present invention.

[0088] The following explanation will refer to both Figure 1 and Figure 5.

[0089] The first neural network 140 receives input from the pixels of the 2D image captured by the q-th camera (e.g., camera 131) at time t and generates a feature vector (s) for each pixel.(k) It can generate ).

[0090] JPEG0007875566000001.jpg67151

[0091] JPEG0007875566000002.jpg39151

[0092] In the world coordinate system, the feature vector of each coordinate can be defined by multiplying the feature vector of the pixel by the probability of the distance value that the pixel can have, as shown in Equation 1.

[0093]

number

[0094] Figure 6 is a diagram illustrating the process of calculating the camera feature vector of each voxel according to one embodiment of the present invention.

[0095] In the world coordinate system, trapezoidal 3D grid data can be formed depending on the camera's field of view. In this case, since the present invention utilizes voxel-type data, voxel data 520 can be generated by overlaying voxels onto the trapezoidal 3D grid data (not shown).

[0096] For example, the feature vector v41 of the pixel in the first column of the fourth row of image 510 can be multiplied by the distance values ​​p1 to p4 that the pixel may have to generate the feature vector of each voxel corresponding to the height of the pixel.

[0097] For example, the feature vector v14 of the fourth enthusiasm pixel in the first row of image 510 can be multiplied by the distance values ​​p11 to p14 that the pixel may have to generate the feature vector of each voxel corresponding to the height of the pixel.

[0098] Referring to both FIGS. 5 and 6, the image captured by the camera q that constitutes the multi-view image frame is converted into a feature vector through the CNN, and each pixel that constitutes the image is mapped in the three-dimensional space by reflecting the distance value in the two-dimensional image feature vector. At this time, since a plurality of distance values can be reflected in each feature vector, each feature vector is converted into a vector with the number of distance values, and the vector thus converted can be positioned in the three-dimensional space. Such operations are repeated for all cameras. When such operations are completed, there will be vectors in the three-dimensional space in which the distance values are reflected, and an average operation is performed on these vectors that exist in the space of the non-empty voxel j based on the lidar to generate a camera voxel feature embedding vector (c j、t )

[0099] At this time, there may be no feature vector that can exist inside an arbitrary voxel (j) depending on the shooting directions of all the cameras. In such a case, the corresponding camera feature vector (c j、t ) can be set to a zero vector (c j、t = 0).

[0100] As described above, there may be a plurality of feature vectors existing in the space of the voxel j. For example, when two pixels correspond to one voxel, an average vector of the two pixel feature vectors can be generated and defined as the feature vector (c (j、t) ) at the corresponding voxel. The mathematical formula 2 of the feature vector (c (j、t) ) may be as follows.

[0101]

Number

[0102] [[ID=?]] JPEG0007875566000005.jpg37161

[0103] JPEG0007875566000006.jpg28161

[0104] Referring to Equation 2, pixel coordinates can be converted to world coordinates by reflecting the distance value. Therefore, the denominator can be the total number of pixels located within a given voxel j (where each pixel increases by the distance value when the distance value is reflected), and the numerator can be the sum of the feature vectors for the corresponding pixels (since the feature vector is obtained by multiplying the distance value by the feature vector of the corresponding pixel).

[0105] The denominator is the number of vectors, and the numerator is the sum of those vectors, so it can be the mean vector. The operation in equation 2 can be considered equivalent to mean pooling in neural networks.

[0106] On the other hand, when using multiple cameras, the camera's field of view may overlap. In this case, for each voxel, an average vector can be generated for all feature vectors generated by each camera present in that voxel space, and this average vector can be defined as the feature vector for that voxel. In this case, a pillar structure that does not divide the voxel in the height direction may be used.

[0107] Figure 7 is a diagram illustrating the process of combining voxel feature vectors for lidar, radar, and camera according to one embodiment of the present invention.

[0108] The following explanation will refer to Figures 1 and 3 through 7.

[0109] The second neural network 150 can include a concat layer and a linear layer.

[0110] JPEG0007875566000007.jpg46161

[0111] JPEG0007875566000008.jpg66161

[0112] JPEG0007875566000009.jpg70161

[0113] JPEG0007875566000010.jpg38161

[0114] Figure 8 is a diagram illustrating voxel data generated in chronological order according to one embodiment of the present invention.

[0115] In the first embodiment of the present invention, a roadside base station can be used to determine whether a fluctuation detected from a point cloud frame or voxel data for a given point in time is due to abnormal activity.

[0116] Alternatively, in a second embodiment of the present invention, a roadside base station can be used to determine whether fluctuations detected from point cloud frames or voxel data for multiple time points, rather than just one, are due to abnormal activity.

[0117] Since roadside base stations are not moved once they are permanently installed, all voxel data acquired over time can be voxelized so that the position and size of each voxel are identical within the same region.

[0118] At this time, Figures 3, 4, and 6 show the first voxel data 320 for the lidar, the second voxel data 430 for the radar, and the voxel data 520 for the camera, all at different sizes. However, when generating the combined voxel data, voxelization is performed only for the region where the radar measurement area and the camera's imaging area coincide with the region measured by the lidar, resulting in the final feature vector (u (j、t) This allows for the generation of voxel data. Specifically, after generating feature embedding vectors for the voxel region corresponding to the radar's virtual point cloud frame and the voxel region for the camera's image frame, which are the same size as the voxel region of the point cloud frame, combined voxel data 610 in the same space can be generated through the second neural network 150.

[0119] For example, the numerous time points (t) for learning can be a predetermined number (T). These predetermined number of time points can be a predetermined number of consecutive time points. For example, if T=10, then t can be from 0 to 9.

[0120] Figure 9 is a flowchart illustrating an RSU-based lidar signal detection and learning method according to one embodiment of the present invention.

[0121] The following steps (S710) to (S750) may be preprocessing steps for generating input to the third neural network 160 included in RSU100 in Figure 1.

[0122] In other embodiments of the present invention, the third neural network 160 may be included in a separate detection device other than the RSU 100.

[0123] The subsequent steps can be performed using the RSU100 or a separate detection device.

[0124] The following explanation will refer to Figures 1, 3, 4, and 6-9.

[0125] In step (S710), first voxel data 320 can be obtained, which is a voxelized version of the point cloud frame measured by the lidar 110 installed at the roadside base station 100. For example, the first voxel data may be generated by the roadside base station 100, or it may be generated by the detection device using the point cloud frame received from the roadside base station.

[0126] In step (S720), second voxel data 430 can be obtained, which is obtained by voxelizing radar frames measured from radar 120 installed at roadside base station 100. For example, the second voxel data may be generated by roadside base station 100, or it may be generated by the detection device using a virtual point cloud frame after generating a virtual point cloud frame using the radar frames received from the roadside base station.

[0127] In step (S730), the first feature vector (c) extracted from the image frames captured by cameras 130 and 131 installed at the roadside base station 100 is calculated. (j、t) ) can be obtained. For example, the first feature vector may be extracted by the roadside base station 100 from the image frame, or it may be extracted by the detection device using the image frame received from the roadside base station.

[0128] At this time, step (S730) extracts a 2D image feature vector (s) from the image frames taken from each of the multiple cameras 131-134 installed at the roadside base station 100, including camera 131. (k) The steps include generating a first feature vector (c = k = pixel index), generating a probability value of the distance value for each pixel in the image frame, and multiplying the probability value of the distance value for each pixel by the two-dimensional image feature vector of each pixel to generate a first feature vector (c = k = pixel index) corresponding to each voxel of the first voxel data 320. (j、t) This may include a step to generate )

[0129] At stage (S740), the first voxel data 320, the second voxel data 430, and the first feature vector (c (j、t) ) can be combined to generate combined voxel data 610 by joining the voxels at their corresponding positions. At this time, a vector for each voxel in the combined voxel data can be calculated.

[0130] At this point, step (S740) is the lidar voxel feature embedding vector (z) for each voxel (j) contained in the first voxel data 320. (j、t) In the stage of generating the radar voxel feature embedding vector (r) for each voxel included in the region of the second voxel data 430 that overlaps with the first voxel data 320, (j、t) The steps involve generating the ) and, for voxels at the same location, the lidar voxel feature embedding vector (z (j、t) ), radar voxel feature embedding vector (r (j、t) ), and the first feature vector (c (j、t) The step may include connecting the voxel data to generate the combined voxel data.

[0131] In step (S750), a predetermined number of voxels 611, 612, and 613 at random positions in the combined voxel data 610 (for example, 3 voxels) can be masked. Preferably, masking can be performed on voxels included in the non-empty voxel set of the combined voxel data. When using multiple time points, masking can be performed on the same positions 611, 612, and 613 in the combined voxel data at each time point from t=0 to T-1.

[0132] In step (S760), vectors for the predetermined number of masked voxels (i.e., masked non-empty voxels) 611, 612, 613 and the remaining voxels other than the masked voxels 611, 612, 613 are input to the third neural network 160 to obtain predicted point information for the masked voxels 611, 612, 613. At this time, the remaining voxels may be voxels from the set of non-empty voxels.

[0133] At this time, the third neural network 160 may be trained to satisfy a predetermined condition by using a loss function between the predicted point information of the masked voxel and the voxel information at the same location. That is, the step of training the neural network 160 to predict the voxels that are not visible due to masking, using only the remaining voxels, may be a step of training to minimize the correlation between the actual voxel information at the corresponding location of all voxels. At this time, the predetermined condition may be a condition that minimizes the correlation.

[0134] Although steps (S710) to (S760) were explained based on a single point in time, as detailed in Figure 8, steps (S710) to (S760) can be performed based on voxel data acquired at multiple point in time.

[0135] The following explanation is based on voxel data acquired at multiple points in time.

[0136] Figure 10 is a diagram illustrating the structure of a third neural network according to one embodiment of the present invention.

[0137] The neural network 160 may include an encoder 161, a decoder 162, and three predictive heads 163, 164, and 165.

[0138] The following explanation will refer to Figures 8 through 10.

[0139] At step (S760), the vector (u) is applied to the remaining voxels other than the time-specific masked (non-empty) voxels 611, 612, and 613. (j、t) The following can be input to encoder 161. In this case, assuming that voxels (j) have values ​​from 0 to 11 and time point (t) has values ​​from 0 to 9, the number of vectors for the remaining voxels can be 12 × 10 = 120, minus the number of masking voxels × T = 30, resulting in 90 vectors being input.

[0140] The input to encoder 161 may be a vector for the remaining voxels, excluding the masked non-empty voxels and voxels for which no points exist (or voxels with a vector of 0) in the combined voxel data at any given time point, i.e., the empty voxels.

[0141] For example, if, using Figure 8 as a reference, there are no points in voxels 614 and 615, which are at the same position at all points in the combined voxel data 610, then those voxels may also be excluded from the input of encoder 161.

[0142] The encoder 161 can encode an input vector that is arranged in voxel position and time order (i.e., spatial order and occurrence order) and output the encoded value.

[0143] At this time, positional embedding vectors may be additionally input to the encoder 161 and decoder 162. For example, we can assume that there are 100 voxels per point cloud frame, and that 10 frames occur sequentially. In this case, the positional embedding vectors that must be input to the transformer neural network may have index values ​​corresponding to 0-99, 100-199, ..., 900-999 sequentially for each voxel per frame.

[0144] At this time, the time-series voxel positions of the voxels excluded from the input of encoder 161 may remain empty, while the encoded values ​​may be placed in time-series / position-series order.

[0145] The empty spaces may be filled with a first mask token of a masked voxel(rar) (i.e., a masked non-empty voxel(rar)) at the corresponding voxel location, and may be filled with a second mask token of a masked empty voxel(rar) that masks one or more empty voxels (or all empty voxels) selected at the corresponding voxel location.

[0146] In this case, the first mask token and the second mask token may be identical. That is, the first mask token and the second mask token input to the decoder may not be distinguishable. However, the position vectors applied to the first mask token and the second mask token may be different from each other.

[0147] Based on Figure 8, the combined voxel data 610 may generate first mask tokens 611, 612, and 613 in voxels at the same position at each time point (t=0 to (T-1)), and in Figure 10, a total of 30 first mask tokens (e.g., 810, 820) (3 per time point, for a total of 10 time points) may be placed at the positions of the voxels at each time point.

[0148] An empty voxel (614, or 615) may have a second mask token generated at the same position at each time point. For example, if there are two empty voxels at each time point, a total of 20 second mask tokens 840, 850 may be placed at the positions of the voxels at each time point. In this case, the number of input vectors to encoder 161 may be 70 (=120-30-20). Alternatively, in other embodiments, a second mask token may be placed at the position of a selected empty voxel (e.g., 614) from among multiple empty voxels.

[0149] The decoder 162 can receive the encoded value 830, the first mask tokens (e) 810 for the masked non-empty voxels (e), and the second mask tokens (e) 840, 850 for masked empty voxels that mask empty voxels where no points exist in the voxel data for the entire time point, and generate reconstructed data for the first mask tokens 810, 820 and the second mask tokens 840, 850.

[0150] The prediction heads 163, 164, and 165 can receive the reconstructed data as input and generate predicted point information.

[0151] For example, there may be three prediction heads. The three prediction heads may be a first prediction head 163 that predicts the coordinates of a point in a voxel, a second prediction head 164 that predicts the number of points in the voxel, and a third prediction head 165 that predicts whether the voxel at the position to be predicted is a masked non-empty voxel or a masked empty voxel that does not contain a point.

[0152] In this case, the loss value can be calculated for each prediction head, equal to the number of mask tokens input to the decoder. For example, if there are two masked non-empty voxels and four time points for one masked empty voxel, 12 mask tokens are input to the decoder 162, and each prediction head can output 12 values. A loss value can be calculated for each of the 12 output values.

[0153] Subsequently, the Chamfer distance loss (LOSS1) can be calculated using the point coordinates within the voxels predicted at each time point output by the first prediction head 163 and the point coordinates within the voxels at the corresponding positions in the first voxel data 320 of the lidar 110 (ground truth).

[0154] For example, for a non-empty voxel (j) measured at observation time t among the mask tokens, it is possible to output a fixed number of Q 3D coordinates (i.e., a total of 3Q). In this case, each output is generated using the tanh function, so it can have a value between -1 and 1. The ground truth coordinates have an unlimited number of point coordinates, and all relevant coordinates can be normalized so that the minimum and maximum dimensions of the corresponding voxel j at time t correspond to -1 and 1, respectively.

[0155] Equation 3 for the chamfer distance loss (CD) can be as follows:

[0156]

number

[0157] In this case, j could mean a voxel, and i could mean a point.

[0158] JPEG0007875566000012.jpg11161

[0159] JPEG0007875566000013.jpg17161

[0160] JPEG0007875566000014.jpg18161

[0161] JPEG0007875566000015.jpg18161

[0162] The number of points loss (LOSS2) can be calculated by using the number of points in a voxel predicted at each time point output by the second prediction head 164 and the number of points in the corresponding voxel (non-empty voxel) (ground truth) in the first voxel data 320 of the lidar 110 at each time point.

[0163] Formula 4 for calculating the loss value (SL) for the number of points mentioned above may be as follows:

[0164]

number

[0165] JPEG0007875566000017.jpg17161

[0166] The cross-entropy loss (LOSS3) can be calculated using the value for whether the time-specific predicted empty voxel output by the third prediction head 165 is an empty voxel (i.e., whether it is a non-empty voxel corresponding to the first mask token or a selected empty voxel corresponding to the second mask token) and the value for whether the voxel at the corresponding position in the time-specific first voxel data 320 of the lidar 110 is an empty voxel (ground truth).

[0167] For example, if the voxel j at time t is a non-empty voxel for all mask tokens input to decoder 162, then label a (j、t) Assigned to =1, if it is an empty voxel, label a (j、t) By assigning it to =0, the cross-entropy loss value can be calculated.

[0168] The formula 5 for calculating the cross-entropy loss value may be as follows:

[0169]

number

[0170] At this time, f(V (j、t) ) is V (j、t) This is a function for predicting the probability that a cell is a non-empty voxel.

[0171] Thereafter, the final loss value (Total Loss) can be calculated by multiplying the chamfer distance loss value, the loss value for the number of points, and the cross-entropy loss value by a weight and adding them together. The third neural network 160 can be trained to minimize the final loss value. By using the final loss value, it can be applied to the prediction heads 163-165 such that the values ​​for the three types of loss are similar.

[0172] Formula 6 for calculating the sum of the aforementioned weighted values ​​may be as follows:

[0173]

number

[0174] JPEG0007875566000020.jpg9161

[0175] JPEG0007875566000021.jpg25161

[0176] JPEG0007875566000022.jpg17161

[0177] JPEG0007875566000023.jpg9161

[0178] T could represent the total number of point cloud time indices.

[0179] Figure 11 is a diagram illustrating voxel data for point cloud frames that occur in chronological order according to one embodiment of the present invention.

[0180] Figure 11 illustrates, for example, the reasoning process that uses the trained third neural network 160 to determine whether the current fluctuation is due to abnormal behavior.

[0181] Figure 12 is a flowchart illustrating an RSU-based lidar signal detection and inference method according to one embodiment of the present invention.

[0182] The following explanation will refer to Figures 1, 3, 4, 7, 10, and 11.

[0183] In step (S1100), first voxel data 320 can be obtained, which is a voxelized version of the point cloud frame at the current time (t) measured by the lidar 110 installed at the roadside base station 100. For example, the first voxel data 320 may be generated by the roadside base station 100, or it may be generated by a separate detection device (not shown) using the point cloud frame received from the roadside base station 100.

[0184] In step (S1200), if a variation is detected based on the first voxel data 320, the first feature vector (c) is extracted from the first voxel data 320, the second voxel data 430 which is obtained by voxing radar frames measured from the radar 120 installed at the roadside base station 100, and the first feature vector (c) is extracted from image frames captured by cameras 130 and 131 installed at the roadside base station 100. (j、t) ) can be combined to generate joined voxel data 910 by joining voxels at their corresponding positions.

[0185] For example, the second voxel data 430 may be generated by the roadside base station 100, or it may be generated by a separate detection device (not shown) using radar frames received from the roadside base station 100.

[0186] At this time, step (S1200) generates a 2D image feature vector (s) from the image frames captured by each of the multiple cameras 131-134 installed at the roadside base station 100, including cameras 130 and 131. (k)The steps include generating a first feature vector (c = k = pixel index), generating a probability value of the distance value for each pixel in the image frame, and multiplying the probability value of the distance value for each pixel by the two-dimensional image feature vector of each pixel to obtain the first feature vector (c = k = pixel index) corresponding to each voxel in the first voxel data 320. (j、t) This may include a step to generate )

[0187] At this time, step (S1200) is the lidar voxel feature embedding vector (z) for each voxel (j) contained in the first voxel data 320. (j、t) In the stage of generating the radar voxel feature embedding vector (r) for each voxel included in the region of the second voxel data 430 that overlaps with the first voxel data 320, (j、t) The steps involve generating the ) and, for voxels at the same location, the lidar voxel feature embedding vector (z (j、t) ), radar voxel feature embedding vector (r (j、t) ), and the first feature vector (c (j、t) The process may further include the step of connecting the voxel data to generate the combined voxel data.

[0188] In step (S1300), based on the combined voxel data 910, the voxels 911 and 912 at the first position where the variation is detected are masked, and a vector (u) is generated for the masked voxels 911 and 912 and the remaining voxels other than the masked voxels 911 and 912. (j、t) By inputting this into the neural network 160, we can obtain predicted point information for the masked voxels 911 and 912.

[0189] In this case, the voxel at the first position where the variation is detected may be the single voxel that is most suspicious among all the voxels in which the variation is detected (for example, voxel 911), or it may be multiple voxels among all the voxels in which the variation is detected (for example, voxel 911, voxel 912).

[0190] The aforementioned fluctuations can be detected during the process of sequentially tracking the movement of an object using a multi-object tracking algorithm. In this case, the fluctuation may mean that an observed object has disappeared or an unobserved object has appeared. At this time, the location of the fluctuation can be determined by the multi-object tracking algorithm at the point in time when the fluctuation is detected. Therefore, the voxel(s) at the first position, which is the determined location of the fluctuation, can be masked.

[0191] At this time, step (S1300) is the step of masking the voxels 911 and 912 at the first position in each combined voxel data 910 generated for the time when the variation was detected and for time prior to the time when the variation was detected (i.e., t=0 to (T-1)), and the neural network 160 generates vectors (u) for the masked (non-empty) voxels 911 and 912 and the remaining voxels at each time point. (j、t) This could include a step where you input )

[0192] In this case, the remaining voxels may be the voxels excluding the voxels at the first position and the voxels in the combined voxel data that do not contain any points.

[0193] For example, if there is one masking voxel 911 and no voxels for each time point where there are no points in the Lidar, radar, or camera (i.e., voxels with a vector of 0, empty voxels), the number of inputs to encoder 161 could be (number of voxels in the voxel data - 1) × number of time points. For example, if there are 20 voxels and a total of 10 time points, the number of input vectors could be 190.

[0194] At this time, the first mask tokens (ra) for the masked (non-empty) voxels 911 and 912 at each point in time are input to the decoder of the neural network 160, and the vector (u) for the remaining voxels is input (j、t)) can be input to the encoder of the neural network 160.

[0195] If, for each time point, there is a voxel (i.e., an empty voxel) in which there are no points in the lidar, radar, or camera, then the second mask token (and so on) for the whole can be input to the decoder of the neural network 160 into the empty voxel(and so on).

[0196] At this time, the operation of the encoder, decoder, and prediction head of the neural network 160 may be as described above through Figure 10.

[0197] At this time, the total number of time points that affect the number of inputs to the encoder 161 of the neural network 160 may be the same as the total number of time points used in the learning process of the neural network 160. For example, if the neural network 160 learned using lidar, radar, and camera voxel data acquired over 10 time points (T=10), then during inference, it can use the lidar, radar, and camera voxel data acquired over the current time point where a change is detected and over 9 consecutive time points prior to the current time point.

[0198] In step (S1400), it can be determined whether the fluctuation detected at the first position is due to abnormal activity, based on the predicted point information of the masked voxels 911 and 912 and the first voxel data 320 of the lidar.

[0199] Specifically, the loss values ​​(LOSS1, LOSS2, LOSS3) can be calculated by comparing the predicted values ​​from prediction heads 163-165 with the correct values.

[0200] First, the chamfer distance loss value LOSS1 is calculated using the point coordinates within the voxel predicted for each time point and the point coordinates within the voxel at the corresponding position in the first voxel data 320 of the lidar 110. Then, it can be determined whether the maximum value among the chamfer distance loss values ​​calculated for each time point is greater than a preset first critical value.

[0201] Furthermore, by using the number of points in a voxel predicted for each time point and the number of points in the voxel at the corresponding position in the first voxel data 320 of the LiDAR 110, a loss value LOSS2 relative to the number of points can be calculated, and it can be determined whether the maximum value among the loss values ​​relative to the number of points calculated for each time point is greater than a pre-set second critical value.

[0202] Furthermore, by using the value predicting whether a voxel is empty at each time point and the value indicating whether the voxel at the corresponding position in the first voxel data 320 of the lidar 110 is empty, the cross-entropy loss value LOSS3 can be calculated, and it can be determined whether the maximum value among the cross-entropy loss values ​​calculated at each time point is greater than a preset third critical value.

[0203] If the determination result indicates that at least one of the loss values ​​is greater than the corresponding first critical value, second critical value, and third critical value, then the fluctuation detected at the first position can be determined to be due to abnormal activity.

[0204] Equation 7 for determining whether the fluctuation detected at the first position is due to abnormal activity may be as follows:

[0205]

number

[0206] JPEG0007875566000025.jpg17161

[0207] JPEG0007875566000026.jpg17161

[0208] JPEG0007875566000027.jpg10161

[0209] For example, in the case of loss with respect to the number of points and cross-entropy loss, if the maximum values ​​of each are smaller than the second critical value and the third critical value, respectively, while in the case of chamfer distance loss, if the maximum value is greater than the first critical value, then it can be determined that the cause of the fluctuation detected at the present time is abnormal activity.

[0210] Figure 13 is a conceptual diagram illustrating an example of a generalized RSU-based signal anomaly detection device or computing system capable of performing at least part of the processes shown in Figures 1 to 12.

[0211] At least a portion of the RSU base signal anomaly detection method and RSU base signal anomaly detection learning method according to one embodiment of the present invention can be performed by the computing system 2000 shown in Figure 13.

[0212] Referring to Figure 13, a computing system 2000 according to one embodiment of the present invention may be configured to include a processor 2100, memory 2200, communication interface 2300, storage device 2400, input interface 2500, output interface 2600, and bus 2700.

[0213] A computing system 2000 according to one embodiment of the present invention may include at least one processor 2100 and a memory 2200 that stores instructions that the at least one processor 2100 perform at least one step. At least some steps of the method according to one embodiment of the present invention may be performed by the at least one processor 2100 loading and executing instructions from the memory 2200.

[0214] The processor 2100 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which the method according to an embodiment of the present invention is performed.

[0215] Each of the memory 2200 and the storage device 2400 may consist of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 2200 may consist of at least one of a read-only memory (ROM) and a random access memory (RAM).

[0216] Furthermore, the computing system 2000 may include a communication interface 2300 that performs communication over a wireless network.

[0217] Furthermore, the computing system 2000 may further include a storage device 2400, an input interface 2500, an output interface 2600, and the like.

[0218] Furthermore, each component included in the computing system 2000 can be connected by bus 2700 to perform communication.

[0219] Examples of the computing system 2000 of the present invention include a communication-enabled desktop computer, laptop computer, notebook computer, smartphone, tablet PC, mobile phone, smart watch, smart glasses, e-book reader, PMP (portable multimedia player), portable game console, navigation device, digital camera, DMB (digital multimedia broadcasting) player, digital audio recorder, digital audio player, digital video recorder, digital video player, PDA (Personal Digital Assistant), and the like.

[0220] The operation of the method according to the embodiment of the present invention can be embodied in a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices on which information that can be read by a computer system is stored. Furthermore, the computer-readable recording medium can be distributed across a network of computer systems, and the computer-readable program or code can be stored and executed in a distributed manner.

[0221] Furthermore, computer-readable recording media can include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Program instructions can include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0222] Some aspects of the present invention have been described in the context of apparatus, but this can also represent a description by a corresponding method, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method can also be represented by a corresponding block or item or a feature of a corresponding apparatus. Some or all of the method steps may be carried out by (or utilizing) a hardware device such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, at least one or more of the most important method steps may be carried out by such a device.

[0223] In embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, a field-programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. Generally, it is preferable that the methods be performed by some hardware device.

[0224] Although preferred embodiments of the present invention have been described above with reference to the present invention, those skilled in the art will understand that the present invention can be modified and altered in various ways without departing from the spirit and scope of the invention as described in the following claims.

Claims

1. Memory to store at least one instruction; A processor that executes at least one of the aforementioned instructions; and Neural networks; including, The aforementioned processor, The point cloud frame measured from a roadside base station is voxelized to obtain the first voxel data. If a variation is detected based on the first voxel data, Combined voxel data is generated by combining the first voxel data, the second voxel data obtained by voxing radar frames measured from the radar installed at the roadside base station, and the first feature vector extracted from image frames captured by the camera installed at the roadside base station, at the voxels at their corresponding positions. Based on the combined voxel data, the voxel at the first position where the variation is detected is masked, and the vectors for the masked voxel and the remaining voxels other than the masked voxel are input to the neural network to obtain predicted point information for the masked voxel. RSU base signal anomaly detection device that determines whether the fluctuation detected at the first position is due to abnormal activity, based on the predicted point information of the masked voxel and the first voxel data of the lidar.

2. The aforementioned roadside base station is equipped with multiple cameras, including the aforementioned camera. The RSU base signal anomaly detection device according to claim 1, wherein the first feature vector is extracted from image frames captured by each of the plurality of cameras.

3. When the processor generates the combined voxel data, A lidar voxel feature embedding vector is generated for each voxel included in the aforementioned first voxel data. Radar voxel feature embedding vectors are generated for each voxel included in the region of the second voxel data that overlaps with the first voxel data. The RSU base signal anomaly detection device according to claim 1, wherein the lidar voxel feature embedding vector, the radar voxel feature embedding vector, and the first feature vector are linked together to generate the combined voxel data for voxels at the same location.

4. The aforementioned processor, A two-dimensional image feature vector is generated from the aforementioned image frame. A probability value of the distance to each pixel of the aforementioned image frame is generated. The RSU substrate signal anomaly detection device according to claim 1, wherein the probability value of the distance value for each pixel is multiplied by the two-dimensional image feature vector of each pixel to generate the first feature vector corresponding to each voxel of the first voxel data and the second voxel data.

5. When the processor masks the voxel at the first position, it masks the voxel at the first position in each combined voxel data generated for a time point prior to the time when the variation was detected. The RSU substrate signal anomaly detection device according to claim 1, wherein the neural network is further input with respect to the masked voxels and the remaining voxels for each of the previous time points.

6. The neural network includes an encoder, a decoder, and a prediction head. The aforementioned processor, The vectors for the empty voxels (voxels that do not have points at the time the aforementioned variation is detected and at all prior time points) and the remaining voxels other than the masked voxels are input to the encoder to obtain encoded values. The encoded value, the first mask token for the masked voxel, and the second mask token for the masked empty voxel that masks the empty voxel are input to the decoder to obtain reconstructed data for the first mask token and the second mask token, respectively. The RSU base signal anomaly detection device according to claim 5, wherein the reconstructed data is input to the prediction head to obtain the predicted point information.

7. The predicted point information includes the point coordinates within the voxel at each time point, the number of points within the voxel at each time point, and a value indicating whether or not it is a masked empty voxel. The aforementioned processor The loss value is calculated using the point coordinates within the voxel at each time point, the number of points within the voxel at each time point, and the value indicating whether it is a masked empty voxel, and the corresponding point coordinates within the voxel at each time point, the number of points within the voxel at each time point, and the value indicating whether it is a masked empty voxel for the first voxel data of the LiDAR. The aforementioned loss value is compared with the corresponding critical value, The RSU base signal anomaly detection device according to claim 6, wherein if it is determined that at least one of the loss values ​​is greater than the corresponding critical value, the fluctuation detected at the first position is determined to be due to an abnormal operation.

8. The aforementioned neural network is First training voxel data is obtained by voxelizing point cloud frames measured at different points in time from the LiDAR installed at the aforementioned roadside base station. Second training voxel data is obtained, which consists of time-series radar frames measured from the radar installed at the aforementioned roadside base station, and then voxelized. A first training feature vector is obtained from an image frame captured by a camera installed at the aforementioned roadside base station. Combined voxel data is generated by combining the first training voxel data, the second training voxel data, and the first training feature vector at time points, with the voxels at corresponding positions at each time point. Although a predetermined number of voxels at random positions are masked using the combined voxel data for each time point, the same voxels at the same positions are masked at each time point. The neural network is input with a predetermined number of masked voxels and vectors for the remaining voxels other than the masked voxels at each time point to obtain predicted point information for the masked voxels. The RSU base signal anomaly detection device according to claim 1, wherein the value calculated using a loss function between the predicted point information of the masked voxel and the voxel information at the same position in the first training voxel data is learned to satisfy a predetermined condition.

9. This stage involves obtaining the first voxel data, which is the current point cloud frame measured by a lidar installed at a roadside base station; If a variation is detected based on the first voxel data, A step of generating combined voxel data by combining the first voxel data, the second voxel data obtained by voxing radar frames measured from radar installed at the roadside base station, and the first feature vector extracted from image frames captured by a camera installed at the roadside base station, at the voxels at their corresponding positions; A step of masking the voxel at the first position where the variation is detected based on the combined voxel data, inputting the vectors for the masked voxel and the remaining voxels other than the masked voxel into a neural network to obtain predicted point information for the masked voxel; and A method for detecting RSU substrate signal anomalies, comprising the step of determining whether a fluctuation detected at a first position is due to an anomaly based on predicted point information of the masked voxel and first voxel data of the lidar.

10. The step of generating the aforementioned combined voxel data is: A step of generating a two-dimensional image feature vector from image frames captured by each of the multiple cameras installed at the roadside base station, including the aforementioned camera; A step of generating a probability value of the distance value for each pixel of the image frame; and The RSU base signal anomaly detection method according to claim 9, comprising the step of multiplying the probability value of the distance value for each pixel by the two-dimensional image feature vector of each pixel to generate the first feature vector corresponding to each voxel of the first voxel data.

11. The step of generating the aforementioned combined voxel data is: A step of generating a lidar voxel feature embedding vector for each voxel included in the first voxel data; The step of generating radar voxel feature embedding vectors for each voxel included in the region of the second voxel data that overlaps with the first voxel data; and The RSU base signal anomaly detection method according to claim 9, comprising the step of generating the combined voxel data by concatenating the lidar voxel feature embedding vector, the radar voxel feature embedding vector, and the first feature vector for voxels at the same location.

12. The step of obtaining the predicted point information is as follows: When masking the voxel at the first position, the steps include: masking the voxel at the first position in each combined voxel data generated for the time when the variation was detected and for time prior to the time when the variation was detected; and The RSU base signal anomaly detection method according to claim 9, comprising the step of inputting time-series vectors for the masked voxels and the remaining voxels into the neural network.

13. The neural network includes an encoder, a decoder, and a prediction head. The step of obtaining the predicted point information is as follows: A step of inputting vectors to the encoder for the remaining voxels other than the empty voxels (voxels that do not have points at the time the aforementioned variation is detected and at all prior time points) and the masked voxels, and obtaining encoded values; A step of inputting the encoded value, the first mask token for the masked voxel, and the second mask token for the masked empty voxel that masks the empty voxel into the decoder to obtain reconstructed data for the first mask token and the second mask token, respectively; and The RSU base signal anomaly detection method according to claim 12, further comprising the step of inputting the reconstructed data to the prediction head to obtain the predicted point information.

14. The predicted point information includes the point coordinates within the voxel at each time point, the number of points within the voxel at each time point, and a value indicating whether or not it is a masked empty voxel. The step of determining whether the fluctuation detected at the first position is due to abnormal activity is: A step of calculating a loss value using the point coordinates within the voxel at each time point, the number of points within the voxel at each time point, and a value indicating whether or not it is a masked empty voxel, and the corresponding point coordinates within the voxel at each time point, the number of points within the voxel at each time point, and a value indicating whether or not it is a masked empty voxel in the first voxel data of the LiDAR; A step of comparing the loss value with the corresponding critical value; and The RSU base signal anomaly detection method according to claim 13, further comprising the step of determining that the fluctuation detected at the first position is due to an abnormal activity if it is determined that at least one of the loss values ​​is greater than the corresponding critical value.

15. The first stage involves obtaining voxelized first-cell data from point cloud frames measured by lidar installed at roadside base stations; The step of acquiring second voxel data, which is obtained by voxelizing radar frames measured from radar installed at the aforementioned roadside base station; A step of obtaining a first feature vector extracted from an image frame captured by a camera installed at the aforementioned roadside base station; A step of generating combined voxel data by joining the first voxel data, the second voxel data, and the first feature vector at the voxels at their corresponding positions; A step of masking a predetermined number of voxels at random positions in the combined voxel data; and The process includes the step of inputting a predetermined number of masked voxels and vectors for the remaining voxels other than the masked voxels into a neural network to obtain predicted point information for the masked voxels; RSU-based signal anomaly detection learning method, wherein the neural network is trained to satisfy predetermined conditions by using a loss function between the predicted point information of the masked voxel and the voxel information at the same location in the first voxel data.

16. The step of obtaining the first feature vector is: A step of generating a two-dimensional image feature vector from image frames captured by each of the multiple cameras installed at the roadside base station, including the aforementioned camera; A step of generating a probability value of the distance value for each pixel of the image frame; and The RSU-based signal anomaly detection learning method according to claim 15, comprising the step of generating the first feature vector corresponding to each voxel of the first voxel data by multiplying the probability value of the distance value for each pixel by the two-dimensional image feature vector of each pixel.

17. The step of generating the aforementioned combined voxel data is: A step of generating a lidar voxel feature embedding vector for each voxel included in the first voxel data; The step of generating radar voxel feature embedding vectors for each voxel included in the region of the second voxel data that overlaps with the first voxel data; and The RSU base signal anomaly detection learning method according to claim 15, comprising the step of generating the combined voxel data by concatenating the lidar voxel feature embedding vector, the radar voxel feature embedding vector, and the first feature vector for voxels at the same location.

18. The steps of acquiring the first voxel data, acquiring the second voxel data, and acquiring the first feature vector are each steps that are performed at multiple points in time. The step of masking the voxels is the step of masking the voxels at the same position at each time point. The RSU-based signal anomaly detection learning method according to claim 15, wherein a predetermined number of masked voxels and vectors for the remaining voxels other than the masked voxels are input to the neural network at each time point.

19. The neural network includes an encoder, a decoder, and a prediction head. The step of obtaining the predicted point information is as follows: A step of inputting vectors to the encoder for empty voxels (voxels that do not have points at all time points prior to the detection of a change in the same position) and the remaining voxels other than the masked voxels, and obtaining encoded values; A step of inputting the encoded value, the first mask token for the masked voxel, and the second mask token for the masked empty voxel that masks the empty voxel into the decoder to obtain reconstructed data for the first mask token and the second mask token, respectively; and The RSU base signal anomaly detection learning method according to claim 18, further comprising the step of inputting the reconstructed data to the prediction head to obtain the predicted point information.

20. After the stage of acquiring the predicted point information, A step of calculating the chamfer distance loss value using the point coordinates within the voxel predicted for each time point and the point coordinates within the voxel at the corresponding position in the first voxel data; A step of calculating a loss value relative to the number of points using the number of points in the voxel predicted for each time point and the number of points in the voxel at the corresponding position in the first voxel data; A step of calculating the cross-entropy loss value using the value for whether or not it is a masked empty voxel predicted at each time point and the value for whether or not the voxel at the corresponding position in the first voxel data is a masked empty voxel; and The process further includes the step of calculating a final loss value obtained by multiplying the chamfer distance loss value, the loss value for the number of points, and the cross-entropy loss value by a weighted value and adding them together; The RSU-based signal anomaly detection learning method according to claim 19, wherein the neural network is trained to minimize the final loss value.

Images