Method, Apparatus and Computer-Readable Medium for Generating Radar-Based AI Training Dataset for Risk Situation Detection Through Vision Sensor-Assisted Learning

Abstract

The present invention relates to a method for generating an image shape for AI analysis based on point data of multiple objects detected by a radar sensor. Point data collected from a radar sensor is preprocessed to remove noise and classify valid points for each object, wherein the point data consists of 5-dimensional time-series data including 3D spatial coordinates (x, y, z), reflection intensity, and Doppler shift. A 3D shape is reconstructed based on the position coordinates, reflection intensity, and velocity information of each object's point, and a micro-Doppler signature is generated from the reconstructed 3D shape. The reconstructed 3D points are mapped to image data acquired from an image sensor to align the contours and positions of the objects, and a plurality of skeleton joint coordinates extracted from the image data are spatially aligned with the radar point data. Based on the alignment data, an image dataset for AI analysis is generated that includes object-specific contours, movement, and distance information, and the dataset is input into an AI model trained by combining radar and image data to train the AI ​​model. By iteratively executing a trained AI model, it is possible to generate image datasets for AI analysis using only radar input, without the need for an image sensor. This eliminates the need for an image sensor during actual operation, enabling cost reduction and privacy protection, while also improving the accuracy and practicality of radar-based risk situation identification.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for generating an AI learning dataset for determining radar-based dangerous situations through image sensor-assisted learning. More specifically, the invention relates to a method and apparatus for generating an image dataset including object-specific contour, movement, and distance information by fusing a radar sensor and an image sensor, and generating an AI learning dataset capable of determining dangerous situations using only radar input through transfer learning. Background Technology

[0002] Recently, there has been a surge in demand for real-time multi-object recognition and risk situation identification technologies in fields such as smart cities, smart homes, autonomous driving, and security surveillance. In these application areas, it is crucial to analyze human behavioral patterns to automatically detect dangerous situations, such as falls, violence, and intrusion, and to respond rapidly.

[0003] Traditionally, image sensors (cameras) have been widely used for object recognition and behavior analysis due to their high resolution and rich visual information. However, image sensors have fundamental limitations, such as sensitivity to lighting conditions, performance degradation at night or in low-light environments, and, above all, serious privacy infringement issues arising from the exposure of personal facial or physical information.

[0004] In contrast, radar sensors utilize radio waves, allowing them to operate regardless of lighting conditions and maintain stable performance even at night or in adverse weather environments. Furthermore, they offer advantages in privacy protection as they do not directly capture images of an individual's face or appearance. Additionally, radar can precisely measure the speed and minute movements of objects using the Doppler effect, making it highly effective for detecting sudden changes in motion, such as falls or violence.

[0005] However, object recognition technology utilizing only radar sensors has the following drawbacks. First, since radar data is provided in the form of point clouds, its spatial resolution is lower compared to image data, making it difficult to accurately identify the detailed external features or orientations of objects. Second, generating training data solely from radar sensors makes it difficult to secure accurate ground truth, which can degrade the training accuracy of the AI ​​model. Third, there is a technical challenge in accurately distinguishing and tracking each object when multiple objects are in close proximity.

[0006] Against this backdrop, developing a system that fuses image sensors and radar sensors to leverage the strengths of each, while ultimately enabling the radar sensor to operate independently, is emerging as an important challenge.

[0007] As a related technology, Korean Registered Patent No. 10-2589987 discloses a technology regarding an online multi-object tracking method and device based on image and radar feature learning. The aforementioned prior art transmits the detection box and amplitude characteristics of an object detected by an object detection unit to a reliability-based data linkage unit, and the reliability-based data linkage unit evaluates the reliability score of a track using the detection box and amplitude characteristics and classifies the track according to the evaluated reliability. Furthermore, it discloses a technology in which a learning unit learns the object's appearance model, motion model, shape model, and amplitude model from visual features while tracking an object on a track, and learns the amplitude model from radar features to evaluate a relevance score for reliability-based data association.

[0008] However, the aforementioned prior art has the following problems. First, since it tracks objects by simultaneously using video and radar sensors, the video sensor is always required during real-time operation, resulting in a complex system configuration, increased costs, and potential privacy infringement issues. Second, it focuses merely on improving tracking performance by combining the features of the two sensors and lacks an autonomous learning mechanism that enables the radar sensor to operate independently. Third, while effective for tracking objects, it has limitations in accurately detecting dangerous situations such as falls or violence because it fails to utilize detailed motion information, such as behavioral pattern analysis or micro-Doppler signatures, for identifying hazardous situations. The problem to be solved

[0009] The present invention aims to solve the aforementioned problems by providing a method for generating an AI training dataset that performs initial learning by fusing a radar sensor and an image sensor, gradually reduces dependence on the image sensor, and ultimately identifies dangerous situations using only the radar sensor. Through this, the invention seeks to reduce system costs and resolve privacy issues by operating solely with the radar sensor without the image sensor during the actual operation phase.

[0010] In addition, the present invention aims to generate an AI training dataset capable of accurately identifying dangerous situations such as falls and violence beyond simple object tracking by generating 5-dimensional time series data including 3-dimensional spatial coordinates, reflection intensity, and Doppler shift from radar point data, and estimating micro-Doppler signatures and joint coordinates based thereon.

[0011] Furthermore, the present invention aims to simultaneously secure learning efficiency and the practicality of the final model by applying a transfer learning technique to enable the generation of an image dataset with high accuracy using radar alone through iterative learning after initial learning utilizing ground truth acquired from an image sensor. means of solving the problem

[0012] To achieve the above objectives, a method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning according to an embodiment of the present invention comprises: a step of preprocessing point data collected from a radar sensor to remove noise and classifying valid points for each object, wherein the point data is composed of 5-dimensional time-series data including 3-dimensional spatial coordinates (x, y, z), reflection intensity, and Doppler shift; a step of restoring a 3-dimensional shape based on position coordinates, reflection intensity, and velocity information of points for each object, wherein a micro-Doppler signature is generated from the restored 3-dimensional shape; a step of mapping the restored 3-dimensional points with image data acquired from the image sensor to align the contours and positions of the objects, wherein a plurality of skeleton joint coordinates extracted from the image data are spatially aligned with the radar point data; a step of generating an image dataset for AI analysis including contours, movement, and distance information for each object based on the alignment data; and a step of inputting the dataset into an AI model trained by combining radar and image data to train the AI ​​model. The method is characterized by including a step of generating an image dataset for AI analysis using only radar input without an image sensor by repeatedly performing the step of training the AI ​​model.

[0013] The above micro-Doppler signature represents the Doppler frequency change pattern according to the movement of an object in the time-frequency domain, and it is possible to generate a normalized Doppler pattern of a point cloud separated by object using DBSCAN (Density-Based Spatial Clustering of Applications with Noise)-based clustering and a Kalman filter.

[0014] The above joint coordinates consist of fewer joint points than the number of joints recognized from the image sensor, are estimated through a convolutional neural network (CNN) that takes the above 5-dimensional time series data as input, and can be learned by using the joint coordinates of the image sensor as ground truth.

[0015] The step of enabling the generation of an image dataset for AI analysis using only the radar input described above can be achieved by correcting the radar-based joint estimation results by referencing joint coordinates obtained from image data during the initial learning phase, gradually reducing the dependency on the image sensor through iterative learning, and finally performing transfer learning to estimate joint coordinates using only radar input without image data and generate an image dataset for AI analysis.

[0016] When the number of repetitions reaches a preset threshold, an AI analysis image generated when radar is input alone and an AI analysis image data set when image data is combined are generated respectively, and an error between the two image data sets is calculated, wherein the error can be calculated as a composite error including a spatial distance error between joint coordinates, a similarity error of micro-Doppler signatures, and an agreement error of object contours.

[0017] The above noise removal may include time-frequency domain filtering and K-means-based cluster separation.

[0018] The above image mapping makes it possible to minimize spatial errors by calibrating the radar coordinate system and the image coordinate system using a matching matrix.

[0019] Meanwhile, an AI learning dataset generation device for radar-based risk situation determination through image sensor-assisted learning according to an embodiment of the present invention comprises: a preprocessing unit that preprocesses point data collected from a radar sensor to remove noise and classify valid points for each object, wherein the point data is composed of 5-dimensional time-series data including 3-dimensional spatial coordinates (x, y, z), reflection intensity, and Doppler shift; a 3-dimensional reconstruction unit that reconstructs a 3-dimensional shape based on position coordinates, reflection intensity, and velocity information of points for each object, and generates a micro-Doppler signature from the reconstructed 3-dimensional shape; a data mapping unit that maps the reconstructed 3-dimensional points with image data acquired from an image sensor to align the contours and positions of objects, and spatially aligns a plurality of skeleton joint coordinates extracted from the image data with the radar point data; and a dataset generation unit that generates an image dataset for AI analysis including contours, movement, and distance information for each object based on the alignment data. It is characterized by including: a model training unit that inputs the above dataset into a trained AI model by combining radar and image data to train the AI ​​model; and an autonomous learning unit that enables the generation of an image dataset for AI analysis using only radar input without an image sensor by repeatedly executing the above-mentioned AI model. Effects of the invention

[0020] The method and apparatus for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning according to the present invention has the following effects.

[0021] First, since the radar can operate independently through transfer learning following initial learning that fuses the radar sensor and the image sensor, the image sensor is unnecessary during actual operation, thereby reducing system costs and fundamentally resolving privacy infringement issues.

[0022] Second, by estimating micro-Doppler signatures and joint coordinates based on 5-dimensional time series data (3-dimensional spatial coordinates, reflection intensity, Doppler shift), it is possible to analyze the detailed movement patterns of objects and accurately identify dangerous situations such as falls and violence.

[0023] Third, highly reliable image datasets can be generated by combining advanced signal processing and AI technologies, such as DBSCAN clustering, point cloud separation using Kalman filters, and CNN-based joint coordinate estimation.

[0024] Fourth, by training a radar-based joint estimation model using the joint coordinates of the image sensor as ground truth and gradually reducing the dependency on the image sensor through iterative training, it is possible to simultaneously secure training efficiency and the accuracy of the final model.

[0025] Fifth, by calculating composite errors (spatial distance error between joint coordinates, micro-Doppler signature similarity error, and object contour matching error) through a comparison of images generated with radar-only input and images combined with video data, it is possible to quantitatively evaluate the model's performance and enable continuous improvement. Brief explanation of the drawing

[0026] FIGS. 1 and 2 are flowcharts of a method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning according to an embodiment of the present invention. FIG. 3 is a block diagram of a device for generating an AI learning dataset for determining a radar-based dangerous situation through image sensor-assisted learning according to an embodiment of the present invention. FIG. 4 is an example illustrating the flow of generating a micro-Doppler signature according to one embodiment of the present invention. FIG. 5 is a diagram illustrating an example of aligning the contour and position of an object by mapping a 3D point restored using a plurality of joint coordinates with image data acquired from an image sensor, according to an embodiment of the present invention. FIG. 6 is an example for explaining a structure in which an AI model is trained according to an embodiment of the present invention. FIG. 7 is a graph showing the change in error according to the number of iterative learning cycles according to an embodiment of the present invention. Specific details for implementing the invention

[0027] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0028] As used herein, terms such as "examples," "examples," "aspects," "examples," etc., may not be interpreted as implying that any aspect or design described is better or more advantageous than other aspects or designs.

[0029] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0030] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0031] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0032] FIGS. 1 and 2 are flowcharts of a method for generating an AI learning dataset for determining a radar-based dangerous situation through image sensor-assisted learning according to an embodiment of the present invention; FIG. 3 is a block diagram of a device for generating an AI learning dataset for determining a radar-based dangerous situation through image sensor-assisted learning according to an embodiment of the present invention; FIG. 4 is an example illustrating the flow of generating a micro-Doppler signature according to an embodiment of the present invention; FIG. 5 is a diagram for explaining an example of matching the contour and position of an object by mapping a 3D point restored using a plurality of joint coordinates with image data acquired from an image sensor according to an embodiment of the present invention; FIG. 6 is an example for explaining the structure in which an AI model is trained according to an embodiment of the present invention; and FIG. 7 is a graph showing the change in error according to the number of iterations of learning according to an embodiment of the present invention.

[0033] Meanwhile, in the following description, some components described in the drawings may be omitted or excessively enlarged or reduced in order to explain the function of each component of the present invention, but it will be understood that such illustrated components do not limit the technical features and scope of rights of the present invention.

[0034] In addition, in the following description, multiple drawings will be referred to simultaneously to explain a single technical feature or a component constituting the invention.

[0035] A method for generating an AI learning dataset for determining a radar-based dangerous situation through image sensor-assisted learning according to one embodiment of the present invention is composed of a plurality of steps as illustrated in FIG. 1. The method of the present invention, implemented by a computing device comprising one or more processors and one or more memories for storing commands that can be executed by said processors, begins with a step (S10) of preprocessing point data collected from a radar sensor (30) to remove noise and classify valid points by object.

[0036] The point data processed in step (S10) consists of 5-dimensional time-series data including 3-dimensional spatial coordinates (x, y, z), reflection intensity, and Doppler shift. The radar sensor (30) can be implemented in various forms, such as mmWave radar, UWB radar, and FMCW radar, and in this embodiment, it is preferable to use a mmWave radar in the 77 GHz band. When the radar sensor (30) transmits radio waves and receives a signal reflected back from an object, the Intermediate Frequency (IF) frequency, phase difference, and reflection intensity of the received signal are analyzed to calculate the distance, speed, direction angle, and reflection intensity to the object. The distance is calculated using the formula d=(f_IFХc) / (2ХS), where f_IF represents the IF frequency, c represents the speed of light, and S represents the chirp slope. The speed is v=( Х It is calculated using the formula ) / (4ðХT_c), where is a phase change, represents the wavelength, and T_c represents the chirp period. The angle of arrival is =sin^(-1)(( Х It is calculated using the formula ) / (2ðХd_Rx)), where d_Rx represents the distance between receiving antennas.

[0037] These five-dimensional time series data are collected continuously over time and organized into frames, and each frame may contain hundreds to thousands of points. The preprocessing unit (11) performs the role of removing noise included in the collected point data and classifying valid points by object. Noise is generated by environmental factors, thermal noise of the sensor itself, multipath reflection, etc., and if it is not removed, the accuracy of object recognition is significantly reduced. In one embodiment of the present invention, noise removal may be configured as an embodiment including time-frequency domain filtering and a K-means-based cluster separation method.

[0038] Time-frequency domain filtering is a technique that removes noise components by simultaneously analyzing radar signals in the time and frequency domains. Methods such as the Short-Time Fourier Transform (STFT), Wavelet Transform, or Fast Fourier Transform (FFT) with applied window functions (Hamming window, Hann window, etc.) can be used. The STFT has the advantage of analyzing frequency changes over time by dividing the time axis into short intervals and performing a Fourier Transform on each interval, as well as the ability to selectively remove noise in specific frequency bands. The Wavelet Transform allows for signal analysis at various scales and locations, enabling simultaneous acquisition of time and frequency resolution, and can effectively eliminate impulse noise or intermittent interference.

[0039] K-means-based cluster separation is an algorithm that classifies point cloud data into multiple clusters based on their spatial distribution. The K-means algorithm forms optimal clusters by repeatedly setting k centroids randomly, assigning each point to the nearest centroid, and recalculating the centroids of each cluster. In addition to K-means, clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), OPTICS (Ordering Points To Identify the Clustering Structure), and Mean Shift may be used in this invention. DBSCAN is a density-based clustering algorithm that considers high-density areas as clusters and classifies low-density points as noise, thereby enabling the effective detection of irregularly shaped clusters. Furthermore, since there is no need to pre-specify the number of clusters, it is suitable for environments where multiple objects change dynamically.

[0040] The preprocessing unit (11) first removes noise by time-frequency domain filtering, and then classifies points by object by applying K-means or DBSCAN clustering. For example, if there are 3 people in a room, the point cloud corresponding to each person is separated into 3 clusters, and each cluster is assigned a unique ID and tracked. The valid points classified by object are then transmitted to the next stage in the form of 5-dimensional time series data.

[0041] Next, in step (S20), a three-dimensional shape is restored based on the position coordinates, reflection intensity, and velocity information of each object's point, and a micro-Doppler signature is generated from the restored three-dimensional shape. The three-dimensional restoration unit (12) generates a point cloud (100) from data (31) obtained through radar signal processing as shown in FIG. 4, and generates a micro-Doppler signature (200) from it.

[0042] 3D shape reconstruction is the process of reconstructing the spatial structure of an object from a distributed point cloud. Since radar point data contains only signals reflected from specific points on the object's surface, missing regions must be interpolated or estimated to reconstruct the entire shape. Point cloud reconstruction algorithms are used for this purpose, and algorithms such as Poisson Surface Reconstruction, Marching Cubes, Alpha Shapes, and Convex Hull can be applied. Poisson Surface Reconstruction constructs an implicit function using the normal vectors of the point cloud and converts it into a mesh to generate a smooth surface. Marching Cubes is an algorithm that divides 3D space into small cubes and approximates the surface within each cube to generate a polygon mesh; it is widely used in the reconstruction of medical images and 3D scan data.

[0043] A key feature of the present invention is the generation of micro-Doppler signatures from a reconstructed three-dimensional shape. In one embodiment of the present invention, the micro-Doppler signature represents a Doppler frequency change pattern according to the movement of an object in the time-frequency domain and is generated as a normalized Doppler pattern of a point cloud separated by object using DBSCAN (Density-Based Spatial Clustering of Applications with Noise) based clustering and a Kalman filter.

[0044] The micro-Doppler effect refers to additional Doppler frequency modulation created by minute internal movement elements of an object (e.g., limb movements, gait, breathing). For instance, in the case of a walking person, in addition to the main Doppler component caused by the movement of the entire body, the periodic movements of the arms and legs create a time-varying Doppler frequency pattern. These micro-Doppler signatures are highly effective in detecting sudden changes in motion, such as falls or violence. In the case of a fall, the movement is initially slow but then suddenly accelerates, manifesting as a rapid increase in Doppler frequency. In the case of violent behavior, sudden movements such as rapid arm swings or kicks create characteristic Doppler patterns.

[0045] To generate a micro-Doppler signature, a complex signal s(t) = I(t) + jQ(t) is first generated from I / Q (In-phase / Quadrature) data. The Doppler frequency is calculated as f_D = (1 / 2ð) * (d / dt) * arg(s(t)), where arg represents the complex phase angle. The micro-Doppler signature is defined as MDpS = v / t and is expressed as the ratio of speed to time. Doppler features over time are collected by applying a sliding window technique and converted into a time-frequency spectrogram. The spectrogram is represented in the form of a two-dimensional image where the x-axis represents time, the y-axis represents the Doppler frequency, and the color represents the signal intensity, which can be seen in the micro-Doppler signature (200) of FIG. 4.

[0046] DBSCAN clustering classifies point clouds based on density to separate point groups corresponding to each object. DBSCAN uses two parameters—neighborhood radius (epsilon) and minimum point count (minPts)—to consider a group as a single cluster if a sufficient number of points exist within a specific radius. Since points in low-density areas are classified as noise and removed, each object can be accurately separated even in environments where multiple objects are close together or there is significant background noise.

[0047] The Kalman filter is a recursive filter that removes noise from time-series data and estimates the state of an object. The Kalman filter consists of a prediction phase and an update phase; in the prediction phase, the current state is predicted based on the previous state and the system model, while in the update phase, the predicted value is corrected using actual measurements. In this invention, the Kalman filter is used to track the movement path of each object and generates an accurate micro-Doppler signature by continuously connecting points belonging to that object. In addition to the Kalman filter, the Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), and Particle Filter may be used. The EKF is a method that applies the Kalman filter to non-linear systems through linearization, while the UKF uses Sigma Points to more accurately estimate the mean and variance of non-linear transformations. The Particle Filter utilizes the Monte Carlo method to effectively estimate the state even in complex non-linear and non-Gaussian systems.

[0048] The Doppler patterns of normalized point clouds serve as signatures for each action and are used to classify different behaviors. For example, actions such as walking, running, sitting, standing, and falling each have unique micro-Doppler signatures; therefore, AI models trained on these signatures can automatically classify these actions in new data.

[0049] Next, in step (S30), radar-based joint coordinates are estimated from the restored 3D points, and the radar-based joint coordinates are mapped to image-based joint coordinates, which are multiple joint coordinates (skeleton joints) extracted from image data obtained from the image sensor (20), thereby spatially aligning the radar-based joint coordinates and the image-based joint coordinates to generate alignment data, which is joint data aligned to the contour and position of an object. The data mapping unit (13) performs the role of aligning the different coordinate systems of the radar sensor (30) and the image sensor (20) and fusing the data obtained from the two sensors.

[0050] The image sensor (20) can be implemented as an RGB camera, an RGB-D camera, an infrared camera, or a depth sensor such as Kinect or RealSense. In this embodiment, it is preferable to use an RGB-D sensor such as Microsoft Kinect V2, which provides 25 three-dimensional joint coordinates. Joint coordinates represent the positions of major joints of the body, such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles, expressed as (x, y, z) coordinates, and are essential information for analyzing a person's posture and behavior.

[0051] In one embodiment of the present invention, image mapping minimizes spatial error by calibrating the radar coordinate system and the image coordinate system with a matching matrix. The radar sensor (30) and the image sensor (20) each have unique coordinate systems, and since the installation position, orientation, and field of view of the two sensors are different, the data cannot be directly compared. Therefore, a transformation matrix representing the transformation relationship between the two coordinate systems must be obtained. The matching matrix is ​​expressed as a 4x4 homogeneous transformation matrix representing rotation and translation, and is calculated through a calibration process.

[0052] Calibration is performed using a standard object (e.g., checkerboard pattern, spherical reflector, etc.) that can be measured simultaneously by two sensors. The position of the standard object is measured by the two sensors, and the correspondence between the measurements is found to calculate the optimal matching matrix. For this purpose, algorithms such as Singular Value Decomposition (SVD), Iterative Closest Point (ICP), and Random Sample Consensus (RANSAC) can be used. ICP is an algorithm that iteratively finds corresponding points between two point clouds to optimize the transformation matrix, and it converges quickly if the initial estimates are appropriate. RANSAC is an algorithm capable of robustly estimating a model even in data with many outliers; it calculates a model by randomly selecting samples and selects the model with the most inliers.

[0053] Once the alignment matrix is ​​obtained, points in the radar coordinate system can be converted to the image coordinate system, or conversely, points in the image coordinate system can be converted to the radar coordinate system. Using the converted coordinates, data from the two sensors is projected into the same space, and optimal alignment is performed to minimize spatial error. Spatial error is measured by the Euclidean distance between corresponding points, and the alignment matrix is ​​adjusted in a direction that minimizes the sum of squares of the error.

[0054] FIG. 5 illustrates an example of aligning the contour and position of an object by mapping a 3D point restored using a plurality of joint coordinates with image data obtained from an image sensor, according to an embodiment of the present invention. The image-based joint coordinates (21) obtained from the image data consist of 25 joint points, each point representing the position of the head, neck, shoulder, elbow, wrist, spine, hip, knee, ankle, etc. The radar-based joint coordinates (121) obtained from the radar point data are values ​​estimated from the restored 3D point.

[0055] In one embodiment of the present invention, radar-based joint coordinates (121) consist of fewer joint points than the number of joints recognized by the image sensor, are estimated through a convolutional neural network (CNN) that takes 5-dimensional time series data as input, and are learned using image-based joint coordinates (21) as ground truth. In this embodiment, 19 joint coordinates are estimated from the radar, which is fewer than the 25 joints of the image sensor. The 19 joints include only major joints such as the head, neck, shoulders (left / right), elbows (left / right), wrists (left / right), spine center, hips (left / right), knees (left / right), and ankles (left / right). Fine joints such as fingers or toes are excluded because they are difficult to measure accurately due to the spatial resolution limitations of the radar.

[0056] A Convolutional Neural Network (CNN) is a deep learning model that learns spatial or temporal patterns in images or time-series data. A CNN consists of convolutional layers, activation functions, pooling layers, and fully connected layers. Convolutional layers apply filters (kernels) to input data to extract feature maps, and the weights of these filters are optimized through learning. Activation functions introduce non-linearity to enable the learning of complex patterns, and functions such as ReLU (Rectified Linear Unit), Leaky ReLU, Sigmoid, and Tanh are commonly used. Pooling layers reduce the size of feature maps to decrease computational load while preserving key features and preventing overfitting. Types include max pooling, average pooling, and global pooling; max pooling extracts the most prominent features by selecting the maximum value in each region.

[0057] The CNN model used in this invention consists of a 4-layer CNN as described in the specific technical data. Each convolutional layer uses a kernel of size 3x3, and the depth increases sequentially to 32, 64, 128, and 256. Depth refers to the number of feature maps output by each layer, and as the depth increases, more complex and high-dimensional features can be learned. The Leaky ReLU activation function mitigates the dying ReLU problem and allows small gradients even for negative inputs, enabling all neurons to participate in learning. Since ReLU is defined as f(x)=max(0, x) and outputs 0 for negative inputs, there is a problem where some neurons may be permanently deactivated. Leaky ReLU solves this problem by defining f(x)=max(αx, x) (where α is a small positive number, e.g., 0.01).

[0058] 2D max pooling with a 2x2 window is applied after each convolutional layer to reduce the size of the feature map by half and decrease computational complexity. The flatten layer converts the multidimensional feature map into a one-dimensional vector and provides it as input to the fully connected layer. The fully connected layer performs a final prediction by connecting all elements of the input vector to the output neurons, and in this embodiment, it consists of a fully connected layer with 512 neurons and an output layer with 57 neurons (19 joints x 3D coordinates).

[0059] The input to the CNN is a 5-channel feature map composed of 5-dimensional time-series data on a frame-by-frame basis. Each channel corresponds to x, y, z, intensity, and Doppler shift, and is arranged along the time axis. The CNN learns spatial and temporal patterns from these 5-channel inputs to output 3D coordinates of 19 joints. During the training process, the coordinates of 25 joints from the image sensor are used as ground truth, but these 25 joints are mapped to 19 major joints. For example, the left fingers and right fingers are combined into the left wrist and right wrist, and the toes are combined into the ankle.

[0060] The loss function measures the difference between the CNN's predicted values ​​and ground truth, using either Mean Squared Error (MSE) or Mean Absolute Error (MAE). MSE is defined as the average of the squared errors and is sensitive to outliers by imposing larger penalties for significant errors. MAE is defined as the average of the absolute values ​​of the errors and is more robust against outliers. CNN weights are updated through backpropagation algorithms and optimization algorithms (such as Adam, SGD, and RMSprop), and training proceeds in the direction that minimizes the loss function. Adam (Adaptive Moment Estimation) automatically adjusts the learning rate and combines the strengths of momentum and RMSprop to provide fast and stable convergence.

[0061] The alignment data (300), which is joint data aligned with the contour and position of an object derived from the mapping result of the restored 3D points and image data, is expressed in a form in which radar-based joint coordinates (121) and image-based joint coordinates (21) are spatially aligned, as shown in FIG. 5. The alignment data (300) thus aligned integrates information from the two sensors to represent the accurate orientation of the object and is used to generate an image dataset in the next step.

[0062] Next, in step (S40), an image dataset for AI analysis is generated based on the matching data, including contour, movement, and distance information for each object. As can be seen in the configuration diagram of FIG. 3, the dataset generation unit (14) integrates the data received from the preprocessing unit (11), the 3D reconstruction unit (12), and the data mapping unit (13) to generate an image dataset suitable for AI model training.

[0063] The contour of an object represents its outline and is extracted from point clouds or joint coordinates. Contour extraction algorithms that can be used include Canny edge detection, the Sobel operator, the Laplacian of Gaussian, and contour tracing. Canny edge detection is a multi-stage algorithm that extracts accurate edges through noise removal using a Gaussian filter, calculation of gradient magnitude and direction, non-maximum suppression, double thresholding, and hysteresis edge tracing. If joint coordinates are available, the contour can be approximated by drawing a skeleton connecting the joints and applying elliptical or cylindrical bounding boxes to each body part.

[0064] Motion information represents changes in an object's position, velocity, and acceleration over time. The position of the same object is tracked across consecutive frames to calculate a motion vector, and velocity and acceleration are calculated by differentiating this vector with respect to time. Optical flow algorithms can also be used, such as the Lucas-Kanade method, the Horn-Schunck method, and the Farneback method. Lucas-Kanade is a local method that assumes motion is constant within a small area and calculates the motion vector using the least squares method. Horn-Schunck is a global method that calculates dense optical flow by adding the constraint that motion is smooth across the entire area. In the present invention, since the micro-Doppler signature already contains velocity information, it can be utilized as motion information.

[0065] Distance information refers to the distance between an object and a sensor, or the relative distance between each part within an object. Since radar sensors inherently possess distance measurement capabilities, they can directly utilize distance information for each point. Distance information is represented by the depth channel of an image and forms an RGB-D image together with the RGB image. Because depth information clearly represents the three-dimensional structure of an object, it significantly improves the accuracy of pose estimation or behavior recognition.

[0066] Image datasets for AI analysis are generated by integrating object-specific contour, motion, and distance information. Image formats can vary and may consist of 2D images, 3D volumes, time-series image sequences, etc. For example, a multi-channel image can be generated by converting micro-Doppler signatures into time-frequency spectrogram images and combining them with a skeleton image in which joint coordinates are projected onto a 2D plane. Additionally, a radar point cloud can be converted into a voxel grid and used as a 3D input. A voxel is a division of 3D space into small cubes, and each voxel stores whether a point exists in the corresponding area and the attributes of the point (reflection intensity, velocity, etc.).

[0067] Labels are assigned to the generated image dataset. These labels indicate the type of behavior represented by the images (e.g., walking, sitting, falling, violence, etc.) and are used for supervised learning. Labeling can be performed manually or automatically extracted from image sensors. By using the results of recognizing human behavior via image sensors as labels, a large amount of training data can be generated efficiently.

[0068] Next, in step (S50), the dataset is combined with radar and image data and input into the trained AI model to train the AI ​​model. The model training unit (15) performs training by inputting image data (21) and radar point data (32) into the initial model (400) as shown in FIG. 6, and generates the trained model (410).

[0069] AI models can be implemented using various structures, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), Gated Recurrent Units (GRUs), Transformers, and Graph Neural Networks (GNNs). CNNs are suitable for learning spatial patterns and are widely used in image classification, object detection, and segmentation. RNNs and LSTMs can learn the temporal dependencies of time-series data and are useful for tracking changes in behavior across consecutive frames. LSTMs are structures designed to solve the vanishing gradient problem of RNNs, controlling the flow of information through input, forget, and output gates. GRUs are a simplified version of LSTMs that reduces computational load by decreasing the number of gates while providing similar performance.

[0070] Transformers are based on attention mechanisms and effectively learn long-range dependencies by simultaneously considering the relationships between all positions in a sequence. Self-attention calculates the relevance of each element in an input sequence to all other elements to create a weighted sum, while multi-head attention performs multiple attentions in parallel to extract information from various perspectives. Graph neural networks process data with a graph structure and can recognize actions by representing skeleton joints as nodes and connections between joints as edges. Examples include Graph Convolutional Networks (GCN), Graph Attention Networks (GAT), and Spatial-Temporal Graph Convolutional Networks (ST-GCN); ST-GCN analyzes dynamic skeleton sequences by performing both spatial and temporal graph convolutions.

[0071] In this invention, a multi-stream structure can be used to process multimodal inputs. The multi-stream structure configures separate network streams for each input modality (e.g., micro-Doppler signatures, joint coordinates, point clouds) and fuses features extracted from each stream in the latter part. Fusion methods include simple concatenation, weighted sum, and attention-based fusion. Since attention-based fusion dynamically calculates the importance of each modality and assigns weights, it can focus on more useful information depending on the situation.

[0072] During the training process, the model's weights are updated in a direction that minimizes the loss function. Cross-entropy loss is used for classification problems, while Mean Squared Error (MSE) is used for regression problems. Regularization techniques such as L1 regularization, L2 regularization, dropout, and batch normalization are applied to prevent overfitting. Dropout randomly disables some neurons during training to prevent the model from relying excessively on specific neurons, while batch normalization normalizes the inputs of each layer to stabilize and accelerate training.

[0073] The image dataset result (401) obtained from the initial model (400) is generated using both image data and radar data and has high accuracy. However, since image sensors cannot be used during actual operation, the model must be trained to produce similar performance using radar alone. To this end, transfer learning is performed in the next step.

[0074] Next, in step (S60), the step of training the AI ​​model is repeated so that an image dataset for AI analysis can be generated using only radar input without an image sensor. As illustrated in FIGS. 2 and FIGS. 6, the autonomous learning unit (16) gradually reduces dependence on the image sensor through a repetitive learning process and finally completes a model that operates solely on radar.

[0075] In one embodiment of the present invention, the step (S60) of generating an image dataset for AI analysis using only radar input can specifically perform the following process. First, in the initial learning stage, the radar-based joint estimation result is corrected by referring to the joint coordinates obtained from image data (S61), the dependency on the image sensor is gradually reduced through iterative learning, and finally, transfer learning is performed to estimate joint coordinates using only radar input without image data and to generate an image dataset for AI analysis.

[0076] Step (S61), as illustrated in FIG. 2, is a process of correcting radar-based joint estimation results by referencing joint coordinates obtained from image data during the initial learning stage. Since it is difficult to estimate accurate joint coordinates using only radar data initially, accurate joint coordinates obtained from the image sensor are used as the teacher signal. The difference between the joint coordinates output by the radar-based model and the joint coordinates of the image sensor is calculated as a loss function, and the model is updated through backpropagation. In this process, the relationship between the pattern of the radar data and the joint coordinates is learned.

[0077] Transfer learning is a method of transferring knowledge learned in one task to another. In this invention, knowledge learned in a multimodal model using both image and radar sensors is transferred to a single-modal model using only a radar sensor. To this end, techniques such as knowledge distillation, fine-tuning, and domain adaptation may be used. Knowledge distillation is a method of training a simple student model to mimic the output of a complex teacher model, inducing the student model's learning using the teacher model's soft labels. Soft labels represent probability distributions by class and provide more information than hard labels (1 for the correct class and 0 for the rest).

[0078] The gradual reduction of dependence on the image sensor through iterative learning is performed in stages. Initially, the weight of the image sensor is set high, and as the number of iterations increases, the weight of the image sensor is gradually lowered while the weight of the radar sensor is increased. For example, the loss function is defined as L = α L_vision + (1-α) L_radar, and α is decreased from 1 to 0. Here, L_vision represents the image-based loss, and L_radar represents the radar-based loss. Initially, α is set to 1 to rely solely on the image sensor, and as α is gradually decreased until it finally becomes α=0, only the radar sensor is used.

[0079] In one embodiment of the present invention, when the number of repetitions reaches a preset threshold, an AI analysis image generated when radar is input alone and an AI analysis image data set when image data is combined are each generated, and an error between the two image data sets is calculated, wherein the error is calculated as a composite error including a spatial distance error between joint coordinates, a similarity error of micro-Doppler signatures, and an agreement error of object contours (S62).

[0080] Step (S62) is a process of calculating a composite error (161) by comparing the image dataset result (402) obtained from the learned model (410) with the image dataset result (401) obtained from the initial model (400), as illustrated in FIGS. 2 and 6. The spatial distance error between joint coordinates is calculated as the Euclidean distance between the 3D positions of each joint, and the errors are summed or averaged for all 19 joints. The formula is E_joint = Σ It is expressed as - x_v)² + (y_r - y_v)² + (z_r - z_v)²), where (x_r, y_r, z_r) are radar-based joint coordinates and (x_v, y_v, z_v) are image-based joint coordinates.

[0081] The similarity error of micro-Doppler signatures measures the similarity between two signature images. Similarity measurement methods that may be used include the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), cosine similarity, and mutual information. SSIM evaluates image quality in a manner similar to human visual perception by comparing three factors: brightness, contrast, and structure. The SSIM value ranges from 0 to 1, where 1 indicates complete identicality. The error is calculated as E_doppler = 1 - SSIM, and a smaller value indicates greater similarity between the two signatures.

[0082] The congruence error of object contours measures the morphological similarity between two contours. Hausdorff distance, Frechet distance, and IoU (Intersection over Union) can be used. Hausdorff distance measures the maximum and minimum distance between two sets and indicates how close all points of the contours are. IoU is the value obtained by dividing the intersection of two contours by their union, ranging from 0 to 1, where 1 signifies a perfect match. The error is calculated as E_contour = 1 - IoU.

[0083] The composite error is calculated by combining these three errors into a weighted sum. It is expressed as E_total = w1 Х E_joint + w2 Х E_doppler + w3 Х E_contour, where w1, w2, and w3 are weights representing the importance of each error. The weights can be adjusted according to the application field; for example, w1 is set high when pose estimation is important, and w2 is set high when behavior recognition is important. In this embodiment, w1 is set to 0.4, w2 to 0.4, and w3 to 0.2.

[0084] FIG. 7 is a graph showing the change in error according to the number of iterations of learning according to an embodiment of the present invention. The x-axis represents the number of learning iterations, and the y-axis represents the composite error. Initially, the error is large, but as learning progresses, the error gradually decreases. yth represents a preset threshold error range, and it is determined that learning has progressed sufficiently when the error decreases below this threshold. t1 represents the point in time (number of times) when the error enters yth.

[0085] Step (S63), as illustrated in FIG. 2, is a process of completing transfer learning to estimate joint coordinates using only radar input without image data and generate an image dataset for AI analysis when the error calculated in S62 enters a preset threshold error range (yth). When the error decreases below the threshold, it is determined that the radar-only model has achieved performance similar to a model using both an image sensor and a radar sensor, and the image sensor is completely removed. Subsequently, data is collected using only the radar sensor, joint coordinates are estimated through the trained model, and an image dataset is generated.

[0086] Even after transfer learning is complete, the model can be improved through continual learning or online learning. Fine-tuning the model by collecting data from new environments or new users can further enhance performance in specific situations. Additionally, the problem of insufficient training data can be mitigated by applying semi-supervised learning or self-supervised learning using unlabeled data.

[0087] As another embodiment of the present invention, an AI learning dataset generation device (10) for determining a radar-based dangerous situation through image sensor-assisted learning is provided, which is implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors. FIG. 3 illustrates the configuration of an AI learning dataset generation device (10) for determining a radar-based dangerous situation through image sensor-assisted learning according to one embodiment of the present invention.

[0088] The device (10) includes a preprocessing unit (11), a three-dimensional reconstruction unit (12), a data mapping unit (13), a dataset generation unit (14), a model learning unit (15), and an autonomous learning unit (16). Additionally, the device (10) receives data by being connected to an image sensor (20) and a radar sensor (30). The image sensor (20) and the radar sensor (30) may be integrated as part of the device (10) or may be connected to an external device.

[0089] The preprocessing unit (11) preprocesses point data collected from the radar sensor (30) to remove noise and classify valid points by object, wherein the point data consists of 5-dimensional time series data including 3-dimensional spatial coordinates (x, y, z), reflection intensity, and Doppler shift. The preprocessing unit (11) may be composed of a signal processing module, a noise filter module, a clustering module, etc., and each module may be implemented in dedicated hardware (FPGA, ASIC, etc.) or software.

[0090] The 3D reconstruction unit (12) reconstructs a 3D shape based on the position coordinates, reflection intensity, and velocity information of points per object, and generates a micro-Doppler signature from the reconstructed 3D shape. The 3D reconstruction unit (12) may be composed of a point cloud processing module, a surface reconstruction module, a Doppler analysis module, etc. The Doppler analysis module includes an FFT processor, a spectrogram generator, a DBSCAN clustering engine, a Kalman filter, etc.

[0091] The data mapping unit (13) estimates radar-based joint coordinates from the restored 3D point and maps the radar-based joint coordinates to image-based joint coordinates, which are multiple joint coordinates (skeleton joints) extracted from image data acquired from the image sensor (20), thereby spatially aligning the radar-based joint coordinates and the image-based joint coordinates to generate alignment data, which is joint data aligned to the contour and position of an object. The data mapping unit (13) may be composed of a calibration module, a coordinate transformation module, a joint estimation module, an alignment module, etc. The joint estimation module includes a CNN network and performs computations at high speed using an accelerator such as a GPU (Graphics Processing Unit) or TPU (Tensor Processing Unit).

[0092] The dataset generation unit (14) generates an image dataset for AI analysis that includes object-specific contours, movement, and distance information based on matching data. The dataset generation unit (14) may be composed of a feature extraction module, an image rendering module, a labeling module, a data storage module, etc. The data storage module stores the generated image dataset in a file system or database and may use formats such as HDF5, TFRecord, LMDB, etc.

[0093] The model training unit (15) inputs the dataset into the AI ​​model trained by combining radar and image data to train the AI ​​model. The model training unit (15) may be composed of a neural network structure definition module, a learning algorithm module, a loss function calculation module, an optimization module, etc. Training is performed in batch units, and the batch size is set considering memory capacity and training stability. Generally, values ​​such as 16, 32, and 64 are used.

[0094] The autonomous learning unit (16) enables the generation of an image dataset for AI analysis using only radar input without an image sensor by repeatedly performing the learned AI model. The autonomous learning unit (16) may be composed of a transfer learning module, an error evaluation module, a weight adjustment module, a convergence determination module, etc. The convergence determination module monitors the complex error and terminates the learning when it decreases below a threshold value, and saves the final model.

[0095] Each component of the device (10) may be integrated into a single physical device or distributed across multiple devices. For example, the sensors (20, 30) and the preprocessing unit (11) may be located on an edge device, and the remaining components may be located on a cloud server to perform cloud-based learning. Alternatively, all components may be integrated into an embedded system to implement a device that operates independently.

[0096] As another embodiment of the present invention, a computer-readable recording medium is provided having a program recorded thereon for executing on a computer the method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning as described in claim 1, as described in claim 9. The computer-readable recording medium may be implemented in various forms such as ROM, RAM, flash memory, CD-ROM, DVD, Blu-ray disc, hard disk, SSD, magnetic tape, optical disc, etc. Additionally, the program may be downloaded via the Internet or stored in cloud storage and executed remotely.

[0097] The program consists of a set of instructions for executing the method of the present invention and runs on an operating system (OS). The program can be written in programming languages ​​such as Python, C++, Java, and MATLAB, and can implement an AI model using deep learning frameworks such as TensorFlow, PyTorch, Keras, and Caffe. TensorFlow is an open-source framework developed by Google that can define and efficiently execute computation graphs and supports GPU acceleration. PyTorch is a framework developed by Facebook that supports dynamic computation graphs, facilitates debugging, and is widely used for research purposes.

[0098] The program may also include libraries for data preprocessing, visualization, and evaluation. NumPy and SciPy are libraries for scientific computing that provide functions such as matrix operations, FFT, and optimization. OpenCV is a computer vision library that provides functions such as image processing, object detection, and tracking. Matplotlib and Seaborn are data visualization libraries used to draw graphs and charts. scikit-learn is a machine learning library that provides algorithms for classification, regression, and clustering.

[0099] The present invention may have various embodiments and variations, and the embodiments described above are merely examples to aid in understanding the invention and do not limit the scope of the invention. For example, the radar sensor may be replaced with radars of various frequency bands and types, such as 24GHz radar, 60GHz radar, and UWB radar, in addition to 77GHz mmWave radar. The image sensor may be replaced with RealSense, LiDAR, ToF cameras, etc., in addition to Kinect. The number of joints may be changed to 15, 17, 25, etc., in addition to 19, and only the necessary joints may be selected and used depending on the application field.

[0100] The structure of the AI ​​model can be varied, and backbone networks such as ResNet, VGG, Inception, and EfficientNet can be used, or object detection networks such as YOLO, SSD, and Faster R-CNN can be combined. In addition, performance can be improved by applying advanced techniques such as attention mechanisms, residual connections, and dense connections.

[0101] The types of hazardous situations can be expanded beyond falls and violence to include various scenarios such as intrusion, loitering, fainting, seizures, fire, and gas leaks. By collecting characteristic micro-Doppler signatures for each situation and building a trained model, a comprehensive hazardous situation identification system can be implemented.

[0102] The present invention can be applied to various environments such as smart homes, smart cities, hospitals, nursing homes, schools, offices, factories, airports, and train stations. In particular, by using radar sensors instead of video sensors in environments where privacy is critical (e.g., bathrooms, bedrooms), safety monitoring and privacy protection can be achieved simultaneously. Furthermore, since it operates reliably even at night or in smoky environments, it can be effectively utilized in fire scenes or disaster situations.

[0103] The method of the present invention can be optimized for real-time processing and can be lightweighted to operate on edge computing devices. Model compression techniques such as pruning, quantization, knowledge distillation, and Neural Architecture Search (NAS) may be used. Pruning reduces the model size by removing weights or neurons of low importance, and quantization reduces memory and computational load by representing weights and activations with a low number of bits (e.g., 8 bits, 4 bits).

[0104] The present invention can also track and analyze multiple objects simultaneously. It can individually monitor the behavior of each person in an environment with multiple people and send an immediate alert when a specific person is in a dangerous situation. To this end, object tracking algorithms (such as the Kalman filter, Hungarian algorithm, SORT, and DeepSORT) may be used together. The Hungarian algorithm is an algorithm that finds the optimal match in a bipartite graph and is used to determine the correspondence relationship between objects in frames. SORT (Simple Online and Realtime Tracking) is a real-time tracking algorithm that combines the Kalman filter and the Hungarian algorithm, and DeepSORT improves tracking accuracy by adding deep learning-based appearance features to SORT.

[0105] The effectiveness of the present invention can be evaluated quantitatively, and indicators such as accuracy, precision, recall, F1 score, ROC curve, and AUC (Area Under the Curve) are used. Accuracy is the proportion of correctly classified samples out of the total samples; precision is the proportion of actually positive samples among those predicted as positive; and recall is the proportion of actually positive samples among those predicted as positive. The F1 score is the harmonic mean of precision and recall, representing the balance between the two indicators. The ROC curve plots the True Positive Rate and False Positive Rate for various threshold values, and the AUC is the area under the ROC curve; a value closer to 1 indicates superior performance.

[0106] FIG. 8 illustrates an example of the internal configuration of a computing device according to an embodiment of the present invention. In the following description, descriptions of unnecessary embodiments that overlap with the descriptions of FIG. 1 to 4 described above will be omitted.

[0107] As illustrated in FIG. 8, the computing device (10000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (10000) may correspond to a user terminal (A) connected to a haptic interface device or the aforementioned computing device (B).

[0108] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (10000).

[0109] At this time, access to memory (11200) from other components, such as the processor (11100) or peripheral device interface (11300), can be controlled by the processor (11100).

[0110] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (10000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (10000) and process data by executing software modules or instruction sets stored in the memory (11200).

[0111] The input / output subsystem (11400) can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem (11400).

[0112] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0113] The communication circuit (11600) can enable communication with another computing device using at least one external port.

[0114] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.

[0115] The embodiment of FIG. 8 is merely an example of a computing device (10000), and the computing device (11000) may have some components shown in FIG. 8 omitted, additional components not shown in FIG. 8 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include, in addition to the components shown in FIG. 8, a touchscreen or a sensor, etc., and the communication circuit (1160) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (10000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0116] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0117] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0118] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed across networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0119] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0120] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below also fall within the scope of the claims.

Claims

Claim 1 The present invention relates to a method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning, implemented as a computing device comprising one or more processors and one or more memories storing instructions executable by said processors, comprising: a step of preprocessing point data collected from a radar sensor to remove noise and classifying valid points by object, wherein the point data is composed of 5-dimensional time-series data including 3-dimensional spatial coordinates (x, y, z), reflection intensity, and Doppler shift; a step of restoring a 3-dimensional shape based on the 3-dimensional spatial coordinates, reflection intensity, and Doppler shift of the classified valid points by object, wherein a micro-Doppler signature is generated from the restored 3-dimensional shape; and a step of estimating radar-based joint coordinates from the restored 3-dimensional points, and mapping the radar-based joint coordinates to image-based joint coordinates, which are a plurality of skeleton joint coordinates extracted from image data acquired from the image sensor, thereby spatially aligning the radar-based joint coordinates and the image-based joint coordinates to obtain the outline of an object A method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning, characterized by comprising: a step of generating alignment data, which is joint data aligned by position; a step of generating an image dataset for AI analysis including object-specific contour, movement, and distance information based on the alignment data; a step of inputting the dataset into an AI model trained by combining radar and image data to train the AI ​​model; and a step of repeating the step of training the AI ​​model so that an image dataset for AI analysis can be generated using only radar input without an image sensor. Claim 2 A method for generating an AI learning dataset for radar-based risk situation identification through image sensor-assisted learning, characterized in that, in claim 1, the micro-Doppler signature represents a Doppler frequency change pattern according to the movement of an object in the time-frequency domain, and is generated as a normalized Doppler pattern of a point cloud separated by object using DBSCAN (Density-Based Spatial Clustering of Applications with Noise) based clustering and a Kalman filter. Claim 3 A method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning, characterized in that, in claim 1, the radar-based joint coordinates consist of fewer joint points than the number of joints recognized from an image sensor, are estimated through a convolutional neural network (CNN) that takes the 5-dimensional time series data as input, and are learned using the image-based joint coordinates as ground truth. Claim 4 A method for generating an AI learning dataset for AI analysis based on radar-assisted learning through image sensor assistance, wherein, in claim 1, the step of enabling the generation of an image dataset for AI analysis using only radar input is characterized by, in an initial learning step, correcting the estimation result of the radar-based joint coordinates by referring to the image-based joint coordinates, gradually reducing the dependency on the image sensor through iterative learning, and finally performing transfer learning to estimate the radar-based joint coordinates using only radar input without image data and generate an image dataset for AI analysis. Claim 5 A method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning, wherein, in claim 4, the step of enabling the generation of an image dataset for AI analysis using only radar input is characterized by generating an image for AI analysis generated when radar is input alone and an image dataset for AI analysis when image data is combined, respectively, when the number of iterations reaches a preset threshold number, and calculating an error between the two image datasets, wherein the error is calculated as a composite error including a spatial distance error between joint coordinates, a similarity error of micro-Doppler signatures, and an accuracy error of object contours. Claim 6 A method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning, characterized in that, in claim 1, the noise removal includes time-frequency domain filtering and K-means-based cluster separation. Claim 7 A method for generating an AI learning dataset for radar-based risk situation determination through image sensor-assisted learning, wherein, in claim 1, the mapping is characterized by minimizing spatial error by calibrating the radar coordinate system and the image coordinate system with a matching matrix. Claim 8 The present invention relates to an AI learning dataset generation device for radar-based risk situation determination through image sensor-assisted learning, implemented as a computing device comprising one or more processors and one or more memories storing instructions executable by said processors, comprising: a preprocessing unit that preprocesses point data collected from a radar sensor to remove noise and classify valid points by object, wherein the point data is composed of 5-dimensional time-series data including 3-dimensional spatial coordinates (x, y, z), reflection intensity, and Doppler shift; a 3-dimensional reconstruction unit that reconstructs a 3-dimensional shape based on the 3-dimensional spatial coordinates, reflection intensity, and Doppler shift of the classified valid points by object, and generates a micro-Doppler signature from the reconstructed 3-dimensional shape; and a radar-based joint coordinate estimation from the reconstructed 3-dimensional point, and maps the radar-based joint coordinate to image-based joint coordinates, which are a plurality of joint coordinates (skeleton joints) extracted from image data acquired from the image sensor, thereby spatially mapping the radar-based joint coordinate and the image-based joint coordinate A device for generating an AI learning dataset for determining dangerous situations based on radar through image sensor-assisted learning, characterized by comprising: a data mapping unit that generates alignment data, which is joint data aligned to the contours and positions of objects by aligning; a dataset generation unit that generates an image dataset for AI analysis including object-specific contours, movement, and distance information based on the alignment data; a model learning unit that inputs the dataset into a trained AI model by combining radar and image data to train the AI ​​model; and an autonomous learning unit that enables the generation of an image dataset for AI analysis using only radar input without an image sensor by repeatedly executing the trained AI model. Claim 9 A computer-readable recording medium having a program recorded thereon for executing on a computer the method for generating an AI learning dataset for radar-based danger situation identification through image sensor-assisted learning as described in paragraph 1.

Images