Remote monitoring system using posture identification code and method thereof
The system addresses network congestion and privacy concerns by performing deep learning inference at the edge to generate a unique identification code for human posture, reducing traffic and server load, and providing real-time, intuitive 3D visualization.
Patent Information
- Application Number
- KR1020260131647
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-07-16
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-07-16
AI Technical Summary
Conventional remote monitoring systems using millimeter wave radar sensors face issues with excessive network bandwidth consumption and server overload due to the transmission of large volumes of raw data, compromising real-time performance and privacy.
A remote monitoring system that performs deep learning inference at the edge using a radar sensor module to generate a unique identification code for human posture, transmitting only this code and metadata over the network, while using a user terminal to render a 3D graphic model for intuitive visualization.
Reduces communication traffic and server load, enabling real-time monitoring and accurate, intuitive 3D visualization of human posture without infringing on privacy, even in unstable Wi-Fi environments.
Smart Images

Figure R1020260131647_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a remote monitoring system and method using a posture identification code. Background Technology
[0003] With the recent entry into an aging society, the need for remote monitoring systems has emerged to prevent lonely deaths among the elderly living alone and to respond quickly to emergencies such as falls. Conventional monitoring systems have primarily utilized cameras (CCTV), but this poses serious privacy infringement issues. To overcome this, there is an increasing number of attempts to determine human posture by utilizing millimeter wave (mmWave) radar sensors instead of vision sensors. However, conventional technology has limitations in that it directly streams large volumes of raw data (point clouds or micro-Doppler signatures) generated by radar to a server or smartphone to perform computational processing centrally, thereby consuming massive network bandwidth and causing server overload and reduced real-time performance when multiple devices are connected.
[0004] The information described above disclosed in the background technology of this invention is intended only to enhance understanding of the background of the present invention and may therefore include information that does not constitute prior art. The problem to be solved
[0006] The technical problem that the present invention aims to solve is to provide a remote monitoring system and method that can precisely determine the posture of a human body without infringing on the privacy of the subject, minimize network traffic by transmitting only a unique identification code calculated by classifying the posture at the edge without transmitting large volumes of raw data generated from a radar sensor to the outside, and provide intuitive 3D visualization on a mobile terminal based on the received identification code.
[0007] However, the technical problems that the present invention aims to solve are not limited to those described above, and other unmentioned problems can be clearly understood by those skilled in the art from the description of the invention below. means of solving the problem
[0009] A remote monitoring system using an attitude identification code according to an embodiment of the present invention for solving the above technical problem may include: a radar sensor module that transmits and receives a millimeter wave (mmWave) signal toward a target located indoors, processes the received signal to calculate a unique identification code corresponding to the attitude, and transmits metadata including the unique identification code and the location coordinates of the target, excluding raw data for the target; a server that receives the unique identification code and metadata from the radar sensor module and relays them; and a user terminal that receives the unique identification code and metadata through the server, extracts a 3D graphic model corresponding to the received unique identification code by referring to a 3D visualization model database for each attitude, and renders and displays the 3D graphic model in an indoor virtual space on a screen.
[0010] In one embodiment, the radar sensor module may include a mmWave transceiver that transmits an electromagnetic wave signal and receives a signal reflected back from the target, and a signal processing unit that performs signal processing operations to extract distance, speed, and angle information from the signal received by the mmWave transceiver to generate three-dimensional point cloud data.
[0011] In one embodiment, the radar sensor module may include a deep learning model storage unit that stores weight data and an attitude database of a pre-trained deep learning model, and an attitude classification code generation unit that inputs the 3D point cloud data into the pre-trained deep learning model stored in the deep learning model storage unit to classify the attitude of a target and calculate the unique identification code.
[0012] In one embodiment, the attitude classification code generation unit is implemented through a microcontroller provided in the radar sensor module, so that inference operations by the deep learning model can be performed within the radar sensor module.
[0013] In one embodiment, the radar sensor module may include a communication unit that packetizes the unique identification code and metadata and transmits them externally via a wireless network.
[0014] In one embodiment, the user terminal may include a code receiving unit that receives the unique identification code and metadata from the server, and a pose 3D model matching unit that loads a 3D graphic model that matches the unique identification code received by the code receiving unit by referring to the pose-specific 3D visualization model database.
[0015] In one embodiment, the user terminal may further include a display unit that maps a 3D graphic model loaded by the pose 3D model matching unit to an indoor virtual space coordinate system and sets the direction to render it on a screen, based on the location coordinates of a target included in the metadata received by the code receiving unit.
[0016] A remote monitoring method using an attitude identification code according to an embodiment of the present invention may include: a step in which a radar sensor module transmits and receives a millimeter wave signal toward a target located indoors and processes the received signal to calculate a unique identification code corresponding to a posture classified as one of the designated postures; a step in which the radar sensor module transmits metadata including the unique identification code and the location coordinates of the target to the outside, excluding raw data for the target; a step in which a server receives the transmitted unique identification code and metadata and relays them to a user terminal; and a step in which the user terminal matches the received unique identification code with a 3D visualization model database for each posture to extract a 3D graphic model and renders and displays the 3D graphic model in an indoor virtual space on a screen.
[0017] In one embodiment, the step of calculating the unique identification code may include a process of generating three-dimensional point cloud data by performing a signal processing operation to extract distance, speed, and angle information for the received signal.
[0018] In one embodiment, the step of calculating the unique identification code includes the process of generating a multidimensional feature vector from the three-dimensional point cloud data, and the multidimensional feature vector may be formed by combining spatial geometric features including elevation statistics, horizontal and vertical spatial dispersion, and aspect ratio of the target bounding box, Doppler kinematic features including average velocity and velocity dispersion of target points, and signal energy features including the sum of the signal-to-noise ratio per frame.
[0019] In one embodiment, the step of calculating the unique identification code may further include the process of constructing input data for a deep learning model by additionally combining the rate of change of target center point elevation between consecutive time frames to model static attitude and dynamic attitude transition processes.
[0020] In one embodiment, the step of calculating the unique identification code may include a process in which a microcontroller equipped in the radar sensor module performs an inference operation based on a pre-trained deep learning model within the radar sensor module.
[0021] In one embodiment, the step of rendering and displaying may include mapping the 3D graphic model to a coordinate system of an indoor virtual space and setting the direction based on the location coordinates of the target included in the metadata.
[0022] In one embodiment, the step of rendering and displaying may further include a process of updating a 3D graphic model on the screen and displaying it as a real-time monitoring interface based on a unique identification code that is updated as the steps of calculating the unique identification code, transmitting metadata externally, and relaying to a user terminal are repeatedly performed.
[0023] A radar sensor module according to one embodiment of the present invention may include: a mmWave transceiver that transmits an electromagnetic wave signal and receives a signal reflected back from a target; a signal processing unit that generates three-dimensional point cloud data by performing a signal processing operation to extract distance, speed, and angle information on the signal received by the mmWave transceiver; a deep learning model storage unit that stores weight data of a pre-trained deep learning model and an attitude database; an attitude classification code generation unit that inputs the three-dimensional point cloud data into a pre-trained deep learning model stored in the deep learning model storage unit to classify the attitude of the target and calculate a unique identification code corresponding to the classified attitude; and a communication unit that packetizes metadata including the unique identification code and the location coordinates of the target, excluding raw data for the target, and transmits it externally through a wireless network.
[0024] In one embodiment, the pose classification code generation unit is implemented through a microcontroller and can perform inference operations by the pre-trained deep learning model.
[0025] In one embodiment, the attitude classification code generation unit generates a multidimensional feature vector from the three-dimensional point cloud data, and the multidimensional feature vector may be configured by combining spatial geometric features including elevation statistics, horizontal and vertical spatial dispersion, and aspect ratio of the target bounding box, Doppler kinematic features including average velocity and velocity dispersion of target points, and signal energy features including the sum of the signal-to-noise ratio per frame.
[0026] In one embodiment, the attitude classification code generation unit may additionally combine the rate of change of target center point elevation between consecutive time frames with the multidimensional feature vector to model the static attitude and dynamic attitude transition process. Effects of the invention
[0028] According to the present invention, instead of transmitting large-capacity point cloud data generated by a radar sensor module to the outside as is, only a unique identification code of a few bytes calculated through an attitude classification code generation unit is transmitted through a communication unit, thereby drastically reducing communication traffic costs and enabling real-time monitoring without delay even in an unstable Wi-Fi environment.
[0029] According to the present invention, since deep learning inference for attitude classification is performed at the edge level from the attitude classification code generation unit (MCU) within the radar sensor module and heavy AI computations are not performed on the server, it is possible to significantly reduce the construction and maintenance costs of the central server even when simultaneously applied to tens of thousands of households of elderly people living alone.
[0030] According to the present invention, a unique identification code corresponding to a subdivided posture is predefined and stored in a deep learning model storage unit, and a 3D visualization model database corresponding thereto is embedded in a mobile terminal, thereby enabling the user to perceive the posture and state of the subject very intuitively and accurately through a high-resolution 3D avatar form rather than a simple text notification.
[0031] However, the effects obtainable through the present invention are not limited to those described above, and other unmentioned technical effects will be clearly understood by those skilled in the art from the description of the invention below. Brief explanation of the drawing
[0033] The following drawings attached to this specification illustrate embodiments of the present invention and serve to further enhance understanding of the technical concept of the present invention together with the detailed description of the invention provided below; therefore, the present invention should not be interpreted as being limited only to the matters described in such drawings. FIG. 1 is a block diagram showing the overall configuration of a remote monitoring system using an attitude identification code according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a remote monitoring method using a posture identification code according to an embodiment of the present invention. Figure 3 is a flowchart illustrating the detailed steps of the posture classification and code generation step (S230) of Figure 2. FIG. 4 is a flowchart illustrating the detailed steps of the 3D model rendering step (S250) of FIG. 2. FIG. 5 is a block diagram showing a computer system for implementing a method according to an embodiment of the present invention. Specific details for implementing the invention
[0034] An embodiment of the present invention will be described in detail below with reference to the attached drawings. Prior to this, terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings. Instead, based on the principle that the inventor can appropriately define the concept of a term to best describe their invention, they should be interpreted in a meaning and concept consistent with the technical spirit of the present invention. Therefore, the embodiments described in this specification and the configurations illustrated in the drawings are merely partial embodiments of the present invention and do not represent all of the technical spirit of the present invention. It should be understood that various equivalents and modifications capable of replacing them may exist at the time of filing this application.
[0035] Additionally, as used herein, “comprise, include” and / or “comprising, including” specify the presence of the mentioned features, numbers, steps, actions, parts, elements, and / or groups thereof, and do not exclude the presence or addition of one or more other features, numbers, actions, parts, elements, and / or groups.
[0036] Additionally, to aid in understanding the invention, the attached drawings are not drawn to actual scale, and the dimensions of some components may be exaggerated. Furthermore, the same reference numerals may be assigned to identical components in different embodiments.
[0037] Although terms such as "first," "second," etc., are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are used merely to distinguish one component from another, and unless specifically stated otherwise, the first component may also be the second component.
[0038] Throughout the specification, unless specifically stated otherwise, each component may be singular or plural.
[0039] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Additionally, the use of “may” when describing embodiments of the invention relates to “one or more embodiments of the invention.” Expressions such as “one or more” and “one or more” preceding a list of elements modify the entire list of elements and do not modify individual elements of the list.
[0040] Throughout the specification, "A and / or B" means A, B, or A and B unless specifically stated otherwise, and "C to D" means C or more and D or less, unless specifically stated otherwise.
[0041] When syntax such as "at least one of A, B, and C", "at least one of A, B, or C", "at least one selected from the group of A, B, and C", or "at least one selected from A, B, and C" is used to specify a list of elements A, B, and C, the syntax can refer to any suitable combination.
[0042] The term "use" may be considered synonymous with the term "utilize." As used herein, terms such as "substantially," "about," and similar terms are used as terms of approximation rather than degree, and are intended to account for the inherent variation of a measured or calculated value that would be recognized by a person skilled in the art.
[0043] The terms used in this specification are intended to describe embodiments of the invention and are not intended to limit the invention.
[0045] Below, the background of the proposed invention is explained, and embodiments of the invention are described.
[0046] With the recent acceleration of the transition to an aging society, there is a continuously emerging need for remote monitoring systems to prevent lonely deaths among the elderly living alone and to respond quickly to emergencies such as falls. Conventional remote monitoring systems have primarily adopted video recording methods using cameras (CCTV); however, this video-based monitoring method has limitations in that it causes serious privacy infringement issues because it involves a structure where the monitored individual's daily life is directly captured, stored, and transmitted in video form. In particular, when cameras must be installed even in private spaces such as bedrooms or bathrooms, the problem becomes even more serious, as it increases the emotional resistance of the monitored individual and the risk of actual privacy exposure. To overcome these privacy infringement issues, there is a growing number of attempts to indirectly determine the human body's posture by utilizing millimeter-wave (mmWave) radar sensors. These sensors use radio waves to detect only signal reflection characteristics rather than the subject's shape or skeletal structure, instead of vision sensors that directly acquire video information.
[0047] mmWave radar sensors are attracting attention as an alternative monitoring method with low privacy concerns, as they can acquire location and motion information by analyzing radio signals reflected from subjects without directly capturing images of them. However, the data generated by radar sensors takes the form of large-capacity raw data, such as point clouds or micro-Doppler signatures. If this method involves directly streaming unprocessed raw data to a central server or a user's smartphone and then performing computational processing for attitude estimation on that central device, there is a problem of excessive network bandwidth consumption because a massive amount of data must be continuously transmitted over the network. Additionally, if multiple radar devices are simultaneously connected to and operated on the same server, the processing load may be concentrated on the server, potentially leading to server overload. Furthermore, there is a limitation in that the time required for data transmission and central processing may degrade real-time detection and response to emergencies such as falls.
[0048] Therefore, there is a need for a new type of remote monitoring technology that can efficiently recognize attitude without infringing on the subject's privacy or directly transmitting large volumes of raw data collected from radar sensors externally, enables simultaneous operation of multiple devices while minimizing network load, and allows for real-time response to emergency situations.
[0050] FIG. 1 is a block diagram showing the overall configuration of a remote monitoring system using an attitude identification code according to one embodiment of the present invention.
[0051] A remote monitoring system according to one embodiment of the present invention may be configured to include a radar sensor module (100) functioning as an edge device, a server (200) that receives and relays data from the radar sensor module (100), and a user terminal (300) that receives data through the server (200). According to an embodiment of the present invention, unlike conventional remote monitoring systems, there is a technical feature in that heavy data computation, i.e., deep learning inference computation, is performed on the radar sensor module (100) itself rather than on the server (200). This configuration is fundamentally distinct from the conventional method in which large volumes of raw data generated from the radar are directly streamed to a server or smartphone to perform computation processing at a central location.
[0052] The radar sensor module (100) is an edge device that transmits and receives millimeter wave (mmWave) signals toward a human body located indoors, i.e., a monitoring subject (hereinafter referred to as the target), and processes the received signals to generate a code for identifying the attitude of the target. It may be configured to include a mmWave transceiver (110), a signal processing unit (120), a deep learning model storage unit (130), an attitude classification code generation unit (140), and a communication unit (150).
[0053] The mmWave transceiver (110) performs the function of transmitting an electromagnetic wave signal in the millimeter wave band toward an indoor space and receiving a signal reflected back from various objects in the indoor space, including a target. Unlike conventional vision sensors or cameras (CCTV), the mmWave transceiver (110) does not acquire the target's appearance or face as direct image information, but detects only the reflection characteristics of the radio waves, thereby enabling the acquisition of location and motion information without infringing upon the target's privacy. The signal received through the mmWave transceiver (110) is transmitted to the signal processing unit (120).
[0054] The signal processing unit (120) can function as a digital signal processing unit (DSP) that processes the received signal received from the mmWave transceiver (110) to generate three-dimensional point cloud data representing the spatial location and dynamic characteristics of the target. The signal processing unit (120) can generate point cloud data, which is a set of points containing coordinates in space where the target is located and velocity information according to the movement of the target, by performing a series of signal processing operations to extract range, Doppler, and angle information for the received signal. The point cloud data generated by the signal processing unit (120) is itself large-capacity raw data, and this raw data is not directly transmitted to the outside of the radar sensor module (100), i.e., to a server (200) or a mobile terminal (300), but is subsequently processed within the radar sensor module (100), thereby reducing network traffic.
[0055] The deep learning model storage unit (130) stores weight data of a deep learning model that has been trained in advance to classify the posture of a target, and a posture database subdivided into more than 100 categories. In the deep learning model storage unit (130), millimeter-wave radar reflection characteristics, such as point cloud distribution and Doppler velocity change amounts corresponding to various postures of the human body, such as walking, running, sitting upright, sitting at an angle, lying on one's back, lying on one's side, or falling, can be stored in a form subdivided into more than 100 classes through a prior training process. Additionally, the deep learning model storage unit (130) can store a unique identification code mapped one-to-one to each subdivided posture, such as a code system ranging from 001 to 100 or more. The deep learning model stored in the deep learning model storage unit (130) may include, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), or a classification model in a lightweight form suitable for real-time computation on an edge device.
[0056] The posture classification code generation unit (140) functions as a microcontroller (MCU) of the radar sensor module (100) and passes the point cloud data received from the signal processing unit (120) through a pre-trained deep learning model stored in the deep learning model storage unit (130) to classify the posture currently taken by the target into one of more than 100 designated postures and extract a unique identification code corresponding to the classified posture. Since the deep learning inference operation performed by the posture classification code generation unit (140) is performed at the edge, it is possible to generate and transmit only the posture identification code without the need to transmit raw data externally. The posture classification code generation unit (140) can improve classification accuracy by generating a multidimensional feature vector to extract the spatial arrangement and dynamic state of the human body from 3D point cloud data from various angles and utilizing this as input data for the deep learning model.
[0057] The communication unit (150) is a component that performs the function of transmitting a unique identification code calculated by the attitude classification code generation unit (140) to the server (200), and can be implemented as a wireless communication module including, for example, a Wi-Fi module. The communication unit (150) packetizes and transmits only minimal metadata, such as the unique identification code calculated by the attitude classification code generation unit (140) and the location coordinates of the target, and does not transmit raw radar data, such as point cloud data generated by the signal processing unit (120), to the outside. By transmitting only identification codes and metadata at the level of a few bytes in this way, the communication unit (150) can drastically reduce communication traffic costs and enables real-time transmission without delay even in an unstable Wi-Fi environment.
[0058] The server (200) receives a unique identification code and metadata from the communication unit (150) of the radar sensor module (100) and performs the function of relaying them to the user terminal (300), and may be implemented in a cloud form. Unlike conventional technology, the server (200) does not perform separate deep learning operations or heavy image processing operations on the received data, but simply performs the role of relaying data, so the processing load of the server (200) can be minimized even if multiple radar sensor modules (100) are connected simultaneously. Accordingly, even if this system is applied simultaneously to tens of thousands of households of elderly people living alone, the cost required for the construction and maintenance of the central server can be significantly reduced.
[0059] The user terminal (300) is a terminal such as a smartphone carried by a user, such as a guardian, and performs the function of receiving a unique identification code transmitted through the server (200) and displaying a posture corresponding to the code in a three-dimensional visual form on the screen, and may be configured to include a code receiving unit (310), a posture 3D model matching unit (320), and a display unit (330).
[0060] The code receiving unit (310) receives metadata, such as a unique identification code and the location coordinates of a target, from the server (200) via a wireless network. The code receiving unit (310) may be implemented as part of an application installed on a user terminal (300).
[0061] The posture 3D model matching unit (320) performs the function of extracting and loading a 3D graphic model that matches the unique identification code received by the code receiving unit (310) by referring to a database of 3D graphic models (assets) that correspond one-to-one to each of 100 or more unique identification codes stored in advance inside the user terminal (300). For example, when the code receiving unit (310) receives a unique identification code "085", the posture 3D model matching unit (320) can load a 3D graphic model from the database that represents a state where the target is lying on its right side, which is mapped to the corresponding code.
[0062] The display unit (330) renders a 3D graphic model loaded by the posture 3D model matching unit (320) onto an indoor house structure diagram displayed on the screen of the user terminal (300), thereby performing the function of visually displaying the current posture of the target to the user, i.e., the guardian. The system according to one embodiment of the present invention displays the status of the target in the form of a high-resolution 3D avatar rather than a simple text-based notification, thereby enabling the user to perceive the status of the subject very intuitively and accurately.
[0063] A remote monitoring system according to one embodiment of the present invention completes processing up to attitude classification and unique identification code generation through deep learning inference operations within a radar sensor module (100), transmits only the identification code excluding raw data to a user terminal (300) via a server (200), and the user terminal (300) visualizes the received identification code by matching it with a stored 3D model database, thereby implementing an edge computing-based system structure that enables accurate and intuitive remote monitoring without infringing on privacy.
[0065] FIG. 2 is a flowchart illustrating a remote monitoring method using a posture identification code according to an embodiment of the present invention.
[0066] A remote monitoring method according to one embodiment of the present invention may be configured to include a radar data collection step (S210), a point cloud generation step (S220), an attitude classification and code generation step (S230), a code transmission step (S240), and a 3D model rendering step (S250). The method according to the present embodiment illustrated in FIG. 2 is a series of processing steps performed in real time in an actual operating environment, and can be performed sequentially by the radar sensor module (100), server (200), and user terminal (300) described above with reference to FIG. 1.
[0067] The radar data collection step (S210) is a step of scanning an indoor space by transmitting a millimeter wave signal toward a target located indoors through the mmWave transceiver (110) of the radar sensor module (100) and receiving a signal reflected back from an indoor object including the target. Unlike conventional video recording methods using cameras (CCTV), this step has technical significance in that it does not acquire the external appearance or shape of the target as direct image information, but only transmits and receives radio signals, thereby enabling the acquisition of information regarding the location and operation of the target without infringing upon the privacy of the monitored person.
[0068] The point cloud generation step (S220) is a step in which the signal processing unit (120) receives the signal received by the mmWave transceiver (110) through the radar data collection step (S210) and performs a series of signal processing operations to extract information on range, speed (Doppler), and angle, thereby generating three-dimensional point cloud data representing the spatial position and dynamic characteristics of the target. The point cloud data generated through this step is used as input data for the subsequent attitude classification and code generation step (S230). Since the data itself corresponds to large-capacity raw data, the present invention adopts a structure in which such data is not directly transmitted outside the radar sensor module (100) but is subsequently processed within the radar sensor module (100).
[0069] The attitude classification and code generation step (S230) is a step in which the attitude classification and code generation unit (140), that is, the microcontroller (MCU) of the radar sensor module (100), passes the point cloud data generated through the point cloud generation step (S220) through a pre-trained deep learning model stored in the deep learning model storage unit (130), thereby classifying the current attitude of the target into one of 100 or more designated attitudes and calculating a unique identification code corresponding to the classified attitude. The deep learning inference operation performed in this step is performed at the radar sensor module (100) itself, that is, at the edge, rather than at the server (200). The attitude classification and code generation step (S230) according to an embodiment of the present invention includes detailed steps for generating a multidimensional feature vector to extract the spatial arrangement and dynamic state of the human body from point cloud data from various angles, and combining additional features to model the static attitude and dynamic attitude transition process.
[0070] The code transmission step (S240) is a step in which the communication unit (150) packets only the unique identification code calculated through the attitude classification and code generation step (S230) and minimal metadata such as the target's location coordinates (X, Y), and transmits them to the user terminal (300) via the server (200). In this step, since only the identification code and minimal metadata are transmitted, excluding the raw point cloud data generated through the point cloud generation step (S220), communication traffic costs can be drastically reduced compared to conventional technology, and real-time transmission without delay is possible even in an unstable Wi-Fi environment. Meanwhile, the server (200) does not perform separate deep learning operations on the unique identification code and metadata received from the radar sensor module (100), but only performs the role of relaying them to the user terminal (300), so server overload does not occur even if multiple radar sensor modules (100) are connected simultaneously.
[0071] The 3D model rendering step (S250) is a step in which the user terminal (300) matches the unique identification code received through the code transmission step (S240) with a 3D graphic model database corresponding to 100 pose identification codes stored internally, and renders and displays the matched 3D graphic model in an indoor virtual space on the screen. This step may include the process of the code receiving unit (310) of the user terminal (300) receiving the unique identification code, the process of the pose 3D model matching unit (320) loading a graphic asset mapped to the corresponding code, and the process of the display unit (330) rendering the corresponding pose on a house structure diagram and displaying it to the user, i.e., the guardian. Through this step, the user can perceive the status of the subject very intuitively and accurately in the form of a high-resolution 3D avatar, rather than a simple text-based notification.
[0072] A remote monitoring method according to one embodiment of the present invention enables intuitive remote monitoring while minimizing network load without infringing on privacy by sequentially performing a series of steps starting from the collection of radar data and the generation of a point cloud, attitude classification and identification code generation at the edge, lightweight transmission of the identification code excluding raw data, and three-dimensional visualization at a user terminal.
[0074] Figure 3 is a flowchart illustrating the detailed steps of the posture classification and code generation step (S230) of Figure 2.
[0075] The attitude classification and code generation step (S230) may be configured to include a model reference step (S231), a feature vector generation step (S232), a transition feature combination step (S233), a feature vector input and matching step (S234), and an identification code extraction step (S235). The detailed steps illustrated in FIG. 3 may be performed by the attitude classification code generation unit (140), i.e., the microcontroller (MCU) of the radar sensor module (100).
[0076] The model reference step (S231) is a step in which the posture classification code generation unit (140) refers to pre-trained deep learning model and weight data stored in the deep learning model storage unit (130) to prepare for inference. The deep learning model storage unit (130) stores the weights of a model that is the result of collecting mmWave radar reflection characteristics according to various postures of the human body, such as walking, running, sitting upright, sitting at an angle, lying on the front, lying on the side, falling, etc., subdividing them into more than 100 classes, and pre-training them with a deep learning model such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a lightweight classification model, and a unique identification code (e.g., 001 to 100 or more) mapped one-to-one to each posture. In this step, by loading the weights that have been learned in advance and stored in the storage unit, immediate inference operations can be performed within the radar sensor module (100) without a separate learning process.
[0077] The feature vector generation step (S232) is a step in which the posture classification code generation unit (140) generates a multidimensional feature vector to extract the spatial arrangement and dynamic state of the human body from three-dimensional point cloud data. The feature vector generated in this step can be constructed by combining spatial geometric features, Doppler kinematic features, and signal energy features.
[0078] Spatial geometric features may include elevation statistics representing the spatial distribution of the point cloud, horizontal and vertical spatial dispersion, and the aspect ratio of the target bounding box. The elevation statistics of the target indicate the vertical distribution characteristics of the points constituting the target and can serve as a valid indicator for distinguishing, for example, between a standing posture and a lying posture. Horizontal and vertical spatial dispersion are statistical indicators representing how widely or narrowly the points are distributed in space, and the aspect ratio of the target bounding box is the ratio of the width to the height of the minimum bounding box surrounding the target, which can reflect the overall shape characteristics according to the target's posture.
[0079] Doppler kinematic features may include the average velocity and velocity variance of target points. The average velocity of target points serves as an indicator of the overall degree of movement of the target and can be used to distinguish between postures that are near rest and those involving dynamic movement, while the velocity variance represents the velocity deviation between each point constituting the target, which can reflect the relative differences in movement of different body parts.
[0080] Signal energy features may include the sum of frame-by-frame signal-to-noise ratios (SNR) to reflect the radar cross-section (RCS) characteristics of the target. By reflecting differences in radio wave reflection characteristics based on the target's body surface area or attitude, signal energy features can be utilized as an auxiliary indicator to complement the accuracy of attitude classification.
[0081] According to an embodiment of the present invention, by combining spatial geometric features, Doppler kinematic features, and signal energy features, a multidimensional feature vector can be generated that can capture even minute differences between similar poses that are difficult to distinguish by a single feature alone.
[0082] The transition feature combination step (S233) is a step in which the posture classification code generation unit (140) additionally combines the rate of change in elevation of the target center point between consecutive time frames with the feature vector generated through the feature vector generation step (S232) in order to model the transition process between static posture and dynamic posture. This step enables the capture of not only static posture at a single point in time but also dynamic posture change processes, such as transitioning from a standing state to a sitting state, or from a sitting state to a falling state, thereby allowing for more accurate identification of emergency situations, particularly such as falls. The rate of change in elevation of the target center point is an indicator showing how the center height of the target changes over a number of consecutive time frames, and can be effectively utilized for distinguishing between static and dynamic postures and for identifying the transition point.
[0083] The feature vector input and matching step (S234) is a step of optimally matching the current attitude of a target to one of more than 100 specified attitudes by providing the multidimensional feature vector finally constructed through the transition feature combination step (S233) as input data to the deep learning model referenced in the model reference step (S231) and performing inference operations through the deep learning model. This step is performed through a deep learning model that runs in a lightweight form on the attitude classification code generation unit (140), that is, the microcontroller (MCU) of the radar sensor module (100), thereby enabling near real-time attitude inference at the edge without a separate high-performance server.
[0084] The identification code extraction step (S235) is a step for extracting a unique identification code corresponding to the posture matched through the feature vector input and matching step (S234). For example, if the posture of the target is matched as lying on its right side, the unique identification code 085 corresponding to that posture can be extracted in this step. The unique identification code extracted through this step can be packetized with minimal metadata, such as the location coordinates of the target, and transmitted to the server (200) through the code transmission step (S240) of FIG. 2, while excluding raw point cloud data.
[0085] The detailed steps of the posture classification and code generation steps according to one embodiment of the present invention extract multidimensional features in terms of spatial, kinematic, and signal energy from point cloud data and generate a multidimensional feature vector that combines temporal transition characteristics, thereby enabling precise classification of dynamic posture changes beyond simple static posture recognition and making it possible to generate accurate unique identification codes for postures subdivided into more than 100 types.
[0087] According to another embodiment of the present invention, the attitude classification and code generation step (S230) illustrated in FIG. 3 may additionally perform the following specific computational processing steps so that real-time inference is possible even under the limited computational resources and memory capacity of the microcontroller (MCU) mounted on the radar sensor module (100).
[0088] Edge Inference Optimization via Integer Quantization and Fixed-Point Operation
[0089] According to another embodiment of the present invention, the model reference step (S231) may include the process of storing weight data of a pre-trained deep learning model stored in a deep learning model storage unit (130) by quantizing it from a 32-bit floating-point (FP32) format to an 8-bit integer (INT8) format. Specifically, in the pre-training step, the maximum and minimum values of the weight values for each layer of the deep learning model are calculated to determine the quantization scale factor and zero-point correction value for each layer, and the floating-point weight W can be converted into an integer weight W_int according to the following [Equation 1].
[0090] [Mathematical Formula 1]
[0091] W_int = round(W / scale) + zero_point
[0092] The quantized integer weights can be stored in the deep learning model storage unit (130) of the radar sensor module (100), thereby reducing the memory occupancy capacity to about one-fourth of that of floating-point operations.
[0093] In the feature vector input and matching step (S234), the same quantization scale factor is applied to the multidimensional feature vector generated through the feature vector generation step (S232) and the transition feature combination step (S233) to convert it into an integer input vector, and then the microcontroller included in the pose classification code generation unit (140) can perform inference by performing fixed-point matrix multiplication and activation function operations composed only of integer operations. At this time, for the activation function operation, instead of using a sigmoid or softmax function that requires floating-point exponential operations, a piecewise linear approximation function that can perform approximate calculations using only integer comparison and bit-shift operations is applied, thereby enabling inference operations even on a low-cost microcontroller that is not equipped with a floating-point operation unit (FPU).
[0094] Such integer quantization and fixed-point operation processing is not related to the abstract attitude classification itself, but rather is a means of quantitatively reducing memory usage and operation latency by reconstructing the conversion method of weight data and the operation step under the hardware constraints unique to edge devices, such as the absence of a floating-point operation unit, and thereby the radar sensor module (100) can independently perform real-time attitude inference without the support of a cloud server.
[0095] Reducing inference latency through an early termination cascade structure
[0096] According to another embodiment of the present invention, the feature vector input and matching step (S234) may include a process in which a pre-trained deep learning model stored in a deep learning model storage unit (130) is configured into a multi-stage cascade structure of a first inference layer with a relatively low amount of computation and a second inference layer with a higher amount of computation and precision than the first inference layer, and a detail classification code generation unit (140) compares a confidence score for a classification result calculated through the first inference layer with a preset threshold.
[0097] Specifically, the pose classification code generation unit (140) inputs the feature vector generated through the transition feature combination step (S233) into the first inference layer to calculate a probability value for each of 100 or more specified poses, and if the maximum value among the calculated probability values is greater than or equal to a preset threshold, e.g., 0.9, the operation for the second inference layer is omitted and the pose corresponding to the maximum value is immediately determined as the final classification result, thereby performing an early-exit process. On the other hand, if the maximum value of the calculated probability value is less than the threshold, additional operation for the second inference layer is performed using the output value of the first inference layer as input, thereby allowing for more precise feature extraction and determining the final classification result.
[0098] This early termination cascade structure reduces the average inference time by rapidly completing classification using only the first inference layer for relatively easy-to-identify postures, such as a target standing or lying upright, while selectively allocating additional computational resources only to difficult-to-identify postures, such as transitional postures immediately after a fall or cases where the distinction between similar postures is ambiguous, thereby enabling classification accuracy to be maintained while minimizing the overall average inference latency even within the limited computational throughput of a microcontroller. In particular, in emergency situations requiring rapid judgment, such as a fall, immediate identification code generation is possible for postures with clear reliability, further enhancing real-time responsiveness to emergency situations.
[0099] Adaptive frame sampling control considering wireless bandwidth constraints
[0100] According to another embodiment of the present invention, the radar data collection step (S210) and the attitude classification and code generation step (S230) may further include a process of dynamically adjusting the signal collection cycle of the mmWave transceiver (110), i.e., the frame sampling rate, by monitoring the wireless network transmission bandwidth of the communication unit (150) and the current computational load status of the attitude classification code generation unit (140).
[0101] Specifically, if the attitude classification code generation unit (140) determines that the rate of change in the target center point elevation between consecutive time frames calculated in the transition feature combination step (S233) is maintained at a level below a preset range of variation, it can reduce computational load and power consumption by operating the frame sampling rate of the mmWave transceiver (110) at a first rate, e.g., 5 frames per second. On the other hand, if the rate of change in the target center point elevation is determined to be a dynamic rate where an attitude transition is detected by exceeding the preset range of variation, the frame sampling rate can be increased to a second rate higher than the first rate, e.g., 20 frames per second or more, thereby enabling more precise and frequent inference for periods where rapid attitude changes occur, such as falls.
[0102] Additionally, in the code transmission step (S240), the communication unit (150) receives information on the current available bandwidth of the wireless network, and if it is determined that the available bandwidth has decreased below a preset threshold, it can perform differential transmission control by delaying the transmission cycle of metadata for static sections where the same unique identification code is generated continuously for more than a preset number of times, while shortening the transmission cycle and transmitting it preferentially for dynamic sections where an attitude transition is detected.
[0103] Such adaptive frame sampling control and differential transmission control minimize power and bandwidth consumption during normal operation and maximize responsiveness in suspected emergency situations by dynamically readjusting the temporal resolution of data collection and transmission based on the degree of target state change, under specific hardware and communication environmental constraints such as wireless network bandwidth limitations and the limited computational processing capacity of microcontrollers. This enables the simultaneous achievement of the potentially conflicting goals of reducing communication traffic and providing real-time emergency response.
[0105] FIG. 4 is a flowchart illustrating the detailed steps of the 3D model rendering step (S250) of FIG. 2.
[0106] As illustrated in FIG. 4, the 3D model rendering step (S250) may be configured to include a code and coordinate reception step (S251), a 3D model matching step (S252), a coordinate mapping and orientation setting step (S253), and a rendering and UI display step (S254). The detailed steps illustrated in FIG. 4 may be performed by a user terminal (300).
[0107] The code and coordinate reception step (S251) is a step in which the code receiving unit (310) of the user terminal (300) receives metadata, such as a unique identification code and the location coordinates of a target, transmitted from the server (200) via a wireless network. The data received by the code receiving unit (310) is configured in the form of a packet of several bytes in size calculated by the radar sensor module (100), and does not include raw point cloud data. The unique identification code received in this step may be a code such as 085, for example, corresponding to the state where the target is lying on its right side, and the metadata received together may include coordinate information indicating the location of the target.
[0108] The 3D model matching step (S252) is a step in which the posture 3D model matching unit (320) of the user terminal (300) refers to a posture-specific 3D visualization model database that corresponds one-to-one to each of 100 or more unique identification codes stored in advance within the user terminal (300), and identifies and loads a 3D graphic model (asset) that matches the unique identification code received through the code and coordinate reception step (S251). Since the application of the user terminal (300) has a 3D graphic model corresponding to each of the 100 posture identification codes in advance, it can immediately extract the corresponding 3D model using only the received identification code without separate external data communication or additional computation. For example, if the received unique identification code is 085, the posture 3D model matching unit (320) can load a graphic asset from the database that represents a state where the target is lying on its right side, which is mapped one-to-one to the corresponding code.
[0109] The coordinate mapping and orientation setting step (S253) is a step of mapping a 3D graphic model loaded through the 3D model matching step (S252) to a virtual indoor space coordinate system displayed on the screen of a user terminal (300), based on the spatial location coordinates of the target, namely X, Y, and Z coordinates, received together through the code and coordinate reception step (S251), and setting the orientation of the 3D avatar model. Through this step, it becomes possible to visualize three-dimensional images that not only visualize the type of pose but also reflect the location within the indoor space where the target assumes the pose. The location coordinates of the target may be included in the metadata transmitted along with a unique identification code through the code transmission step (S240) of FIG. 2, based on spatial coordinate information calculated by the signal processing unit (120) of the radar sensor module (100) from the distance, velocity, and angle information of the mmWave signal. In this step, this coordinate information is utilized to ensure that the 3D avatar model is positioned at an accurate location and orientation within the indoor virtual space.
[0110] The rendering and UI display step (S254) is a step in which the display unit (330) of the user terminal (300) renders a 3D avatar model, whose position and direction are set through the coordinate mapping and direction setting step (S253), onto a house structure diagram displayed on the screen of the user terminal (300), and displays it in the form of a user interface (UI) capable of real-time monitoring. Through this step, the user, i.e., the guardian, can intuitively and accurately recognize the target's current posture and location through visual information implemented in the form of a high-resolution 3D avatar, rather than a simple text-based notification message. Since this step is configured as a real-time monitoring UI, as the series of processes from the radar data collection step (S210) to the code transmission step (S240) of FIG. 2 are repeatedly performed, the 3D avatar model on the screen can also be updated and displayed whenever a change in the target's posture occurs, thereby enabling the guardian to quickly recognize the situation and respond in the event of an emergency such as a fall.
[0111] The detailed steps of the 3D model rendering step according to one embodiment of the present invention match a received unique identification code with a stored 3D model database and, by reflecting the spatial coordinate information of the target together, render an avatar in an accurate position and orientation in a virtual indoor space, thereby providing the user with an intuitive and spatially contextual real-time monitoring environment.
[0113] FIG. 5 is a block diagram showing a computer system for implementing a method according to an embodiment of the present invention.
[0114] Referring to FIG. 5, a computer system (1300) may include at least one of a processor (1310), memory (1330), an input interface device (1350), an output interface device (1360), and a storage device (1340) that communicate via a bus (1370). The computer system (1300) may also include a communication device (1320) coupled to a network. The processor (1310) may be a central processing unit (CPU) or a semiconductor device that executes instructions stored in memory (1330) or storage device (1340). Memory (1330) and storage device (1340) may include various forms of volatile or non-volatile storage media. For example, memory may include read-only memory (ROM) and random access memory (RAM). In the embodiments of this description, memory may be located inside or outside the processor, and memory may be connected to the processor through various known means. Memory is a volatile or non-volatile storage medium of various forms, and for example, memory may include read-only memory (ROM) or random access memory (RAM).
[0115] Accordingly, embodiments of the present invention may be implemented as a method implemented on a computer or as a non-transient computer-readable medium storing computer-executable instructions. In one embodiment, when executed by a processor, the computer-readable instructions may perform a method according to at least one aspect of the present description.
[0116] The communication device (1320) can transmit or receive wired or wireless signals.
[0117] In addition, the method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium.
[0118] The above computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for embodiments of the present invention, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory, etc. The program instructions may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer through an interpreter, etc.
[0119] Although the present invention has been described above with reference to limited embodiments and drawings, the present invention is not limited thereto, and it is obvious that various modifications and variations are possible within the scope of the technical spirit of the present invention and the equivalent scope of the claims described below by those skilled in the art to which the present invention belongs.
Claims
Claim 1 A remote monitoring system using attitude identification codes, comprising: a radar sensor module that transmits and receives millimeter wave (mmWave) signals toward a target located indoors, processes the received signals to calculate a unique identification code corresponding to an attitude, and transmits metadata including the unique identification code and the location coordinates of the target, excluding raw data for the target; a server that receives and relays the unique identification code and metadata from the radar sensor module; and a user terminal that receives the unique identification code and metadata through the server, extracts a 3D graphic model corresponding to the received unique identification code by referring to a 3D visualization model database for each attitude, and renders and displays the 3D graphic model in an indoor virtual space on a screen, wherein the radar sensor module constructs deep learning model input data by combining the rate of change of the target center point elevation between consecutive time frames to model static attitude and dynamic attitude transition processes. Claim 2 A remote monitoring system using an attitude identification code according to claim 1, wherein the radar sensor module comprises: a mmWave transceiver that transmits an electromagnetic wave signal and receives a signal reflected back from the target; and a signal processing unit that performs signal processing operations to extract distance, speed, and angle information on the signal received by the mmWave transceiver to generate three-dimensional point cloud data. Claim 3 A remote monitoring system using an attitude identification code, wherein the radar sensor module comprises: a deep learning model storage unit that stores weight data of a pre-trained deep learning model and an attitude database; and an attitude classification code generation unit that inputs the 3D point cloud data into a pre-trained deep learning model stored in the deep learning model storage unit to classify the attitude of a target and calculate the unique identification code. Claim 4 In paragraph 3, the attitude classification code generation unit is implemented through a microcontroller equipped in the radar sensor module, and performs inference operations by the deep learning model within the radar sensor module, a remote monitoring system using an attitude identification code. Claim 5 A remote monitoring system using an attitude identification code, wherein the radar sensor module comprises a communication unit that packetizes the unique identification code and metadata and transmits them externally via a wireless network. Claim 6 A remote monitoring system using a posture identification code according to claim 1, wherein the user terminal comprises: a code receiving unit that receives the unique identification code and metadata from the server; and a posture 3D model matching unit that loads a 3D graphic model matching the unique identification code received by the code receiving unit by referring to the posture-specific 3D visualization model database. Claim 7 A remote monitoring system using a posture identification code according to claim 6, wherein the user terminal includes a display unit that maps a 3D graphic model loaded by the posture 3D model matching unit to an indoor virtual space coordinate system and sets the direction to render it on a screen based on the location coordinates of a target included in the metadata received by the code receiving unit. Claim 8 In a remote monitoring system using a posture identification code, the system comprises: (a) a step in which a radar sensor module transmits and receives a millimeter wave signal toward a target located indoors, processes the received signal to calculate a unique identification code corresponding to a posture classified as one of the designated postures; (b) a step in which the radar sensor module transmits metadata including the unique identification code and the location coordinates of the target to the outside, excluding raw data regarding the target; (c) a step in which a server receives the unique identification code and metadata transmitted in step (b) and relays them to a user terminal; and (d) the user terminal extracts a 3D graphic model by matching a unique identification code received through step (c) with a 3D visualization model database for each attitude, and renders and displays the 3D graphic model in an indoor virtual space on a screen, wherein step (a) generates 3D point cloud data by performing a signal processing operation to extract distance, speed, and angle information for the received signal, and generates a multidimensional feature vector from the 3D point cloud data, and the multidimensional feature vector is composed of spatial geometric features including elevation statistics, horizontal and vertical spatial dispersion, and the aspect ratio of a target bounding box, Doppler kinematic features including the average speed and speed dispersion of target points, and signal energy features including the sum of the signal-to-noise ratio per frame, a remote monitoring method using an attitude identification code. Claim 9 delete Claim 10 delete Claim 11 A remote monitoring system using attitude identification codes, performed by a remote monitoring system using attitude identification codes, comprising: (a) a step in which a radar sensor module transmits and receives a millimeter wave signal toward a target located indoors and processes the received signal to calculate a unique identification code corresponding to a posture classified as one of the designated postures; (b) a step in which the radar sensor module transmits metadata including the unique identification code and the location coordinates of the target to the outside, excluding raw data for the target; (c) a step in which a server receives the unique identification code and metadata transmitted in step (b) and relays them to a user terminal; and (d) a step in which the user terminal matches the unique identification code received through step (c) with a 3D visualization model database for each posture to extract a 3D graphic model and renders and displays the 3D graphic model in an indoor virtual space on a screen, wherein step (a) further combines the rate of change of the target center point elevation between consecutive time frames to model the static posture and dynamic posture transition process to constitute input data for a deep learning model. Claim 12 In claim 8, the above step (a) is a remote monitoring method using an attitude identification code, wherein a microcontroller equipped in the radar sensor module performs inference operations based on a pre-trained deep learning model within the radar sensor module. Claim 13 In claim 8, the above step (d) is a remote monitoring method using an attitude identification code, which maps the 3D graphic model to a coordinate system of an indoor virtual space and sets the orientation based on the position coordinates of the target included in the metadata. Claim 14 In claim 8, the above step (d) is a remote monitoring method using a posture identification code, wherein the above step (a) to the above step (c) is updated based on a unique identification code that is updated as steps (a) to (c) are repeatedly performed, and the three-dimensional graphic model on the screen is updated and displayed as a real-time monitoring interface. Claim 15 A mmWave transceiver that transmits electromagnetic wave signals and receives signals reflected back from a target; a signal processing unit that generates 3D point cloud data by performing signal processing operations to extract distance, velocity, and angle information from the signals received by the mmWave transceiver; a deep learning model storage unit that stores weight data of a pre-trained deep learning model and a pose database; and a pose classification code generation unit that inputs the 3D point cloud data into a pre-trained deep learning model stored in the deep learning model storage unit to classify the pose of the target and calculate a unique identification code corresponding to the classified pose. A radar sensor module comprising a communication unit that packetizes metadata including the unique identification code and the location coordinates of the target, excluding raw data for the target, and transmits it externally via a wireless network, wherein the attitude classification code generation unit generates a multidimensional feature vector from the 3D point cloud data, and the multidimensional feature vector is configured by combining spatial geometric features including elevation statistics, horizontal and vertical spatial dispersion, and the aspect ratio of the target bounding box, Doppler kinematic features including the average velocity and velocity dispersion of target points, and signal energy features including the sum of the signal-to-noise ratio per frame. Claim 16 In claim 15, the attitude classification code generation unit is implemented through a microcontroller and performs inference operations based on the pre-trained deep learning model, a radar sensor module. Claim 17 delete Claim 18 In claim 15, the attitude classification code generation unit is a radar sensor module that additionally combines the rate of change of target center point elevation between consecutive time frames with the multidimensional feature vector to model static attitude and dynamic attitude transition processes.
Citation Information
Patent Citations
Apparatus and method for transmitting simplified motion information from which background image is excluded and displaying information using avatar
KR1020140022670A
3D positioning device capable of multi-unit identification using 5G radar and artificial intelligence
KR1020200094598A
Non-face-to-face non-contact fall detection system based on skeleton vector and method therefor
KR102413893B1
Server for user behavior analysis in store system and method therof
KR102525234B1
Intelligent video surveillance system using skeleton detection from pose estimation model for objects
KR102772343B1
