Human body recognition method and device based on radar sensing and RGB camera fusion

Abstract

The present application relates to the field of intelligent terminal perception, and discloses a human body recognition method and device based on radar sensing and RGB camera fusion. The radar sensor analyzes the target micro-Doppler characteristic fingerprint vector and compares it with the pre-stored interference characteristic library, generates a wake-up signal after filtering the known interference source. The processor responds to the wake-up, calculates the predicted coordinates of the target at the exposure moment in combination with the system time delay, maps the predicted coordinates to the light measurement area parameters before starting the camera, converts the target speed into the upper limit parameters of the exposure time and the focus driving value and issues them. The camera collects images according to the parameters, and the processor classifies using a neural network: if it is not a human body, the interference suppression logic is triggered and the current characteristics are written into the interference library, and if it is a human body, the business response is executed. Through radar bottom filtering and camera parameter feedforward configuration, the present application solves the problems of high radar false alarm rate and poor cold start imaging quality of the camera, reduces the system power consumption and improves the recognition accuracy and response speed.

Description

Technical Field

[0001] This invention relates to the field of intelligent terminal sensing technology, specifically to a human body recognition method and device based on the fusion of radar sensing and RGB camera. Background Technology

[0002] With the development of artificial intelligence technology, more and more AI-powered smart agent products based on the Android system are integrating radar sensing functions to monitor the presence of humans. Current technical solutions typically involve the radar sensor outputting a digital signal to the central processing unit when it detects movement of an object within its effective range, indicating the presence of movement.

[0003] However, this single radar sensing solution has technical limitations in practical applications. Radar can only sense changes in movement within space but cannot accurately identify the specific type of moving object. In complex home or office environments, movement from humans, pets, or devices with moving parts (such as oscillating fans or running robot vacuums) will trigger the radar's sensing mechanism. This triggering logic, lacking category recognition capabilities, causes the intelligent agent to frequently misjudge situations where no human is present. This leads to frequent wake-ups of the main processor and incorrect activation of high-power business modules such as voice interaction or screen activation, resulting in wasted system computing resources and increased overall power consumption. Furthermore, although some solutions attempt to incorporate cameras for auxiliary verification, the lack of underlying coordination between sensors often makes it difficult for the system to acquire clear, accurately exposed images in a very short time during cold start capture, thus affecting the success rate of subsequent human recognition algorithms. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a human body recognition method and device based on the fusion of radar sensing and RGB camera. This solves the problems of high false alarm rate and high system power consumption caused by the inability of single radar sensing to distinguish the type of moving target, as well as the problems of slow focusing, inaccurate exposure and motion blur caused by the lack of parameter guidance during cold start of camera capture, which reduce the recognition accuracy.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The first aspect of this invention provides a human body recognition method based on the fusion of radar sensing and RGB camera data. This method aims to solve the problems of high false alarm rate from a single radar sensor and poor cold-start imaging quality of the camera by integrating data fusion and temporal coordination from underlying sensors.

[0007] This method begins with the radar sensor independently undertaking the initial screening task. The radar sensor obtains a range-Doppler map by performing a multidimensional Fourier transform on the detected echo, extracts the target point using a constant false alarm rate (CFAR) algorithm, and calculates the target's range, velocity, and angle data. To distinguish between human and non-human interference, the radar further performs a short-time Fourier transform on the signal sequence of the target range cell to generate a time-frequency distribution map. By extracting energy features from the time-frequency map, it fits the upper and lower envelopes and calculates the mean Doppler bandwidth, envelope period, and standard deviation between the envelopes, constructing a micro-Doppler feature fingerprint vector reflecting the target's motion characteristics. Subsequently, the weighted Euclidean distance between this vector and a reference vector in a pre-stored interference feature library is calculated. The target's nature is then determined using nonlinear mapping and a double-threshold hysteresis comparison logic: a wake-up signal is generated only when the comprehensive decision score indicates that the target is not a known interference source; otherwise, the camera remains off, thus filtering out invalid targets at the lower level.

[0008] After the processor responds to the wake-up signal, this invention employs a feedforward control strategy to optimize the camera's startup parameters. The processor obtains the time difference between the radar interrupt trigger moment and the expected image sensor exposure moment as the total system delay. It then superimposes the original radial distance detected by the radar with the product of the target's radial velocity and the total system delay to calculate the predicted radial distance of the target at the exposure moment. Based on a perspective projection model, this predicted distance and azimuth angle are mapped to predicted pixel coordinates in the image coordinate system, and the area corresponding to these coordinates is designated as the photometric region, achieving accurate photometric measurement of moving targets.

[0009] Meanwhile, to address motion blur, the method determines whether the target speed exceeds a preset threshold. If it does, it calculates a dynamic exposure time limit inversely proportional to the target speed based on the maximum motion blur pixel count threshold, pixel size, and focal length, and writes this limit to the camera's exposure limit register. Furthermore, the processor utilizes an inverse mapping model between distance and lens voice coil motor drive value to directly calculate and set the target drive value for the voice coil motor based on radar ranging results. This completes open-loop focusing configuration before the image sensor outputs the image, eliminating the search process required for contrast-detection or phase-detection autofocus.

[0010] After image acquisition, the processor uses a convolutional neural network containing depthwise separable convolutional layers and an attention mechanism to classify the image. The system executes bidirectional logic based on classification confidence: if the image is confirmed to be a human, a service response is executed; if it is confirmed to be non-human interference, interference suppression logic is triggered, calculating the consistency between the image target depth and radar distance. Within the allowable deviation range, the distance and velocity of the current target are extracted, and a dynamic shielding interval is generated within the radar's range-velocity phase space and sent to the radar sensor. This allows the radar to automatically filter signals within this interval during subsequent monitoring, achieving adaptive anti-interference evolution of the system. Furthermore, the system includes an online calibration mechanism that dynamically updates the baseline parameters of the inverse mapping model based on the deviation between the actual position after camera autofocus convergence and the radar's predicted value.

[0011] A second aspect of the present invention provides a human body recognition device based on the fusion of radar sensing and RGB camera. The device includes a radar sensor, a camera, and a processor.

[0012] The radar sensor is configured to independently perform signal processing and feature analysis, responsible for transmitting detection signals and receiving reflected signals. It integrates micro-Doppler feature extraction logic and interference feature library comparison logic to filter non-human interference and generate wake-up signals at the hardware level. The camera is connected to the processor and configured to respond to the processor's configuration instructions and start-up enable signals, acquiring image data of the monitoring area according to preset parameters.

[0013] The processor, as the core control unit, communicates with both the radar sensor and the camera. Internally, the processor integrates a position prediction module, which responds to wake-up signals and calculates the target's predicted coordinates at the exposure time based on system delay parameters and radar data; a parameter mapping module, which converts the predicted coordinates into metering area parameters and the target velocity into exposure time upper limit parameters and focus drive values, sending these to the camera via a bus feedforward; and a verification feedback module, which runs a neural network algorithm to classify and determine image data. When non-human interference is detected, the module feeds back the current micro-Doppler feature fingerprint vector and corresponding spatial shielding parameters to the radar sensor, achieving closed-loop suppression of the interference source.

[0014] This invention provides a human body recognition method and device based on the fusion of radar sensing and RGB camera. It has the following beneficial effects: 1. This invention performs micro-Doppler feature analysis at the underlying hardware level using radar sensors to extract fingerprint vectors such as the target's mean Doppler bandwidth and envelope period, and compares and filters them with a pre-stored interference feature library. This predictive mechanism overcomes the limitation of traditional radar, which can only detect motion but cannot distinguish target types. It can intercept false triggering signals from non-human interference sources such as fans and curtains before waking up the high-power processor, thereby reducing the frequency of invalid wake-ups and overall standby power consumption.

[0015] 2. This invention utilizes target distance and velocity information detected by radar, combined with system time delay parameters, to calculate the predicted coordinates of the imaging time. Furthermore, it pre-configures the metering area, exposure time limit, and lens focus position before the camera starts. This feedforward control strategy solves the problem of image blurring or abnormal brightness caused by the lag in autofocus and autoexposure convergence during the cold start phase of conventional cameras. It ensures that the first frame meets the recognition requirements of the neural network, thereby improving the clarity and recognition response speed of the AI ​​agent for capturing dynamic human targets.

[0016] 3. This invention constructs a closed-loop mechanism for online calibration and interference suppression of radar parameters based on visual confirmation. When the visual algorithm determines that the target is not human, the system automatically extracts the radar signal features at that moment and updates them to the interference feature library, thereby achieving adaptive shielding against specific environmental interference sources (such as robot vacuum cleaners and pets). At the same time, the precise position of visual recognition is used to reverse correct the spatial mapping parameters between the radar and the camera and the focus model reference value, eliminating system errors caused by device aging or temperature drift, and ensuring the detection stability and anti-interference capability of the device during long-term operation. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the human body recognition device based on the fusion of radar sensing and RGB camera according to the present invention. Figure 2 This is a structural block diagram of the human body recognition device based on the fusion of radar sensing and RGB camera according to the present invention; Figure 3 This is the main flowchart of the human body recognition method based on the fusion of radar sensing and RGB camera of the present invention; Figure 4 This is a schematic diagram of scene decision logic based on micro-Doppler features in a specific application embodiment of the present invention; Figure 5 This diagram illustrates a multi-dimensional performance comparison and ISP feedforward strategy in a specific application embodiment of the present invention.

[0018] Among them, 10 is the radar sensor; 20 is the camera; 30 is the processor; 31 is the position prediction module; 32 is the parameter mapping module; and 33 is the verification feedback module. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See attached document Figure 1 and attached Figure 2 This invention provides a human body recognition device based on the fusion of radar sensing and an RGB camera, comprising: a radar sensor 10, a camera 20, and a processor 30. The radar sensor 10 is communicatively connected to the processor 30, and the camera 20 is communicatively connected to the processor 30.

[0021] Radar sensor 10 is used to detect targets within the monitoring area. Radar sensor 10 employs a frequency-modulated continuous wave (FMCV) system and is configured to transmit detection signals and receive reflected signals. Radar sensor 10 integrates signal processing circuitry to analyze the reflected signals and output target status data. The target status data includes the target's distance, angle, velocity, and spectral characteristics. Radar sensor 10 is configured to send a trigger signal to processor 30 when a target signal is detected.

[0022] Camera 20 is used to acquire image data of the monitored area. Camera 20 includes an image sensor and an image signal processor. Camera 20 is equipped with a control interface for receiving external configuration commands. The configuration commands include metering area commands and exposure time commands. Camera 20 is configured to enter an operating state in response to an enable signal from processor 30 and to transmit the acquired image data to processor 30.

[0023] The processor 30 is used to perform system logic operations and control. The processor 30 includes an application processor and a neural network operation unit. The processor 30 is internally configured with a logic operation submodule, which includes: a position prediction module 31, a parameter mapping module 32, and a verification feedback module 33.

[0024] The position prediction module 31 is connected to the data output terminal of the radar sensor 10. The position prediction module 31 stores system delay parameters. The position prediction module 31 is configured to calculate the predicted coordinate position of the target at the exposure time of the camera 20 based on the velocity value in the target state data and the system delay parameters.

[0025] The parameter mapping module 32 connects the position prediction module 31 to the control interface of the camera 20. The parameter mapping module 32 stores a coordinate calibration matrix. The parameter mapping module 32 is configured to convert the predicted coordinate position into metering area parameters of the camera 20 and convert the velocity value into an upper limit parameter for exposure time of the camera 20, and send these parameters to the camera 20 through the control interface.

[0026] The verification feedback module 33 connects the recognition result output terminal of the processor 30 to the configuration terminal of the radar sensor 10. The verification feedback module 33 is configured to extract the spectral feature data of the current target and store it in the interference feature library in response to the recognition result of the non-human category; the verification feedback module 33 is also configured to calculate the deviation value between the predicted coordinate position and the actual recognition position in response to the recognition result of the human category, and update the coordinate calibration matrix in the parameter mapping module 32 using the deviation value.

[0027] See attached document Figure 3 The identification method provided by this invention includes the following steps: The radar sensor 10 is in continuous monitoring mode, while the camera 20 is either off or in a low-power sleep mode. The radar sensor 10 periodically transmits frequency-modulated continuous wave signals and receives echoes to detect targets within the monitoring area. During detection, the radar sensor 10 analyzes the target's distance, angle, and velocity information in real time and extracts the target's micro-Doppler spectral characteristics.

[0028] The radar sensor 10 internally stores an interference feature database, which records the spectral characteristics and corresponding spatial location information of known non-human interference sources. The radar sensor 10 compares the spectral characteristics of the current target extracted in step S1 with the data in the interference feature database. If the comparison result shows that the current target is a known interference source, the radar sensor 10 filters the signal and keeps the camera 20 in the off state; if the comparison result shows that the current target is a newly appearing moving object or the features do not match, the radar sensor 10 determines that the wake-up condition is met, generates a wake-up signal and sends it to the processor 30.

[0029] In response to the wake-up signal, the processor 30 acquires the target state data output by the radar sensor 10 through the position prediction module 31, and calculates the predicted position of the target at the imaging time based on the system delay parameters. The position prediction module 31 reads the pre-stored camera 20 startup delay constant. Since there is a physical delay from power-on to outputting the first frame image, directly using the radar coordinates at the trigger time will lead to focusing and metering area deviations. The position prediction module 31 uses a linear motion model to extrapolate and compensate for the target position. Specifically, the position prediction module 31 adds the radial distance of the target output by the radar sensor 10 at the trigger time to the product of the radial velocity and the system startup delay constant to obtain the predicted radial distance at the imaging time; simultaneously, it adds the azimuth angle at the trigger time to the product of the tangential angular velocity and the system startup delay constant to obtain the predicted azimuth angle at the imaging time.

[0030] Before activating the camera 20, the processor 30 pre-configures the image signal processing parameters of the camera 20 based on the predicted position and target velocity obtained in step S3 via the parameter mapping module 32. The parameter mapping module 32 sends metering area and exposure time commands to the camera 20 through the control interface. Specifically, the parameter mapping module 32 uses a calibration matrix to map the predicted radial distance and predicted azimuth angle into rectangular region coordinates on the image sensor plane, and sets this region as the region of interest for the automatic exposure algorithm. Simultaneously, to prevent image blurring due to target movement, the parameter mapping module 32 calculates the maximum permissible exposure time based on the absolute value of the target's radial velocity, writes this upper limit value into the camera 20's exposure limit register, and then sends a start-up enable signal.

[0031] In response to the start-up enable signal, camera 20 acquires the first frame of image data using the preset metering area and exposure time upper limit in step S4. Thanks to the feedforward configuration of the parameters, camera 20 can acquire a target image with suitable brightness and clarity without the need for multi-frame convergence, and transmits the image data to processor 30.

[0032] After receiving image data, the processor 30 uses a pre-set neural network algorithm to detect and classify targets in the image, outputting the target's category label and the coordinates of its actual detection box in the image. For the specific structure and training process of the neural network algorithm, those skilled in the art can use existing convolutional neural network models.

[0033] Processor 30 determines whether the category label is non-human. When it is determined to be non-human (such as a pet or a moving part of a household appliance), processor 30 determines it as a false alarm and does not trigger a service response operation. At this time, the verification feedback module 33 executes the interference suppression logic: extracts the micro-Doppler spectrum features and spatial location information of the target recorded by radar sensor 10, marks it as an interference feature, and writes it into the interference feature library of radar sensor 10 for direct filtering in subsequent monitoring in step S2.

[0034] When the category label is "human body," the processor 30 determines that the recognition is successful and triggers a preset business response operation, such as activating the screen display or initiating voice interaction. Simultaneously, the verification feedback module 33 executes online calibration logic: calculating the positional deviation between the actual detection box coordinates obtained in step S6 and the radar-based predicted pixel coordinates calculated in step S3. If this positional deviation is within the valid update range, the verification feedback module 33 updates the translation parameters of the calibration matrix in the parameter mapping module 32 according to the deviation value to correct the spatial mapping error between the radar sensor 10 and the camera 20.

[0035] See attached document Figure 3As a specific implementation method for the radar sensor 10 to detect targets within the monitoring area in the aforementioned step S1, the radar sensor 10 performs the following hierarchical processing steps through its internally integrated microcontroller unit: Step S101: The radar sensor 10 is configured to periodically transmit a linear frequency modulated continuous wave (FMCW) signal through its radio frequency front end. In this embodiment, the frequency of the transmitted signal changes linearly with time. When the electromagnetic wave encounters a non-homogeneous medium (such as a human body or object) in the monitoring space, it is reflected to form an echo signal. The receiving link captures this echo and introduces it into a mixer to perform beat processing with the current transmitted signal to generate an intermediate frequency (IF) signal. This IF signal carries in the time domain the frequency difference information caused by the round-trip time of the electromagnetic wave, as well as the phase change information caused by the Doppler effect. The analog-to-digital converter inside the radar sensor 10 discretizes the IF signal to satisfy the sampling rate of the sampling theorem, converting the continuous analog voltage into a digital baseband data sequence.

[0036] Step S102: To analyze the range and velocity components superimposed on the time-domain signal, the radar sensor 10 performs a two-dimensional frequency domain transformation on the digital baseband data. The processing logic performs a first-dimensional Fast Fourier Transform (FFT) in the fast time dimension, mapping the time-domain echo to a spectrum in the range dimension, where the peak frequency corresponds to the radial range of the target; then, it performs a second-dimensional Fast Fourier Transform (FFT) in the slow time dimension to analyze the phase rotation rate of the same range cell over multiple consecutive linear frequency modulation cycles, thereby extracting the Doppler frequency. After this processing, the original data is converted into a range-Doppler matrix, where the magnitude of each element represents the signal energy intensity at the corresponding range gate and velocity gate.

[0037] Step S103: To address the system thermal noise and environmental clutter mixed in the radar echo, the radar sensor 10 uses a constant false alarm rate (CFAR) detector to adaptively threshold the range-Doppler matrix. The CFAR detector traverses the target cells in the matrix, selects the average power of its surrounding reference cells as the local noise basis estimate, and combines it with a preset false alarm probability factor to determine the dynamic detection threshold. When the signal power of the target cell exceeds the dynamic detection threshold, the cell is determined as a valid target point; otherwise, it is set to zero to suppress background noise.

[0038] Step S104: Radar sensor 10 calculates the physical space state parameters based on the frequency domain index of the effective target point. Based on the ranging principle of linear frequency modulated continuous wave, the processing logic maps the radial distance of the target to its frequency position in the intermediate frequency spectrum, combined with the frequency modulation slope parameter. Simultaneously, based on the frequency shift of the effective target point in the Doppler spectrum, and combined with the radar carrier wavelength parameter mapping, the radial velocity of the target is obtained. Furthermore, the radar sensor 10 utilizes the phase difference between adjacent antenna elements in the receiving antenna array to calculate the target's azimuth angle using a direction-of-arrival (DOA) estimation algorithm. The above calculations are all based on conventional physical models in the field of radar signal processing, which can convert frequency domain features into physical space coordinates.

[0039] Step S105: The radar sensor 10 converts the radial distance RR and azimuth angle obtained above into a single value. radial velocity and the corresponding Doppler spectral energy characteristics Encapsulated as a target state vector : ;in, The target state vector is a standard format used for data interaction within the system. Radial distance of the target (Range), usually in meters (m); The azimuth of the target, usually in degrees (°) or radians (rad); The radial velocity of the target, usually expressed in meters per second (m / s); Doppler spectrum energy characteristics characterize signal strength.

[0040] This state vector As a standard data interaction interface within the system: on the one hand, it is used to compare features with the interference feature library in step S1 to filter out non-human targets; on the other hand, after confirming that the target is valid, the vector is transmitted to the processor 30 as a wake-up data packet through the communication interface, so that it can perform spatiotemporal trajectory prediction and camera parameter configuration in step S2.

[0041] See attached document Figure 3 To further refine the specific implementation mechanisms of feature extraction in step S1, signal filtering using the interference feature library in step S2, and interference feature library update in step S7, this embodiment, after obtaining the target primary state vector in step S105, continues to execute the following signal feature analysis and interference screening steps S106 to S110: Step S106: Radar sensor 10 extracts the slow-time dimension complex signal sequence of the target's range cell (RangeBin). Since a single Doppler frequency shift can only characterize the target's centroid translational velocity and is insufficient to distinguish between rigid mechanical motion and non-rigid biological motion, the signal processing unit performs a short-time Fourier transform (STFT) on this complex signal sequence. This process uses a window function with a preset overlap rate (e.g., 50% to 75%) to truncate signal segments, mapping the one-dimensional time-domain echo signal into a two-dimensional time-frequency distribution map (Spectrogram). In this Spectrogram, the horizontal axis maps the time dimension, the vertical axis maps the instantaneous frequency dimension, and the grayscale value of each pixel characterizes the signal's energy density, thus presenting the micro-Doppler effect of the target's scattering cross-section changing with limb movement on a microscopic time scale.

[0042] Step S107: Based on the generated time-frequency distribution map, the signal processing unit executes envelope extraction logic to construct micro-Doppler feature curves. Considering that the Doppler frequency shift of the torso is relatively stable during human movement, while the limbs (such as arm swings and leg steps) will generate periodic and large-amplitude frequency shift modulations, the signal processing unit traverses each discrete time frame of the time-frequency distribution map, extracts the highest and lowest frequency points with energy intensities higher than the preset noise floor threshold, and fits them to form upper envelopes. and lower envelope The envelope formed by these two envelope lines precisely defines the range of instantaneous velocity distribution of all scattering points of the target along the line of sight.

[0043] Step S108: The signal processing unit calculates the quantized micro-Doppler feature parameters based on the extracted envelope, focusing on extracting the average Doppler bandwidth. and envelope period .

[0044] Among them, the mean Doppler bandwidth This is used to characterize the dispersion of the motion velocities of various components of a target. For rigid bodies, the relative velocities of their components are constant, resulting in a narrow and stable bandwidth; however, for the human body, the swinging of limbs causes the instantaneous bandwidth to broaden. The calculation formula is as follows: In the formula, : Mean Doppler bandwidth, used to characterize the dispersion of the motion velocity of each part of the target (human limb movement usually causes bandwidth broadening). The total number of sampling points within the observation time window.

[0045] : The discrete-time index within the current window; : No. The upper envelope frequency value of the micro-Doppler envelope at time step; : No. The lower envelope frequency value of the micro-Doppler envelope at time step.

[0046] For envelope period The signal processing unit performs autocorrelation on the upper envelope sequence and detects the position of the second peak of the autocorrelation function to determine the main period. If the autocorrelation function does not show a peak within a preset time range, then... This parameter is marked as a non-periodic motion characteristic value. Based on the gait rhythm principle of human walking, it can effectively distinguish between interference sources with fixed mechanical frequencies and the human body with biological rhythms.

[0047] Step S109: The radar sensor 10 normalizes the calculated feature parameters and combines them with the standard deviation of the envelope. Construct the micro-Doppler feature fingerprint vector of the current target. : In the formula: The target's micro-Doppler fingerprint vector is used for comparison with the interference database; The average Doppler bandwidth calculated above; Envelope period, characterizing the rhythmicity of movement (such as breathing or gait cycles). The standard deviation of the envelope characterizes the dispersion of signal fluctuations.

[0048] This feature fingerprint vector, as a digital summary of the target's motion pattern, can compress complex time-varying signals into low-dimensional feature space data that can be compared.

[0049] Step S110: The radar sensor 10 executes interference source matching and signal filtering logic. In this embodiment, the radar sensor 10 has a pre-stored reference feature vector library containing various typical environmental interference sources (such as air conditioner deflectors, electric curtains, and electric fans). The processing logic calculates the weighted Euclidean distance between the current target feature fingerprint vector and each reference feature vector in the library, and selects the minimum value as the optimal matching distance.

[0050] Subsequently, the optimal matching distance is compared with a preset similarity threshold. This threshold is determined based on the statistical distribution characteristics of a large number of calibrated interference source samples, taking the boundary of a confidence interval three times the standard deviation.

[0051] If the optimal matching distance is less than the threshold, it indicates that the motion characteristics of the current target highly match a known interference source recorded in the database, and the radar sensor 10 determines that the target is a non-human interference source. In response to this determination, the radar sensor 10 directly intercepts the wake-up signal at the underlying level, maintaining the sleep state of the subsequent processor 30 and camera 20, thereby achieving the filtering of the signal described in step S2.

[0052] Conversely, if the optimal matching distance is greater than or equal to the threshold, the target is determined to be a potential human body or an unknown moving object. In response to this determination, the radar sensor 10 triggers a wake-up process, generates a wake-up interrupt signal containing the aforementioned target state vector, and sends it to the processor 30. The processor 30 then instructs the RGB camera to perform further visual recognition.

[0053] See attached document Figure 3 To further refine the specific execution mechanism of the wake-up decision in the aforementioned main process step S2, and to implement the core design goal of this invention regarding effectively reducing power consumption and minimizing system resource waste, this embodiment constructs a decision model based on temporal probability within the interference suppression module. This model follows the feature extraction step and executes the following steps S201 to S204, aiming to accurately distinguish between mechanical interference that should be filtered and potential human bodies that should trigger wake-up at the signal processing level through multi-dimensional logical verification, thereby maintaining the processor 30's sleep state as much as possible.

[0054] Step S201: Establish a similarity probability model based on nonlinear mapping.

[0055] After obtaining the minimum Euclidean distance between the current target feature vector and the feature vectors of each reference interference source, directly using a linear hard threshold often fails to adapt to complex electromagnetic environments, easily leading to decision flickering in critical states. To address this, the interference suppression module introduces a nonlinear mapping mechanism, utilizing a kernel function with radial basis function properties to transform the unbounded Euclidean distance into a normalized confidence probability. The physical meaning of this mapping process lies in simulating the gravitational field in the feature space; that is, the closer a sample point is to the center of the interference source, the higher its probability density of belonging to that type of interference, exhibiting a nonlinear exponential increase. This amplifies the signal strength of strongly correlated features while rapidly attenuating the influence of weakly correlated noise.

[0056] Step S202: Perform timing consistency verification based on sliding time window.

[0057] Given that radar echo signals may experience instantaneous distortion due to multipath effects and environmental clutter, the probability score for a single frame may contain random noise. To improve the system's stability in determining continuous movement, the interference suppression module constructs a time sliding window of preset length and performs weighted smoothing on the probability scores of consecutive frames within the window, thereby calculating a comprehensive decision score. This step utilizes the principle of continuity in the motion state of macroscopic objects in the physical world, meaning that the target's motion attributes do not change abruptly within milliseconds. In the calculation logic, the system assigns higher weights to feature frames closer to the current moment, thus ensuring the smoothness of the decision, avoiding interference from outliers in a single frame, and retaining sensitivity to state changes.

[0058] Step S203: Perform dual-threshold hysteresis comparison and wake-up decision.

[0059] To prevent the decision result from repeatedly jumping at the critical point, the interference suppression module pre-sets two decision parameters: a high decision threshold and a low decision threshold. Both thresholds are dimensionless scalars between 0 and 1, and satisfy the numerical relationship that the high decision threshold is greater than the low decision threshold, thus logically constructing a Schmitt trigger-type hysteresis comparison interval. This logic determines the system's triage strategy based on the interval in which the comprehensive decision score falls. When the overall decision score is greater than the high decision threshold, it indicates that the system has a very high confidence that the current target is a known interference source. At this time, the radar sensor determines that there is no need to wake up the processor 30 and keeps the RGB camera in the off state. This logic directly corresponds to the reduction of system operating load requirements mentioned in the technical background, that is, digesting invalid signals inside the radar sensor and maintaining the system in a deep sleep state.

[0060] When the overall decision score is less than the low decision threshold, it indicates that the current target differs significantly from the characteristics of known interference sources and is classified by the system as a potential human. At this time, the radar sensor immediately generates an external interrupt signal to wake up the processor 30 and instructs the RGB camera to start visual confirmation, thus entering the subsequent image acquisition and analysis process.

[0061] When the overall decision score is between the high decision threshold and the low decision threshold, the system maintains the decision state of the previous moment to ensure the continuity of system operation and avoid frequent state switching caused by small signal fluctuations.

[0062] Step S204: Closed-loop adaptive update in response to RGB visual verification results.

[0063] This step specifically achieves deep integration of radar and vision, solving the technical pain point of radar sensors' difficulty in accurately identifying object types, and continuously optimizing the radar's predictive capabilities through a feedback mechanism. As a preferred approach, if the system wakes up the processor 30 due to a potential human body, but the RGB camera, through image analysis algorithms, clearly determines that no human body exists in the image (e.g., the target is a newly appeared pet or a robot vacuum cleaner not recorded in the database), the processor 30 generates a negative sample marker signal and sends it back to the radar sensor. Responding to this signal, the radar sensor automatically extracts the radar feature vector sequence within the time period of the false trigger, performs weighted fusion with historical feature templates to update it, and then stores it in non-volatile memory. Through this adaptive update mechanism, the agent can automatically learn about unique interference sources in the home environment during use. As the feature database is continuously enriched, the decision score for similar objects will gradually increase and exceed a high decision threshold, thus directly intercepting the target at the radar end in the future, avoiding repeated false wake-ups, and achieving power consumption optimization throughout the entire lifecycle.

[0064] See attached document Figure 3 As a specific implementation detail of the aforementioned main process step S3 (wake-up and position prediction), this embodiment constructs a closed-loop control model that includes delay calibration, parameter feedforward, and validity verification. This model is executed by the processor 30 after completing the underlying driver initialization. It aims to solve the spatiotemporal mismatch problem between radar-detected targets and visually acquired images caused by the physical delay of sensor cold start in existing technologies. Based on prior radar data, it performs physical-level parameter optimization on the camera 20 to ensure that the first frame image acquired in low-power wake-up mode meets the algorithm's recognition requirements. The specific execution logic is as follows: Step S301: Construct system-level wake-up delay calibration and coordinate compensation. During the process of processor 30 receiving the radar interrupt signal and instructing camera 20 to start, due to physical processes such as power domain switching, clock stabilization, and I2C bus initialization, there is an objective system lag of tens to hundreds of milliseconds. To establish an accurate spatiotemporal synchronization reference, processor 30 reads the high-precision hardware timer at the system's underlying layer to obtain the time difference between the radar interrupt trigger moment and the expected image sensor exposure moment, which is recorded as the total system delay. .

[0065] Based on the principle of continuity of motion of macroscopic objects, in Within this tiny time window, the target's motion is approximately uniform linear motion. The system utilizes the target's radial velocity reported by the radar sensor. Linear extrapolation compensation is performed on the target's true position at the time of image acquisition. The calculation formula is as follows: In the formula, Compensated theoretical radial distance of the target corresponding to the image acquisition time (unit: meters); : The original radial distance detected by the radar at the wake-up trigger moment (unit: meters); Target's radial velocity relative to the radar (unit: meters per second). Negative for approaching and positive for moving away (or as defined by the specific coordinate system); Total system delay, which is the time difference between the radar interruption trigger time and the expected image sensor exposure time (unit: seconds).

[0066] The physical technology of this step aims to fill the data blind spot caused by the sensor startup gap, eliminate spatial position prediction deviation caused by time asynchrony, and thus provide timely coordinate input for subsequent camera parameter configuration.

[0067] Step S302: Adaptive pre-configuration of camera parameters based on predicted position. To address the issue that conventional cameras default to full-screen average metering and default exposure time during cold start, resulting in blurred or underexposed human targets in backlit or dynamic scenes, the processor 30, through the parameter mapping module 32, uses the compensation coordinates obtained in step S301 before officially starting the image data stream (Stream On). Actively configure the ISP (Image Signal Processing) pipeline of camera 20: On one hand, dynamic locking of the metering area (AEROI) is performed. The parameter mapping module 32 retrieves a pre-stored calibration matrix, which contains the camera's intrinsic parameters (focal length). Optical center coordinates And the rotation and translation extrinsic parameters between the radar and the camera. Utilizing the principle of perspective projection, the predicted radial distance in three-dimensional space is... And the radar azimuth angle is mapped to the coordinates of a rectangular area on the plane of the two-dimensional image sensor. The parameter mapping module 32 writes the coordinates of this region into the automatic exposure control register of the image sensor via the I2C interface, forcing the ISP algorithm to concentrate the metering weights on this prediction region. This ensures that even if there is background highlight interference, the system can still prioritize ensuring that the grayscale value of the potential human body region is in the linear response zone, avoiding the loss of effective features.

[0068] On the other hand, adaptive calculation of the maximum permissible exposure time is performed. To prevent motion blur caused by excessively fast relative speed of the target, which would reduce the feature extraction efficiency of the subsequent neural network, the system calculates the upper limit of the exposure time under the current operating conditions based on the principle of optical flow. The calculation model is as follows: In the formula: : Maximum integration time written to the camera (maximum exposure time, unit: seconds); The maximum number of motion blur pixels allowed by the system (e.g., 1.5-3.0 pixels) ensures that the blur does not exceed the receptive field of the CNN convolutional kernel; : Physical pixel size of the image sensor (PixelPitch, in meters); : Target radial distance after compensation (unit: meters); Equivalent physical focal length of a camera lens (unit: meters); : Absolute value of the target radial velocity (unit: meters per second); Minimum speed threshold to prevent the denominator from approaching 0 and causing calculation overflow.

[0069] In this embodiment, the threshold is set to 1.5 to 3.0 pixels, and its value is based on the size of the first convolutional kernel of the subsequent convolutional neural network (CNN) (usually 3x3 or 5x5), to ensure that the blurring does not exceed the receptive field of the convolutional kernel, thereby maintaining the sharpness of the edge features.

[0070] Step S303: Mapping the visual region of interest (ROI) and validating its effectiveness. After the camera 20 completes the acquisition of the first frame image based on the above optimized parameters, the processor 30 does not directly process the full-resolution image, but instead utilizes... The calibration information is logically blocked and data is pruned to specifically implement the core design goals of this invention regarding reducing system load and minimizing invalid actions: Before entering the deep learning analysis process, the system performs a field-of-view validity gating judgment. If the calculated... If the distance is less than the camera's minimum focusing distance, or if the center coordinates of the ROI calculated by projection fall outside the effective resolution boundary of the image sensor, it indicates that although the radar detected the moving target, the target has moved out of the effective monitoring range of the visual sensor. In such cases, the processor 30 will determine that the acquisition is invalid, directly terminate the execution of the subsequent step S4, and immediately send a command to shut down the camera 20, causing the system to re-enter sleep mode.

[0071] If the Region of Interest (ROI) is within the effective range of the image, the processor 30 will crop the ROI based on the projected coordinates and perform necessary preprocessing operations. Specifically, the system expands the pixel region outward by a preset proportion (e.g., 20%) centered on the mapped coordinates to accommodate prediction errors, and then resizes the cropped region to the standard input dimension required by the subsequent neural network model (e.g., a 224×224×3 tensor). Through this radar-guided spatial attention mechanism, the system can filter out more than 90% of irrelevant background data in the image, allowing the AI ​​agent to perform inference only on extremely low-resolution image patches. This not only significantly reduces CPU memory usage and power consumption but also improves the real-time response speed of human body recognition.

[0072] See attached document Figure 3 To specifically implement the technical requirements of reducing system load and improving human recognition success rate in the technical solution of this invention, the processor 30 executes the algorithm described in this embodiment to accurately map the spatial position of the target detected by the radar sensor onto the two-dimensional image plane of the RGB camera 20, and constructs a lightweight neural network model to perform feature identification on the mapped area. This process combines the ranging and angle measuring capabilities of the radar with the visual resolution advantage of the camera, enabling direct locking of the target area during the image analysis stage, avoiding the waste of computing power caused by blindly searching the full-frame image. This logic includes the following execution steps: Step S401: Spatiotemporal alignment and rigid body transformation of the multi-sensor coordinate system. The target position output by the radar sensor is usually based on a spherical coordinate system with the radar antenna center as the origin, while the imaging of the camera 20 depends on a Cartesian coordinate system with the optical center as the origin. In order to unify the spatial reference, the processor 30 first converts the radar spherical coordinates to radar Cartesian coordinates. During this process, to prevent invalid data transmission, the system performs validity filtering on the raw radar data, initiating subsequent transformations only when the detection range is greater than zero and the echo signal-to-noise ratio (SNR) is greater than a preset threshold (e.g., 15 dB). Subsequently, the points in the radar coordinate system are transformed to the camera coordinate system using a pre-calibrated extrinsic parameter matrix. Let the target's position vector in the camera coordinate system be... The transformation model is as follows: In the formula, : The three-dimensional coordinate vector of the target in the camera coordinate system ; The target's three-dimensional coordinate vector in the radar Cartesian coordinate system; : A 3×3 rotation matrix, representing the rotation angle (external parameter) of the radar coordinate axes relative to the camera coordinate axes; : 3×13×1 translation vector, representing the physical displacement (extrinsic parameter) of the radar center relative to the camera optical center.

[0073] The technical purpose of this step is to eliminate parallax caused by the different physical installation positions of the two sensors, ensuring that the target position calculated by the system is relative to the true spatial orientation of the camera.

[0074] Step S402: Perspective projection and pixel coordinate calculation. Obtain the target's 3D coordinates in the camera coordinate system. Then, the processor 30 uses the pinhole imaging model to project the three-dimensional point onto the two-dimensional pixel plane, thereby determining the center anchor point of the target in the image.

[0075] To ensure the stability of mathematical operations, in this embodiment, the processor 30 verifies the depth value before performing the projection operation. .like in If the distance is extremely small (e.g., 0.01 meters), the system determines that the target is too close to the lens to be imaged, and will directly discard the frame data and reset the algorithm state to prevent division by zero errors from causing program crashes. Through this step, the system can establish a one-to-one mapping relationship between the physical entities sensed by the radar and the pixel indices in the image data, providing accurate center positioning for subsequent local image extraction.

[0076] Step S403: Adaptive ROI bounding based on depth perception. Since the target detected by radar is only a point coordinate, while the subsequent human recognition algorithm requires a rectangular image patch containing complete human features as input, the system needs to adaptively generate the region of interest (ROI) bounding box based on the target's distance. The processor 30 calculates the theoretical projection size of the target on the image plane based on the principle of similar triangles. Let the statistical average physical width of the human target be... (For example, the preset height is 0.5 meters), the statistical average physical height is (For example, the default is 1.7 meters), then the pixel width of the ROI region and pixel height The calculation is as follows: ; In the formula, The region of interest (ROI) is defined by its pixel width and height in the image. The total resolution width and height (boundary constraints) of the image sensor; Normalized focal length (focal length in pixels); Statistical average physical width (e.g., 0.5 meters) and height (e.g., 1.7 meters) of human targets; : Target depth distance (i.e., in camera coordinates) (axis coordinates); The safety redundancy factor (e.g., 1.2~1.5) is used to expand the interception range to accommodate attitude changes or errors. In this embodiment, the value range is set to 1.2 to 1.5.

[0077] The technical basis for introducing redundancy coefficients lies in expanding the cropping range to accommodate target contour overflow caused by radar measurement errors, changes in human posture (such as waving or bending over), or non-rigid deformation, preventing critical features (such as the head or hands) from being truncated. After calculating the ROI parameters, the system also performs boundary compliance checks (Clamping) to ensure that the cropping area falls entirely within the physical boundaries of the image, avoiding memory out-of-bounds access.

[0078] Step S404: Neural Network Feature Recognition and Classification. As the core decision-making step in determining whether an object is human, the processor 30 constructs a lightweight convolutional neural network (CNN) model based on cropped and normalized ROI image patches. This model aims to solve the problem of radar's inability to distinguish target types. Specific implementation details are as follows: Model Architecture Design: This embodiment employs a lightweight backbone network based on depthwise separable convolution (e.g., a cropped version of MobileNetV3-Small) to adapt to the limited CPU computing power of the AI ​​agent. The network input layer receives an RGB tensor of size 224×224×3; the intermediate layers contain several inverted residual blocks, each consisting of a 1×1 dilated convolution, a 3x3 depthwise convolution, and a 1×1 projective convolution. Squeeze-and-Excitation (SE) attention modules are embedded between layers to enhance feature extraction capabilities; the final layers are connected to a global average pooling layer and a fully connected layer (FC).

[0079] Output definition and physical meaning: The output layer of the network uses the Softmax activation function, outputting a two-dimensional probability vector. .in, This represents the confidence level that the input image patch belongs to the human body category. This indicates the confidence level for belonging to a non-human entity (such as a pet, robot vacuum cleaner, fan, etc.). The system sets a judgment threshold. (e.g., 0.85), only when Only when the CPU determines that the current situation is a valid human trigger, will it execute voice interaction or screen lighting operation; otherwise, it will be determined as a false trigger and the system will remain silent.

[0080] The model training process is disclosed: To enable those skilled in the art to reproduce the model, this invention discloses its training strategy: constructing a hybrid dataset containing positive samples (human ROI images under different poses and lighting conditions) and negative samples (common moving objects in a home environment). The loss function used is binary cross-entropy loss. The Adam optimizer is used during training, with a learning rate set to... Furthermore, data augmentation techniques such as random cropping, rotation, and color jitter are introduced to improve the model's generalization ability.

[0081] See attached document Figure 3 This embodiment details how the processor 30 utilizes the prior characteristics of radar sensing data to execute the aforementioned main process step S4 (camera parameter feedforward configuration) to actively feedforward control the underlying hardware parameters of the camera 20. This control logic aims to overcome the time lag problem caused by the passive convergence of traditional cameras relying on image statistical information during the cold start phase, ensuring that the first frame image after system wake-up meets the algorithm's requirements for clarity and brightness recognition, thereby fulfilling the technical goal of improving the success rate of AI intelligent agent human recognition as outlined in the technical background. This logic specifically includes the following execution steps: Step S501: Lens focusing open-loop control based on range measurement values. Processor 30 utilizes the precise target distance provided by the radar sensor. The system directly calculates the target drive current value of the voice coil motor (VCM) in the camera module to eliminate the search latency of traditional hill climbing algorithms. The system pre-constructs an inverse mapping model between distance and VCM position. In the formula, : The register value written to the lens driver chip (dimensionless integer, representing the position of the voice coil motor). Optical sensitivity coefficient (unit: meter·DAC), characterizing the coupling relationship between lens focal length and motor stroke; Real-time target distance measured by radar (unit: meters); : Base Value (DAC value) when the lens is focused at infinity; : Integer function.

[0082] The physical significance of this step lies in linearizing the nonlinear optical imaging law, so that the lens is in the theoretical focus position before exposure, thus achieving zero-delay focusing at the physical level.

[0083] Step S502: Dynamic gain clamping under low illumination.

[0084] In the hardware latency compensation model of the aforementioned embodiment, the system has calculated the dynamic exposure upper limit to prevent motion blur based on the target radial velocity detected by radar. In order to compensate for the It may be much shorter than the usual default exposure time. The processor 30 calculates the required analog gain based on the reciprocity principle to account for the resulting brightness loss. To prevent excessive gain in extremely low-light environments from causing image noise to overwhelm texture features, the system executes the following gain clamping control logic: In the formula, The final digital gain ratio command issued; : Preset target brightness value (grayscale value); Ambient light intensity (unit: Lux); The upper limit of dynamic exposure time calculated based on radar velocity (i.e., the aforementioned) ); The reference sensitivity of the sensor; : Maximum allowable gain threshold (e.g., 16dB) is used to prevent excessive noise.

[0085] Introduction The technical objective is to balance the physical contradiction between motion blur and image noise: when the ambient light is extremely dark and the target is moving at a high speed (leading to...) When the required gain exceeds the sensor's signal-to-noise ratio (SNR) tolerance (for extremely short gain), the system prioritizes clamping the gain to suppress noise rather than unrestrictedly increasing brightness. This weighted decision strategy ensures that the image input to the neural network achieves an optimal balance between contour sharpness and SNR.

[0086] Step S503: I2C Bus Timing Control and Instruction Issuance. After completing the above parameter calculations, the processor 30 transmits the calculated parameters through the I2C (Inter-Inteqrated Circuit) bus interface. Dynamic exposure limit and gain command It is encapsulated as a hardware instruction package.

[0087] In a preferred embodiment, the processor 30 utilizes the group hold feature of the I2C bus to perform atomic operations; before sending the Stream On command to the image sensor, the focus position, exposure line count, and gain parameters are packaged and written. This timing control ensures that all optical and photoelectric parameters take effect within the same vertical sync signal (VSYNC) cycle, avoiding brightness banding or color drift in the first frame due to asynchronous parameter updates, thus ensuring that the first frame image can be used as effective AI inference input.

[0088] See attached document Figure 3 To further implement the technical objective of this invention—using an RGB camera to successfully identify human bodies and reduce resource waste caused by misjudgments—the processor 30 executes the image classification algorithm described in this embodiment after receiving the wake-up signal from the radar sensor. This algorithm extracts depth features from the image captured by the camera 20 by constructing a lightweight neural network model adapted to the computing power of the Android smart agent, thereby accurately distinguishing at the pixel level whether the target triggering the radar sensor is a human body or a non-human moving object (such as a pet, fan, or robot vacuum cleaner). This process specifically includes the following steps: Step S601: Tensor Construction and Normalization Preprocessing of Input Data. The processor 30 first acquires image frames output by the camera 20. Considering the potential pixel value distribution drift caused by differences in lighting conditions (such as backlighting or low light) in different application scenarios, the system performs normalization processing based on statistical characteristics before feeding the data into the neural network. The system adjusts the image to a fixed input size for the model. (e.g., 224×224 pixels) and construct a four-dimensional input tensor. Subsequently, the system performs a normalization operation to map the pixel distribution of the original image to the standard normal distribution range, ensuring that the activation response of the intermediate layers of the neural network is in the non-saturation region, thereby improving the robustness of the feature extraction algorithm to changes in ambient lighting.

[0089] Step S602: Construct a feature extraction network based on depthwise separable convolutions. Considering the limited hardware resources and real-time response requirements of the Android system's AI agent, this embodiment constructs a lightweight convolutional neural network based on inverted residuals (e.g., a customized MobileNetV3 architecture).

[0090] This network extracts human features through a pipeline logic of expansion-extraction-compression: The data stream first enters a 1×1 point convolutional layer for channel upsizing, using the high-dimensional space to decouple feature information. Then, it passes through a 3×3 or 5×5 depthwise convolutional layer to extract spatial features independently for each channel. Finally, a 1×1 point convolutional layer projects the features back into the low-dimensional space. To enhance the model's sensitivity to key human features (such as head contours and limb structures), this embodiment embeds a Squeeze-and-Excitation (SE) attention mechanism in the residual module. This mechanism compresses spatial information through global average pooling and adaptively learns the weight coefficients of each channel using fully connected layers, thereby automatically suppressing the response weights of background noise (such as moving curtains) during the feature extraction stage and enhancing the response weights of human features.

[0091] Step S603: Weighted training strategy under imbalanced samples.

[0092] To ensure that technicians can reproduce the model and address the sample imbalance problem caused by the significantly higher frequency of non-human moving objects compared to humans in the home environment, this embodiment discloses a specific model training strategy. During the training phase, the system employs a weighted binary cross-entropy loss function to optimize model parameters, preventing overfitting to the negative sample class (non-human). In this embodiment, positive sample weights are set. Greater than negative sample weights (For example, take) The value of this parameter is derived from the reciprocal of the statistical probability of falsely triggered events in typical home scenarios. The technical purpose of this strategy is to force the model to impose a larger gradient penalty on missed human bodies during backpropagation, thereby improving the recall rate of human body recognition while ensuring a low false alarm rate.

[0093] Step S604: Decision output based on hysteresis comparator. The neural network output layer, after Softmax processing, yields the confidence score that the current frame belongs to the human body. Considering the continuity of radar sensing and the instantaneous nature of image recognition, in order to avoid high-frequency jitter between yes and no recognition results due to object occlusion or posture changes, processor 30 introduces hysteresis logic to generate the final control signal.

[0094] The system is set to determine a high threshold. (e.g., 0.85) and low threshold (e.g., 0.60), the final output signal The determination rule is as follows: In the formula, : Current frame The final decision output signal; Previous frame The decision-making process remains unchanged; The confidence (probability value) that the current frame output by the neural network belongs to the human category; : Determine a high threshold (e.g., 0.85) to confirm it as the threshold for the human body; : If the threshold is low (e.g., 0.60), it is considered a non-human threshold.

[0095] The physical meaning of this logic lies in constructing a decision dead zone with memory characteristics, triggering voice interaction or screen activation only when the confidence level strongly exceeds a high threshold; once a human is identified, the system maintains the state of human presence even if the confidence level slightly decreases in subsequent frames (as long as it does not fall below a low threshold). This mechanism effectively eliminates signal jitter, ensuring the consistency and stability of the AI ​​agent's user experience.

[0096] See attached document Figure 3 This embodiment, as a specific implementation of the aforementioned main process step S7 (interference feature library update), aims to deepen the execution mechanism of the radar sensor not responding after determining that the object is not human in the technical disclosure scheme. Specifically, it establishes a radar signal suppression mechanism based on visual confirmation for persistent interference sources such as swaying curtains, pet activity, or robot vacuum cleaners. This mechanism prevents the system from falling into a high-power, dead-loop state of radar wake-up-visual denial-sleep-immediate radar wake-up again, thereby effectively reducing the system's operating load. This processing logic is executed by processor 30 and specifically includes the following steps: Step S701: Spatiotemporal consistency verification of heterogeneous sensor data.

[0097] Before performing the shielding operation, the processor 30 needs to confirm that the non-human target identified by the camera 20 is indeed the physical source that triggered the radar sensing, in order to eliminate the risk of missed detection in complex scenarios such as a stationary human body in the distance and a pet running nearby.

[0098] The system utilizes the pinhole imaging principle to map a visual bounding box to three-dimensional depth space and then matches and verifies it with radar ranging data. To prevent computational anomalies caused by the extremely small bounding box, the system introduces a numerical stability term and a consistency judgment index. The calculation is as follows: In the formula, Consistency deviation value, used to determine whether a radar target and a visual target are the same object; : The distance of the target detected by the radar; : Camera optical focal length; : Identify the prior physical height of the target (from a preset library, such as the height of a cat); : The vertical pixel height of the target bounding box in the image; Image sensor pixel size; : A minimum value stabilizing term set to prevent the denominator from approaching 0.

[0099] When the calculated deviation value Less than the preset spatial tolerance threshold When the distance is 0.5 meters (for example), the system determines that the visually captured non-human target and the radar-sensed moving source are the same physical object, thereby activating the subsequent interference shielding logic.

[0100] Step S702: Interference source feature extraction and parameterized shielding gating.

[0101] Once it is confirmed that the moving object is not a human body and passes the consistency check, the processor 30 extracts the multi-dimensional features of the radar sensing signal at the current moment and constructs dynamic shielding parameters for the specific interference source.

[0102] The system defines a dynamic rejection zone within the radar's range-velocity two-dimensional phase space, denoted as . : In the formula, The set of shielding intervals within the radar range-velocity phase space; The distance and velocity values ​​currently detected by the radar; Mean distance and mean radial velocity of the interference source (such as a fan); , : Tolerance, which refers to the allowable fluctuation range of distance and speed, respectively.

[0103] This step sets up a parameterized filter in the radar detection logic using a software-defined method. Its physical purpose is to issue specific filtering instructions to the underlying signal processing unit (DSP or MCU): any Doppler signal falling within this range and velocity range is considered as known invalid interference, and there is no need to send a wake-up interrupt to the main CPU again.

[0104] Step S703: Conditional wake-up control based on negative feedback mechanism.

[0105] The system will generate The parameters are written back to the radar module or its associated low-power coprocessor via a communication interface (such as UART or SPI). During subsequent monitoring cycles, the system executes conditional wake-up logic based on negative feedback, waking up only when newly detected radar signal features are detected. The main CPU and RGB camera will only be reactivated when the shielding range is exceeded.

[0106] Logical criteria for wake-up trigger Described as: In the formula, : Wake-up trigger flag. When the value is 1, it means that the system has identified a valid human body and will send an interrupt signal to wake up the main processor (CPU) and RGB camera. When the value is 0, it means that the system has identified interference or noise and will continue to maintain a low-power sleep state. Newly detected radar signal characteristics typically refer to features including the target's current distance ( ) and speed ( ) feature combination The parameterized filter generated in S702 based on known interference sources (such as fans, sweepers). If If the target falls within this interval (∈∈), it means that the current target is a known interfering object; Radar echo energy intensity; characterizing the power of the currently detected radar signal; Background noise floor threshold.

[0107] The system has a preset minimum energy threshold. Only signals with an intensity exceeding this threshold will be considered as potential targets; signals below this threshold will be considered as ambient background noise.

[0108] / : Set affixation symbol, This indicates that the signal does not belong to the shielded area (i.e., the signal is not within the shielded area and may be a new target). This indicates that it belongs to (i.e., the signal is within the shielded area, and it is interference).

[0109] As a preferred embodiment, to prevent slow drift of the interference source location from causing shielding failure or false shielding, the system sets a shielding validity period. When the validity period expires, the system automatically clears the shield. It also forces a visual verification. This time-dimensional negative feedback reset mechanism ensures that the system can adapt to changes in the environment during long-term operation, achieving both continuous power reduction against non-human moving objects and ensuring agile response to newly entering human bodies.

[0110] See attached document Figure 3 This embodiment, as a specific implementation of the aforementioned main process step S8 (business response and online calibration), corresponds to scenario c described in the technical disclosure: After the system successfully identifies a human body and executes a specific application, the processor 30 does not stop there. Instead, it uses the post-identification data to construct a positive feedback loop and perform online correction of the underlying collaborative parameters of the radar sensor and camera. This mechanism aims to solve the problem of system performance degradation caused by device aging, temperature drift, or changes in the installation environment, ensuring that the AI ​​agent maintains high sensitivity and high accuracy throughout its entire lifecycle. This processing logic is executed by the processor 30 and specifically includes the following steps: Step S801: Dynamic remapping of radar sensitivity based on visual anchor points.

[0111] When the image classification algorithm of the RGB camera confirms with high confidence that the current target is a human body, the processor 30 uses the physical result of the image recognition at this time as the ground truth to back-verify the detection threshold of the radar sensor. The system reads the echo energy value detected by the radar at the current moment. The system considers distance. If, within a certain distance range, the radar signal exceeds the factory-preset detection threshold but with a very small margin (e.g., only 5% higher), it indicates significant attenuation of the radar signal by the current environment (possibly due to enclosure obstruction or environmental absorption). To avoid missed detections in similar scenarios later, the system adaptively updates the sensitivity threshold downwards. In the formula, : No. After the next update, at a distance Radar energy detection threshold at the location; : The previous moment (the first moment) The threshold for (times); The forgetting factor (or learning rate) determines the weight of the impact of new data on historical parameters. The measured echo energy when a human body is successfully identified; : Reserved safety noise margin (e.g., 3dB).

[0112] The physical meaning of this formula lies in using an exponentially weighted moving average (EWMA) algorithm to slowly correct the radar's detection baseline using reliable human body signals. This long-cycle adaptive process can automatically compensate for signal attenuation caused by dust accumulation or aging of the product casing, ensuring that the radar always maintains optimal detection sensitivity.

[0113] Step S802: Temperature drift compensation for the VCM focusing model. In the aforementioned ISP parameter feedforward instruction embodiment, the system utilizes radar range... Calculate the preset position of the lens voice coil motor (VCM). However, the physical focal length and back focal length of the lens module expand and contract with changes in ambient temperature, causing deviations in the open-loop control model. In this successful recognition process, the camera's autofocus (AF) algorithm ultimately locks onto the sharpest actual focus position, denoted as... Processor 30 calculates the prediction bias. Based on this, the infinity reference value in the focusing model is corrected. : In the formula, , : Update the infinity reference DAC value in the focusing model before and after; Learning rate controls the step size of each adjustment. Prediction bias, which is the difference between the actual sharp focus position and the predicted position; : Sign function, extracts the direction of the deviation; : The maximum allowable adjustment amount (clamping value) in a single operation, to prevent parameter divergence.

[0114] The technical objective of this step is to eliminate systematic errors in the system using closed-loop feedback. This is achieved through continuous fine-tuning. The system can sense the current temperature environment and mechanical status in real time, making the open-loop focusing more accurate during the next cold start, thereby shortening the focusing convergence time.

[0115] Step S803: Posterior optimization of multimodal confidence weights. The system maintains a dynamic weight vector for fusion decision, used to weight the radar and visual signals. After successfully completing a radar wake-up to visual confirmation process, it indicates that the data consistency between the radar and camera is high in the current scene. The processor 30 dynamically adjusts the weight of radar features in the next wake-up decision based on the latency performance of this interaction. : In the formula, The weights of radar signals in the wake-up determination before and after the update; : Reward coefficient, the amount by which the weight increases when the conditions are met; Indicator function: 1 if the condition is met, 0 otherwise; The total response time from radar triggering to visual confirmation. : The preset optimal response time threshold (e.g., 200ms); : The maximum upper limit of the weight.

[0116] This logic is based on the simple idea of ​​reinforcement learning (Reinforcement Learninq). When the system responds quickly and accurately, the confidence level of the radar signal is appropriately increased, allowing the system to adopt a more aggressive wake-up strategy in subsequent detections (such as reducing the number of consecutive confirmation frames), thereby further improving the human recognition response speed and user experience of the AI ​​agent.

[0117] Specific application examples: Reference Appendix Figure 4 and attached Figure 5 The above-mentioned invention is applied to an Android-based smart central control screen installed on the living room wall. The device is normally in a screen-off standby state (deep sleep), and only automatically lights up and enters voice listening mode when it detects a family member approaching with an intention to operate it. It remains silent to interference sources such as robot vacuums, pet dogs, or swaying curtains.

[0118] in, Figure 4 This diagram illustrates the scene decision logic based on micro-Doppler features in a specific application embodiment of the present invention. The diagram visually demonstrates how the system distinguishes between interference sources (Scene A) and valid human targets (Scene B) using micro-Doppler features at the underlying radar signal processing stage.

[0119] The image above (Scenario A: Interference Source Filtering) illustrates the echo signal analysis for mechanical devices such as robotic vacuum cleaners. The curves in the image show that the target's Doppler energy / frequency characteristics exhibit a flat trend on the time axis (0-2 seconds), with values ​​consistently below the interference threshold indicated by the dashed line (approximately 1.15). This characteristic matches the narrow bandwidth and non-rhythmic properties described in the image, indicating that the target is in rigid body motion. Therefore, the system outputs a decision to remain silent and not trigger a wake-up call, effectively filtering out such false alarms.

[0120] The image below (Scenario B: User wake-up upon returning home) illustrates signal analysis for human walking. The solid line represents the micro-Doppler spectral characteristics, and the dashed line represents the fitted limb swing envelope. It can be seen that the signal characteristic values ​​fluctuate significantly and repeatedly exceed the interference threshold, exhibiting obvious periodic changes (corresponding to the limb swing period in the image). This feature matches the wide bandwidth indicated by the text in the image and exhibits limb swing periodicity. Therefore, the system outputs a decision to wake up the CPU and start the camera, achieving accurate human body recognition.

[0121] Figure 5 This diagram illustrates a multi-dimensional performance comparison and ISP feedforward strategy in a specific application embodiment of the present invention. The diagram is divided into two parts, left and right, respectively demonstrating the advantages of the present invention in system energy efficiency and the specific control strategy for solving motion blur problems.

[0122] The left chart (system performance comparison) compares the three solutions using a dual-axis chart. The left Y-axis represents the average standby power consumption (mW), and the right Y-axis represents the non-human accidental touch rate (%).

[0123] Option A (pure radar): Although it has the lowest power consumption (150mW as shown in the bar chart), the line graph shows that its false touch rate is as high as 85% (mainly due to interference from robot vacuums, etc.).

[0124] Option B (purely visual always on): Although the false touch rate is low, the power consumption is extremely high (the bar chart shows 2800mW), making it unsuitable for battery-powered or energy-saving scenarios.

[0125] Solution C (the present invention): The bar chart shows that its power consumption is only 185mW, slightly higher than the pure radar solution but far lower than the pure vision solution; at the same time, the line chart shows that its false touch rate is reduced to <1% through dual filtering. This data proves that the present invention achieves extremely high recognition accuracy while maintaining low power consumption.

[0126] The right figure (Scenario B: ISP feedforward strategy based on radar velocity) shows the inverse relationship between the target's moving speed and the maximum exposure time of the ISP.

[0127] The coordinate axes represent the target velocity detected by the radar (m / s) and the maximum exposure time allowed by the ISP (ms).

[0128] Motion blur risk zone: The gray shaded area in the image (area with speed > 1.2m / s) is marked as the motion blur risk zone, indicating that high-speed moving objects are very prone to blurring under the conventional automatic exposure (AE) strategy.

[0129] Feedforward Control Point: An asterisk marks a specific example in the diagram. When the radar detects a user's homeward speed of 1.2 m / s, the system does not use conventional exposure but instead forces the upper limit of the exposure time to 5 ms based on a curve. The text in the diagram explains that this feedforward control ensures clear image capture even when the user is moving quickly, effectively solving the problem of poor cold-start imaging quality.

[0130] I. Scene logic mapping.

[0131] The following describes the internal processing flow of the device when facing different targets, corresponding to the steps and modules mentioned above: Scenario A: Interference source filtering (the passing of the robot vacuum cleaner).

[0132] Status: The central control screen is in off state, camera 20 is off, and radar sensor 10 is polling at low power.

[0133] Detection (corresponding to steps S106-S110): The radar detects an object moving laterally at a constant speed of 0.3 m / s at a distance of 0.1 meters from the ground.

[0134] Feature extraction: The radar's internal DSP extracts the micro-Doppler features of the target (step S107), and finds that its spectral bandwidth is extremely narrow (rigid body motion) and lacks the rhythmicity of limb swinging (step S108, no Tcycle features).

[0135] Judgment (corresponding to step S203): Calculate the matching degree between its feature fingerprint and the pre-stored robot vacuum cleaner interference library, and obtain the comprehensive judgment score > high judgment threshold.

[0136] Result: The radar determined the interference to be non-human interference and did not send a wake-up signal to the processor. The screen remained black, and the system power consumption remained at the mW level.

[0137] Scenario B: User wakes up upon returning home (high-speed capture and recognition).

[0138] Status: The user is walking towards the central control screen from a distance of 3 meters, at a speed of approximately 1.2 m / s.

[0139] Wake-up and prediction (corresponding steps S2 / S301): The radar determines a potential human body and wakes up the CPU. The position prediction module 31 calculates the user's position after the system startup delay (approximately 150ms), at which point the user has moved to a distance of 2.8 meters.

[0140] ISP parameter feedforward (corresponding steps S302 / S502): Since the living room is in a backlit environment and the user moves quickly, the parameter mapping module 32 directly sets the upper limit of the camera's exposure time to 1 / 200 second (anti-momentation) based on a speed of 1.2m / s, and locks the position of the focus motor based on the distance.

[0141] ROI cropping and recognition (corresponding steps S403 / S602): The camera starts up, and the first frame image is transmitted directly. The CPU, based on the radar coordinates, only crops a 224x224 region (ROI) in the center of the image and sends it to the lightweight CNN network.

[0142] Result: The CNN outputs a Human confidence score of 0.92 (step S604). The screen lights up momentarily before the user arrives, displaying a welcome message.

[0143] Feedback (corresponding to step S8): The system records the characteristics of the radar signal and updates the radar sensitivity parameters in a positive direction.

[0144] II. Experimental verification and effect comparison.

[0145] To verify the advantages of the present invention (Solution C) over the traditional solution, we conducted a comparative test in a standard laboratory environment (simulating a living room, with an illuminance of 300 Lux, including interference from a robot vacuum cleaner and an electric fan).

[0146] 1. Comparison group settings.

[0147] Option A (pure radar solution): Only radar sensing is used; the screen will wake up and light up when movement is detected.

[0148] Option B (Pure Vision Always On Solution): The camera is always on at a low frame rate for human detection.

[0149] Scheme C (Invention Fusion Scheme): Radar Micro-Doppler Pre-screening + Visual ROI Linkage + ISP Parameter Feedforward.

[0150] 2. The experimental data results are shown in Table 1.

[0151] Table 1

[0152] 3. Conclusion.

[0153] While retaining the advantages of ultra-low power consumption of radar, the present invention solves the false alarm problem through visual fusion; and through the unique ISP parameter feedforward and coordinate mapping ROI technology, the recognition rate of the first frame image after cold start is greatly improved, solving the pain points of slow wake-up and blurry capture in traditional fusion solutions.

Claims

1. A human body recognition method based on the fusion of radar sensing and RGB camera, characterized in that the method include: The radar sensor (10) analyzes the target state data and micro-Doppler feature fingerprint vector based on the detection echo, compares the micro-Doppler feature fingerprint vector with the pre-stored interference feature library, and if it determines that the target is not a known interference source, it generates a wake-up signal and sends it to the processor (30). The processor (30) responds to the wake-up signal, calculates the predicted coordinate position of the target at the exposure time in combination with the system delay, and before starting the camera (20), maps the predicted coordinate position to the metering area parameter, converts the target speed into the upper limit parameter of the exposure time and sends it down; The camera (20) collects and transmits image data based on the metering area parameters and the upper limit of exposure time parameters. The processor (30) uses a neural network to classify the image data: if the classification result is non-human, it triggers the interference suppression logic and writes the current micro-Doppler feature fingerprint vector into the interference feature library; if the classification result is human, it executes the service response.

2. The human body recognition method based on radar sensing and RGB camera fusion according to claim 1, characterized in that, The process of analyzing the target state data and micro-Doppler feature fingerprint vector specifically includes the following steps: A multidimensional Fourier transform is performed on the received echo signal to obtain a range Doppler map. The target point is extracted using a constant false alarm rate (CFAR) detection algorithm. The target's distance, velocity, and angle are calculated based on the phase difference of the antenna array to generate the target state data. Locate the distance cell where the target point is located, perform a short-time Fourier transform on the slow-time dimension complex signal sequence of the distance cell, and generate a time-frequency distribution map; Traverse the discrete time frames of the time-frequency distribution map, extract the highest and lowest frequency points with energy intensity higher than the noise floor threshold, and fit them to form an upper envelope and a lower envelope. Calculate the mean Doppler bandwidth between the upper envelope and the lower envelope, and perform autocorrelation operation on the upper envelope sequence to calculate the envelope period; The mean Doppler bandwidth, the envelope period, and the standard deviation of the upper envelope are used to construct the micro-Doppler feature fingerprint vector, thereby completing the analysis of the target state data and the micro-Doppler feature fingerprint vector.

3. The human body recognition method based on radar sensing and RGB camera fusion according to claim 1, characterized in that, The process of comparing the micro-Doppler feature fingerprint vector with the pre-stored interference feature library specifically includes: Calculate the weighted Euclidean distance between the micro-Doppler feature fingerprint vector and the reference feature vector in the interference feature library, and select the optimal matching distance; The optimal matching distance is converted into a comprehensive decision score based on a nonlinear mapping mechanism; The comprehensive decision score is determined using a dual-threshold hysteresis comparison logic: if the comprehensive decision score is less than the low decision threshold, the monitoring target corresponding to the micro-Doppler feature fingerprint vector is determined to be a potential human body and the wake-up signal is allowed to be generated. If the comprehensive decision score is greater than the high decision threshold, the monitoring target is determined to be a known interference source. The determination result of keeping the camera (20) in a closed or dormant state is used as the comparison result between the micro-Doppler feature fingerprint vector and the interference feature library.

4. The human body recognition method and apparatus based on radar sensing and RGB camera fusion according to claim 1, characterized in that, The process of calculating the predicted coordinates of the target at the exposure time of the camera (20) specifically includes the following steps: The time difference between the radar interruption trigger time and the expected image sensor exposure time is used as the total system delay. The original radial distance in the target state data is added to the product of the target's radial velocity and the total system delay to obtain the predicted radial distance at the imaging time. Using a perspective projection model, the predicted radial distance and the azimuth of the target are converted into predicted pixel coordinates in the image coordinate system, and the predicted pixel coordinates are used as the predicted coordinate position of the target at the exposure time of the camera (20).

5. The human body recognition method and apparatus based on radar sensing and RGB camera fusion according to claim 1, characterized in that, The process of converting the target speed into the upper limit parameter of exposure time specifically includes the following steps: Determine whether the absolute value of the target speed is greater than a preset minimum speed threshold; If so, the upper limit of dynamic exposure time is calculated based on the maximum number of motion blur pixels threshold, the physical size of the image sensor pixels, the equivalent physical focal length of the camera lens, and the target speed. The upper limit of dynamic exposure time is inversely proportional to the target speed. If not, then obtain the preset default exposure time limit value; The processor (30) writes the calculated dynamic exposure time limit or the obtained default exposure time limit value into the exposure limit register of the camera (20) and converts it into the exposure time limit parameter.

6. The human body recognition method and apparatus based on radar sensing and RGB camera fusion according to claim 1, characterized in that, Before activating the camera (20), the processor (30) also performs a feedforward configuration of the focus position, specifically including the following steps: An inverse mapping model is established between distance and lens voice coil motor drive value. The inverse mapping model includes optical sensitivity coefficient and reference value when the lens is focused at infinity. Using the distance value in the target state data, the target drive value of the voice coil motor is calculated based on the inverse mapping model; The processor (30) writes the target driving value into the camera (20) through the bus interface and performs open-loop focus control as a feedforward configuration for the focus position.

7. The human body recognition method and apparatus based on radar sensing and RGB camera fusion according to claim 1, characterized in that, The process of classifying image data using a neural network specifically includes the following steps: The image data is input into a convolutional neural network containing depthwise separable convolutional layers and an attention mechanism module, and the confidence score of the current frame belonging to the human body category is output. The final decision output signal is generated using hysteresis comparison logic: when the confidence level is greater than the high threshold for judgment, the classification result is confirmed as human. When the confidence level is less than the low threshold for judgment, the classification result is confirmed as non-human. When the confidence level is between the high decision threshold and the low decision threshold, the decision state of the previous frame is maintained, and the final decision output signal is used as the result of classifying the image data using the neural network.

8. The human body recognition method and apparatus based on radar sensing and RGB camera fusion according to claim 6, characterized in that, The triggering interference suppression logic specifically includes the following steps: Calculate the spatial depth corresponding to the target recognition box in the image data and compare it with the distance value detected by the radar sensor (10) to obtain the consistency deviation value; If the consistency deviation value is less than the preset spatial tolerance threshold, the distance and velocity values ​​of the target at the current moment are extracted, and a dynamic shielding interval is generated in the radar range and velocity phase space. The shielded area is sent to the radar sensor (10) so that when the radar sensor (10) filters signals falling into the shielded area in subsequent monitoring, it will no longer trigger the wake-up signal. The operation of sending the shielded area to the radar sensor (10) is used as the triggering interference suppression logic.

9. The human body recognition method and apparatus based on radar sensing and RGB camera fusion according to claim 6, characterized in that, The method further includes performing online calibration logic, specifically comprising the following steps: Obtain the actual focus position of the camera (20) after autofocus convergence; Calculate the prediction deviation between the actual focus position and the target drive value; The reference value in the inverse mapping model is updated with weighted values ​​using the prediction bias, and the process of updating the reference value with weighted values ​​is used as the logic for performing online calibration.

10. The human body recognition device based on radar sensing and RGB camera fusion according to claim 1, applied to the human body recognition method based on radar sensing and RGB camera fusion according to any one of claims 1-9, characterized in that, include: Radar sensor (10) is used to transmit detection signals and receive reflected signals, analyze micro-Doppler feature fingerprint vectors and compare them with internal interference feature library, and generate wake-up signals after filtering known interference; The camera (20), which is communicatively connected to the processor (30), is used to acquire image data of the monitored area in response to configuration instructions and start enable signals; The processor (30) is communicatively connected to the radar sensor (10) and the camera (20), respectively, and is used to execute a human body recognition method based on the fusion of radar sensing and RGB camera.

Images