Multi-mode sensing fusion edge intelligent control method and system

Abstract

The invention relates to the technical field of multi-mode perception, and discloses a multi-mode perception fused edge intelligent control method and system, and the method comprises the steps: carrying out the frequency calibration of RGB-D camera data, laser radar data and ultrasonic sensor data, and obtaining a first sensor data set; performing timestamp synchronization based on the first sensor data set to obtain a second sensor data set; carrying out adaptive time window adjustment and synchronous optimization on the second sensor data set according to a detection result of the movement speed of the old person to obtain a third sensor data set; and navigation path calculation is performed based on the third sensor data set, and a navigation control instruction of the intelligent old-age care robot is output, so that the heterogeneous frequency data matching precision is improved, and a personalized navigation decision is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal perception, in particular to a multi-modal perception fusion edge intelligent control method and system. BACKGROUND

[0002] An intelligent elderly care robot needs to accurately identify the behavior state of an old person in a complex indoor environment, respond to the care needs of the old person in real time, and provide safe and reliable navigation services, which puts high technical requirements on the multi-modal perception ability and edge computing response speed of the robot. However, existing elderly care robots generally have problems such as insufficient perception accuracy, high response delay, poor environmental adaptability, and the like, and are difficult to meet the strict requirements of the elderly on safety and real-time performance. SUMMARY

[0003] The present application provides a multi-modal perception fusion edge intelligent control method and system, thereby improving the heterogeneous frequency data matching accuracy and realizing personalized navigation decision-making.

[0004] The first aspect of the present application provides a multi-modal perception fusion edge intelligent control method, which comprises: Frequency calibration is performed on RGB-D camera data, laser radar data and ultrasonic sensor data to obtain a first sensor data set; Timestamp synchronization is performed based on the first sensor data set to obtain a second sensor data set; According to the old person's moving speed detection result, the second sensor data set is adaptively adjusted and synchronized to obtain a third sensor data set; Based on the third sensor data set, a navigation path is calculated, and a navigation control instruction of an intelligent elderly care robot is output.

[0005] In combination with the first aspect, in a first implementation manner of the first aspect of the present application, the frequency calibration of the RGB-D camera data, the laser radar data and the ultrasonic sensor data to obtain the first sensor data set comprises: Heterogeneous frequency characteristic analysis is performed on the RGB-D camera data, the laser radar data and the ultrasonic sensor data to obtain sensor frequency characteristic parameters; A frequency ratio constraint equation set is constructed according to the sensor frequency characteristic parameters, and a timestamp calibration integer programming model is established based on the frequency ratio constraint equation set; Optimization is solved based on the timestamp calibration integer programming model to obtain a frequency calibration parameter set; The RGB-D camera data, the laser radar data and the ultrasonic sensor data are hierarchically timestamp interpolated according to the frequency calibration parameter set, to obtain a first sensor data set.

[0006] In a second implementation manner of the first aspect, the frequency proportion constraint equation set is constructed according to the sensor frequency characteristic parameters, and the timestamp calibration integer programming model is established based on the frequency proportion constraint equation set, including: The frequency proportion relationship parameters are calculated according to the sensor frequency characteristic parameters, including an RGB-D camera frequency, a laser radar frequency and an ultrasonic sensor frequency. The frequency proportion constraint equation set is constructed based on the frequency proportion relationship parameters, and the timestamp calibration objective function is established according to the frequency proportion constraint equation set. The timestamp calibration integer programming model is constructed based on the timestamp calibration objective function and the frequency proportion constraint equation set, to obtain the timestamp calibration integer programming model.

[0007] In a third implementation manner of the first aspect, the RGB-D camera data, the laser radar data and the ultrasonic sensor data are hierarchically timestamp interpolated according to the frequency calibration parameter set, to obtain a first sensor data set, including: The sensor frequency density weight parameters are calculated according to the frequency calibration parameter set, including the RGB-D camera data, the laser radar data and the ultrasonic sensor data. The laser radar interpolation data are calculated according to the sensor frequency density weight parameters, including the laser radar data. The high-frequency sensor interpolation data are calculated according to the sensor frequency density weight parameters, including the RGB-D camera data and the ultrasonic sensor data. The first sensor data set is obtained by hierarchically integrating the laser radar interpolation data and the high-frequency sensor interpolation data.

[0008] In a fourth implementation manner of the first aspect, the timestamp synchronization is performed based on the first sensor data set, to obtain a second sensor data set, including: The first timestamp loop configuration is obtained by performing timestamp sequence loop initialization on the first sensor data set. The timestamp deviation optimization objective function is established according to the first timestamp loop configuration, and the second timestamp loop configuration is obtained by performing local search based on the timestamp deviation optimization objective function. performing time stamp neighborhood search and best synchronization sequence generation based on the second time stamp loop configuration to obtain a second sensor data set.

[0009] With reference to the first aspect, in a fifth implementation form of the first aspect of the present application, the establishing a time stamp deviation optimization objective function according to the first time stamp loop configuration, and performing local search based on the time stamp deviation optimization objective function to obtain a second time stamp loop configuration, comprises: performing deviation calculation based on adjacent time stamp nodes in the first time stamp loop configuration to obtain a time stamp deviation optimization objective function; performing exchange operation on any two edges in the first time stamp loop configuration according to the time stamp deviation optimization objective function to obtain a candidate time stamp loop configuration; performing deviation improvement amount calculation on the candidate time stamp loop configuration based on the time stamp deviation optimization objective function to obtain a time stamp deviation improvement amount parameter; performing threshold value judgment and loop configuration selection based on the time stamp deviation improvement amount parameter to obtain a second time stamp loop configuration.

[0010] With reference to the first aspect, in a sixth implementation form of the first aspect of the present application, the performing adaptive time window adjustment and synchronization optimization on the second sensor data set according to the old person moving speed detection result to obtain a third sensor data set, comprises: performing old person moving speed calculation based on the RGB-D data between continuous frames in the second sensor data set to obtain an old person moving speed detection result; performing hierarchical time window length matching according to the old person moving speed detection result to obtain a basic time window length parameter; performing old person abnormal behavior detection and emergency time window adjustment based on the second sensor data set to obtain an adjusted time window length parameter; inputting the adjusted time window length parameter into an adaptive time window generation module to perform sensor data reorganization to obtain a window adjusted sensor data set; performing hybrid evolutionary multi-frequency synchronization optimization based on the window adjusted sensor data set to obtain a third sensor data set.

[0011] With reference to the first aspect, in a seventh implementation form of the first aspect of the present application, the performing hybrid evolutionary multi-frequency synchronization optimization based on the window adjusted sensor data set to obtain a third sensor data set, comprises: performing heuristic rule task decomposition based on the window adjusted sensor data set to obtain visual high-frequency sub-task data, laser medium-frequency sub-task data and ultrasonic high-frequency sub-task data; According to the visual high-frequency sub-task data, the laser medium-frequency sub-task data and the ultrasonic high-frequency sub-task data, genetic algorithm population initialization and evolution optimization are respectively performed to obtain sub-task optimization population data; Based on the sub-task optimization population data, optimal individual migration and information exchange are performed to obtain evolution synchronization optimization population data; The evolution synchronization optimization population data is subjected to convergence judgment and multi-frequency synchronization parameter processing to obtain a third sensor data set.

[0012] In combination with the first aspect, in an eighth implementation manner of the first aspect of the application, the navigation path calculation based on the third sensor data set is performed to output the navigation control instruction of the intelligent elderly care robot, including: Based on the third sensor data set, an old person behavior state recognition is performed to obtain an old person behavior state parameter; According to the old person behavior state parameter, a behavior weight distribution is performed on the third sensor data set to obtain a sensor behavior weight fusion parameter; Based on the sensor behavior weight fusion parameter, a weighted Bayesian position estimation is performed to obtain an old person position fusion result; Based on the old person position fusion result, a behavior preference path planning is performed to obtain an edge navigation path parameter, and based on the edge navigation path parameter, a motion control is performed to output the navigation control instruction of the intelligent elderly care robot.

[0013] The second aspect of the application provides a multi-modal perception fusion edge intelligent control system, which comprises: A calibration module is configured to perform frequency calibration on RGB-D camera data, laser radar data and ultrasonic sensor data to obtain a first sensor data set; An optimization module is configured to perform timestamp synchronization based on the first sensor data set to obtain a second sensor data set; An adjustment module is configured to perform adaptive time window adjustment and synchronization optimization on the second sensor data set according to an old person moving speed detection result to obtain a third sensor data set; An output module is configured to perform navigation path calculation based on the third sensor data set to output the navigation control instruction of the intelligent elderly care robot.

[0014] Compared with the prior art, the application has the following beneficial effects: by establishing a 2.5:1:4 frequency ratio constraint equation set for the RGB-D camera 25 Hz, the laser radar 10 Hz and the ultrasonic sensor 40 Hz, the heterogeneous frequency characteristics are first introduced into the integer programming model as a hard constraint condition, the heterogeneous frequency characteristic analysis technology can accurately identify the working frequency difference of different sensors, the hierarchical interpolation algorithm based on the frequency density weight mechanism can select the most suitable interpolation strategy according to the sensor frequency characteristics, and the matching accuracy of the heterogeneous frequency data is significantly improved; the cubic spline interpolation algorithm is specially designed for low-frequency laser radar data for time stamp compensation, and the smoothness and continuity of the time sequence are maintained, which is better than the traditional linear interpolation method; the high-frequency sensor adopts frequency density weighted interpolation, which fully utilizes the time resolution advantage of high-frequency data. The spatial distance optimization of the traditional TSP problem is innovatively changed to the minimum timestamp deviation optimization, which is specially designed for time domain problems; combined with the timestamp neighborhood search mechanism, the local optimal solution of timestamp synchronization can be quickly converged; compared with the traditional spatial path optimization, the time window length is dynamically adjusted according to the characteristics of the slow movement speed and the relatively fixed behavior mode of the old people, which not only ensures the data synchronization accuracy but also optimizes the utilization of computing resources; the abnormal behavior detection mechanism can immediately shorten the time window when detecting the fall or sharp movement of the old people, thereby improving the response speed in emergency situations; compared with the fixed time window method, the adaptive strategy can optimize the time synchronization accuracy of edge computing according to the specific behavior characteristics of the old people. The heuristic rule divides the heterogeneous frequency calibration task into three parallel sub-tasks, which significantly improves the processing efficiency compared with the traditional serial processing mode; each sub-task independently runs the genetic algorithm, and the optimal individual migration mechanism is used to realize information exchange and collaborative optimization; the parallel evolution strategy can simultaneously consider the frequency characteristic differences of multiple sensors to realize more accurate timestamp synchronization. The behavior characteristics of the old people are used as the data fusion weight, the sensor fusion strategy is dynamically adjusted according to different behavior states, and personalized navigation decision is realized; compared with the fixed weight fusion, the weighted Bayesian estimation method can optimize the position estimation accuracy according to the actual behavior state of the old people; the distance cost, behavior preference cost and safety cost are comprehensively considered, and the optimization is specially designed for the old people care scene; the fused position estimation and target position are converted into smooth robot motion instructions. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] The structures, proportions, sizes, etc. shown in the drawings of the specification are merely used to cooperate with the content disclosed in the specification, to enable those skilled in the art to understand and read, and are not used to limit the conditions that the application can be implemented, so they do not have technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, which does not affect the effects that the application can produce and the purposes that the application can achieve, should still fall within the scope of the technology disclosed by the application.

[0017] Figure 1 is a flowchart of the edge intelligent control method of multi-modal perception fusion provided by the embodiment of the application; Figure 2 is a structural schematic block diagram of the edge intelligent control system of multi-modal perception fusion provided by the embodiment of the application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0019] The flowchart shown in the drawings is only an example description, and does not necessarily include all the contents and operations / steps, nor does it necessarily execute in the order described. For example, some operations / steps can be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

[0020] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in the specification and the appended claims of the application, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0021] It should be further understood that the term "and / or" used in the specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. Please refer to Figure 1 One embodiment of the multi-modal perception fusion edge intelligent control method in the embodiments of the application includes: Step 100, frequency calibration is performed on RGB-D camera data, laser radar data and ultrasonic sensor data to obtain a first sensor data set; It can be understood that the execution subject of the present application can be an edge intelligent control system for multi-modal perception fusion, and can also be a terminal or a server, which is not limited here. The server is taken as an example for description in the embodiments of the present application.

[0022] Specifically, the heterogeneous frequency characteristics of RGB-D camera data, laser radar data and ultrasonic sensor data are analyzed. The number of valid data frames generated by each sensor per unit time is collected and counted on the edge side, so as to extract the working frequency of each type of sensor, wherein the frequency of the RGB-D camera is stable at 25 Hz, the frequency of the laser radar data is 10 Hz, and the ultrasonic sensor has a relatively higher data sampling frequency of about 40 Hz. According to the above analysis results, a frequency characteristic parameter set is constructed to clearly define the distribution law and sampling sparsity of each type of perception module in the time domain, thereby forming a structured parameter representation describing the frequency density difference. A frequency ratio constraint equation set is constructed according to the frequency characteristic parameters of the sensors. The simplest integer ratio relationship between the three frequency sources, i.e. 2.5:1:4, respectively corresponding to RGB-D, laser radar and ultrasonic sensor, forms a frequency consistency constraint equation set; on this basis, time window constraint and data integrity constraint are introduced to ensure that in any sliding window, there is at least one available data frame for each type of sensor, and the maximum delay does not exceed a preset threshold (such as 200 milliseconds) to maintain a balance between real-time and integrity. The above multiple constraints are embedded into an integer programming model, so that the model objective not only optimizes the alignment accuracy, but also includes strong constraint management of frequency coordination and time consistency. The objective function of the model is designed to minimize the sum of the absolute deviations between the current timestamps of each sensor and the reference timestamps, i.e. to compress the discrete error of the sensor sampling points on the time axis. Based on the integer programming model, a global optimization solving process is performed, and a mixed integer linear programming algorithm or a heuristic optimization algorithm is used to iteratively approximate the timestamp alignment solution space, and search for the alignment scheme with the minimum global deviation under the strict frequency ratio constraint. In this process, the optimization solution not only outputs the theoretical optimal timestamp matching solution vector, but also generates a set of frequency calibration parameter sets describing the offset of each type of sensor relative to the reference time. Specifically, the frequency calibration parameter set is taken as the basic input to quantitatively evaluate the frequency density of the three types of heterogeneous sensor data, and the interpolated laser radar data and the interpolated high-frequency sensor data are hierarchically aggregated and integrated according to their respective frequency levels to form a unified first sensor data set.

[0023] Step 200, performing timestamp synchronization based on the first sensor data set to obtain a second sensor data set; Specifically, the timestamp sequence in the first sensor dataset is organized and initially sorted. A closed loop is constructed by using the timestamp of each interpolated data point as a path node. An initial traversal order is set as the first timestamp loop configuration. This configuration provides a starting point for optimization iterations in the time dimension. This initial path is generated according to strategies such as time sequence, frequency density priority, or behavior trigger weight priority, thus giving the optimization process a certain convergence direction. A timestamp deviation optimization objective function is established based on the first timestamp loop configuration. The absolute value of the time difference between any adjacent data points in the entire time path is used as the evaluation index, and the optimization objective is to minimize the total deviation. This objective function is similar in form to minimizing the path length in the classic Traveling Salesman Problem, but in this embodiment, the optimization object is transformed into the offset distance between timestamps of multiple sensors. Therefore, the timestamp optimization path not only concerns the difference between nodes but also reflects the time coordination between data sources. Guided by the objective function, a local path search operation is performed, employing a 2-opt optimization mechanism similar to the Lin-Kernighan improvement strategy. This involves randomly selecting two connecting edges in the current path and attempting to swap their connections to form a new path sequence. After evaluating whether the new path reduces the overall time deviation, a decision is made whether to accept the new configuration, and this locally updated loop is used as the second timestamp loop configuration. To overcome local extrema and improve synchronization accuracy, timestamp neighborhood search and optimal sequence generation operations are performed based on the second timestamp loop configuration. The neighborhood search mechanism prioritizes reorganizing data pairs with similar time values, thereby improving the synchronization quality of potential pairings. A search criterion based on synchronization error gradient descent is introduced, allowing for non-optimal steps within a certain threshold range to enhance the algorithm's ability to escape local minima. After multiple rounds of neighborhood search iterations, a second sensor dataset is output based on the current optimal time alignment result. This dataset achieves minimum deviation pairing between RGB-D cameras, LiDAR, and ultrasonic sensors on the time axis, exhibiting high time consistency, high data alignment, and low heterogeneous latency.

[0024] Step 300: Based on the elderly person's movement speed detection results, perform adaptive time window adjustment and synchronization optimization on the second sensor dataset to obtain the third sensor dataset; Specifically, the velocity of the elderly person's current movement is extracted based on RGB-D image data between consecutive frames in the second sensor dataset. At the edge, by analyzing the displacement changes between the RGB images corresponding to consecutive timestamps and depth information, the trajectory of the elderly person's center of mass in the three-dimensional coordinate system is extracted. Multiple consecutive frames are then smoothed to obtain a relatively stable and physically meaningful result for detecting the elderly person's movement velocity, reflecting different activity states such as stillness, slow movement, or rapid movement. The obtained movement velocity is matched to a time window length according to a set velocity classification rule. When the detected velocity is below 0.5 m / s, it is determined to be a still or low-speed movement stage, and the time window is set to a longer period, such as 300 ms, to gather more data frames and improve fusion redundancy. When the velocity is between 0.5 m / s and 1.0 m / s, the elderly person is considered to be in a moderate activity state, and the time window is shortened to 200 ms to ensure real-time synchronization. When the velocity exceeds 1.0 m / s, the window is adjusted to 100 ms to reduce the risk of delayed perception. This yields the basic time window length parameter. Based on the second sensor dataset, abnormal behavior detection in the elderly is performed by monitoring rapid changes in posture or dramatic shifts in body contours in RGB-D images. Combined with acceleration calculation methods, when an acceleration greater than 2 m / s² is detected... 2 If the posture rapidly transitions to a falling mode, an emergency response strategy is triggered, significantly shortening the current base window length to a minimum of 50ms to ensure high-frequency response capabilities in unexpected situations. This results in an adjusted time window length parameter that incorporates both routine and emergency response mechanisms. This adjusted time window length parameter is input into an adaptive time window generation module, which reorganizes synchronized multi-source sensor data based on the new time window boundaries, constructing a window-adjusted sensor dataset through a sliding window strategy and data resampling mechanism. In this dataset, various sensor data are re-paired and rearranged within the newly defined time period, forming a window-based data structure to ensure local consistency and real-time response capabilities in the time domain. To improve the quality of temporal coordination under non-equal frequency conditions, a hybrid evolutionary multi-frequency synchronization optimization algorithm is run on the window-adjusted dataset. Independent genetic optimization processes are constructed for the visual high-frequency subtask, the lidar mid-frequency subtask, and the ultrasonic high-frequency subtask, respectively, and an information transfer mechanism is introduced to enhance the interaction and sharing of local optimal solutions. The objective function for each subtask takes into account both time synchronization accuracy and response delay penalty. The optimal synchronization parameters are generated through continuous multi-generation evolution and fitness convergence detection, and finally the third sensor dataset is obtained.

[0025] Step 400: Calculate the navigation path based on the third sensor dataset and output the navigation control commands for the intelligent elderly care robot.

[0026] Specifically, behavioral perception analysis is performed on the third sensor dataset. By integrating and analyzing multi-source information such as RGB-D image sequences, LiDAR point clouds, and ultrasonic ranging, the behavioral state of the elderly at the current moment is identified. This identification process relies on a behavior recognition model embedded in the edge computing node. This model takes continuous frame feature changes as input and combines multiple feature dimensions such as pose estimation, motion trajectory, spatial displacement, and image region changes to output the elderly's current behavioral state parameters in real time. These parameters represent one class from a predefined finite set of behaviors, such as "walking," "standing," "sitting," "lying down," or "emergency / abnormality." Behavioral weights are assigned to the third sensor dataset based on the elderly's behavioral state parameters, that is, the contribution of the three types of sensor data to the fusion process is dynamically adjusted according to the behavioral state. For example, when an elderly person is walking, the precise environmental structure information provided by LiDAR is more valuable for obstacle avoidance and path determination, thus its weight will be significantly increased. When the elderly person is sitting, the RGB-D camera's ability to capture static posture in detail is more crucial, and its weight will be increased accordingly. When the elderly person is in an abnormal or emergency situation, to ensure fusion redundancy and robustness, the fusion weights of the three types of sensors will be set to approximately equal weights to maximize the balance of information failure risk, thus forming sensor behavior weight fusion parameters. Based on the above weight configuration results, a weighted Bayesian position estimation algorithm is initiated. In each fusion cycle, the position information observed by different sensors is used as a conditional probability input, combined with the behavioral state as a prior modulation factor, to estimate the credible spatial region of the current elderly person's position in a way that maximizes the posterior probability, and the position fusion result is output. Based on the position fusion, a path planning operation is performed in conjunction with the elderly person's current behavioral preferences. Specifically, an improved A* navigation algorithm with behavioral cost weights is adopted. The path cost function considers traditional distance cost and introduces behavioral constraint cost and safety buffer cost. For example, when the elderly person is standing or walking, shorter paths are allowed to traverse the activity area; while when lying down or fallen, path planning prioritizes avoiding the area where the elderly person is located and increases the safety distance. Simultaneously, the planning results are converted into an edge navigation path parameter set, containing multi-dimensional path data such as path point sequences, behavioral state weights, and turning point prompts. Based on the navigation path parameter set, a fuzzy logic controller is activated, calculating navigation control quantities based on the distance deviation between the current position and the target path point, changes in direction and angle, and trends in the elderly person's behavioral state. This results in real-time output of control commands to drive the elderly care robot forward, turn, decelerate, or stop.

[0027] In one specific embodiment, the process of performing step 100 may specifically include the following steps: Heterogeneous frequency characteristic analysis was performed on RGB-D camera data, LiDAR data, and ultrasonic sensor data to obtain sensor frequency characteristic parameters; A set of frequency ratio constraint equations is constructed based on the sensor frequency characteristic parameters, and a timestamp calibration integer programming model is established based on the set of frequency ratio constraint equations. The frequency calibration parameter set is obtained by performing an optimization solution based on the timestamp calibration integer programming model. Based on the frequency calibration parameter set, the RGB-D camera data, LiDAR data, and ultrasonic sensor data are subjected to hierarchical timestamp interpolation to obtain the first sensor dataset.

[0028] Specifically, based on the multimodal perception system of the intelligent elderly care robot, the raw data streams of the three types of sensor data sources—RGB-D camera, LiDAR, and ultrasonic sensor—are analyzed. By statistically analyzing the number of effective data frames per unit time, the independent operating frequencies of each sensor are extracted. Frequency confidence intervals are then fitted based on the stability of the statistical distribution to obtain frequency characteristic parameters reflecting the sensor's frequency performance under typical operating conditions. Specifically, the RGB-D camera uses structured light or time-of-flight technology to acquire depth information, with a relatively fixed frame rate and a standard operating frequency of 25Hz; the LiDAR outputs data based on the beam rotation period, with a frequency of 10Hz; and the ultrasonic sensor uses periodic pulse echo sampling, with a frequency reaching 40Hz, exhibiting high-density, short-period ranging information. Based on this, the frequencies of the three sensors are normalized and their simplest integer ratios are calculated to form a vector representing the frequency ratio. In this example, the frequency ratio is 2.5:1:4, which is further transformed into an integer relationship of 10:4:16 as a hard structural parameter in the subsequent model, constituting the basic expression of the frequency ratio constraint. To establish an optimization framework that constrains the time alignment process for various types of data, a frequency constraint expression group is constructed based on the aforementioned frequency ratio relationship. This expression group takes minimizing timestamp alignment error as its core objective, defining the goal as minimizing the sum of the absolute values ​​of the deviations between the timestamps of various sensors and the unified reference time, and constructing a unified time alignment logic throughout the entire observation period. Simultaneously, frequency ratio constraints are introduced into the expression logic with a fixed-ratio time mapping relationship. For example, the number of RGB-D data points per unit time window is set to be 2.5 times that of LiDAR, and the number of ultrasonic data points is set to be 4 times that of LiDAR, mapped to a resolvable data distribution structure through an integer structure. This integer programming model not only includes frequency constraints but also introduces time window integrity constraints. That is, within any specified sliding time period, each type of sensor must contribute at least one valid data point, thereby ensuring the structural integrity and information continuity of time alignment and avoiding irrecoverable breakpoint errors in synchronization accuracy caused by the loss of low-frequency data. Furthermore, the model is extended to introduce a redundancy tolerance threshold, for example, by relaxing the lower limit of data coverage density for certain low-importance sensors under extreme conditions, improving the model's practical robustness and operational stability. After setting the frequency and window integrity constraints, the structural optimization problem is input into an integer programming solver, and a mixed-integer linear programming method is used to search for the optimal solution. In practice, this optimization process combines heuristic initialization and constraint strength adjustment strategies: on the one hand, it uses the statistical sequence of historical timestamps from sensors to construct an approximate solution as the initial structure; on the other hand, it dynamically adjusts the frequency constraint tolerance, allowing linear backfilling of non-integer frequency time offsets within certain limits, thereby guiding the optimization process to quickly converge to an approximate global optimum.During the optimization process, the algorithm outputs a data structure comprising three main parts: the optimal timestamp alignment solution vector, the synchronization deviation matrix of each type of sensor relative to the reference time base, and the set of frequency calibration parameters used for interpolation and synchronization evaluation. This parameter set clarifies the reference sampling structure of multimodal data within a unified time frame.

[0029] In one specific embodiment, the process of constructing a set of frequency proportionality constraint equations based on the sensor frequency characteristic parameters and establishing a timestamp calibration integer programming model based on the set of frequency proportionality constraint equations can specifically include the following steps: Based on the sensor frequency characteristic parameters, the proportional relationship between the RGB-D camera frequency, the lidar frequency, and the ultrasonic sensor frequency is calculated to obtain the frequency proportional relationship parameters. A set of frequency ratio constraint equations is constructed based on the frequency ratio relationship parameters, and a timestamp calibration objective function is established based on the set of frequency ratio constraint equations. An integer programming model is constructed based on the timestamp calibration objective function and the frequency ratio constraint equations to obtain the timestamp calibration integer programming model.

[0030] Specifically, the fundamental frequency characteristic parameters of three typical sensors in a multimodal sensing system are defined, and a unified cross-source ratio analysis is performed on them. During the operation of the intelligent elderly care robot, edge nodes continuously collect raw data from RGB-D cameras, LiDAR, and ultrasonic sensors. Based on the number of effective output frames per unit time period, the stable sampling frequency of the three data sources under standard operating conditions is calculated. Among them, the RGB-D camera, due to its structured light or TOF mechanism, has a stable frame rate and is less affected by image processing load, with an average frequency of 25Hz; the LiDAR outputs data in a periodic scanning manner with a rotating beam, with a typical frequency of 10Hz; and the ultrasonic sensor adopts a periodic emission and echo reception mechanism, with an output frequency of up to 40Hz. Based on the above three frequency values, the simplified integer ratio calculation is performed, resulting in a frequency ratio of RGB-D:LiDAR:ultrasonic sensor of 25:10:40. In order to construct an analytical constraint model structure, this ratio is further standardized to a minimum integer ratio relationship, i.e., 2.5:1:4, and then converted into an integer mapping form of 10:4:16. A set of frequency proportionality constraint equations is constructed based on the frequency proportionality relationship parameters, serving as the core constraint input in the model structure. This constraint means that within any observation time window, the number of data samples taken by the RGB-D camera should be 2.5 times that of the LiDAR, while the number of ultrasonic data samples should be 4 times that of the LiDAR. After quantization, this is set to 4 LiDAR frames and 16 ultrasonic frames for every 10 RGB-D frames. This constraint is used to maintain the correspondence of data structures in the time series and to control the participation weight of various data types during sensor fusion, avoiding increased alignment deviation or prolonged delay response due to frame redundancy of high-frequency sources. A timestamp calibration objective function is constructed based on the logical structure of the frequency proportionality constraint equations. This objective function aims to unify the time axis starting point and time step of various sensors to ensure minimal offset during alignment of various data points within the same time window. Using a reference time as a unified benchmark, this objective function measures the offset difference between the sampling time of RGB-D, LiDAR, and ultrasonic data and the benchmark time, and uses the sum of all offset differences as the optimization objective to minimize this difference, thereby achieving accurate synchronization of cross-modal timestamps. To improve alignment accuracy and convergence performance, an acceptable deviation threshold is set based on the statistical characteristics of historical data. Constraints are imposed on the offset difference to control fluctuations in time differences within a set range, preventing extreme individual points from causing global disturbances. The objective function described above is integrated with the frequency proportion constraint equations to jointly construct an integer programming model. This model describes the coupling relationship between the structure, constraints, and optimization objectives of the entire timestamp calibration process, thus forming the timestamp calibration integer programming model.The model aims to minimize the total time deviation and uses the frequency ratio as a structural constraint. It also introduces engineering conditions such as data integrity requirements, sliding window coverage rules, and boundary consistency to ensure the feasibility, reducibility, and edge computing adaptability of the model during the solution process.

[0031] In one specific embodiment, the process of performing step 200 may specifically include the following steps: Based on the frequency calibration parameter set, frequency density weights are calculated for the RGB-D camera data, the lidar data, and the ultrasonic sensor data to obtain sensor frequency density weight parameters. The lidar data is subjected to cubic spline interpolation based on the sensor frequency density weighting parameters to obtain lidar interpolated data. Based on the sensor frequency density weighting parameter, frequency density weighted interpolation is performed on the RGB-D camera data and the ultrasonic sensor data to obtain high-frequency sensor interpolation data. The lidar interpolation data and the high-frequency sensor interpolation data are integrated in a hierarchical manner to obtain the first sensor dataset.

[0032] Specifically, a frequency calibration parameter set is used as the mathematical benchmark for time synchronization. Based on this parameter set, the operating frequencies of various sensors are density-normalized to construct frequency density weight parameters representing the density of their temporal distribution. According to the sampling frequencies of the three sensors per unit time, the frequencies are extracted as follows: 25Hz for the RGB-D camera, 10Hz for the LiDAR, and 40Hz for the ultrasonic sensor. Using the sum of these three frequencies as a unified reference total (75Hz), each frequency value is divided by the total frequency, resulting in frequency density weights of 0.333, 0.133, and 0.533 for the RGB-D camera, LiDAR, and ultrasonic sensor, respectively. For the lower-frequency LiDAR data, cubic spline interpolation is preferentially performed because the sampling granularity of LiDAR in the time domain is relatively sparse, with large intervals between data points. Directly using linear interpolation would lead to data jumps, discontinuous temporal structures, or inaccurate target trajectories. To this end, continuous interpolation segments are constructed based on the original timestamp sequence of the LiDAR and corresponding distance or angle feature data. By constraining the boundary continuity between interpolation segments, cubic polynomial curves satisfying the boundary conditions and the continuity of the first and second derivatives are fitted within each interval. This ensures that the interpolation results have a smooth, stable, and physically consistent trajectory on the time axis, yielding LiDAR interpolated data. Using the same frequency density weighting parameters as input, frequency density weighted interpolation is performed on two high-frequency modes: RGB-D camera data and ultrasonic sensor data. Since these two types of sensors have high frame rates and small sampling point intervals, a direct interpolation strategy based on frequency density weighting is adopted. This involves setting a unified interpolation node sequence on the time axis and constructing a weighted averaging process with a dynamic smoothing coefficient based on the time interval and frequency weight relationship between the original data points and the target interpolation points. This ensures that the interpolation results fully retain high-frequency information while controlling abnormal abrupt changes caused by frequency fluctuations, thus maintaining data stability and temporal consistency, yielding high-frequency sensor interpolated data. Layered data integration is then performed on the LiDAR interpolated data and the high-frequency sensor interpolated data. Based on sensor frequency hierarchy, LiDAR is used as the mid-frequency layer, and RGB-D and ultrasound as the high-frequency layers. Under a unified timestamp index framework, the three types of interpolated data are paired and arranged according to their corresponding time points. For each standardized timestamp, the system ensures that the spatial scanning features closest to the current moment can be extracted from the interpolated LiDAR data. Simultaneously, the RGB-D image frame features and ultrasound distance data at the current time point are paired to form a data structure unit containing three-modal information. This structure unit is constructed sequentially in chronological order and written into the first sensor dataset, forming a fusion input set that is temporally continuous, has consistent frequency ratios, and complementary data types.

[0033] In one specific embodiment, the process of performing step 300 may specifically include the following steps: The first sensor dataset is initialized with a timestamp sequence loop to obtain the first timestamp loop configuration. A timestamp deviation optimization objective function is established based on the first timestamp loop configuration, and a local search is performed based on the timestamp deviation optimization objective function to obtain the second timestamp loop configuration; Based on the second timestamp loop configuration, a timestamp neighborhood search and optimal synchronization sequence generation are performed to obtain the second sensor dataset.

[0034] Specifically, a timestamp sequence loop is initialized for the first sensor dataset. An initial traversal path connecting all interpolated time nodes is constructed. During this process, all timestamp indices are linearly sorted in chronological order, and a closed loop is built, making each interpolated data point a node on the path. These nodes are connected in ascending chronological order, and the loop ends at the beginning, forming the initial time series path structure, i.e., the first timestamp loop configuration. Based on the first timestamp loop configuration, a timestamp deviation optimization objective function is established. This function uses the time difference between the timestamps of all adjacent nodes as a variable to evaluate the magnitude of the deviation in time alignment of the current path. The goal is to minimize the time interval and fluctuation between any two time-adjacent nodes in the path, thereby forming a near-synchronous shortest time path. The time intervals between all interpolated data points are accumulated, and the objective is expressed as minimizing the total time deviation, forming the evaluation criterion. Subsequently, a local path search strategy is initiated to explore structural improvements to the current timestamp loop. In each round of the search, any two non-adjacent connection segments in the path are selected, and their connection order is attempted to be swapped to form a new path structure. The change in total deviation before and after optimization is compared to determine whether to retain the modification. This optimization method is logically equivalent to the improved 2-opt strategy, and its operating mechanism ensures that each local adjustment of the path structure moves towards reducing the overall deviation. When the deviation improvement value generated by a certain round of swapping is greater than a preset threshold, the new path is adopted as the second timestamp loop configuration; when the path structure no longer shows effective deviation improvement after several consecutive rounds of optimization, the local search process terminates. Although this optimization stage is a local search, it can effectively reduce the local time misalignment between cross-modal data and enhance synchronization consistency based on the original time order. After obtaining the second timestamp loop configuration, a timestamp neighborhood search mechanism is introduced. This mechanism expands the local search range of the path, identifies a group of candidate nodes with similar times from the current path structure, and performs complex path structure transformations such as rearrangement, insertion, and intersection in the neighboring time domain to seek a better synchronization path scheme. This neighborhood search differs from the aforementioned 2-opt strategy. Its goal is to overcome the limitations of the original path structure and attempt to construct multi-node path combinations within locally time-dense segments to achieve lower global time alignment errors. To ensure computational efficiency, two adjustable parameters, search depth and search width, are set for the neighborhood search. An adaptive temperature coefficient is also introduced for step size control to prevent getting trapped in local optima. After the neighborhood search is completed, synchronization deviations are evaluated for all generated candidate path structures, and the path with the smallest total deviation is selected as the optimal synchronization sequence for the current round. Based on this, the final second sensor dataset is output.Each timestamp in this dataset corresponds to a fusion node, which contains RGB-D image information with optimal time synchronization accuracy, LiDAR spatial point cloud information, and ultrasonic distance information. The modal information is collaboratively expressed around the same time reference point in this node.

[0035] In one specific embodiment, the process of establishing a timestamp deviation optimization objective function based on the first timestamp loop configuration and performing a local search based on the timestamp deviation optimization objective function to obtain the second timestamp loop configuration can specifically include the following steps: Based on the deviation calculation of adjacent timestamp nodes in the first timestamp loop configuration, the timestamp deviation optimization objective function is obtained. Based on the timestamp deviation optimization objective function, any two edges in the first timestamp loop configuration are swapped to obtain a candidate timestamp loop configuration; Based on the timestamp deviation optimization objective function, the deviation improvement amount of the candidate timestamp loop configuration is calculated to obtain the timestamp deviation improvement amount parameter. Based on the timestamp deviation improvement parameter, threshold judgment and loop configuration selection are performed to obtain the second timestamp loop configuration.

[0036] Specifically, a time error structure analysis operation is performed based on the first timestamp loop configuration. This first loop configuration is obtained after the first sensor dataset is constructed by forming a closed time traversal path by arranging all timestamp nodes in chronological order. Each node in the path represents a fusion point for cross-modal data alignment, and the interval between any two adjacent nodes directly reflects the degree of data synchronization balance on the current time axis. To measure the synchronization rationality of this path, the time interval is calculated for all adjacent node pairs in the loop, and a unified timestamp deviation optimization objective function is established based on these interval results. This function is expressed as the sum of the cumulative deviations of the time differences of all adjacent nodes in the entire path, reflecting the overall performance of the path in terms of temporal continuity and local synchronization balance. After the objective function is established, the path structure exploration and optimization phase begins. Specifically, a local path exchange mechanism is adopted, that is, any two non-overlapping edges in the current first timestamp loop configuration are exchanged to construct a new path connection mode. This operation is similar to the boundary flipping strategy in 2-opt optimization. The logic involves randomly selecting two edges from the current loop, setting their endpoint sequences, breaking the connections in the original path, and reconnecting them to form a new closed path, thus obtaining a candidate timestamp loop configuration. This candidate path changes the order of node connections while maintaining the same node composition, achieving a more compact alignment in the time structure. The candidate loop configuration is re-evaluated based on the initially established timestamp deviation optimization objective function, calculating the change in deviation between it and the original loop configuration to obtain a timestamp deviation improvement parameter. This parameter measures the improvement in time alignment performance of the candidate path relative to the current path. Specifically, the total deviation of the current loop configuration and the candidate loop configuration is evaluated separately, and the difference between the two is compared as the improvement amount. If this improvement amount is positive and exceeds a preset lower threshold, the current candidate path is considered to have structural improvement value in time alignment. To prevent path optimization from getting stuck in local optima or resulting in invalid path replacements, a threshold judgment mechanism is implemented based on the improved parameters. This mechanism dynamically sets a lower limit threshold for deviation improvement based on factors such as the current time alignment accuracy, path convergence speed, and local perturbation redundancy. When the reduction in deviation brought by a candidate path exceeds this threshold value, the current candidate loop configuration is accepted as the officially updated path, i.e., a second timestamp loop configuration is generated. If the candidate path does not meet the threshold condition, the original path remains unchanged, and the process returns to continue trying to swap other edges. This process unfolds continuously in the path structure space, and each local structure adjustment approaches the value of the objective function towards its minimum, thereby gradually converging to a stable configuration near the current optimal solution.During the above operations, parameters such as the maximum number of attempts, the threshold for consecutive no-improvement counts, and the neighborhood exploration depth are set to avoid getting stuck in excessively long ineffective calculation cycles, while also ensuring that the optimization process has a structural termination mechanism. The entire path search and configuration switching process embodies a structural iteration idea dominated by time alignment accuracy. Without changing the data nodes themselves, the timestamp organization method is finely optimized through path structure adjustment and boundary order reconstruction, ultimately forming a second timestamp loop configuration that is more balanced and synchronous in the time domain structure.

[0037] In one specific embodiment, the process of performing step 400 may specifically include the following steps: The elderly person's movement speed is calculated based on the continuous inter-frame RGB-D data in the second sensor dataset to obtain the elderly person's movement speed detection result. Based on the detection results of the elderly's movement speed, a graded time window length matching is performed to obtain the basic time window length parameter; Based on the second sensor dataset, abnormal behavior of the elderly is detected and emergency time window is adjusted to obtain the adjustment time window length parameter. The adjustment time window length parameter is input into the adaptive time window generation module to reorganize the sensor data and obtain the window-adjusted sensor dataset. Based on the window adjustment of the sensor dataset, hybrid evolutionary multi-frequency synchronization optimization is performed to obtain the third sensor dataset.

[0038] Specifically, based on the second sensor dataset, to determine the elderly person's behavioral state in the current physical space in real time, depth change and position displacement analysis are performed on continuous frame image data in RGB-D modality to extract the skeletal coordinates or contour center of the elderly person in the target area. Three-dimensional spatial motion estimation is performed between consecutive frames, and the current movement speed of the elderly person is derived by calculating the ratio of the coordinate difference between consecutive frames to the time interval. This speed detection module can identify the state difference between minute movements and large-scale walking, and can predict the behavioral transition state through speed change trends, outputting quantitative results of the elderly person's movement speed detection. An adaptive sampling control mechanism is constructed by matching the graded time window lengths based on the speed values. This mechanism divides the elderly person's movement speed into multiple behavioral intervals. For example, when the speed is below 0.5 meters per second, it is judged as a stationary or low-speed state, and a longer observation time window, such as 300 milliseconds, is set to increase data coverage and improve information integrity. When the speed is between 0.5 meters per second and 1.0 meter per second, the elderly person is considered to be in a medium-speed walking stage, and the window length is adjusted to 200 milliseconds to balance reaction speed and data density. If the speed exceeds 1.0 meter per second, it is considered a rapid movement or sudden action state, and the window length is shortened to 100 milliseconds to enhance the sensitivity to high dynamic states. This hierarchical matching process outputs the basic time window length parameter. Considering the risk of sudden abnormal behavior in the elderly, such as falls, rapid imbalances, and abrupt turns leading to sudden increases in speed or abrupt changes in posture, an abnormal behavior detection mechanism is implemented on the synchronous dataset simultaneously with the basic time window matching. This is achieved by jointly analyzing features such as contour abrupt changes in RGB-D images, obstacle disturbances in LiDAR point clouds, and instantaneous amplitude anomalies in ultrasonic echoes. An acceleration threshold judgment and a posture angle abrupt change model are also introduced. When an abnormal event is detected, an emergency time window adjustment strategy is immediately triggered, compressing the current basic window length to a minimum threshold (e.g., 50 milliseconds or less). This rapidly refreshes the data cache and improves the edge computing system's processing frequency for sudden events. The output of this process is the adjusted time window length parameter, which includes two types of numerical configurations: a basic hierarchical window and an emergency reduced window. The adjusted time window length parameter is input into the adaptive time window generation module, which performs a reorganization of the perceived data within the edge computing framework. Under the control of the new window parameters, the time axis of the second sensor dataset is re-slid, and a multimodal observation unit including RGB-D images, LiDAR point clouds, and ultrasonic data is reconstructed using the newly set window length as the boundary. During the reorganization process, boundary alignment, data imputation, and temporal resampling are performed simultaneously to ensure that the data within each time window has structural integrity and temporal consistency, and a new window-adjusted sensor dataset is constructed. Hybrid evolutionary multi-frequency synchronization optimization is then performed based on the window-adjusted sensor dataset, and a frequency-hierarchical genetic evolution algorithm is introduced to improve the synchronization accuracy of cross-modal data.The data sources are divided into high-frequency bands (RGB-D and ultrasound) and mid-frequency bands (LiDAR), and independent optimization subtasks are constructed for each. Each subtask uses minimizing the synchronization error and time lag within a window as its fitness function, searching the parameter space through mechanisms such as genetic selection, crossover, and mutation. Simultaneously, a cross-task optimal migration mechanism is introduced during the optimization cycle, allowing high-fitness individuals to share and propagate across different frequency layers to accelerate the convergence speed of the globally optimal synchronization configuration. After several consecutive rounds of evolutionary iterations, when the error improvement between frequency layers falls below a set convergence threshold, the evolutionary process is terminated, and the optimal results of each subtask are integrated to generate a multi-frequency synchronization configuration. This configuration is written into the third sensor dataset as the final optimized output.

[0039] In one specific embodiment, the process of performing hybrid evolutionary multi-frequency synchronization optimization based on the windowed sensor dataset to obtain a third sensor dataset can specifically include the following steps: Based on the window-adjusted sensor dataset, heuristic rule-based task decomposition was performed to obtain visual high-frequency subtask data, laser mid-frequency subtask data, and ultrasonic high-frequency subtask data. Genetic algorithm population initialization and evolution optimization are performed on the visual high-frequency subtask data, the laser mid-frequency subtask data, and the ultrasonic high-frequency subtask data respectively to obtain subtask optimized population data; Based on the sub-tasks, optimize the population data to perform optimal individual migration and information exchange, and obtain evolutionarily synchronized optimized population data. The evolutionary synchronization optimization population data is subjected to convergence judgment and multi-frequency synchronization parameter processing to obtain the third sensor dataset.

[0040] Specifically, based on window-adjusted sensor datasets, an initial decomposition strategy for synchronous optimization tasks is implemented using heuristic rules as the partitioning criterion. Data from different sensor modalities are assigned to different sub-task domains, and tasks are categorized according to the temporal density, perceptual feature dimension, and synchronization sensitivity level of each modality. Specifically, visual data sources mainly refer to RGB-D cameras, with a frequency of 25Hz, belonging to the high-frequency modality. Due to their large data volume and rich variation information, they are classified as the high-frequency visual sub-task. LiDAR data, with a frequency of 10Hz, belongs to the medium-frequency range. Its point cloud data structure is relatively sparse but its spatial positioning accuracy is high, thus constituting the medium-frequency laser sub-task. Ultrasonic sensor data, with a frequency of 40Hz, has a fast response but a simple numerical structure, constituting the high-frequency ultrasonic sub-task. After task partitioning, the process enters the parallel optimization stage using a genetic algorithm, performing population initialization and evolutionary operations on the three types of sub-tasks respectively. Each subtask constructs an independent genetic algorithm execution unit. During initialization, the chromosome structure is defined as a set of time synchronization adjustment variables, each representing an interpolation correction factor or time lag compensation. The population size is set according to the task complexity; for example, the vision subtask population is set to 60 individuals, while laser and ultrasound are set to 40 and 50 individuals respectively. Initial chromosome values ​​are generated by random sampling within the historical alignment deviation range to ensure diversity in the solution space. During optimization, independent selection, crossover, and mutation operations are performed within each subtask. The fitness function is set based on the synchronization error, alignment delay, and data continuity of each modality, with synchronization error as the primary indicator, delay penalty as a secondary indicator, and data jump as a screening inhibitor, thus focusing the evolutionary objective on minimizing time consistency. After multiple rounds of independent evolution, the population data generated from the optimization of the three subtasks are processed uniformly and enter the optimal individual migration and information exchange stage. The core purpose of this stage is to build horizontal cooperative relationships between different modalities to prevent the evolution of each task from getting trapped in local optima. Every ten generations, the individual with the best fitness in the current population is migrated from one subtask to the corresponding position in another subtask, replacing the individual with the lowest fitness. During the migration, the chromosome coding structure remains unchanged; only the fitness weight calculation method is adjusted to adapt to the evaluation function of the new task. Furthermore, an information exchange mechanism allows different subtasks to call high-fitness structural segments from other tasks as crossover sources during the crossover process, thereby breaking modal isolation and improving global convergence speed. This evolutionary process integrates a hybrid strategy of local and global search, constructing a cross-modal time-synchronized optimal path structure through the collaborative migration of optimal solutions between tasks, enhancing the overall linkage and robustness of the evolutionary algorithm. After several rounds of evolution and exchange, a global convergence judgment is performed on the current evolutionarily synchronized optimization population data. Specifically, termination is determined based on whether the improvement in the best fitness over consecutive generations is lower than a set convergence threshold. For example, if the fitness improvement is less than 0.001 for 20 consecutive generations, the global solution is considered basically stable.Simultaneously, error distribution assessment and task synchronization consistency checks are performed to confirm whether the synchronization points among subtasks converge to the same time window. If the consistency constraint is not met, an additional migration and re-evolution cycle is added until the synchronization points stabilize. After the convergence judgment is completed, synchronization parameters are extracted from the best individuals in the population. The corresponding time adjustment variables in each chromosome are output as synchronization optimization parameters, labeled according to modal structure as visual synchronization parameters, laser synchronization parameters, and ultrasound synchronization parameters, and integrated into a unified multi-frequency synchronization parameter set. Guided by this parameter set, the interpolation time nodes and data alignment methods are reconfigured, and the final output third sensor dataset is the high-precision time fusion structure optimized by the evolutionary synchronization mechanism.

[0041] In one specific embodiment, the process of performing step 500 may specifically include the following steps: Based on the third sensor dataset, the behavior state of the elderly is identified to obtain the behavior state parameters of the elderly. Based on the elderly's behavioral state parameters, behavioral weights are assigned to the third sensor dataset to obtain sensor behavioral weight fusion parameters. Weighted Bayesian location estimation is performed based on the sensor behavior weight fusion parameters to obtain the elderly person's location fusion result. Based on the location fusion results of the elderly, behavioral preference path planning is performed to obtain edge navigation path parameters. Based on the edge navigation path parameters, motion control is performed to output navigation control commands for the intelligent elderly care robot.

[0042] Specifically, this method identifies the behavioral state of the elderly based on a third-sensor dataset, jointly analyzing feature changes in multimodal data to infer the current behavioral state. The identification mechanism employs a fusion model based on temporal pattern recognition and spatial posture modeling. It jointly models the skeleton posture, movement amplitude, joint velocity changes extracted from RGB-D images, and displacement trajectories and obstacle avoidance behaviors from laser point clouds. This is further validated by abrupt changes in target distance or stationary state features detected by ultrasound. A stable behavioral state classification standard is established through a combination of long-term and short-term features. The elderly person's behavioral state is output as a parameterized result and labeled as one of a finite set of states, such as "still," "walking slowly," "sitting," "lying down," "falling," and "emergency turning." This identification result is encoded as behavioral state parameters. Based on the elderly person's behavioral state parameters, behavioral sensitivity weights are assigned to the current third-sensor dataset. This allocation mechanism uses behavioral state as the criterion to dynamically adjust the contribution ratio of different modal sensing sources in the fusion calculation. For example, when the elderly person is in a "walking" state, the point cloud environmental perception provided by LiDAR is more meaningful for navigation, so the fusion weight of LiDAR data is increased to 0.5, while the weight of RGB-D is set to 0.3 and the weight of ultrasound is 0.2. If the elderly person is in a "still" or "sitting" state, the RGB-D image dominates in detail recognition and posture modeling, and its weight is increased to 0.6, ultrasound is set to 0.3, and LiDAR is compressed to 0.1. If the elderly person enters a "fall" or "emergency state," a full-modal balance strategy is adopted, and the weights of all three are set to 0.33 to enhance data redundancy and error correction capabilities. Through this behavior-driven weight allocation mechanism, a set of real-time updated sensor behavior weight fusion parameters are formed. Based on the fusion weight control, the position estimation operation is performed, and a weighted Bayesian inference mechanism is used to integrate data inputs from different modalities to estimate the maximum a posteriori probability of the elderly person's current location. This process uses modal observations as conditional inputs and behavioral states as prior guiding variables to calculate the conditional probability density functions of the elderly person's location under different modalities. These are then weighted and superimposed using fusion weights to obtain the most confident location fusion estimate. Based on the elderly person's location fusion results, a behavioral preference path planning model is constructed according to behavioral states and location coordinates, and solved in real time using edge computing nodes as the execution platform. This path planning module employs an improved A* search algorithm and introduces behavioral preference factors matching the elderly person's behavioral states into the traditional cost function. For example, when the elderly person is walking, the path will prioritize avoiding high-load areas such as corners and slopes; when the elderly person is sitting or lying down, the path planning will move as far away from the current area as possible and set up virtual safety zones to prevent the robot from approaching; if the elderly person is in an abnormal state, the path planning will combine the maximum achievable speed, path obstacle confidence, and spatial safety distance to generate the optimal avoidance scheme.During path generation, the system continuously reads the real-time updated location and behavioral state parameters of the elderly person, and readjusts the path direction and cost gradient at each time step to generate edge navigation path parameters. These parameters include multi-dimensional control quantities such as the target path point sequence, obstacle avoidance vector, behavior correction coefficient, and speed adjustment factor. The edge navigation path parameters are input into the motion control module to generate navigation control commands under multi-modal drive. The control module employs a fuzzy control strategy, which, based on the fusion of the navigation path and position deviation, adjusts the dynamic range and control response speed of the control commands according to the behavioral state, generating specific speed commands, steering angles, and acceleration / deceleration rates. These commands are then sent in real-time to the drive controller of the intelligent elderly care robot, ensuring that the robot smoothly, quickly, and flexibly completes the navigation objective while ensuring the safety of the elderly person.

[0043] The above describes the edge intelligent control method of multimodal perception fusion in the embodiments of this application. The following describes the edge intelligent control system of multimodal perception fusion in the embodiments of this application. Please refer to [link / reference]. Figure 2 One embodiment of the multimodal perception fusion edge intelligent control system in this application includes: Calibration module 21 is used to perform frequency calibration on RGB-D camera data, LiDAR data and ultrasonic sensor data to obtain the first sensor dataset; Optimization module 22 is used to perform timestamp synchronization based on the first sensor dataset to obtain the second sensor dataset; Adjustment module 23 is used to perform adaptive time window adjustment and synchronous optimization on the second sensor dataset based on the elderly movement speed detection results to obtain the third sensor dataset; Output module 24 is used to calculate the navigation path based on the third sensor dataset and output navigation control commands for the intelligent elderly care robot.

[0044] Through the collaborative efforts of the aforementioned components, a 2.5:1:4 frequency ratio constraint equation set was established for RGB-D cameras (25Hz), LiDAR (10Hz), and ultrasonic sensors (40Hz). This introduced heterogeneous frequency characteristics as hard constraints into the integer programming model for the first time. The heterogeneous frequency characteristic analysis technology can accurately identify the differences in the operating frequencies of different sensors. The hierarchical interpolation algorithm based on the frequency density weighting mechanism can select the most suitable interpolation strategy according to the sensor frequency characteristics, significantly improving the matching accuracy of heterogeneous frequency data. The cubic spline interpolation algorithm is specifically designed for timestamp compensation of low-frequency LiDAR data, maintaining the smoothness and continuity of the time series, which is superior to traditional linear interpolation methods. For high-frequency sensors, frequency density weighted interpolation is used to fully utilize the temporal resolution advantage of high-frequency data. The traditional Time-of-Sight (TSP) problem innovatively transforms spatial distance optimization into timestamp deviation minimization optimization, specifically designed for time-domain problems. Combined with a timestamp neighborhood search mechanism, it can quickly converge to a local optimum for timestamp synchronization. Compared to traditional spatial path optimization, timestamp synchronization dynamically adjusts the time window length to address the slow movement speed and relatively fixed behavior patterns of the elderly, ensuring data synchronization accuracy while optimizing computational resource utilization. An abnormal behavior detection mechanism immediately shortens the time window when a fall or sudden movement is detected, improving response speed in emergencies. An adaptive strategy, compared to fixed time window methods, optimizes the time synchronization accuracy of edge computing based on the specific behavioral characteristics of the elderly. Heuristic rules decompose the heterogeneous frequency calibration task into three parallel subtasks, significantly improving processing efficiency compared to traditional serial processing. Each subtask runs a genetic algorithm independently, achieving information exchange and collaborative optimization through an optimal individual migration mechanism. The parallel evolutionary strategy can simultaneously consider the frequency characteristics differences of multiple sensors, achieving more accurate timestamp synchronization. By using elderly behavioral characteristics as data fusion weights, the sensor fusion strategy is dynamically adjusted according to different behavioral states to achieve personalized navigation decisions. Compared with fixed-weight fusion, the weighted Bayesian estimation method can optimize the position estimation accuracy according to the actual behavioral state of the elderly. Taking into account distance cost, behavioral preference cost and safety cost, it is specifically optimized for elderly care scenarios. The fused position estimate and target position are converted into smooth robot motion commands.

[0045] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0046] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0047] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal perception fusion edge intelligent control method, characterized in that, include: Frequency calibration was performed on RGB-D camera data, LiDAR data, and ultrasonic sensor data to obtain the first sensor dataset. The second sensor dataset is obtained by synchronizing the timestamps based on the first sensor dataset; Based on the elderly person's movement speed detection results, the second sensor dataset is adaptively adjusted and synchronized to obtain the third sensor dataset. Based on the third sensor dataset, the navigation path is calculated, and navigation control commands for the intelligent elderly care robot are output.

2. The edge intelligent control method based on multimodal perception fusion according to claim 1, characterized in that, The frequency calibration of RGB-D camera data, LiDAR data, and ultrasonic sensor data yields a first sensor dataset, including: Heterogeneous frequency characteristic analysis was performed on RGB-D camera data, LiDAR data, and ultrasonic sensor data to obtain sensor frequency characteristic parameters; A set of frequency ratio constraint equations is constructed based on the sensor frequency characteristic parameters, and a timestamp calibration integer programming model is established based on the set of frequency ratio constraint equations. The frequency calibration parameter set is obtained by performing an optimization solution based on the timestamp calibration integer programming model. Based on the frequency calibration parameter set, the RGB-D camera data, LiDAR data, and ultrasonic sensor data are subjected to hierarchical timestamp interpolation to obtain the first sensor dataset.

3. The edge intelligent control method based on multimodal perception fusion according to claim 2, characterized in that, The step of constructing a set of frequency proportionality constraint equations based on the sensor frequency characteristic parameters, and establishing a timestamp calibration integer programming model based on the set of frequency proportionality constraint equations, includes: Based on the sensor frequency characteristic parameters, the proportional relationship between the RGB-D camera frequency, the lidar frequency, and the ultrasonic sensor frequency is calculated to obtain the frequency proportional relationship parameters. A set of frequency ratio constraint equations is constructed based on the frequency ratio relationship parameters, and a timestamp calibration objective function is established based on the set of frequency ratio constraint equations. An integer programming model is constructed based on the timestamp calibration objective function and the frequency ratio constraint equations to obtain the timestamp calibration integer programming model.

4. The edge intelligent control method based on multimodal perception fusion according to claim 3, characterized in that, The first sensor dataset is obtained by performing hierarchical timestamp interpolation on the RGB-D camera data, LiDAR data, and ultrasonic sensor data according to the frequency calibration parameter set, including: Based on the frequency calibration parameter set, frequency density weights are calculated for the RGB-D camera data, the lidar data, and the ultrasonic sensor data to obtain sensor frequency density weight parameters. The lidar data is subjected to cubic spline interpolation based on the sensor frequency density weighting parameters to obtain lidar interpolated data. Based on the sensor frequency density weighting parameter, frequency density weighted interpolation is performed on the RGB-D camera data and the ultrasonic sensor data to obtain high-frequency sensor interpolation data. The lidar interpolation data and the high-frequency sensor interpolation data are integrated in a hierarchical manner to obtain the first sensor dataset.

5. The edge intelligent control method based on multimodal perception fusion according to claim 1, characterized in that, The process of synchronizing the timestamps based on the first sensor dataset to obtain the second sensor dataset includes: The first sensor dataset is initialized with a timestamp sequence loop to obtain the first timestamp loop configuration. A timestamp deviation optimization objective function is established based on the first timestamp loop configuration, and a local search is performed based on the timestamp deviation optimization objective function to obtain the second timestamp loop configuration; Based on the second timestamp loop configuration, a timestamp neighborhood search and optimal synchronization sequence generation are performed to obtain the second sensor dataset.

6. The edge intelligent control method based on multimodal perception fusion according to claim 5, characterized in that, The step of establishing a timestamp deviation optimization objective function based on the first timestamp loop configuration, and performing a local search based on the timestamp deviation optimization objective function to obtain the second timestamp loop configuration includes: Based on the deviation calculation of adjacent timestamp nodes in the first timestamp loop configuration, the timestamp deviation optimization objective function is obtained. Based on the timestamp deviation optimization objective function, any two edges in the first timestamp loop configuration are swapped to obtain a candidate timestamp loop configuration. Based on the timestamp deviation optimization objective function, the deviation improvement amount of the candidate timestamp loop configuration is calculated to obtain the timestamp deviation improvement amount parameter. Based on the timestamp deviation improvement parameter, threshold judgment and loop configuration selection are performed to obtain the second timestamp loop configuration.

7. The edge intelligent control method based on multimodal perception fusion according to claim 1, characterized in that, The third sensor dataset is obtained by adaptively adjusting the time window and synchronizing the data based on the elderly person's movement speed detection results, and includes: The elderly person's movement speed is calculated based on the continuous inter-frame RGB-D data in the second sensor dataset to obtain the elderly person's movement speed detection result. Based on the detection results of the elderly's movement speed, a graded time window length matching is performed to obtain the basic time window length parameter; Based on the second sensor dataset, abnormal behavior of the elderly is detected and emergency time window is adjusted to obtain the adjustment time window length parameter. The adjustment time window length parameter is input into the adaptive time window generation module to reorganize the sensor data and obtain the window-adjusted sensor dataset. Based on the window adjustment of the sensor dataset, hybrid evolutionary multi-frequency synchronization optimization is performed to obtain the third sensor dataset.

8. The edge intelligent control method based on multimodal perception fusion according to claim 7, characterized in that, The hybrid evolutionary multi-frequency synchronization optimization based on the window-adjusted sensor dataset yields a third sensor dataset, including: Based on the window-adjusted sensor dataset, heuristic rule-based task decomposition was performed to obtain visual high-frequency subtask data, laser mid-frequency subtask data, and ultrasonic high-frequency subtask data. Genetic algorithm population initialization and evolution optimization are performed on the visual high-frequency subtask data, the laser mid-frequency subtask data, and the ultrasonic high-frequency subtask data respectively to obtain subtask optimized population data; Based on the sub-tasks, optimize the population data to perform optimal individual migration and information exchange, and obtain evolutionarily synchronized optimized population data. The evolutionary synchronization optimization population data is subjected to convergence judgment and multi-frequency synchronization parameter processing to obtain the third sensor dataset.

9. The edge intelligent control method based on multimodal perception fusion according to claim 1, characterized in that, The step of calculating the navigation path based on the third sensor dataset and outputting navigation control commands for the intelligent elderly care robot includes: Based on the third sensor dataset, the behavior state of the elderly is identified to obtain the behavior state parameters of the elderly. Based on the elderly person's behavioral state parameters, behavioral weights are assigned to the third sensor dataset to obtain sensor behavioral weight fusion parameters. Weighted Bayesian location estimation is performed based on the sensor behavior weight fusion parameters to obtain the elderly person's location fusion result. Based on the location fusion results of the elderly, behavioral preference path planning is performed to obtain edge navigation path parameters. Based on the edge navigation path parameters, motion control is performed to output navigation control commands for the intelligent elderly care robot.

10. A multimodal perception fusion edge intelligent control system, characterized in that, An edge intelligent control method for performing multimodal perception fusion as described in any one of claims 1-9, wherein the multimodal perception fusion edge intelligent control system comprises: The calibration module is used to perform frequency calibration on RGB-D camera data, LiDAR data, and ultrasonic sensor data to obtain the first sensor dataset. The optimization module is used to perform timestamp synchronization based on the first sensor dataset to obtain the second sensor dataset; The adjustment module is used to perform adaptive time window adjustment and synchronous optimization on the second sensor dataset based on the elderly person's movement speed detection results to obtain the third sensor dataset. The output module is used to calculate the navigation path based on the third sensor dataset and output navigation control commands for the intelligent elderly care robot.

Images