System and method for cross-modal time-series data alignment and behavior fusion for smart elderly care, and storage medium

Abstract

The present application relates to the technical field of multi-modal data processing and time series analysis, and particularly relates to a system and method for cross-modal time series data alignment and behavior fusion for smart elderly care and a storage medium, the method comprising: acquiring multi-source heterogeneous data within a set time range; performing time calibration on timestamps of the multi-source heterogeneous data to obtain time-aligned multi-source heterogeneous data; performing clustering fusion processing on the time-aligned multi-source heterogeneous data based on time dimension, space dimension and entity dimension to obtain a behavior-related data set of each corresponding entity; constructing a behavior timeline in chronological order for each entity's behavior-related data set; and converting the behavior timeline into a structured report and saving the report. The present application solves the problems of time alignment and behavior portrayal of multi-source heterogeneous data in the smart elderly care scenario, realizes accurate alignment and behavior portrayal of multi-source heterogeneous data, and thus generates a complete, accurate, interpretable and real-time old person behavior timeline report.

Description

Technical Field

[0001] This invention relates to the technical field of multimodal data processing and time series analysis, and specifically to a system, method, and storage medium for cross-modal time series data alignment and behavior fusion for smart elderly care. Background Technology

[0002] With the rapid development of IoT and AI technologies, modern intelligent systems need to process massive amounts of data from various data sources. These data typically have the following characteristics: 1. Diverse data sources, including sensor data, video data, access control systems, business databases, and other types, with different data formats and storage methods; 2. Inconsistent time granularity, with different devices having different time precision and timestamp formats, leading to difficulties in time alignment; 3. Clock skew issues, with clock storage drift and deviation between devices, meaning that timestamps of records at the same moment may differ by several seconds or even minutes in different data sources; 4. Semantic fragmentation, where records from different data sources are semantically mutually managed but physically independent.

[0003] In smart elderly care scenarios, access control and video surveillance equipment operate continuously, typically 24 / 7. Clock drift in these devices is often subtle, cumulative, and inconsistent. Existing time synchronization methods, such as NTP (Network Time Protocol), can only eliminate current-time discrepancies and cannot predict or compensate for long-term drift trends. In smart elderly care, elderly behavior exhibits temporal continuity, spatial constraints (limited activity range and highly correlated behavior records within the same space), and fragmented semantics from multiple sources (the same behavior is recorded across access control, video, and nursing systems). One existing approach is to stitch the data together in chronological order, but this results in inaccurate fusion due to temporal discrepancies and differences in data sources. Another approach relies on matching based on fixed rules, which lack adaptability and struggle to handle complex semantic relationships. Therefore, improvements are necessary. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a system, method and storage medium for cross-modal time-series data alignment and behavior fusion for smart elderly care. It solves the problem that existing time synchronization methods can only eliminate the deviation at the current moment and cannot predict and compensate for the long-term drift trend of the device. It also solves the problems that the actual sorting process makes the fusion results inaccurate and that fixed rule matching lacks adaptability and is difficult to handle complex semantic associations.

[0005] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a method for cross-modal time-series data alignment and behavior fusion for smart elderly care, comprising the following steps: Acquire multi-source heterogeneous data that is legally used within a set time range. The acquired multi-source heterogeneous data includes access control system access record data, video surveillance system video surveillance data, nursing management system medication record data, nursing management system nursing record data, and nursing management system visitor record data. The timestamps of the acquired multi-source heterogeneous data are time-calibrated to obtain time-aligned multi-source heterogeneous data; Time-aligned multi-source heterogeneous data are clustered and fused based on time, space, and entity dimensions to obtain a set of behavioral data related to each corresponding entity. For each entity's behavior-related data set, construct a behavior timeline in chronological order; Convert the aforementioned timeline of behavior into a structured report and save it.

[0006] A further improvement of the method for cross-modal time-series data alignment and behavior fusion for smart elderly care in this invention lies in the following steps for time calibration of the timestamps of the acquired multi-source heterogeneous data: The clock deviation of the equipment in the access control system, video surveillance system, and nursing management system is detected to obtain the equipment clock drift rate; Obtain historical calibration data from access control systems, video surveillance systems, and nursing management systems; The time deviation correction is calculated using a time calibration function, which is: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, where ΔT represents the time deviation correction amount in seconds, w_source represents the data source weight coefficient, K_source represents the data source base deviation coefficient in seconds, w_drift represents the drift rate weight, drift represents the device clock drift rate in seconds / hour, w_history represents the historical weight, and ΔT_history represents the exponentially weighted average of historical calibration deviations in seconds; The timestamps of multi-source heterogeneous data are calibrated using time deviation correction. The time window is determined according to the dynamic time window division rule after the timestamp is calibrated, and the multi-source heterogeneous data is aligned according to the time window to obtain time-aligned multi-source heterogeneous data.

[0007] A further improvement of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care in this invention lies in the following steps: clustering and fusion processing of time-aligned multi-source heterogeneous data based on time, space, and entity dimensions includes the following steps: With the consent of the individuals, facial features are extracted from video data in time-aligned multi-source heterogeneous data, and then the video data is associated with entities based on the results of facial feature extraction. Clustering results are obtained by clustering and fusion processing of multi-source heterogeneous data using a clustering function. The clustering function is: C = w1·(Δt / T_ref) + w2·(Δs / S_ref) + w3·(1-Δe), where C represents the clustering distance and is a dimensionless value; w1, w2, and w3 represent the weights of the time dimension, spatial dimension, and entity dimension, respectively, and w1 + w2 + w3 = 1; Δt represents the time distance in the time dimension, in seconds, and T_ref represents the preset time reference scale, in seconds; Δs represents the spatial distance in the spatial dimension, in meters, and S_ref represents the preset spatial reference scale, in meters; Δe represents the entity similarity in the entity dimension, with a value range of [0,1], and (1-Δe) represents the entity distance in the entity dimension; Δt / T_ref, Δs / S_ref, and (1-Δe) are all dimensionless values, and the value range of C is [0,1].

[0008] A further improvement of the present invention on the method for cross-modal temporal data alignment and behavior fusion for smart elderly care is that it further includes: Calculate the confidence score based on the clustering results; Temporal and spatial conflict detection is performed on the clustering results. Detected conflict behaviors are sorted by confidence level and then processed, retaining behaviors with high confidence levels and deleting behaviors with low confidence levels. For conflicting behaviors with the same confidence level, processing is based on data source priority: data from the nursing management system > data from the video surveillance system > data from the access control system.

[0009] A further improvement of the present invention on the method for cross-modal temporal data alignment and behavior fusion for smart elderly care is that it further includes: Obtaining captured images from a legally used video surveillance system; With the individual's consent, facial features are extracted from the captured images, and the captured images are then associated with corresponding entities based on the results of the facial feature extraction. The time of acquisition of the captured image and the name of the camera that captured the captured image; Based on the camera's name, locate the storage channel for video surveillance data; combine this with the set video capture time range, extract the corresponding video frames, and then generate video clips. The generated video clips are associated with entities and time and stored accordingly.

[0010] To achieve the above objectives, another technical solution provided by the present invention is: a storage medium storing a program for a method of cross-modal temporal data alignment and behavior fusion for smart elderly care. When the program of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care is executed by the processor, it implements the steps of the method.

[0011] To achieve the above objectives, another technical solution provided by the present invention is: a system for cross-modal temporal data alignment and behavior fusion for smart elderly care, comprising: The data acquisition unit is used to acquire multi-source heterogeneous data that is legally used within a set time range. The acquired multi-source heterogeneous data includes access control system access record data, video surveillance system video surveillance data, nursing management system medication record data, nursing management system nursing record data, and nursing management system visitor record data. A time calibration unit is connected to the data acquisition unit. The time calibration unit is used to calibrate the timestamps of the acquired multi-source heterogeneous data to obtain time-aligned multi-source heterogeneous data. A clustering fusion unit, connected to the time calibration unit, is used to perform clustering and fusion processing on time-aligned multi-source heterogeneous data based on time dimension, spatial dimension and entity dimension to obtain a set of behavior-related data for each corresponding entity. The processing unit, connected to the clustering and fusion unit, is used to construct a behavior timeline for each entity's behavior-related data set in chronological order; it is also used to convert the behavior timeline into a structured report and save it.

[0012] A further improvement of the system for cross-modal temporal data alignment and behavior fusion for smart elderly care in this invention is that the time calibration unit includes a deviation detection module, a historical data acquisition module, a calibration processing submodule, and an alignment processing submodule. The deviation detection module is used to detect the clock deviation of the equipment in the access control system, video surveillance system, and nursing management system to obtain the equipment clock drift rate. The historical data acquisition module is used to acquire historical calibration data from the access control system, video surveillance system, and nursing management system. The calibration processing submodule is connected to the deviation detection module and the historical data acquisition module. The calibration processing submodule is used to calculate the time deviation correction using a time calibration function, which is: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, where ΔT represents the time deviation correction in seconds, w_source represents the data source weight coefficient, K_source represents the data source basic deviation coefficient in seconds, w_drift represents the drift rate weight, drift represents the device clock drift rate in seconds / hour, w_history is the historical weight, and ΔT_history represents the exponentially weighted average of historical calibration deviations in seconds. The calibration processing submodule is also used to calibrate the timestamps of multi-source heterogeneous data using the time deviation correction. The alignment processing submodule is connected to the calibration processing submodule. The alignment processing submodule is used to determine the corresponding time window for the calibrated timestamp according to the dynamic time window division rule, and to perform alignment operation on the multi-source heterogeneous data according to the time window, thereby obtaining time-aligned multi-source heterogeneous data.

[0013] A further improvement of the system for cross-modal temporal data alignment and behavior fusion for smart elderly care in this invention is that the clustering fusion unit includes a face recognition module and a clustering processing sub-module. The face recognition module is used to extract face features from video data in time-aligned multi-source heterogeneous data with the consent of the individual, and then associate the video data with the corresponding entities through the results of the face feature extraction. The clustering processing submodule is connected to the face recognition module. The clustering processing submodule is used to perform clustering and fusion processing on multi-source heterogeneous data using a clustering function to obtain clustering results. The clustering function is: C=w1·(Δt / T_ref)+w2·(Δs / S_ref)+w3·(1-Δe), where C represents the clustering distance and is a dimensionless value; w1, w2, and w3 represent the time dimension weight, spatial dimension weight, and entity dimension weight, respectively, and w1+w2+w3=1; Δt represents the time distance in the time dimension, in seconds (s), and T_ref represents the preset time reference scale, in seconds (s); Δs represents the spatial distance in the spatial dimension, in meters (m), and S_ref represents the preset spatial reference scale, in meters (m); Δe represents the entity similarity in the entity dimension, with a value range of [0,1], and (1-Δe) represents the entity distance in the entity dimension; Δt / T_ref, Δs / S_ref, and (1-Δe) are all dimensionless values, and C has a value range of [0,1].

[0014] A further improvement of the system for cross-modal temporal data alignment and behavior fusion for smart elderly care in this invention is that it also includes a video intelligent capture unit, which is connected to the data acquisition unit; The data acquisition unit is also used to acquire captured images from a legally used video surveillance system. The intelligent video capture unit is used to extract facial features from captured images with the individual's consent, and then associate the captured images with entities based on the results of the facial feature extraction; it is also used to obtain the time of the captured image and the name of the camera that captured the captured image; it is also used to locate the storage channel of the video surveillance data according to the name of the camera, extract the corresponding video frames based on the set video capture time range, generate video segments, and associate and store the generated video segments with entities and time.

[0015] The beneficial effects of the system and method for cross-modal temporal data alignment and behavior fusion for smart elderly care, as described in this invention, are as follows: The present invention relates to a system and method for cross-modal time-series data alignment and behavior fusion for smart elderly care. It addresses the problem of time alignment and behavior characterization of multi-source heterogeneous data in smart elderly care scenarios, and achieves accurate alignment and behavior characterization of multi-source heterogeneous data, thereby generating a complete, accurate, interpretable, and real-time timeline report of elderly behavior.

[0016] This invention performs time calibration on timestamps of multi-source heterogeneous data to achieve time alignment. Specifically, it solves the clock drift problem caused by long-term operation of equipment by establishing a time calibration mechanism. Through time calibration functions and dynamic time windows, it effectively eliminates the influence of clock deviation and achieves millisecond-level time alignment accuracy.

[0017] This invention addresses the problem of scattered records of elderly people's behavior across multiple systems by achieving unified integration of video images, access control records, and business data. Through clustering and fusion processing, it automatically identifies and associates the complete behavior of the same elderly person. Each behavior record includes the complete data source, confidence level, and fusion basis, supporting result traceability and interpretation.

[0018] This invention addresses the problem of semantic fragmentation in elderly behavior by constructing a complete behavioral timeline and supporting the generation of structured reports, making it easier for caregivers and family members to understand the elderly's living conditions.

[0019] This invention supports real-time data fusion and report generation through parallel data acquisition and pipelined processing.

[0020] The present invention also provides a video intelligent capture function, which associates the captured images with the original video recordings to achieve automatic capture and storage of videos of key events. Attached Figure Description

[0021] Figure 1 This is a system diagram of the system for cross-modal temporal data alignment and behavior fusion for smart elderly care, as described in this invention.

[0022] Figure 2 This is a system architecture diagram of another preferred embodiment of the system for cross-modal temporal data alignment and behavior fusion for smart elderly care according to the present invention.

[0023] Figure 3 This is a flowchart of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care according to the present invention.

[0024] Figure 4 This is a flowchart of the time alignment method in the cross-modal time-series data alignment and behavior fusion method for smart elderly care of the present invention.

[0025] Figure 5 This is a flowchart of the cross-modal behavior fusion method in the cross-modal temporal data alignment and behavior fusion method for smart elderly care of the present invention.

[0026] Figure 6 This is a flowchart of the three-dimensional weighted clustering algorithm in the method for cross-modal temporal data alignment and behavior fusion for smart elderly care in this invention.

[0027] Figure 7 This is a schematic diagram illustrating the implementation of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care in this invention, specifically in the scenario of tracking elderly behavior in a nursing home.

[0028] Figure 8 This is a flowchart of the intelligent video recording extraction method in the cross-modal temporal data alignment and behavior fusion method for smart elderly care of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0030] See Figure 1 This invention provides a system, method, and storage medium for cross-modal time-series data alignment and behavior fusion for smart elderly care. Addressing the specific needs of smart elderly care scenarios, it achieves accurate alignment and behavior characterization of multi-source heterogeneous time-series data, thereby generating a complete, accurate, interpretable, and real-time timeline report of elderly behavior. The aim is to eliminate the impact of device clock drift, identify complete behaviors, and construct behavior timelines. The system, method, and storage medium for cross-modal time-series data alignment and behavior fusion for smart elderly care according to this invention are described below with reference to the accompanying drawings.

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. All data involved in the user data collection and processing described in this invention are legally used data.

[0032] See Figure 1 This diagram illustrates the system diagram of the cross-modal temporal data alignment and behavior fusion system for smart elderly care according to the present invention. The following section combines... Figure 1 This paper describes the system for cross-modal temporal data alignment and behavior fusion for smart elderly care.

[0033] like Figure 1 As shown, the system for cross-modal temporal data alignment and behavior fusion for smart elderly care of the present invention includes a data acquisition unit 21, a time calibration unit 22, a clustering fusion unit 23, and a processing unit 24. The data acquisition unit 21 is connected to the time calibration unit 22, the time calibration unit 22 is connected to the clustering fusion unit 23, and the clustering fusion unit 23 is connected to the processing unit 24. The data acquisition unit 21 is used to acquire legally used multi-source heterogeneous data within a set time range. The acquired multi-source heterogeneous data includes access record data from a legally used access control system, video surveillance data from a legally used video surveillance system, medication record data from a legally used nursing management system, nursing record data from a legally used nursing management system, and visitor record data from a legally used nursing management system. The set time range is configurable, with a default value of 00:00:00 to 23:59:59 of the current day. By configuring the set time range, a structured report on the elderly's life within the corresponding time range can be obtained. The time calibration unit 22 is used to calibrate the timestamps of the acquired multi-source heterogeneous data to obtain time-aligned multi-source heterogeneous data; the clustering and fusion unit 23 is used to perform clustering and fusion processing on the time-aligned multi-source heterogeneous data based on the time dimension, spatial dimension and entity dimension to obtain the behavior-related data set of each corresponding entity; the processing unit 24 is used to construct a behavior timeline for each entity's behavior-related data set in chronological order; and is also used to convert the behavior timeline into a structured report and save it.

[0034] The data acquisition unit 21 communicates with legally used access control systems, video surveillance systems, and nursing management systems, acquiring raw or initial multi-source heterogeneous data from these systems. Each piece of data in the acquired multi-source heterogeneous data includes a timestamp, subject identifier, behavior type, and location information. The timestamp indicates the precise time of the event. The subject identifier includes personnel ID (collected and counted with individual consent), device ID, etc. The behavior type includes access, medication, nursing, visitor, etc., and the location information includes room number, device number, etc. Furthermore, the access control system uses MongoDB + disk storage, with key data source fields including access time, access control ID, personnel ID, and door location. The video surveillance system uses NVR / DVR storage, with key data source fields including capture time, camera ID, and image data. The business system uses MySQL storage, with key data source fields including medication records, nursing records, visitor records, and blood pressure data. The data source is also configured with a facial feature database (the collection and establishment of this facial feature database were also carried out with the consent of the individuals), and the storage method is local storage. The main fields of the data source include 256-dimensional feature vectors, personnel profile photos, etc.

[0035] Furthermore, the time calibration unit 22 is used to uniformly calibrate the timestamps of the data source, eliminating the effects of device constant deviation and drift.

[0036] The time calibration unit 22 includes a deviation detection module, a historical data acquisition module, a calibration processing submodule, and an alignment processing submodule; The deviation detection module is used to detect the clock deviation of equipment in access control systems, video surveillance systems, and nursing management systems in order to obtain the equipment clock drift rate; The historical data acquisition module is used to acquire historical calibration data from the access control system, video surveillance system, and nursing management system. The calibration processing submodule is connected to the deviation detection module and the historical data acquisition module. It calculates the time deviation correction using a time calibration function: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, where ΔT represents the time deviation correction in seconds (s), w_source represents the data source weight coefficient, K_source represents the data source basic deviation coefficient in seconds, w_drift represents the drift rate weight, drift represents the device clock drift rate in seconds / hour, w_history is the historical weight, and ΔT_history represents the exponentially weighted average of historical calibration deviations in seconds. The calibration processing submodule also calibrates the timestamps of multi-source heterogeneous data using the time deviation correction. The data source weight coefficient w_source is set as follows: 0.1 for the access control system, 0.15 for the video surveillance system, and 0.05 for the nursing management system. The drift rate weight w_drift is set to 0.3, and the historical weight w_history is set to 0.4. The data source baseline deviation coefficient K_source is set to 30ms. This value represents the typical baseline time deviation of each data source device under uncalibrated conditions. It is used in conjunction with the data source weighting coefficient w_source to ensure that the contribution of each data source to the time deviation correction is in the millisecond range, thus meeting the millisecond-level time alignment accuracy design goal of this invention.

[0037] The basis for determining the value of the basic deviation coefficient for this data source is explained as follows: According to the time calibration function: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, In the formula, w_source (data source weight coefficient) has been assigned values ​​according to the data source (access control 0.1, video 0.15, nursing 0.05), and the differences between data sources have been reflected by w_source; therefore, K_source, as the basic deviation coefficient, takes a uniform value, and multiplying it by w_source can reflect the differentiated contribution of each data source to the time deviation correction.

[0038] The basis for setting K_source to 30ms is as follows: - Access control systems, video surveillance systems, and nursing management systems in smart elderly care scenarios are all IoT / server devices, and their base time deviation under uncalibrated conditions is usually on the order of 10ms to 50ms; - Taking 30ms as the common basic deviation coefficient for all data sources, which is located in the upper-middle level of this range, can reasonably reflect the typical clock deviation of the nursing home equipment group; - When combined with the pre-set weighting coefficients, the contribution of the w_source × K_source term is in the millisecond range, which meets the technical goal of "millisecond-level time alignment accuracy" of this invention.

[0039] Calibration contribution verification: Substitute K_source = 30ms into the w_source × K_source term in the time calibration function:

[0040] Table 1: Data source time calibration values ​​for each system.

[0041] It can be seen that the contribution of each data source's basic deviation term is in the millisecond range (1.5ms~4.5ms). After working together with the drift rate term (w_drift=0.3) and the history term (w_history=0.4), the overall time deviation correction amount ΔT remains within the millisecond range, which meets the design goal of the present invention of millisecond-level time alignment accuracy.

[0042] The alignment processing submodule is connected to the calibration processing submodule. The alignment processing submodule is used to determine the corresponding time window for the calibrated timestamp according to the dynamic time window division rules, and to perform alignment operation on the multi-source heterogeneous data according to the time window, thereby obtaining time-aligned multi-source heterogeneous data.

[0043] Furthermore, the calculation of clock drift rate also considers historical clock drift rate. Specifically, the formula for calculating clock drift rate is: drift = a × current_drift + (1-a) × historical_drift, where a is a smoothing coefficient with a value ranging from 0.1 to 0.3. Here, current_drift represents the current clock drift rate, and historical_drift represents the historical clock drift rate.

[0044] Furthermore, the dynamic time window segmentation rules include sub-rules for data density factors, sub-rules for time importance factors, and sub-rules for historical pattern factors. The data density factor segmentation sub-rule is as follows: for the first density region (high-density region greater than or equal to a set density), the first set duration of the time window is automatically reduced, for example, to 5 seconds; for the second density region (low-density region less than a set density), the second set duration of the time window is expanded, for example, to 60 seconds. The event importance factor segmentation sub-rule is as follows: for critical events (such as medication, abnormal events, etc.), a fine-grained window is used, for example, a time window of 1 second; for general events (such as walking, leisure events, etc.), a coarse-grained window is used, for example, a time window of 30 seconds. The sub-rules for classifying historical pattern factors are as follows: For regular behaviors (such as taking medication at fixed times, eating, etc.), a fixed time window is adopted. For example, for the behavior of taking medication at 08:00, the fixed time window is 07:55:00-08:05:00, that is, 5 minutes before and 5 minutes after the medication time, with a window length of 10 minutes; for the behavior of eating at 12:00, the fixed time window is 11:55:00-12:05:00, with a window length of 10 minutes.

[0045] Examples of fixed time windows are shown in Table 2 below:

[0046] Table 2: Example table of fixed time windows.

[0047] Taking scheduled medication as an example: If the care plan specifies that the elderly person takes nifedipine sustained-release tablets at 08:00, then the fixed time window is 07:55:00 - 08:05:00 (i.e., 5 minutes before and 5 minutes after the medication time, with a window length of 10 minutes). Within this time window, the system performs time alignment and clustering fusion of access control records, video surveillance capture records, and medication records from the care management system to confirm the elderly person's actual medication behavior.

[0048] The duration of the fixed time window in this invention is based on the fact that the time fluctuation of regular behavior is usually within ±5 minutes. Therefore, the window is set to 5 minutes before and after the medication time point, for a total of 10 minutes. This can cover the reasonable time range of the behavior and avoid introducing irrelevant data by making the window too large.

[0049] For random activities (such as visitors, walks, etc.), a flexible window is used. For example, for visitor visits, the flexible window is dynamically determined based on the time the visitor enters and leaves through the access control. If the visitor enters at 10:30:00 and leaves at 11:15:00, the flexible window is 10:30:00-11:15:00, with a window duration of 45 minutes. For walks, the flexible window is dynamically determined based on the time the elderly person leaves and returns to the room through the access control. If the elderly person leaves at 15:00:00 and returns at 15:30:00, the flexible window is 15:00:00-15:30:00, with a window duration of 30 minutes.

[0050] Flexible windows are suitable for random activities, such as visitor arrivals or walks, where the timing and duration are not fixed. The start and end times of a flexible window are dynamically determined based on the actual timestamps of the event's trigger and end.

[0051] Taking a visitor's visit as an example: Visitor Zhang Qiang enters the nursing home through the access control at 10:30:00 and leaves through the access control at 11:15:00. The flexible window is dynamically determined from 10:30:00 to 11:15:00, with a window duration of 45 minutes. Within this flexible window, the system performs time alignment and clustering fusion of visitor access control records, visitor lobby video surveillance records, and visitor records from the nursing management system to generate a complete "visitor visit" behavior unit.

[0052] Taking a walk as an example: If an elderly person, Zhang San, leaves room 201 of Building 1 through the room access control at 15:00:00 and returns to room 201 through the same access control at 15:30:00, the flexible window is dynamically determined to be from 15:00:00 to 15:30:00, with a window duration of 30 minutes. Within this flexible window, the system performs time alignment and clustering fusion of access control records and video surveillance records from corridors and public areas to generate a "walk" behavior unit.

[0053] like Figure 4 As shown, the processing flow of the time alignment method is illustrated. Starting from multi-source data input, the process involves obtaining timestamps, detecting device clock deviation, calculating the relative standard time offset, obtaining historical calibration data, recording and analyzing historical deviations and trends, calculating the time deviation correction using a time calibration function, calculating the calibration time, dividing the time window dynamically, using an adaptive window based on data density, and finally outputting time-aligned data and a unified time axis record.

[0054] In one specific embodiment of the present invention, the clustering fusion unit 23 includes a face recognition module and a clustering processing submodule; This facial recognition module is used to extract facial features from video data in time-aligned multi-source heterogeneous data with the consent of the individual, and then associate the video data with the corresponding entities based on the results of facial feature extraction. The clustering processing submodule is connected to the face recognition module. This submodule uses a clustering function to perform clustering and fusion processing on multi-source heterogeneous data to obtain the clustering result. The clustering function is: C = w1·(Δt / T_ref) + w2·(Δs / S_ref) + w3·(1-Δe), where C represents the clustering distance (a dimensionless value); w1, w2, and w3 represent the time dimension weight, spatial dimension weight, and entity dimension weight, respectively, and w1 + w2 + w3 = 1; Δt represents... The time distance in the time dimension is in seconds (s), where T_ref represents the preset time reference scale in seconds; Δs represents the spatial distance in the spatial dimension in meters (m), where S_ref represents the preset spatial reference scale in meters; Δe represents the entity similarity in the entity dimension, with a value range of [0,1], and (1-Δe) represents the entity distance in the entity dimension; Δt / T_ref, Δs / S_ref, and (1-Δe) are all dimensionless values, and C has a value range of [0,1].

[0055] Furthermore, with the individual's consent, the face recognition module performs face detection on the video data images. Specifically, it can use the Retina Face model to detect the face location in the image and then return the face bounding box coordinates. Then, it performs feature extraction and matching. Specifically, it can use the Arc Face model to extract 256-dimensional feature vectors and then match the extracted features with elderly entities in the face feature database based on feature similarity. For example, it can calculate the cosine similarity. When the similarity is greater than or equal to the matching threshold, it associates the face in the image with the corresponding elderly entity, establishing a correspondence between the video data and the elderly entity.

[0056] Furthermore, the clustering submodule performs three-dimensional weighted clustering on multi-source heterogeneous data, merging relevant records from different data sources to obtain clustering results. Each clustering result represents a specific behavior, forming a behavioral unit data structure. Clustering fusion processing is performed using a clustering function. The weights w1, w2, and w3 for the time dimension are set to 0.4, w2 to 0.3, and w3 to 0.3, respectively. These weights are determined based on the constraints of the smart elderly care scenario. Elderly behavior exhibits clear temporal regularity, such as scheduled medication, meals, and bedtime. Temporal continuity is the most important indicator for judging the integrity of behavior, and the sequence of events is crucial for anomaly identification; therefore, the time dimension weight is set to the highest value. Elderly individuals have limited activity ranges, and spatial distance is an important basis for judging behavioral relevance. Multiple behavioral records within the same room usually belong to the same behavioral unit, and spatial information can be used to assist in identifying abnormal behavior, such as an elderly person appearing in an inappropriate location. Facial recognition is a key means of linking records from different data sources. The entity dimension has the same weight as the spatial dimension, reflecting the importance of identity verification. In scenarios with many elderly people in nursing homes, accurate entity recognition can avoid confusion in behavioral records. Validated using actual data from nursing homes, this weight combination (w1=0.4, w2=0.3, w3=0.3) achieved the following results: an average silhouette coefficient of 0.85, a single time dimension clustering score of 0.7 (a 21% improvement), a behavioral unit recognition accuracy of 92%, and an abnormal behavior detection recall rate of 88%.

[0057] Combination Figure 6 As shown, the clustering algorithm process is as follows: Initialize K cluster centers; calculate the weighted distance of each record to each cluster center and assign it to the nearest cluster; recalculate each cluster center; repeat until convergence (distance change < ε = 0.01) or the maximum number of iterations (e.g., 100 times) is reached. The clustering result is a behavioral unit data structure: {"timestamp": "2024-01-15 08:30:00", "subject": "Elderly ID-Zhang San", "action": "medication / nursing care / visitor / mobility", "location": "Room 201, Building 1", "confidence": 0.95, "sources": ["Access Control", "Nursing System"], "rawData": [...]}.

[0058] Furthermore, the clustering processing submodule is also used to perform behavioral semantic fusion on the clustering results and calculate the fusion confidence score. The formula for calculating the fusion confidence score is: Conf=Σ(w_k·score_k), where w_k represents the weight and score_k is the rating. For access control records, the weight w_k is 0.3, the base score_k is 0.95, and the rating factor is device reliability; for video capture, the weight w_k is 0.35, the base score_k is 0.85, and the rating factor is facial similarity; for business systems, the weight w_k is 0.35, the base score_k is 1.00, and the rating factor is authoritative manually entered data.

[0059] Furthermore, the clustering processing submodule also performs conflict detection on the clustering results, specifically detecting temporal and spatial conflicts between behavioral unit data structures, and resolving conflicts through the following strategies: confidence ranking, prioritizing the retention of behavioral records with high confidence; rule priority, processing according to data source priority, for conflicting behaviors with the same confidence, processing is based on data source priority, with data from the nursing management system having priority > data from the video surveillance system > data from the access control system.

[0060] like Figure 5 The diagram illustrates the complete workflow of the cross-modal behavior fusion method, including multi-source data input (access control + video + business records), facial feature extraction (ArcFace 256-dimensional vector), similarity matching (threshold ≥ 0.75), three-dimensional clustering fusion (time + space + entity), confidence calculation (Conf = Σ(w_k × score_k)), behavior conflict detection (temporal conflict / spatial conflict), behavior unit generation (time + subject + behavior + location + confidence), behavior timeline construction (event sequence + causal relationship), and output fusion results (structured behavior report).

[0061] Furthermore, the processing unit organizes behavioral units chronologically, constructs a complete behavioral timeline, establishes causal relationships between events, and converts the behavioral timeline into a structured, highly readable natural language report, such as... Figure 7 As shown, the activity trajectory of the elderly from 07:30 to 21:00 is displayed, including access control records, medication records, visitor records, care records, etc., as well as the final generated elderly life report.

[0062] In one specific embodiment of the present invention, the system of the present invention further includes a video intelligent capture unit, which is connected to the data acquisition unit; The data acquisition unit is also used to acquire images captured by legally used video surveillance systems; The intelligent video capture unit is used to extract facial features from captured images with the individual's consent, and then associate the captured images with entities based on the results of facial feature extraction; it is also used to obtain the time of the captured image and the name of the camera that captured the image; it is also used to locate the storage channel of the video surveillance data based on the camera name, extract the corresponding video frames based on the set video capture time range, generate video segments, and associate and store the generated video segments with entities and time.

[0063] This intelligent video capture unit is used to realize the intelligent capture function of video recording. By associating images captured by the video surveillance system with the original video recording, it realizes the automatic capture and storage of videos of key events. The specific process is as follows: feature matching is performed on the captured image to achieve face recognition matching; the capture time T_capture and camera ID of the matched image are extracted; the capture range is calculated according to the capture time and preset parameters to realize the determination of the time window, T_start=T_capture-Δt_before, T_end = T_capture + Δt_after, where T_start is the start time of video capture, T_end is the end time of video capture, Δt_before is the preset before time margin, and Δt_after is the preset after time margin. The default values ​​for Δt_before and Δt_after are 5 seconds, which can be configured according to actual conditions. Access the network video recorder via the RTSP real-time streaming protocol or SDK / API interface; convert the system time to a video frame index, and calculate the frame index corresponding to T_start and T_end respectively: Frame_Index = (T_target-T_start_of_file)×Frame_Rate, where T_target is the target time (i.e., the time point corresponding to the video frame to be located, T_start and T_end are taken respectively when extracting), T_start_of_file is the recording start time of the original video recording file to be extracted, and Frame_Rate is the video frame rate (usually 25fps or 30fps); Video segment extraction: Extract video frames of a specified time period and encode them to generate short videos; Associative storage: Establish an index relationship of "elderly ID-behavior time-video segment"; Report embedding: Embed video links or thumbnails in structured reports.

[0064] like Figure 8The diagram illustrates the complete process of the intelligent video recording extraction method. It begins by acquiring legally captured surveillance images, followed by facial recognition matching (with the individual's consent), extraction of capture time and camera ID, calculation of the video extraction time window, judgment of usage parameters, access to the disk recorder (NVR / DVR), location of the video file (RTSP / API access), calculation of time-to-frame index mapping, and execution of video segment extraction (encoding to generate a short video). Finally, it associates and stores the elderly person's ID, behavior time, and video segment.

[0065] In a preferred embodiment, such as Figure 2As shown, the system comprises a data source layer, a data acquisition and preprocessing layer, a core processing layer, and an application output layer. The data source layer includes legally used access control / turnstile systems, video surveillance systems, network video recorders, business system data, and a facial feature database. These data source systems are existing systems within the nursing home. The data acquisition and preprocessing layer communicates with each system in the data source layer to acquire their data. Data from the legally used access control / turnstile system includes communication records and captured images, stored in MongoDB + disk. Data from the legally used video surveillance system includes surveillance footage and captured images, stored in MongoDB + disk. The system stores raw video from the legally used network video recorder (NVR / DVR). Business system data includes medication, nursing, visitor, and blood pressure information, stored in MySQL. The legally used facial feature database (established with individual consent) includes elderly residents' photos and 256-dimensional feature vectors, used for facial recognition. The data acquisition and preprocessing layer includes a timestamp standardization module for multi-source time synchronization, deviation calculation, and compensation; a face detection and recognition module for Retina Face detection and Arc Face feature extraction from video surveillance data and captured images; entity association mapping for elderly ID extraction and cross-table association matching; and an NVR access interface for time synchronization calibration according to the RTSP protocol / API. The core processing layer includes a time alignment engine, a multi-dimensional clustering engine, a behavior fusion engine, a timeline construction engine, and a video capture engine. The time alignment engine includes dynamic time windows and time calibration functions for time alignment of data from data sources; the multi-dimensional clustering engine includes weighted clustering algorithms for time / space / entity dimensions for clustering related records from different data sources; the timeline construction engine is used for behavior sequence generation and event correlation analysis; and the video capture engine is used for time window calculation, frame index mapping, and video segment encoding. The application output layer includes structured report generation, an intelligent question-answering system, video report embedding, and abnormal behavior alerts. Structured report generation converts behavior timelines into structured, readable natural language reports, characterized by natural language descriptions, the ability to add video links, and timeline visualization effects. The intelligent question-answering system supports intent recognition and entity extraction. Users can ask questions to obtain the information they need, such as "How was Mr. Zhang's day?" The system recognizes the intent and extracts a structured report of Mr. Zhang's day, outputting it to the user. Video report embedding includes video thumbnail display and short video clip playback. Abnormal behavior alerts include risk identification and real-time alarms, analyzing behavior timelines to identify abnormal behaviors and generate alert information.The storage layer includes behavior record storage (MongoDB), video clip storage (including short-term cache + long-term archive), and related indexes (elderly ID-time-video).

[0066] The following section uses the example of tracking the behavior of elderly people in nursing homes to explain in detail the implementation process of this invention.

[0067] Query request: How is Mr. Zhang San's day? Expected output: A complete daily life report for Mr. Zhang, including his daily routine, medication, care, and visitors.

[0068] Processing flow: Phase 1: Intent recognition, 1. Intent Classification: Recognized as [Life Status Query]; 2. Entity Extraction: Extract entities such as [Old Man Zhang San] and [Today]; 3. Parameter Analysis: The time range is defined as 00:00:00 to 23:59:59 of the current day; Phase Two: Parallel Data Acquisition 1. Access Control Records: All access records for Mr. Zhang San on that day (door opening and closing times, locations); 2. Medication Records: Retrieve all medication records for the day from the business system (medication, dosage, time, and person responsible); 3. Nursing Records: Obtain nursing procedure records (nursing items, time, and performer); 4. Visitor Records: Retrieve visitor visit records (visitor name, visit time, departure time); 5. Video screenshots: Screenshots of Zhang San taken by surveillance cameras (approximately 20-50 images) were selected using facial recognition. Phase 3: Time Alignment and Behavioral Integration 1. Time calibration: Timestamps from all data sources are standardized to the standard timeline using a calibration function; 2. Clustering and Fusion: Based on the three-dimensional similarity of time, space, and entity, related records are clustered into behavioral units; 3. Conflict resolution: Handling situations involving overlapping time periods or spatial contradictions; 4. Timeline Construction: Organize behavioral units in chronological order; 8.3 Example of output results [Zhang San, an elderly man, reports on his daily life] Report Date: January 15, 2024 Daily schedule: • Woke up at 7:30 AM, Room 201, Building 1 (access control record, confidence level 0.98) • 08:00 Dinner at Restaurant in Building 1 (confirmed by video, confidence level 0.92) • 21:00 Go to bed and rest, Room 201, Building 1 (access control record, confidence level 0.98) Medication usage: • 08:00 Nifedipine sustained-release tablets 10mg, Executed by: Caregiver Li Ming (Business System, Confidence Level 1.00) • 12:00 Metformin tablets 500mg, Executed by: Caregiver Li Ming (Business System, Confidence Level 1.00) • 20:00 Aspirin enteric-coated tablets 100mg, Executed by: Caregiver Wang Fang (Business System, Confidence Level 1.00) Nursing care status: • 09:00 Routine turning over, executor: caregiver Wang Fang (business system, confidence level 1.00) • 14:00 Bathing assistance, performed by caregiver Li Ming, with good cooperation (business system, confidence level 1.00) Visitor statistics: • 10:30 Son Zhang Qiang visited and stayed for 45 minutes. Visitor lobby (visitor system + access control, confidence level 0.95) • 15:00 Daughter Zhang Li visited and stayed for 60 minutes. Visitor lobby (visitor system + video, confidence level 0.92) Overall assessment: Today's routine was regular, mood was good, and interaction with family members was pleasant.

[0069] This invention also provides a method for cross-modal temporal data alignment and behavior fusion for smart elderly care, which is described below.

[0070] like Figure 3 As shown, the method includes the following steps: Execute step S11 to obtain legally used multi-source heterogeneous data within a set time range. The obtained multi-source heterogeneous data includes access control system access record data, video surveillance system video surveillance data, nursing management system medication record data, nursing management system nursing record data, and nursing management system visitor record data; then execute step S12. Execute step S12 to perform time calibration on the timestamps of the acquired multi-source heterogeneous data to obtain time-aligned multi-source heterogeneous data; then execute step S13. Step S13 is executed, where time-aligned multi-source heterogeneous data are clustered and fused based on time, space, and entity dimensions to obtain a set of behavior-related data for each corresponding entity; then step S14 is executed. Execute step S14 to construct a behavior timeline for each entity's behavior-related data set in chronological order; then execute step S15. Perform step S15 to convert the behavior timeline into a structured report and save it.

[0071] In one specific embodiment of the present invention, time calibration of the timestamps of the acquired multi-source heterogeneous data includes the following steps: The clock deviation of the equipment in the access control system, video surveillance system, and nursing management system is detected to obtain the equipment clock drift rate; Obtain historical calibration data from access control systems, video surveillance systems, and nursing management systems; The time deviation correction is calculated using a time calibration function, which is: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, where ΔT represents the time deviation correction amount in seconds, w_source represents the data source weight coefficient, K_source represents the data source base deviation coefficient in seconds, w_drift represents the drift rate weight, drift represents the device clock drift rate in seconds / hour, w_history represents the historical weight, and ΔT_history represents the exponentially weighted average of historical calibration deviations in seconds; The timestamps of multi-source heterogeneous data are calibrated using time deviation correction. The time window is determined according to the dynamic time window division rule after the timestamp is calibrated, and the multi-source heterogeneous data is aligned according to the time window to obtain time-aligned multi-source heterogeneous data.

[0072] In one specific embodiment of the present invention, the clustering and fusion processing of time-aligned multi-source heterogeneous data based on time, spatial, and entity dimensions includes the following steps: With the consent of the individuals, facial features are extracted from video data in time-aligned multi-source heterogeneous data, and then the video data is associated with entities based on the results of facial feature extraction. Clustering results are obtained by clustering and fusion processing of multi-source heterogeneous data using a clustering function. The clustering function is: C = w1·(Δt / T_ref) + w2·(Δs / S_ref) + w3·(1-Δe), where C represents the clustering distance and is a dimensionless value; w1, w2, and w3 represent the weights of the time dimension, spatial dimension, and entity dimension, respectively, and w1 + w2 + w3 = 1; Δt represents the time distance in the time dimension, in seconds, and T_ref represents the preset time reference scale, in seconds; Δs represents the spatial distance in the spatial dimension, in meters, and S_ref represents the preset spatial reference scale, in meters; Δe represents the entity similarity in the entity dimension, with a value range of [0,1], and (1-Δe) represents the entity distance in the entity dimension; Δt / T_ref, Δs / S_ref, and (1-Δe) are all dimensionless values, and the value range of C is [0,1].

[0073] Furthermore, it also includes: Calculate the confidence score based on the clustering results; Temporal and spatial conflict detection is performed on the clustering results. Detected conflict behaviors are sorted by confidence level and then processed, retaining behaviors with high confidence levels and deleting behaviors with low confidence levels. For conflicting behaviors with the same confidence level, processing is based on data source priority: data from the nursing management system > data from the video surveillance system > data from the access control system.

[0074] In one specific embodiment of the present invention, it further includes: Obtaining captured images from a legally used video surveillance system; With the consent of the individual, facial features are extracted from the captured images, and the captured images are then associated with the corresponding entities based on the results of the facial feature extraction. The time of acquisition of the captured image and the name of the camera that captured the captured image; Based on the camera's name, locate the storage channel for video surveillance data; combine this with the set video capture time range, extract the corresponding video frames, and then generate video clips. The generated video clips are associated with entities and time and stored accordingly.

[0075] To achieve the above objectives, another technical solution provided by the present invention is: a storage medium storing a program for a method of cross-modal temporal data alignment and behavior fusion for smart elderly care. When the program of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care is executed by the processor, it implements the steps of the method.

[0076] The beneficial effects of the system and method for cross-modal temporal data alignment and behavior fusion for smart elderly care of the present invention include: This invention effectively eliminates the influence of clock deviation and achieves millisecond-level time alignment accuracy through a time calibration function and a dynamic time window; This invention uses exponential smoothing weighting based on historical data, which can dynamically adapt to changes in data distribution without the need for manual intervention. The data structure of each behavioral unit in this invention records the complete data source, confidence level, and fusion basis, supporting result traceability and interpretation; This invention supports real-time data fusion and report generation through parallel data acquisition and pipelined processing; This invention unifies the processing of multiple modal data such as video images, access control records, and business data, enabling cross-modal behavior association; The video capture function of this invention has the advantages of strong evidentiary value, good behavioral continuity, strong configurability, and good storage optimization. This invention automatically generates a complete life report for the elderly, including their daily routine, medication, care, and visitor information. It supports natural language queries, provides confidence level annotations to facilitate manual verification, and links key video clips to support post-event evidence collection.

[0077] The present invention has been described in detail above with reference to the accompanying drawings and embodiments. Those skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention, and the scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A method for cross-modal temporal data alignment and behavior fusion for smart elderly care, characterized in that, Includes the following steps: Acquire multi-source heterogeneous data that is legally used within a set time range. The acquired multi-source heterogeneous data includes access control system access record data, video surveillance system video surveillance data, nursing management system medication record data, nursing management system nursing record data, and nursing management system visitor record data. The timestamps of the acquired multi-source heterogeneous data are time-calibrated to obtain time-aligned multi-source heterogeneous data; Time-aligned multi-source heterogeneous data are clustered and fused based on time, space, and entity dimensions to obtain a set of behavioral data related to each corresponding entity. For each entity's behavior-related data set, construct a behavior timeline in chronological order; Convert the aforementioned timeline of behavior into a structured report and save it.

2. The method for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 1, characterized in that, Time calibration of the timestamps of the acquired multi-source heterogeneous data includes the following steps: The clock deviation of the equipment in the access control system, video surveillance system, and nursing management system is detected to obtain the equipment clock drift rate; Obtain historical calibration data from access control systems, video surveillance systems, and nursing management systems; The time deviation correction is calculated using a time calibration function, which is: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, where ΔT represents the time deviation correction amount in seconds, w_source represents the data source weight coefficient, K_source represents the data source base deviation coefficient in seconds, w_drift represents the drift rate weight, drift represents the device clock drift rate in seconds / hour, w_history represents the historical weight, and ΔT_history represents the exponentially weighted average of historical calibration deviations in seconds; The timestamps of multi-source heterogeneous data are calibrated using time deviation correction. The time window is determined according to the dynamic time window division rule after the timestamp is calibrated, and the multi-source heterogeneous data is aligned according to the time window to obtain time-aligned multi-source heterogeneous data.

3. The method for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 1, characterized in that, Clustering and fusion processing of time-aligned multi-source heterogeneous data based on time, spatial, and entity dimensions includes the following steps: With the consent of the individuals, facial features are extracted from video data in time-aligned multi-source heterogeneous data, and then the video data is associated with entities based on the results of facial feature extraction. Clustering results are obtained by clustering and fusion processing of multi-source heterogeneous data using a clustering function. The clustering function is: C = w1·(Δt / T_ref) + w2·(Δs / S_ref) + w3·(1-Δe), where C represents the clustering distance and is a dimensionless value; w1, w2, and w3 represent the weights of the time dimension, spatial dimension, and entity dimension, respectively, and w1 + w2 + w3 = 1; Δt represents the time distance in the time dimension, in seconds, and T_ref represents the preset time reference scale, in seconds; Δs represents the spatial distance in the spatial dimension, in meters, and S_ref represents the preset spatial reference scale, in meters; Δe represents the entity similarity in the entity dimension, with a value range of [0,1], and (1-Δe) represents the entity distance in the entity dimension; Δt / T_ref, Δs / S_ref, and (1-Δe) are all dimensionless values, and the value range of C is [0,1].

4. The method for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 3, characterized in that, Also includes: Calculate the confidence score based on the clustering results; Temporal and spatial conflict detection is performed on the clustering results. Detected conflict behaviors are sorted by confidence level and then processed, retaining behaviors with high confidence levels and deleting behaviors with low confidence levels. For conflicting behaviors with the same confidence level, processing is based on data source priority: data from the nursing management system > data from the video surveillance system > data from the access control system.

5. The method for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 1, characterized in that, Also includes: Obtaining captured images from a legally used video surveillance system; With the individual's consent, facial features are extracted from the captured images, and the captured images are then associated with corresponding entities based on the results of the facial feature extraction. The time of acquisition of the captured image and the name of the camera that captured the captured image; Based on the camera's name, locate the video surveillance data storage channel, and combine it with the set video capture time range to extract the corresponding video frames and generate video clips. The generated video clips are associated with entities and time and stored accordingly.

6. A storage medium, characterized in that, The storage medium stores a program for a method of cross-modal time-series data alignment and behavior fusion for smart elderly care. When the program of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care is executed by a processor, it implements the steps of the method for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in any one of claims 1 to 5.

7. A system for cross-modal temporal data alignment and behavior fusion for smart elderly care, characterized in that, include: The data acquisition unit is used to acquire multi-source heterogeneous data that is legally used within a set time range. The acquired multi-source heterogeneous data includes access control system access record data, video surveillance system video surveillance data, nursing management system medication record data, nursing management system nursing record data, and nursing management system visitor record data. A time calibration unit is connected to the data acquisition unit. The time calibration unit is used to calibrate the timestamps of the acquired multi-source heterogeneous data to obtain time-aligned multi-source heterogeneous data. A clustering fusion unit, connected to the time calibration unit, is used to perform clustering and fusion processing on time-aligned multi-source heterogeneous data based on time dimension, spatial dimension and entity dimension to obtain a set of behavior-related data for each corresponding entity. A processing unit, connected to the clustering and fusion unit, is used to construct a behavior timeline for each entity's behavior-related data set in chronological order. It is also used to convert the behavioral timeline into a structured report and save it.

8. The system for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 7, characterized in that, The time calibration unit includes a deviation detection module, a historical data acquisition module, a calibration processing submodule, and an alignment processing submodule. The deviation detection module is used to detect the clock deviation of the equipment in the access control system, video surveillance system, and nursing management system to obtain the equipment clock drift rate. The historical data acquisition module is used to acquire historical calibration data from the access control system, video surveillance system, and nursing management system. The calibration processing submodule is connected to the deviation detection module and the historical data acquisition module. The calibration processing submodule is used to calculate the time deviation correction using a time calibration function, which is: ΔT = w_source × K_source + w_drift × drift + w_history × ΔT_history, where ΔT represents the time deviation correction in seconds, w_source represents the data source weight coefficient, K_source represents the data source basic deviation coefficient in seconds, w_drift represents the drift rate weight, drift represents the device clock drift rate in seconds / hour, w_history is the historical weight, and ΔT_history represents the exponentially weighted average of historical calibration deviations in seconds. The calibration processing submodule is also used to calibrate the timestamps of multi-source heterogeneous data using the time deviation correction. The alignment processing submodule is connected to the calibration processing submodule. The alignment processing submodule is used to determine the corresponding time window for the calibrated timestamp according to the dynamic time window division rule, and to perform alignment operation on the multi-source heterogeneous data according to the time window, thereby obtaining time-aligned multi-source heterogeneous data.

9. The system for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 7, characterized in that, The clustering fusion unit includes a face recognition module and a clustering processing sub-module; The face recognition module is used to extract face features from video data in time-aligned multi-source heterogeneous data with the consent of the individual, and then associate the video data with the corresponding entities through the results of the face feature extraction. The clustering processing submodule is connected to the face recognition module. The clustering processing submodule is used to perform clustering and fusion processing on multi-source heterogeneous data using a clustering function to obtain clustering results. The clustering function is: C=w1·(Δt / T_ref)+w2·(Δs / S_ref)+w3·(1-Δe), where C represents the clustering distance and is a dimensionless value; w1, w2, and w3 represent the time dimension weight, spatial dimension weight, and entity dimension weight, respectively, and w1+w2+w3=1; Δt represents the time distance in the time dimension, in seconds (s), and T_ref represents the preset time reference scale, in seconds (s); Δs represents the spatial distance in the spatial dimension, in meters (m), and S_ref represents the preset spatial reference scale, in meters (m); Δe represents the entity similarity in the entity dimension, with a value range of [0,1], and (1-Δe) represents the entity distance in the entity dimension; Δt / T_ref, Δs / S_ref, and (1-Δe) are all dimensionless values, and C has a value range of [0,1].

10. The system for cross-modal temporal data alignment and behavior fusion for smart elderly care as described in claim 7, characterized in that, It also includes a video intelligent cropping unit, which is connected to the data acquisition unit; The data acquisition unit is also used to acquire captured images from a legally used video surveillance system. The intelligent video capture unit is used to extract facial features from captured images with the individual's consent, and then associate the captured images with entities based on the results of the facial feature extraction; it is also used to obtain the time of the captured image and the name of the camera that captured the captured image; it is also used to locate the storage channel of the video surveillance data according to the name of the camera, extract the corresponding video frames based on the set video capture time range, generate video segments, and associate and store the generated video segments with entities and time.

Images