A Method and System for Identifying Abnormal Behavior of Elderly People Living Alone Based on Multimodal Data Fusion

Abstract

This application provides a method and system for identifying abnormal behavior in elderly people living alone based on multimodal data fusion, relating to the field of intelligent recognition of abnormal behavior. The method includes real-time acquisition of smart door lock data containing arrival and departure times, as well as images of items carried by the elderly person; acquisition of voice interaction data such as voice messages and recorded conversations from the elderly person via a smart wearable device; activation of the wearable device's positioning function after detecting the elderly person leaving home, plotting the positioning data into a real-time route trajectory; determination of the travel intention, including the destination, by combining the smart door lock data and the voice interaction data; further determination of whether the elderly person is traveling normally or wandering aimlessly by combining the real-time route trajectory and travel intention; if wandering aimlessly, triggering a lost response, and sending a lost alert to emergency contacts upon successful response. Implementing this method improves the timeliness and accuracy of identifying abnormal behavior in the elderly, providing more comprehensive protection for the safety of elderly people living alone and reducing the risk of getting lost.

Description

Technical Field

[0001] This application relates to the field of intelligent recognition of abnormal behavior, and in particular to a method and system for recognizing abnormal behavior of elderly people living alone based on multimodal data fusion. Background Technology

[0002] With the increasing aging of the population, the phenomenon of elderly people living alone getting lost and forgetting their way home due to cognitive decline, sudden illness, and other reasons is becoming more and more frequent. According to statistics, the number of cases of elderly people with Alzheimer's disease getting lost is increasing by 15% every year, while ordinary elderly people may also lose their sense of direction when they go out due to physical reasons such as fainting from hypoglycemia or sudden stroke. Many elderly people who are prone to getting lost (such as Alzheimer's patients) may also experience short-term memory loss when they get lost, making it impossible for them to actively call for help from their family. Traditional ways of dealing with this often rely on family members to search for them afterward, or for the elderly to carry a simple identification card and wait for passersby to help them. These methods not only fail to intervene in time when the elderly are lost, but may also lead to delays in rescue due to missing information (such as lost cards or the elderly being unable to describe their address due to confusion), missing the golden time for rescue.

[0003] To address the aforementioned issues, existing technologies utilize GPS-based smart wearable devices for real-time tracking of the elderly, such as smart bracelets and location badges. These devices, through their built-in GPS modules and mobile communication chips, upload the elderly person's location information to a cloud-based anomaly detection system in real time. Family members can then view the elderly person's trajectory via a mobile app. If the device detects that the elderly person has deviated from a preset safe zone for a certain period, it automatically triggers an alert, prompting family members to take action. Compared to traditional methods, this approach significantly improves the efficiency of locating lost elderly individuals and reduces search time.

[0004] However, existing technology has difficulty in accurately distinguishing whether the deviation from the preset safety range is due to the elderly person's normal travel or abnormal behavior. If the system misjudges that the elderly person is in a safe state, it will miss the opportunity for timely rescue, which will seriously affect the rescue efficiency and the safety of the elderly person. Summary of the Invention

[0005] This application provides a method and system for identifying abnormal behavior of elderly people living alone based on multimodal data fusion. It is used to achieve accurate monitoring and abnormal identification of the travel status of elderly people living alone by integrating multi-source data such as smart door locks and smart wearable devices.

[0006] Firstly, this application provides a method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion, applied to an anomaly identification system. The method includes: real-time acquisition of smart lock data, including the time of arrival home, the time of departure home, and images of items carried by the elderly person captured by the smart lock camera; acquisition of the elderly person's voice interaction data via a smart wearable device, including voice messages between the elderly person and the smart wearable device and recordings of conversations with others; if the elderly person is detected leaving home, activation of the location function in the smart wearable device, and plotting the acquired location data into a real-time route trajectory; determination of the elderly person's travel intention by combining the smart lock data and the voice interaction data, the travel intention including at least one destination; determination of whether the elderly person is in a normal travel state or is lost and wandering, based on the real-time route trajectory and the travel intention; if the elderly person is lost and wandering, triggering a lost response; if the lost response is triggered, sending a lost alert to emergency contacts.

[0007] By employing the above technical solution, real-time data from smart locks (including travel time and images of carried items) and voice interaction data from smart wearable devices are acquired, and the status is determined by combining location trajectory and travel intention. Smart lock data provides basic travel time and item information, voice data reflects subjective intention, and location trajectory tracks movement in real time. Multimodal data fusion enables comprehensive perception of the travel status of elderly people living alone. When an elderly person becomes lost and wanders around, a response is triggered, and an alert is sent to emergency contacts, forming a closed loop of "monitoring-judgment-response-alert," improving the timeliness and accuracy of anomaly identification and effectively ensuring the safety of the elderly.

[0008] In some embodiments of the first aspect, the determination of the elderly person's travel intention by combining the smart lock data and the voice interaction data includes: determining whether the elderly person left a voice message within a set time interval before and after the departure time; if so, performing semantic recognition on the voice message, determining the extracted travel intention as the travel intention, and marking it as a normal travel status; if not, obtaining the elderly person's travel intention data for each time period within a set time period in the past; determining the time interval corresponding to the elderly person's travel intention data based on the departure time; and determining multiple travel intentions in the time interval as a first travel intention.

[0009] By employing the above technical solution, the travel intention is first determined by checking voice messages left before and after leaving home. If no messages are left, historical data is retrieved to match the intention for the corresponding time period. Voice messages directly reflect the immediate intention, while historical data reflects travel patterns; the combination of the two forms a dual basis for judgment. For those with messages, the status is directly marked as normal; for those without messages, the initial intention is determined based on historical patterns. This reduces judgment bias caused by the lack of immediate information, improves the efficiency and reliability of travel intention recognition, and lays an accurate foundation for subsequent status judgments.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, after determining the multiple travel intentions in the time interval as the first travel intention, the method further includes: performing feature recognition on the image of the elderly person's carried items to determine the elderly person's carried items and clothing features; based on the carried items and clothing features, determining the elderly person's second travel intention through an intention association feature library, which is pre-set and stores the elderly person's travel intentions and corresponding carried items and clothing features; if the second travel intention matches the first travel intention, then the travel intention is determined and marked as a normal travel state; if the second travel intention does not match the first travel intention, then each of the first travel intentions and the second travel intention is marked as a state to be verified by multimodal data.

[0011] By employing the above technical solution, after determining the primary travel intention from historical data, the secondary travel intention is identified by combining the characteristics of carried items and clothing, and then verified by matching with an intention-related feature database. Items and clothing are objective manifestations of travel intention, forming cross-validation with historical intentions. During matching, the intention is confirmed and marked as normal; if there is a mismatch, it is marked as needing verification. This avoids the one-sidedness of judging from a single data point, improves the accuracy of travel intention recognition through multi-dimensional feature comparison, and reduces the probability of misjudgment.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of marking each of the first and second travel intentions as pending multimodal data verification states if the second travel intention does not match the first travel intention, the method further includes: after determining that the elderly person has left home and has not left home beyond a set distance, controlling the infrared sensor array in the wearable device to turn on; when the infrared sensor array detects the existence of a second body temperature field within a set infrared range and the existence time exceeds a set time threshold, controlling the wearable device to turn on the recording function; acquiring a recording of the elderly person's conversation with others, and performing semantic recognition on the recording to determine whether it contains travel intention features; if so, determining the travel intention based on the travel intention features, and changing the status information from pending multimodal data verification state to normal travel state; when the infrared sensor array detects the disappearance of the second body temperature field, turning off the infrared sensor and stopping the recording.

[0013] By adopting the above technical solution, when the intent does not match, the infrared sensor is activated if the elderly person has not exceeded a set distance from home. Upon detecting others, the sensor records audio and semantically recognizes their travel characteristics. The infrared sensor precisely triggers the recording, and the recorded dialogue supplements the real-time intent information, correcting the pending verification state to a normal state. This mechanism utilizes external interaction information to fill the gaps in intent judgment, dynamically improves the recognition logic, enhances adaptability to complex scenarios, and further improves the completeness of intent recognition.

[0014] In some embodiments of the first aspect, the determination of whether an elderly person is in a normal travel state or is lost and wandering by combining the real-time route trajectory with the travel intention includes: acquiring historical travel data and identifying all routes to the target location in the historical travel data; identifying route segments that appear more frequently than a preset frequency threshold as the necessary path for the current trip; extending the necessary path outward with the farthest point of the historical trip as the radius to determine the reachable area for the current trip; if the real-time route trajectory is located on the necessary path, or deviates from the necessary path but is still within the reachable area, and the direction of movement is continuously towards the target location, then it is determined to be a normal travel state; if the real-time route trajectory continuously deviates from the necessary path by more than a preset distance threshold and does not enter the reachable area, or repeatedly turns back outside the reachable area, then it is determined that the elderly person is lost and wandering.

[0015] By employing the above technical solution, the necessary routes and reachable areas are determined based on historical data. The status is then assessed by comparing the real-time trajectory with the route, area, and target direction. The necessary routes and reachable areas define the normal travel range, while the degree of trajectory deviation and reversal behavior serve as anomaly indicators. A normal trajectory is within the range and faces the target; an abnormal trajectory indicates deviation or wandering. By quantifying spatial and behavioral characteristics, a clear distinction is achieved between normal travel and wandering / lost travel, improving the objectivity of status assessment.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, if the elderly person is lost and wandering, a lost response is triggered, including: controlling the wearable device to send a lost response prompt to the elderly person; if no response is received from the elderly person, sending a second lost response prompt to the elderly person after a set time, and controlling the wearable device to flash a warning; after receiving the elderly person's lost response information, performing semantic recognition on the lost response information to obtain a lost response result; and determining whether the elderly person has passed the lost response based on the lost response result.

[0017] By employing the above technical solution, when an elderly person becomes lost and wanders around, a prompt is first issued. If there is no response, a second prompt is issued and a flashing warning is displayed. Upon receiving a response, semantic recognition is performed. Tiered prompts ensure the elderly person's awareness, flashing warnings enhance the alerting effect, and semantic recognition verifies the actual situation. This process confirms the elderly person's condition through progressive interaction, avoids false alarms, balances the timeliness and accuracy of warnings, and provides a reliable basis for subsequent actions.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of issuing a second lost response prompt to the elderly after a set time and controlling the wearable device to flash a warning if no response is received from the elderly, the method further includes: if no lost response information is received from the elderly, automatically confirming that the elderly have passed the lost response.

[0019] By adopting the above technical solution, the system automatically confirms a lost response if no response is received from the elderly person. This design takes into account the possibility that the elderly person may be unable to respond due to being lost, avoiding delays in early warning due to lack of response. After multiple unsuccessful prompts, subsequent warnings are automatically triggered, ensuring timely response in emergencies, filling the gap in response when the elderly person becomes disabled, and further protecting their safety.

[0020] In a second aspect, this application provides an anomaly identification system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the anomaly identification system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on an anomaly detection system, cause the anomaly detection system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product that, when run on an anomaly detection system, causes the anomaly detection system to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. By adopting real-time acquisition of smart door lock data (including travel time and images of carried items), voice interaction data and location trajectory of smart wearable devices, and combining multimodal data fusion to judge travel status and form a closed loop of "monitoring-judgment-response-early warning", the technology effectively solves the problems of single monitoring of the travel status of elderly people living alone and lagging abnormal identification in existing technologies. In this way, it realizes timely and accurate identification of abnormal behavior of the elderly and improves the comprehensiveness of safety protection.

[0025] 2. By employing a technical approach that uses infrared sensors to monitor others and trigger recording when the travel intentions do not match and the elderly person has not exceeded a set distance from home, and uses the recorded conversations to supplement the travel intention information, the problem of incomplete travel intention judgment caused by insufficient single data in existing technologies is effectively solved. This enables dynamic and comprehensive recognition of the elderly person's travel intentions in complex scenarios, and enhances the scenario adaptability of the technical solution.

[0026] 3. By employing a technical approach that uses historical data to determine the necessary routes and reachable areas, and then comparing real-time route trajectories with the route, area, and target direction to distinguish between normal travel and wandering, the technology effectively solves the problem of existing technologies lacking quantitative judgment standards for the travel status of the elderly and being prone to misjudgment. This achieves an objective and accurate definition of the travel status of the elderly and improves the reliability of anomaly identification. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of an application framework for a method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion in this application embodiment;

[0028] Figure 2 This is a flowchart illustrating a method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion in an embodiment of this application.

[0029] Figure 3 This is another flowchart illustrating the method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion in this application embodiment;

[0030] Figure 4 This is a schematic diagram of the physical device structure of an anomaly recognition system in the embodiments of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] For ease of understanding, the application framework of the method provided in this implementation is described below. Please refer to [link / reference]. Figure 1 This is a schematic diagram illustrating the application framework of the method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion in this application embodiment.

[0034] exist Figure 1In this system, the server in the anomaly detection system communicates with the smart lock in the elderly person's room. The smart lock communicates with a camera. Each time the elderly person opens the door, the smart lock records the time of opening and closing the door and sends it to the server. Simultaneously, the camera is activated to capture images of the elderly person's belongings. This camera can be installed outside the door lock, near the upper corner of the wall, or it can be installed by relevant personnel in other indoor or outdoor locations to monitor the elderly person's behavior and belongings. There are no restrictions on this. The elderly person can carry a smart wearable device prepared in advance by their family. This smart wearable device can be a small device like a watch for easy carrying. The smart wearable device has a positioning module for basic positioning functions and a voice recording function to record conversations between the elderly person and others at set times. All of the above content and the functions of each device are known and used in advance by the elderly person and their family, and will not infringe on personal privacy information.

[0035] The smart wearable device sends the elderly person's location information and voice recordings or recorded voice interaction data to the server. If the server determines that the elderly person is lost based on the above information, it sends a lost person alert to the receiving end of the pre-prepared emergency contact.

[0036] For ease of understanding, the method provided in this implementation is described in process below. Please refer to [link / reference]. Figure 2 This is a flowchart illustrating a method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion in an embodiment of this application.

[0037] S201. Real-time acquisition of smart lock data, including home arrival time, departure time, and images of items carried by the elderly person captured by the smart lock camera;

[0038] Among them, "smart lock data" refers to the data set generated by smart locks and their accessories that reflects the elderly’s entry and exit behavior and related status, such as the elderly’s daily entry and exit records and the items they carry; "smart lock camera" refers to the image acquisition device integrated into the smart lock, which usually has infrared night vision function and is used to capture the scene in front of the door; "image of items carried by the elderly" refers to the visual data captured by the smart lock camera that includes the items carried by the elderly when they leave or return home, such as the image of the elderly carrying a shopping basket when they leave home.

[0039] This step is executed when the smart lock undergoes a status change (such as unlocking or locking) or when the camera completes image acquisition. Specifically, the anomaly detection system establishes a stable communication connection with the smart lock. First, regarding the acquisition of arrival and departure times, the system connects to the smart lock's operation log module. When the elderly person uses fingerprints, passwords, cards, or a mobile app to open or close the lock, the smart lock immediately records the specific time of the operation. If some locks are not so smart, and elderly people living alone only use physical keys to open the lock, the camera can send the corresponding time period video to the server after capturing the elderly person taking out the key to open the door. The server determines the time when the door is opened as the arrival time. Similarly, if it detects that no one opened the door from the outside, but the camera captures the door opening and the elderly person leaving from the inside, the time when the elderly person closes the door after leaving is determined as the departure event.

[0040] For images of elderly people carrying items captured by smart door lock cameras, the camera will start shooting according to preset trigger conditions (such as within 10 seconds of detecting the door lock opening action). It typically captures multiple frames of images or short video clips to ensure clear capture of the entire process of the elderly person entering and leaving the house. Afterwards, the system uses edge computing or local AI processing modules to perform preliminary analysis on these images, such as detecting the elderly person's outline, movement trajectory, and items in their hands, arms, and surroundings. To improve recognition accuracy, the system automatically filters background clutter and other irrelevant personnel, extracts and categorizes features of items such as canes, handbags, shopping baskets, medicine bags, and water bottles, and binds the extracted item tags to the timestamp of the entry / exit event, forming a structured "entry / exit event" record. All collected data, including the time of arrival and departure, along with associated images of carried items and recognition results, will be synchronously uploaded to the central server of the anomaly detection system and managed in association with the elderly person's personal identity.

[0041] S202. Obtain the elderly’s voice interaction data through smart wearable devices. The voice interaction data includes the elderly’s voice messages with the smart wearable devices and recordings of conversations with others.

[0042] Among them, "voice interaction data" refers to the collection of audio information generated when the elderly communicate with the device itself or others through voice using smart wearable devices. This includes both voice information recorded by the elderly actively operating the device and environmental voices automatically collected by the device in daily life. "Voice messages" refers to the voice content left by the elderly when they actively record messages on the device. This content usually reflects the elderly's immediate intentions, plans, or needs. "Dialogue recordings" refers to voice dialogue segments that are automatically collected and saved by the device during natural communication between the elderly and others. These segments contain the content of the communication between the two parties and can help analyze the elderly's social and behavioral purposes.

[0043] Specifically, this step is typically performed when the elderly wear smart wearable devices at home or while out and about. After the device is powered on and establishes communication with the backend system, the anomaly detection system continuously monitors the device's audio input module according to a pre-set collection strategy. The smart wearable device collects voice data in various modes, including when the elderly actively operate the device to leave voice messages, such as by pressing buttons, touching, or using voice activation. The system automatically tags the recordings with time, location, and identity information, categorizing them as "voice messages." To prevent the elderly from forgetting or being unable to use the smart wearable device, the system can also automatically send a recording activation message to the audio input module of the connected smart wearable device after the server detects that the elderly person has left home. For conversation recordings, the device uses an automatic voice acquisition algorithm to monitor changes in ambient sound sources in real time through sound activity detection. When it detects voice interaction between the elderly person and others, it automatically separates the voices of both parties and marks them as "conversation recording" data. All collected audio data undergoes preliminary noise reduction and voice enhancement processing locally to ensure recording quality before being encrypted and uploaded to the anomaly detection system backend. The backend system uses ASR (Automatic Speech Recognition) technology to transcribe the audio stream into text. Through natural language processing algorithms such as keyword extraction and intent recognition, it further extracts core information related to the elderly person's daily life, travel plans, and health status. The system also associates the collected voice data with metadata such as the elderly person's identity, device serial number, collection time, and geographical location. To protect the elderly person's privacy, all raw audio data and transcribed text are encrypted and stored, with strictly controlled access permissions.

[0044] S203. If the elderly person is detected to have gone out, the positioning function in the smart wearable device will be turned on, and the obtained positioning data will be plotted as a real-time route trajectory.

[0045] Among them, "real-time route trajectory" is used to represent the actual movement path of the elderly formed by splicing these location data in chronological order, which facilitates subsequent abnormal behavior analysis and visual tracking.

[0046] Specifically, this step is executed after the elderly person leaves their residence and the system determines that they have left. The anomaly detection system monitors various signals in real time, including changes in door lock status, wearable device accelerometer data, and changes in the door camera image. Once it determines that the elderly person has left, it immediately issues a command to activate the wearable device's positioning function. The device automatically switches to high-precision positioning mode and continuously acquires positioning points according to the sampling frequency set by the system. All positioning data is first cached locally and then uploaded to the backend server in real time via an encrypted communication module. The backend system performs trajectory stitching, map matching, and error correction on the received positioning point data. It uses algorithms such as trajectory smoothing, outlier removal, and missing point interpolation to generate a continuous and visualized route trajectory. The trajectory data can not only be used to display the elderly person's movement process in real time, but also be linked with data models such as historical trajectories, target locations, and necessary paths to achieve subsequent identification of behaviors such as deviation, wandering, and abnormal stays.

[0047] In some embodiments, elderly people may find their smart wearable devices uncomfortable when they go out, or forget to bring them back for other reasons, such as leaving them on a chair outdoors. Leaving the smart wearable device unattended makes it difficult to accurately track the elderly person's location data, making it hard to determine whether the elderly person is lost or simply out and about. In this situation, by continuously analyzing the location data, if the duration of the elderly person's current real-time location exceeds the longest historical duration for that location, a determination can be made as to whether the extended stay is normal (such as talking to someone) or due to a lost smart wearable device causing location anomalies, thus determining whether the elderly person is lost.

[0048] First, the system receives location data uploaded by the smart wearable device in real time, determines the elderly person's current location through coordinate matching, and starts a timer to count the real-time duration of the elderly person's stay at that location. Simultaneously, the system retrieves the longest historical stay duration for that location from the historical database (if it's a new location, it refers to historical data from similar nearby locations, such as the stay data for a new park visited for the first time, referring to similar parks). When the real-time stay duration exceeds the longest historical stay duration, the system triggers the first judgment: the system sends a "start infrared sensor array" command to the smart wearable device. Upon receiving the command, the device starts infrared detection within a preset range (e.g., a 0.3-0.8 meter area facing the body), collecting infrared temperature signals at a frequency of 2 times per second, and transmitting the temperature data back to the system in real time. The system compares the returned temperature data with a pre-established target body temperature field threshold range (e.g., 36.2-36.8℃) specifically for the elderly. If more than 80% of the temperature data falls within the target body temperature field threshold range during a continuous 1-minute detection period, and the temperature field distribution conforms to the human body contour (e.g., when worn on the wrist, the temperature field is elongated and matches the arm contour), then it is determined that "a target body temperature field exists." This indicates that the smart wearable device is still being worn by the elderly person and has not been lost. The current location data accurately reflects the elderly person's location, and the location data is in a normal positioning state. The system will not determine any abnormality at this time, but will continue to monitor the elderly person's subsequent movement trajectory.

[0049] If, within a 1-minute detection period, the temperature data is consistently below or above the target body temperature field threshold (e.g., a detected temperature of 25°C is ambient temperature, or 38.5°C is abnormally high, neither of which corresponds to a normal body temperature for the elderly), or if the temperature field distribution lacks human body contour features (e.g., it appears as dots or is irregularly distributed, possibly indicating that the device was placed on the ground and detected an ambient heat source), then it is determined that "no target body temperature field exists." This means that the smart wearable device has been separated from the elderly person (e.g., the elderly person left the device on a bench or forgotten it on a store counter). The current location data only reflects the device's location and cannot represent the elderly person's actual location. The location data is not in a normal location state, and the system will immediately mark it as "device lost."

[0050] After the system completes the device status determination, it enters the "Return Home Confirmation" stage, waiting until the set latest time of the day (e.g., 21:00). When the set latest time arrives, the system queries the smart lock's operation records for the day through the communication interface with the smart lock: if a valid door opening record before the set latest time is found (e.g., a fingerprint unlocking record at 20:50, and the unlocking fingerprint matches the elderly person's preset fingerprint), it means the elderly person has returned home normally, and regardless of whether the device was lost before, it is determined to be "no risk of getting lost," and the process ends; if the query finds that there are no door opening records for the day up to the set latest time (or only exit records, no entry records), it further analyzes the previous device status determination results: if the device was previously determined not to be lost, it means the elderly person may be unable to return home due to cognitive impairment, physical discomfort, or other reasons, and has become lost; if the device was previously determined to be lost, it means the elderly person may be lost in an unknown area after losing the device and has not returned home because they cannot be located. If both scenarios meet the criteria for "elderly person getting lost", the system will immediately initiate the lost person alert process. It will send information such as the elderly person's last valid location (current location if the device is not lost, and last location before the device was lost) and device status (whether it is lost) to the emergency contact via SMS, APP push, telephone notification, etc.

[0051] S204. Combine the smart lock data and the voice interaction data to determine the elderly person's travel intention, which includes at least one target location;

[0052] Among them, "travel intention" refers to the main purpose and goal of the elderly person's trip, including but not limited to shopping, medical treatment, taking a walk, visiting relatives, etc., and is usually characterized by "target location" such as supermarket, hospital, park, etc.

[0053] The detailed method for determining travel intentions will be described in steps S301-S315, and will not be repeated here.

[0054] S205. Combine the real-time route trajectory with the travel intention to determine whether the elderly person is in a normal travel state or is lost and wandering.

[0055] Among them, "normal travel status" means that the elderly person's current movement trajectory is highly consistent with their historical patterns and current travel intentions, and there are no abnormal deviations or abnormal behaviors. "Lost and wandering" means that the trajectory analysis reveals that the elderly person has abnormal movement characteristics such as long-term deviation from the established path, trajectory reversal, and wandering, and these behaviors are obviously inconsistent with the travel intentions, and there is a risk of getting lost or cognitive impairment.

[0056] Specifically, this step is executed as follows: after determining the elderly person's travel intention, the system starts generating a real-time route trajectory on the smart wearable device and continues until the elderly person returns home or is confirmed safe. Its core scenario is that during the elderly person's outing, the system needs to dynamically compare the match between the route and the intention to promptly identify potential risks of getting lost, preventing the elderly person from getting lost due to cognitive impairment, memory decline, or other reasons.

[0057] First, the system needs to establish a judgment benchmark based on historical travel data. The first step is to extract all routes from historical travel data that lead to the target location included in the current travel intention. Through big data statistical analysis, route segments that appear more frequently than a preset frequency threshold (e.g., 60%) are marked as "must-pass routes"—that is, the route segments that the elderly person is likely to have taken when traveling to that target location in the past. The second step is to define the "accessible area": ​​with the must-pass route as the center, an area with a radius equal to the "farthest point of historical travel" (the "farthest point of historical travel" refers to the maximum distance the elderly person has deviated from the must-pass route when traveling to that target location in the past) is formed, creating a dynamic geographical range as the boundary for judging whether the route is reasonable. For example, assuming an elderly person's travel intention is "to the vegetable market at the east gate of the community," the system extracts their historical route data from the past 30 trips to that market and analyzes it using the following steps: It counts the frequency of each route segment in the 30 trips: Starting from the south gate of the community → heading east along the main road within the community → exiting the east gate → turning right 50 meters to the vegetable market. This route segment appeared 21 times in the 30 trips, a frequency of 70% (exceeding the preset threshold of 60%), and is therefore marked as a "must-pass route." Other occasionally occurring routes (such as detouring from the north gate of the community) are not included in the must-pass routes because their frequency is less than 30%. The system analyzes the maximum deviation distance the elderly person made when traveling along the above must-pass routes in historical data: Past records show that the elderly person once deviated 15 meters from the route in the "east gate to vegetable market" section of the must-pass route due to a last-minute breakfast purchase (entering a roadside breakfast shop), which is the "farthest point in their historical trip." Therefore, the area formed by extending a 15-meter radius outward from the must-pass route (including the must-pass route and the area within 15 meters) is defined as the "reachable area." As can be seen from the above examples, "must-pass routes" are the core routes that the elderly frequently choose, while "accessible areas" provide a reasonable buffer range for temporary deviations. The combination of the two can accurately determine whether the current trip is normal.

[0058] Subsequently, the system compares the current real-time route trajectory with the aforementioned benchmark in real time. If the real-time route trajectory meets one of the following conditions, it is determined to be in a normal travel state: 1) the trajectory falls entirely on the required path; 2) although the trajectory deviates from the required path, it remains within the reachable area, and direction vector analysis (e.g., calculating the angle between the movement direction and the target location every 5 minutes) confirms that the movement direction continues towards the target location. If the real-time route trajectory exhibits one of the following situations, it is determined to be lost and wandering: 1) the distance continuously deviating from the required path exceeds a preset distance threshold (e.g., 200 meters), and the system has not entered the reachable area; 2) repeated back-and-forth behavior occurs outside the reachable area (e.g., reversing direction more than 3 times within 10 minutes), or the time spent at the same location exceeds a preset dwell time threshold without a clear movement trend.

[0059] The system also performs statistical analysis on characteristics such as the distribution density, dwell time, and movement speed of trajectory points, and uses anomaly detection algorithms to determine whether the anomalies are habitual or sudden. For newly emerging target locations or trajectory patterns, the system can automatically expand the path library using semi-supervised clustering, dynamically updating the mandatory paths and reachable area models to improve the system's adaptability to new scenarios. If a trajectory changes state multiple times within a short period, the system will use a sliding window mechanism to smooth the judgment and avoid false alarms. All judgment results and trajectory analysis processes are automatically archived for subsequent behavior tracking and security intervention.

[0060] S206. If the elderly person is lost and wandering, then trigger the lost response.

[0061] Among them, "lost and wandering" refers to the state of the elderly person's route trajectory deviating from the reasonable path or turning back aimlessly, as defined in S205; "triggering a lost response" refers to the system actively initiating an interaction with the elderly person through a smart wearable device after determining that the elderly person is lost and wandering, in order to confirm whether the elderly person is actually lost.

[0062] After determining in step S205 that the elderly person is in a lost and wandering state, the system activates within a set time limit, such as 10 seconds, to ensure that the elderly person's status is confirmed in time before any potential danger arises. The core scenario is that the system infers that the elderly person is at risk of getting lost through trajectory analysis, but it needs to rule out false alarms (such as the elderly person temporarily taking a detour to buy something, resulting in a brief deviation in the trajectory that does not constitute a true loss). Therefore, it is necessary to verify the elderly person's actual status through direct interaction.

[0063] Specifically, the detailed execution process of this step is as follows: First, the system sends a command to the smart wearable device to trigger the initial lost response prompt. The prompt should be concise and clear, suitable for the elderly person's comprehension. For example, the device might play a voice message: "We have detected a possible deviation in your route. Are you currently unable to find your way?" Simultaneously, if the device has a screen, it should display the text "Do you need help?", ensuring both visual and auditory prompts. Second, the device enters a waiting phase, setting the initial waiting time to a preset, relatively long duration, such as 40 seconds (considering the elderly person's potentially slower reaction time). During this period, the microphone is activated to listen for voice input, and the operation signals of physical buttons (such as the "Yes / No" button on the side of the device) are monitored. Third, if no response is received within 40 seconds (no voice, no button operation), the system triggers a second lost response prompt: the voice prompt volume is increased by 20%, the initial question is repeated, and an LED flashing warning (red LED) is activated. The waiting time is then shortened to a preset, shorter duration, such as 25 seconds (to avoid excessive waiting and potential delays). The fourth step involves processing the response from the elderly person if a response is received between the two prompts. For voice responses, the system extracts semantics using a pre-trained semantic recognition model (optimized for common elderly speech and supporting dialect recognition). If the response contains keywords such as "lost," "cannot find," or "don't know where to go," it is determined as "confirmed lost." If the response contains keywords such as "it's okay," "I know," or "take a detour," it is determined as "not lost." For button responses, pressing the "Yes" button confirms "confirmed lost," while pressing the "No" button confirms "not lost." The fifth step generates the final lost response result: if the result is "confirmed lost" or "no response," it is determined as "passed the lost response"; if the result is "not lost," it is determined as "failed the lost response," and the system terminates subsequent operations and continues to monitor the trajectory.

[0064] In some embodiments, the triggering and interaction of the lost response can be achieved in a variety of ways: Optionally, with voice interaction as the core, the smart wearable device uses a dual-microphone array to achieve noise reduction (adapting to noisy outdoor environments), and the voice prompts can use voices familiar to the elderly (such as recording the voices of relatives) to improve acceptance. The semantic recognition model optimizes the recognition accuracy of the elderly's accent and unclear pronunciation through transfer learning; Optionally, combining voice and tactile interaction, the device vibrates synchronously when the first prompt is made (3 short vibrations), and the vibration intensity is increased when the second prompt is made (2 long vibrations). For elderly people with hearing loss, it is supported to directly trigger the "confirm lost" signal by pressing and holding the device button (such as pressing and holding for 5 seconds), while blocking accidental touches (such as no response within 1 second of short press).

[0065] In some embodiments, there may be accidental triggers of the lost response, leading to elderly users becoming resistant and turning off their devices due to frequent prompts. To avoid this, the system can establish a "Lost Response Trigger-Feedback" log database, recording in real time the trigger time, trigger reason (such as trajectory deviation or prolonged stay), and elderly user feedback result for each lost response (categorized as "Confirmed Not Lost," "No Response but Trajectory Resumed," "False Trigger Warning," and "Confirmed Lost"). When this step is required, the system extracts the records of the "most recent n triggers" from the log database (n is 7 by default; if the elderly user's weekly travel frequency is low, the emergency contact can adjust it to 5 in the system backend), counts the total number of triggers for these n times (i.e., the value of n), and counts the number of times the elderly user's feedback result is "Confirmed Not Lost," "False Trigger Warning," or "No Response but Trajectory Resumed Normal." The system compares the statistical results with a preset tolerance threshold: First, it determines whether the number of triggers exceeds the preset tolerance threshold (e.g., if the preset tolerance threshold is 4 times, and 5 out of the last 7 triggers occurred within 7 days, then the "number of triggers > preset tolerance threshold" condition is met). Second, it calculates the total percentage of "confirmed not lost" and "false alarm triggers" (e.g., if 3 out of 5 triggers are confirmed not lost + 1 is a false alarm trigger, the total percentage is 80%). If this total percentage exceeds the preset percentage threshold (e.g., 60%), it indicates that most recent triggers are false alarms, and the frequency has exceeded the elderly person's tolerance range, triggering the "judgment threshold increase" mechanism. The preset tolerance threshold refers to the maximum number of triggers that the elderly person can accept (usually 3-5 times / 7 days, set by the emergency contact person based on the elderly person's psychological tolerance), used to determine whether the triggering frequency is too high.

[0066] The system performs a threshold-raising operation by expanding the radius of the reachable area based on the total percentage. Specifically, the expansion rule is: "For every 10 percentage points the total percentage exceeds the preset threshold, the reachable area radius increases by 10%" (the expansion ratio can be adjusted linearly, with an upper limit of 50% of the original radius to avoid excessive expansion leading to security vulnerabilities). For example: if the original reachable area radius is 100 meters, the preset percentage threshold is 60%, and the current total percentage is 80% (exceeding the threshold by 20 percentage points), the radius will increase by 20%, adjusting to 120 meters; if the total percentage is 90% (exceeding the threshold by 30 percentage points), since the upper limit is 50%, the radius will increase to 150 meters (100 meters × 150%). After expanding the radius, the system's criteria for determining "lost and wandering" are correspondingly relaxed: the elderly person's real-time route trajectory must deviate beyond the new radius range to trigger the judgment (e.g., previously triggered at 100 meters, now triggered at 120 meters). At the same time, the system will record the time and reason for this threshold adjustment (e.g., "total percentage 80% > 60%, triggering expansion by 20%) and the new radius value, and simultaneously push it to the emergency contact's APP, informing them that "to reduce accidental triggers, the reachable area has been expanded, and the current radius is 120 meters," ensuring that family members are aware of the adjustment.

[0067] Exceeding the threshold for trigger counts only indicates "frequent prompts," but frequent prompts can have two distinct causes: Cause 1 is primarily accidental triggers – most prompts are misjudged by the system (e.g., location jumps causing trajectory deviations, temporary detours being judged as wandering), while the elderly person is actually in a normal travel state (e.g., 4 out of 5 triggers are accidental triggers); Cause 2 is primarily genuine anomalies – most prompts indicate the elderly person is indeed at risk of getting lost (e.g., 4 out of 5 triggers are genuine loss of consciousness, with only 1 accidental trigger). If the total percentage of accidental triggers is not verified, and the judgment threshold is raised directly due to "high number of triggers" (e.g., expanding the reachable area), it will lead to serious security vulnerabilities. For example, an elderly person with cognitive decline may have recently experienced frequent brief periods of disorientation (5 triggers in 7 days, all genuine anomalies). If the system expands the reachable area simply because "number of triggers exceeds the threshold" (e.g., expanding from 100 meters to 120 meters), the system may delay warnings when the elderly person actually gets lost due to an excessively high judgment threshold, missing the golden rescue time.

[0068] The purpose of setting a preset percentage threshold (e.g., 60%) is to verify whether "frequent prompts are mainly caused by accidental touches": when the total percentage of "confirmed not lost + accidental triggering of warnings" exceeds the threshold (e.g., 80%), it indicates that the core reason for the recent frequent prompts is that the system's judgment criteria are too strict (e.g., the original reachable area radius is too small, and normal detours trigger the judgment), rather than an increase in actual abnormalities among the elderly. In this case, raising the judgment threshold (expanding the reachable area) is essentially to "make the judgment criteria adapt to the actual travel habits of the elderly" and reduce meaningless accidental touch prompts. If the total percentage does not exceed the threshold (e.g., only 40%), it indicates that the frequent prompts are caused by "an increase in actual abnormalities among the elderly" (e.g., a recent decline in the elderly's cognitive function, increasing the risk of getting lost). In this case, not only should the judgment threshold not be raised, but monitoring should be strengthened (e.g., shortening the response prompt interval) to ensure timely capture of actual abnormalities.

[0069] After adjustment, the system continuously monitors subsequent triggering, and re-counts the most recent n triggering data every 3 days: if the number of triggers drops below the preset tolerance threshold and the total percentage is lower than the preset percentage threshold (e.g., 3 triggers, 50% total percentage), the original judgment threshold is gradually restored (10% each time, until the original radius is reached); if the conditions for improvement are still met, the radius is maintained or further expanded (not exceeding the upper limit), forming a dynamic adjustment closed loop, which avoids frequent accidental triggers that cause resistance from the elderly, while ensuring effective monitoring of real disorientation.

[0070] S207. If the missing person response is received, a missing person alert will be sent to the emergency contact.

[0071] Among them, "passing the lost response" refers to the lost response result obtained in S206 being "confirmed lost" or "no response" (i.e., the system determines that the elderly person is indeed at risk of getting lost); "emergency contact" refers to the person that the elderly person or their relatives have set in the system in advance and who should be notified first in case of an emergency; "sending a lost warning" refers to the system transmitting a warning message containing the elderly person's lost status, real-time location, and relevant data to the emergency contact in order to trigger a rescue operation.

[0072] If the system confirms that the elderly person is lost, it is necessary to promptly notify the emergency contact person to shorten the rescue response time and reduce the risk of the elderly person getting lost or accidentally injured.

[0073] Specifically, the detailed execution process of this step is as follows: First, the system verifies the validity of the "responding to being lost" result, ruling out device malfunctions (such as a damaged microphone causing a false "no response"), and confirms data reliability by checking the device status log (e.g., whether the positioning module is functioning properly, whether the microphone is blocked). Second, the system automatically collects the necessary information for the warning, including: the elderly person's real-time location coordinates (accurate to within 10 meters), current time, criteria for determining the lost state (e.g., "continuous deviation from the expected path by 300 meters, two unresponsive responses"), a 5-minute route trajectory segment (presented as a dynamic link), and the remaining battery power of the smart wearable device (to assess the device's battery life). Third, a structured warning content is generated, using a "priority + details" format, for example: "[Urgent] Your relative Zhang XX may be lost. Location: Intersection of XX Road and XX Street (location link), Status: Abnormal trajectory since 10:20, no response. Please contact or go for assistance as soon as possible." The system also includes the trajectory segment link collected in the previous step and the device battery information. Fourth, the system prioritizes emergency contacts (e.g., children first, community grid workers second) and selects a notification method: it first calls the first contact (playing a voice alert simultaneously). If the call is not answered within 30 seconds, it sends an SMS containing the above information and pushes a notification through a linked app (e.g., WeChat mini-program notification). If the first contact does not confirm receipt within 10 minutes (e.g., app read confirmation), the system automatically repeats the notification process for the next highest-priority contact. Fifth, the system records the alert sending time and reception status (e.g., "answered" or "read SMS"). If all contacts do not respond within 30 minutes, it automatically sends a request for coordinated assistance to the local elderly care service hotline or community emergency center.

[0074] In the above embodiment, by comprehensively utilizing multimodal data fusion technology and combining smart lock data, voice interaction data, and real-time location trajectory, a complete closed loop is constructed from inferring travel intention to recognizing a lost state and then to emergency response. Therefore, it can accurately infer the elderly's travel intention and quickly distinguish between normal travel and a lost wandering state by dynamically comparing real-time trajectory with intention. At the same time, a lost response mechanism is introduced for secondary verification, which effectively solves the problems of high false alarm rate, delayed recognition of lost state, and untimely emergency response caused by relying on single data (such as location alone) in the traditional monitoring of elderly people living alone. Thus, it realizes intelligent and refined prevention and control of the risk of getting lost when elderly people living alone go out, improves the safety and autonomy of elderly people living at home, and provides timely and reliable emergency early warning support for emergency contacts.

[0075] Based on the above, the implementation process of step S204 will be described in detail below. Please refer to [link / reference]. Figure 3 This is another flowchart illustrating the method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion in this application.

[0076] S301. Determine whether the elderly person left a voice message within a set time interval before and after leaving home;

[0077] The time range for setting the duration refers to a fixed time range extending forward and backward from the time of departure (e.g., 30 minutes before and after). This duration can be personalized according to the elderly person's daily habits (e.g., 45 minutes before and after for elderly people with slower mobility). Voice messages refer to voice information that the elderly person actively records and stores through smart wearable devices (e.g., smart bracelets, watches). These messages usually include the travel destination, estimated return time, etc., and are used to convey the travel intention to the system.

[0078] Specifically, the system needs to parse the departure time uploaded by the smart lock (e.g., "08:30") and generate a time range based on preset rules—for example, if the duration is set to 30 minutes, the time range is "08:00-09:00". Then, the system accesses the voice interaction database, retrieves all voice files uploaded by the elderly's smart wearable device within that time range, and filters out voice data that meets the criteria based on file attributes (such as recording time and device identifier). Next, the system performs preliminary verification on the filtered voice data, excluding invalid audio recorded by accident (such as audio containing only noise or less than 3 seconds in length), and retaining valid voice messages that may contain travel information.

[0079] If a voice message is left, proceed to step S303;

[0080] If no voice message is left, proceed to step S302.

[0081] S302. If not, then obtain the travel intention data of the elderly in each time period within the past set time.

[0082] Among them, the past set time refers to the time range for extracting historical data (such as the past 3 months or 6 months) preset by the system, which can be dynamically adjusted according to the amount of data (automatically extended when the amount of data is insufficient); each time period refers to dividing a day into multiple fixed time segments (such as "06:00-08:00", "08:00-12:00", etc., each time period is 2-4 hours) to match the time period characteristics of the current time away from home.

[0083] Specifically, the immediate execution after S301 determines "no" is a crucial step in inferring travel intentions based on historical behavioral patterns when the elderly person has not left a message. The core scenario is for elderly people who do not frequently use voice messaging functions (such as those who are not accustomed to operating smart devices), using the correlation between their past travel time and intentions to initially identify possible travel destinations.

[0084] Specifically, the system reads a preset "past set time" parameter (e.g., 3 months by default) and verifies whether there is sufficient travel intention data (e.g., at least 10 records) within this time range. If the data volume is insufficient, the system automatically extends the time range (e.g., to 6 months) until the minimum data volume requirement is met, avoiding inference bias due to insufficient samples. Subsequently, the system divides the historical travel intention data into time periods, dividing the day into 12 time periods at "2 hours / segment" and labeling each data record with a corresponding time period tag (e.g., a trip at "08:30" is labeled "08:00-10:00"). Next, the system extracts the time period corresponding to the current departure time (e.g., "08:30" corresponds to "08:00-10:00") and filters all travel intention data belonging to the same time period from historical data, forming a preliminary dataset. Finally, the system cleans this dataset, removing abnormal data (e.g., incorrect intention records due to equipment malfunction) and retaining valid data, providing a foundation for subsequent S304 time interval matching.

[0085] S303. If so, perform semantic recognition on the voice message, determine the extracted travel intention as the travel intention, and mark it as a normal travel status.

[0086] Among them, the normal travel status refers to the marker when the system confirms that the elderly person's travel intention is clear and does not significantly conflict with their historical behavior, which is used to indicate that no further multimodal verification is needed.

[0087] If the S301 detects a valid voice message and initiates the process immediately, it is the fastest way to confirm travel intentions. The core scenario is when the elderly proactively inform the system of their travel plans via voice (e.g., "I'm going to the community supermarket to buy some groceries and will be back at 10 o'clock"). The system directly identifies the intention through semantic parsing, reducing resource consumption in subsequent processes.

[0088] Specifically, the system converts voice messages into text—processing audio signals using an ASR (Acoustic Speech-to-Text) model. Considering the potential accents and slow speech of elderly users, a model optimized for elderly speech (e.g., incorporating dialect data for training) is used to accurately transcribe the speech into text. Subsequently, the text undergoes semantic analysis: the first step is word segmentation and part-of-speech tagging (e.g., "go to / verb supermarket / noun buy / verb vegetable / noun"); the second step is entity recognition, extracting target location (e.g., "supermarket" or "park"), time information (e.g., "10 o'clock"), and activity type (e.g., "grocery shopping" or "stroll"); the third step is intent integration, combining the extracted entities into a travel intent in the format of "target location + activity" (e.g., "target location: community supermarket; activity: grocery shopping"). Next, the system verifies the completeness of the extracted travel intent—if it contains at least one explicit target location, it is directly identified as the current travel intent; if the target location is vague (e.g., "go for a walk there"), it is supplemented into a complete intent through contextual completion (e.g., combining historically frequented locations). Finally, the system stores the travel intention in the current travel record and marks the status as "normal travel status".

[0089] S304. Based on the time of departure from home, determine the corresponding time interval in the elderly's travel intention data;

[0090] The corresponding time interval refers to the time range that is highly similar to the current departure time in terms of time period characteristics, selected from historical travel intention data. It is used to focus on the most likely historical intentions to match (e.g., the current departure time is Wednesday 09:15, which corresponds to the historical travel intentions of Wednesday 09:00-09:30).

[0091] Specifically, the system analyzes the multidimensional features of the current departure time—including time (e.g., "09:15"), day of the week (e.g., "Wednesday"), whether it is a holiday (e.g., "non-holiday"), and season (e.g., "summer"), which together constitute the time tag. Subsequently, the system performs the same feature analysis on the historical travel intention data obtained from S302, generating a corresponding time tag for each historical record. Next, the system calculates the similarity between the current time tag and the time tags of each historical record—using a weighted scoring method: time similarity (e.g., 80% similarity between 09:15 and 09:00) accounts for 60% of the weight, day of the week similarity (e.g., 100% similarity between Wednesday and Wednesday) accounts for 30% of the weight, and holiday / seasonal similarity accounts for 10% of the weight. Historical records with a total similarity exceeding 70% are included in the candidate set. Finally, the system sorts the travel times in the candidate set in ascending order and takes the earliest and latest times to form the "corresponding time interval" (e.g., if the candidate set times are 09:00, 09:10, and 09:20, then the time interval is "09:00-09:20"), ensuring that the historical intentions within this interval are most similar to the behavioral patterns of the current departure time.

[0092] S305. Determine the primary travel intention from multiple travel intentions within a time interval;

[0093] Among them, multiple travel intentions within a time interval refer to all travel intention records filtered from historical travel intention data within the corresponding time interval determined by S304.

[0094] Specifically, the system extracts all travel intention records within a time interval, removes duplicates, and then counts the frequency of each intention—for example, if there are 5 records in the interval, "go to the community park to exercise" appears 3 times (frequency 60%), and "go to the convenience store to shop" appears 2 times (frequency 40%). The system then sorts these intentions by frequency from highest to lowest, including those with frequencies exceeding a preset threshold (e.g., 30%) in the first travel intention set, and marks them as primary or secondary (the highest frequency is the primary intention, the rest are secondary intentions). If all intention frequencies are below the threshold (e.g., all are 20%), all intentions are included in the set without being marked as primary or secondary, and are only considered as candidate targets to be verified. Next, the system performs a consistency check on the first travel intentions—checking whether there is a geographical correlation between the target locations of each intention (e.g., the park and the supermarket are close, possibly along the same route). If a correlation exists, they are merged into a composite intention (e.g., "go to the park to exercise and then go to the supermarket to shop"); if the target locations are not related (e.g., the park and the hospital are far apart), they remain independent intentions.

[0095] S306. Perform feature recognition on the image of the items carried by the elderly to determine the characteristics of the items carried and clothing of the elderly;

[0096] Among them, the items carried refer to the items that the elderly carry with them when they go out (such as shopping baskets, water cups, and medicine boxes), which are directly related to their travel intentions (such as shopping baskets are often associated with "shopping"); clothing characteristics refer to the attributes of the clothes worn by the elderly (such as sportswear, raincoats, and thick coats), which can help infer the travel scenario (such as raincoats are associated with "going out in the rain" and sportswear is associated with "exercising").

[0097] Specifically, the system acquires images of the items carried and clothing information of the elderly through a camera and preprocesses them. Then, it extracts features from the preprocessed images: for carried items, it uses a target detection model to locate the item's position and extracts features such as shape (e.g., a round shopping basket with handles), color (e.g., a red cloth bag), and texture (e.g., a mesh backpack); for clothing features, it uses a human keypoint detection model to locate parts such as tops, pants, and shoes, and identifies styles (e.g., sports jackets, casual pants), materials (e.g., cotton, waterproof fabric), and additional features (e.g., wearing a hat, having reflective strips). Next, the system classifies and identifies the extracted features—matching item features with a pre-set item database (e.g., "shopping basket," "medicine box," "umbrella") to determine the item's category and function (e.g., "medicine box" corresponds to "needs to take medication"); and matching clothing features with a clothing database (e.g., "sports clothes," "formal wear," "raincoat"), and combining real-time weather (e.g., obtaining "rainy day" via API) to infer the clothing's purpose (e.g., "raincoat + sneakers" associated with "traveling in the rain and needing to walk"). Finally, the system organizes the recognition results into structured data (such as "items carried: shopping basket (confidence 90%); clothing features: blue sportswear, white sneakers (confidence 85%)"), providing visual modal data for inferring the second travel intention of S307.

[0098] S307. Based on the characteristics of the items carried and clothing, determine the elderly person's second travel intention through the intention association feature library. The intention association feature library is pre-set and stores the elderly person's travel intention as well as the corresponding characteristics of the items carried and clothing.

[0099] The second travel intention refers to the potential travel goal of the elderly inferred by matching the characteristics of carried items and clothing through the feature database, which serves as another modal basis for comparison with the first travel intention.

[0100] Specifically, the system converts the features of carried items and clothing identified by S306 into feature vectors—for example, "grocery basket" corresponds to vector [1, 0, 0, ...], and "sports clothes" corresponds to vector [0, 1, 0, ...], which are combined into a joint feature vector. Then, the system accesses an intent-related feature database, where each record contains "travel intent ID + carried item feature vector + clothing feature vector + association strength" (association strength is a score of 0-100, with higher scores indicating a stronger association between the intent and the feature). The system calculates the cosine similarity between the joint feature vector and the feature vectors of each record in the database, filtering out records with similarity exceeding a preset threshold (e.g., 70%) as candidate intents. Next, the system performs a weighted ranking of the candidate intents—using association strength as the primary weight and the completeness of feature matching as a secondary weight (e.g., records matching both carried items and clothing features have a higher weight than records matching only items), selecting the top-ranked candidate intent as the initial second travel intent. If the similarity of candidate intentions is below the threshold, the matching probability of each intention is calculated (e.g., "go to the park" has a matching probability of 45%, "go to the supermarket" has a matching probability of 30%), and the two with the highest probabilities are taken as the second set of travel intentions.

[0101] S308. If the second travel intention matches the first travel intention, then the travel intention is determined and marked as a normal travel status.

[0102] First, the system establishes matching criteria, evaluating consistency from three dimensions: First, the target location dimension, calculating the overlap of geographical coordinates of the target locations in the first and second travel intentions (e.g., "community supermarket" and "supermarket at the east gate of the residential area" are considered the same location); an overlap of ≥70% is considered a location match. Second, the activity type dimension, determining whether the travel activities described in both (e.g., "shopping," "exercise," "medical treatment") belong to the same category (e.g., "grocery shopping" and "shopping" both belong to the shopping category). Third, scenario relevance, combining real-time environment (e.g., weather, date) to determine if the intention is logically consistent (e.g., "going to the park" on a rainy day matches the clothing feature of "bringing an umbrella"). If all three dimensions meet the matching conditions (or the core dimension such as location matching), it is judged as a "match." Subsequently, the system integrates the core elements of the two intentions into the final "travel intention" (e.g., taking the time pattern of the first intention and the scenario details of the second intention), and marks the current travel status as "normal travel status" in the system database, while terminating subsequent verification processes, retaining only the monitoring of real-time route trajectories.

[0103] S309. If the second travel intention does not match the first travel intention, then mark each first travel intention and second travel intention as pending multimodal data verification.

[0104] The system judges mismatches based on preset criteria: if the geographical distance to the target location exceeds a preset distance threshold (e.g., 500 meters), or if the activity types belong to different categories (e.g., "exercise" and "medical treatment") and have no logical connection (e.g., "exercise first, then buy medicine" is not mentioned in any intent), then it is judged as a mismatch. Subsequently, the system stores the first and second travel intentions and their confidence levels (e.g., first intention 60% confidence, second intention 70%) in a temporary verification database, and marks the current travel record as "awaiting multimodal data verification." Next, the system triggers preparatory instructions for subsequent verification processes, including activating the sensor standby state of the smart wearable device (e.g., infrared sensor, microphone).

[0105] S310. After confirming that the elderly person has left home but has not left home beyond the set distance, control the infrared sensor array in the wearable device to turn on.

[0106] In wearable devices, the infrared sensor array refers to a detection module composed of multiple infrared sensors integrated on a smart bracelet / watch, which can sense the body temperature field within a certain range by receiving infrared radiation.

[0107] After confirming that the elderly person has left home (i.e., obtaining the time of departure), the system calculates the straight-line distance between the elderly person and their residence in real time through the positioning module of the wearable device. When the distance is less than or equal to the set distance from home, the system immediately activates. The core scenario is when the elderly person has just left home and may be communicating with neighbors or family members in nearby areas such as the stairwell or the entrance of the community. Activating the infrared sensor at this time can promptly detect whether anyone else is present, providing a trigger condition for obtaining subsequent conversation recordings.

[0108] Specifically, after recording the time the elderly person leaves home, the system receives GPS / BeiDou positioning data from the wearable device in real time and calculates the straight-line distance between the elderly person's current location and the coordinates of their residence using geographic coordinates. If the distance is less than or equal to the set distance from home (e.g., 50 meters), the system sends a "activate infrared sensor array" command to the wearable device, while simultaneously setting the sensor's sampling frequency (e.g., twice per second) and detection angle (e.g., a horizontal 120° range). This is because when the elderly person has not left home beyond the set distance, they are usually in high-frequency communication scenarios near their residence (e.g., in hallways, at the entrance of the community, or downstairs in their building). In these scenarios, the elderly person is more likely to have close conversations with familiar people such as neighbors, family members, and community workers (e.g., "I'm going to the market" or "Could you bring me something?"), and the content of these conversations often includes a clear intention to travel.

[0109] After receiving the command, the wearable device activates the sensor array, enters a low-power monitoring mode (to avoid excessive power consumption), and provides real-time feedback on the sensor status to the system (e.g., "On, working normally"). If the elderly person leaves home and travels beyond the set distance (e.g., goes outside the community), the system will temporarily not activate the infrared sensor.

[0110] S311. When the infrared sensor array detects a second body temperature field within the set infrared range and the duration of the field exceeds the set time threshold, control the wearable device to enable the recording function.

[0111] The second body temperature field refers to the infrared radiation area within the set infrared range that conforms to the human body temperature range, excluding the elderly person themselves (the first body temperature field), and is used to indicate the presence of other people.

[0112] Specifically, an infrared sensor array collects ambient infrared radiation data in real time and filters out non-human heat sources (such as heaters or objects exposed to direct sunlight) using a temperature threshold. The system performs cluster analysis on the filtered signals to distinguish the elderly person's primary body temperature field (which can be calibrated using the wearable device's position, e.g., if the sensor is on the wrist, the primary body temperature field corresponds to the body position) from other body temperature fields. If a signal cluster exists that is separate from the primary body temperature field but has a temperature within the human body's range, it is identified as the "second body temperature field." Subsequently, the system times the duration of this second body temperature field. If the duration is greater than or equal to a set time threshold (e.g., 30 seconds), it is determined that "a continuous communication object exists," and a "start recording function" command is sent to the wearable device. At the same time, recording parameters are set. Upon receiving the command, the wearable device activates the microphone to start recording and caches audio data in real time, while simultaneously sending feedback to the system that "recording has started." If the duration of the second body temperature field is less than the set time threshold, recording is not initiated, and monitoring continues.

[0113] S312. Obtain recordings of conversations between the elderly and others, and perform semantic recognition on the recordings to determine whether they contain travel intention features.

[0114] The system's server receives real-time recordings of conversations uploaded by wearable devices, preprocesses the audio, and then converts the preprocessed audio into text using an ASR model. To address the characteristics of elderly speech (such as slow speech rate and heavy accent), a model trained with elderly speech data is used to improve transcription accuracy (e.g., transcribing "cut medicine" as "take medicine"). Next, semantic analysis is performed on the text; this step is similar to step S204 and will not be described further here.

[0115] If the travel intention feature is included, proceed to step S314;

[0116] If the travel intention feature is not included, proceed to step S313.

[0117] S313. Control smart wearable devices for continuous monitoring;

[0118] S314. Determine the travel intention based on the travel intention characteristics, and change the status information from the status of waiting for multimodal data verification to the status of normal travel.

[0119] The core scenario of this step is to confirm the travel intention through the conversation between the elderly person and others (e.g., if the first intention was "to go to the park" and the second intention was "to go to the hospital", and the conversation mentions "to go to the hospital for a check-up", then the intention is confirmed as "to go to the hospital"), thus completing the transition from pending verification to a clear state.

[0120] The system extracts core elements from travel intention features, including destination location, travel purpose, and possible time information, and verifies the accuracy of these elements. Then, it associates the extracted elements with the first and second travel intentions to be verified. If it matches one of the intentions (e.g., the feature points to "hospital," the second intention is "go to pharmacy," but "hospital" may include the "pharmacy" scenario, or it matches directly), the intention extracted from the feature is taken as the final result. If it does not match either of the two intentions (e.g., the first intention is "park," the second intention is "supermarket," and the feature points to "community service center"), the intention extracted from the feature is taken as the final result.

[0121] Next, the system generates a structured "travel intent" record, including the coordinates of the destination location, the expected activity type, and related evidence, and stores it in the current travel profile. Finally, the system updates the status flag, changing "awaiting multimodal data verification status" to "normal travel status," and simultaneously sends a command to the wearable device to disable the infrared sensor and recording function.

[0122] S315. When the infrared sensor array detects the disappearance of the second body temperature field, the infrared sensor is turned off and recording is stopped.

[0123] The infrared sensor array continuously monitors the signal strength and position changes of the second body temperature field. When the signal strength continuously decreases or disappears completely (no signal matching human body temperature is detected for 5 consecutive seconds), it is determined that "the second body temperature field has disappeared." The system simultaneously uses the microphone input from the wearable device to assist in confirming the end of the conversation, avoiding false judgments due to temporary sensor malfunctions. Subsequently, the system sends "turn off infrared sensor" and "stop recording" commands to the wearable device. Upon receiving the commands, the device sequentially turns off the microphone and then shuts down the infrared sensor array.

[0124] In this embodiment, a multimodal data fusion-based hierarchical verification mechanism is constructed. From prioritizing voice messages to cross-referencing historical data and the characteristics of carried items, and then to environmental interaction verification triggered by infrared sensors, a complete closed loop for travel intention inference is formed. Therefore, the travel intention of the elderly can be accurately inferred through cross-verification of multi-dimensional data such as voice messages, historical behavior, carried items, and environmental interactions. The verification intensity is dynamically adjusted (e.g., clear messages can be directly confirmed, while conflicting data triggers deep verification). This effectively solves the problems of high misjudgment rate, delayed verification, and poor adaptability to sudden intentions caused by relying on single data (such as location or fixed schedule) in the traditional monitoring of elderly people living alone. This achieves intelligent and refined identification of the travel intention of elderly people living alone, reducing unnecessary intervention while providing accurate intention benchmarks for subsequent abnormal behavior warnings (such as getting lost), thus improving the safety and autonomy of the elderly's travel.

[0125] The anomaly detection system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference needed]. Figure 4 This is a schematic diagram of the physical device structure of an anomaly recognition system in this application embodiment.

[0126] It should be noted that, Figure 4 The structure of the anomaly identification system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0127] like Figure 4 As shown, the anomaly detection system includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 402 or a program loaded from storage section 408 into Random Access Memory (RAM) 403, such as performing the methods described in the above embodiments. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0128] The following components are connected to I / O interface 405: input section 406 including audio input devices, push-button switches, etc.; output section 407 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 408 including a hard disk, etc.; and communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0129] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the various functions defined in the present invention.

[0130] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0132] Specifically, the anomaly recognition system in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the method for recognizing abnormal behavior of elderly people living alone based on multimodal data fusion provided in the above embodiment.

[0133] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the anomaly recognition system described in the above embodiments; or it may exist independently and not assembled into the anomaly recognition system. The storage medium carries one or more computer programs, which, when executed by a processor of the anomaly recognition system, cause the anomaly recognition system to implement the method for identifying abnormal behavior of elderly people living alone based on multimodal data fusion provided in the above embodiments.

[0134] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0135] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for identifying abnormal behavior in elderly people living alone based on multimodal data fusion, applied to an anomaly identification system, characterized in that: The method includes: Real-time acquisition of smart lock data, including home arrival time, departure time, and images of items carried by the elderly person captured by the smart lock camera; The elderly’s voice interaction data is obtained through smart wearable devices. The voice interaction data includes the elderly’s voice messages with the smart wearable devices and recordings of conversations with others. If the system detects that an elderly person has gone out, the system will activate the location function on the smart wearable device and plot the acquired location data as a real-time route trajectory. The elderly person's travel intention is determined by combining the smart lock data and the voice interaction data, and the travel intention includes at least one target location; By combining the real-time route trajectory with the travel intention, it can be determined whether the elderly person is in a normal travel state or is lost and wandering. If the elderly person is lost and wandering, then the lost response will be triggered; If the lost response is successful, a lost alert will be sent to the emergency contact; the lost response means that the lost response result confirms the person is lost or there is no response. The step of determining the elderly person's travel intention by combining the smart lock data and the voice interaction data includes: determining whether the elderly person left a voice message within a set time interval before and after the departure time; if so, performing semantic recognition on the voice message, identifying the extracted travel intention as the travel intention, and marking it as a normal travel status; if not, obtaining the elderly person's travel intention data for each time period within a set time period in the past; determining the corresponding time interval in the elderly person's travel intention data based on the departure time; and identifying multiple travel intentions in the time interval as a first travel intention. After the step of determining multiple travel intentions within the time interval as the first travel intention, the method further includes: performing feature recognition on the image of the elderly person's carried items to determine the elderly person's carried items and clothing features; based on the carried items and clothing features, determining the elderly person's second travel intention through an intention association feature library, which is pre-set and stores the elderly person's travel intentions and corresponding carried items and clothing features; if the second travel intention matches the first travel intention, then the travel intention is determined and marked as a normal travel state; if the second travel intention does not match the first travel intention, then each of the first and second travel intentions is marked as pending multimodal data. According to the verification status; after confirming that the elderly person has left home, and has not left home beyond the set distance, the infrared sensor array in the wearable device is activated; when the infrared sensor array detects a second body temperature field within the set infrared range, and the field exists for a period exceeding a set time threshold, the wearable device is activated to record audio; the conversation recording between the elderly person and others is acquired, and semantic recognition is performed on the conversation recording to determine whether it contains travel intention features; if so, the travel intention is determined based on the travel intention features, and the status information is changed from pending multimodal data verification to normal travel status; when the infrared sensor array detects that the second body temperature field disappears, the infrared sensor is turned off, and recording stops.

2. The method according to claim 1, characterized in that, After the step of plotting the acquired location data into a real-time route trajectory, the following steps are also included: The system determines whether the elderly person has returned home based on the smart lock data at the latest set time of the day. If it is determined that the elderly person has not returned home, then it is determined that the elderly person is lost, and a lost person alert is sent to the emergency contact.

3. The method according to claim 1, characterized in that, The process of determining whether an elderly person is in a normal travel state or lost and wandering by combining the real-time route trajectory with the travel intention includes: Obtain historical travel data and determine all routes to the target location in the historical travel data. Determine route segments that appear more frequently than a preset frequency threshold as the necessary path for the current trip. Extend the necessary path outwards with the radius of the farthest point of the historical trip to determine the reachable area of ​​the current trip; If the real-time route trajectory is located on the necessary path, or deviates from the necessary path but is still within the reachable area, and the direction of movement continues towards the target location, it is determined to be a normal travel status; If the real-time route trajectory continuously deviates from the required path by more than a preset distance threshold and fails to enter the reachable area, or if repeated back-and-forth behavior occurs outside the reachable area, then the elderly person is determined to be lost and wandering.

4. The method according to claim 1, characterized in that, If the elderly person is lost and wandering, the following response will be triggered: Control wearable devices to send lost response alerts to elderly people; If no response is received from the elderly person, a second prompt for a lost response will be sent to the elderly person after a set time, and the wearable device will be controlled to flash as a warning. After receiving the elderly person's lost response information, semantic recognition is performed on the lost response information to obtain the lost response result; Determine whether the elderly person passed the lost response based on the lost response results; If no lost response information is received from the elderly person, the system will automatically confirm that the elderly person has passed the lost response.

5. The method according to claim 1, characterized in that, If the elderly person is lost and wandering, the steps to trigger the lost response include: The number of times the lost response was triggered and the feedback results of the elderly were counted in the most recent n times. The feedback results of the elderly included at least the following: confirmation that they were not lost, no response but the subsequent trajectory returned to normal, and false alarm. If the number of triggers exceeds the preset tolerance threshold, and the total proportion of confirmed non-lost and falsely triggered warnings exceeds the preset proportion threshold, then the threshold for determining subsequent lost and wandering will be automatically raised. Specific ways to raise the threshold for judgment include expanding the radius of the reachable area based on the total proportion.

6. An anomaly detection system, characterized in that, The anomaly detection system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the anomaly detection system to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the anomaly detection system, it causes the anomaly detection system to perform the method as described in any one of claims 1-5.

8. A computer program product, characterized in that, When the computer program product is run on the anomaly detection system, the anomaly detection system performs the method as described in any one of claims 1-5.

Images