An intelligent assistance system for amnesiac elderly people based on AR glasses

Abstract

This invention discloses an intelligent assistance system for elderly people with amnesia based on AR glasses. Using the AR glasses, the system uses a voice-activated wake-up module to activate the glasses' camera to capture images for face detection, face recognition, and object detection. This system enables the identification of acquaintances and the location and traceability of key items, with the results displayed visually on the AR screen. This system can replace the basic functions of caregivers, increase the independence and self-care abilities of Alzheimer's patients, and reduce the cost of patient care.

Description

Technical Field

[0001] The present invention relates to the field of intelligent wearable devices and human-computer interaction technology, and in particular to an intelligent assistance system for amnesiac elderly people based on AR glasses. Background Art

[0002] Alzheimer's disease (AD) is the most common form of dementia and the primary cause of dementia. It is also one of the most common chronic diseases in the elderly, affecting 30% of the world's elderly population. It is projected that by 2050, the number of people with AD worldwide will reach 130 million. Memory loss is the most common symptom in the early and middle stages of AD. Patients experience a gradual loss of cognitive abilities, including but not limited to facial memory, object memory, and directional memory. This can lead to Alzheimer's patients frequently forgetting familiar names and faces, the location of frequently used items, and even forgetting what they were doing. This phenomenon not only impacts the patients themselves but also places a heavy burden on their families. To address this memory loss, many are working to design intelligent systems to reduce expensive labor costs and assist Alzheimer's patients in their daily lives. These systems include intelligent living environment-based assistance systems and wearable devices.

[0003] Assistance systems based on intelligent living environments typically consist of a series of sensors, cameras, terminals, or cloud-based processors. These sensors or cameras are installed in the daily lives of elderly individuals with amnesia, capturing relevant data and transmitting it to terminals for processing. Using technologies like artificial intelligence, these systems implement features like behavioral assistance prompts and abnormal behavior detection. For example, researchers have deployed a simple sensor network in the homes of elderly individuals with amnesia, using recursive neural networks to predict future sensor values ​​and compare them with actual values, thereby detecting abnormal behavior. However, this assistance system requires the pre-deployment of a large number of sensors, resulting in high labor and equipment costs, and the system can fail if the user leaves the coverage area.

[0004] In recent years, research on intelligent assistance systems for elderly individuals with amnesia has increasingly focused on wearable devices, due to their compact size, portability, and adaptability to environmental and location constraints. These systems typically integrate data-collecting sensors such as accelerometers, gyroscopes, and cameras into wearable devices. After processing the data, they assess the elderly's behavior and provide assistance and prompts to compensate for cognitive deficiencies. However, most wearable assistance systems fail to simultaneously meet the following three requirements: 1. ease of use for elderly individuals with amnesia; 2. portable, comfortable, and non-interfering wearable devices; and 3. cognitive prompts covering facial, object, and directional memory. Furthermore, voice prompts, commonly used in these assistance systems, can often embarrass or frighten the elderly.

[0005] Therefore, although existing technologies have made certain progress in elderly assistance prompt systems, they still face challenges such as being not easy to use, uncomfortable to wear, and prompt methods that are not friendly to elderly people with amnesia, resulting in no large-scale promotion. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies and provide an intelligent assistance system for elderly people with amnesia based on AR glasses. This invention can replace some of the functions of caregivers, helping Alzheimer's patients increase their independence and improve their self-care ability, while reducing the cost of care for patients at home or in nursing homes.

[0007] The present invention is achieved through the following technical solutions:

[0008] An intelligent assistance system for amnesiac elderly people based on AR glasses includes AR glasses and a face detection module, a face recognition module, an object detection module, a voice wake-up module, and a voice recognition module integrated in the AR glasses. The AR glasses have an AR display part, an internal storage module, and a processor. The specific contents are as follows:

[0009] Step 1: Select a dataset for model training and unify the saving format of the model file;

[0010] Step 2: Use the Android development platform to build a face database containing familiar faces in the internal storage module of the AR glasses, and implement dynamic addition, deletion, and modification functions for the face database;

[0011] Step 3: Deploy the trained model to the AR glasses system. Use the camera to capture each frame, scale it to fit the model input size, and use it as input for the face recognition module for inference. The obtained face coordinates are displayed on the AR page in the form of a face frame. The original image is then captured and scaled according to the position of the face frame. This is used as input for the face recognition module and compared with the identities in the pre-established face library to obtain the person's name displayed on the AR page.

[0012] Step 4: In the Android-developed interface, the user selects the indoor map provided by the user through the system album and marks the location of fixed objects in the room by clicking on the area on the map.

[0013] Step 5: Use the existing indoor map provided by the user and the locations of fixed objects in the marked room to achieve coarse-grained positioning of key objects. When a key object appears during the object detection process, the object detection module determines its coarse-grained location and saves it.

[0014] Step 6: Implement the voice wake-up module through the plug-in provided by the Picovoice solution to wake up the system. The voice recognition module automatically captures audio for voice recognition and activates the corresponding function after recognizing the specified keyword.

[0015] Select the dataset described in step 1 for model training and unify the format for saving the model files. Specifically, select the public widerface and emore datasets for training the face detection and face recognition models respectively. Training is performed on the Ubuntu 18.04 operating system and the PyTorch platform according to the default hyperparameters of the open source repository. The face detection module is trained for 200 epochs with a batch_size of 24 and a learning_rate of 0.02. The face recognition model is trained for 100 epochs with a batch_size of 64 and a learning_rate of 0.003. Finally, the model files are saved in the unified format of onnx.

[0016] The overall system framework is built on the AR glasses using Flutter and Android hybrid development technology.

[0017] The AR glasses are integrated with a camera, a microphone, and are equipped with an independent operating system and internal storage.

[0018] The speech recognition module described in step 4 automatically captures audio for speech recognition and activates the corresponding function after recognizing the specified keyword, as follows:

[0019] The speech recognition module uses the glasses camera to capture real-time images as model input. The images are checked at regular intervals. If a key object is detected in the image, the module observes the nearest adjacent images to see if there is an annotated fixed object. If so, the coarse-grained location of the key object is saved near the fixed object.

[0020] When the user's voice is recognized to contain the set search keyword, the most recently saved key item location is displayed on the page as a flashing red dot on the indoor map, instructing the user to search within a certain area;

[0021] When the user's voice is recognized to contain the set face recognition keywords, face detection and face recognition are started on the real-time image captured by the camera. The face detection module is responsible for drawing a frame around the face in the image and providing output to the face recognition model. The face recognition module returns the extracted facial feature values ​​and compares them with the faces in the face library, returning a person's name. The returned results are displayed on the AR screen using the Android canvas tool for visualization.

[0022] The advantages of this invention are that it uses AR glasses that can be developed, and uses a voice wake-up module to activate the glasses' camera to capture images for face detection, face recognition, object detection, and other processing. It can also identify acquaintances and locate and trace key items, and the results are displayed visually on the AR screen. This invention can replace the basic functions of caregivers, increase the independence and self-care ability of Alzheimer's patients, and reduce the cost of patient care. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the AR glasses device;

[0024] Figure 2 Flowchart for system operation;

[0025] Figure 3 This is a schematic diagram of acquaintance identification. DETAILED DESCRIPTION

[0026] like Figure 1 As shown in FIG, an intelligent assistance system for amnesiac elderly people based on AR glasses includes AR glasses and a face detection module, a face recognition module, a target detection module, a voice wake-up module and a voice recognition module integrated in the AR glasses. The AR glasses have an AR display part, internal storage and a processor, as shown in FIG. Figure 2 The specific contents are as follows:

[0027] Step 1: Select commercially available AR glasses. The device should have simple AR display capabilities, internal storage and a processor, as well as integrated components such as a camera, microphone, speaker, accelerometer, and gyroscope. Use Flutter and Android hybrid development techniques to build the overall system framework for the AR glasses, streamlining the logical relationships between various modules. Considering the limited storage and computing power of the glasses, research and select models that are as lightweight as possible for face detection, recognition, and object detection. Select or create a suitable dataset for model training to ensure high accuracy, and standardize the format for saving model files.

[0028] The public widerface and emore datasets were used for training face detection and face recognition models, respectively. Training was performed on the PyTorch platform on Ubuntu 18.04 using the default hyperparameters from the open source repository. The face detection module was trained for 200 epochs with a batch size of 24 and a learning rate of 0.02. The face recognition model was trained for 100 epochs with a batch size of 64 and a learning rate of 0.003. The model files were saved in the onnx format.

[0029] Step 2: Use the Android development platform to build a face database containing familiar faces in the internal storage module of the AR glasses, and implement dynamic addition, deletion, and modification functions for the face database;

[0030] Step 3: Deploy the trained model to the AR glasses system. Use the camera to capture each frame, scale it to fit the model input size, and use it as input for the face recognition module for inference. The obtained face coordinates are displayed on the AR page in the form of a face frame. The original image is then captured and scaled according to the position of the face frame. This is used as input for the face recognition module and compared with the identities in the pre-established face library to obtain the person's name displayed on the AR page.

[0031] Step 4: In the Android-developed interface, the user selects the indoor map provided by the user through the system album and marks the location of fixed objects in the room by clicking on the area on the map.

[0032] Step 5: Use the existing indoor map provided by the user and the locations of fixed objects in the marked room to achieve coarse-grained positioning of key objects. When a key object appears during the object detection process, the object detection module determines its coarse-grained location and saves it.

[0033] Step 6: Implement the voice wake-up module through the plug-in provided by the Picovoice solution to wake up the system. The voice recognition module automatically captures audio for voice recognition and activates the corresponding function after recognizing the specified keyword.

[0034] The following further explains how the intelligent assistance system can identify acquaintances and trace the origin of key items. Figure 1 It is the device that the system runs on. Figure 2 It is the overall process of system operation.

[0035] (1) First, the user clicks the system app icon on the AR page of the glasses to start the auxiliary system. The system then initializes the face library, target detection, and voice monitoring threads.

[0036] (2) During the voice monitoring process, the target detection thread, also known as the key item traceability thread, begins working. This thread uses the glasses camera to capture real-time images as model input. The thread detects the images at regular intervals. If the image contains the set key item, it observes the nearest adjacent images to see if there are any marked fixed items. If so, the coarse-grained location of the key item is saved near the fixed item. The above process runs continuously to continuously track the placement of key items.

[0037] (3) When the voice monitoring recognizes that the user's voice contains keywords such as "where" (keywords can be set freely), the system will display the most recently saved location of the key item on the page in the form of a flashing red dot on the indoor map, instructing the user to search within a certain area.

[0038] (4) When the voice monitoring recognizes that the user's voice contains keywords such as "Who are you" (keywords can be set freely), the system will start face detection and face recognition on the real-time image captured by the camera. The face detection module is responsible for drawing a frame around the face in the image and providing output to the face recognition model. The face recognition module returns the extracted face feature value and compares it with the faces in the face database, and returns a name. The returned results are displayed on the AR screen using the Android canvas tool for visualization. The display page is as follows: Figure 3 As shown, this is to remind the old man of the identity of the familiar person in front of him, such as Figure 3 shown.

[0039] (5) When these threads exit, the system will automatically restart monitoring and prepare to repeat the above process.

[0040] The present invention is an easy-to-use wearable intelligent assistance system based on AR glasses, which has the following advantages:

[0041] (1) More comprehensive intelligent prompts

[0042] The system integrates two major functions: acquaintance identity recognition and key item location and traceability. The intelligent prompts given to the elderly cover most scenarios of memory loss at home for amnesiac elderly people.

[0043] (2) Easy for the elderly to use

[0044] Considering that elderly people with amnesia may not be able to use AR glasses or find it difficult to understand the system, the present invention adds a voice wake-up module to the system to determine whether it is necessary to start a specified function by capturing what the elderly say.

[0045] (3) Visual prompts

[0046] Taking into account the embarrassment and suddenness caused by sudden voice prompts in daily life, the present invention uses the method of displaying prompt information on the AR screen instead of voice prompts. This is also a major reason why we chose AR glasses for system development.

Claims

1. An intelligent assistance system for elderly people with amnesia based on AR glasses, characterized by: The AR glasses include a face detection module, a face recognition module, a target detection module, a voice wake-up module, and a voice recognition module integrated in the AR glasses. The AR glasses have an AR display part, an internal storage module, and a processor. The specific contents are as follows: Step 1: Select a dataset for model training and unify the saving format of the model file; Step 2: Use the Android development platform to build a face database containing familiar faces in the internal storage module of the AR glasses, and implement dynamic addition, deletion, and modification functions for the face database; Step 3: Deploy the trained model to the AR glasses system. Use the camera to capture each frame, scale it to fit the model input size, and use it as input for the face recognition module for inference. The obtained face coordinates are displayed on the AR page in the form of a face frame. The original image is then captured and scaled according to the position of the face frame. This is used as input for the face recognition module and compared with the identities in the pre-established face library to obtain the person's name displayed on the AR page. Step 4: In the Android-developed interface, the user selects the indoor map provided by the user through the system album and marks the location of fixed objects in the room by clicking on the area on the map. Step 5: Use the existing indoor map provided by the user and the locations of fixed objects in the marked room to achieve coarse-grained positioning of key objects. When a key object appears during the object detection process, the object detection module determines its coarse-grained location and saves it. Step 6: Implement the voice wake-up module through the plug-in provided by the Picovoice solution to wake up the system. The voice recognition module automatically captures audio for voice recognition and activates the corresponding function after recognizing the specified keyword.

2. The intelligent assistance system for elderly people with amnesia based on AR glasses according to claim 1, characterized in that: Select the dataset described in step 1 for model training and unify the format for saving the model files. Specifically, select the public widerface and emore datasets for training the face detection and face recognition models respectively. Training is performed on the Ubuntu 18.04 operating system and the PyTorch platform according to the default hyperparameters of the open source repository. The face detection module is trained for 200 epochs with a batch_size of 24 and a learning_rate of 0.

02. The face recognition model is trained for 100 epochs with a batch_size of 64 and a learning_rate of 0.

003. Finally, the model files are saved in the unified format of onnx.

3. The intelligent assistance system for elderly people with amnesia based on AR glasses according to claim 1 is characterized by: The overall system framework is built on the AR glasses using Flutter and Android hybrid development technology.

4. The intelligent assistance system for elderly people with amnesia based on AR glasses according to claim 1, characterized in that: The AR glasses are integrated with a camera, a microphone, and are equipped with an independent operating system and internal storage.

5. The intelligent assistance system for elderly people with amnesia based on AR glasses according to claim 1, characterized in that: The speech recognition module described in step 4 automatically captures audio for speech recognition and activates the corresponding function after recognizing the specified keyword, as follows: The speech recognition module uses the glasses camera to capture real-time images as model input. The images are checked at regular intervals. If a key object is detected in the image, the module observes the nearest adjacent images to see if there is an annotated fixed object. If so, the coarse-grained location of the key object is saved near the fixed object. When the user's voice is recognized to contain the set search keyword, the most recently saved key item location is displayed on the page as a flashing red dot on the indoor map, instructing the user to search within a certain area; When the user's voice is recognized to contain the set face recognition keywords, face detection and face recognition are started on the real-time image captured by the camera. The face detection module is responsible for drawing a frame around the face in the image and providing output to the face recognition model. The face recognition module returns the extracted facial feature values ​​and compares them with the faces in the face library, returning a person's name. The returned results are displayed on the AR screen using the Android canvas tool for visualization.

Images