Intelligent auxiliary system for memory-losing old people based on AR (Augmented Reality) glasses

Abstract

The invention discloses an intelligent auxiliary system for memory-losing old people based on AR glasses, which uses developable AR glasses, starts a glasses camera through a voice wake-up module to collect pictures to carry out face detection, face recognition, target detection and other processing, and realizes the functions of acquaintance identity recognition and key article positioning and tracing. And the result is displayed on the AR screen in a visual mode. Basic functions of nursing personnel can be replaced, independence is increased for a patient suffering from the Alzheimer's disease, the self-care ability is improved, and the nursing cost of the patient is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent wearable devices and human-computer interaction technology, and in particular to an intelligent assistance system for amnesiac elderly people based on AR glasses. Background Art

[0002] Alzheimer's disease (AD) is the most common type of dementia in the elderly, the main cause of dementia, and one of the most common chronic diseases in the elderly, accounting for 30% of the world's total elderly population. It is predicted that by 2050, the number of people suffering from Alzheimer's disease worldwide will reach 130 million. Memory loss is the most common symptom of Alzheimer's patients in the early and middle stages. The patient's cognitive ability will be gradually destroyed, including but not limited to their facial memory, object memory, and direction memory. This causes Alzheimer's elderly people to often forget their familiar names and faces, forget the placement of certain commonly used items, and even forget what they are doing. This phenomenon not only affects the patients themselves, but also brings a heavy burden to their families. In response to this amnesia phenomenon, many people are trying to design intelligent systems to reduce expensive labor costs and assist Alzheimer's elderly people in their daily lives, including auxiliary systems based on the intelligence of living environments and auxiliary systems based on the intelligence of wearable items.

[0003] The auxiliary system based on intelligent living environment usually consists of a series of sensors, cameras, terminals or cloud processors. The sensors or cameras are installed in the daily living environment of the elderly with amnesia, capturing the relevant data of the elderly and transmitting it to the terminal for processing, and realizing the functions of behavior assistance prompts and abnormal behavior detection through technologies such as artificial intelligence. For example, some researchers have equipped the homes of elderly people with amnesia with a simple sensor network, and compared the future values ​​of the sensors with the actual values ​​through recursive neural networks, so as to realize the function of abnormal behavior detection for the elderly. However, this auxiliary system requires the deployment of a large number of sensor devices in the environment in advance, and the labor and equipment costs are very high, and there is a defect that the system fails when the user leaves the coverage area.

[0004] Based on the intelligent assistance system of wearable items, due to its small size and portability and not being restricted by the environment and location, in recent years, more and more research on intelligent assistance systems for elderly people with amnesia has begun to turn to wearable devices. These systems usually integrate accelerometers, gyroscopes, cameras and other devices that collect data into wearable devices, and after processing the data, they judge the behavior of the elderly and provide auxiliary prompts to make up for the defects of their cognitive functions. However, most auxiliary systems based on wearable devices cannot meet the following three requirements at the same time: 1. It is easy to use for elderly people with amnesia 2. The wearable device is portable and comfortable and does not interfere with daily life 3. The cognitive prompts for elderly people with amnesia cover facial, object and direction memory at the same time. In addition, voice prompts, which are commonly used in such auxiliary systems, often make the elderly feel embarrassed or frightened.

[0005] Therefore, although the existing technology has made certain progress in the elderly auxiliary prompt system, it still faces challenges such as being not easy to use, uncomfortable to wear, and the prompt method is not friendly to the elderly with amnesia, resulting in no large-scale promotion. Summary of the invention

[0006] The purpose of the present invention is to make up for the defects of the existing technology and provide an intelligent assistance system for amnesiac elderly people based on AR glasses. The present invention can replace some functions of caregivers, help patients with Alzheimer's disease increase their independence, improve their self-care ability, and reduce the cost of care for patients by families or nursing homes.

[0007] The present invention is achieved through the following technical solutions: An intelligent assistance system for amnesiac elderly people based on AR glasses, including AR glasses and a face detection module, a face recognition module, a target detection module, a voice wake-up module and a voice recognition module integrated in the AR glasses. The AR glasses have an AR display part, an internal storage module and a processor. The specific contents are as follows: Step 1: Select a data set for model training and unify the saving format of the model file; Step 2: Use the Android development platform to build a face database containing acquaintances in the internal storage module of the AR glasses, and implement the dynamic addition, deletion and modification functions of the face database; Step 3: Deploy the trained model to the AR glasses system, call the camera to capture each frame of the image, scale it to a size suitable for the model input, and use it as the input of the face recognition module for inference. Display the obtained face coordinates on the AR page in the form of a face frame. Then, collect and scale the original image according to the position of the face frame as the input of the face recognition module, compare it with the identity in the pre-established face library, and then obtain the person's name displayed on the AR page; Step 4: In the interface developed by Android, the user selects the indoor map provided by the user through the system album, and marks the location of fixed objects in the room by clicking on the area on the map. Step 5: Use the existing indoor map provided by the user and the locations of the fixed objects in the marked room to achieve coarse-grained positioning of key objects. When a key object appears during the target detection process, the target detection module determines its coarse-grained location and saves it; Step 6: Implement the voice wake-up module through the plug-in provided by the Picovoice solution to wake up the system. The voice recognition module automatically captures the audio for voice recognition and enables the corresponding function after recognizing the specified keyword.

[0008] The data set selected in step 1 is used for model training, and the saving format of the model file is unified, as follows: the public widerface and emore data sets are selected for face detection and face recognition model training respectively, and the training is performed on the Ubuntu 18.04 operating system and the Pytorch platform according to the default hyperparameters of the open source warehouse. The face detection module is trained for 200 epochs with a batch_size of 24 and a learning_rate of 0.02, and the face recognition model is trained for 100 epochs with a batch_size of 64 and a learning_rate of 0.003. Finally, the model file is saved in the unified format of onnx.

[0009] The overall system framework is built on the AR glasses using flutter and Android hybrid development technology.

[0010] The AR glasses are integrated with a camera, a microphone, and are equipped with an independent operating system and internal storage.

[0011] The speech recognition module described in step 4 automatically captures audio for speech recognition, and activates the corresponding function after recognizing the specified keyword, as follows: The speech recognition module uses the eyeglass camera to capture real-time images as model input, and detects the images at regular intervals. If a set key object is detected in the image, it observes whether there is an annotated fixed object in the nearest adjacent images. If so, the coarse-grained position of the key object is saved near the fixed object. When the user's voice is recognized to contain the set search keyword, the most recently saved key item location is displayed on the page as a flashing red dot on the indoor map, instructing the user to search within a certain area; When the user's voice is recognized to contain the set face recognition keywords, face detection and face recognition are started on the real-time image captured by the camera. The face detection module is responsible for drawing a frame around the face in the image and providing output to the face recognition model. The face recognition module returns the extracted face feature values ​​and compares them with the faces in the face library, and returns a name. The returned results are visualized on the AR screen using the Android canvas tool.

[0012] The advantages of the present invention are: the present invention uses the developable AR glasses, and uses the voice wake-up module to open the glasses camera to collect pictures for face detection, face recognition, target detection and other processing, realizing the recognition of acquaintances, the positioning and tracing of key items, and displaying the results on the AR screen in a visual way. The present invention can replace the basic functions of nursing staff, increase the independence of patients with Alzheimer's disease, improve their self-care ability, and reduce the cost of nursing for patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the AR glasses device; Figure 2 Run a flowchart for the system; Figure 3 This is a schematic diagram of acquaintance identification. DETAILED DESCRIPTION

[0014] like Figure 1 As shown, an intelligent assistance system for amnesiac elderly people based on AR glasses includes AR glasses and a face detection module, a face recognition module, a target detection module, a voice wake-up module and a voice recognition module integrated in the AR glasses. The AR glasses have an AR display part, an internal storage and a processor, such as Figure 2 The specific contents are as follows: Step 1: Select AR glasses that can be developed on the market. The required equipment needs to have simple AR display functions, internal storage and processors, integrated cameras, microphones, speakers, accelerometers, gyroscopes and other components. Use flutter and Android hybrid development technology to build the overall system framework on AR glasses and sort out the logical relationship between various modules. Considering the limited storage and computing power of glasses devices, research and select models that are as lightweight as possible for face detection, recognition, and target detection. Select or create a suitable data set for model training to ensure a high accuracy rate and unify the saving format of model files.

[0015] The public widerface and emore datasets were selected for training face detection and face recognition models respectively. The training was performed on the Ubuntu 18.04 operating system and the Pytorch platform according to the default hyperparameters of the open source repository. The face detection module was trained for 200 epochs with a batch_size of 24 and a learning_rate of 0.02, and the face recognition model was trained for 100 epochs with a batch_size of 64 and a learning_rate of 0.003. Finally, the unified model file was saved in the onnx format.

[0016] Step 2: Use the Android development platform to build a face database containing acquaintances in the internal storage module of the AR glasses, and implement the dynamic addition, deletion and modification functions of the face database; Step 3: Deploy the trained model to the AR glasses system, call the camera to capture each frame of the image, scale it to a size suitable for the model input, and use it as the input of the face recognition module for inference. Display the obtained face coordinates on the AR page in the form of a face frame. Then, collect and scale the original image according to the position of the face frame as the input of the face recognition module, compare it with the identity in the pre-established face library, and then obtain the person's name displayed on the AR page; Step 4: In the interface developed by Android, the user selects the indoor map provided by the user through the system album, and marks the location of fixed objects in the room by clicking on the area on the map. Step 5: Use the existing indoor map provided by the user and the locations of the fixed objects in the marked room to achieve coarse-grained positioning of key objects. When a key object appears during the target detection process, the target detection module determines its coarse-grained location and saves it; Step 6: Implement the voice wake-up module through the plug-in provided by the Picovoice solution to wake up the system. The voice recognition module automatically captures the audio for voice recognition and enables the corresponding function after recognizing the specified keyword.

[0017] The following further describes the method of using the intelligent assistance system to identify acquaintances and trace the source of key items. Figure 1 It is the device that the system runs on. Figure 2 It is the overall process of system operation.

[0018] (1) First, the user clicks the system app icon on the AR page of the glasses to start the auxiliary system. The system then initializes the face library, target detection, and voice monitoring threads.

[0019] (2) During the voice monitoring process, the target detection thread, that is, the key item tracing thread, starts working. This thread uses the glasses camera to capture real-time images as model input and detects the images at regular intervals. If the set key items are detected in the images, the thread observes whether there are fixed items marked in the nearest adjacent images. If so, the coarse-grained position of the key items is saved near the fixed items. The above work runs continuously to continuously track the placement of key items.

[0020] (3) When the voice monitoring recognizes that the user's voice contains keywords such as "where" (the keywords can be set freely), the system will display the most recently saved location of the key item on the page in the form of a flashing red dot on the indoor map, instructing the user to search within a certain area.

[0021] (4) When the voice monitoring recognizes that the user's voice contains keywords such as "Who are you" (keywords can be set freely), the system will start face detection and face recognition on the real-time image captured by the camera. The face detection module is responsible for drawing a frame around the face in the image and providing output to the face recognition model. The face recognition module returns the extracted face feature value and compares it with the faces in the face database, and returns a name. The returned results are visualized on the AR screen using the Android canvas tool. The display page is as follows: Figure 3 As shown, this is used to remind the elderly of the identity of the familiar person in front of them, such as Figure 3 shown.

[0022] (5) When these threads exit, the system will automatically restart listening and prepare to repeat the above process.

[0023] The present invention is an easy-to-use wearable intelligent assistance system based on AR glasses, which has the following advantages: (1) More comprehensive intelligent prompts The system integrates two major functions: acquaintance identity recognition and key item location and traceability. The intelligent prompts given to the elderly cover most scenarios of memory loss for amnesiac elderly people at home.

[0024] (2) Easy for the elderly to use Considering that the elderly with amnesia may not use AR glasses or find it difficult to understand the system, the present invention adds a voice wake-up module to the system to determine whether it is necessary to start a specified function by capturing what the elderly say.

[0025] (3) Visual prompts Taking into account the embarrassment and suddenness caused by sudden voice prompts in daily life, the present invention uses a method of displaying prompt information on the AR screen instead of voice prompts, which is also a major reason why we choose AR glasses for system development.

Claims

1. An intelligent assistance system for amnesiac elderly people based on AR glasses, characterized by: The invention comprises AR glasses and a face detection module, a face recognition module, a target detection module, a voice wake-up module and a voice recognition module integrated in the AR glasses. The AR glasses have an AR display part, an internal storage module and a processor. The specific contents are as follows: Step 1: Select a data set for model training and unify the saving format of the model file; Step 2: Use the Android development platform to build a face database containing acquaintances in the internal storage module of the AR glasses, and implement the dynamic addition, deletion and modification functions of the face database; Step 3: Deploy the trained model to the AR glasses system, call the camera to capture each frame of the image, scale it to a size suitable for the model input, and use it as the input of the face recognition module for inference. Display the obtained face coordinates on the AR page in the form of a face frame. Then, collect and scale the original image according to the position of the face frame as the input of the face recognition module, compare it with the identity in the pre-established face library, and then obtain the person's name displayed on the AR page; Step 4: In the interface developed by Android, the user selects the indoor map provided by the user through the system album, and marks the location of fixed objects in the room by clicking on the area on the map. Step 5: Use the existing indoor map provided by the user and the locations of the fixed objects in the marked room to achieve coarse-grained positioning of key objects. When a key object appears during the target detection process, the target detection module determines its coarse-grained location and saves it; Step 6: Implement the voice wake-up module through the plug-in provided by the Picovoice solution to wake up the system. The voice recognition module automatically captures the audio for voice recognition and enables the corresponding function after recognizing the specified keyword.

2. According to claim 1, the intelligent assistance system for amnesiac elderly people based on AR glasses is characterized by: The data set selected in step 1 is used for model training, and the saving format of the model file is unified, as follows: the public widerface and emore data sets are selected for face detection and face recognition model training respectively, and the training is performed on the Ubuntu 18.04 operating system and the Pytorch platform according to the default hyperparameters of the open source warehouse. The face detection module is trained for 200 epochs with a batch_size of 24 and a learning_rate of 0.02, and the face recognition model is trained for 100 epochs with a batch_size of 64 and a learning_rate of 0.

003. Finally, the model file is saved in the unified format of onnx.

3. According to claim 1, the intelligent assistance system for amnesiac elderly based on AR glasses is characterized by: The overall system framework is built on the AR glasses using flutter and Android hybrid development technology.

4. According to claim 1, the intelligent assistance system for amnesiac elderly people based on AR glasses is characterized by: The AR glasses are integrated with a camera, a microphone, and are equipped with an independent operating system and internal storage.

5. According to claim 1, the intelligent assistance system for amnesiac elderly people based on AR glasses is characterized by: The speech recognition module described in step 4 automatically captures audio for speech recognition, and activates the corresponding function after recognizing the specified keyword, as follows: The speech recognition module uses the eyeglass camera to capture real-time images as model input, and detects the images at regular intervals. If a set key object is detected in the image, it observes whether there is an annotated fixed object in the nearest adjacent images. If so, the coarse-grained position of the key object is saved near the fixed object. When the user's voice is recognized to contain the set search keyword, the most recently saved key item location is displayed on the page as a flashing red dot on the indoor map, instructing the user to search within a certain area; When the user's voice is recognized to contain the set face recognition keywords, face detection and face recognition are started on the real-time image captured by the camera. The face detection module is responsible for drawing a frame around the face in the image and providing output to the face recognition model. The face recognition module returns the extracted face feature values ​​and compares them with the faces in the face library, and returns a name. The returned results are visualized on the AR screen using the Android canvas tool.

Images