An intelligent accompanying system for the elderly based on computer vision technology

Abstract

The application discloses an intelligent accompanying system for the old people based on computer vision technology. The system is mainly composed of an indoor monitoring system, a robot system and a mobile device display terminal. The indoor monitoring system is installed with multiple integrated cameras according to the indoor space layout, and realizes dynamic tracking of real-time information of the old people at day and night through a video and infrared combined mode; the robot system is composed of an internal intelligent analysis and control system, a voice interaction system, an indoor navigation system, a vital sign detection system and the like; the accompanying robot is composed of an integrated camera, a motion system, a humanoid robot arm system, a partitioned storage box (for medicines, nutrients and the like), a sensor module; and the mobile device display terminal includes a mobile phone, a smart watch, a computer and the like, and is used for displaying video information according to a task, and simple gesture interaction can be realized. The accompanying system meets the demands of children caring about the health psychology of the old people and reducing the consumption of manpower and material resources, and improves the timeliness and accuracy of emergency rescue.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and companion robots, and particularly relates to an intelligent companion system for the elderly based on computer vision technology. BACKGROUND

[0002] More and more young people go to big cities to seek development, so that the elderly are less accompanied, and various problems such as the quality of life of the elderly, daily life, chronic diseases, etc. are becoming more and more prominent. At the same time, children are under pressure and spend a lot of time working, lacking communication and communication with the elderly, and cannot understand the physical condition of the elderly in the first time, and the elderly understand the difficulties of children and do not want to bother children in most cases, which to some extent leads to the situation of the elderly being left alone and helpless.

[0003] The existing companion robots are not convenient for interaction with the accompanying personnel when in use, and the companion robots cannot actively identify the position and physical condition of the indoor elderly, so as to play a better accompanying effect. Patent document CN114029948A records an indoor companion robot based on deep learning, including a body and a bionic leg, etc., but it is difficult to apply in the indoor. Patent document CN114227712A records an interactive pension companion robot based on a computer device, containing multiple modules, but the accuracy of the infrared wave detection of the body condition is difficult to guarantee. Therefore, an indoor companion robot that can better fit the actual situation and solve the living problems of the elderly is urgently needed. SUMMARY

[0004] The present application provides an intelligent companion system for the elderly based on computer vision technology, which solves the problem of intelligent pension for the elderly at home.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is: an intelligent companion system for the elderly based on computer vision technology, which comprises a central control module, an indoor monitoring module, a companion robot module and a mobile terminal display module.

[0006] In the indoor monitoring module, a plurality of integrated cameras are installed in the indoor monitoring module according to the indoor space layout, the total visual field covers all indoor activity positions that need to be monitored, the information of the same information source is encapsulated and marked as target person 1 signal, other person signals are processed in the same way, the dynamic tracking of the person activity in the video information is realized through the tracking algorithm based on deep learning, and the information is sent to the central control module; different objects and different small animals are distinguished through a pre-trained model; the central control module analyzes the features according to the collected information, detects and analyzes the abnormal state of the elderly, when the video person activity analysis result or detection result prompts that the elderly activity is abnormal or needs help, the signal is transmitted to the companion robot module through 5G communication technology;

[0007] In a preferred solution, the integrated camera has two modes, day and night, which are automatically switched according to the ambient light intensity. When the ambient light intensity is higher than a preset value, the normal camera detects the old person's movement information in real time. When the ambient light intensity is lower than the preset value, the infrared camera is switched to detect the old person's movement information in real time.

[0008] In a preferred solution, the feature analysis includes analysis of facial features, gait features, and voiceprint features.

[0009] The facial feature analysis and the voiceprint feature analysis compare the collected information with the pre-stored information to extract abnormal features.

[0010] The gait feature generates dynamic walking information from the video image, which is compared with the pre-stored information to extract abnormal features.

[0011] The facial feature analysis includes the following steps:

[0012] S1, data collection: first, collect the facial data of the person in the video, and perform image filtering and histogram equalization on the image for preprocessing;

[0013] S2, feature extraction: calculate the Gaussian difference pyramid, detect local extreme points in the Gaussian difference pyramid, and use scale space extreme points to fit the exact position and scale to obtain key points. Use RPN to generate face candidate regions, i.e. based on different scales and feature maps of the input image, use RPN to generate candidate regions (face objects), and adjust their size and shape to adapt to different facial morphologies. Based on the facial feature point coordinates, extract the face shape, nose shape, and eye shape ratio features, and construct a face ratio feature vector;

[0014] S3, feature matching: use Keras to build a twin network model, which includes convolution layers and two pooling layers. By inputting two face images, the corresponding similarity scores are obtained. During model training, a contrast loss function is used to optimize the model, and ROC curve and other indicators are used to evaluate its performance.

[0015] The voiceprint analysis includes the following steps:

[0016] S11, import the collected voice information from the monitor;

[0017] S22, pre-process the voice information, including pre-emphasis, framing, windowing, and endpoint detection;

[0018] S33, after the voice information preprocessing, the frequency spectrum diagram is obtained by fast Fourier transform, and the mel frequency cepstrum coefficient (MFCC) is calculated and normalized. TDNN is adopted, the input sequence is shifted by using time delay line, the neural network extracts features from the input of different time steps, and the pooling layer is further used to extract features, such as using statistical pooling to take mean, variance, maximum, minimum, median and other statistical quantities as the feature representation of this dimension. Finally, the features are integrated through the full connection layer to obtain the final voiceprint feature representation. The interval-based classification loss function LAMS and the small sample-based prototype cosine loss function LCP are selected as the loss function to realize the goal of increasing the distance between classes and reducing the distance within the class;

[0019] S44, the siamese network includes two identical networks, each network has the same structure and parameters, the input is a pair of voiceprint feature vectors, and the output is the similarity between the two voiceprint feature vectors. Each network is composed of multiple convolutional layers, pooling layers, fully connected layers and activation functions, which are used to extract voiceprint features;

[0020] S55, the siamese network model is built using Keras, which includes convolutional layers and two pooling layers, and the corresponding similarity score is obtained by inputting two face images. The contrast loss function is used to optimize the model during model training, and the ROC curve and other indicators are used to evaluate its performance;

[0021] S66, according to the matching result as one of the identity identification indicators. Based on the TOPSIS algorithm, the factors of the feature vector are weighted, the comprehensive similarity probability G is calculated, and the identity identification is taken as the identity identification. In the preferred scheme, the face analysis and voiceprint analysis are realized based on the siamese network.

[0022] In the preferred scheme, the face feature analysis and voiceprint feature analysis adopt one or more combinations of tracking algorithms based on deep learning, such as siamese network (twin neural network) or region proposal network.

[0023] In the preferred scheme, the accompanying robot module is also provided with an intelligent analysis and control system, a voice interaction system (voice through remote control and accompanying robot action), an indoor navigation system (autonomous or through remote control), a vital sign detection system (autonomous or through remote control).

[0024] In a preferred solution, the intelligent analysis and control system acts as the brain of the accompanying robot, by understanding the data characteristics transmitted by the indoor monitoring system, inferring various situations that the old person may encounter, and making intervention plans and taking action through decision trees. If the intelligent analysis and control system feels "unconfident" about the intervention plan to be implemented, it actively calls the video information of the scene where the target is located, and performs algorithm detection and investigation of other most likely dangerous situations. At the same time, through the indoor navigation system, it actively goes to the room where the old person is located. The "unconfident" state is determined by establishing a behavior-result analysis model, using artificial intelligence algorithms to infer the result from the behavior, and if the inferred result is one of the following three:

[0025] 1. Cannot solve the existing problem;

[0026] 2. The inferred result may cause harm to the target person;

[0027] 3. The inferred result has no effect.

[0028] When the probability of the above results is greater than a preset value, it is determined that the preset plan is "unconfident".

[0029] In a preferred solution, the indoor navigation system can be autonomously navigated or remotely controlled. The self-protection navigation uses image processing and computer vision technology to detect and track objects in the video, identify pets and record their positions and motion trajectories, and uses SLAM algorithm to construct a travel map based on object detection and tracking data collected by multiple integrated cameras at multiple positions. The generated map data is transmitted to the accompanying robot in the room through 5G technology.

[0030] The accompanying robot uses algorithms to process and analyze the map to plan the path.

[0031] The accompanying robot uses the camera mounted on it to obtain image information of the surrounding environment, matches the current position of the accompanying robot with the position on the map through image processing technology, realizes autonomous positioning, and further obtains the position of the accompanying robot through the positioning information in the map.

[0032] The accompanying robot uses path planning algorithms such as A* algorithm to plan the optimal travel route on the map based on the current position and the position of the target point.

[0033] The accompanying robot autonomously navigates according to the motion ability and task requirements, follows the path or performs other tasks.

[0034] The accompanying robot uses convolutional neural networks or recurrent neural networks to analyze the data captured by the integrated camera, accurately identifies and tracks different environmental features, and optimizes the local map.

[0035] The companion robot detects the movement and rotation of the ground through feedback data from its own sensors, updating the map of the area it is in;

[0036] The companion robot controls its forward movement, turning, and stopping through an internal control system.

[0037] In a preferred solution, the intelligent analysis and control system extracts video information from the room where the companion robot is located and the room where the elderly person is located, as well as other relevant video information, and performs preprocessing operations such as denoising, stabilization, and filtering to improve the accuracy of subsequent image analysis and recognition;

[0038] Using image processing and computer vision techniques, such as object detection algorithms including YOLO and tracking algorithms including KCF, the objects in the video are detected and tracked, and the pets are identified and their positions and motion trajectories are recorded;

[0039] SLAM algorithm is used to construct a travel map based on the object detection and tracking data collected by the integrated cameras at multiple locations;

[0040] The generated map data is transmitted to the companion robot in the room through 5G technology, and the companion robot processes the data using algorithms to analyze the map, plan the path, and perform other operations;

[0041] The companion robot uses the integrated cameras to obtain image information of the surrounding environment, matches the current position of the companion robot with the position on the map through image processing technology, realizes autonomous positioning, and further obtains the position of the companion robot through the positioning information in the map;

[0042] The companion robot uses path planning algorithms such as A* algorithm to plan the optimal travel route on the map based on the current position and the position of the target point;

[0043] The companion robot performs autonomous navigation based on its movement ability and task requirements, follows the path or performs other tasks;

[0044] The companion robot uses deep learning models such as convolutional neural networks (CNN) or recurrent neural networks (RNN) to analyze the data captured by the integrated cameras, accurately identify and track different environmental features (such as walls, furniture, and obstacles), and optimize the local map;

[0045] The companion robot detects the movement and rotation of the ground through feedback data from its own sensors (such as encoders, gyroscopes, etc.), updating the map of the area it is in;

[0046] The companion robot controls its forward movement, turning, and stopping through an internal control system.

[0047] The accompanying robot is characterized in that:

[0048] The accompanying robot is composed of an integrated camera, a motion system, a humanoid robot arm system, a partitioned storage box (such as storing medicines, nutrients, etc.), and a sensor module. It is characterized in that:

[0049] The visual camera is installed at the left eye position of the head of the accompanying robot;

[0050] The thermal imaging camera is installed at the right eye position of the head of the accompanying robot;

[0051] The interactive screen is installed on the body of the accompanying robot, using a 9-inch LCD screen, driven by a mainboard, powered by an external BEC, connected to an STM32 master control chip, combined with the visual main camera of the head of the accompanying robot, to realize network video calls with relatives;

[0052] In the preferred scheme, the sensor module includes a motion sensor module, an interactive sensor module, an environmental sensor module, and a vital sign sensor module.

[0053] The motion sensor module includes a laser sensor, a camera, and a gyroscope;

[0054] The interactive sensor module includes a camera, a projector, an input button and a panel, and a sound pickup;

[0055] The environmental sensor module includes an indoor thermometer, an anemometer, a hygrometer, and a harmful gas sensor;

[0056] The vital sign sensor includes one or a combination of more than one of a thermometer, a sphygmomanometer, a heart rate, and a blood oxygen meter.

[0057] In the preferred scheme, for each detected face, a face feature extraction algorithm such as LBP, HOG, CNN, etc. is used to extract the feature vector of the face;

[0058] For each face feature vector, a machine learning algorithm such as support vector machine (SVM), random forest (Random Forest), etc. is used for face recognition or sentiment analysis tasks.

[0059] The motion system is characterized in that:

[0060] The accompanying robot is equipped with two or more wheels driven by electric motors to realize forward movement, backward movement, turning, and drifting;

[0061] In the preferred scheme, the accompanying robot moves according to a preset route driven by the motor-driven wheels;

[0062] In a preferred solution, the accompanying robot is loaded with ultrasonic sensors and contact sensors, and adjusts the current motion state of the accompanying robot according to the ground state, and automatically avoids obstacles by adjusting the motor drive through the master control chip. The contact sensor is used to avoid collision. The cycle is executed until the obstacle is passed.

[0063] In a preferred solution, when the accompanying robot encounters an item label identified by the pre-trained model during the journey, the human-like hand robot arm system is used to complete actions such as opening the door and moving the object.

[0064] The human-like hand robot arm system is characterized by:

[0065] The shape is simulated according to the human arm, including a mechanical arm and mechanical fingers, and is characterized by the flexibility of the mechanical fingers and the multi-directional movement of the arm.

[0066] The mechanical arm is controlled by the built-in intelligent analysis and control system of the accompanying robot.

[0067] In a preferred solution, after the accompanying robot reaches the target room, the integrated camera is used to obtain the orientation of the old person's heat source and move to the target heat source according to the preset route.

[0068] The vital sign detection system is characterized by:

[0069] The pre-installed sensor module is used to detect vital signs (such as heart rate, respiration, body temperature, blood pressure, etc.), and the intelligent analysis collects whether the vital sign data is within the normal range and the degree of deviation. Warning information is transmitted to the mobile device display terminal in a timely manner to prompt the emergency contact person;

[0070] If the situation is very urgent, the emergency plan is immediately started, including delivering basic disease drugs, contacting 120 emergency services, and 119 fire services.

[0071] The voice interaction system is characterized by:

[0072] In addition to verifying emergency situations, the voice interaction system based on Baidu API DE voice interaction system also meets the needs of alleviating the loneliness of the elderly, coping with the decline in vision, lack of flexibility in fingers, and insufficient knowledge reserves of the elderly, which hinder the quality of pension needs;

[0073] It can answer the questions of the elderly, play music, control home smart devices, and interact with other smart home devices, and can be configured and controlled through a smart phone application;

[0074] Voice recognition algorithms are used to convert voice signals into text form, including end-to-end speech recognition algorithms based on Hidden Markov Models (HMM) and deep learning.

[0075] Utilize natural language processing algorithms to convert text-form speech recognition results into understandable semantic information, including syntax analysis, entity recognition, sentiment analysis, and dialogue management.

[0076] Utilize speech synthesis algorithms to convert computer-generated text into speech signals, such as deep learning-based neural network speech synthesis algorithms.

[0077] Utilize dialogue management algorithms to control the flow of dialogue, including dialogue state tracking, strategy generation, and response generation.

[0078] Through cloud knowledge base queries, utilize hybrid recommendation algorithms to provide reasonable recommendations for problems encountered by the elderly.

[0079] After obtaining authorization from the elderly, actively dispatch intelligent home devices and other resources available through internet technology.

[0080] If sensitive information is involved, double identification must be taken, such as joint confirmation by the elderly and emergency contacts before execution.

[0081] Mobile device display terminals include mobile phones, tablets, smart watches, laptops, or desktop computers, etc., for displaying video information according to task calls.

[0082] The accompanying robot screen can achieve simple gesture interaction, such as setting "draw a straight line on the screen" and dialing a phone call to a specific contact as a binding instruction, etc., to handle special situations encountered during actual operation.

[0083] In a preferred scenario, the companion robot acts as a first-line "savior" in the event of an emergency for the elderly. After the companion robot is activated, it quickly completes the preliminary diagnosis of critically ill patients in need of monitoring through a set of intelligent and automated analysis systems. This is achieved by detecting the basic vital signs of the elderly through detectors and matching their current signs and symptoms with the symptoms of common and frequently occurring diseases in various systems through computer vision technology, resulting in information 1 (including possible complaints, current medical history, etc.). After the indoor monitoring device sends an early warning, the companion robot successfully reaches the side of the elderly and conducts basic vital sign detection, resulting in information 2 (basic physical examination). Through the electronic archive system, information 3 is obtained (obtaining the patient's general information such as age, occupation, etc.), and all obtained information is automatically converted into text information to generate a virtual case. A virtual intelligent "intern" trained through a neural network makes a preliminary diagnosis, automatically generates a diagnosis order based on the probability size, and generates a dispatch plan for nearby available medical resources. All information obtained (including location coordinate information, which can be specific to the house number, such as obtained from the ID number of the monitor) and treatment recommendations are fed back to the elderly's indoor video and emergency video window. Professional emergency doctors inquire and confirm the emergency level and required emergency resources through the voice of the monitor, further completing the classification of the emergency and the dispatch of the emergency team. In this case, the companion robot can be used as a companion device recommended by the hospital for the prognosis and rehabilitation of critically ill patients, and the specific prognosis and rehabilitation recommendations are fed back to the companion robot, which completes the subsequent human-inaccessible follow-up care and nursing work. The companion robot, designed by professional graphic designers, can be put into market use.

[0084] In a preferred scenario, the companion robot is an effective treatment "intervener" that acts as an "effective treatment" intervener for patients with serious underlying diseases but are able to self-care. This includes:

[0085] 1. Disease control and treatment: Control and treatment of the patient's underlying disease, such as hypertension, diabetes, lung disease, etc.

[0086] 2. Nutritional support: Nutritional assessment of the patient and provision of appropriate nutritional support to ensure that the patient receives adequate nutrition and energy.

[0087] 3. Exercise and rehabilitation: Proper physical exercise and rehabilitation measures help improve the patient's physical condition and enhance their self-care ability.

[0088] 4. Psychological support: Assessment of the patient's emotions and provision of necessary psychological support to help them cope with possible emotional and psychological stress.

[0089] 5. Drug treatment: If necessary, the patient needs to receive drug treatment to alleviate disease symptoms and control disease progression.

[0090] 6. Regular follow-up: Regular follow-up is conducted on the patient to monitor disease progression and treatment effect, and necessary adjustments and management are made.

[0091] Although the above 1-5 aspects are not completely completed by the accompanying robot, the accompanying robot can serve as a reminder of the timely use of drugs, a consultant for adaptation and contraindication during treatment and rehabilitation, a small helper for communication between doctors and patients, and can comprehensively obtain information to formulate reasonable and personalized prognosis management. The prognosis management scheme can be immediately fed back and adjusted.

[0092] The beneficial effects of the present application are: by arranging indoor real-time integrated cameras, the dynamic tracking and danger monitoring of the behavior of the old people are realized, the accompanying robot can be awakened to reach the location of the old people as soon as possible and further intervene when danger occurs, or feedback the danger information to the emergency contact person in the first time, actively provide emergency plans, automatically complete emergency help after authorization, and have the timely and practical characteristics of practical application scenarios; through the built-in intelligent analysis and control system of the accompanying robot, in addition to effectively preventing the old people from danger, the emotional needs of the old people such as accompanying, dignity, and the practical needs of undertaking housework and reducing the overwork of the old people can be met, especially for the old age group with declining vision and unagile fingers. At the same time, the accompanying robot reminds the old people to take medicine regularly and supplement nutrients, which can realize effective tertiary prevention of basic diseases. In general, the accompanying system meets the demands of children caring about the health of the old people and reducing the consumption of manpower and material resources on the one hand, improves the timeliness and accuracy of emergency rescue, and ensures the health of the old people; on the other hand, promotes the important application of 5G technology, artificial intelligence and other frontier technologies in the aging society, and can effectively improve the quality of old-age care. BRIEF DESCRIPTION OF DRAWINGS

[0093] The present application will be further described below in conjunction with the drawings and examples:

[0094] Figure 1 is a three-view of the intelligent accompanying robot of the present application;

[0095] Figure 2 is a three-view of the intelligent accompanying robot of the present application after the mechanical arm is parked;

[0096] Figure 3 is a three-view of the intelligent accompanying robot of the present application;

[0097] Figure 4 is a visual pose recognition joint point schematic diagram of the present application.

[0098] Figure 1 M: accompanying robot head 1; accompanying robot body 4; accompanying robot large arm 5; accompanying robot small arm 6; accompanying robot palm 7.

[0099] Example 1:

[0100] like Figures 1 to 3 As shown, the intelligent companion robot is composed of three main parts: the head, the body, and the robotic arm. The head's main function is to monitor the elderly person's physical condition in real time. The body is responsible for maintaining the robot's normal operation, supporting the entire body structure, and facilitating intelligent interaction. The robotic arm consists of an upper arm, a forearm, and a hand. The shoulder connects the robotic arm to the body, and motors enable rotation between the upper arm, forearm, and hand. The fingers on the hand bend and grasp using motors and linkages. In terms of details, in addition to the chassis, there are four wheels connected to a differential motor, enabling the robot's movement. The side of the body has a slot for the robotic arm to park when not in use, preventing it from bumping into obstacles.

[0101] Basic functions of companion robots:

[0102] 1. Picking up items: According to the elderly's medical orders, the care robot pops out a medicine box, and the robotic arm operates to take out the medicine box and hand it to the elderly, as well as handing the elderly a water cup; if the elderly drop something, the care robot can identify and pick up the item and hand it to the elderly without the elderly having to bend over.

[0103] 2. Laundry: The care robot identifies the color of the clothes and sorts them, then puts the sorted clothes into the washing machine. After washing, it takes them out and puts them in a basket to remind the elderly to hang them to dry.

[0104] 3. Cooking: The care robot asks the elderly what they want to eat today, orders groceries online and delivers them to their door, puts the vegetables in a washing basket, rinses them with water while shaking the basket, puts the washed vegetables into the cooking care robot to start cooking, and reminds the elderly of the progress of cooking.

[0105] 4. Entertainment: When an elderly person has no one to accompany them, ask if they want to play chess. The chessboard can be projected onto a TV screen via the internet, and the elderly person can operate the screen of the companion robot to play chess with the companion robot or other chess players for entertainment purposes.

[0106] Example 2:

[0107] A smart companionship system for the elderly based on computer vision technology, characterized by:

[0108] The system mainly consists of an indoor monitoring system, a robot system, and a mobile device display terminal;

[0109] The indoor monitoring system installs multiple integrated cameras according to the indoor space layout, characterized by:

[0110] Multiple cameras are installed in one housing unit, with a total field of view covering all indoor activity locations that need to be monitored;

[0111] The integrated camera has two modes of day and night, which automatically switches according to the visibility. The normal camera detects the action information of the old people in real time in the daytime, and switches to the infrared camera to detect the action information of the old people in real time when the visibility is low.

[0112] Assuming that the ambient brightness is L, the threshold of the visible light camera is T1, and the threshold of the infrared camera is T2, the switching formula is represented as:

[0113] When L>T1, the visible light camera is used; when L<T2, the infrared camera is used; when T2≤L≤T1, it is decided according to the specific requirements which camera to use;

[0114] The collected video information is converted into digital information, and the face feature, gait feature and voiceprint feature are analyzed in the cloud through 5G technology. The information from the same source is packaged and marked as person 1 signal, and other person signals are processed in the same way.

[0115] The collected video images are preprocessed, such as adjusting image brightness and contrast, removing noise, etc.

[0116] Face detection algorithm such as Haar feature classifier or deep learning algorithm is used to detect faces in video images and obtain face position and size.

[0117] Gait feature recognition consists of feature extraction and feature matching:

[0118] Linear discriminant analysis (LDA) is used for feature extraction, and image data conversion is performed through a feature transformation matrix, whose formula is as follows:

[0119] Assuming that the gait image sample x is an n-dimensional vector, there are m categories of samples: w1, w2, …, w m .

[0120] Then the inter-class dispersion matrix s b and the intra-class dispersion s w can be defined as formula (1.1), (1.2).

[0121]

[0122]

[0123] In the formula, p(w i ) is the prior probability of the i-th sample, represents the mean vector of all sample images, represents the mean vector of the i-th sample image, and the LDA criterion is represented by formula (1.3).

[0124]

[0125] To maximize Jw, w must satisfy: s b w = λs w w. When s w Each column of the optimal projection space w is the eigenvector corresponding to the k-1 largest eigenvalues of

[0126] Feature matching is performed using the Euclidean distance method. If the Euclidean distance between the feature to be matched and the existing feature in the database is less than the set threshold, it is considered as a successful feature matching;

[0127] An acoustic feature extraction method based on Gaussian mixture model is used for voiceprint recognition:

[0128] MFCC (Mel Frequency Cepstral Coefficient) is used to extract voiceprint features from audio signals;

[0129] A Gaussian mixture model is established for each sound source. The EM algorithm (Expectation Maximization algorithm) is used to estimate the model parameters, such as the mean and covariance matrix of the Gaussian distribution, and the weight of each Gaussian component;

[0130] The new sound sample is compared with each sound source model, the likelihood under each model is calculated, and the model with the highest likelihood is selected as the recognition result.

[0131] The composite criterion is encapsulated to judge different person information, which can accurately distinguish between the elderly and other family members, and objects, pets, etc.

[0132] Different objects and small animals (pets, mosquitoes, and flies) are distinguished by pre-training models:

[0133] Convolution and pooling operations are performed on images based on convolutional neural networks (CNN), and classification is performed based on the extracted features to distinguish different objects and species in the images;

[0134] Feature extraction and classification are performed on audio signals based on recurrent neural networks (RNN);

[0135] Through logical judgment, the elderly and other objects, pets, etc. can be accurately distinguished.

[0136] Deep learning-based tracking algorithms such as Siamese network and region proposal network (RPN) are used to realize dynamic tracking of human activities in video information.

[0137] ​The abnormal state of the old people is detected through a fall detection algorithm, a radio spectrum analysis algorithm, etc., and when the video character activity analysis result and / or the algorithm detection result indicates that the old people's activity is abnormal or needs help, a signal is transmitted to the robot system through 5G communication technology to realize intelligent and automatic management and scheduling of indoor site information.

[0138] The Siamese network is a model design framework, which is a coupled framework based on two artificial neural networks. The main use of the Siamese neural network is to input two samples, output the representation of the embedding high-dimensional space, and compare the similarity of the two samples. The Siamese neural network usually has a deep structure, which can be composed of convolutional neural networks, recurrent neural networks, etc.

[0139] Under the supervised learning paradigm, the Siamese neural network maximizes the representation of different labels and minimizes the representation of the same label. Under the self-supervised or unsupervised learning paradigm, the Siamese neural network can minimize the representation between the original input and the interference input (for example, the original image and the cropped image of the image).

[0140] The Siamese network can mainly solve two types of problems. The first type is that the number of categories is small, and the amount of data of each category is large, such as ImageNet, VOC, etc. This classification problem can be solved by using a neural network or SVM, as long as all the categories are known in advance. The second type is that the number of categories is large (or the specific number cannot be confirmed), and the amount of data of each category is small, such as face recognition and face verification tasks. In order to solve this kind of problem, we will train a model to output the similarity of two given samples. The model learns the similarity function, and by setting a threshold for the similarity, it can be determined whether it belongs to the same person.

[0141] The Siamese network outputs a vector. If the input sample vectors X1 and X2 are the same person, the Euclidean distance of the one-dimensional feature vectors output by the upper and lower models is small. If the input sample vectors X1 and X2 are not the same person, the Euclidean distance of the one-dimensional feature vectors output by the upper and lower models is large. Therefore, by calculating the Euclidean distance of the vectors output by the upper and lower models, the similarity of the two input samples can be obtained.

[0142] Siamese-RPN can be used for small sample / single sample learning, and is not easily disturbed by false samples, and can be used for pattern recognition problems with strict fault tolerance requirements, such as portrait recognition, fingerprint recognition, target tracking, etc. The Siamese network accepts two inputs, one called template frame, which is the object or audio position artificially framed from the first frame of video or audio; the other input is called detection frame, which is the frame other than the first frame of the video or audio segment to be detected.

[0143] In a preferred scheme, the elderly abnormal state is detected by a fall detection algorithm, a radio spectrum analysis algorithm, etc., and when the video character activity analysis result and / or the algorithm detection result prompts that the elderly activity is abnormal or needs help, a signal is transmitted to the robot system through 5G communication technology to realize intelligent and automatic management and scheduling of indoor site information.

[0144] The fall detection algorithm extracts the target skeleton by background difference method and morphological algorithm, and the skeleton extraction goes through nine steps: image graying, target contour extraction by background difference method, contrast enhancement using CLAHE algorithm, Gaussian filtering, edge detection using Solel operator, wavelet denoising, maximum inter-class error method binarization, morphological operation and median filtering. In the time domain window, first, the image is downgraded to obtain a gray image, and the gray image is processed by difference and mathematical morphology to obtain the motion skeleton of the moving target. This algorithm can have a good segmentation result for the moving area of the static background sequence. After layer-by-layer processing of the above seven steps, the motion target skeleton graph is obtained, and the subsequent recognition detection is performed on this basis.

[0145] Four-connected or eight-connected is a basic concept in image processing: 8-connected, which means that if a pixel is connected to other pixels in the upper, lower, left, right, upper left corner, lower left corner, upper right corner or lower right corner, they are considered to be connected; 4-connected means that if the position of a pixel is adjacent to the upper, lower, left or right of another pixel, they are considered to be connected, and if they are connected in the upper left corner, lower left corner, upper right corner or lower right corner, they are not considered to be connected.

[0146] There are many connected region labeling algorithms, some of which can complete labeling once by traversing the image, and some of which need to traverse the image 2 or more times, which causes differences in time efficiency of different algorithms. The algorithm used in this paper is the algorithm used in the connected region labeling function bwlabel in matlab, which traverses the image once and records the continuous team (run) and the equivalent pair of labels in each row (or column), and then re-labels the original image through the equivalent pair.

[0147] Because the human body has a certain proportion, for example, the arm is in the upper half of the body, and the foot is in the lower half of the body. It can be imagined that if the number of white points (white pixel target) in a certain area is relatively large, it means that there is a limb in the area. If the set of all possible postures can be defined in advance, and the area combination of each posture is determined, the posture can be judged in advance. This algorithm first needs to determine the standard position of the human body, which is marked with a red rectangular frame, and the height H and width L of the rectangular frame are calculated, and the posture of the human body is judged according to the height-width ratio. Because the speed of a person when actually falling is relatively fast, and there is a tendency for the center of gravity to drop, according to such characteristics of the human body, many false falls can be excluded. The time of a video frame is about 0.03 seconds, and the two images separated by 10 frames are taken as the judgment standard of the motion trend, that is, the motion trend of the target object is judged every 0.3 seconds. This algorithm first extracts the motion target skeleton, then labels the skeleton, and preliminarily judges whether the target has fallen according to the proportion of the skeleton. In the state of judging that it has fallen, the motion trend is used to screen false falls, and the real fall state is left.

[0148] The radio spectrum analysis algorithm is a method for real-time monitoring of the physical condition of the elderly by using sensors, radio communication technology and data analysis technology. The core idea of this algorithm is to monitor the physiological state of the elderly by analyzing the radio signals around them, so as to timely discover whether the elderly have abnormal conditions such as sudden cardiac arrest, abnormal heart rate, etc.

[0149] The implementation of the radio spectrum analysis algorithm needs to rely on some sensors and wireless communication technology, such as heart rate sensors, respiration sensors, radio signal receivers, etc. These sensors will collect the physiological data of the elderly and transmit them to the data processing unit through radio signals. The data processing unit extracts the physiological data of the elderly by performing spectrum analysis on the received radio signals, and performs real-time monitoring and analysis.

[0150] Example 3:

[0151] The robot system is composed of an internal intelligent analysis and control system, a voice interaction system, an indoor navigation system, a vital sign detection system, etc., and is characterized in that:

[0152] The internal intelligent analysis and control system serves as the brain of the accompanying robot, understands the data characteristics transmitted by the indoor monitoring system, infers various possible situations of the elderly, formulates intervention plans through decision trees, and takes action;

[0153] If the built-in intelligent analysis and control system feels "unconfident" about the impending intervention plan, it actively calls the video information of the scene where the target is located, conducts algorithm detection and investigation of other most likely dangerous conditions, and at the same time, actively goes to the room where the old person is located in time through the indoor autonomous navigation system. The "unconfident" state is determined by establishing a behavior-result analysis model, using artificial intelligence algorithms to infer results from behavior, and if the inferred results are the following three:

[0154] 1. Cannot solve the existing problem;

[0155] 2. The inferred result may cause harm to the target person;

[0156] 3. The inferred result has no effect.

[0157] When the probability of the above results is greater than the preset value, it is determined that the preset plan is "unconfident";

[0158] A percentage scoring method is used, that is, the prediction of the accompanying robot is assigned a score of 0-100. If the accompanying robot's intuition is completely correct, the accompanying robot will get a score of 100. If it is partially correct, the accompanying robot may get a score of 60-90, and if it is completely wrong, the accompanying robot can only get a score of 0-50.

[0159] Indoor elderly people may appear dangerous and be classified, and pre-trained models for corresponding dangerous situations are trained using Internet resources, including fire, theft, etc. According to actual conditions, constantly improve, including emergency drug, nutrient shelf life inspection, etc.

[0160] Indoor autonomous navigation system:

[0161] In the preferred scheme, the built-in intelligent analysis and control system extracts video information of the room where the accompanying robot is located and the room where the old person is located and other video information related to it, and performs preprocessing operations such as denoising, stabilization, and filtering to improve the accuracy of subsequent image analysis and recognition;

[0162] Image processing and computer vision techniques, such as object detection algorithms including YOLO and tracking algorithms including KCF, are used to detect and track objects in the video, identify pets, and record their positions and movement trajectories;

[0163] SLAM algorithm is used to construct a travel map based on object detection and tracking data collected by multiple integrated cameras at different locations;

[0164] Map data generated by 5G technology is transmitted to the accompanying robot in the room, and the accompanying robot uses algorithms to process and analyze the map, plan a path, etc.

[0165] The accompanying robot uses the camera to obtain image information of the surrounding environment, matches the current position of the accompanying robot with the position on the map through image processing technology, realizes autonomous positioning, and further obtains the position of the accompanying robot through the positioning information in the map;

[0166] The accompanying robot plans the optimal route on the map according to the current position and the position of the target point using a path planning algorithm such as A* algorithm;

[0167] The accompanying robot autonomously navigates according to the motion ability and task requirements, follows the path or performs other tasks;

[0168] The accompanying robot uses a deep learning model such as convolutional neural network (CNN) or recurrent neural network (RNN) to analyze the data captured by the integrated camera, accurately recognize and track different environmental features (such as walls, furniture and obstacles, etc.), and realize local map optimization;

[0169] The accompanying robot detects the movement and rotation of the ground through the feedback data of its own sensors (such as encoders, gyroscopes, etc.), and updates the map of the area where it is located;

[0170] The accompanying robot completes the travel control through the internal control system, such as controlling the accompanying robot to move forward, turn, stop, etc.

[0171] The accompanying robot, characterized in that:

[0172] The accompanying robot is composed of an integrated camera, a motion system, a humanoid robot arm system, a partitioned storage box (such as storing medicines, nutrients, etc.), and a sensor module. Its characteristics are:

[0173] The visual camera is installed at the left eye position of the head of the accompanying robot;

[0174] The thermal imaging camera is installed at the right eye position of the head of the accompanying robot;

[0175] The interactive screen is installed on the body of the accompanying robot, using a 9-inch lcd screen, driven by the mainboard, powered by the external bec, connected to the stm32 main control chip, combined with the visual main camera of the head of the accompanying robot, realizing the network video call with the relatives;

[0176] The motion system, characterized in that:

[0177] The accompanying robot is equipped with two or more wheels driven by electric motors to realize forward, backward, turning and drifting;

[0178] In the preferred scheme, the accompanying robot moves according to the preset route driven by the motor;

[0179] In a preferred solution, the accompanying robot is loaded with ultrasonic sensors and contact sensors, and adjusts the current motion state of the accompanying robot according to the ground state, and automatically avoids obstacles by adjusting the motor drive through the master control chip. The contact sensor is used to avoid collision. The cycle is executed until the obstacle is passed.

[0180] In a preferred solution, when the accompanying robot encounters an item label obtained through a pre-trained model during the journey, the human-like hand robot arm system is used to complete actions such as opening the door and moving the object.

[0181] The human-like hand robot arm is characterized by:

[0182] The shape is simulated according to the human arm, including a mechanical arm and mechanical fingers, and is characterized by the flexibility of the mechanical fingers and the multi-directional movement of the mechanical arm.

[0183] The mechanical arm is controlled by the built-in intelligent analysis and control system of the accompanying robot.

[0184] In a preferred solution, after the accompanying robot reaches the target room, the integrated camera obtains the orientation of the old person's heat source and moves to the target heat source according to the preset route.

[0185] Vital sign detection system:

[0186] Through the pre-installed sensor module, vital signs such as heart rate, respiration, body temperature, blood pressure, etc. are detected, and the collected vital sign data is intelligently analyzed to determine whether it is within the normal range and the degree of deviation. Early warning information is transmitted to the mobile device display terminal in a timely manner to prompt the emergency contact person.

[0187] If the situation is very urgent, an emergency plan is immediately activated, including delivering basic disease drugs, contacting 120 emergency services, and 119 fire services, etc.

[0188] Voice interaction system:

[0189] Relying on the Baidu APIE voice interaction system, in addition to verifying emergency situations, it also meets the needs of alleviating the loneliness of the elderly, coping with the decline in vision, lack of flexibility in fingers, and insufficient knowledge reserves of the elderly, which hinder the quality of pension needs.

[0190] For each detected face, a face feature extraction algorithm such as LBP, HOG, CNN, etc. is used to extract the feature vector of the face.

[0191] For each face feature vector, a machine learning algorithm such as support vector machine (SVM), random forest (Random Forest), etc. is used for face recognition or sentiment analysis tasks.

[0192] The face recognition process of the accompanying robot is as follows:

[0193] S111, data collection: first, collect the three-dimensional facial data of the person in the video, and use a depth camera or other device to collect. Perform a different size scaling operation on the input image to form an image pyramid.

[0194] S222, feature extraction: the collected three-dimensional facial data needs to be extracted. The MTCNN algorithm is used for face detection to obtain a set of coordinates containing face position and key point information, and the external contour of the face. The composite method based on deep learning expression feature is used for feature extraction, that is, the method of combining face external features and face key point geometric features.

[0195] S333, expression classification: an output layer is added behind the deep feature extraction network layer, so that the number of last output neurons is the same as the expression categories to be classified, and then the Softmax loss function is used to calculate the cross-entropy between the correct label and the prediction result, and finally the classification result is obtained.

[0196] The mobile device display terminal includes a mobile phone, a tablet computer, a smart watch, a notebook computer or a desktop computer, etc., and is used for displaying the display of the video information according to the task call.

[0197] Embodiment 4:

[0198] The visual camera is installed at the left eye position of the head of the accompanying robot;

[0199] The thermal imaging camera is installed at the right eye position of the head of the accompanying robot;

[0200] The interactive screen is installed on the body of the accompanying robot, uses a 9-inch lcd screen, is driven by a mainboard, is powered by an external bec, is connected with an stm32 main control chip, combines the visual main camera of the head of the accompanying robot, and realizes the network video call with relatives.

[0201] In the preferred scheme, the residence of the old people is modeled by graph theory;

[0202] In the preferred scheme, the A* algorithm is used for global planning on the basis of graph theory modeling;

[0203] In the preferred scheme, a partially observable Markov decision process (POMDP) is used to make decisions on the route;

[0204] In the preferred scheme, the accompanying robot establishes a specific scene graph of the indoor scene by combining the SLAM algorithm and the three-dimensional reconstruction technology according to the read RGB image;

[0205] In the preferred scheme, the relative positions of the obstacles are marked by the depth map to obtain the distribution of the obstacles;

[0206] In a preferred solution, the model obtained through graph theory modeling is corrected by the ORB-SLAM algorithm for obstacle marking, distinguishing between room structure and obstacles, and further obtaining a feature point map of the obstacle;

[0207] In a preferred solution, the feature point map distribution of the obstacle is read to improve local route planning through the PPO algorithm to determine the final travel route of the accompanying robot;

[0208] In a preferred solution, the wheel driven by the brushless motor moves according to the preset route;

[0209] In a preferred solution, according to the ground state, the current motion state of the accompanying robot is adjusted, and if the distance between the accompanying robot and the obstacle reaches the sensing range of the ultrasonic range finder, the duty cycle of the servo motor pwm is adjusted through the main control chip to automatically avoid obstacles. Loop execution until passing through the obstacle.

[0210] In a preferred solution, the machine is trained by collecting a large number of door photos through the network, and the accompanying robot will identify the door when the distance to the door marker is less than a certain threshold when moving to the target room, and perform image scaling to make the door occupy the entire camera background.

[0211] In a preferred solution, the Laplace operator is used for image segmentation to identify the door handle, and the main control chip generates a PWM signal to drive the servo motor to press the door handle to complete the door opening action.

[0212] In a preferred solution, after reaching the target room, the heat source orientation, body temperature and heart rate of the old person are obtained through thermal imaging technology, and the wheel driven by the brushless motor moves to the target heat source according to the preset route;

[0213] In a preferred solution, the visual camera obtains real-time video of the old person;

[0214] In a preferred solution, the main control device is provided with a communicator, a memory, an operator and a power supply, the communicator collects data of the visual camera through a near field communication protocol, the near field communication protocol includes Bluetooth, WIFI, Zigbee or LoRa protocol, and the communicator is connected with a terminal through the Internet;

[0215] In a preferred solution, the memory is used to store the data of the visual camera;

[0216] In a preferred solution, an XGBoost classifier is used to classify the key nodes of the person, and a classification standard is formed according to the content of deep learning;

[0217] As Figure 4As shown, preferably, the operation unit obtains the old man model to be judged by image segmentation technology and morphological filtering, and identifies the joint points on the body. The OpenPose pose estimation algorithm is used to extract 19 key points, and further modeling is performed on the skeleton to set the threshold range corresponding to the length of each segment. The image of the collected image is fitted to the old man to determine the specific model corresponding to the old man.

[0218] In the preferred scheme, the collected old man posture is further classified according to the content of deep learning to obtain the current posture state information of the old man; the body temperature, heart rate and other information obtained by the thermal imaging camera are combined to comprehensively judge the risk expectation value of the old man, if the risk expectation value of the old man is large, the image obtained by the target camera is transmitted to the client, and the child or designated relative can be seen through the strong prompt, and the first aid medicine box is popped out, and the child or designated relative takes further processing measures, finally, when the rescue personnel arrive, the risk is removed and returns to the charging pile. If the risk expectation value of the old man is small, further interaction verification is performed by relying on the Baidu api de voice interaction system, and the demand of the old man is further met through the mobile Internet, and finally returns to the charging pile.

[0219] In the preferred scheme, the charging device includes a wireless receiving coil and a wireless transmitting device;

[0220] In the preferred scheme, the wireless receiving coil is fixed on the shell at the bottom of the accompanying robot, and after being connected to the power ic of the main control pmu, the battery is inputted and charged;

[0221] In the preferred scheme, the wireless transmitting device is fixed in the room and connected to the power supply, including a built-in voltage conversion device, a voltage stabilizing module, a on-off circuit detection device and a transmitting coil;

[0222] In the preferred scheme, the voltage conversion device reduces the input 220v high voltage to 12v for the charging system based on safety principles; after the 12v power supply reduced by the voltage conversion device, the voltage stabilizing module controls the voltage to be constant in a suitable range, and the charging ic modulates the voltage to flow to the transmitting coil; the wireless transmitting coil emits electromagnetic waves to realize the wireless charging function with the receiving coil within a certain range.

[0223] Embodiment 5: an old man A living alone

[0224] One day, the accompanying robot is located at the charging pile in the northwest corner of the living room of the old man and waits for the alarm of the system to prompt the old man to leave the bedroom at 9 o'clock in the morning.

[0225] Step 1: wake up the accompanying robot. Send a wake-up instruction to the accompanying robot through the system, and the accompanying robot starts the camera after receiving the wake-up instruction, plans the path to the old man's bedroom through the algorithm, and drives the stepping motor to travel to the old man's bedroom door according to the planned path;

[0226] Second step: path planning and automatic obstacle avoidance. After the camera collects images when traveling to the door, the handle is recognized through image segmentation, and the arm is aligned with the handle by driving the stepping motor. The arm servo motor drives the arm to press the door handle to complete the door opening action;

[0227] Third step: preliminary measurement of vital signs. After entering the old man's bedroom, the accompanying robot starts thermal imaging scanning, identifies human heat sources, records the old man's body temperature and heart rate, and compares with the deep learning database data. It is found that the old man's basic vital signs have a large abnormality. The abnormal data is reported to the terminal app, and the old man's children receive a strong reminder from the app;

[0228] Fourth step: video call and automatic return after exclusion. At the same time of reporting, the accompanying robot travels to the old man's side, uses the camera to identify the old man's posture and asks the old man whether he feels uncomfortable, and after waiting for 10 seconds, the old man does not give a response. The accompanying robot immediately sends a warning to the terminal, and the old man's children receive an app alarm and call the old man through real-time video. The old man does not give a response, and the children quickly dial 120 emergency number. After the emergency personnel arrive, the accompanying robot identifies the path to the charging pile through the camera and algorithm planning, and drives the stepping motor to automatically return to the charging pile according to the planned path.

[0229] Example 6: Old man B

[0230] One day, the accompanying robot is located at the charging pile at the intersection of the old man's kitchen and living room, and receives a reminder that the old man has not left the bathroom for too long at 3 pm;

[0231] First step: the accompanying robot wakes up the system to send a wake-up instruction to the accompanying robot. After receiving the wake-up instruction, the accompanying robot starts the camera and goes to the bathroom door according to the algorithm planning path;

[0232] Second step: path planning and automatic obstacle avoidance. After recognizing the door through image segmentation, the field of view is zoomed in to focus on the door, and then image segmentation is performed again to recognize the handle, and then the arm is aligned with the handle, and the handle is pressed open by driving the servo motor with the main control chip;

[0233] Third step: preliminary measurement of vital signs. After entering the old man's bedroom, the accompanying robot starts thermal imaging scanning, identifies human heat sources, measures the old man's body temperature and heart rate, and compares with the deep learning database data. It is found that the old man's basic vital signs have a large abnormality. The abnormal data is reported to the terminal app, and the old man's children receive a strong reminder from the app;

[0234] Fourth step: At this time, the old man is groaning painfully on the ground, the accompanying robot issues a voice prompt to inquire about the old man's condition, and according to the voice information released by the old man, the accompanying robot captures the keyword "save my life", at this time the accompanying robot realizes that danger has occurred, according to the electronic file content registered and stored in the database in advance, the first-aid medicine box is popped out and sent to the side of the old man, and the real-time image of the danger is transmitted to the management personnel of the old man, then the rescue personnel quickly come forward to eliminate the danger and save the life of the old man. After the danger is eliminated, it returns to the charging pile and waits for orders.

[0235] Example 7: Old man C

[0236] Living in Haidian District, Beijing, 69 years old, suffering from diabetes, the accompanying robot is located in the charging pile on the balcony of the old man's home, around 9 o'clock in the morning, the system alarm prompts that the old man has not left the bedroom for a certain period of time;

[0237] First step: The accompanying robot wakes up. The system sends a wake-up instruction to the accompanying robot, which starts the camera after receiving the wake-up instruction, plans the path to the old man's bedroom through algorithm, and drives the stepper motor to travel to the bedroom door according to the planned path;

[0238] Second step: Path planning and automatic obstacle avoidance. After identifying the door through image segmentation, the field of view is zoomed in to focus on the door, and then image segmentation is performed again to identify the handle, and then the arm is aimed at the handle and the handle is pressed open by the main control chip driving the servo motor;

[0239] Third step: Preliminary measurement of vital signs. After entering the old man's bedroom, the accompanying robot starts the thermal imaging scan to identify the human heat source and records the old man's body temperature and heart rate. After comparison with the deep learning database data, it is found that the old man's basic life signs are normal;

[0240] Fourth step: Automatic homing after danger is eliminated. The accompanying robot travels to the old man's side and uses the camera to identify the old man's posture and ask if the old man feels uncomfortable. After waiting for 5 seconds, the old man responds that there is no abnormality. The accompanying robot identifies the path to the charging pile through camera recognition and algorithm planning, and drives the stepper motor to automatically return to the charging pile according to the planned path.

[0241] Example 8: Old man D

[0242] 69 years old, suffering from diabetes, suddenly feels dizzy and eyesight blurred at home, then faint on the ground.

[0243] After the video monitoring found the abnormality, the accompanying robot was dispatched to check, and it was found that the old man fainted, so information 1 was generated:

[0244] (The diseases that may be associated with fainting are:

[0245] Stroke: A temporary or persistent fainting caused by lack of blood flow or bleeding in the brain, often accompanied by symptoms such as headache, nausea, and vomiting.

[0246] Heart disease: Heart disease can cause the heart to beat too fast or too slow, or cause irregular heart rhythms, which can lead to fainting.

[0247] Abnormal blood pressure: Both low and high blood pressure can cause fainting, as abnormalities in blood pressure can affect blood supply to the brain.

[0248] Nervous system disorders: Nervous system disorders such as epilepsy and Parkinson's disease can also cause fainting.

[0249] Anemia: Lack of enough red blood cells or hemoglobin in the blood can lead to insufficient oxygen supply, causing fainting.

[0250] Drug side effects: Certain medications can cause fainting, especially if used with other medications or not as recommended by a doctor.

[0251] Diabetes: Loss of consciousness caused by low or high blood sugar, especially in cases of uncontrolled or inadequate management.

[0252] According to the ABCDE principles of emergency care, the sensor module detects the basic vital signs of the elderly, generating information 2

[0253] (ABCDE principles of emergency care refer to:

[0254] A-Check airway: Ensure the patient's airway is unobstructed, including clearing any blockages or secretions in the mouth, throat, and trachea.

[0255] B-Check breathing: Check if the patient can breathe independently and assess the frequency and depth of their breath. If the patient has difficulty breathing or is not breathing enough, they need to be given respiratory support.

[0256] C-Check circulation: Check the patient's circulatory system, including heart rate and blood pressure. If the patient's heart rate or blood pressure is abnormal, take immediate action to intervene.

[0257] D-Perform neurological examination: Assess the patient's neurological function, including consciousness, pupil response, and limb mobility. If the patient's neurological function is abnormal, take immediate action to intervene.

[0258] E-Environmental control: Ensure the patient's surroundings are safe, including maintaining the patient's body temperature, preventing infection, and preventing further injury.

[0259] Access and retrieve information through the electronic archive system, and obtain information 3:

[0260] (Important information found through search: the patient who fainted has a history of diabetes, etc.

[0261] The accompanying robot automatically converts all the information obtained into text information, generates a virtual case, and makes a preliminary diagnosis by a virtual intelligent "intern" trained through a neural network. In this case, the effective information obtained is that the patient's basic vital signs are normal, but he fainted and his blood sugar is low.

[0262] Automatically generate the most likely diagnosis order based on the probability size, such as the diagnosis order in this case:

[0263] 1. Diabetes: Diabetic patients may experience hypoglycemia when they take too much insulin or oral medication after injecting insulin or taking oral medication, which can lead to hypoglycemia.

[0264] 2. Insulinoma: Insulinoma is a type of endocrine tumor of the pancreas that secretes large amounts of insulin, leading to low blood sugar. Patients may experience fainting, dizziness, sweating, and other symptoms.

[0265] 3. Adrenal gland tumor: Adrenal gland tumor is a type of pheochromocytoma that secretes large amounts of adrenaline, leading to high blood pressure, palpitations, and other symptoms, as well as low blood sugar. Patients may experience fainting, weakness, headache, and other symptoms.

[0266] 4. Pancreatitis: Acute pancreatitis can lead to insufficient insulin secretion and low blood sugar. Patients may experience abdominal pain, nausea, vomiting, and other symptoms.

[0267] 5. Drug or alcohol poisoning: Certain drugs or excessive alcohol consumption can cause low blood sugar, and patients may experience fainting, dizziness, shortness of breath, and other symptoms.

[0268] (Retrieve the coordinates of the nearest available ambulance, call for resources on the vehicle with a medical checklist, and generate the best recommended vehicle dispatch. The configuration required for an ambulance includes:

[0269] 1. Glucose injection: This is the preferred drug for treating low blood sugar, which can quickly raise blood sugar levels and ensure patient safety. Different countries may have different regulations, but generally, ambulances should have enough glucose injection to be used in emergency situations.

[0270] 2. Respiratory monitoring equipment: Low blood sugar can cause patients to experience coma, difficulty breathing, and other symptoms, so ambulances should be equipped with respiratory monitoring equipment to detect respiratory problems and take timely measures.

[0271] 3. Oxygen cylinder: If the patient experiences difficulty breathing, the ambulance should be equipped with an oxygen cylinder to provide timely oxygen inhalation to maintain respiration.

[0272] 4. Electrocardiogram monitoring device: Hypoglycemia can cause arrhythmia, so the ambulance should be equipped with an electrocardiogram monitoring device to timely detect arrhythmia and take appropriate measures.

[0273] 5. Infusion device: In case of illness, the ambulance should be equipped with an infusion device to administer other medications and treatments.

[0274] All the information obtained (including location coordinate information, which can be specific to the house number, such as from the monitor at No. I) and treatment recommendations are fed back to the elderly room video and emergency video window. Professional emergency doctors inquire and confirm the emergency level and required emergency resources through the voice attached to the monitor. Emergency "link" personnel determine the main diagnosis of diabetes with neurological complications through professional investigation, such as thorough inquiry and judgment, because the patient has symptoms such as urinary incontinence.

[0275] Further complete the classification of emergency situations (such as emergency situations) and the final dispatch of emergency teams.

[0276] Supplement: If the elderly do not have typical symptoms such as fainting, but specific signs of each system of the human body (usually physical examination) can still be detected by computer vision, it can still provide effective information for the next stages of the accompanying robot's work. The specific signs of each system in the clinic include:

[0277] 1. Cardiovascular system: abnormal heart sounds, arrhythmia, pericardial friction rub, heart murmur, edema, etc.

[0278] 2. Respiratory system: abnormal breath sounds, abnormal respiratory rhythm, chest deformity, lung rales, airway obstruction, etc.

[0279] 3. Digestive system: abdominal mass, tenderness, hepatosplenomegaly, cholecystalgia, abnormal bowel sounds, etc.

[0280] 4. Urinary system: low back pain, renal tenderness, urinary obstruction, hematuria, proteinuria, etc.

[0281] 5. Nervous system: neurological dysfunction, sensory impairment, muscle weakness, decreased reflexes, etc.

[0282] 6. Blood system: anemia, bleeding tendency, lymph node enlargement, etc.

[0283] 7. Immune system: rash, joint pain, lymphadenopathy, etc.

[0284] 8. Endocrine system: goiter, acromegaly, diabetic foot, metabolic acidosis, etc.

[0285] 9. Skin and subcutaneous tissue: skin pigmentation, skin herpes, ulcers, nodules, scars, etc.

[0286] Specifically, specific signs are identified mainly through computer vision technology and companion robot voice interaction module. If the patient has obvious discomfort but is conscious, the virtual "attending physician" displayed on the interactive screen of the companion robot and the patient himself / herself determine that the patient has a relapse of a previous disease, such as angina pectoris. The companion robot can actively provide the patient with a pre-stored medicine box (containing nitroglycerin, etc.), and after taking the medicine, the symptoms are relieved. The companion robot provides the patient with authoritative treatment guidelines for angina pectoris and monitors the patient's heart-related indicators, including electrocardiogram, blood pressure, heart rate, etc., to realize the whole-process consultation, monitoring and accompanying of the patient. In addition, the companion robot can also complete the physical examination of the conscious patient through the guidance of the interactive screen, such as displaying a schematic diagram of human body dissection (processed to be acceptable to the public) on the screen, guiding the patient to touch the body parts through a hand mouse, and recording the corresponding conditions through voice interaction system and patient dialogue, so as to perfect the content of virtual case, so that subsequent analysis and recommendation are more accurate and more in line with the situation at that time.

[0287] It is particularly necessary to note that specific signs appearing in each system can be recognized by computer vision, trained by neural network and deep learning, and finally automatically recognized and judged. For example, pale skin and mucosa and nail bed cyanosis in anemia:

[0288] In this case, the recognition object is refined as the lips and the nail bed.

[0289] The identification of the color of the lips can be completed by using neural networks and deep learning.

[0290] The following is a possible implementation step:

[0291] 1. Data collection: Collect a large number of photos of lips of people of different ages, genders and health conditions, and label their corresponding color categories (such as red, light red, purple, etc.).

[0292] 2. Data preprocessing: Preprocess the collected lip photos, including image size normalization, color space conversion, contrast and brightness adjustment, etc.

[0293] 3. Model training: Select an appropriate deep learning model, such as Convolutional Neural Network (CNN) or Autoencoder, and train it using the preprocessed dataset. Some common optimization algorithms can be used during training, such as Gradient Descent (GD), Adam, etc.

[0294] 4. Model testing: Another part of the data is used to test the model performance, techniques such as cross-validation can be used to evaluate the accuracy, recall rate and F1 score of the model.

[0295] 5. Deployment and optimization: The trained model is deployed to practical applications, and some optimization may be needed, such as accelerating inference speed, reducing model size, etc.

[0296] Other similar parts are implemented with similar ideas, such as skin pigmentation, skin herpes, limb edema, painful face, facial flushing, etc.

[0297] Example 9:

[0298] Male, 70 years old, with a history of hypertension, long-term ineffective control, eventually leading to a stroke. After the stroke, Mr. Zhang was sent to the hospital for emergency treatment, and after comprehensive treatment and care by neurologists, his life was saved.

[0299] After a period of rehabilitation treatment, Mr. Zhang's brain function has been restored to some extent, but his body still has some problems, such as mild speech disorder and mild limb movement limitation. In order to further improve the physical condition, Mr. Zhang actively cooperates with the guidance of the rehabilitation physician, and gradually recovers certain limb function and self-care ability in daily life. During the treatment process, Mr. Zhang's family also gave him a lot of support and love, helping him to overcome the difficulties of rehabilitation.

[0300] After discharge, Mr. Zhang took the hospital's advice and used the accompanying robot to assist in controlling hypertension. Mr. Zhang began to pay attention to healthy diet, appropriate exercise, and regular medication according to the doctor's advice, keeping his blood pressure stable.

[0301] The accompanying robot will remind Mr. Zhang to take medication on time and regularly. If the accompanying robot is not with Mr. Zhang, it will send a message to the communication tool (such as a mobile phone) of the accompanying personnel;

[0302] For the initial accompanying personnel (such as family members), the accompanying robot will call the authoritative database in the field of hypertension to complete the knowledge education of family members and Mr. Zhang and answer some accompanying questions; when there are requests beyond the scope of ability, appropriate suggestions will also be given, and the user can schedule time to make an appointment with the nearest attending physician if allowed;

[0303] The accompanying robot can quantify the basic vital signs monitored and draw conclusions, and communicate with the patient's relatives and friends according to the current situation to develop a suitable schedule for visiting, thereby promoting the patient's recovery to the greatest extent and providing psychological support for the patient's needs;

[0304] The accompanying robot can perform the regular follow-up questionnaire of the attending physician, which takes the form of logical matching, that is, the previous question is associated with the next question, and different replies will get different questionnaire results. The analysis results are fed back to the attending physician, and the treatment plan is adjusted by the attending physician. And make a schedule to contact the attending physician and remind when the attending physician is busy or the conditions allow, contact the patient's family doctor, community doctor, etc.

[0305] Later, Mr. Zhang basically recovered completely, and his family returned to their posts. The accompanying robot served as the main companion for Mr. Zhang to go out for exercise, including detecting that there were few people nearby, suitable for slow walking or moderate physical exercise, etc.

[0306] When the patient reaches an agreement with the hospital and the high blood pressure is effectively controlled, the patient can decide to continue to rent the accompanying robot or purchase the accompanying robot or return the accompanying robot.

[0307] The above-mentioned embodiments are only preferred technical solutions of the present application, and should not be regarded as a limitation of the present application. The protection scope of the present application should be based on the technical solutions recited in the claims, including the equivalent replacement solutions of the technical features recited in the claims. That is, within this scope, equivalent replacement improvements are also within the protection scope of the present application.

Claims

1. An intelligent companionship system for the elderly based on computer vision technology, characterized in that: Includes a central control module, an indoor monitoring module, a companion robot module, and a mobile device display terminal; In the indoor monitoring module, multiple integrated cameras are installed in the indoor monitoring system according to the indoor space layout. The total field of view covers all indoor activity locations that need to be monitored. Information from the same source is encapsulated and marked as target person signals. Dynamic tracking of the target person's activities in the video information is achieved through a deep learning-based tracking algorithm, and the information is sent to the central control module. The central control module performs feature analysis based on the collected information, detects and analyzes abnormal states of the target person, and when the video activity analysis results or detection results indicate that the target person's activity is abnormal or that they need help, it transmits a signal to the companion robot module through 5G communication technology. Feature analysis includes one of the following: facial features, gait features, and voiceprint features. Facial analysis and voiceprint analysis compare the collected information with pre-stored information to extract abnormal features; Gait features generate dynamic walking information from video images and compare it with pre-stored information to extract abnormal features; Facial and voiceprint analysis employs a combination of one or more deep learning-based tracking algorithms, such as Siamese networks or region proposal networks; gait feature analysis is obtained through fall detection algorithms and radio spectrum analysis algorithms. The companion robot module is equipped with an intelligent analysis and control system, a voice interaction system, an indoor navigation system, and a vital signs detection system; The intelligent analysis and control system acts as the brain of the care robot. By understanding the data characteristics transmitted from the indoor monitoring system, it infers the possible situations of the elderly, formulates intervention plans through decision trees, and takes action. When the built-in intelligent analysis and control system feels "unconfident" about the intervention plan to be implemented, it will proactively call up video information of the target scene to conduct algorithm detection and investigation of other most likely dangerous situations. At the same time, it will proactively go to the room where the elderly person is located through the indoor navigation system. The "lack of confidence" state is identified by establishing a behavior-outcome analysis model and using artificial intelligence algorithms to infer outcomes from behavior. The preset solution is judged as "lack of confidence" when the probability of the following outcomes exceeds a preset value: It cannot solve the existing problems; The predicted result may lead to harm to the target. The predicted results had no effect.

2. The intelligent companionship system for the elderly based on computer vision technology according to claim 1, characterized in that: The integrated camera has two modes: day and night, which automatically switch according to the ambient light. When the ambient light is higher than the preset value, the visible light camera detects the elderly person's movement information in real time. When the ambient light is lower than the preset value, it switches to the infrared camera to detect the elderly person's movement information in real time.

3. The intelligent companionship system for the elderly based on computer vision technology according to claim 1, characterized in that: The companion robot module also includes an integrated camera, motion system, humanoid robotic arm, compartmentalized storage box, and sensor module; The integrated camera system includes a visual camera and a thermal imaging camera, used to collect environmental information and track the movement of the target person. The motion system is used to control the walking of the companion robot module; Humanoid robotic arms are used to assist target individuals in performing corresponding actions; Partitioned storage boxes are used to store medicines and medical devices; The sensor module is used to collect the physiological characteristics and environmental parameters of the target person.

4. The intelligent companionship system for the elderly based on computer vision technology according to claim 1, characterized in that: In addition to verifying emergency situations, the voice interaction system also includes mechanisms to alleviate loneliness among the elderly and to address issues such as declining eyesight, dexterity, and insufficient knowledge. It has the functions of answering the elderly's questions, playing music, and controlling smart home devices, and can interact with other smart home devices, and can be configured and controlled through a smartphone application; Speech recognition algorithms are used to convert speech signals into text, including end-to-end speech recognition algorithms based on hidden Markov models and deep learning. Natural language processing algorithms are used to transform speech recognition results in text form into understandable semantic information, including syntactic analysis, entity recognition, sentiment analysis, and dialogue management. Using speech synthesis algorithms to convert computer-generated text into speech signals; Algorithms that utilize dialogue management algorithms to control the dialogue flow include dialogue state tracking, policy generation, and response generation; By querying the cloud knowledge base, a hybrid recommendation algorithm is used to provide suggestions and recommendations for the problems encountered by the elderly. After obtaining authorization from the elderly, the system proactively manages smart home devices and other resources available through internet technology. Mobile device display terminals include mobile phones, tablets, smartwatches, laptops, or desktop computers, used to display video information called according to tasks; The companion robot module is equipped with an interactive screen for interaction.

5. The intelligent companionship system for the elderly based on computer vision technology according to claim 1, characterized in that: The indoor navigation system uses image processing and computer vision technology to detect and track objects in videos, identify pets and record their location and movement trajectory. It uses the SLAM algorithm to build a travel map based on object detection and tracking data collected by integrated cameras in multiple locations, and transmits the generated map data to the companion robot in the room via 5G technology. The companion robot uses algorithms to process data, analyze maps, and plan routes. The companion robot uses its onboard camera to acquire image information of the surrounding environment. Through image processing technology, it matches the current position of the companion robot with its position on the map to achieve autonomous positioning. The location of the companion robot is obtained through the positioning information in the map. The companion robot uses a path planning algorithm to plan the optimal route on the map based on its current location and the location of the target point. The companion robot navigates autonomously based on its mobility and task requirements, moving along the path or performing tasks. When the companion robot moves, it uses convolutional neural networks or recurrent neural networks to analyze the data captured by the integrated camera, accurately identify and track different environmental features, and achieve local map optimization. The companion robot uses feedback data from its own sensors to detect ground movement and rotation, and updates the map of its area. The companion robot is controlled by an internal control system to move forward, turn, and stop.

6. The intelligent companionship system for the elderly based on computer vision technology according to claim 1, characterized in that: The vital signs detection system uses pre-installed sensor modules to detect heart rate, respiration, body temperature, and blood pressure. It intelligently analyzes whether the collected vital signs data are within the normal range and the degree of deviation, and promptly transmits early warning information to the mobile device display terminal to alert emergency contacts. Based on the preset triggering procedure and the detection information from the sensor module, the preset emergency plan is activated, which includes delivering medication and contacting emergency numbers.

Images