Intelligent interaction method and device for nursing homes
Through intelligent interaction methods and neural network technology, the visual and hearing impairment and image blurring of the elderly when using intelligent interactive devices are solved, personalized interaction strategies and emergency responses are provided, and the availability and safety of nursing home services are improved.
Patent Information
- Application Number
- CN202410911223.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-09
AI Technical Summary
Existing intelligent interactive devices cannot effectively serve elderly people with low vision, low hearing or low education, and the blurred picture caused by the elderly's shaking and movement when taking images has not been effectively solved.
Using an intelligent interaction method in nursing home, by obtaining the sensing information of the perception device, generating shooting information and passing it to the camera device, using a trained repair neural network to extract facial image features, perform similarity calculations to judge interaction strategies, and Image repair is performed in the multi-level fusion module, and interaction strategies are selected based on the environmental brightness information, voice or text interaction is provided, falling state is recognized in a timely manner and alarm information is generated.
Provide each elderly person with personalized interaction strategies, improve the user experience of visually impaired and hearing-impaired elderly people, reduce image blur problems, timely identify and deal with critical situations of the elderly, and improve the efficiency and safety of nursing home services.
Smart Images

Figure CN118470290B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an intelligent interaction method and an intelligent interaction device for a nursing home. Background Art
[0002] The elderly in existing nursing homes usually have various underlying diseases. For example, some elderly people have presbyopia or cataracts, which lead to low vision. Another example is that some elderly people have hearing degeneration, which leads to low hearing. And some elderly people may have a low level of education and cannot read normally due to their old age.
[0003] Nowadays, smart interactive devices have become the main equipment for nursing home services in the later stage. However, most elderly people with special conditions (such as low vision, low hearing or low cultural level) are unable to use smart interactive devices normally, making it impossible for smart interactive devices to serve the elderly.
[0004] In addition, when performing facial recognition on the elderly through interactive devices, since most of the elderly have low endurance, the image may be blurred due to shaking or movement of the elderly when taking images.
[0005] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned defects of the prior art. Summary of the invention
[0006] The purpose of the present application is to provide an intelligent interaction method for nursing homes to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.
[0007] To achieve the above objectives, the present application provides an intelligent interaction method for a nursing home, and the intelligent interaction method for a nursing home includes:
[0008] Obtain the sensing information transmitted by the sensing device;
[0009] Generate shooting information according to the sensing information and transmit it to the camera device;
[0010] Acquire an initial captured image transmitted by the camera device according to the capturing information;
[0011] Acquire an initial face image from the initial captured image;
[0012] Acquire a trained repair neural network, wherein the repair neural network includes an encoder and a decoder, and the decoder includes a multi-stage fusion module;
[0013] Inputting the initial face image into the encoder to extract features of the face image;
[0014] Inputting the facial image features into the decoder to obtain a restored facial image;
[0015] Acquire a preset face database, wherein the preset face database includes at least one preset face image;
[0016] The restored face image is similar to each of the preset face images to determine whether a similarity value exceeds a preset threshold. If so,
[0017] Acquire an interaction strategy library, wherein the interaction strategy library includes at least one preset face image and an interaction strategy group, and one preset face image is associated with one interaction strategy group;
[0018] Obtain an interaction strategy group corresponding to a preset face image whose similarity value exceeds a preset threshold.
[0019] Optionally, the multi-stage fusion module includes:
[0020] A pooling and splicing module, wherein the pooling and splicing module is used to perform pooling and splicing on the facial image features provided by the encoder to form a first feature;
[0021] A first processing module, the first processing module is used to perform a first processing on the first feature, so as to obtain a second feature;
[0022] a second processing module, the second processing module being used for performing a second processing on the first feature, thereby forming a third feature;
[0023] a third processing module, configured to add the second feature to the third feature and perform a convolution operation to obtain a fourth feature;
[0024] A linear transformation module, wherein the linear transformation module is used to perform a linear transformation on the fourth feature, thereby obtaining a linear transformation feature;
[0025] An activation feature module, wherein the activation feature module is used to perform Sigmoid activation on the fourth feature, thereby obtaining an activation feature;
[0026] A multiplication module, wherein the multiplication module is used to multiply the linear transformation feature and the activation feature to obtain a fifth feature;
[0027] An adding module, wherein the adding module is used to add the fifth feature to the fourth feature to obtain a final fusion feature.
[0028] Optionally, the first processing module includes:
[0029] An average pooling module, wherein the average pooling module is used to perform average pooling on the first feature, thereby obtaining an average pooling feature;
[0030] A maximum pooling module, wherein the maximum pooling module is used to perform maximum pooling on the first feature, thereby obtaining a maximum pooling feature;
[0031] A first splicing module, wherein the first splicing module is used to splice the average pooling feature and the maximum pooling feature to obtain a first splicing feature;
[0032] A position attention mechanism module, wherein the position attention mechanism module is used to perform a position self-attention operation on the first splicing feature;
[0033] An upsampling module is used to upsample the first splicing feature that performs the position self-attention operation, so as to obtain a second feature.
[0034] Optionally, the second processing module includes:
[0035] an anti-overflow module, the anti-overflow module being used to perform an anti-overflow process on the first feature;
[0036] A channel attention mechanism module, wherein the channel attention mechanism module is used to perform a channel self-attention operation on the first feature that has been processed with anti-overflow, so as to obtain a third feature.
[0037] Optionally, each of the interaction strategy groups includes a dim environment interaction strategy and a clear environment interaction strategy, and the intelligent interaction method for a nursing home further includes:
[0038] Get the ambient brightness information in the room;
[0039] Determine whether the ambient brightness information in the room exceeds a preset brightness threshold, if not,
[0040] Selecting the dim environment interaction strategy;
[0041] Determine whether the ambient brightness information in the room exceeds a preset brightness threshold, and if so,
[0042] The clear environment interaction strategy is selected.
[0043] Optionally, when the interaction strategy group is a dim environment interaction strategy, the intelligent interaction method for a nursing home further comprises:
[0044] Obtain basic user information pre-stored in the dim environment interaction strategy;
[0045] Determine whether the user can communicate in a dim environment based on the user's basic information. If not,
[0046] Generate auxiliary lighting information and send it to the auxiliary lighting device.
[0047] Optionally, the intelligent interaction method for a nursing home further comprises:
[0048] Extracting an image of a human figure part in the image according to the acquired initial captured image;
[0049] Get the trained behavior classifier;
[0050] Extracting posture features of posture information of the image of the human figure part;
[0051] The posture feature is input into the behavior classifier to obtain a behavior classification label, wherein the behavior classification label includes a fall label. When the classification label is a fall label, it is determined that the human figure in the image is in a fall state.
[0052] Optionally, when the human figure in the image is in a fallen state and the similarity between the restored face image and each of the preset face images is calculated, and it is determined that a similarity value of a similarity exceeds a preset threshold, the intelligent interaction method for the nursing home further includes:
[0053] Acquire an emergency response strategy library, wherein the emergency response strategy library includes at least one preset face image and basic information of a user corresponding to each face image;
[0054] Obtain interaction methods based on user basic information;
[0055] Generate interactive actions using the acquired interactive methods;
[0056] After a preset time, determine whether someone has fed back interaction information based on the interaction action, and if so,
[0057] Analyze the interaction information to determine whether help is needed from others. If so,
[0058] Determine whether the help information can be obtained according to the interaction information, and if so,
[0059] Generate alarm information and send the alarm information and help information to the main control console.
[0060] Optionally, the intelligent interaction method for a nursing home further comprises:
[0061] After a preset time, determine whether someone has fed back interaction information based on the interaction action. If not,
[0062] Generate alarm information and send the alarm information and user basic information to the main console.
[0063] The present application also provides an intelligent interactive device for a nursing home, and the intelligent interactive method for a nursing home further includes:
[0064] A sensing information acquisition module, which is used to acquire sensing information transmitted by a sensing device;
[0065] A shooting information generation module, the shooting information generation module is used to generate shooting information according to the sensing information and transmit it to the camera device;
[0066] A shooting information transmission module, the shooting information transmission module is used to obtain an initial shooting image transmitted by the camera device according to the shooting information;
[0067] An initial face image acquisition module, the initial face image acquisition module is used to acquire an initial face image from the initial captured image;
[0068] A repair neural network acquisition module, the repair neural network acquisition module is used to acquire a trained repair neural network, the repair neural network includes an encoder and a decoder, and the decoder includes a multi-stage fusion module;
[0069] A facial image feature acquisition module, the facial image feature acquisition module is used to input the initial facial image into the encoder, so as to extract facial image features;
[0070] A repaired face image acquisition module, the repaired face image acquisition module is used to input the face image features into the multi-stage fusion module, so as to obtain a repaired face image;
[0071] A preset face database acquisition module, the preset face database acquisition module is used to acquire a preset face database, the preset face database includes at least one preset face image;
[0072] A similarity comparison module, the similarity comparison module is used to calculate the similarity between the restored face image and each of the preset face images, so as to determine whether a similarity value exceeds a preset threshold;
[0073] An interaction strategy library acquisition module, wherein the interaction strategy library acquisition module is used to acquire an interaction strategy library when the similarity comparison module determines that the result is yes, wherein the interaction strategy library includes at least one preset face image and an interaction strategy group, and one preset face image is associated with one interaction strategy group;
[0074] The interaction strategy group acquisition module is used to acquire the interaction strategy group corresponding to the preset face image whose similarity value exceeds the preset threshold.
[0075] The nursing home of the present application uses an intelligent interaction method to obtain the information of the elderly through image recognition, and reads different interaction strategies according to the different information of the elderly, so as to provide a suitable interaction strategy for each elderly person, such as providing text interaction strategies for deaf-mute elderly people, and providing voice interaction strategies for visually impaired elderly people, etc. In addition, the present application repairs facial images by setting up a multi-level fusion module, thereby preventing the problem of blurred facial images caused by shaking of the elderly. BRIEF DESCRIPTION OF THE DRAWINGS
[0076] Figure 1 It is a flowchart of an intelligent interaction method for nursing homes according to an embodiment of the present application.
[0077] Figure 2 It is a schematic diagram of the structure of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION
[0078] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.
[0079] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present application.
[0080] Figure 1 It is a flowchart of an intelligent interaction method for nursing homes according to an embodiment of the present application.
[0081] like Figure 1 The smart interaction methods shown for nursing homes include:
[0082] Step 1: Obtain the sensing information transmitted by the sensing device;
[0083] Step 2: Generate shooting information based on the sensing information and transmit it to the camera device;
[0084] Step 3: obtaining an initial captured image transmitted by the camera device according to the captured information;
[0085] Step 4: Obtaining an initial face image from the initial captured image;
[0086] Step 5: Obtain a trained repair neural network, wherein the repair neural network includes an encoder and a decoder, and the decoder includes a multi-stage fusion module;
[0087] Step 6: Inputting the initial face image into the encoder to extract face image features;
[0088] Step 7: Inputting the facial image features into the decoder to obtain a restored facial image;
[0089] Step 8: Acquire a preset face database, wherein the preset face database includes at least one preset face image;
[0090] Step 9: Calculate the similarity between the restored face image and each of the preset face images to determine whether a similarity value exceeds a preset threshold. If so,
[0091] Step 10: Acquire an interaction strategy library, wherein the interaction strategy library includes at least one preset face image and an interaction strategy group, and one preset face image is associated with one interaction strategy group;
[0092] Step 11: Obtain the interaction strategy group corresponding to the preset face image whose similarity value exceeds the preset threshold.
[0093] The nursing home of the present application uses an intelligent interaction method to obtain the information of the elderly through image recognition, and reads different interaction strategies according to the different information of the elderly, so as to provide a suitable interaction strategy for each elderly person, such as providing text interaction strategies for deaf-mute elderly people, and providing voice interaction strategies for visually impaired elderly people, etc. In addition, the present application repairs facial images by setting up a multi-level fusion module, thereby preventing the problem of blurred facial images caused by shaking of the elderly.
[0094] In this embodiment, if the initial face image cannot be acquired in the initial captured image, prompt information is generated, and the prompt information may be voice prompt information and / or text prompt information.
[0095] In this embodiment, the present application adopts a GAN network. The GAN network in this embodiment includes an encoder and a decoder, wherein the decoder further includes a multi-stage fusion module.
[0096] In this embodiment, the initial face image enters the encoder to obtain face image features, and the face image features are input into the decoder. After the fusion features are generated in the multi-level fusion module in the decoder, the fusion features are input into the fully connected layer to obtain the repaired face image.
[0097] In this embodiment, the multi-stage fusion module includes a pooling splicing module, a first processing module, a second processing module, a third processing module, a linear transformation module, an activation feature module, a multiplication module and an addition module, wherein:
[0098] The pooling and splicing module is used to pool and splice the facial image features provided by the encoder to form a first feature;
[0099] The first processing module is used for performing a first processing on the first feature, so as to obtain a second feature;
[0100] The second processing module is used for performing a second processing on the first feature to form a third feature;
[0101] The third processing module is used to add the second feature and the third feature and perform a convolution operation to obtain a fourth feature;
[0102] The linear transformation module is used to perform a linear transformation on the fourth feature, thereby obtaining a linear transformation feature;
[0103] The activation feature module is used to perform Sigmoid activation on the fourth feature, so as to obtain the activation feature;
[0104] The multiplication module is used to multiply the linear transformation feature with the activation feature to obtain a fifth feature;
[0105] The adding module is used to add the fifth feature to the fourth feature to obtain a final fusion feature.
[0106] In this embodiment, the first processing module includes an average pooling module, a maximum pooling module, a first splicing module, a position attention mechanism module and an upsampling module, wherein:
[0107] The average pooling module is used to perform average pooling on the first feature, so as to obtain an average pooling feature;
[0108] The maximum pooling module is used to perform maximum pooling on the first feature, so as to obtain the maximum pooling feature;
[0109] The first splicing module is used to splice the average pooling feature and the maximum pooling feature to obtain a first splicing feature;
[0110] The position attention mechanism module is used to perform position self-attention operation on the first splicing feature;
[0111] The upsampling module is used to upsample the first splicing feature that performs the position self-attention operation to obtain the second feature.
[0112] In this embodiment, the second processing module includes an anti-overflow module and a channel attention mechanism module, wherein:
[0113] The overflow prevention module is used to perform overflow prevention processing on the first feature;
[0114] The channel attention mechanism module is used to perform channel self-attention operation on the first feature after anti-overflow processing to obtain the third feature.
[0115] The multi-level fusion module of the present application is adopted, and an upsampling module is added after the position self-attention operation through parallel average pooling and maximum pooling operations and serial channel dimension Concat splicing operations; an anti-overflow module operation is added in front of the Softmax activation function of the channel self-attention operation, and the minimum value of the activation value is limited to zero to enhance the features, thereby solving the problem that the position attention mechanism has a large amount of calculation due to the large image size and the nonlinear activation of the channel attention mechanism deviates from the sensitive area. And through the multi-level fusion module, the multi-resolution convolution features extracted from the low-definition input image and the generative face prior features extracted from the generator are fused, which can better balance the restoration of the face area and the image background, fully restore the details of the original low-definition image, and maintain the fidelity of the face.
[0116] In this embodiment, after acquiring the sensing information transmitted by the sensing device, the method of the present application further includes:
[0117] Get the surrounding brightness information;
[0118] Determine whether the ambient brightness information exceeds a preset brightness threshold, if not,
[0119] First fill light information is generated and sent to a first fill light device, so that the first fill light device adjusts the brightness according to the first fill light information.
[0120] In this embodiment, the first fill light device is a fill light device built into the intelligent interactive device, that is, the first fill light device will not illuminate the entire room, but will only illuminate the area in front of the intelligent interactive device, that is, the area captured by the camera device.
[0121] In this embodiment, each of the interaction strategy groups includes a dim environment interaction strategy and a clear environment interaction strategy, and the intelligent interaction method for the nursing home further includes:
[0122] Get the ambient brightness information in the room;
[0123] Determine whether the ambient brightness information in the room exceeds a preset brightness threshold, if not,
[0124] Selecting the dim environment interaction strategy;
[0125] Determine whether the ambient brightness information in the room exceeds a preset brightness threshold, and if so,
[0126] The clear environment interaction strategy is selected.
[0127] In a scenario, a user may use an interactive device in the dark or in dim light (for example, the curtains are completely or partially drawn). At this time, although the human body image may be captured by the first fill light device, the surrounding environment may still be dim, and this dimness may cause some adverse effects, such as damaging the user's eyesight, especially when the user's eyesight is poor, the surrounding environment is dim, and only the fluorescent screen is bright, it will be more difficult to see things clearly. Therefore, by adopting different interaction strategies in different situations, it is possible to distinguish the way to treat users in dim environments and clear environments.
[0128] In this embodiment, when the interaction strategy group is a dim environment interaction strategy, the intelligent interaction method for the nursing home further includes:
[0129] Obtain basic user information pre-stored in the dim environment interaction strategy;
[0130] Determine whether the user can communicate in a dim environment based on the user's basic information. If not,
[0131] Determine whether to generate auxiliary lighting information based on the ambient brightness information. If so,
[0132] Generate auxiliary lighting information and send it to the auxiliary lighting device.
[0133] In a dim environment, if the user uses voice interaction, the actual environment does not affect the user's interaction. Therefore, the pre-stored basic information of the user can be used to understand whether the user can use voice and other environments that do not require human vision.
[0134] In this embodiment, the perception device may be a sensor, for example, a foot-kick AR sensor commonly used in vehicles.
[0135] In this embodiment, a preset face image is associated with an interaction strategy group. For example, a user named Zhang San is provided with a preset face image and an interaction strategy group. The interaction strategy group may have multiple interaction strategies. For example, in one embodiment, Zhang San's interaction strategy group includes a dim environment interaction strategy and a clear environment interaction strategy. In this embodiment, both the dim environment interaction strategy and the clear environment interaction strategy pre-store Zhang San's user basic information. The user basic information may include the following:
[0136] Zhang San's interaction preferences (for example, voice interaction or text interaction);
[0137] Zhang San's physical condition (e.g., deafness, poor eyesight, difficulty moving, etc.);
[0138] Zhang San's underlying medical conditions (e.g., heart disease, cerebral thrombosis, etc.);
[0139] Zhang San's language preference (e.g., Mandarin, Cantonese).
[0140] In this embodiment, the specific interactive content of the interactive device of the present application when the user interacts is not within the protection scope of the present application, and therefore, it will not be described in detail here.
[0141] In this embodiment, judging whether the user can communicate in a dim environment according to the basic information of the user may be performed in the following manner:
[0142] Example 1: Determine whether Zhang San is deaf based on his physical condition. If not, it is considered that he can communicate through voice, and is judged to be able to communicate in a dim environment.
[0143] Example 2: Determine whether Zhang San likes voice interaction based on interaction preferences. If so, it is considered that he can communicate through voice, and it is determined that he can communicate in a dim environment.
[0144] It is understandable that other judgment conditions may also be set, which will not be described in detail here.
[0145] In this embodiment, if it is determined that the user is unable to communicate in a dim environment, auxiliary lighting information is generated and sent to the auxiliary lighting device.
[0146] In this embodiment, the auxiliary lighting device here can be the light of the room itself or other auxiliary lighting device that can illuminate the room and is set in the room and is connected to the communication of this application.
[0147] In this embodiment, the intelligent interaction method for a nursing home further includes:
[0148] Extracting an image of a human figure part in the image according to the acquired initial captured image;
[0149] Get the trained behavior classifier;
[0150] Extracting posture features of posture information of the image of the human figure part;
[0151] The posture feature is input into the behavior classifier to obtain a behavior classification label, wherein the behavior classification label includes a fall label. When the classification label is a fall label, it is determined that the human figure in the image is in a fall state.
[0152] When the human figure in the image is in a fallen state and the similarity between the restored face image and each of the preset face images is calculated, and it is determined that a similarity value of a similarity exceeds a preset threshold, the intelligent interaction method for the nursing home further includes:
[0153] Acquire an emergency response strategy library, wherein the emergency response strategy library includes at least one preset face image and basic information of a user corresponding to each face image;
[0154] Obtain interaction methods based on user basic information;
[0155] Generate interactive actions using the acquired interactive methods;
[0156] After a preset time, determine whether someone has fed back interaction information based on the interaction action, and if so,
[0157] Analyze the interaction information to determine whether help is needed from others. If so,
[0158] Determine whether the help information can be obtained according to the interaction information, and if so,
[0159] Generate alarm information and send the alarm information and help information to the main control console.
[0160] In some embodiments, the reason why the user wakes up the interactive device may be because the user needs help, such as when the user suffers from a sudden illness (fall, fracture, cerebral thrombosis, myocardial infarction, etc.). At this time, the user may be able to perform simple operations (wake up the smart interactive device), but may not be able to communicate much, for example, he or she may not be able to speak a lot or interact in text.
[0161] At this time, the user's status can be identified through image recognition. If the user is judged to have fallen through the classification label, the interaction method is obtained according to the user's basic information. For example, if the user is judged to be deaf according to the user's basic information, the interaction action is generated by text interaction, that is, the text is displayed on the screen. At this time, the database used for the interaction is the emergency response strategy library. The interaction strategies stored in the emergency response strategy library are all simple and urgent strategies. For example, in the interaction process, whether it is voice interaction or text interaction, multiple-choice questions or question-and-answer questions are used. For example, in one example, the voice interaction is: Do you need to call the police? Or do you need to notify the management staff? In this way, the user only needs to say yes or no.
[0162] Alternatively, several selection boxes can be directly displayed in text format: 1. Call the police; 2. First aid; 3. Notify the management staff.
[0163] This method allows users to interact in the simplest way.
[0164] In this embodiment, if the user can provide feedback based on the interactive action, that is, feedback interactive information, it means that the user can also perform some simple communication. At this time, interactive actions can be generated, such as asking about the condition, asking about the situation, and other voice or text information. At this time, if the user can also provide feedback on the condition, rest, and situation information, these condition information or situation information will be sent to the main console as help information.
[0165] If you cannot obtain this help information, you can also send the alarm information directly to the main console.
[0166] In some cases, even the simplest method may not be able to achieve interaction between the user and the interactive device. For example, if the user suddenly has asthma, he may not be able to speak at all. At this time, if voice interaction is used, the user may not be able to provide any feedback. In this case, the method of the present application further includes:
[0167] After a preset time, determine whether someone has fed back interaction information based on the interaction action. If not,
[0168] Generate alarm information and send the alarm information and user basic information to the main console.
[0169] In this way, the safety of users can be guaranteed to the greatest extent.
[0170] The present application also provides an intelligent interactive device for a nursing home, and the intelligent interactive method for a nursing home further includes a sensing information acquisition module, a shooting information generation module, a shooting information transmission module, an initial face image acquisition module, a repair neural network acquisition module, a face image feature acquisition module, a repair face image acquisition module, a preset face database acquisition module, a similarity comparison module, an interaction strategy library acquisition module, and an interaction strategy group acquisition module, wherein:
[0171] The sensing information acquisition module is used to acquire the sensing information transmitted by the sensing device;
[0172] The shooting information generation module is used to generate shooting information according to the sensing information and transmit it to the camera device;
[0173] The shooting information transmission module is used to obtain the initial shooting image transmitted by the camera device according to the shooting information;
[0174] The initial face image acquisition module is used to acquire the initial face image from the initial captured image;
[0175] The repair neural network acquisition module is used to acquire a trained repair neural network, wherein the repair neural network includes an encoder and a decoder, and the decoder includes a multi-stage fusion module;
[0176] The facial image feature acquisition module is used to input the initial facial image into the encoder to extract facial image features;
[0177] The repaired face image acquisition module is used to input the face image features into the multi-stage fusion module, so as to obtain the repaired face image;
[0178] The preset face database acquisition module is used to acquire a preset face database, wherein the preset face database includes at least one preset face image;
[0179] The similarity comparison module is used to calculate the similarity between the restored face image and each of the preset face images, so as to determine whether a similarity value of a similarity exceeds a preset threshold;
[0180] The interaction strategy library acquisition module is used to acquire the interaction strategy library when the similarity comparison module determines that it is yes, the interaction strategy library includes at least one preset face image and an interaction strategy group, and one preset face image is associated with one interaction strategy group;
[0181] The interaction strategy group acquisition module is used to acquire the interaction strategy group corresponding to the preset face image whose similarity value exceeds the preset threshold.
[0182] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.
[0183] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above-mentioned intelligent interaction method for nursing homes is implemented.
[0184] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned intelligent interaction method for nursing homes.
[0185] Figure 2 It is an exemplary structural diagram of an electronic device that can implement the intelligent interaction method for nursing homes provided according to an embodiment of the present application.
[0186] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.
[0187] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1 Described nursing home using intelligent interaction methods.
[0188] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program code; and one or more processors 503 configured to run the executable program code stored in the memory 504 to execute the intelligent interaction method for nursing homes in the above-mentioned embodiment.
[0189] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0190] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0191] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0192] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0193] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.
[0194] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0195] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0196] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.
[0197] Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0198] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An intelligent interaction method for a nursing home, characterized in that: The intelligent interaction method for nursing homes includes: Obtain the sensing information transmitted by the sensing device; Generate shooting information according to the sensing information and transmit it to the camera device; Acquire an initial captured image transmitted by the camera device according to the capturing information; Acquire an initial face image from the initial captured image; Acquire a trained repair neural network, wherein the repair neural network includes an encoder and a decoder, and the decoder includes a multi-stage fusion module; Inputting the initial face image into the encoder to extract features of the face image; Inputting the facial image features into the decoder to obtain a restored facial image; Acquire a preset face database, wherein the preset face database includes at least one preset face image; The restored face image is similar to each of the preset face images to determine whether a similarity value exceeds a preset threshold. If so, Acquire an interaction strategy library, wherein the interaction strategy library includes at least one preset face image and an interaction strategy group, and one preset face image is associated with one interaction strategy group; Obtaining an interaction strategy group corresponding to a preset face image whose similarity value exceeds a preset threshold; The intelligent interaction method for nursing homes further comprises: Extracting an image of a human figure part in the image according to the acquired initial captured image; Get the trained behavior classifier; Extracting posture features of posture information of the image of the human figure part; Inputting the posture feature into the behavior classifier to obtain a behavior classification label, wherein the behavior classification label includes a fall label, and when the classification label is a fall label, determining that the human figure in the image is in a fall state; When the human figure in the image is in a fallen state and the similarity between the restored face image and each of the preset face images is calculated, and it is determined that a similarity value of a similarity exceeds a preset threshold, the intelligent interaction method for the nursing home further includes: Acquire an emergency response strategy library, wherein the emergency response strategy library includes at least one preset face image and basic information of a user corresponding to each face image; Obtain interaction methods based on user basic information; Generate interactive actions using the acquired interactive methods; After a preset time, determine whether someone has fed back interaction information based on the interaction action, and if so, Analyze the interaction information to determine whether help is needed from others. If so, Determine whether the help information can be obtained according to the interaction information, and if so, Generate alarm information and send the alarm information and help information to the main control console.
2. The intelligent interaction method for nursing homes according to claim 1, characterized in that: The multi-stage fusion module includes: A pooling and splicing module, wherein the pooling and splicing module is used to perform pooling and splicing on the facial image features provided by the encoder to form a first feature; A first processing module, the first processing module is used to perform a first processing on the first feature, so as to obtain a second feature; a second processing module, the second processing module being used for performing a second processing on the first feature, thereby forming a third feature; a third processing module, configured to add the second feature to the third feature and perform a convolution operation to obtain a fourth feature; A linear transformation module, wherein the linear transformation module is used to perform a linear transformation on the fourth feature, thereby obtaining a linear transformation feature; An activation feature module, wherein the activation feature module is used to perform Sigmoid activation on the fourth feature, thereby obtaining an activation feature; A multiplication module, wherein the multiplication module is used to multiply the linear transformation feature and the activation feature to obtain a fifth feature; An adding module, the adding module is used to add the fifth feature and the fourth feature to obtain a final fusion feature; The first processing module comprises: An average pooling module, wherein the average pooling module is used to perform average pooling on the first feature, thereby obtaining an average pooling feature; A maximum pooling module, wherein the maximum pooling module is used to perform maximum pooling on the first feature, thereby obtaining a maximum pooling feature; A first splicing module, wherein the first splicing module is used to splice the average pooling feature and the maximum pooling feature to obtain a first splicing feature; A position attention mechanism module, wherein the position attention mechanism module is used to perform a position self-attention operation on the first splicing feature; An upsampling module, the upsampling module is used to upsample the first splicing feature that performs the position self-attention operation, so as to obtain a second feature; The second processing module comprises: an anti-overflow module, the anti-overflow module being used to perform an anti-overflow process on the first feature; A channel attention mechanism module, wherein the channel attention mechanism module is used to perform a channel self-attention operation on the first feature that has been processed with anti-overflow, so as to obtain a third feature.
3. The intelligent interaction method for nursing homes according to claim 1, characterized in that: Each of the interaction strategy groups includes a dim environment interaction strategy and a clear environment interaction strategy, and the intelligent interaction method for the nursing home further includes: Get the ambient brightness information in the room; Determine whether the ambient brightness information in the room exceeds a preset brightness threshold, if not, Selecting the dim environment interaction strategy; Determine whether the ambient brightness information in the room exceeds a preset brightness threshold, and if so, The clear environment interaction strategy is selected.
4. The intelligent interaction method for nursing homes as claimed in claim 3, characterized in that: When the interaction strategy group is a dim environment interaction strategy, the intelligent interaction method for the nursing home further comprises: Obtain basic user information pre-stored in the dim environment interaction strategy; Determine whether the user can communicate in a dim environment based on the user's basic information. If not, Generate auxiliary lighting information and send it to the auxiliary lighting device.
5. The intelligent interaction method for nursing homes according to claim 4, characterized in that: The intelligent interaction method for nursing homes further comprises: After a preset time, determine whether someone has fed back interaction information based on the interaction action. If not, Generate alarm information and send the alarm information and user basic information to the main console.
6. An intelligent interactive device for a nursing home, used to implement the intelligent interactive method for a nursing home as claimed in any one of claims 1 to 5, characterized in that: The intelligent interactive device for nursing homes further comprises: A sensing information acquisition module, which is used to acquire sensing information transmitted by a sensing device; A shooting information generation module, the shooting information generation module is used to generate shooting information according to the sensing information and transmit it to the camera device; A shooting information transmission module, the shooting information transmission module is used to obtain an initial shooting image transmitted by the camera device according to the shooting information; An initial face image acquisition module, the initial face image acquisition module is used to acquire an initial face image from the initial captured image; A repair neural network acquisition module, the repair neural network acquisition module is used to acquire a trained repair neural network, the repair neural network includes an encoder and a decoder, and the decoder includes a multi-stage fusion module; A facial image feature acquisition module, the facial image feature acquisition module is used to input the initial facial image into the encoder, so as to extract facial image features; A repaired face image acquisition module, the repaired face image acquisition module is used to input the face image features into the multi-stage fusion module, so as to obtain a repaired face image; A preset face database acquisition module, the preset face database acquisition module is used to acquire a preset face database, the preset face database includes at least one preset face image; A similarity comparison module, the similarity comparison module is used to calculate the similarity between the restored face image and each of the preset face images, so as to determine whether a similarity value exceeds a preset threshold; An interaction strategy library acquisition module, wherein the interaction strategy library acquisition module is used to acquire an interaction strategy library when the similarity comparison module determines that the result is yes, wherein the interaction strategy library includes at least one preset face image and an interaction strategy group, and one preset face image is associated with one interaction strategy group; The interaction strategy group acquisition module is used to acquire the interaction strategy group corresponding to the preset face image whose similarity value exceeds the preset threshold.
