Multi-sensory integrated AI and VR nursing method, device, medium and program product

Abstract

The invention provides an AI and VR nursing method and device integrating multiple senses, a medium and a program product, and relates to the field of intelligent medical treatment. The method comprises the steps that a first instruction for entering a virtual scene is received, and a subject enters the virtual scene to generate a virtual subject; the virtual scene comprises a foretaste area; receiving a second instruction that the virtual subject moves to the foretaste area in the virtual scene and moving the virtual subject to the foretaste area; receiving an instruction for a fake feeding object placed in the foretaste area, wherein the fake feeding object appears in the hand of the virtual subject; the category of the fake feeding object is recognized, and according to whether the distance between the fake feeding object and the mouth of the virtual subject in the virtual scene exceeds a threshold value or not or whether a feeding instruction is received or not, the smell spraying device receives a smell simulation instruction of the fake feeding object and starts the device to generate corresponding smell. A voice control module of the false feeding device is developed through AI, and a sniffing intention recognition algorithm is developed, so that accurate analysis and conversion of signals are realized, and the equipment intelligence is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical care, and more specifically, to a multi-sensory integrated AI and VR nursing method, device, medium and program product. Background Art

[0002] The prevalence of malnutrition in surgical operations remains high. Reduced food intake caused by disease, preoperative preparations such as fasting before surgery, inability to eat normally for a long time after surgery, surgical trauma stress, gastrointestinal insufficiency, and adverse reactions of various treatments can all cause increased body catabolism and self-tissue consumption, resulting in malnutrition, damage to the body's tissue and organ functions, increase the incidence of complications and mortality, and affect the patient's clinical outcomes and quality of life. Nutritional therapy is an important means for accelerated recovery surgery to promote safe and rapid recovery of perioperative patients. The guidelines recommend that enteral nutrition should be the first choice for trauma patients with existing or partial gastrointestinal function and certain absorption function, and for patients who still cannot take in enough nutrition through oral nutritional preparations, tube feeding enteral nutrition is the first choice. Enteral nutrition can provide key macro- and micronutrients for individuals who cannot meet the oral nutritional requirements, protect the gastric mucosal barrier, and reduce the incidence of enterogenic infection. However, the commonly used tube feeding enteral nutrition supply method in clinical practice lacks the stimulation of food color, aroma and taste, and it is difficult to effectively promote the secretion of digestive juices and gastrointestinal motility, which affects the patient's recovery effect.

[0003] Sham feeding refers to the phenomenon that after food enters the mouth, even if it does not reach the stomach, the color, aroma and other sensory stimulations of the food stimulate the vagus nerve in the brain and then stimulate the stomach and intestines to secrete gastric juice. The ancient Chinese allusion of "looking at plums to quench thirst" suggests that the conditioned reflex of sour plums is used to stimulate the secretion of saliva. The research team found in the early animal experiments on traumatic SD rats that multi-sensory stimulation assisted enteral nutrition can improve the structure of intestinal flora and reduce serum inflammatory factors. Based on this, the research team innovatively proposed a new concept of "sham feeding", that is, simulating a normal eating state, stimulating brain neurons through integrated visual, auditory and olfactory sensory stimulation, and solving the problem that surgical patients' fasting parenteral nutrition and tube feeding enteral nutrition supply methods lack color, aroma and taste stimulation, and cannot effectively mobilize brain neurons to induce gastrointestinal related physiological reactions and functional expressions. Based on the integration of medicine and engineering, the team has developed a virtual reality (VR) sham feeding device with integrated visual, auditory and olfactory stimulation in the early stage. At present, the device has been authorized with a national invention patent (patent number: ZL 202010753133.1). The device provides a panoramic dining scene and chewing, swallowing and environmental sound effects through the immersive display and built-in speakers of the VR headset set, enabling multi-angle viewing of food and simulating the dynamic presentation of food when it is clamped. Patients can pull the trigger to achieve the displacement of the dining scene and press the call bell in the scene to call the waiter for interaction. Patients can immersively experience the audio-visual and olfactory stimulation close to real dining in a virtual environment, and preliminary clinical trials have verified good results.

[0004] However, the team found in the experiment that the VR sham feeding device and nursing technology in the existing patents have the following problems in application and promotion: First, the patient has limited activities and significantly reduced self-care ability due to body damage and medical catheter placement after surgery, which makes it inconvenient to operate the handle with both hands, but the existing VR sham feeding device must use the handle to press buttons and three-dimensional space control, and the operation process is cumbersome and error-prone; second, the VR sham feeding device still faces challenges in simulating the real food experience, especially the lack of accuracy and delay gap of the odor spray function based on food recognition, and the patient needs to manually click the button to trigger the spray of specific food odors during sham feeding treatment. The above-mentioned current problems reduce the immersive experience of patients during sham feeding, affect the patient's compliance with sham feeding treatment, and thus affect the quality and effectiveness of the patient's postoperative digestive function rehabilitation. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides an AI and VR nursing method, device, medium and program product that integrates multiple senses; the method of the present invention develops a voice control module of a VR sham feeding device through AI, improves the intelligence, convenience, autonomy and question-answer interactivity of the VR sham feeding device, so that patients do not need to operate with both hands, but only need to control the device through simple voice commands, solving the problem that patients with restricted populations cannot smoothly use the handle to press buttons and operate the VR sham feeding device in three-dimensional space. And by developing an AI algorithm for sniffing intention recognition, the accurate analysis and conversion of the device signal is realized, so that when the user clamps the food to the mouth through the handle, the device can automatically identify and trigger the food sniffing accurate identification odor spraying device to accurately spray the corresponding odor, solve the problem of insufficient accuracy of the odor spraying function and the existence of delay gap, thereby improving the pleasure of use, and is expected to enhance the use effect of the VR sham feeding integrated device, so that patients with inconvenient operation can fully enjoy the treatment of the VR sham feeding device. It is of practical significance to solve the current application defects of the VR sham feeding integrated device through AI empowerment.

[0006] The first aspect of the present application discloses a multi-sensory integrated AI and VR nursing method, the method comprising: S101, receiving a first instruction to enter a virtual scene, and the subject enters the virtual scene to generate a virtual subject; the virtual scene includes a tasting area; S102, receiving a second instruction for the virtual subject to move to a tasting area in the virtual scene, and the virtual subject moves to the tasting area; at least one fake feeding object is placed in the tasting area; S103, receiving an instruction for a fake feeding object, the fake feeding object appears in the hand of the virtual subject; identifying the category of the fake feeding object, the virtual subject gradually brings the fake feeding object closer to the mouth of the virtual subject, when the distance between the fake feeding object and the mouth of the virtual subject in the virtual scene exceeds a threshold or a feeding instruction is received, the odor spraying device receives an odor simulation instruction of the fake feeding object, starts the odor spraying device and generates a corresponding odor according to the category of the fake feeding object; In some embodiments, if there is only one fake feeding object, the odor spraying device receives a command to simulate the odor of one fake feeding object, and releases a single odor corresponding to the fake feeding object through the movement of the structure in the odor spraying device; Optionally, if the fake feeding objects include at least two kinds, the odor spraying device receives odor simulation commands of at least two kinds of fake feeding objects, and releases the mixed odors corresponding to the fake feeding objects through the movement of the structure in the odor spraying device; Optionally, the distance between the fake feeding object and the virtual subject's mouth in the virtual scene is the physical distance between the external handle and the subject's mouth (i.e., the VR helmet worn by the subject).

[0007] In some embodiments, the virtual scene includes any one or more of the following: a virtual Chinese restaurant scene, a virtual fruit store scene, and a virtual pastry shop scene; each virtual scene also includes a visiting area; Optionally, after S101 and before S102, the method further includes: receiving instructions for the virtual subject to move to different positions in the virtual scene, and displaying corresponding fake feeding objects at different positions; receiving instructions for selecting different fake feeding objects in the virtual scene, and obtaining the selected fake feeding objects; the selected fake feeding objects will appear in the tasting area; Optionally, the instruction may be in the form of voice or external handle button. If the instruction is in the form of voice, the voice is recognized and analyzed to generate an instruction for a corresponding operation, and the virtual scene is switched.

[0008] In some embodiments, the method further comprises: if there is a fake feeding object that needs to be added during the tasting process in the tasting area, receiving an instruction to add the fake feeding object in the virtual scene to obtain the selected fake feeding object; Optionally, the type of fake fed object is identified to give a result on whether tableware is needed. If tableware is not needed, the virtual subject can directly pick it up (such as a drink, click on the drink to directly pick up the cup). If tableware is needed, tableware will appear in the left or right hand of the virtual subject when picking up the fake fed object.

[0009] In some embodiments, between S102 and S103, the method further includes S1021, determining whether the perspective of the virtual subject is parallel to the ground in the virtual scene, and if so, executing step S103; if not, giving a prompt to switch the perspective, and after the perspective is switched, the perspective of the virtual subject is parallel to the ground in the virtual scene, and then executing step S103; Optionally, the tasting area includes a turntable table; between S1021 and S103, the method further includes: receiving an instruction for a fake feeding object in the tasting area, and changing the fake feeding object in front of the virtual subject; when the tasting area is a turntable table, changing the fake feeding object in front of the subject by rotating the turntable; In some embodiments, the odor spraying device includes: a shell, a odor box assembly, an odor box inner sealing plate assembly, and an odor box top cover (ABS); a placement space is reserved inside the shell, and the odor box inner sealing plate assembly is placed in the placement space; the first side of the shell is movably connected to the odor box top cover, and an opening is reserved on the second side of the shell for the odor box assembly to extend into, and the odor box assembly is movably installed in the odor box inner sealing plate assembly along the opening; the odor box assembly includes: at least one cotton swab soaked in liquids with different odors, a buffer member arranged at the first end of the cotton swab, and a first protective shell, and the first protective shell is provided inside the first protective shell. The first receiving groove for receiving a single cotton swab and the second receiving groove for receiving a single buffer are provided at least one at the first receiving groove; the other end of the buffer is connected to the first protective shell; the cotton swab extends out of the opening of the first receiving groove, and a sealing ring is provided at the position corresponding to the single cotton swab at the opening end of the first receiving groove; the extended cotton swab is connected to the sealing plate assembly inside the odor box; the sealing plate assembly inside the odor box includes an oscillation wave plate, and the vibration of the ultrasonic oscillation wave plate atomizes the liquid in the cotton swab into mist and releases it; the odor box is equipped with a cotton swab filling hole and an oscillation wave plate, etc., to ensure that the odor can be released evenly and effectively; The odor box inner sealing plate assembly also includes: a second protective shell and a bottom plate connected, the upper end of the second protective shell is provided with an opening for the first protective shell to extend into, and the opening of the second protective shell is consistent in size with the opening of the outer shell; Optionally, the shock wave plate is arranged on a bottom plate, and the bottom plate is provided with a spray hole for releasing mist; Optionally, the number of the cotton swabs, sealing rings, and shock wave plates corresponds one to one; Optionally, the buffer member is configured as a spring, the upper end of the spring is connected to the inside of the first protective shell, and the lower end of the spring is connected to the upper end of the cotton swab.

[0010] In some embodiments, the method further comprises: obtaining background knowledge of the subject's medical records, and providing nutritional matching suggestions based on the background knowledge of the medical records; Optionally, the method further includes: obtaining the subject's dining intention, and recommending a matching menu according to the dining intention; Optionally, the method further comprises: detecting a virtual environment usage scenario set by the subject, determining a current environment type according to the light and noise levels of the environment, and adjusting the usage scenario in the virtual scene according to the current environment type; Optionally, the method further includes: obtaining a physiological reaction of the user, and adjusting the sensory experience according to the physiological reaction; the physiological reaction includes: heart rate and electromyography.

[0011] The second aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store a computer program; and the processor executes the computer program to implement the steps of the above method.

[0012] A third aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0013] A fourth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0014] This application has the following beneficial effects: 1. Develop a voice interaction module for an artificial feeding intelligent nursing device integrating AI+VR; In order to further solve the limitations of limited patient activities, inconvenient operation and insufficient device interactivity, this project intends to add an intelligent voice interaction module based on the VR fake feeding device that integrates audio, visual and olfactory senses. This module adopts a dynamic maximum matching algorithm based on dictionary improvement. By constructing a domain dictionary, dynamically selecting the maximum matching length and the priority matching principle, it can effectively reduce invalid matching and ambiguity, and improve the efficiency and accuracy of the algorithm. On this basis, a simple startup interface is designed to guide users to start the VR program. In the VR environment, an easily recognizable AI assistant icon is designed, and users can activate the AI ​​assistant through voice or visual prompts. A dialogue interface is designed to display the interaction history and current dialogue content between the user and the AI ​​assistant. Functions such as voice recognition, voice question and answer, and voice synthesis are realized, so that patients can control the device through simple voice commands, communicate with the device in real time, and improve the system's question and answer interactivity.

[0015] 2. Develop an accurate odor emission algorithm for the artificial feeding intelligent nursing device integrating AI+VR; In order to achieve the goal that when the user picks up food to the mouth through the handle, the device can intelligently identify and trigger the olfactory smell box to accurately spray the corresponding smell, the following R&D steps are systematically carried out. First, start to design and make the olfactory smell box. Use ZW software for 3D modeling and print out the physical object. The olfactory smell box will be equipped with key components such as cotton swabs to fill holes and oscillation wave plates to ensure that the smell can be released evenly and effectively. Connect the olfactory smell box to the computer through a gateway device to achieve real-time signal transmission. Then write a signal conversion program to ensure that the patient voice interaction issues a command to eat a certain food, or when the handle is close to the head display device within 32 cm (the specific threshold is not limited to 32 cm), the olfactory smell box can synchronously receive the signal and trigger the spray of the smell. Finally, the written odor precise spray AI algorithm will be used to perfectly integrate the virtual and real worlds. It is planned that through this technology, patients can not only taste food in virtual reality, but also smell the corresponding smell of real food. During use, VR devices will help patients easily switch between various scenes and seamlessly experience different foods, thereby completing the nutritional supply process under nearly realistic visual, auditory and olfactory stimulation. The completion of this research and development step will greatly enhance the user experience of VR fake feeding equipment, making it closer to the sensory experience of the real world.

[0016] 3. Form the operation technology of artificial feeding intelligent nursing device integrating AI+VR; Based on the AI+VR sham feeding intelligent nursing device with integrated vision, hearing and smell, we develop AI algorithms for dining intention matching, food picking intention recognition, health consultation Q&A and environmental interaction for patients who are fasting or receiving enteral and parenteral nutrition alone, and develop related supporting software and technologies (such as AI+VR interactive scenes, feedback animation technology, related function libraries, etc.), present specific scenes in a targeted manner, so that patients can interact with humans and machines in virtual scenes and choose food independently through AI intelligent voice dialogue. At the same time, it assists visual, auditory and olfactory stimulation, so that patients who are fasting or receiving enteral and parenteral nutrition alone can deeply experience the simulated state of normal life and eating through sensory stimulation with high motion feedback, thereby promoting the secretion of gastrointestinal digestive juices in the body and promoting the early recovery of gastrointestinal function in patients. At the same time, it changes the traditional patient education model, increases the fun and immersion of health guidance through VR technology, improves self-nutrition management capabilities, and improves the quality of life. Based on this technology, a randomized controlled trial was conducted to study the effects of the panoramic nutrition management system under the intervention of AI+VR technology on the intestinal flora balance, inflammatory immune function and gastrointestinal function recovery of patients who were fasting or receiving simple enteral and parenteral nutrition, and transformation was carried out on the basis of proven effectiveness.

[0017] Compared with the existing patents studied by the team in the early stage: This invention uses AI empowerment to design a matching algorithm for intelligent voice interaction, add an intelligent voice control module, develop an algorithm for accurate AI odor spraying, increase the intelligent human-computer interaction experience, and innovate equipment functions. Provide convenient voice-controlled sham feeding care for patients who do not eat orally and have limited activities, and shorten the duration of the patient's non-physiological state. Change the current situation that the commonly used parenteral and enteral nutrition supply methods in clinical practice lack the sensory stimulation of food color, aroma and taste, which makes it difficult to mobilize related physiological reactions and functional expressions, affects the normal secretion of digestive glands and gastrointestinal motility, and thus affects the patient's rehabilitation effect. Provide more effective and convenient nutritional treatment for patients who do not eat orally. At the same time, the research and development of sham feeding intelligent devices and nursing technologies that integrate AI+VR can not only help patients recover safely, but also effectively improve the quality and efficiency of clinical nurses' care. The following defects of previous studies are well resolved: the operation process is cumbersome and error-prone due to the limited activity of patients after surgery and significantly reduced self-care ability; there are challenges in simulating the experience of real things, especially the lack of accuracy and delay of the odor emission function based on food recognition, and the need to manually click a button to trigger the emission of specific food odors during sham feeding treatment; and the dishes of the technical solution disclosed in the patent are unprocessed products (the dishes ordered in the present invention are mature menus), which mainly increase the results of the display screen, and do not involve changes in sitting posture, external auxiliary modules, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic diagram of a method flow chart provided by the first aspect of an embodiment of the present invention; Figure 2 is a schematic diagram of an integrated multi-sensory AI and VR nursing system provided by a second aspect of an embodiment of the present invention; Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention; Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention; Figure 5 is a schematic diagram of a storage medium provided by an embodiment of the present invention; Figure 6 It is a technical roadmap for the research and development of a voice interaction module for a nursing device provided by an embodiment of the present invention; Figure 7 It is a technical roadmap for the research and development of nursing device operation technology provided by an embodiment of the present invention; Figure 8 is a flow chart of a multi-sensory fusion algorithm provided by an embodiment of the present invention; Fig. 9 It is an exploded schematic diagram of the overall structure of the odor spraying device provided by an embodiment of the present invention; Fig.10 It is an enlarged schematic diagram of the inner sealing plate assembly of the odor box provided by an embodiment of the present invention.

[0020] In the figure, 1. odor box assembly; 11. cotton swab; 12. spring; 13. first protective shell; 14. sealing ring; 2. outer shell; 3. main board; 4. battery; 5. odor box inner sealing plate assembly; 51. second protective shell; 52. oscillation wave plate; 53. bottom plate; 6. odor box top cover; 7. rubber plug; 8. Velcro; 9. cross recessed countersunk flat tail self-tapping screw; 10. cross recessed pan head flat tail self-tapping screw. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0022] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] Figure 1 1 is a flow chart of a multi-sensory integrated AI and VR nursing method provided by an embodiment of the present invention. Specifically, the method comprises the following steps: S101: receiving a first instruction to enter a virtual scene, and the subject enters the virtual scene to generate a virtual subject; the virtual scene includes a tasting area; In some embodiments, the term "subject" or "person to be tested" or "sample to be tested" as used herein refers to any animal (e.g., mammal), including but not limited to humans, non-human primates, rodents, etc., which will be the recipient of a specific treatment. Generally, the terms "subject" and "patient" are used interchangeably herein when referring to a human subject. Preferably, the subject is a human.

[0025] In some embodiments, the virtual scene includes any one or more of the following: a virtual Chinese restaurant scene, a virtual fruit store scene, and a virtual pastry shop scene; each virtual scene also includes a visiting area; Optionally, after S101 and before S102, the method further includes: receiving an instruction to move a virtual subject to different positions in the virtual scene, and displaying corresponding sham objects at different positions; first observing the environment, using a handle to move to different positions in the visiting area, and displaying different sham objects in front of different positions; receiving an instruction to select different sham objects in the virtual scene, and obtaining the selected sham objects; the selected sham objects will appear in the tasting area; wherein the sham object selection instruction can be voice, or a menu can be called up by touching a bell in the virtual scene, and the subject can independently select the sham objects appearing on the menu through an external handle or other device; medical staff can also assist in the operation of selecting the required sham objects on a computer connected to the VR device. The menu of the virtual Chinese restaurant scene includes two categories: signature Chinese food (cold tofu, Xiangxi bacon, spicy crayfish, maocai, sugar-coated fruits, scallion pancakes, etc.) and special foods. The menu can be switched according to needs, and the selected fake objects will appear in the tasting area; in the virtual fruit store scene, different fake objects can be directly taken in the visiting area, or different fake objects can be selected and the selected fake objects will appear in the tasting area; the menu of the virtual pastry shop scene includes desserts (tiramisu, strawberry ice cream, cake, whole wheat biscuits, etc.) and drinks (cappuccino, milk, barley tea, etc.). The menu can be switched according to needs, and the selected fake objects will appear in the tasting area.

[0026] S102: receiving a second instruction for the virtual subject to move to a tasting area in the virtual scene, and the virtual subject moves to the tasting area; at least one fake feeding object is placed in the tasting area; In some embodiments, between S102 and S103, the method further includes S1021, determining whether the perspective of the virtual subject is parallel to the ground in the virtual scene, if so, executing step S103; if not, giving a prompt to switch the perspective, after switching the perspective, the perspective of the virtual subject is parallel to the ground in the virtual scene, and then executing step S103; specifically, the visual switching can be a subject or external auxiliary operation, and the visual switching includes two modes: default and ground parallel. After clicking, the VR perspective can be switched. In the default state, the user can sit or stand; the ground parallel mode makes the direction of the virtual helmet parallel to the ground, which is suitable for lying down or lying on your back; the external assistance module includes switching perspectives and quick transmission, and the switching perspective includes two modes: default / ground parallel. After clicking and lighting up, the perspective in VR is switched. The default is sitting or standing. The ground parallel mode is to make the virtual direction of the current helmet parallel to the ground, which is used when lying down or lying on your back. Quick transmission is an action in which an external person clicks on different transmission points to transmit the virtual player to the front of the food when the patient is not familiar with VR movement operations. The initial point is not around the food. The teleportation points include 1 / 2 / 3 / 4, each corresponding to a different location.

[0027] In some embodiments, the tasting area includes a turntable table; between S1021 and S103, the method further includes: receiving instructions for fake feeding objects in the tasting area, and changing the fake feeding objects in front of the virtual subject; when the tasting area is a turntable table, changing the fake feeding objects in front of the subject by rotating the turntable; autonomous operation, the subject uses a handle to touch the turntable, clicks and holds it, and moves the handle left and right to rotate the turntable.

[0028] S103: receiving an instruction for a fake feeding object, the fake feeding object appears in the hand of the virtual subject; identifying the category of the fake feeding object, the virtual subject gradually brings the fake feeding object closer to the mouth of the virtual subject, when the distance between the fake feeding object and the mouth of the virtual subject in the virtual scene exceeds a threshold or receiving an eating instruction, the odor spraying device receives an odor simulation instruction of the fake feeding object, starts the odor spraying device and generates a corresponding odor according to the category of the fake feeding object; In some embodiments, when the patient issues a command to eat a certain food through voice interaction, or when the handle is close to the VR helmet within 32 cm, the odor box receives the signal synchronously, which is the recognition of sniffing intention. The system transmits the electrical signal of the odor simulation command to the odor spraying device, which parses the command to identify the odor type and converts the electrical signal into an oscillation signal, and executes the corresponding odor simulation stimulation (transmitted to the ultrasonic chip in the odor spraying device, which atomizes the odor into tiny odor particles of 1 to 5 μm through 2.4 MHz high-frequency oscillation; In some embodiments, in some embodiments, if the fake feeding object is one kind, the odor spraying device receives the odor simulation command of one fake feeding object, and releases the single odor corresponding to the fake feeding object through the movement of the structure in the odor spraying device; optionally, if the fake feeding object includes at least two kinds, the odor spraying device receives the odor simulation command of at least two fake feeding objects, and releases the mixed odor corresponding to the fake feeding object through the movement of the structure in the odor spraying device; Optionally, the distance between the fake feeding object and the virtual subject's mouth in the virtual scene is the physical distance between the external handle and the subject's mouth (i.e., the VR helmet worn by the subject).

[0029] In some embodiments, the instructions are issued by the wearer or medical staff; if it is in the form of an external handle button, the purpose of issuing instructions is achieved by long pressing or short pressing different operation keys of the handle; the index finger position of the external handle is provided with a trigger key or operation key. When the hand of the virtual subject touches an object in the virtual scene, the object will be highlighted. The form of highlighting includes but is not limited to luminescence, magnification, etc. At this time, press the trigger key to confirm the picking up or dragging of the object. The instruction can be an autonomous operation or an auxiliary operation. Autonomous operation: the patient uses the handle to click the seat transfer point to sit down; auxiliary operation: the medical staff clicks the pre-made transfer point to transfer the virtual subject to different fake feeding objects.

[0030] In some embodiments, the form of the instruction includes: voice form, external handle button form. If the form of the instruction is voice, the voice is recognized and analyzed to generate the corresponding operation instruction, and the virtual scene is switched. The voice form adopts a dynamic maximum matching algorithm based on dictionary improvement. By constructing a domain dictionary, dynamically selecting the maximum matching length and the priority matching principle, it can effectively reduce invalid matching and ambiguity, and improve the efficiency and accuracy of the algorithm. A simple startup interface can be designed to guide the start of the VR program, such as an easily recognizable AI assistant icon, which can activate the AI ​​assistant through voice or visual prompts, display the user's interaction history and current conversation content with the AI ​​assistant, and realize voice recognition, voice question and answer, voice synthesis and other functions.

[0031] In some embodiments, the method further includes: if there is a fake feeding object that one wants to add during the tasting process in the tasting area, receiving an instruction to add the fake feeding object in the virtual scene to obtain the selected fake feeding object; the operator issues the instruction by approaching the virtual waiter in the virtual scene or clicking on the bell pop-up menu of the table in the tasting area; if there is a dessert and / or drink that one wants to add during the tasting process, one can approach the virtual waiter in the virtual scene or click on the bell pop-up menu of the table in the tasting area.

[0032] Optionally, the type of the fake object is identified to give a result on whether tableware is needed. If tableware is not needed, the virtual subject can pick it up directly (such as for drinks, click on the drink to directly pick up the cup). If tableware is needed, tableware will appear in the left or right hand of the virtual subject when picking up the fake object; tableware includes: chopsticks (for Chinese food, etc.) or spoons (for cakes, etc.).

[0033] In some embodiments, Fig. 9 and 10 As shown, the odor spraying device includes: a shell 2, a odor box assembly 1, an odor box inner sealing plate assembly 5, and a odor box top cover 6 (ABS); a placement space is reserved inside the shell 2, and the odor box inner sealing plate assembly 5 is placed in the placement space; the first side of the shell 2 is movably connected to the odor box top cover 6, and an opening is reserved on the second side of the shell 2 for the odor box assembly 1 to extend into, and the odor box assembly 1 is movably installed in the odor box inner sealing plate assembly 5 along the opening; the odor box assembly 1 includes: at least one cotton swab 11 soaked in liquids with different odors, a buffer member arranged at the first end of the cotton swab 11 and a first protective shell 13, and a sachet 13 is arranged inside the first protective shell 13 At least one first receiving groove for receiving a single cotton swab 11 and at least one second receiving groove for receiving a single buffer; the other end of the buffer is connected to the first protective shell 13; the cotton swab 11 extends out of the opening of the first receiving groove, and a sealing ring 14 is arranged at the position corresponding to the single cotton swab 11 at the opening end of the first receiving groove; the extended cotton swab 11 is connected to the sealing plate assembly 5 in the odor box; the sealing plate assembly 5 in the odor box includes an oscillation wave plate 52, and the vibration of the ultrasonic oscillation wave plate 52 atomizes the liquid in the cotton swab 11 into mist and releases it; the odor box is equipped with a cotton swab 11 filling hole and an oscillation wave plate, etc., to ensure that the odor can be released evenly and effectively; In some embodiments, Fig.10 The figure shows an enlarged schematic diagram of the odor box inner sealing plate assembly 5, which includes: the odor box inner sealing plate assembly 5 also includes: a second protective shell 51 and a bottom plate 53 connected, the upper end of the second protective shell 51 is provided with an opening for the first protective shell 13 to extend into, and the opening of the second protective shell 51 is consistent in size with the opening of the outer shell 2; Optionally, the oscillating wave plate 52 is disposed on a bottom plate 53, and the bottom plate 53 is provided with a spray hole for releasing mist; Optionally, the number of the cotton swab 11, the sealing ring 14, and the oscillation wave plate 52 corresponds one to one; Optionally, the buffer member is configured as a spring 12 , the upper end of the spring 12 is connected to the interior of the first protective shell 13 , and the lower end of the spring 12 is connected to the upper end of the cotton swab 11 .

[0034] In some embodiments, fixed structures are respectively arranged at corresponding positions on both sides of the odor box assembly 1 and on both sides of the odor box inner sealing plate assembly 5; the fixed structures include: snap structures, sliding structures, hook structures; the fixed structures are not limited here, and can be snap structures, sliding structures, hook structures, etc. that can achieve the fixing effect.

[0035] In some embodiments, the odor box inner sealing plate assembly 5 is connected to the outer shell 2 through a fixing member; in a more specific embodiment, the fixing member includes a screw, and the fixing member at the connection between the odor box inner sealing plate assembly 5 and the outer shell 2 is two cross-slot countersunk flat-tail self-tapping screws 9, which are made of stainless steel; In some embodiments, a fixing is provided at the connection between the first side of the outer shell 2 and the top cover 6 of the odor box; in a more specific embodiment, the fixing includes a screw, and the fixing at the connection between the first side of the outer shell 2 and the top cover 6 of the odor box is 7 cross-slot pan head flat tail self-tapping screws 10, which are made of stainless steel.

[0036] In some embodiments, a connector for connecting an interactive device is provided on the outside of the scent box top cover 6; the connector is configured as a 3M Velcro 8. In this embodiment, the interactive device is an XR head display device, and the function of the 3M Velcro 8 is to connect the XR head display device.

[0037] In some embodiments, decorative silicone rubber plugs 7 are respectively provided at the corners of the top cover 6 of the odor box.

[0038] In some embodiments, the placement space is provided with: a mainboard 3 (VRE100 mainboard), a battery 4; the components provided on the mainboard 3 include: a control chip, a CPU, a memory, a hard disk, and an odor detection sensor.

[0039] In some embodiments, the method further comprises: obtaining medical record background knowledge of the subject in a knowledge base, and providing nutritional matching suggestions based on the medical record background knowledge; Optionally, the method further includes: obtaining the subject's dining intention, and recommending a matching menu according to the dining intention; Optionally, the method further includes: detecting the virtual environment usage scenario set by the subject, judging the current environment type according to the light and noise levels of the environment, and adjusting the usage scenario in the virtual scene according to the current environment type; specifically, the usage scenario or environment type includes daytime, nighttime, quiet environment, noisy environment, etc.; specifically, if it is daytime, adjusting the brightness of the display device, if it is nighttime, adjusting the contrast of the display device, if it is a quiet environment, maintaining normal auditory output, if it is a noisy environment, enhancing the auditory noise reduction function, and synchronizing other sensory outputs to maintain consistency; optionally, the method further includes: collecting user feedback on the adjusted sensory parameters, that is, whether the user is satisfied, if satisfied, maintaining the output of the current sensory parameters, if not satisfied, analyzing the feedback information and adjusting the strategy, and adjusting the sensory parameters according to the strategy to optimize the user experience; Optionally, the method further includes: obtaining a physiological reaction of the user, and adjusting the sensory experience according to the physiological reaction; the physiological reaction includes: heart rate and electromyography.

[0040] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 3 As shown, the device 2000 may include: one or more processors 2010, and one or more memories 2020; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0041] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0042] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0043] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0044] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5 As shown, it is a schematic diagram of a storage medium 4000 provided in an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by a processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0045] The embodiments of the present disclosure further provide a computer program product or system, including a computer program, which implements the steps of the above method when executed by a processor.

[0046] In some embodiments, this embodiment also discloses an integrated multi-sensory AI and VR nursing system, such as Figure 2 As shown, the system comprises: The virtual subject generation module 201 is used or configured to receive a first instruction to enter a virtual scene, and the subject enters the virtual scene to generate a virtual subject; the virtual scene includes a tasting area; The virtual scene moving module 202 is used or configured to receive a second instruction for the virtual subject to move to a tasting area in the virtual scene, and the virtual subject moves to the tasting area; at least one fake feeding object is placed in the tasting area; The sniffing intention recognition module 203 is used or configured to receive instructions for a fake feeding object, the fake feeding object appears in the hands of the virtual subject; identify the category of the fake feeding object, the virtual subject gradually brings the fake feeding object close to the mouth of the virtual subject, and when the distance between the fake feeding object and the mouth of the virtual subject in the virtual scene exceeds a threshold or an eating instruction is received, the odor spraying device receives an odor simulation instruction of the fake feeding object, starts the odor spraying device, and generates a corresponding odor according to the category of the fake feeding object; In some embodiments, the system further includes: a fake feeding object selection module located between 201 and 202, which is used or configured to receive instructions for moving the virtual subject to different positions in the virtual scene, and display the corresponding fake feeding objects at different positions; first observe the environment, use the handle to move to different positions in the visiting area, and display different fake feeding objects in front of different positions; receive instructions for selecting different fake feeding objects in the virtual scene, and obtain the selected fake feeding objects; the selected fake feeding objects will appear in the tasting area; In some embodiments, the system further includes: a switching vision module located between 202 and 203, used for or configured to determine whether the perspective of the virtual subject is parallel to the ground in the virtual scene, if so, execute 203; if not, give a prompt to switch the perspective, after which the perspective of the virtual subject is parallel to the ground in the virtual scene, and then execute 203; a fake object adjustment module, used for or configured to receive instructions for fake objects in the tasting area, and change the fake objects in front of the virtual subject; when the tasting area is a turntable table, change the fake objects in front of the subject by rotating the turntable.

[0047] In some embodiments, the system further includes the following modules, which can be activated at any time during use: A dining intention acquisition module, used for or configured to acquire the dining intention of the subject and recommend a matching menu according to the dining intention; A usage scenario adjustment module is used or configured to detect a virtual environment usage scenario set by a subject, determine a current environment type according to the light and noise levels of the environment, and adjust the usage scenario in the virtual scene according to the current environment type; The sensory parameter adjustment module is used or configured to: collect user feedback on the adjusted sensory parameters, that is, whether the user is satisfied, if satisfied, maintain the output of the current sensory parameters, if not satisfied, analyze the feedback information and adjust the strategy, and adjust the sensory parameters according to the strategy to optimize the user experience; The physiological response acquisition module is used or configured to acquire the user's physiological response and adjust the sensory experience according to the physiological response; the physiological response includes: heart rate and electromyography. Specific embodiment: Sham feeding operation steps: After the patient with enteral nutrition intolerance underwent VR pre-use nursing assessment and signed the informed consent form, VR sham feeding was used to assist enteral nutrition, three times a day, 20 minutes each time, at 7:30-7:50; 12:00-12:20; 18:00-18:20. The specific operation process is as follows: 1.1 Patient assessment: Check the patient's name, bed number, and hospitalization number; Assess the patient's physiological indicators, level of consciousness, whether the surgical incision is bleeding, etc., to ensure that the patient is in a stable and comfortable state and is qualified to use VR equipment; Assess the patient's cognitive ability and education level to determine whether they are able to operate and use VR equipment; Ensure that the patient or their legal guardian has signed a consent form for the use of VR equipment; Before use, introduce yourself to the patient and explain the purpose and precautions of the equipment.

[0049] 1.2 Preparation before use: a. Odor box: Add the corresponding odor liquid according to the name of the odor box, check whether the power of the sprayer is sufficient, and press the switch to start the spray. Only one sprayer can be turned on at a time, and the indicator light will be green after it is turned on. b. Network and connection: Connect the computer to the Wifi6 router (with or without network), find the Wifi connection in the VR helmet, and connect to the Wifi6 router at the same time. c. Open the Steam software on the computer, wait for the link to fail, and then select offline mode to start. d. Wirelessly connect to the computer. e. Run the enterprise connection in the wireless VR helmet application in offline state and wait for the device to search. After searching for the device, the gray two icon link status indicates that the link is successful. f. After the link is successful, click the VRE-100 icon on the desktop to enter the fake feeding software interface. g. Check whether the power of each device is sufficient and whether it can be used normally. Set the autonomous operating system to off, and then push the device to the patient's bedside.

[0050] 1.3 Steps for using the device: a. Assist the patient to wear the VR helmet and olfactory device, grab the handle, and instruct the patient to use the handle correctly; b. Use the VR device to enter the virtual food court scene, use the handle to control the patient to walk in the food court, and choose a Chinese restaurant / fruit shop or dessert shop; during use, the system displays the operation interface of 3 virtual scenes; c. Enter the virtual restaurant scene, use the handle to click on the seat for the patient to sit down; d. The following are the operation procedures for the three scenes: (1) Chinese restaurants: Ordering: The Chinese restaurant menu is divided into two types: VR menu and external menu (after turning off the autonomous operation button, the VR menu will not pop up when entering the scene). The menu includes two categories: signature Chinese food and special cuisine. You can click to switch. Click the "+" sign to add a dish and the "-" sign to reduce a dish. You can order up to three servings of each dish and up to seven servings of all dishes. After clicking the Submit button, the corresponding food will be displayed on the table. The menu will pop up after approaching the waiter or clicking the bell on the table.

[0051] Eating: After ordering, walk to the table, pull the trigger to pick up the chopsticks, or pick up the food for the first time, and the chopsticks will automatically appear in your hand. Use the chopsticks to touch the food again, and the food will show a yellow prompt. Pull the trigger to pick up the food, and release the trigger to put the food down. Bring the food close to the helmet, the spray device starts to simulate eating and release the corresponding Chinese food smell. During use, the turntable desktop interface of the virtual scene tasting area of ​​the Chinese restaurant is displayed.

[0052] (2) Fruit shop: In the fruit shop, patients do not need to check the menu. The external assistance system will help patients move to the tasting position. There are three rows of fruit racks in the tasting area. Patients only need to pull the trigger to grab the fruit and bring it close to the helmet to start simulating eating and release the corresponding fruit smell. During use, a schematic diagram of the virtual scene interface of the fruit shop is displayed.

[0053] (3) Bakery: Ordering: There are two forms of pastry shop menus, VR menu and external menu (VR menu will not pop up after turning off the autonomous operation button and entering the scene). The menu includes two categories: desserts and drinks, which can be switched by clicking. Click the "+" sign to add a dish, and the "-" sign to reduce a dish. After clicking the Submit button, the corresponding food will be displayed on the table. Approach the waiter or click the bell on the table to make the menu pop up.

[0054] Eating: divided into drinks and desserts. When grabbing food, some food can be taken directly, and for the other part, a spoon will be added to the hand when it is first taken. It will automatically switch when grabbing different foods. When the food is close to the helmet, the spray device will start to simulate eating and release the corresponding drink or dessert smell. During use, the turntable desktop interface of the virtual scene tasting area of ​​the pastry shop; The main operating components of the device are: trigger button (located at the index finger position). When the virtual hand touches an object, the object will light up. Pressing the trigger button can confirm picking up or dragging the object. Additional auxiliary functions include: (requires external operators to be familiar with) Press the external auxiliary button to pop up the external auxiliary interface, and click again to hide the interface; in the external auxiliary interface, the VR operation guide interface can be turned on or off; the operation guide interface provides basic voice tutorial pictures of the VR handle; the perspective switching includes two modes: default and parallel to the ground. Click to switch the VR perspective. In the default state, the user can sit or stand. The parallel to the ground mode makes the direction of the virtual helmet parallel to the ground, which is suitable for use when lying down or lying on your back; the fast transmission function allows external personnel to teleport the virtual player to the front of the food by clicking different transmission points when the patient is not familiar with VR movement operations. The initial point is not around the food.

[0055] The device uses a human-computer interaction mode. The operator can communicate and interact with the virtual waiter through the head-mounted device. The patient uses the handle to click on the menu and select the dishes he likes in the scene. The user needs to continue the experience until the end of the meal.

[0056] Equipment maintenance: The intervention time was recorded and the patient was asked if he / she felt any discomfort; Arrange VR equipment: take out the scent box, disinfect the VR mask part with alcohol spray, and charge the scent box and VR.

[0057] After using the device, please turn off the power and charge it, disinfect it with alcohol wipes, and replace the consumed virtual scent cartridges.

[0058] 1.4 Post-operation evaluation: Evaluation time: after using VR virtual feeding assistance; 1.4.1 Post-operation equipment-related nursing assessment: assess the patient's vital signs, state of consciousness, whether the surgical wound is bleeding, etc., to ensure that the patient is safe and comfortable and able to wear and use VR; ask the patient how they feel after using VR virtual feeding and whether they experience adverse reactions such as dizziness, vomiting, and nausea.

[0059] 1.4.2 Post-operative disease-related nursing assessment: Assess the patient's enteral nutrition intolerance, including the degree and frequency of abdominal distension, nausea, vomiting and diarrhea; re-evaluate the patient's vital signs and consciousness state to see if they are normal, and whether there is any bleeding or exudate from the surgical wound; check that the patient's postoperative gastrointestinal decompression is within the normal range and the gastric tube is in the stomach, and confirm that the patient is in a safe and comfortable state.

[0060] 2.1 Key technologies of voice interaction module of artificial feeding intelligent nursing device integrating AI+VR: Users activate AI assistants through voice, and VR program converts voice into text and sends it to general conversation AI. General conversation AI determines user intention, obtains relevant information from the knowledge base, and generates answers. VR program converts answers into audio streams and performs dialogue response actions through virtual characters. The key technologies of this part are as follows: 2.1.1 Speech recognition and conversion: It is necessary to accurately recognize the user's voice commands and be able to handle different accents and speaking speeds.

[0061] 2.1.2 Knowledge base integration: AI assistants need to be able to access the knowledge base to obtain background knowledge of medical records and nutritional matching recommendations.

[0062] 2.1.3 User Intent Determination: General conversational AI needs to be able to accurately determine the user’s intent and provide corresponding feedback.

[0063] 2.1.4 Dish operation guidance: The AI ​​assistant needs to be able to guide users to perform operations such as rotating and picking up dishes.

[0064] 2.1.5 Conversation interface design: It is necessary to design a conversation interface that is easy to understand and operate, and display the interaction history and current conversation content.

[0065] 2.1.6 Audio stream playback: The VR program needs to be able to convert text responses into audio streams and drive the virtual character to perform corresponding dialogue response actions.

[0066] 2.2 Key technologies of precise odor emission algorithm for artificial feeding intelligent nursing device integrating AI+VR: Through the olfactory odor box, gateway and simulated food odor perfume combined with AI emission algorithm, precise odor emission of artificial feeding intelligent nursing device integrating AI+VR is realized. Key technologies of olfactory odor box include: 2.2.1 Sniffing intention recognition: when the user picks up food through the handle and brings it to his mouth, the olfactory odor box sprays the corresponding odor.

[0067] 2.2.2 Signal analysis and conversion: When the handle is 32 cm close to the head-mounted display device, the olfactory smell box converts the electrical signal that controls the smell spray into an oscillation signal and transmits it to the ultrasonic chip in the smell box; 2.2.3 Ultrasonic chip high-frequency atomization: The ultrasonic chip atomizes the odor into tiny odor particles of 1 to 5 μm through 2.4 MHz high-frequency oscillation; 2.2.4 Virtual reality scene linkage: Combining the virtual and the real, allowing users to taste food in the virtual reality world and smell the corresponding real food odors. During use, VR can help patients easily switch between various scenes, seamlessly experience different foods, and complete the nutritional supply process under nearly real audio-visual and olfactory stimulation.

[0068] 2.3 Key technologies of operation technology of artificial feeding intelligent nursing device integrating AI+VR: Based on the user intention judgment of the general conversation AI function, more tasks are expanded to greatly improve the applicability of intelligent fake feeding care products. The response intention includes dining intention, food picking action, health consultation, environmental interaction and emotional communication. Based on this, personalized services suitable for different patients can be built. The key technologies of this part are as follows: 2.3.1 Dining intention: The AI ​​assistant will recommend menus that match the user’s health status and nutritional recommendations.

[0069] 2.3.2 Intention to pick up food: The AI ​​assistant will guide actions in the virtual environment, such as rotating dishes or simulating the action of picking up food. For patients with limited mobility, voice instructions can be used to assist patients in eating.

[0070] 2.3.3 Health consultation: AI assistants will provide relevant health information and advice.

[0071] 2.3.4 Environmental Interaction: If you adjust the lighting or music in the virtual environment, the AI ​​assistant will make corresponding environmental setting adjustments.

[0072] When users seek emotional communication, AI assistants provide emotional support and responses.

[0073] Based on the above key technologies, clinical applications are carried out in order to achieve the interaction between virtual smell and vision, so that patients can see the food in the virtual scene and smell the odor at the same time, and realize intelligent human-computer interaction with the equipment, stimulate the body's gastrointestinal digestive juice secretion, and promote the early recovery of gastrointestinal function, providing a basis for the operation technology of the integrated AI+VR fake feeding intelligent nursing device.

[0074] like Figure 6 As shown, the research and development technology roadmap of the voice interaction module of the nursing device designed corresponding to the method steps is shown. The user opens the application VR program, the VR program environment is initialized, the VR program obtains and returns the knowledge base medical record background knowledge, the VR program assembles the virtual role play, obtains and returns the nutrition matching suggestions, and the VR program assembles the medical staff role; when the user inputs the voice, the voice server receives the voice and converts it into text, and then returns the text content to the VR program. At this time, the VR program updates the conversation message with the history record, and the VR program sends the conversation message to the general conversation AI module. The general conversation AI module determines the user's intention and transmits the activated answer to the VR program; the VR program receives the expression of dining intention and / or the expression of food picking intention. When the expression of dining intention is received, the matching menu is recommended according to the feedback of dining intention; when the expression of food picking intention is received, the appropriate dish is rotated according to the feedback of food picking intention and please use it. The expression of dining intention and the expression of food picking intention can be performed at the same time; the VR program can convert the text answer into an audio stream and drive the virtual character to perform the corresponding dialogue response action. The above dialogue cycle can be repeated.

[0075] like Figure 7 As shown, this is a technical roadmap for the research and development of the nursing device operation technology designed in the method steps, the user starts the application, the AI ​​assistant is initialized, the user interacts with the AI ​​assistant, voice recognition, and user intention determination. If the dining intention is determined, a matching menu is recommended based on the dining intention. If the intention to pick up dishes is determined, the dishes are rotated / the picking up dishes action is performed based on the picking up intention. If health consultation is determined, health information is provided based on the consultation content. If the need for environmental interaction is determined, the environmental settings are adjusted based on the environmental interaction needs. If emotional communication is determined, emotional support is provided based on the communication intention. The above-determined user intention voice is synthesized for feedback, and the corresponding virtual action is performed in the virtual environment. The user interacts with the AI ​​assistant after receiving the feedback.

[0076] like Figure 8As shown, a flowchart of a multi-sensory fusion algorithm provided by an embodiment of the present invention; wherein, from the "start" node, the system first receives input signals from the XR head display, the operating handle (including the virtual handle simulated by gesture recognition) and the odor spray module. Each signal is sent to the corresponding processing module (vision, hearing, smell, control) respectively. The control signal is processed separately to generate control feedback, which will be sent back to each processing module for adjustment. Visual, auditory and olfactory data are fused to synchronize multi-sensory data. An adaptive learning algorithm is applied to adjust the sensory output according to the user's historical data and preferences. The adjusted sensory output is sent to the corresponding module (display device, headphones, odor spray module). The system ends the entire process through feedback from the odor spray module. In some embodiments, a deep learning neural network can also be used to replace the traditional multi-sensory fusion algorithm to more efficiently process and fuse visual, auditory and olfactory data to achieve a more natural sensory fusion effect.

[0077] This project is based on the integrated audio-visual and olfactory VR fake feeding device developed in the early stage. Aiming at the problems of limited operation of patients in special populations, food recognition and precise odor spraying functions of VR fake feeding devices, this project innovatively develops AI+VR intelligent fake feeding nursing devices and technologies. The dynamic maximum matching algorithm based on dictionary improvement is adopted to realize the efficient and accurate voice recognition function of "whatever you want to eat, just say it"; the precise sniffer intention recognition algorithm is adopted to automatically trigger the olfactory odor box to accurately spray the corresponding odor when selecting food. The above-designed matching algorithm of intelligent voice interaction and precise odor spraying further enriches the functions of VR fake feeding devices, improves the interactivity and question-answering efficiency of the equipment, and provides users with a more realistic and natural immersive VR dining experience, making it closer to the sensory experience of the real world.

[0078] The project focuses on the innovative research and development and application of AI+VR intelligent sham feeding care devices and technologies, which is expected to shorten the duration of non-physiological states of patients undergoing clinical fasting, enteral nutrition through tube feeding or parenteral nutrition, accelerate the recovery of patients' gastrointestinal function, shorten the average length of hospital stay, promote rapid recovery of patients, reduce hospitalization costs, and increase hospital capacity.

[0079] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0081] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0082] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A multi-sensory AI and VR nursing method, characterized in that: The method comprises: S101, receiving a first instruction to enter a virtual scene, and the subject enters the virtual scene to generate a virtual subject; the virtual scene includes a tasting area; S102, receiving a second instruction for the virtual subject to move to a tasting area, and the virtual subject moves to the tasting area; at least one fake feeding object is placed in the tasting area; S103, receiving an instruction for a fake fed object, the fake fed object appears in the hands of the virtual subject; identifying the category of the fake fed object, the virtual subject gradually brings the fake fed object closer to the mouth of the virtual subject, when the distance between the fake fed object and the mouth of the virtual subject in the virtual scene exceeds a threshold or an eating instruction is received, the odor spraying device receives an odor simulation instruction of the fake fed object to start the device and generate a corresponding odor according to the category of the fake fed object.

2. The multi-sensory integrated AI and VR nursing method according to claim 1, characterized in that: If the fake feeding object is one, the odor spraying device receives an odor simulation command of one fake feeding object, and releases a single odor corresponding to the fake feeding object through the movement of the structure in the odor spraying device; If the fake feeding objects include at least two kinds, the odor spraying device receives the odor simulation commands of at least two kinds of fake feeding objects, and releases the mixed odors corresponding to the fake feeding objects through the movement of the structure in the odor spraying device; The distance between the fake feeding object and the virtual subject's mouth in the virtual scene is the physical distance between the external handle and the subject's mouth.

3. The multi-sensory integrated AI and VR nursing method according to claim 1, characterized in that: The virtual scene includes any one or more of the following: a virtual Chinese restaurant scene, a virtual fruit store scene, and a virtual pastry shop scene; each virtual scene also includes a visiting area; After 101 and before 102, the method further includes: receiving instructions for the virtual subject to move to different positions in the virtual scene, and displaying corresponding fake feeding objects at different positions; An instruction to select different fake feeding objects in the virtual scene is received to obtain the selected fake feeding objects; the selected fake feeding objects will appear in the tasting area.

4. The multi-sensory integrated AI and VR nursing method according to claim 1, characterized in that: The method further comprises: if there is a fake feeding object that needs to be added during the tasting process in the tasting area, receiving an instruction to add the fake feeding object in the virtual scene to obtain the selected fake feeding object; The method further comprises: identifying the type of the fake feeding object and determining whether tableware is needed; if tableware is not needed, the virtual subject can directly pick it up; if tableware is needed, tableware appears in the left or right hand of the virtual subject when picking up the fake feeding object.

5. The multi-sensory integrated AI and VR nursing method according to claim 3, characterized in that: Between 102 and 103, the method further includes 1021, determining whether the perspective of the virtual subject is parallel to the ground of the virtual scene, if so, executing step 103; if not, giving a prompt to switch the perspective, after which the perspective of the virtual subject is parallel to the ground in the virtual scene, then executing step 103; The tasting area includes a turntable table; after 1021 and before 103, the method further includes: receiving instructions for fake feeding objects in the tasting area, changing the fake feeding objects in front of the virtual subject; when the tasting area is a turntable table, changing the fake feeding objects in front of the subject by rotating the turntable.

6. The multi-sensory integrated AI and VR nursing method according to claim 3, characterized in that: The odor spraying device comprises: a shell, a odor box assembly, an odor box inner sealing plate assembly, and a odor box top cover; a placement space is reserved inside the shell, and the odor box inner sealing plate assembly is placed in the placement space; a first side of the shell is movably connected to the odor box top cover, and an opening is reserved on the second side of the shell for the odor box assembly to extend into, and the odor box assembly is movably installed in the odor box inner sealing plate assembly along the opening; the odor box assembly comprises: at least one cotton swab soaked in liquids with different odors, a buffer member arranged at the first end of the cotton swab, and a first protective shell, at least one first receiving groove for accommodating a single cotton swab and at least one second receiving groove for accommodating a single buffer member are arranged inside the first protective shell; the other end of the buffer member is connected to the first protective shell; the cotton swab extends out of the opening of the first receiving groove, and a sealing ring is arranged at the position of the single cotton swab at the opening end of the first receiving groove; the extended cotton swab is connected to the odor box inner sealing plate assembly; the odor box inner sealing plate assembly comprises an oscillation wave plate, and the vibration of the ultrasonic oscillation wave plate atomizes the liquid in the cotton swab into mist and releases it.

7. The multi-sensory integrated AI and VR nursing method according to claim 1, characterized in that: The method also includes: detecting the virtual environment usage scenario set by the subject, determining the current environment type according to the light and noise levels of the environment, and adjusting the usage scenario in the virtual scene according to the current environment type.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images