Intelligent accompanying method, intelligent accompanying system and computer storage medium
By obtaining and analyzing the behavior and emotional information of people living alone, targeted accompanying strategies are generated and personalized emotional interaction services are provided, and the existing smart home devices are solved, and the problem of poor experience and effectiveness of accompanying people living alone is achieved accurately Status judgment and full-day emotional care.
Patent Information
- Application Number
- CN202411159025.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-31
AI Technical Summary
The existing smart home terminal devices have problems of poor experience and poor results in accompanying people living alone, and it is difficult to accurately judge the abnormal behavior and emotional state of people living alone, resulting in the inability to provide targeted emotional care services.
By obtaining behavioral information, voice information and indoor environment information of individuals living alone, analyzing their current status information, and generating targeted companion strategies based on abnormal behavior or emotional information, providing human-computer interaction services, including video collection, voice communication and virtual avatars, Cloning, a vertical field dialogue model combining massive psychological knowledge and personal information training, realizes personalized emotional interaction.
It realizes accurate status judgment and personalized emotional care for individuals living alone, reduces the false alarm rate, provides uninterrupted affectionate emotional care services throughout the day, and improves user experience and mental health level.
Smart Images
Figure CN119229335A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of intelligent terminal technology, and in particular, relates to an intelligent accompanying method, an intelligent accompanying system and a computer storage medium. Background Art
[0002] People living alone mainly include elderly people living alone, disabled people who cannot take care of themselves, members of extremely poor families, and elderly people left behind in rural areas. The problems faced by these people living alone include but are not limited to difficulties in life care, medical services, spiritual comfort, etc., and they need more care and help from the society.
[0003] In view of the above phenomenon, the existing technology provides some smart home terminal devices that can accompany people living alone, and some smart home terminal devices can also monitor people living alone. When abnormalities are detected, the abnormal results will be sent to caring people in the community, so that caring people in the community can serve them.
[0004] For people living alone, existing smart home devices still have some shortcomings, resulting in a poor experience, and further improvements are needed. Summary of the invention
[0005] The embodiments of the present application provide an intelligent accompanying method, an intelligent accompanying system and a computer storage medium, which aim to solve the technical problems that the current smart home devices for people living alone have some shortcomings, resulting in poor experience and poor results.
[0006] In a first aspect, an embodiment of the present application provides an intelligent accompanying method, the method comprising:
[0007] Acquire current first information and current second information of a target person living alone; the current first information includes behavior information, sound information, and indoor environment information of the target person living alone; the current second information includes emotional information of the target person living alone; determine current state information of the target person living alone based on the current first information or the current second information;
[0008] When the current state information is an abnormal behavior state or the emotion information is within a preset emotion change range, a target accompanying strategy corresponding to the current state information is obtained, and a target human-computer interaction service corresponding to the target accompanying strategy is provided.
[0009] In the embodiment of the present application, the current status information of the person living alone can be analyzed through the behavior information, sound information, indoor environment information or emotional information of the person living alone, and the abnormal behavior state of the person living alone or the emotional information being within a preset emotional change range can be determined immediately, and a targeted care strategy can be generated according to the current status information, so as to provide targeted emotional care services for the person living alone in a timely manner.
[0010] In one embodiment, the behavior information and emotion information of the target person living alone are obtained by:
[0011] Capturing video of the target person living alone to obtain video data;
[0012] Acquire facial expression information, action information, posture information, and position information of the target person living alone in the room from the video data;
[0013] The behavior information and emotion information of the target single person are determined according to the facial expression information, the action information, the posture information and the position information.
[0014] In an embodiment of the present application, by acquiring the facial expression information, movement information, posture information, and indoor position information of the target person living alone, the behavior information and emotional information of the target person living alone can be more accurately determined to accurately judge the current state of the target person living alone, which helps to more accurately distinguish whether the current state information of the target person living alone is an abnormal behavior state.
[0015] In one embodiment, after the step of determining the current status information of the target person living alone based on the current first information, the method further includes:
[0016] In the case where the current state information is an abnormal behavior state, determining the severity and duration of the abnormal behavior state;
[0017] In the case where the severity and the duration meet the preset conditions, executing the step of acquiring the target accompanying strategy corresponding to the current state information;
[0018] or,
[0019] When the severity and the duration meet preset conditions, an alarm message is remotely sent to the second terminal.
[0020] In the embodiments of the present application, the alarm strategy can be dynamically adjusted to reduce the false alarm rate according to the severity and duration of the abnormal behavior state, or the timing of providing human-computer interaction services corresponding to the accompanying strategy can be dynamically adjusted to provide a better user experience.
[0021] In one embodiment, when the current state information is an abnormal behavior state or the emotion information is within a preset emotion change range, obtaining a target accompanying strategy corresponding to the current state information, and providing a target human-computer interaction service corresponding to the target accompanying strategy include:
[0022] When the current state information is an abnormal behavior state, obtaining personal information of the target person living alone from a database, the personal information at least including age information, gender information, hobby information, and living habit information of the target person living alone;
[0023] Inputting the personal information of the target person living alone and the current status information into a target model, so that the target model outputs a corresponding target human-computer interaction service, wherein the target model is a vertical field dialogue model that integrates psychological knowledge and the personal information of the target person living alone as a training set to complete model training;
[0024] The target human-computer interaction service is provided to the target person living alone, and the target human-computer interaction service includes voice communication with the target person living alone, and at least one of playing voice communication and playing story information, wherein the voice communication, the voice communication and the story information are related to the current status information, the interest and hobby information and the living habit information.
[0025] In the embodiment of this application, based on massive psychological knowledge and personal information of single people, high-quality vertical field dialogue models can be trained to achieve context-related and empathetic emotional interaction for the target single people; according to the user's emotional state, interests, hobbies, living habits and other information, personalized care words and companionship strategies are dynamically generated to support the provision of targeted and caring emotional care services throughout the day. Support a variety of human-computer interaction forms, including but not limited to listening to users' confessions, sharing inspirational stories, telling bedtime stories, recommending background music, etc., to create a soothing and relaxing atmosphere for the target single people.
[0026] In one embodiment, the sound information includes intonation information and semantic information, and the method further includes:
[0027] Determining a target dialect of the target single person based on the intonation information and the semantic information;
[0028] A voice communication is provided with the target solitary person based on the target dialect.
[0029] In the embodiments of the present application, the dialect information of people living alone can be identified based on the intonation information and semantic information, and human-computer interaction with the single user can be performed through the dialect, allowing the single user to feel more intimate care.
[0030] In one embodiment, the method further comprises:
[0031] Determining the preferred person information of the target person living alone;
[0032] Determine the clone information corresponding to the preferred character information from a database, wherein the clone information includes image clone information and voice clone information of the preferred character information;
[0033] The providing of a target human-computer interaction service corresponding to the target accompanying strategy includes:
[0034] The image clone information and the voice clone information are used to provide a target human-computer interaction service corresponding to the target escort strategy.
[0035] In the embodiment of the present application, a virtual image and voice information clone of a preferred character that the single person likes can be created, and the image clone information and voice clone information can be used to perform human-computer interaction with the single person, further allowing the single user to feel more intimate care.
[0036] In one embodiment, the method further comprises:
[0037] Respond to the settings information entered by the user;
[0038] The personal information of the target person living alone and / or the preferred person information of the target person living alone are adjusted based on the setting information.
[0039] In the embodiment of the present application, the user can set the information of the person living alone according to his or her own needs, which helps to regenerate a new targeted care strategy.
[0040] In a second aspect, an embodiment of the present application provides an intelligent accompanying system, which includes an image acquisition device and a main control terminal deployed indoors; the image acquisition device is configured to capture video of the target person living alone; the main control terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the intelligent accompanying method described in the first aspect are implemented.
[0041] The intelligent care system provided in the embodiment of the present application deploys a full-coverage, no-dead-angle image acquisition device in a home environment to collect multimodal data in real time, and uses artificial intelligence technologies such as computer vision, speech recognition, and natural language processing to achieve functions such as user location tracking, abnormal behavior detection, and emotion perception and analysis. It can determine the abnormal behavior status of people living alone in the first place, generate targeted care strategies based on current status information, and provide targeted emotional care services for people living alone in a timely manner.
[0042] In a third aspect, the present invention further proposes a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the intelligent accompanying method described in the first aspect above are implemented.
[0043] For the beneficial effects of the third aspect of the present application, please refer to the beneficial effects of the first and second aspects mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0045] Figure 1 The structural framework diagram of the intelligent accompanying system provided for this application;
[0046] Figure 2 This is a schematic diagram of the first embodiment of the intelligent accompanying method provided by the embodiment of the present application;
[0047] Figure 3 It is a flowchart of a second embodiment of the intelligent accompanying method provided in an embodiment of the present application;
[0048] Figure 4 Another structural framework diagram of the intelligent accompanying system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0050] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0051] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0052] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0053] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0054] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0055] Embodiment 1
[0056] refer to Figure 1 , Figure 1 This is a structural block diagram of the intelligent accompanying system provided by the present invention, wherein the system includes an image acquisition device 1004 and a main control terminal 100 deployed indoors; the image acquisition device 1004 is configured to perform video acquisition of the target person living alone; the system may also include a sound sensor and an environmental sensor, wherein the sound sensor is configured to collect sound information of the target person living alone, and the environmental sensor is configured to collect environmental information of the room.
[0057] In a specific implementation, the image acquisition device of the embodiment of the present application can be a smart camera with full coverage and no blind spots deployed in a home environment, and by reasonably planning the camera position, it can ensure that every corner can be effectively monitored.
[0058] The camera integrates high-definition lens and infrared night vision function to ensure the quality of image and video acquisition at all times of day and night. Both the smart camera and the main control terminal can have built-in microphone arrays and audio processing chips to support high-quality voice extraction and playback, and realize voice interaction throughout the house.
[0059] The main control terminal 100 includes a memory 1002, a processor 1001, and a computer program stored in the memory 1002 and executable on the processor 1001; of course, the main control terminal 100 also has a built-in image acquisition device camera, which can be used to collect video data of the target solitary person at close range; of course, the main control terminal 100 also has a built-in sound sensor, which can be used to collect sound information of the target solitary person at close range;
[0060] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0061] In some embodiments, the memory 1002 may be an internal storage unit of the main control terminal 100, such as a hard disk or memory of the main control terminal 100. In other embodiments, the memory 1002 may also be an external storage device of the main control terminal 100. When the processor 1001 executes the computer program, the following steps of the intelligent accompanying method are implemented. Specifically, refer to Figure 2 The intelligent accompanying method of the embodiment of the present application includes steps S10 to S30:
[0062] Step S10: obtaining current first information and current second information of the target person living alone; the current first information includes behavior information, sound information, and indoor environment information of the target person living alone; the current second information includes emotional information of the target person living alone;
[0063] Step S20: determining the current status information of the target person living alone based on the current first information or the current second information;
[0064] Step S30: When the current state information is an abnormal behavior state or the emotional information is within a preset emotional change range, obtain a target accompanying strategy corresponding to the current state information, and provide a target human-computer interaction service corresponding to the target accompanying strategy.
[0065] The embodiment of the present application can obtain the behavior information and emotional information of the target person living alone in the following ways:
[0066] Capture video of a target person living alone to obtain video data; obtain facial expression information, action information, posture information, and position information of the target person living alone in the room from the video data; and determine behavior information and emotion information of the target person living alone based on the facial expression information, the action information, the posture information, and the position information.
[0067] The sound information includes intonation information and semantic information. The method further includes: determining a target dialect of the target single person based on the intonation information and the semantic information; and providing voice communication with the target single person based on the target dialect.
[0068] In the specific implementation, the camera collects image video data and voice data of the home environment in real time, and performs front-end preprocessing, such as image enhancement and voice noise reduction.
[0069] Through video analysis algorithms, the facial expression information, movement information, posture information, and indoor location information of the target lone person are tracked and identified in real time. Using speech recognition technology, the user's voice commands and conversation content are converted into text, supporting dialect recognition and semantic understanding, and realizing natural and smooth human-computer interaction.
[0070] Specifically, the best working camera can be determined and the data collection strategy can be optimized. For example, the camera suitable for the current scene can be selected based on the camera's resolution, frame rate, viewing angle and other performance indicators; the camera's installation position and angle can be reasonably planned according to the user's activity area and path to ensure that important areas are covered without blind spots. Based on the analysis results, the camera's focal length and viewing angle can be dynamically adjusted or specific cameras can be enabled / disabled to ensure optimal monitoring of the target area.
[0071] Understandably, conventional home security systems have some technical limitations in detecting and responding to abnormal events: these systems are easily disturbed by environmental factors and prone to false alarms, causing unnecessary troubles to users. At the same time, due to the limitations of algorithm performance and hardware capabilities, the abnormal detection and warning of traditional security systems often have a certain delay, and the real-time performance is not ideal, which may miss the best time to deal with it. Therefore, there is an urgent need for a smarter and more efficient abnormal detection solution to reduce the false alarm rate, improve the real-time response capability, and create a safer and more reassuring home environment for users.
[0072] The embodiments of the present application can effectively overcome the above technical defects. In the specific implementation, the embodiments of the present application support the speech recognition and synthesis of dialects. The system can realize intelligent speech interaction in multiple dialects and maintain stable and reliable performance when introducing new dialects. Based on dialect adaptation and continuous learning strategies, users can freely expand the types of dialects supported by the system and enjoy personalized and regionalized speech interaction experience. The specific implementation schemes include:
[0073] ① Based on the speech model trained on Mandarin, the model’s adaptability to specific dialects is improved through transfer learning and parameter tuning, so as to achieve accurate recognition and natural synthesis of dialects.
[0074] ② In order to ensure the original performance of the system when introducing new dialects, this embodiment introduces a continuous learning strategy. By designing intelligent parameter updates and gradient control mechanisms, while adapting to new dialects, it prevents significant impact on the recognition and synthesis capabilities of learned dialects, thus achieving a balance between dialect expansion and system stability.
[0075] ③ In the training of speech recognition model, the embodiment of the present application introduces a data augmentation method based on text-to-speech (TTS). Using a high-quality speech synthesis engine, a large number of synthesized speech with the same content is generated according to the text content of the dialect corpus, enriching the dialect speech training data and improving the generalization ability and robustness of the model.
[0076] ④ Aiming at the differences in acoustic features of different dialects, an adaptive feature adjustment module (DialectAdapter) is designed. In the front-end processing stage of speech recognition, the acoustic features of the input speech are automatically adjusted and optimized through the adaptive network to make it more suitable for the characteristics of the target dialect and enhance the recognition accuracy of dialect speech.
[0077] ⑤ In the speech synthesis part, a multi-language / multi-dialect speech synthesis architecture is adopted, and by sharing key components such as pronunciation dictionaries and prosody models, the reuse and rapid expansion of dialect synthesis resources are realized. At the same time, the dialect prosody feature extraction and migration algorithm is introduced to automatically capture the prosody characteristics of the dialect and generate more natural and authentic synthesized speech.
[0078] In specific applications, the intelligent accompanying method of the embodiment of the present application can have two application modes (i.e., active care mode and passive care mode), that is, after step S20:
[0079] The first "active care mode": that is, when the current state information is that the emotional information is within the preset emotional change range, the target accompanying strategy corresponding to the current emotional state is directly obtained, and the target human-computer interaction service corresponding to the target accompanying strategy is provided. For example, in the active care mode, the intelligent accompanying system actively pays attention to the emotional changes of the target single person and gives condolences through emotional detection and the chat function of the main control terminal. The preset emotional change range can be understood as an emotional threshold range for triggering the target accompanying strategy. The preset emotional change range includes multiple emotional point values. The values of these emotional points can be increased from low to high. The higher the value of the emotional point, the more intense the emotional fluctuation of the target single person; for example, when the emotional information is at the lowest emotional point of the preset emotional change range, the intelligent accompanying system will actively greet, chat with the user, and actively initiate a topic. And the intelligent accompanying system will adjust the interactive mode (i.e., the speaking and chatting mode) according to the emotional point value of the current emotional information. For example, when the user is in a good mood, he can make more jokes. If the user is emotionally unstable, the intelligent accompanying system will avoid sensitive topics. The second "passive care mode" determines the severity and duration of the abnormal behavior state when the current state information is an abnormal behavior state; and executes the step of obtaining the target accompanying strategy corresponding to the current state information when the severity and the duration meet the preset conditions;
[0080] Or, when the severity and the duration meet a preset condition, an alarm message is remotely sent to the second terminal. The preset condition may be that the severity is greater than a preset severity threshold and the duration exceeds a preset time threshold.
[0081] It is understandable that the embodiment of the present application performs real-time analysis on the video data collected by the camera based on image and video understanding technology. If it is identified from the video data that the target person living alone has abnormal behavior patterns such as falling, staying still for a long time, and not returning home on time, then the current status information of the target person living alone is determined to be an abnormal behavior state.
[0082] This embodiment develops an intelligent alarm algorithm to dynamically adjust the alarm strategy according to the severity and duration of abnormal behavior to reduce the false alarm rate. After an abnormal situation is discovered, it automatically confirms to the user. If no timely response is received, the alarm mechanism is immediately triggered to notify family members or emergency contacts, and at the same time, community medical and security resources are linked.
[0083] In the specific implementation, the method of checking the abnormal behavior status of this embodiment is mainly to build a comprehensive, accurate and robust abnormal behavior recognition and early warning mechanism by comprehensively utilizing multi-source heterogeneous data such as video, audio, and environmental sensing. First, through the camera, microphone array and various sensors arranged by the system, the video, audio and environmental parameter data of the user's living environment are collected in real time, and pre-processing such as background modeling and noise suppression is performed. Then, the key features of each modal data are extracted, such as human posture and motion trajectory in the video, voice and footsteps in the audio, and abnormal events of the environmental sensor. In the feature fusion stage, a multi-level feature fusion strategy is adopted to realize information interaction and supplementation between modalities at different levels such as data, features, and decisions. Based on the fusion features, a multimodal anomaly detection model is constructed, such as a multimodal attention network, a cross-modal generative adversarial network, etc., to establish a mapping relationship between multimodal data and abnormal behavior. In the detection and reasoning stage, the features of each modal data are fused in real time, the occurrence of abnormal behavior is comprehensively judged, and the abnormal confidence is quantified. Finally, adaptive thresholds are set according to the confidence level of anomalies to trigger different levels of alarms and countermeasures, while supporting user feedback and incremental learning to continuously improve the comprehensiveness and accuracy of the system. Through multimodal fusion, anomaly detection is more comprehensive and reliable, able to cope with complex and changeable abnormal behavior patterns, and provide solid decision-making support for the intelligent accompanying system.
[0084] Furthermore, the step 30 further comprises:
[0085] When the current state information is an abnormal behavior state, obtaining personal information of the target person living alone from a database, the personal information at least including age information, gender information, hobby information, and living habit information of the target person living alone;
[0086] Inputting the personal information of the target person living alone and the current status information into a target model, so that the target model outputs a corresponding target human-computer interaction service, wherein the target model is a vertical field dialogue model that integrates psychological knowledge and the personal information of the target person living alone as a training set to complete model training;
[0087] The target human-computer interaction service is provided to the target person living alone, and the target human-computer interaction service includes voice communication with the target person living alone, and at least one of playing voice communication and playing story information, wherein the voice communication, the voice communication and the story information are related to the current status information, the interest and hobby information and the living habit information.
[0088] It is understandable that although existing conventional voice interaction technologies provide convenience for users in terms of information acquisition and task processing, they still have obvious deficiencies in emotional interaction. These conventional systems often ignore the emotional needs of users, lack the ability to pay attention to and respond to the emotional state of users, and cannot provide intimate emotional companionship and care. It is also difficult to accurately perceive and understand user emotions, and lacks personalized and contextualized interactive experience;
[0089] The embodiments of the present application can overcome the above technical defects. In the specific implementation, the embodiments can use multimodal emotion analysis technology to integrate multi-source information such as facial expression information, voice intonation, semantic content, etc. to perceive and track the user's emotional state in real time. Use deep learning algorithms to build emotion recognition models to provide personalized emotion judgment services for users of different ages, genders, and cultural backgrounds. Combine the emotion analysis results with user behavior data to comprehensively evaluate the user's mental health status and provide a basis for subsequent emotional care.
[0090] Based on a vast amount of psychological knowledge and the personal information of the target single person, we train high-quality vertical field dialogue models to achieve context-related and empathetic emotional interactions. We dynamically generate personalized care words and companionship strategies based on the user's emotional state, interests, hobbies, living habits, etc., and provide 24-hour uninterrupted and caring emotional care services. We support a variety of interactive forms, such as listening to users' confessions, sharing inspirational stories, telling bedtime stories, recommending background music, etc., to create a soothing and relaxing atmosphere.
[0091] In the specific implementation, the target model of the embodiment of the present application needs to build a large-scale language model based on massive psychology literature and case data training, and integrate psychological counseling dialogue materials, including psychology literature, professional books and journal articles, diagnosis and treatment cases, popular science articles, forums and communities, standards and specifications, etc., so that the model can master the communication skills and strategies of psychological counseling, such as empathy, active attention, awareness, etc. At the same time, the introduction of personalization and privacy protection mechanisms allows the model to generate personalized and highly relevant response content based on the user's personal background, problems and demands. The specific technical innovation plan is:
[0092] ① Divide the text corpus of the entire knowledge base into multiple blocks;
[0093] ② Use the Embedding model to convert each block into a vector;
[0094] ③Store all vectors in the vector database;
[0095] ④Using the same embedding model used to embed the knowledge base itself, embed the question / query to be asked and convert it into a vector;
[0096] ⑤ Use the generated vector to run a query in the index of the vector database, returning the top k most similar vectors in the given embedding / latent space;
[0097] ⑥ Combine the question and the retrieved text block into prompt words according to the template;
[0098] ⑦ Pass the prompt word to the large language model (LLM), instructing the large language model LLM to answer the given question using only the provided context.
[0099] ⑧The large language model LLM generates an answer based on the prompt word and returns it to the user.
[0100] The beneficial effects of the embodiments of the present application are as follows: by deploying an image acquisition device in a home environment, the system can monitor the location and behavior of the target person living alone in real time, accurately identify the abnormal behavior status of the person living alone, and promptly trigger the vertical field dialogue model, so as to accurately understand the emotional state of the user, and provide 24-hour uninterrupted, personalized emotional support and companionship services based on the user's emotional state, effectively alleviate the user's negative emotions, and improve the level of mental health.
[0101] Embodiment 2
[0102] refer to Figure 3 Based on the technical solution of the above-mentioned embodiment 1, the embodiment 2 of the present application further includes the following steps:
[0103] Step S01: Determine the preferred person information of the target single person;
[0104] Step S02: determining the clone information corresponding to the preferred character information from a database, wherein the clone information includes image clone information and voice clone information of the preferred character information;
[0105] Accordingly, the step of providing the target human-computer interaction service corresponding to the target accompanying strategy in step S30 further includes:
[0106] Step S03: using the image cloning information and the voice cloning information to provide a target human-computer interaction service corresponding to the target escort strategy.
[0107] In the embodiment of the present application, a virtual image and voice information clone of a preferred character that the single person likes can be created, and the image clone information and voice clone information can be used to perform human-computer interaction with the single person, further allowing the single user to feel more intimate care.
[0108] Understandably, reference Figure 4The embodiment of the present application can achieve the realistic image shaping of the digital assistant by introducing a realistic 3D digital human model through motion capture and expression transfer technology. By using voice cloning and speech synthesis technology, the digital human's speaking style and voice are highly similar to the user's favorite character, enhancing the sense of intimacy. By building an immersive virtual interactive environment, the digital human can make movements and expression feedback that fit the scene according to the content of the conversation and the user's emotions, simulate face-to-face communication, and enhance the sense of reality of companionship.
[0109] In a specific implementation, the embodiment of the present application can obtain the image clone information and voice clone information corresponding to the preferred character information in the following manner:
[0110] First, it is necessary to collect relevant data of the user's relatives, including voice recordings and image / video materials. These data will become the basis for building digital human images and voice cloning. Next, advanced algorithms are used to map the audio to a 3D mesh, a process that is handled by VividTalk. Through the technology of blending shapes and vertex offsets, this embodiment can simulate precise facial expressions and head movements, providing the model with a high degree of expressiveness. Subsequently, this embodiment will use dual-branch motion-VAE and generators to convert 3D mesh motion into 2D video. This conversion process not only ensures the smoothness of the motion, but also maintains visual coherence and realism. Ultimately, this embodiment can generate high-quality speaking videos with personalized images and sounds, presenting the real expressions and head movements of the digital human image, further allowing single users to feel more intimate care.
[0111] Furthermore, in some other embodiments, the method further includes: responding to setting information input by a user; and adjusting the personal information of the target single person and / or the preferred person information of the target single person based on the setting information.
[0112] This embodiment can provide users with flexible system configuration options. Users can customize the parameter settings of emotional care and security functions according to their own needs, such as interaction style, sensitivity threshold, etc. The user can be the person living alone; the user can also be the relatives and friends of the person living alone. Relatives and friends can use the intelligent care system of the embodiment of the present application to regularly observe the current situation of the person living alone and customize a set of care plan strategies for the person living alone.
[0113] In addition, the intelligent accompanying system of the embodiment of the present application can also establish a complete user privacy protection mechanism, desensitize the collected data, and users can independently manage the authorization scope and validity period of their personal information.
[0114] In addition, an embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0118] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0119] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. An intelligent accompanying method, characterized in that: The method comprises: Acquire current first information and current second information of a target person living alone; the current first information includes behavior information, sound information, and indoor environment information of the target person living alone; the current second information includes emotional information of the target person living alone; Determine the current state information of the target person living alone based on the current first information or the current second information; When the current state information is an abnormal behavior state or the emotion information is within a preset emotion change range, a target accompanying strategy corresponding to the current state information is obtained, and a target human-computer interaction service corresponding to the target accompanying strategy is provided.
2. The method according to claim 1, characterized in that The behavior information and emotional information of the target person living alone are obtained by: Capturing video of the target person living alone to obtain video data; Acquire facial expression information, action information, posture information, and position information of the target person living alone in the room from the video data; The behavior information and emotion information of the target single person are determined according to the facial expression information, the action information, the posture information and the position information.
3. The method according to claim 1, characterized in that After the step of determining the current status information of the target person living alone based on the current first information, the method further includes: In the case where the current state information is an abnormal behavior state, determining the severity and duration of the abnormal behavior state; In the case where the severity and the duration meet the preset conditions, executing the step of acquiring the target accompanying strategy corresponding to the current state information; or, When the severity and the duration meet preset conditions, an alarm message is remotely sent to the second terminal.
4. The method according to any one of claims 1 to 3, characterized in that When the current state information is an abnormal behavior state or the emotion information is within a preset emotion change range, obtaining a target accompanying strategy corresponding to the current state information, and providing a target human-computer interaction service corresponding to the target accompanying strategy, including: When the current state information is an abnormal behavior state, obtaining personal information of the target person living alone from a database, the personal information at least including age information, gender information, hobby information, and living habit information of the target person living alone; Inputting the personal information of the target person living alone and the current status information into a target model, so that the target model outputs a corresponding target human-computer interaction service, wherein the target model is a vertical field dialogue model that integrates psychological knowledge and the personal information of the target person living alone as a training set to complete model training; The target human-computer interaction service is provided to the target person living alone, and the target human-computer interaction service includes voice communication with the target person living alone, and at least one of playing voice communication and playing story information, wherein the voice communication, the voice communication and the story information are related to the current status information, the interest and hobby information and the living habit information.
5. The method according to claim 4, characterized in that The sound information includes intonation information and semantic information, and the method further includes: Determining a target dialect of the target single person based on the intonation information and the semantic information; A voice communication is provided with the target solitary person based on the target dialect.
6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determining the preferred person information of the target person living alone; Determine the clone information corresponding to the preferred character information from a database, wherein the clone information includes image clone information and voice clone information of the preferred character information; The providing of a target human-computer interaction service corresponding to the target accompanying strategy includes: The image clone information and the voice clone information are used to provide a target human-computer interaction service corresponding to the target escort strategy.
7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Respond to the settings information entered by the user; The personal information of the target person living alone and / or the preferred person information of the target person living alone are adjusted based on the setting information.
8. An intelligent accompanying system, characterized in that: The system includes an image acquisition device and a main control terminal deployed indoors; the image acquisition device is configured to capture video of the target person living alone; the main control terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the intelligent accompanying method described in any one of claims 1 to 7 are implemented.
9. The intelligent accompanying system according to claim 8, characterized in that: The system further includes a sound sensor and an environment sensor. The sound sensor is configured to collect sound information of the target person living alone, and the environment sensor is configured to collect environment information in the room.
10. A computer storage medium storing a computer program, wherein the processor implements the steps of the intelligent nursing method according to any one of claims 1 to 7 when executing the computer program.