AI-based juvenile accompanying method, system and device and storage medium

Abstract

The embodiment of the invention discloses an AI-based juvenile accompanying method, system and equipment and a storage medium, which can simulate the sound of a father and a mother to interact with a child, really accompany the child as a separate body of a parent, see the dialogue interaction between the AI separate body and the child in the whole course, and can be inserted into a dialogue at any time, so that the interestingness of the child is improved. And after the parents exit the conversation, the AI copies continue to carry out conversation interaction with the children based on the content inserted by the parents. According to the embodiment of the invention, whether the child speaks with himself or herself is judged by simulating human beings, so that the most natural and comfortable awakening is realized, and a specific name does not need to be shouked for calling. According to the method, drawing interaction is simplified, all interaction menus and buttons are removed, all interactions are completed completely through voice and a touch screen, and most of functions of an existing diffusion model are kept.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and specifically to an AI-based method, system, device, and storage medium for accompanying minors. Background Art

[0002] In the existing AIGC chat environment, most of the time, AI and customers ask and answer each other. AI can only answer customers' questions but cannot make more other responses. In addition, the existing AI interaction requires calling out names such as "Xiaox classmate", "Xiaod Xiaod", "heysiri", etc. to wake up before proceeding to the next step of interaction. The steps are cumbersome, the startup time is long, and it is not smart enough.

[0003] The Diffusion model is a type of generative model based on the probabilistic diffusion process. In recent years, it has shown great potential and superiority in generating images, texts and other data types. The model uses the inverse process of the diffusion process, that is, gradually restoring from a simple distribution to a complex data distribution, and generates high-quality data samples through a gradual denoising method.

[0004] The basic idea of ​​the diffusion model comes from the diffusion process in physics, which is a natural phenomenon that describes the movement of particles from high concentration areas to low concentration areas in a medium. In machine learning, the diffusion model gradually transforms data into a noise distribution by introducing random noise, and then gradually restores the data from the noise through the inverse process.

[0005] However, the existing diffusion-type raw image model interactions are relatively complex, and most of the existing painting interactions for minors have many very cumbersome menu interaction operations. Summary of the invention

[0006] To this end, embodiments of the present invention provide an AI-based method, system, device and storage medium for accompanying minors to solve the technical problems of the prior art using diffusion-type raw image models, which have complex interactions, cumbersome steps and long startup time.

[0007] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, there is provided an AI-based method for accompanying minors, the method being applied to an AIGC large model and comprising:

[0009] S1. Monitor children's voices in the surrounding environment and obtain STT audio signals, extract text data from the STT audio signals and input them into a preset stable diffusion model;

[0010] S2. Use the preset stablediffusion model to build the underlying structure of the large model and add the RAG framework on the engineering side, obtain the labeled data and use the labeled data to perform supervised training on the preset stablediffusion model, and use prompt to tune the training results;

[0011] S3, after performing modal fusion and prompt tuning on the text data through the trained stable diffusion model, the text tuning result is input into the multimodal LLM model, and at the same time, the multimodal LLM model obtains the parent history information and generates a natural language text according to the parent history information;

[0012] S4. Use the natural language text and acoustic model to synthesize the TTS audio signal of the AI ​​avatar that is consistent with the parent's voice and output it.

[0013] Furthermore, the method further comprises:

[0014] After receiving the text optimization result, the multimodal LLM model determines whether it is necessary to generate a picture;

[0015] If it is not necessary to generate an image, obtaining the parent's historical information and generating a natural language text according to the parent's historical information;

[0016] If it is necessary to generate a picture, the picture is generated and sent to the child's end. At the same time, the picture is sent to the stable diffusion model after image enhancement, semantic segmentation and feature encoding.

[0017] Further, generating a picture and sending the picture to the child end and sending the picture to the stable diffusion model after image enhancement, semantic segmentation and feature encoding, including:

[0018] Create a new white canvas with a cartoon interaction button at the lower left corner;

[0019] Determine whether there is user voice input / graffiti input;

[0020] If there is user voice input / graffiti input, determine whether it is a color in the preset number of enumerations that is spoken alone, if yes, modify the brush color, if not, determine whether it is a style in the preset number of enumerations that is spoken alone;

[0021] If it is a style in the enumeration, modify the checkpoint;

[0022] If it is not a style in the enumeration, the prompt word information is filtered out from the voice input content and the image is generated after the voice / graffiti input pauses for 5 seconds.

[0023] Furthermore, the child's voice is monitored in the surrounding environment and the STT audio signal is obtained, including:

[0024] Acquire a large amount of audio annotation data and use the audio annotation data for targeted training to obtain STT audio signals;

[0025] The children's voice monitoring includes voice detection, echo cancellation, sound source localization, and sound enhancement, and the STT audio signal is obtained through feature extraction, acoustic model, audio decoding, and semantic analysis.

[0026] Furthermore, the AIGC big model also includes a LangChain framework, and the memory module in the LangChain framework is used to enhance the model memory;

[0027] The multimodal LLM model sends the interactive dialogue to the parent's mobile terminal in real time, and the parent's mobile terminal can send data information to the child's terminal to insert it into the dialogue in real time;

[0028] The AIGC macro model continues to interact with the child based on the content inserted by the parent.

[0029] According to a second aspect of an embodiment of the present invention, there is provided an AI-based minor companionship system, the system comprising:

[0030] A voice monitoring module, used to monitor the voices of minors in the surrounding environment and obtain STT audio signals, extract text data from the STT audio signals and input them into a preset stable diffusion model;

[0031] The training and tuning module is used to build the underlying structure of the large model using the preset stable diffusion model and add the RAG framework on the engineering side, obtain the labeled data and use the labeled data to perform supervised training on the preset stable diffusion model, and use prompt to tune the training results;

[0032] A natural language text generation module is used to perform modal fusion and prompt optimization on the text data through a trained stable diffusion model, and then input the text optimization result into a multimodal LLM model. Meanwhile, the multimodal LLM model obtains parent history information and generates natural language text according to the parent history information;

[0033] The voice output module is used to synthesize and output a TTS audio signal of the AI ​​avatar that is consistent with the parent's voice using the natural language text and the acoustic model.

[0034] According to a third aspect of an embodiment of the present invention, there is provided an AI-based minor companion device, the device comprising: a processor and a memory;

[0035] The memory is used to store one or more program instructions;

[0036] The processor is used to run one or more program instructions to execute the steps of an AI-based method for accompanying minors as described in any of the above items.

[0037] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of an AI-based method for accompanying minors as described in any one of the above items are implemented.

[0038] The embodiments of the present invention have the following advantages:

[0039] The embodiments of the present invention can simulate the voices of parents to interact with children, and can truly accompany children as a parent's avatar. The conversation and interaction between the AI ​​avatar and the child can be seen throughout the process, and the AI ​​avatar can be inserted into the conversation at any time. After the parent exits the conversation, the AI ​​avatar will continue to interact with the child based on the content inserted by the parent. The embodiments of the present invention simulate humans to determine whether the child is talking to them, and achieve the most natural and comfortable wake-up, without the need to call a specific name. The drawing interaction is made simpler, all interactive menus and buttons are removed, and all interactions are completed completely through voice and touch screen, but most of the functions of the existing diffusion model are maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0042] Figure 1 A schematic diagram of the logical structure of an AI-based minor companionship system provided in an embodiment of the present invention;

[0043] Figure 2 A flowchart of an AI-based method for accompanying minors provided in an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the technical architecture of an AI-based method for accompanying minors provided in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the drawing interaction logic in an AI-based method for accompanying minors provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The Diffusion model is a type of generative model based on the probabilistic diffusion process. In recent years, it has shown great potential and superiority in generating images, texts and other data types. The model uses the inverse process of the diffusion process, that is, gradually restoring from a simple distribution to a complex data distribution, and generates high-quality data samples through a gradual denoising method.

[0048] The basic idea of ​​the diffusion model comes from the diffusion process in physics, which is a natural phenomenon that describes the movement of particles from high concentration areas to low concentration areas in a medium. In machine learning, the diffusion model gradually transforms data into a noise distribution by introducing random noise, and then gradually restores the data from the noise through the inverse process.

[0049] However, the existing diffusion-type raw image model interactions are relatively complex, and most of the existing painting interactions for minors have many very cumbersome menu interaction operations.

[0050] In order to solve the technical problems of complex interaction, cumbersome steps and long startup time in the above-mentioned diffusion-type image generation model method.

[0051] refer to Figure 1 The embodiment of the present invention discloses an AI-based juvenile companionship system, which includes: a voice monitoring module 1; a training and tuning module 2; a natural language text generation module 3; and a voice output module 4.

[0052] Corresponding to the above-disclosed AI-based minor companionship system, the embodiment of the present invention further discloses an AI-based minor companionship method. The following describes in detail the AI-based minor companionship method disclosed in the embodiment of the present invention in combination with the above-described AI-based minor companionship system.

[0053] refer to Figure 2 and Figure 3 The present invention discloses an AI-based method for accompanying minors, which is applied to an AIGC large model and includes:

[0054] S1. Monitor children's voices in the surrounding environment and obtain STT audio signals, extract text data from the STT audio signals and input them into a preset stable diffusion model.

[0055] S2. Use the preset stable diffusion model to build the underlying structure of the large model and add the RAG framework on the engineering side, obtain the labeled data and use the labeled data to perform supervised training on the preset stable diffusion model, and use prompt to tune the training results.

[0056] S3. After performing modal fusion and prompt optimization on the text data through the trained stable diffusion model, the text optimization result is input into the multimodal LLM model. Meanwhile, the multimodal LLM model obtains the parent history information and generates natural language text according to the parent history information.

[0057] S4. Use the natural language text and acoustic model to synthesize the TTS audio signal of the AI ​​avatar that is consistent with the parent's voice and output it.

[0058] Among them, the use of the natural language text and the acoustic model to synthesize the TTS audio signal of the AI ​​avatar that is consistent with the parent's voice includes: text analysis, acoustic model, filter coding, and speech synthesis.

[0059] refer to Figure 3 , the parent historical information includes user portrait, historical conversation and memory retrieval, etc.

[0060] Furthermore, the method also includes: after the multimodal LLM model receives the text tuning result, it determines whether it is necessary to generate a picture; if it is not necessary to generate a picture, the parent's historical information is obtained and a natural language text is generated according to the parent's historical information; if it is necessary to generate a picture, the picture is generated and sent to the child end, and at the same time, the picture is sent to the stable diffusion model after image enhancement, semantic segmentation and feature encoding.

[0061] The stable diffusion model is used at the bottom layer, and the prompt enhancement of the RAG framework is added on the engineering side based on the stable diffusion model, so as to carry out supervised training using labeled data in a targeted manner.

[0062] Enhance the LLM model's recognition of stablediffusion related actions, identify the action part by coding, convert the specific checkpoint and LORA parameters of stablediffusion into specific parameters and pass them into the stablediffusion model to achieve simpler control.

[0063] Further, refer to Figure 4 , generate a picture and send the picture to the child end, and send the picture to the stable diffusion model after image enhancement, semantic segmentation and feature encoding, including: creating a new white canvas with a cartoon interaction button in the lower left corner of the white canvas; judging whether there is user voice input / graffiti input; if there is user voice input / graffiti input, judging whether it is a color in the preset number of enumerations said separately, if so, modifying the brush color, if not, judging whether it is a style in the preset number of enumerations said separately; if it is a style in the enumeration, modifying the checkpoint; if it is not a style in the enumeration, filtering the prompt word information from the voice input content and starting to generate pictures after the voice / graffiti input pauses for 5 seconds.

[0064] Furthermore, children's voice monitoring is performed on the surrounding environment and STT audio signals are obtained, including: obtaining a large amount of audio annotation data and using the audio annotation data for targeted training to obtain STT audio signals; the children's voice monitoring includes voice detection, echo cancellation, sound source localization, and sound enhancement, and the STT audio signal is obtained through feature extraction, acoustic model, audio decoding, and semantic analysis.

[0065] Targeted training through reinforcement learning using labeled data enables AI to accurately distinguish who is speaking to "it", automatically judge and wake up the service, and enable real-time voice interaction with AI without having to call out its name.

[0066] Furthermore, the AIGC big model also includes a LangChain framework, and the memory module in the LangChain framework is used to enhance the model memory; the multimodal LLM model sends the interactive dialogue to the parent's mobile terminal in real time, and the parent's mobile terminal can send data information to the child's terminal to insert it into the dialogue in real time; the AIGC big model continues to interact with the child based on the content inserted by the parent.

[0067] The memory module in the Langchain framework is used to enhance model memory, allowing AI to truly accompany children as a parent's clone. Parents can remotely see the conversations and interactions between the AI ​​clone and their children on the mobile terminal, and can insert themselves into the conversation at any time. The AI ​​clone will continue to interact with the child based on the content inserted by the parents.

[0068] In addition, an embodiment of the present invention also provides an AI-based minor companionship device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of an AI-based minor companionship method as described in any of the above items.

[0069] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of an AI-based method for accompanying minors as described in any of the above items are implemented.

[0070] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0071] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0072] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0073] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0074] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0075] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0076] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0077] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for accompanying minors based on AI, characterized in that: The method is applied to the AIGC large model, which includes: S1. Monitor children's voices in the surrounding environment and obtain STT audio signals, extract text data from the STT audio signals and input them into a preset stable diffusion model; S2. Use the preset stablediffusion model to build the underlying structure of the large model and add the RAG framework on the engineering side, obtain the labeled data and use the labeled data to perform supervised training on the preset stablediffusion model, and use prompt to tune the training results; S3, after performing modal fusion and prompt tuning on the text data through the trained stable diffusion model, the text tuning result is input into the multimodal LLM model, and at the same time, the multimodal LLM model obtains the parent history information and generates a natural language text according to the parent history information; S4. Use the natural language text and acoustic model to synthesize the TTS audio signal of the AI ​​avatar that is consistent with the parent's voice and output it.

2. The AI-based method for accompanying minors as claimed in claim 1, characterized in that: The method further comprises: After receiving the text optimization result, the multimodal LLM model determines whether it is necessary to generate a picture; If it is not necessary to generate an image, obtaining the parent's historical information and generating a natural language text according to the parent's historical information; If it is necessary to generate a picture, the picture is generated and sent to the child's end. At the same time, the picture is sent to the stable diffusion model after image enhancement, semantic segmentation and feature encoding.

3. The AI-based method for accompanying minors as claimed in claim 2, characterized in that: Generate an image and send it to the child terminal. At the same time, send the image to the stable diffusion model after image enhancement, semantic segmentation and feature encoding, including: Create a new white canvas with a cartoon interaction button at the lower left corner of the canvas; Determine whether there is user voice input / graffiti input; If there is user voice input / graffiti input, determine whether it is a color in the preset number of enumerations that is spoken alone, if yes, modify the brush color, if not, determine whether it is a style in the preset number of enumerations that is spoken alone; If it is a style in the enumeration, modify the checkpoint; If it is not a style in the enumeration, the prompt word information is filtered out from the voice input content and the image is generated after the voice / graffiti input pauses for 5 seconds.

4. The AI-based method for accompanying minors as claimed in claim 3, characterized in that: Monitor children's voices in the surrounding environment and obtain STT audio signals, including: Acquire a large amount of audio annotation data and use the audio annotation data for targeted training to obtain STT audio signals; The children's voice monitoring includes voice detection, echo cancellation, sound source localization, and sound enhancement, and the STT audio signal is obtained through feature extraction, acoustic model, audio decoding, and semantic analysis.

5. The AI-based method for accompanying minors as claimed in claim 4, characterized in that: The AIGC big model also includes a LangChain framework, and the memory module in the LangChain framework is used to enhance the model memory; The multimodal LLM model sends the interactive dialogue to the parent's mobile terminal in real time, and the parent's mobile terminal can send data information to the child's terminal to insert it into the dialogue in real time; The AIGC macro model continues to interact with the child based on the content inserted by the parent.

6. An AI-based juvenile companionship system, characterized in that: The system comprises: A voice monitoring module, used to monitor the voices of minors in the surrounding environment and obtain STT audio signals, extract text data from the STT audio signals and input them into a preset stable diffusion model; The training and tuning module is used to build the underlying structure of the large model using the preset stable diffusion model and add the RAG framework on the engineering side, obtain the labeled data and use the labeled data to perform supervised training on the preset stable diffusion model, and use prompt to tune the training results; A natural language text generation module is used to perform modal fusion and prompt optimization on the text data through a trained stable diffusion model, and then input the text optimization result into a multimodal LLM model. Meanwhile, the multimodal LLM model obtains parent history information and generates natural language text according to the parent history information; The voice output module is used to synthesize and output a TTS audio signal of the AI ​​avatar that is consistent with the parent's voice using the natural language text and the acoustic model.

7. An AI-based companion device for minors, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of an AI-based method for accompanying minors as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an AI-based method for accompanying minors as described in any one of claims 1 to 5.

Images