A desktop device interaction method and system based on a local vector knowledge base

Abstract

This application discloses a desktop device interaction method and system based on a local vector knowledge base. The method includes: retrieving related question texts from a pre-built local vector knowledge base based on user question text; sending the user question text and related question texts to a cloud server to obtain an answer text generated and fed back by the cloud server based on the user question text and related question texts; generating digital human driving parameters based on the answer text, and sending the answer text and digital human driving parameters to the cloud server to obtain digital human audiovisual interactive content generated and fed back by the cloud server based on the answer text and digital human driving parameters; and playing the digital human audiovisual interactive content. This technical solution eliminates the need to upload user privacy data to a cloud server, reducing the risk of privacy leakage and improving the cloud server's efficiency in generating digital human audiovisual interactive content, thereby reducing interaction response latency.

Description

Technical Field

[0001] This application belongs to the field of electronic digital data processing technology, specifically relating to a desktop device interaction method and system based on a local vector knowledge base. Background Technology

[0002] Desktop devices are intelligent human-computer interaction terminals designed for office or home scenarios, primarily used on desktops, and supporting multimodal interaction. Examples include desktop speakers and desktop control screens. Through multimodal interaction, they can efficiently fulfill diverse needs such as information retrieval, schedule management, task collaboration, smart home control, and remote communication and collaboration, significantly improving office efficiency and the level of intelligence in home life.

[0003] Current desktop devices generally adopt a full-chain cloud processing architecture: the cloud server processes user questions uploaded by the desktop device sequentially through semantic understanding, knowledge retrieval, and content generation, and finally returns the interactive content to the desktop device for playback. While this architecture fully utilizes the powerful computing capabilities of the cloud server, core information closely related to the user must be stored in the cloud server for knowledge retrieval, posing risks to user privacy and security. Moreover, with the increasing complexity of interactive content, especially the widespread adoption of digital human interaction modes, the complete generation of interactive content by the cloud server significantly increases the computing power consumption and processing pressure on the cloud server, reducing the efficiency of interactive content generation and thus lengthening the interaction response latency. Summary of the Invention

[0004] This application provides a desktop device interaction method and system based on a local vector knowledge base. The purpose is to reduce the risk of privacy leakage by retrieving user privacy data locally without uploading it to a cloud server, and to improve the efficiency of cloud server in generating digital human audiovisual interactive content by generating digital human driving parameters locally first, thereby reducing interaction response latency.

[0005] Firstly, this application provides a desktop device interaction method based on a local vector knowledge base, the method comprising:

[0006] Obtain the user's question text, and search the pre-built local vector knowledge base based on the user's question text to obtain the associated question text corresponding to the user's question text;

[0007] The user's question text and the associated question text are sent to the cloud server, whereby the cloud server generates and responds with a response text based on the user's question text and the associated question text.

[0008] Upon receiving the response text, digital human driving parameters are generated based on the response text, and the response text and the digital human driving parameters are sent to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters.

[0009] Upon receiving the digital human audiovisual interactive content, play the digital human audiovisual interactive content.

[0010] Optionally, the digital human driving parameters include a lip-shape time series;

[0011] Accordingly, the generation of digital human driving parameters based on the answer text includes:

[0012] The response text is segmented to obtain multiple word units, and a phoneme time series is generated based on the multiple word units; wherein, the phoneme time series includes multiple phonemes and the timestamps and durations corresponding to the phonemes;

[0013] Based on the pre-constructed mapping relationship between phonemes and lip shape identifiers, each phoneme in the phoneme time series is mapped to a lip shape identifier, and the lip shape identifiers, along with the timestamps and durations corresponding to the phonemes, are integrated to obtain the basic lip shape time series.

[0014] The basic lip shape time series is subjected to a smooth transition processing to obtain the lip shape time series.

[0015] Optionally, after sending the user's question text and the associated question text to the cloud server, the method further includes:

[0016] Receive sentiment description information generated and fed back by the cloud server based on the user's question text and the associated question text;

[0017] Accordingly, the smoothing process performed on the basic lip shape time series to obtain the lip shape time series includes:

[0018] The target transition frame density and the target interpolation motion curve are determined based on the emotional description information.

[0019] The base lip-shaped time series is smoothed according to the target transition frame density and the target interpolation motion curve to obtain the lip-shaped time series.

[0020] Optionally, the digital human driving parameters may further include a head-nodding action time series;

[0021] Accordingly, after segmenting the response text into multiple word units, the method further includes:

[0022] Based on multiple word units, prosodic pause information and prosodic stress information are determined;

[0023] A nodding action sequence is generated based on the prosodic pause information and the prosodic stress information; wherein, the nodding action sequence includes the word unit and the nodding action description information corresponding to the word unit;

[0024] Accordingly, after performing smoothing transition processing on the basic lip shape time series to obtain the lip shape time series, the method further includes:

[0025] Based on the word units in the nodding action sequence, the nodding action description information is time-aligned with the lip shape time sequence to obtain a nodding action time sequence, which is used to drive the digital human to synchronously complete lip shape changes and nodding actions with the lip shape time sequence.

[0026] Optionally, the process of pre-building a local vector knowledge base includes:

[0027] Obtain a preset question document and store it in a local encrypted storage area. Then, parse the preset question document locally to obtain the preset question text.

[0028] The preset question text is segmented to obtain multiple related question texts;

[0029] The multiple associated question texts are converted into associated question vectors, and the associated question vectors and their corresponding associated question texts are associated and stored to obtain a local vector knowledge base.

[0030] Optionally, before segmenting the preset question text to obtain multiple related question texts, the method further includes:

[0031] Obtain core issue keywords and user scenario tags;

[0032] Accordingly, the preset question text is segmented to obtain multiple related question texts, including:

[0033] The preset question text is segmented into scene-related paragraph text based on the core question keywords; wherein, the scene-related paragraph text includes at least one core question keyword;

[0034] The context-related paragraph text is segmented based on the user context tags to obtain multiple related question texts.

[0035] Optionally, the step of segmenting the scene-related paragraph text according to the user scene tags to obtain multiple related question texts includes:

[0036] Based on the user scenario tags and the preset semantic segmentation rule library, the target semantic segmentation rule is determined; wherein, the preset semantic segmentation rule library includes multiple user scenario tags and the question sentence features, punctuation weights and semantic pause weights corresponding to the user scenario tags;

[0037] The scene-related paragraph text is segmented according to the question's sentence structure characteristics to obtain short sentence text;

[0038] Calculate the similarity between two adjacent short sentence texts, and calculate the splitting determination coefficient of the two adjacent short sentence texts based on the similarity, the punctuation mark weight, and the semantic pause weight;

[0039] The two adjacent short sentence texts whose splitting determination coefficient exceeds the preset splitting critical threshold are merged to obtain multiple related question texts.

[0040] Secondly, this application provides a desktop device interaction system based on a local vector knowledge base, the system comprising:

[0041] The question text acquisition module is used to acquire user question text and search in a pre-built local vector knowledge base based on the user question text to obtain the associated question text corresponding to the user question text;

[0042] The answer text generation module is used to send the user question text and the associated question text to the cloud server, and the cloud server generates and returns answer text based on the user question text and the associated question text;

[0043] The interactive content generation module is used to generate digital human driving parameters based on the received response text, and send the response text and the digital human driving parameters to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters.

[0044] The interactive content playback module is used to play the digital human audiovisual interactive content when the digital human audiovisual interactive content is received.

[0045] Optionally, the digital human driving parameters include a lip-shape time series;

[0046] Accordingly, the interactive content generation module is specifically used for:

[0047] The response text is segmented to obtain multiple word units, and a phoneme time series is generated based on the multiple word units; wherein, the phoneme time series includes multiple phonemes and the timestamps and durations corresponding to the phonemes;

[0048] Based on the pre-constructed mapping relationship between phonemes and lip shape identifiers, each phoneme in the phoneme time series is mapped to a lip shape identifier, and the lip shape identifiers, along with the timestamps and durations corresponding to the phonemes, are integrated to obtain the basic lip shape time series.

[0049] The basic lip shape time series is subjected to a smooth transition processing to obtain the lip shape time series.

[0050] Optionally, the interactive content generation module is further configured to:

[0051] Receive sentiment description information generated and fed back by the cloud server based on the user's question text and the associated question text;

[0052] The target transition frame density and the target interpolation motion curve are determined based on the emotional description information.

[0053] The base lip-shaped time series is smoothed according to the target transition frame density and the target interpolation motion curve to obtain the lip-shaped time series.

[0054] Optionally, the digital human driving parameters may further include a head-nodding action time series;

[0055] Accordingly, the interactive content generation module is also used for:

[0056] Based on multiple word units, prosodic pause information and prosodic stress information are determined;

[0057] A nodding action sequence is generated based on the prosodic pause information and the prosodic stress information; wherein, the nodding action sequence includes the word unit and the nodding action description information corresponding to the word unit;

[0058] Based on the word units in the nodding action sequence, the nodding action description information is time-aligned with the lip shape time sequence to obtain a nodding action time sequence, which is used to drive the digital human to synchronously complete lip shape changes and nodding actions with the lip shape time sequence.

[0059] Optionally, the system is also used for:

[0060] Obtain a preset question document and store it in a local encrypted storage area. Then, parse the preset question document locally to obtain the preset question text.

[0061] The preset question text is segmented to obtain multiple related question texts;

[0062] The multiple associated question texts are converted into associated question vectors, and the associated question vectors and their corresponding associated question texts are associated and stored to obtain a local vector knowledge base.

[0063] Optionally, the system is also used for:

[0064] Obtain core issue keywords and user scenario tags;

[0065] Accordingly, the preset question text is segmented to obtain multiple related question texts, including:

[0066] The preset question text is segmented into scene-related paragraph text based on the core question keywords; wherein, the scene-related paragraph text includes at least one core question keyword;

[0067] The context-related paragraph text is segmented based on the user context tags to obtain multiple related question texts.

[0068] Optionally, the system is also used for:

[0069] Based on the user scenario tags and the preset semantic segmentation rule library, the target semantic segmentation rule is determined; wherein, the preset semantic segmentation rule library includes multiple user scenario tags and the question sentence features, punctuation weights and semantic pause weights corresponding to the user scenario tags;

[0070] The scene-related paragraph text is segmented according to the question's sentence structure characteristics to obtain short sentence text;

[0071] Calculate the similarity between two adjacent short sentence texts, and calculate the splitting determination coefficient of the two adjacent short sentence texts based on the similarity, the punctuation mark weight, and the semantic pause weight;

[0072] The two adjacent short sentence texts whose splitting determination coefficient exceeds the preset splitting critical threshold are merged to obtain multiple related question texts.

[0073] Thirdly, this application provides an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method described in the first aspect.

[0074] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the method described in the first aspect.

[0075] In this application, a user question text is obtained, and a search is performed in a pre-built local vector knowledge base based on the user question text to obtain associated question texts corresponding to the user question text. The user question text and the associated question texts are then sent to a cloud server, whereby the cloud server generates and responds with an answer text based on the user question text and the associated question texts. Upon receiving the answer text, digital human driving parameters are generated based on the answer text, and the answer text and the digital human driving parameters are sent to the cloud server, whereby the cloud server generates and responds with digital human audiovisual interactive content based on the answer text and the digital human driving parameters. Upon receiving the digital human audiovisual interactive content, the digital human audiovisual interactive content is played. This desktop device interaction method based on a local vector knowledge base reduces the risk of privacy leakage by eliminating the need to upload user privacy data to a cloud server through local vectorized knowledge retrieval, and improves the efficiency of the cloud server in generating digital human audiovisual interactive content by generating digital human driving parameters locally first, thereby reducing interaction response latency. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating a desktop device interaction method based on a local vector knowledge base provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram of the hardware structure of a desktop device provided in an embodiment of this application;

[0078] Figure 3 This is a flowchart illustrating another desktop device interaction method based on a local vector knowledge base provided in an embodiment of this application;

[0079] Figure 4 This is a flowchart illustrating a process for pre-building a local vector knowledge base, as provided in an embodiment of this application.

[0080] Figure 5 This is a schematic diagram of the structure of a desktop device interaction system based on a local vector knowledge base provided in an embodiment of this application;

[0081] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0083] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0084] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0085] The desktop device interaction method and system based on a local vector knowledge base provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0086] First, this application applies to scenarios involving human-computer interaction using desktop devices. Based on the aforementioned usage scenarios, it is understood that the executing entity of this application can be the aforementioned desktop device.

[0087] Figure 1 This is a flowchart illustrating a desktop device interaction method based on a local vector knowledge base, as provided in an embodiment of this application. Figure 1 As shown, the specific steps include the following:

[0088] S101, obtain the user's question text, and search in a pre-built local vector knowledge base based on the user's question text to obtain the associated question text corresponding to the user's question text.

[0089] Desktop devices can be intelligent human-computer interaction terminals designed for office or home scenarios, primarily used for desktop placement, and supporting multimodal interaction, such as desktop speakers and desktop control screens.

[0090] Figure 2 This is a schematic diagram of the hardware structure of a desktop device provided in an embodiment of this application. For example... Figure 2 As shown, the desktop device includes a local processing module 210, an audio acquisition module 220, a display output module 230, and a cloud communication module 240. Specifically, the local processing module may include an RK3588 processor, paired with 4GB of LPDDR4 memory and 64GB of eMMC storage; the audio acquisition module may include a microphone and speakers; the display output module supports a 1920×1200 resolution and a 60Hz refresh rate; the cloud communication module supports the WiFi 6 protocol, with a transmission rate of up to 1.2Gbps and a latency controlled within 20ms, and is also equipped with an Ethernet interface; the local processing module connects to the cloud communication module via PCIe, to the audio acquisition module via I2S, and to the display output module via HDMI 2.0.

[0091] Among them, user question text can be natural language text proposed by users based on their actual needs.

[0092] In one embodiment, the method for obtaining user question text can be to send the received user input voice command to the cloud server through the cloud communication module, and the ASR (Automatic Speech Recognition) function unit of the cloud server generates and feeds back the user question text based on the user input voice command.

[0093] Among them, the user input voice command can be an audio signal in which the user verbally expresses their needs in natural language, which can be acquired through the microphone in the audio acquisition module; the cloud server can be an edge cloud server or a public cloud server with an artificial intelligence computing power cluster; the ASR functional unit can be an end-to-end speech recognition model based on the Transformer architecture.

[0094] The local vector knowledge base can be a lightweight semantic index database deployed on the local storage medium of a desktop device, used to store associated question vectors and their corresponding associated question texts. Specifically, the associated question vectors can be vectors obtained by converting the associated question texts into vector format; the associated question texts can be a set of high-frequency question texts covering the core requirements of the target scenario.

[0095] In one embodiment, the method of retrieving the associated question text corresponding to the user question text from a pre-built local vector knowledge base based on the user question text can be achieved by converting the user question text into a vector format to obtain a user question vector (e.g., a 384-dimensional vector), calculating the similarity between the user question vector and each associated question vector in the local vector knowledge base, and determining the associated question text corresponding to the three associated question vectors with the highest similarity as the associated question text corresponding to the user question text.

[0096] S102, the user question text and the associated question text are sent to the cloud server, so that the cloud server can generate and provide a response text based on the user question text and the associated question text.

[0097] In one embodiment, the method of sending the user's question text and related question text to the cloud server can be to combine the user's question text and related question text into JSON format data, and then send the JSON format data to the cloud server through the cloud communication module.

[0098] The answer text can be natural language text generated based on the user's question text and related question texts, used to answer the user's question text.

[0099] In one embodiment, the cloud server generates and returns the answer text based on the user's question text and related question text. This can be achieved by the LLM (Large Language Model) functional unit in the cloud server generating and returning the answer text based on the user's question text and related question text.

[0100] Among them, the LLM functional unit can be a dedicated large model based on Llama 2 or GPT-4 fine-tuning.

[0101] S103, upon receiving the response text, generating digital human driving parameters based on the response text, and sending the response text and the digital human driving parameters to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters.

[0102] In one embodiment, the response text can be received by receiving the response text sent by the cloud server through a cloud communication module.

[0103] Among them, a digital human can be a virtual image with an anthropomorphic appearance and facial expressions and movements; digital human driving parameters can be a set of structured parameters used to drive the digital human to perform anthropomorphic interactions.

[0104] In one embodiment, the method of generating digital human driving parameters based on the response text can be achieved by extracting the dialogue intent, sentiment tendency, and core keywords of the response text through a local lightweight NLP (Natural Language Processing) model. Based on the extracted dialogue intent, sentiment tendency, and core keywords, as well as the pre-built association between dialogue intent, sentiment tendency, core keywords, and digital human driving parameters, the digital human driving parameters corresponding to the current response text are determined.

[0105] In one embodiment, the method of sending the response text and digital human driving parameters to the cloud server can be to combine the response text and digital human driving parameters into a preset format data, and then send the preset format data to the cloud server through a cloud communication module.

[0106] Among them, digital human audiovisual interactive content can be multimodal interactive content with anthropomorphic expression characteristics and precise audio-visual synchronization, including digital human video signals and digital human audio signals.

[0107] In one embodiment, the cloud server generates and feeds back digital human audiovisual interactive content based on the response text and digital human driving parameters. This can be achieved by the TTS (Text-to-Speech) function unit in the cloud server generating and feeding back digital human audio signals based on the response text, and the digital human rendering function unit in the cloud server generating and feeding back digital human video signals based on the response text and digital human driving parameters.

[0108] Among them, the TTS functional unit can be a high-fidelity speech synthesis model based on VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech); the digital human rendering functional unit can be a cloud computing service based on a real-time graphics rendering engine and an AI-driven anthropomorphic content generation unit, capable of generating digital human video signals based on the answer text and digital human driving parameters.

[0109] S104, Upon receiving the digital human audiovisual interactive content, play the digital human audiovisual interactive content.

[0110] In one embodiment, the method of receiving digital human audiovisual interactive content can be by receiving digital human audiovisual interactive content (digital human video signal and digital human audio signal) sent by a cloud server through a cloud communication module.

[0111] In one embodiment, the method of playing digital human audiovisual interactive content can be to perform time alignment processing on the received digital human video signal and digital human audio signal, play the digital human audio signal through the speakers in the audio acquisition module, and play the digital human video signal through the display output module.

[0112] In this embodiment, a user question text is obtained, and a search is performed in a pre-built local vector knowledge base based on the user question text to obtain associated question texts corresponding to the user question text. The user question text and the associated question texts are then sent to a cloud server, whereby the cloud server generates and responds with an answer text based on the user question text and the associated question texts. Upon receiving the answer text, digital human driving parameters are generated based on the answer text, and the answer text and the digital human driving parameters are sent to the cloud server, whereby the cloud server generates and responds with digital human audiovisual interactive content based on the answer text and the digital human driving parameters. Upon receiving the digital human audiovisual interactive content, the digital human audiovisual interactive content is played. This desktop device interaction method based on a local vector knowledge base reduces the risk of privacy leakage by eliminating the need to upload user privacy data to a cloud server through local vectorized knowledge retrieval, and improves the efficiency of the cloud server in generating digital human audiovisual interactive content by generating digital human driving parameters locally, thereby reducing interaction response latency.

[0113] Figure 3 This is a flowchart illustrating another desktop device interaction method based on a local vector knowledge base provided in an embodiment of this application. For example... Figure 3 As shown, the specific steps include the following:

[0114] S301, Obtain the user's question text, and search in a pre-built local vector knowledge base based on the user's question text to obtain the associated question text corresponding to the user's question text.

[0115] S302, the user question text and the associated question text are sent to the cloud server, so that the cloud server can generate and provide a response text based on the user question text and the associated question text.

[0116] S303, upon receiving the response text, the response text is segmented to obtain multiple word units, and a phoneme time sequence is generated based on the multiple word units; wherein, the phoneme time sequence includes multiple phonemes and the timestamps and durations corresponding to the phonemes.

[0117] Among them, a word unit can be the smallest linguistic unit obtained by splitting the response text according to semantic integrity.

[0118] In one embodiment, the method of segmenting the answer text to obtain multiple word units can be achieved by using a local lightweight Chinese word segmentation model to segment the answer text to obtain multiple word units.

[0119] The phoneme time series can be a structured data sequence used to characterize the temporal features of pronunciation in the response text, and can include multiple phonemes and their corresponding timestamps and durations. Specifically, a phoneme can be the basic unit of pronunciation in Mandarin Chinese; a timestamp can be the absolute time value at which the phoneme begins to be pronounced; and the duration can be the duration of pronunciation of a single phoneme.

[0120] In one embodiment, the method of generating a phoneme time series based on multiple word units can be achieved by converting each word unit into pinyin with tone using a local pinyin conversion library (e.g., pypinyin-mobile), splitting the pinyin into corresponding phonemes based on the Mandarin phoneme mapping rules, assigning a duration to each phoneme according to a preset speech rate, and calculating the timestamp corresponding to each phoneme based on the duration of each phoneme and the preset pause duration corresponding to the pause punctuation, thereby obtaining the phoneme time series.

[0121] S304, based on the pre-constructed mapping relationship between phonemes and lip shape identifiers, each phoneme in the phoneme time series is mapped to a lip shape identifier, and the lip shape identifiers and the timestamps and durations corresponding to the phonemes are integrated to obtain the basic lip shape time series.

[0122] Among them, lip shape identifiers can be standardized codes used to uniquely identify specific lip shape states of digital humans; the mapping relationship between phonemes and lip shape identifiers can be a one-to-one or many-to-one mapping table between phonemes and lip shape identifiers constructed based on the physiology of speech articulation. By collecting lip shape data during real-person pronunciation, the association between phonemes and lip shape states can be established, thereby combining the lip shape identifiers assigned to each lip shape state to establish the mapping relationship between phonemes and lip shape identifiers.

[0123] In one embodiment, each phoneme in the phoneme time series is mapped to a lip shape identifier according to the pre-built mapping relationship between phonemes and lip shape identifiers. This can be done by querying the stored data of the pre-built mapping relationship between phonemes and lip shape identifiers using phonemes as the query condition. The query results include the lip shape identifiers corresponding to each phoneme.

[0124] The basic lip-shape time series can be a time series data sequence obtained by structurally integrating the lip shape identifiers, timestamps, and durations corresponding to phonemes.

[0125] In one embodiment, the method of integrating lip shape identifiers and the timestamps and durations corresponding to phonemes to obtain a basic lip shape time series can be achieved by structurally encapsulating the lip shape identifiers and the timestamps and durations corresponding to phonemes in the ProtoBuf format to obtain the basic lip shape time series.

[0126] S305, perform smooth transition processing on the basic lip shape time series to obtain the lip shape time series.

[0127] In one embodiment, the method of smoothing the transition of the basic lip shape time series to obtain the lip shape time series can be achieved by extracting lip shape parameters (such as opening degree, lip thickness, and corner angle) corresponding to two adjacent lip shape identifiers in the basic lip shape time series, generating transition lip shape parameters based on the lip shape parameters corresponding to the two adjacent lip shape identifiers, selecting lip shape identifiers that match the transition lip shape parameters, calculating the average of the timestamps of the two adjacent lip shape identifiers as the timestamp of the transition lip shape identifier, updating the duration of the previous original lip shape identifier based on the timestamp of the transition lip shape identifier, and determining the duration of the transition lip shape identifier to obtain the lip shape time series.

[0128] Optionally, after sending the user's question text and the associated question text to the cloud server, the method further includes:

[0129] Receive sentiment description information generated and fed back by the cloud server based on the user's question text and the associated question text;

[0130] Accordingly, the smoothing process performed on the basic lip shape time series to obtain the lip shape time series includes:

[0131] The target transition frame density and the target interpolation motion curve are determined based on the emotional description information.

[0132] The base lip-shaped time series is smoothed according to the target transition frame density and the target interpolation motion curve to obtain the lip-shaped time series.

[0133] The emotional description information can be structured information generated by a cloud server based on the user's question intent and the semantics of the answer text, used to characterize the emotional tendency and intensity of digital human interaction, and may include emotional type labels and emotional intensity values.

[0134] In one embodiment, the cloud server generates and feeds back sentiment description information based on the user's question text and related question text. This can be achieved by having the LLM functional unit in the cloud server first generate the answer text based on the user's question text and related question text, and then generate and feed back sentiment description information based on the user's question text and answer text.

[0135] In one embodiment, the emotional description information can be received by receiving the emotional description information sent by the cloud server through a cloud communication module.

[0136] Among them, the target transition frame density can be the core parameter used to control the smoothness of the lip transition, that is, the number of transition frames inserted between two adjacent original lip markers; the target interpolation motion curve can be a curve describing the change law of the lip parameters from the current state to the next state.

[0137] In one embodiment, the method for determining the target transition frame density and the target interpolation motion curve based on the sentiment description information can be to predefine the mapping relationship between the sentiment description information, the transition frame density, and the interpolation motion curve, and then determine the target transition frame density and the target interpolation motion curve based on the current sentiment description information and the mapping relationship.

[0138] In one embodiment, the method of smoothing the basic lip shape time series according to the target transition frame density and the target interpolation motion curve to obtain the lip shape time series can be achieved by extracting the lip shape parameters (such as opening degree, lip thickness, and corner angle) corresponding to two adjacent lip shape identifiers in the basic lip shape time series, calculating the number of transition frames to be inserted between two adjacent lip shape identifiers based on the target transition frame density, calculating the lip shape parameters of each transition frame based on the target interpolation motion curve and the lip shape parameters corresponding to two adjacent lip shape identifiers through a curve fitting algorithm and selecting the lip shape identifier that matches the lip shape parameters of the transition frame, determining the duration corresponding to each transition lip shape identifier based on the timestamps of two adjacent lip shape identifiers and the number of transition frames, and determining the timestamp corresponding to each transition lip shape identifier and the duration of the previous original lip shape identifier based on the duration corresponding to each transition lip shape identifier, thus obtaining the lip shape time series.

[0139] The advantage of this approach is that by introducing emotional description information to drive the dynamic adaptation of lip-shape transition parameters, the changes in the digital human's mouth shape not only match the speech pronunciation logic but are also deeply bound to the interactive emotions, significantly improving the digital human's ability to convey emotions and its anthropomorphism.

[0140] Optionally, the digital human driving parameters may further include a head-nodding action time series;

[0141] Accordingly, after segmenting the response text into multiple word units, the method further includes:

[0142] Based on multiple word units, prosodic pause information and prosodic stress information are determined;

[0143] A nodding action sequence is generated based on the prosodic pause information and the prosodic stress information; wherein, the nodding action sequence includes the word unit and the nodding action description information corresponding to the word unit;

[0144] Accordingly, after performing smoothing transition processing on the basic lip shape time series to obtain the lip shape time series, the method further includes:

[0145] Based on the word units in the nodding action sequence, the nodding action description information is time-aligned with the lip shape time sequence to obtain a nodding action time sequence, which is used to drive the digital human to synchronously complete lip shape changes and nodding actions with the lip shape time sequence.

[0146] Among them, prosodic pause information can be structured information that represents the pause position and duration when the answer text is read aloud, including the pause position and duration following word units; prosodic stress information can be information that represents the word units that need to be emphasized in the answer text and the intensity of the emphasis, including stress position and stress level.

[0147] In one embodiment, the method for determining prosodic pause information and prosodic stress information based on multiple word units can be achieved by analyzing word units using a local lightweight prosodic analysis model (e.g., a Chinese prosodic prediction model based on convolutional neural networks) to obtain prosodic pause information and prosodic stress information.

[0148] The nodding sequence can be a structured dataset of associated word units and nodding action features, including word units and corresponding nodding action descriptions. The nodding action descriptions can be core parameters that quantify the nodding action, including nodding amplitude and speed.

[0149] In one embodiment, the method of generating a nodding action sequence based on prosodic pause information and prosodic stress information can be achieved by pre-defining the mapping relationship between prosodic pause information, prosodic stress information and nodding action description information, and determining the nodding action description information corresponding to each word unit based on the current prosodic pause information, prosodic stress information and the mapping relationship, thereby obtaining the nodding action sequence.

[0150] The nodding action time series can be time-driven data obtained by precisely aligning the nodding action description information with the time axis of the lip-shape time series, and can include the nodding action description information and the timestamp corresponding to the nodding action description information.

[0151] In one embodiment, the method of aligning the nodding action description information to the lip-sync time sequence based on the word units in the nodding action sequence to obtain the nodding action time sequence can be achieved by extracting the phoneme timestamp range corresponding to each word unit in the lip-sync time sequence, determining the starting point of the phoneme timestamp range as the timestamp corresponding to the nodding action description information of that word unit, and integrating the nodding action description information and its corresponding timestamp to obtain the nodding action time sequence.

[0152] The advantage of this approach is that by driving the nodding motion through prosodic features and precisely aligning it with lip movements, the digital human not only achieves lip-sync but also possesses body language such as stressed nodding and nodding after pauses, which conform to human communication habits, significantly improving the naturalness of anthropomorphic expression and the efficiency of information transmission.

[0153] S306, the response text and the digital human driving parameters are sent to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters.

[0154] S307, Upon receiving the digital human audiovisual interactive content, play the digital human audiovisual interactive content.

[0155] The advantage of this solution is that it accurately matches the physiological characteristics and temporal logic of human speech, achieving naturalness, synchronization, and adaptability in digital lip-reading expression, while also taking into account the local processing capabilities and interactive experience requirements of desktop devices.

[0156] Figure 4 This is a flowchart illustrating a process for pre-building a local vector knowledge base, as provided in an embodiment of this application. Figure 4 As shown, the specific steps include the following:

[0157] S401, Obtain a preset question document and store the preset question document in a local encrypted storage area, and perform local parsing on the preset question document to obtain the preset question text.

[0158] The preset question document can be a structured knowledge document set by the user that covers the core needs of the target scenario. It includes frequently asked questions by users in the target scenario and corresponding background information. The preset question document can be a PDF, Word, TXT or other format files.

[0159] In one embodiment, the preset problem document can be obtained by having the user import the preset problem document via screen touch or USB interface.

[0160] The local encrypted storage area can be an independent encrypted partition within the eMMC (Embedded Multi-Media Card) storage medium of a desktop device.

[0161] In one embodiment, storing a preset problem document in a local encrypted storage area can be achieved by encrypting the preset problem document in real time using the AES-256 encryption algorithm and then storing the encrypted preset problem document in the local encrypted storage area.

[0162] The preset question text can be a collection of plain text content extracted from a preset question document.

[0163] In one embodiment, the method of parsing a preset issue document locally to obtain the preset issue text can be achieved by using a lightweight parsing library corresponding to the document format of the preset issue document to parse the preset issue document locally and obtain the preset issue text.

[0164] S402, the preset question text is segmented to obtain multiple related question texts.

[0165] In one embodiment, the method of segmenting a preset question text to obtain multiple related question texts can be achieved by using a Langchain-based RecursiveCharacterTextSplitter to segment the preset question text according to a preset segmentation size, thereby obtaining multiple related question texts.

[0166] Among them, Langchain can be an open-source framework for building large language model applications, providing a wealth of text processing and data integration tools. Its core advantages lie in its modular design and lightweight adaptability, supporting the deployment of core functional modules in embedded devices (such as desktop devices) without the need for a complete framework installation; RecursiveCharacterTextSplitter can be a recursive character text splitter in the Langchain framework.

[0167] Optionally, before segmenting the preset question text to obtain multiple related question texts, the method further includes:

[0168] Obtain core issue keywords and user scenario tags;

[0169] Accordingly, the preset question text is segmented to obtain multiple related question texts, including:

[0170] The preset question text is segmented into scene-related paragraph text based on the core question keywords; wherein, the scene-related paragraph text includes at least one core question keyword;

[0171] The context-related paragraph text is segmented based on the user context tags to obtain multiple related question texts.

[0172] Among them, core issue keywords can be high-frequency words, professional terms, or key concepts that characterize the core needs of the target scenario; user scenario tags can be standardized identifiers used to clarify the usage scenario of desktop devices, such as home, office, and education.

[0173] In one embodiment, the core issue keywords and user scenario tags can be obtained by having the user import the core issue keywords and user scenario tags via screen touch or USB interface.

[0174] Among them, the scene-related paragraph text can be a semantically coherent paragraph text divided from the preset question text based on the core question keywords, and the scene-related paragraph text should include at least one core question keyword.

[0175] In one embodiment, the method of segmenting the preset question text into scene-related paragraph texts based on core question keywords can be achieved by traversing the preset question text using a string matching algorithm to locate the occurrence positions of all core question keywords. Taking the sentence containing the core question keyword as the center, the algorithm extends forward and backward to the semantic integrity boundary (e.g., strong semantic separators such as paragraph separators, periods, exclamation marks, etc.) to obtain the initial paragraph text. Adjacent initial paragraph texts containing the same core question keyword are then merged to finally obtain multiple scene-related paragraph texts.

[0176] In one embodiment, the method of segmenting scene-related paragraph text according to user scene tags to obtain multiple related question texts can be achieved by determining a target preset segmentation size based on the current user scene tags and a preset segmentation size predefined for each user scene tag, and then segmenting each scene-related paragraph text according to the preset segmentation size to obtain multiple related question texts.

[0177] Optionally, the step of segmenting the scene-related paragraph text according to the user scene tags to obtain multiple related question texts includes:

[0178] Based on the user scenario tags and the preset semantic segmentation rule library, the target semantic segmentation rule is determined; wherein, the preset semantic segmentation rule library includes multiple user scenario tags and the question sentence features, punctuation weights and semantic pause weights corresponding to the user scenario tags;

[0179] The scene-related paragraph text is segmented according to the question's sentence structure characteristics to obtain short sentence text;

[0180] Calculate the similarity between two adjacent short sentence texts, and calculate the splitting determination coefficient of the two adjacent short sentence texts based on the similarity, the punctuation mark weight, and the semantic pause weight;

[0181] The two adjacent short sentence texts whose splitting determination coefficient exceeds the preset splitting critical threshold are merged to obtain multiple related question texts.

[0182] The preset semantic segmentation rule base can be a set of structured rules pre-built based on industry scenario language data; the target semantic segmentation rule can be a set of semantic segmentation rules matched from the preset semantic segmentation rule base that completely corresponds to or has the highest similarity to the current user scenario tag; the problem sentence feature can be the sentence-ending symbols of typical short sentences in the scenario corresponding to the user scenario tag, including sentence-ending punctuation marks and semantic sentence-ending characters; the punctuation mark weight can be the quantitative score corresponding to the sentence-ending punctuation mark that reflects its influence on the semantic segmentation boundary; the semantic pause weight can be the quantitative score corresponding to the semantic sentence-ending character that reflects its influence on the semantic segmentation boundary.

[0183] In one embodiment, the method for determining the target semantic segmentation rule based on user scenario tags and a preset semantic segmentation rule library can be to use user scenario tags as query conditions to query the preset semantic segmentation rule library, and the query results include the target semantic segmentation rule.

[0184] Among them, the short sentence text can be a text fragment obtained after preliminary segmentation of the scene-related paragraph text according to the question sentence characteristics, and each short sentence text contains complete grammatical components.

[0185] In one embodiment, the method of segmenting the scene-related paragraph text according to the problem sentence features to obtain short sentence text can be achieved by using a string matching algorithm to identify the scene-related paragraph text. When any sentence-ending punctuation mark or semantic sentence-ending character in the problem sentence features is identified, the scene-related paragraph text is segmented to obtain short sentence text.

[0186] The similarity between two adjacent short sentences can be a quantitative value of the semantic relevance between them.

[0187] In one embodiment, the similarity between two adjacent short sentence texts can be calculated by converting the two adjacent short sentence texts into vector format and calculating the cosine similarity value of the two short sentence text vectors as the similarity between the two adjacent short sentence texts.

[0188] Among them, the splitting determination coefficient of two adjacent short sentence texts can be used to determine whether two adjacent short sentence texts belong to the same association problem.

[0189] In one embodiment, the method for calculating the splitting determination coefficient of two adjacent short sentence texts based on similarity, punctuation weight, and semantic pause weight can be achieved by subtracting the similarity between the two adjacent short sentence texts from 1, multiplying the subtraction result by the punctuation weight, and finally multiplying the multiplication result by the semantic pause weight to obtain the splitting determination coefficient of the two adjacent short sentence texts.

[0190] Among them, the preset splitting threshold can be a splitting judgment coefficient threshold value pre-set based on industry scenario language data. If the splitting judgment coefficient of two adjacent short sentence texts exceeds the preset splitting threshold, it means that the semantic correlation between the two adjacent short sentence texts is strong and they belong to the same correlation problem. Therefore, these two adjacent short sentence texts can be merged. After performing the merging and verification operation on all adjacent short sentence texts, the resulting text fragment is the correlation problem text.

[0191] The advantage of this approach is that it enables precise segmentation of context-related paragraphs and ensures the semantic integrity and context-appropriateness of the related question text.

[0192] The advantage of this approach is that by using the dual constraints of core question keywords and user scenario tags, a two-level segmentation of the pre-set question text is achieved. This ensures that the associated question text focuses on core knowledge and that the associated question text accurately matches the usage scenario, significantly improving the retrieval relevance of the subsequent local vector knowledge base.

[0193] S403, convert the multiple associated question texts into associated question vectors, and store the associated question vectors and the associated question texts corresponding to the associated question vectors in association to obtain a local vector knowledge base.

[0194] In one embodiment, converting multiple related question texts into related question vectors can be achieved by using the MiniLM-L6 lightweight model adapted to the RK3588 processor to convert the related question texts into vector format, thus obtaining related question vectors.

[0195] In one embodiment, the method of storing the associated question vectors and their corresponding associated question texts to obtain a local vector knowledge base can be achieved by using a relational database to store the associated question vectors and their corresponding associated question texts.

[0196] The advantage of this approach is that by building and storing the vector knowledge base locally, it ensures the storage security of user privacy and adapts to the hardware computing power limitations of desktop devices.

[0197] Figure 5This is a schematic diagram of the structure of a desktop device interaction system based on a local vector knowledge base, provided in an embodiment of this application. Figure 5 As shown, the system includes:

[0198] The question text acquisition module 510 is used to acquire user question text and search in a pre-built local vector knowledge base based on the user question text to obtain the associated question text corresponding to the user question text;

[0199] The answer text generation module 520 is used to send the user question text and the associated question text to the cloud server, and the cloud server generates and returns answer text based on the user question text and the associated question text;

[0200] The interactive content generation module 530 is used to generate digital human driving parameters based on the received response text, and send the response text and the digital human driving parameters to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters.

[0201] The interactive content playback module 540 is used to play the digital human audiovisual interactive content when it receives the digital human audiovisual interactive content.

[0202] Optionally, the digital human driving parameters include a lip-shape time series;

[0203] Accordingly, the interactive content generation module 530 is specifically used for:

[0204] The response text is segmented to obtain multiple word units, and a phoneme time series is generated based on the multiple word units; wherein, the phoneme time series includes multiple phonemes and the timestamps and durations corresponding to the phonemes;

[0205] Based on the pre-constructed mapping relationship between phonemes and lip shape identifiers, each phoneme in the phoneme time series is mapped to a lip shape identifier, and the lip shape identifiers, along with the timestamps and durations corresponding to the phonemes, are integrated to obtain the basic lip shape time series.

[0206] The basic lip shape time series is subjected to a smooth transition processing to obtain the lip shape time series.

[0207] Optionally, the interactive content generation module 530 is further configured to:

[0208] Receive sentiment description information generated and fed back by the cloud server based on the user's question text and the associated question text;

[0209] The target transition frame density and the target interpolation motion curve are determined based on the emotional description information.

[0210] The base lip-shaped time series is smoothed according to the target transition frame density and the target interpolation motion curve to obtain the lip-shaped time series.

[0211] Optionally, the digital human driving parameters may further include a head-nodding action time series;

[0212] Accordingly, the interactive content generation module 530 is also used for:

[0213] Based on multiple word units, prosodic pause information and prosodic stress information are determined;

[0214] A nodding action sequence is generated based on the prosodic pause information and the prosodic stress information; wherein, the nodding action sequence includes the word unit and the nodding action description information corresponding to the word unit;

[0215] Based on the word units in the nodding action sequence, the nodding action description information is time-aligned with the lip shape time sequence to obtain a nodding action time sequence, which is used to drive the digital human to synchronously complete lip shape changes and nodding actions with the lip shape time sequence.

[0216] Optionally, the system is also used for:

[0217] Obtain a preset question document and store it in a local encrypted storage area. Then, parse the preset question document locally to obtain the preset question text.

[0218] The preset question text is segmented to obtain multiple related question texts;

[0219] The multiple associated question texts are converted into associated question vectors, and the associated question vectors and their corresponding associated question texts are associated and stored to obtain a local vector knowledge base.

[0220] Optionally, the system is also used for:

[0221] Obtain core issue keywords and user scenario tags;

[0222] Accordingly, the preset question text is segmented to obtain multiple related question texts, including:

[0223] The preset question text is segmented into scene-related paragraph text based on the core question keywords; wherein, the scene-related paragraph text includes at least one core question keyword;

[0224] The context-related paragraph text is segmented based on the user context tags to obtain multiple related question texts.

[0225] Optionally, the system is also used for:

[0226] Based on the user scenario tags and the preset semantic segmentation rule library, the target semantic segmentation rule is determined; wherein, the preset semantic segmentation rule library includes multiple user scenario tags and the question sentence features, punctuation weights and semantic pause weights corresponding to the user scenario tags;

[0227] The scene-related paragraph text is segmented according to the question's sentence structure characteristics to obtain short sentence text;

[0228] Calculate the similarity between two adjacent short sentence texts, and calculate the splitting determination coefficient of the two adjacent short sentence texts based on the similarity, the punctuation mark weight, and the semantic pause weight;

[0229] The two adjacent short sentence texts whose splitting determination coefficient exceeds the preset splitting critical threshold are merged to obtain multiple related question texts.

[0230] In this embodiment, the question text acquisition module is used to acquire user question text and search a pre-built local vector knowledge base based on the user question text to obtain associated question text corresponding to the user question text; the answer text generation module is used to send the user question text and the associated question text to a cloud server, whereby the cloud server generates and feeds back answer text based on the user question text and the associated question text; the interactive content generation module is used to generate digital human driving parameters based on the answer text upon receiving it, and send the answer text and the digital human driving parameters to the cloud server, whereby the cloud server generates and feeds back digital human audiovisual interactive content based on the answer text and the digital human driving parameters; the interactive content playback module is used to play the digital human audiovisual interactive content upon receiving it. The above-mentioned desktop device interaction system based on a local vector knowledge base, through local vectorized knowledge retrieval, eliminates the need to upload user privacy data to a cloud server, reducing the risk of privacy leakage, and by generating digital human driving parameters locally first, improves the efficiency of the cloud server in generating digital human audiovisual interactive content, thereby reducing interaction response latency.

[0231] The desktop device interaction system based on a local vector knowledge base in this application embodiment can be a system, or a component, integrated circuit, or chip in a terminal. This system can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0232] The desktop device interaction system based on a local vector knowledge base in this application embodiment can be a system with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0233] The desktop device interaction system based on a local vector knowledge base provided in this application can realize the various processes implemented in the above embodiments. To avoid repetition, these processes will not be described again here.

[0234] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described desktop device interaction method embodiment based on a local vector knowledge base and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0235] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0236] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described desktop device interaction method embodiment based on a local vector knowledge base and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0237] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0238] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element. Furthermore, it should be noted that the scope of the methods and systems in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0240] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0241] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A desktop device interaction method based on a local vector knowledge base, characterized in that, The method includes: Obtain the user's question text, and search the pre-built local vector knowledge base based on the user's question text to obtain the associated question text corresponding to the user's question text; The user's question text and the associated question text are sent to the cloud server, whereby the cloud server generates and responds with a response text based on the user's question text and the associated question text. Upon receiving the response text, digital human driving parameters are generated based on the response text, and the response text and the digital human driving parameters are sent to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters; Upon receiving the digital human audiovisual interactive content, play the digital human audiovisual interactive content; The process of pre-constructing a local vector knowledge base includes: acquiring a preset question document and storing it in a local encrypted storage area; parsing the preset question document locally to obtain preset question text; acquiring core question keywords and user scenario tags; segmenting the preset question text into scenario-related paragraph text based on the core question keywords; wherein the scenario-related paragraph text includes at least one core question keyword; and determining target semantic segmentation rules based on the user scenario tags and a preset semantic segmentation rule library; wherein the preset semantic segmentation rule library includes multiple user scenario tags and questions corresponding to the user scenario tags. The process involves: segmenting the scene-related paragraph text according to the sentence structure features, punctuation weights, and semantic pause weights; calculating the similarity between two adjacent short sentence texts and determining the splitting coefficient based on the similarity, punctuation weights, and semantic pause weights; merging adjacent short sentence texts whose splitting coefficients exceed a preset splitting threshold to obtain multiple related question texts; converting these multiple related question texts into related question vectors and storing the related question vectors and their corresponding related question texts together to obtain a local vector knowledge base.

2. The desktop device interaction method based on a local vector knowledge base according to claim 1, characterized in that, The digital human driving parameters include lip-shape time series; Accordingly, the generation of digital human driving parameters based on the answer text includes: The response text is segmented to obtain multiple word units, and a phoneme time series is generated based on the multiple word units; wherein, the phoneme time series includes multiple phonemes and the timestamps and durations corresponding to the phonemes; Based on the pre-constructed mapping relationship between phonemes and lip shape identifiers, each phoneme in the phoneme time series is mapped to a lip shape identifier, and the lip shape identifiers, along with the timestamps and durations corresponding to the phonemes, are integrated to obtain the basic lip shape time series. The basic lip shape time series is subjected to a smooth transition processing to obtain the lip shape time series.

3. The desktop device interaction method based on a local vector knowledge base according to claim 2, characterized in that, After sending the user's question text and the associated question text to the cloud server, the process further includes: Receive sentiment description information generated and fed back by the cloud server based on the user's question text and the associated question text; Accordingly, the smoothing process performed on the basic lip shape time series to obtain the lip shape time series includes: The target transition frame density and the target interpolation motion curve are determined based on the emotional description information. The base lip-shaped time series is smoothed according to the target transition frame density and the target interpolation motion curve to obtain the lip-shaped time series.

4. The desktop device interaction method based on a local vector knowledge base according to claim 2, characterized in that, The digital human driving parameters also include a head nodding motion time series; Accordingly, after segmenting the response text into multiple word units, the method further includes: Based on multiple word units, prosodic pause information and prosodic stress information are determined; A nodding action sequence is generated based on the prosodic pause information and the prosodic stress information; wherein, the nodding action sequence includes the word unit and the nodding action description information corresponding to the word unit; Accordingly, after performing smoothing transition processing on the basic lip shape time series to obtain the lip shape time series, the method further includes: Based on the word units in the nodding action sequence, the nodding action description information is time-aligned with the lip shape time sequence to obtain a nodding action time sequence, which is used to drive the digital human to synchronously complete lip shape changes and nodding actions with the lip shape time sequence.

5. A desktop device interaction system based on a local vector knowledge base, characterized in that, The system includes: The question text acquisition module is used to acquire user question text and search in a pre-built local vector knowledge base based on the user question text to obtain the associated question text corresponding to the user question text; The answer text generation module is used to send the user question text and the associated question text to the cloud server, and the cloud server generates and returns answer text based on the user question text and the associated question text; The interactive content generation module is used to generate digital human driving parameters based on the received response text, and send the response text and the digital human driving parameters to the cloud server, so that the cloud server can generate and feed back digital human audiovisual interactive content based on the response text and the digital human driving parameters. An interactive content playback module is used to play the digital human audiovisual interactive content upon receiving it. The system is further configured to: acquire a preset question document and store the preset question document in a local encrypted storage area; parse the preset question document locally to obtain preset question text; acquire core question keywords and user scenario tags; segment the preset question text into scenario-related paragraph text based on the core question keywords; wherein the scenario-related paragraph text includes at least one core question keyword; determine target semantic segmentation rules based on the user scenario tags and a preset semantic segmentation rule library; wherein the preset semantic segmentation rule library includes multiple user scenario tags and question sentence features and tags corresponding to the user scenario tags. Punctuation weights and semantic pause weights are used to segment the scene-related paragraph text according to the question sentence features, resulting in short sentence texts. The similarity between two adjacent short sentence texts is calculated, and the splitting determination coefficient between the two adjacent short sentence texts is calculated based on the similarity, the punctuation weights, and the semantic pause weights. Two adjacent short sentence texts whose splitting determination coefficients exceed a preset splitting threshold are merged to obtain multiple related question texts. The multiple related question texts are converted into related question vectors, and the related question vectors and the related question texts corresponding to the related question vectors are associated and stored to obtain a local vector knowledge base.

6. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the desktop device interaction method based on a local vector knowledge base as described in any one of claims 1-4.

7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the desktop device interaction method based on a local vector knowledge base as described in any one of claims 1-4.

Images