Digital human interaction method and system based on naked eye 3D visualization

Abstract

The invention provides a digital human interaction method and system based on naked eye 3D visualization, and the method comprises the steps: obtaining query content provided by a user in a multi-modal manner, carrying out the semantic understanding and intention analysis of the query content through a natural language processing technology, and obtaining user intention data; performing matching search in a preset content database according to the user intention data, and if a highly related answer template or video clip exists, calling preset content and feeding back the preset content to the user; according to different answer sources, corresponding digital human video visualization schemes are adopted, a digital human driving engine is used for matching mouth shapes, expressions and actions, and matched digital human videos are synthesized; and displaying the digital human video through a transparent LCD screen.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a digital human interaction method and system based on naked-eye 3D visualization. Background Art

[0002] When building an intelligent digital human interactive visualization system, ensuring efficient and accurate responses to user queries is a core challenge. When matching the corresponding answers to different types of questions based on a preset template database, the following specific problems are often encountered in actual use: Although a rich set of answer templates are prepared in advance, due to the complexity of the enterprise business and the diversity of user questions, there are always some questions that cannot find fully matched answers in the preset template database. In this case, if you directly rely on the pre-trained general language model to search massive information from the Internet to generate reply text, it will not only consume a lot of computing resources, but may also cause the generated answers to conflict with the internal data of the enterprise, and even leak sensitive information. Even if a certain number of relevant answers are obtained through the general language model, these answers may not meet the specific needs, industry specifications, or the latest internal regulations of the enterprise. For example, some answers may cite outdated product information, terms of service, or legal interpretations that are not applicable to the enterprise. This not only affects the accuracy of the answer, but may also bring potential risks to the enterprise and users. In addition, whenever an unmatched question is encountered, the system needs to call the pre-trained language model for large-scale information retrieval and text generation. This process will take up a lot of computing resources, especially in the case of high concurrent requests, the server load will increase significantly, which may lead to system performance degradation and user experience. At the same time, after finding the answer content, it also needs to be converted into an adaptive three-dimensional digital human video and presented by imitating the expressions and movements of real humans, while providing users with an efficient and natural communication experience. Summary of the invention

[0003] The present invention provides a digital human interaction method based on naked eye 3D visualization, which mainly includes:

[0004] Acquire query content provided by the user in a multimodal manner, use natural language processing technology to perform semantic understanding and intent analysis on the query content, and obtain user intent data;

[0005] According to the user intention data, a matching search is performed in the preset content database, and if there is a highly relevant answer template or video clip, the preset content is called to feedback to the user;

[0006] If the preset content database fails to meet the matching search requirements, the query content is passed to the enterprise knowledge base, and key information in the enterprise knowledge base is extracted using information retrieval technology. If the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold, the query content is submitted to the pre-trained language model;

[0007] After receiving the query content, the general large model uses a pre-trained large-scale language model and massive information resources on the Internet to input the question and perform reasoning based on the context information, and dynamically constructs the answer through text generation technology;

[0008] According to different sources of answers, corresponding digital human video visualization solutions are adopted, in which the digital human driving engine is used to match lip shape, expression, and action to synthesize the corresponding digital human video;

[0009] The digital human video is displayed through a transparent LCD screen.

[0010] The present invention provides a digital human interaction system based on naked eye 3D visualization, which mainly includes:

[0011] A multimodal input processing module is used to obtain the query content provided by the user in a multimodal manner, and use natural language processing technology to perform semantic understanding and intent analysis on the query content to obtain user intent data;

[0012] A preset content matching module is used to perform a matching search in a preset content database according to the user intention data, and if there is a highly relevant answer template or video clip, the preset content is called to feedback to the user;

[0013] An enterprise knowledge base retrieval module is used to transfer the query content to the enterprise knowledge base if the preset content database fails to meet the matching search requirements, extract key information from the enterprise knowledge base using information retrieval technology, and submit the query content to a pre-trained language model if the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold;

[0014] The general large model underlying module is used for receiving the query content, using the pre-trained large-scale language model, combined with the massive information resources on the Internet, inputting the question and combining the context information for reasoning, and dynamically constructing the answer through text generation technology;

[0015] The digital human video synthesis module is used to adopt corresponding digital human video visualization solutions according to different answer sources, using the digital human driving engine to match lip shape, expression, and action to synthesize the appropriate digital human video;

[0016] The naked eye 3D display module is used to display the digital human video through a transparent LCD screen.

[0017] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0018] The present invention discloses a digital human interaction method and system based on naked-eye 3D visualization. The method understands the user's intention through multimodal input, and adopts a hierarchical retrieval strategy to obtain the answer content: first, match in the preset content library, if it fails, search the enterprise knowledge base, and finally generate it by a large language model. For answers from different sources, the present invention adopts corresponding visualization schemes: directly call preset videos, or synthesize new videos using text-to-speech and digital human driving engines. In the process of generating digital human behavior, real-person video data is modeled and imitated by machine learning algorithms, and the interaction strategy is optimized using reinforcement learning technology. Finally, the present invention displays digital human videos through a special transparent LCD screen, uses the principle of optical refraction to create a naked-eye 3D effect, and dynamically adjusts display parameters according to the environment. This method realizes an intelligent, natural, and immersive human-computer interaction experience, which can be widely used in customer service, guided tours and other scenarios to improve service efficiency and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the digital human interaction method based on naked-eye 3D visualization of the present invention.

[0020] Figure 2 Schematic diagram of the digital human interaction system based on naked-eye 3D visualization of the present invention. DETAILED DESCRIPTION

[0021] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0022] like Figure 1 The digital human interaction method based on naked eye 3D visualization in this embodiment may specifically include:

[0023] Step S101, obtaining query content provided by the user in a multimodal manner, using natural language processing technology to perform semantic understanding and intent analysis on the query content to obtain user intent data.

[0024] A voice signal and a facial image sequence are acquired through a voice acquisition device and a face acquisition device, and a phoneme sequence, a pitch curve, facial key point coordinate data and expression feature parameters are extracted from the voice signal and the facial image sequence; the voice content is segmented and annotated according to the phoneme sequence and the pitch curve, and a feature fusion algorithm is used to perform multimodal feature fusion on the pre-trained word vector corresponding to the segmentation result and the facial key point coordinate data to obtain a fused feature input matrix; a semantic template is matched in a preset dialogue knowledge base for the fused feature input matrix to obtain a semantic understanding result and an intent label.

[0025] Specifically, the voice signal is acquired through the voice acquisition device and the phoneme sequence, pitch curve, and speech speed parameters are extracted. At the same time, the facial image sequence is acquired through the face acquisition device and the facial key point coordinate data and expression feature parameters are extracted. According to the voice signal phoneme sequence and pitch curve, the voice content is segmented and annotated by the conditional random field algorithm, and the semantic feature vector is generated according to the pre-trained word vector corresponding to the segmentation result. The semantic feature vector and the facial key point coordinate data are multimodally fused by the feature fusion algorithm to generate a fusion feature input matrix. According to the fusion feature input matrix, the semantic template stored in the preset dialogue knowledge base is matched to generate the semantic understanding result and the corresponding intention label. According to the intention label, the corresponding digital human expression parameters and action parameter sequence are retrieved from the preset action library. According to the expression parameters and action parameter sequence, the three-dimensional digital human model is driven, and the digital human expression and body movements are rendered on the naked eye three-dimensional display device. The semantic understanding result is converted into speech waveform data by the speech synthesis module, and the sound and picture synchronization output is completed in combination with the digital population type parameters. In the multimodal human-computer interaction scenario, activation is supported by voice wake-up or face recognition. For voice wake-up, the audio signal in the environment is monitored. Once the preset wake-up word is detected, the voice signal processing process is started. For face recognition wake-up, the camera continuously collects image sequences. When the face of the registered user is recognized, the facial image processing and other related modules are initialized. The voice signal is first preprocessed to extract the phoneme sequence. For each acoustic frame, a 25-millisecond analysis window is used, and the frame shift is set to 10 milliseconds. The 13-dimensional fundamental frequency features and first-order difference features are extracted through the Mel cepstral coefficients. At the same time, the fundamental frequency contour features in the pitch curve are extracted. When extracting the speech rate parameters, the syllable duration normalization calculation is used. In the process of facial image sequence acquisition, a 68-point annotation scheme is used to extract the coordinate data of facial key points. The facial deformation parameters are obtained by calculating the Euclidean distance between adjacent key points. The principal component analysis method is used to reduce the dimension of facial expression changes to obtain the expression feature parameters. Based on the synchronous clock of the audio and video acquisition hardware, the voice signal and the facial image are time-aligned. After word segmentation, the speech content generates a word sequence. Each word corresponds to a 300-dimensional pre-trained word vector. A bidirectional long short-term memory network is used to extract features from the word vector sequence. The facial key point coordinate data is extracted through a convolutional neural network to extract spatial features, and a 128-dimensional expression feature vector is obtained. The speech feature vector and the expression feature vector are concatenated and then fused through a fully connected layer to output a 256-dimensional fused feature vector. The preset dialogue knowledge base stores structured semantic templates, each of which contains an intent category, a keyword set, and a corresponding response strategy. The fused feature vector is matched with the semantic template for similarity, and the cosine similarity calculation method is used. The similarity threshold is set to 0.85, and the template with the highest similarity is selected as the semantic understanding result.The digital human expression parameters include the weight values ​​of 17 basic action units, ranging from 0 to 1. The action parameter sequence uses a frequency of 25 frames per second to describe the position information of joint points such as the head rotation angle and the torso posture. Based on the naked-eye 3D display technology, the parallax rendering method is used to generate the parallax images of the left and right eyes. The parallax angle range is controlled between 2 and 8 degrees to ensure viewing comfort. The speech synthesis adopts the vocoder model to convert the text into acoustic parameters and then generate a speech waveform with a sampling rate of 16khz. The synchronization error between the lip parameters and the speech is calculated through the visual and auditory perception model, and the error is controlled within 40 milliseconds. The digital human expression parameters are adjusted according to the fundamental frequency and loudness characteristics of the speech to achieve an expressive multimodal interaction effect.

[0026] Step S102: perform a matching search in a preset content database according to the user intention data, and if there is a highly relevant answer template or video clip, call the preset content to feed back to the user.

[0027] Keywords and context features are extracted according to user intention data to generate a feature vector of the intention, and a semantic search is performed on a preset content database through a vector space model to obtain a similarity score; the similarity score is sorted in descending order, and if the similarity is greater than a similarity threshold, the corresponding action data is extracted from the preset content database; a three-dimensional space transformation algorithm is used to align the coordinate system of the action data, and the key frames of the action data are smoothly interpolated through Bezier curves to obtain a continuous action trajectory; motion parameters between adjacent frames are calculated according to the continuous action trajectory, and if the motion parameter change exceeds a motion parameter change threshold, transition action data is obtained from the action library, and a left-eye stereoscopic image is generated for the action data through a parallax rendering method.

[0028] Specifically, keywords and context features are extracted from user intention data through deep learning algorithms to generate a 512-dimensional intention feature vector. The answer templates and video materials in the preset content database are semantically retrieved based on the vector space model to generate a similarity score between the intention and the content. The search results are sorted in descending order according to the similarity score, and a similarity threshold of 0.8 is set. The answer templates and video materials with similarity greater than the threshold are selected as candidate content, and the action data corresponding to the candidate content is extracted. For the extracted action data, the three-dimensional space transformation algorithm is used to align the coordinate system, and the key frames are smoothly interpolated through the Bezier curve to generate a continuous action trajectory. Based on the action trajectory data, the motion parameters between adjacent frames are calculated. If the change in the motion parameters exceeds the preset threshold, the transition action data is retrieved from the action library for supplementation. The supplemented action sequence is reprojected in three dimensions, and the left and right eye stereo images are generated in combination with parallax rendering, and the digital human animation is output through a naked eye three-dimensional display device. According to the lip key frames in the action sequence, the corresponding phoneme sequence is extracted, and a matching speech waveform is generated through a speech synthesizer to achieve synchronous output of sound and picture. In understanding user intent, a multi-layer perceptron structure is used when extracting features through deep learning. The input layer receives the word vector sequence of the user's query text, and the middle layer is set with 512 neurons. The activation function ReLU is used for nonlinear transformation, and finally a 512-dimensional intent feature vector is output. For the query "check today's weather", the extracted feature vector contains semantic information such as "check", "weather", and "time", and the value range of each dimension of the vector is between -1 and 1. The similarity calculation adopts the cosine similarity method, and the similarity score is obtained by calculating the cosine value of the angle between the intent feature vector and the preset content feature vector. When the query intent and the "weather query" answer template in the preset content library are calculated to obtain a similarity score of 0.92, which is higher than the set threshold of 0.8, the digital human video material corresponding to the template can be selected. The video material contains the expression, gesture and other action data of the digital human when explaining the weather information. The action data is represented by key frames, and each key frame contains the three-dimensional coordinate information of 67 human joints. Adjacent key frames are interpolated by cubic Bezier curves, and the interpolation function is P(t) = (1-t) 3 P 0 +3t(1-t) 2 P 1 +3t 2 (1-t)P 2 +t 3 P 3 , where t ranges from 0 to 1, P 0 and P 3 is the key frame position, i.e. the starting and ending point of the curve, P 1 and P 2is the control point. When it is detected that the displacement of the joint points between adjacent key frames exceeds the preset threshold of 30 cm, it is necessary to retrieve the transition action from the action library. The transition action retrieval calculates the similarity based on the Euclidean distance of the joint point position, selects the transition action sequence with the closest starting posture to supplement, and realizes the smooth transition of the action. In the process of 3D reprojection, the left and right eye images are generated by the binocular parallax principle. The parallax value is related to the observation distance and the pupil distance. At a 2-meter observation distance, the baseline parallax value is set to 6.5 cm. The naked eye 3D display projects the left and right eye images to the observer's eyes respectively through the grating structure to achieve a stereoscopic visual effect. Speech synthesis uses phonemes as the basic unit, and extracts the phoneme labels corresponding to the lip keyframes from the action sequence. The duration of each phoneme is about 50 to 200 milliseconds. By adjusting the duration parameters of the phonemes, the speech output is synchronized with the lip shape changes, and the time deviation between the phoneme and the lip shape is controlled within 40 milliseconds. Based on the duration information of the phonemes, the playback speed of the action sequence is adjusted accordingly to ensure the coordination and consistency of sound and action.

[0029] Step S103: If the preset content database fails to meet the matching search requirements, the query content is transferred to the enterprise knowledge base, and key information in the enterprise knowledge base is extracted using information retrieval technology. If the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold, the query content is submitted to the pre-trained language model.

[0030] A text classifier is used to divide the query content into topics, and a query semantic vector is obtained from the query content through a keyword extraction method; relevant documents are retrieved in the enterprise knowledge base according to the query semantic vector, and entity information is extracted from the relevant documents using a named entity recognition method, and the entity information is labeled and classified using a conditional random field algorithm; for the labeled and classified entity information, the cosine similarity between the entity information and the query semantic vector is calculated, and if the cosine similarity is higher than the cosine similarity threshold, a candidate knowledge matrix is ​​generated; the entity information in the candidate knowledge matrix is ​​sorted in descending order of similarity, and the entity information combination with the highest similarity is selected to generate an answer semantic vector, the answer text is constructed through a template filling method, and the digital human motion data is rendered using a three-dimensional reprojection method.

[0031] Specifically, a text classifier is used to divide the user's query content into topics, and the core entity and attribute information in the query is obtained through the keyword extraction method. The query semantic vector is constructed, and the match is retrieved from the preset content layer. If the matching similarity is lower than the 0.7 threshold, the knowledge base layer query is triggered. According to the query semantic vector, relevant documents are retrieved in the enterprise knowledge base, and the named entity recognition method is used to extract the entity information such as people, organizations, time, place, and values ​​contained in the document. The entity information is annotated and classified by the conditional random field algorithm. For the annotated and classified entity information, the cosine similarity between the query semantic vector and the query is calculated, and the entity information is screened by the 0.6 similarity threshold to generate a candidate knowledge matrix. The entity information in the candidate knowledge matrix is ​​sorted in descending order of similarity, and the top three entity information with the highest similarity are selected for combination. Based on the selected entity information, the answer semantic vector is generated, and the answer text is constructed by the template filling method. According to the answer text, the corresponding digital human expression and action data sequence is retrieved from the action library. The digital human action data is rendered by the three-dimensional reprojection method and displayed on a naked-eye three-dimensional display device. If the answer similarity is lower than the 0.5 threshold, it is switched to the general large language model. When a user queries a company's product return and exchange policy, the text classifier first identifies the topic of the query content and extracts keywords through the word frequency-inverse document frequency method, including core words such as "return", "exchange", and "policy", to construct a 300-dimensional query semantic vector. The similarity calculated at the preset content layer match is 0.65, which is lower than the 0.7 threshold, triggering a query at the knowledge base layer. The knowledge base retrieval locates the company's after-sales service documents, and named entity recognition extracts key entity information in the document, including the return period of "7 days", the exchange condition of "intact packaging", and the refund method of "original return". The conditional random field algorithm annotates these entities, with the return period annotated as a time type, the exchange condition annotated as a status type, and the refund method annotated as a method type. The cosine similarity between the annotated entity information and the query semantic vector is calculated using the following formula:

[0032] cos(θ)=(A·B) / (||A||·||B||), where A is the query vector and B is the entity information vector. The similarity of return-related entity information is 0.85 and that of exchange-related entity information is 0.78, both of which are higher than the threshold of 0.6, and a candidate knowledge matrix is ​​generated. The knowledge matrix is ​​sorted in descending order of similarity, and the three most relevant entity information groups, namely, return time limit, exchange conditions, and refund method, are selected. Based on the selected entity information, the preset answer template "the product is processed in Z method within X days from the date of receipt under Y conditions" is used to fill in the answer text to generate a standardized answer text. According to the semantic category of the answer text, the corresponding digital human expression sequence and gesture action sequence are retrieved from the action library. The expression sequence contains basic action units such as eyebrow raising, eye gaze, and smile, and each action unit contains the coordinate data of 17 facial feature points. The gesture action sequence contains directional gestures and explanatory gestures, and each gesture contains the three-dimensional coordinate data of 21 hand joints. The digital human action data is presented on the naked eye three-dimensional display device after three-dimensional reprojection. The reprojection process uses the parallax rendering principle. At a 2-meter observation distance, the parallax value of the left and right eyes is set to 6.5 cm to generate a binocular parallax image. If the similarity between the generated answer text and the query is 0.45, which is lower than the 0.5 threshold, it will automatically switch to the general large language model to generate a more general answer content. The answer text is converted into a speech waveform through a speech synthesizer, and the sampling rate is set to 16khz. At the same time, the phoneme sequence is extracted to drive the digital population type. The action sequence is resampled based on the phoneme duration to achieve accurate synchronization between sound and action, and the time error is controlled within 40 milliseconds.

[0033] Step S104, after receiving the query content, the general large model bottom layer uses the pre-trained large-scale language model, combined with the massive information resources on the Internet, inputs the question and performs reasoning based on the context information, and dynamically constructs the answer through text generation technology.

[0034] Part-of-speech tagging data is obtained according to the user query content, and the part-of-speech tagging data is obtained by performing word segmentation and part-of-speech tagging on the user query content by a question understander; the part-of-speech tagging data is used to generate a question vector and a context vector, and a retrieval vector is formed by splicing the question vector and the context vector, and the retrieval vector is used to obtain information fragments; a subject-predicate-object grammatical structure is extracted for the information fragment, and the subject-predicate-object grammatical structure is used to construct a knowledge triple based on a syntactic dependency relationship, and after constructing the knowledge triple, an initial answer knowledge graph is obtained; knowledge expansion is performed on the initial answer knowledge graph to obtain an extended answer knowledge graph, and the extended answer knowledge graph is used to extract key facts, and the key facts are filled in through a sentence template to obtain an answer text.

[0035] Specifically, the question understander performs word segmentation and part-of-speech tagging on the user query content, extracts the question keywords and context features, generates a question vector, and extracts the last five rounds of dialogue content from the dialogue record to construct a context vector. The retrieval vector is generated by splicing the question vector and the context vector, and the relevant information fragments are retrieved from the Internet knowledge base, and the information fragments are screened and sorted by relevance calculation. The subject-predicate-object grammatical structure is extracted from the screened information fragments, and knowledge triples are constructed based on syntactic dependencies to generate the initial answer knowledge graph. The knowledge graph reasoning method is used to expand the initial answer knowledge graph and supplement the complete knowledge nodes required for the answer. Based on the expanded answer knowledge graph, key facts are extracted, and a standardized answer text is generated by the sentence template filling method. Emotional tendencies and tone features are identified from the answer text and mapped to the corresponding facial expression parameters in the digital human expression library. According to the facial expression parameters, matching body movement data is retrieved from the action library to generate a complete digital human action sequence. The parallax rendering method is used to reproject the digital human action sequence in three dimensions and present it in real time on a naked-eye three-dimensional display device. When a user asks "the application of artificial intelligence in the medical field", the question understander first performs word segmentation processing, extracts keywords such as "artificial intelligence", "medical care", and "application", and marks "medical care" as a domain word and "application" as a subject word, and constructs a question vector with a dimension of 512. Contextual information is extracted from the last five rounds of conversation records, including the basic concepts of artificial intelligence that the user has asked about, and the two are spliced ​​to form a 768-dimensional retrieval vector. Based on the retrieval vector, relevant content is searched in the Internet knowledge base, and the relevance is calculated by cosine similarity. The similarity threshold is set to 0.75, and the top 10 information fragments with the highest relevance are extracted. Syntactic analysis is performed on each information fragment to identify grammatical components such as the subject "artificial intelligence", the predicate "application", and the object "aided diagnosis", and knowledge triplets such as "artificial intelligence-application-medical imaging" and "deep learning-diagnosis-pulmonary disease" are extracted. The initial answer is expanded through knowledge graph reasoning, and specific data nodes such as "CT image recognition accuracy reaches 95%" and "diagnosis time is shortened by 60%" are supplemented based on association rules. Using the preset professional answer template, the medical scene is divided into three dimensions: diagnosis, treatment, and management. The standardized answer text is generated according to the structure of "field-method-effect". The answer text is identified with positive and professional tone characteristics through sentiment analysis, and mapped to the "confidence" and "focus" expression parameter groups in the digital human expression library. The facial expression parameters include 17 action unit weight values, such as 0.7 for eyebrow raising and 0.5 for mouth corner raising. Based on the expression parameters, the matching gestures are retrieved in the action library to generate an action sequence including explanatory gestures and indicative gestures. The action sequence is presented on the naked eye 3D display device after 3D reprojection. The reprojection adopts the principle of binocular parallax. At a 2-meter observation distance, the parallax of the left and right eyes is set to 6.5 cm. When the digital human explains the medical application, it cooperates with gestures such as raising hands and pointing, and maintains a professional smile on the face.The speech synthesizer generates speech output with a sampling rate of 16khz, and accurately controls the changes in lip shape according to the phoneme sequence, with the time error controlled within 40 milliseconds. Based on the sound loudness and speech rate parameters, the amplitude and rhythm of the digital human's movements are dynamically adjusted to achieve natural expression with synchronized sound and picture. Through the preset general sentence template, a hierarchical answer structure is constructed. When explaining medical application cases, it is expanded layer by layer from the dimensions of basic diagnosis, specialized treatment, smart hospitals, etc., and the logical connection between the parts is established through semantic connection vocabulary. Specific statistical data are inserted in a timely manner during the answer process to enhance persuasiveness and credibility, while maintaining the professionalism and accuracy of language expression.

[0036] According to the specific needs of different enterprises, lightweight private models are trained to replace or supplement general large models to generate response texts while protecting data privacy.

[0037] A text segmenter is used to perform word segmentation on professional documents, and a dependency syntax analyzer is used to obtain the conceptual relationship in the word segmentation result, and a knowledge vector is generated according to the conceptual relationship; semantic understanding parameters are extracted from a general language model according to the knowledge vector, and the semantic understanding parameters are subjected to matrix pruning and compression to obtain a lightweight semantic understanding model; cosine similarity is calculated for the knowledge vector, and if the similarity is higher than a similarity threshold, the knowledge vectors are clustered to the same knowledge node, and an encrypted knowledge base is constructed through the knowledge nodes; a lightweight semantic understanding model is used to perform semantic analysis on the query content to obtain a query intent vector, and the knowledge content with the highest matching degree is retrieved in the encrypted knowledge base according to the query intent vector.

[0038] Specifically, based on the professional documents and historical question-and-answer records provided by the enterprise, a text segmenter is used to extract professional terms, and the concept relationship is obtained through dependency syntactic analysis to generate an initial knowledge vector. The knowledge distillation method is used to transfer the semantic understanding ability from the general language model, and the parameter matrix is ​​pruned and compressed to generate a lightweight semantic understander. A domain knowledge base is constructed based on the initial knowledge vector, and the knowledge is clustered based on the cosine similarity calculation method to establish the association relationship between knowledge nodes. The hash algorithm is used to desensitize the sensitive data in the knowledge base to generate an encrypted knowledge base. The user's query content is understood through a lightweight semantic understander to extract the query intent and key attributes. Relevant content is retrieved from the encrypted knowledge base based on the query intent, and a list of candidate answers is generated by similarity sorting. According to the tone and intonation characteristics of the candidate answers, matching digital human expression parameters are retrieved from the expression library. Digital human action sequences are generated based on the expression parameters, and parallax rendering method is used to display them on naked-eye 3D display devices. In the financial enterprise scenario, when extracting professional terms from internal documents of the enterprise, the word segmenter is used to identify the core concepts in the financial field such as "financial products", "yield", and "risk level". The dependency syntax analysis obtains the concept relationships such as "financial products-inclusion-yield" and "risk level-impact-yield". Each concept is represented by a 300-dimensional vector to form an initial knowledge vector set. In the knowledge distillation process, the original language model is compressed by the structural pruning method, and the 12-layer transformer structure is compressed to 4 layers. The number of attention heads is reduced from 12 to 4, the hidden layer dimension is reduced from 768 to 256, and the number of model parameters is reduced from 150 million to 15 million, while maintaining the core semantic understanding ability. Based on the knowledge vector, a financial domain knowledge base is constructed, and the K-means clustering method is used to group the knowledge. The K value is set to 20, and the correlation strength between knowledge points is calculated by cosine similarity. The similarity calculation formula is cos(θ)=(A·B) / (||A||·||B||), where A and B are knowledge vectors. When the similarity is greater than 0.8, an association relationship is established. Hash encryption is performed on sensitive data such as customer information and transaction amount. The SHA-256 algorithm is used to generate a fixed-length ciphertext. The original data "Zhang San-purchase-financial product A-amount 1 million" is converted into a hash value and stored in the encrypted knowledge base. When the user asks "high-yield financial product recommendation", the lightweight semantic understander extracts the query intent "product recommendation" and the key attribute "high yield", retrieves relevant content in the encrypted knowledge base, and generates a list of candidate answers in descending order of similarity. Based on the tone characteristics of the answer content, the matching digital human expression parameters are retrieved from the expression library. In the professional explanation scenario, facial parameters such as eyebrow raising degree 0.6, eye gaze degree 0.8, and smile degree 0.4 are set to create a professional and credible expression effect. Combined with speech synthesis, a speech output with a sampling rate of 16khz is generated, and the lip shape changes are controlled by the phoneme sequence.The digital human action sequence includes explanatory gestures and indicative gestures, and each gesture is described by the three-dimensional coordinate data of 21 hand joints. The action data is presented on the naked eye three-dimensional display device after three-dimensional reprojection, and a parallax value of 6.5 cm is set at a 2-meter observation distance to generate left and right eye stereo images. The gestures are synchronized with the rhythm of the voice, and the error is controlled within 40 milliseconds to achieve a natural and smooth expression effect. Enterprise knowledge forms a complete chain from document extraction, knowledge base construction to answer generation. Through knowledge distillation and parameter compression, the lightweight and efficient private deployment is guaranteed. At the same time, encryption algorithms are combined to ensure data security, and professional and accurate knowledge transfer is achieved through digital humans.

[0039] Step S105, according to different sources of answers, adopt corresponding digital human video visualization solutions, wherein a digital human driving engine is used to match lip shape, expression, and action to synthesize a suitable digital human video.

[0040] A content recognition method is used to classify and recognize the answer content. If the recognition result comes from a preset content database or an enterprise knowledge base, a corresponding video material identifier is obtained from the preset content database or the enterprise knowledge base; video material data is extracted according to the video material identifier, and a digital human expression sequence and audio data are obtained from the video material data through a video framing method; a standard audio waveform is generated for the audio data using a speech synthesis method, and a phoneme label sequence and lip shape parameters are extracted from the standard audio waveform through a phoneme recognizer; a standard lip shape library is retrieved based on the phoneme label sequence to obtain a lip shape parameter sequence, digital human animation data is generated according to the lip shape parameter sequence and the expression sequence, and the digital human animation data is three-dimensionally reprojected through a parallax rendering method.

[0041] Specifically, the source of the answer is classified by the content recognition method, and the type of the answer source is determined by calculating the text feature vector. If it comes from the preset content library or the enterprise knowledge base, the corresponding material identifier is retrieved from the video material library. The video material is extracted according to the material identifier, and the digital human expression sequence, action sequence and corresponding audio data are obtained by the video frame method. For the text generated by the language model, the speech synthesis method is used to generate a standard audio waveform, and the phoneme label and duration information are extracted by the phoneme recognizer. The standard lip shape library is retrieved based on the phoneme label sequence to generate a lip shape parameter sequence, and the mutation between adjacent lip shapes is eliminated by parameter smoothing. The speech emotion recognizer is used to extract the intonation feature from the audio waveform, and the expression library is retrieved according to the intonation feature to generate the facial expression parameter. The corresponding body movement data is retrieved from the action library based on the expression parameter sequence, and the motion trajectory smoothing method is used to interpolate the action data. The lip shape parameters, expression parameters and body movement data are integrated by the action fusion method to generate a complete digital human animation sequence. The parallax rendering method is used to reproject the digital human animation in three dimensions, and the stereoscopic visual effect is output in real time on the naked eye three-dimensional display device. In the enterprise service scenario, when a user inquires about the product return process, the content identifier first classifies the source of the answer through the text feature vector, extracts word frequency features, syntactic features, etc. to construct a 512-dimensional feature vector, and determines that the answer comes from the preset content library through cosine similarity calculation, and the similarity threshold is set to 0.85. The video material with the identifier "return_policy_001" is retrieved from the preset content library, and the video is parsed at a frame rate of 25 frames per second. The coordinate data of 17 facial feature points in the digital human expression sequence are extracted, and the action sequence contains the three-dimensional coordinate information of 21 human joints. The audio data sampling rate is 16khz. When the answer comes from the language model, the speech synthesizer converts the text into an audio waveform, and uses the phoneme segmentation algorithm to divide the speech into basic phoneme units according to acoustic features. Each phoneme is annotated with the start time and duration, and a phoneme label sequence is generated, such as " / h / / e / / l / / o / ". Based on the phoneme sequence, the corresponding lip shape parameters are retrieved in the standard lip shape library, and each phoneme is mapped to a specific lip shape parameter, including values ​​such as mouth opening and closing degree and lip angle stretching degree. Cubic spline interpolation can be used to smooth adjacent lip parameters. Speech emotion recognition extracts acoustic features such as fundamental frequency and energy from the audio, calculates emotional dimension scores such as positivity and activation, and maps them to facial expression parameter groups. For example, positive tone corresponds to parameter values ​​such as 0.7 for eyebrow uplift and 0.5 for mouth corner uplift. The action library stores standardized gesture action sequences, each of which contains parameters such as arm joint angle and finger curvature. The action data is interpolated by smoothing the motion trajectory, and the joint position of the intermediate frame is calculated using the Bezier curve to ensure a natural and smooth transition of the action. During the action fusion process, the parameters such as lip shape, expression, and gesture are unified into a three-dimensional coordinate space, and the final vertex position is calculated based on the weight superposition method.The weight of lip shape is dynamically adjusted with the loudness of speech, the weight of expression is positively correlated with the intensity of emotion, and the weight of gesture peaks at the key frame. When rendering parallax, a 6.5 cm reference parallax is set at a 2-meter observation distance, and the horizontal offset of the left and right eye images is calculated based on the depth information. The left and right eye images are projected to the observer's eyes respectively through the grating structure to achieve a naked eye 3D effect. The rendering resolution is 3840x2160 and the refresh rate is 90hz, which meets the requirements of real-time interaction.

[0042] In the process of synthesizing digital human videos, massive amounts of real-life video data are modeled and imitated for learning, while the digital human's response strategy and interaction method are optimized based on user feedback and dialogue.

[0043] The human body key point coordinate sequence and facial feature point sequence in real-person video data are obtained to obtain a human body motion feature matrix; the human body motion feature matrix is ​​converted using a joint angle mapping method to obtain a standardized motion feature vector; an action generator is constructed using a recurrent neural network for the standardized motion feature vector, and an action frame sequence is interpolated and completed; the completed action frame sequence is smoothed using a Bezier curve to obtain a smooth action sequence; a state transfer matrix is ​​constructed based on user satisfaction scores, the state transfer probability is updated using a reinforcement learning method, and the optimal action response sequence is selected.

[0044] Specifically, the real-life video data is processed by deep neural network, and 67 human key point coordinate sequences and 68 facial feature point sequences are extracted to generate a human motion feature matrix. The human motion feature matrix is ​​converted to the digital human bone structure by using the joint angle mapping method to generate a standardized motion feature vector. Based on the standardized motion feature vector, an action generator is constructed by using a recurrent neural network to interpolate and complete the continuous action frames. For the completed action frame sequence, the action trajectory is smoothed by using Bezier curves to generate a smooth action sequence. The voice intonation and text emotions in the user feedback are analyzed by the emotion recognizer to generate a user satisfaction score. The action state transition matrix is ​​constructed by using the reinforcement learning method, and the state transition probability is updated according to the user satisfaction score. The optimal action response sequence is selected based on the state transition matrix to generate the digital human expression and action control parameters. The parallax rendering method is used to reproject the digital human in three dimensions, and the interactive screen is presented in real time on the naked eye three-dimensional display device. In the customer service scenario, the deep neural network extracts features from the recorded live reception video, including 67 human body key point coordinates and 68 facial feature points. Each key point is represented by a three-dimensional coordinate (x, y, z). The spatial features are extracted through the convolution layer, and the pooling layer outputs a 512-dimensional action feature matrix after dimensionality reduction. These feature points describe the standard action postures of the receptionist, such as the corners of the mouth rising when smiling and the head turning when nodding. The joint angle mapping adopts the Euler angle representation method to convert the spatial position of the human joint points into a joint angle sequence. For example, for the arm raising action, the standardized angle vector is obtained by calculating the rotation angles of the shoulder joint, elbow joint, and wrist joint. The mapped action feature vector contains the Euler angle values ​​of each joint, and the value range is normalized to between -1 and 1. The recurrent neural network adopts a long short-term memory network structure. The input layer receives the standardized action feature vector, and the hidden layer contains 256 neurons. The action parameters of the intermediate frame are generated through time series prediction. For an action sequence of 25 frames per second, the network predicts the transition action between adjacent key frames. The cubic Bezier curve is used for action trajectory smoothing, and the curve equation is P(t) = (1-t) 3 P 0 +3t(1-t) 2 P 1 +3t 2 (1-t)P 2 +t 3 P 3 , where t ranges from 0 to 1, P 0 and P 3 is the key frame position, P 1 and P 2is the control point. For example, when waving an arm, the acceleration and deceleration effects can be achieved by adjusting the position of the control point. When analyzing user feedback, the emotion recognizer extracts acoustic features such as the pitch curve and energy distribution of the voice, and matches the text with the emotional dictionary to calculate the positive score. The satisfaction score adopts a 5-point system, and is weighted according to the weight of the voice emotion feature of 0.6 and the weight of the text emotion feature of 0.4. The state transition matrix constructed by reinforcement learning has a dimension of 1000x1000, and each state corresponds to a set of standard action postures. The transition probability is updated based on the user satisfaction score. The state transition with a higher score obtains a greater reward value, and the state transition strategy is optimized through the value iteration method. In the process of generating digital human expressions and action parameters, the action state transition matrix and real-time feedback score are combined to select the optimal action response sequence. For example, when the user feedback satisfaction is high, the weight of the smiling expression is increased, and the gesture amplitude is increased to enhance the effect of positive interaction. Parallax rendering adopts the principle of binocular parallax, sets a 6.5 cm benchmark parallax at a 2-meter observation distance, and calculates the horizontal offset of the left and right eye images in combination with the action depth information. The rendering resolution is 3840x2160, and the left and right eye images are projected to the observer's eyes respectively through the grating structure to achieve real-time stereoscopic display effect.

[0045] Step S106, displaying the digital human video through a transparent LCD screen.

[0046] The light data in the display environment is obtained through a light sensor array, and an environment parameter matrix including an environment light intensity value, a light incident angle value, and a color temperature distribution value is generated according to the light data; the environment parameter matrix is ​​processed by a light parameter mapping method to obtain a display parameter vector including a display brightness value, a contrast value, and a color saturation value; brightness gain processing is performed on the original image of the digital human according to the display parameter vector to obtain a brightness compensation image, and color correction is performed on the brightness compensation image according to the color temperature distribution value in the environment parameter matrix to obtain a color correction image; the left and right eye disparity values ​​are calculated for the color correction image, a stereo disparity image pair is generated by a light refraction compensation method, and the stereo disparity image pair is compensated for viewing angle according to the observation distance value to obtain a final display image.

[0047] Specifically, the illumination data in the display environment, including the ambient light intensity, the incident angle of light, and the color temperature distribution parameters, are collected through the light sensor array to generate an environment parameter matrix. The environment parameter matrix is ​​converted into a display parameter vector using the illumination parameter mapping method, including the display brightness value, the contrast value, and the color saturation value. The original image of the digital human is subjected to brightness gain processing based on the display parameter vector to generate a brightness compensation image. The brightness compensation image is subjected to color temperature correction, and the image color balance is adjusted according to the ambient color temperature value to generate a color correction image. In the digital human holographic cabin, this color correction is crucial to ensure the authenticity and immersion of the virtual character, because the holographic projection technology requires accurate color reproduction to maintain the quality of the 3D image. The left and right eye parallax values ​​are calculated based on the color correction image, and the light refraction compensation method is used to generate stereoscopic parallax image pairs, which are used to create a realistic three-dimensional effect, so that the participants feel that they are in the environment of the digital human. The parallax image is separated by the grating structure of the transparent liquid crystal screen to generate independent left and right eye images. This is not only applicable to traditional 3D displays, but also particularly suitable for transparent LCD screens in holographic cabins, ensuring that clear stereoscopic images can be observed within a 45-degree viewing angle, and users can experience stereoscopic visual effects without wearing special glasses. The parallax angle is calculated according to the observation distance, and the viewing angle compensation is performed on the independent images of the left and right eyes to generate the final display image. The real-time monitoring method is used to continuously collect environmental parameter changes, and the display parameter vector is dynamically updated according to the magnitude of the change. In the shopping mall display scene, the light sensor array consists of 16 light intensity sensors and 4 color temperature sensors, which are evenly distributed around the LCD transparent screen or the digital human holographic cabin display. The light intensity sensor has a measurement range of 0-100000 lux, the color temperature sensor has a measurement range of 2000-10000 Kelvin, and the sampling frequency is 10 Hz. After filtering, the sensor data forms a 16x3 environmental parameter matrix, which contains light intensity, incident angle and color temperature values. The mapping of environmental parameters to display parameters uses a piecewise linear function. When the ambient light intensity is in the range of 0-1000 lux, the display brightness is proportionally increased by 0.5 times, and in the range of 1000-10000 lux, it is proportionally increased by 0.3 times, which helps to optimize the visibility and contrast of the digital person in the LCD transparent screen or holographic cabin. The contrast value increases with the increase of ambient light intensity. The mapping function is contrast = 1 + log (ambient_light / 1000), where contrast is the image contrast value, ambient_light is the ambient light intensity, and log is the natural logarithm function, which is used to convert the linear change of ambient light intensity into a change on a logarithmic scale. Color saturation is inversely proportional to the ambient color temperature. In the holographic cabin, in order to make the digital person look more natural, the color saturation needs to be finely adjusted according to the ambient color temperature. The brightness gain processing uses pixel-level multiplication operations, and the RGB values ​​of the original image are multiplied by the brightness gain coefficient respectively.In an indoor environment of 5000 lux, the brightness gain coefficient is 1.5, and the corresponding display brightness reaches 450 nits. Color temperature correction is achieved by adjusting the relative intensity of the three RGB channels, mapping the standard white point of 6500 Kelvin to the actual ambient color temperature. The parallax image generation adopts the binocular parallax principle. At a standard observation distance of 2 meters, the parallax reference value of the left and right eyes is set to 6.5 cm. The grating structure of the transparent LCD screen consists of a strip lens array with a lens spacing of 0.5 mm, a focal length of 2 mm, and a refractive index of 1.5. After the light is refracted by the lens array, 9 viewing angles are formed, and the difference between each viewing angle is 8 degrees. When the observation distance changes, the parallax angle is adjusted in an inverse relationship. When the observation distance increases from 2 meters to 3 meters, the parallax angle decreases from 8 degrees to 5.3 degrees. The display parameters are updated every time the ambient light intensity changes by 1000 lux. The display brightness can be adjusted from 150 to 600 nits, the contrast can be adjusted from 1000:1 to 3000:1, and the color temperature can be adjusted from 5000 to 7500 Kelvin. Through real-time parameter adjustment, the digital human image always maintains the best display effect in different scenes such as natural light in the morning, strong light at noon, and dim light in the evening. The grating structure ensures that a clear stereoscopic image can be observed within a 45-degree viewing angle, and the observer can move freely to experience a realistic three-dimensional effect. The response delay of the real-time monitoring system is controlled within 100 milliseconds, and the observer will not perceive the adjustment process of the display parameters.

[0048] The present invention provides a digital human interaction system based on naked eye 3D visualization, which mainly includes:

[0049] A multimodal input processing module is used to obtain the query content provided by the user in a multimodal manner, and use natural language processing technology to perform semantic understanding and intent analysis on the query content to obtain user intent data;

[0050] A preset content matching module is used to perform a matching search in a preset content database according to the user intention data, and if there is a highly relevant answer template or video clip, the preset content is called to feedback to the user;

[0051] An enterprise knowledge base retrieval module is used to transfer the query content to the enterprise knowledge base if the preset content database fails to meet the matching search requirements, extract key information from the enterprise knowledge base using information retrieval technology, and submit the query content to a pre-trained language model if the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold;

[0052] The general large model underlying module is used for receiving the query content, using the pre-trained large-scale language model, combined with the massive information resources on the Internet, inputting the question and combining the context information for reasoning, and dynamically constructing the answer through text generation technology;

[0053] The digital human video synthesis module is used to adopt corresponding digital human video visualization solutions according to different answer sources, using the digital human driving engine to match lip shape, expression, and action to synthesize the appropriate digital human video;

[0054] The naked eye 3D display module is used to display the digital human video through a transparent LCD screen.

[0055] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A digital human interaction method based on naked eye 3D visualization, characterized in that: The method comprises: Acquire query content provided by the user in a multimodal manner, use natural language processing technology to perform semantic understanding and intent analysis on the query content, and obtain user intent data; According to the user intention data, a matching search is performed in the preset content database, and if there is a highly relevant answer template or video clip, the preset content is called to feedback to the user; If the preset content database fails to meet the matching search requirements, the query content is passed to the enterprise knowledge base, and key information in the enterprise knowledge base is extracted using information retrieval technology. If the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold, the query content is submitted to the pre-trained language model; After receiving the query content, the general large model uses a pre-trained large-scale language model and massive information resources on the Internet to input the question and perform reasoning based on the context information, and dynamically constructs the answer through text generation technology; According to different sources of answers, corresponding digital human video visualization solutions are adopted, in which the digital human driving engine is used to match lip shape, expression, and action to synthesize the corresponding digital human video; The digital human video is displayed through a transparent LCD screen.

2. The method according to claim 1, characterized in that The obtaining of the query content provided by the user in a multimodal manner, and using natural language processing technology to perform semantic understanding and intent analysis on the query content to obtain user intent data includes: Acquire a voice signal and a facial image sequence through a voice acquisition device and a facial acquisition device, and extract a phoneme sequence, a pitch curve, facial key point coordinate data, and expression feature parameters from the voice signal and the facial image sequence; Segmenting and annotating the speech content according to the phoneme sequence and the pitch curve, and using a feature fusion algorithm to perform multimodal feature fusion on the pre-trained word vector corresponding to the segmentation result and the facial key point coordinate data to obtain a fusion feature input matrix; The fusion feature input matrix is ​​matched with semantic templates in a preset dialogue knowledge base to obtain semantic understanding results and intent labels.

3. The method according to claim 1, characterized in that According to the user intention data, a matching search is performed in a preset content database, and if there is a highly relevant answer template or video clip, the preset content is called to feedback to the user, including: Extract keywords and context features based on user intent data, generate feature vectors of intent, perform semantic retrieval on the preset content database through the vector space model, and obtain similarity scores; Sorting the similarity scores in descending order, and if the similarity is greater than a similarity threshold, extracting corresponding action data from the preset content database; A three-dimensional space transformation algorithm is used to align the coordinate system of the motion data, and a Bezier curve is used to smoothly interpolate the key frames of the motion data to obtain a continuous motion trajectory; Motion parameters between adjacent frames are calculated according to the continuous motion trajectory. If the motion parameter change exceeds a motion parameter change threshold, transition motion data is obtained from an action library, and left-eye and right-eye stereoscopic images are generated for the motion data using a parallax rendering method.

4. The method according to claim 1, characterized in that: If the preset content database fails to meet the matching search requirements, the query content is transmitted to the enterprise knowledge base, and key information in the enterprise knowledge base is extracted using information retrieval technology; If the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold, submitting the query content to a pre-trained language model includes: Using a text classifier to divide the query content into topics, and obtaining a query semantic vector from the query content through a keyword extraction method; Retrieve relevant documents from the enterprise knowledge base according to the query semantic vector, extract entity information from the relevant documents using a named entity recognition method, and annotate and classify the entity information using a conditional random field algorithm; For the labeled and classified entity information, the cosine similarity between the entity information and the query semantic vector is calculated, and if the cosine similarity is higher than the cosine similarity threshold, a candidate knowledge matrix is ​​generated; The entity information in the candidate knowledge matrix is ​​sorted in descending order of similarity, the entity information combination with the highest similarity is selected to generate an answer semantic vector, the answer text is constructed through a template filling method, and the digital human action data is rendered using a three-dimensional reprojection method.

5. The method according to claim 1, characterized in that After receiving the query content, the general large model uses a pre-trained large-scale language model, combined with massive information resources on the Internet, inputs the question and performs reasoning based on context information, and dynamically constructs the answer through text generation technology, including: Acquire part-of-speech tagging data according to the user query content, wherein the part-of-speech tagging data is obtained by performing word segmentation and part-of-speech tagging on the user query content by the question understander; Generate a question vector and a context vector using the part-of-speech tagging data, and form a search vector by concatenating the question vector and the context vector, wherein the search vector is used to obtain information fragments; Extracting a subject-predicate-object grammatical structure from the information fragment, wherein the subject-predicate-object grammatical structure is used to construct a knowledge triple based on a syntactic dependency relationship, and after constructing the knowledge triple, an initial answer knowledge graph is obtained; Performing knowledge expansion on the initial answer knowledge graph to obtain an extended answer knowledge graph, wherein the extended answer knowledge graph is used to extract key facts, and filling the key facts with a sentence template to obtain an answer text; It also includes: training lightweight private models to replace or supplement general large models according to the specific needs of different enterprises, and generating response texts while protecting data privacy.

6. The method according to claim 5, characterized in that The method trains lightweight private models to meet the specific needs of different enterprises, replaces or supplements general large models, and generates reply texts while protecting data privacy, including: Use a text segmenter to segment professional documents, obtain the conceptual relationship in the segmentation results through a dependency syntactic analyzer, and generate a knowledge vector based on the conceptual relationship; Extracting semantic understanding parameters from a general language model according to the knowledge vector, and performing matrix pruning and compression on the semantic understanding parameters to obtain a lightweight semantic understanding model; Performing cosine similarity calculation on the knowledge vectors, if the similarity is higher than a similarity threshold, clustering the knowledge vectors into the same knowledge node, and constructing an encrypted knowledge base through the knowledge nodes; A lightweight semantic understanding model is used to perform semantic analysis on the query content to obtain a query intention vector, and the knowledge content with the highest matching degree is retrieved in the encrypted knowledge base according to the query intention vector.

7. The method according to claim 1, characterized in that The method adopts corresponding digital human video visualization schemes according to different sources of answers, wherein a digital human driving engine is used to match lip shape, expression, and action, and synthesize a corresponding digital human video, including: Using a content recognition method to classify and recognize the answer content, if the recognition result comes from a preset content database or an enterprise knowledge base, obtaining a corresponding video material identifier from the preset content database or the enterprise knowledge base; Extracting video material data according to the video material identifier, and obtaining a digital human expression sequence and audio data from the video material data by a video frame segmentation method; A standard audio waveform is generated by using a speech synthesis method for the audio data, and a phoneme label sequence and lip shape parameters are extracted from the standard audio waveform by using a phoneme recognizer; Based on the phoneme tag sequence, a standard lip shape library is retrieved to obtain a lip shape parameter sequence, digital human animation data is generated according to the lip shape parameter sequence and expression sequence, and the digital human animation data is three-dimensionally reprojected by a parallax rendering method; It also includes: in the process of synthesizing digital human videos, modeling and imitating learning of massive real-life video data, and optimizing the digital human's response strategy and interaction method based on user feedback dialogue.

8. The method according to claim 7, characterized in that In the process of synthesizing digital human videos, massive amounts of real-person video data are modeled and imitated for learning, and the digital human's response strategy and interaction method are optimized based on user feedback dialogue, including: Obtain the human body key point coordinate sequence and facial feature point sequence in real-person video data to obtain the human body action feature matrix; According to the human body motion feature matrix, a joint angle mapping method is used to convert the human body motion feature matrix to obtain a standardized motion feature vector; Constructing an action generator through a recurrent neural network for the standardized action feature vector, and performing interpolation and completion processing on the action frame sequence; The completed action frame sequence is smoothed using Bezier curve to obtain a smooth action sequence; A state transition matrix is ​​constructed based on user satisfaction scores, the state transition probability is updated through a reinforcement learning method, and the optimal action response sequence is selected.

9. The method according to claim 1, characterized in that: The step of displaying the digital human video through a transparent LCD screen comprises: Acquire illumination data in a display environment through a light sensor array, and generate an environmental parameter matrix including ambient light intensity values, light incident angle values, and color temperature distribution values ​​according to the illumination data; Processing the environment parameter matrix using a lighting parameter mapping method to obtain a display parameter vector including display brightness value, contrast value, and color saturation value; Performing brightness gain processing on the original digital human image according to the display parameter vector to obtain a brightness compensated image, and performing color correction on the brightness compensated image according to the color temperature distribution value in the environmental parameter matrix to obtain a color corrected image; The left-eye and right-eye parallax values ​​are calculated for the color-corrected image, a light refraction compensation method is used to generate a stereoscopic parallax image pair, and visual angle compensation is performed on the stereoscopic parallax image pair according to the observation distance value to obtain a final display image.

10. A digital human interaction system based on naked eye 3D visualization, characterized in that: The system comprises: A multimodal input processing module is used to obtain the query content provided by the user in a multimodal manner, and use natural language processing technology to perform semantic understanding and intent analysis on the query content to obtain user intent data; A preset content matching module is used to perform a matching search in a preset content database according to the user intention data, and if there is a highly relevant answer template or video clip, the preset content is called to feedback to the user; An enterprise knowledge base retrieval module is used to transfer the query content to the enterprise knowledge base if the preset content database fails to meet the matching search requirements, extract key information from the enterprise knowledge base using information retrieval technology, and submit the query content to a pre-trained language model if the relevance of the information extracted from the enterprise knowledge base is lower than a preset threshold; The general large model underlying module is used for receiving the query content, using the pre-trained large-scale language model, combined with the massive information resources on the Internet, inputting the question and combining the context information for reasoning, and dynamically constructing the answer through text generation technology; The digital human video synthesis module is used to adopt corresponding digital human video visualization solutions according to different answer sources, using the digital human driving engine to match lip shape, expression, and action to synthesize the appropriate digital human video; The naked eye 3D display module is used to display the digital human video through a transparent LCD screen.

Images