Response generation method and device based on digital portrait, equipment and storage medium

Abstract

The embodiment of the invention provides a response generation method and device based on a digital portrait, equipment and a storage medium, belongs to the technical field of artificial intelligence, and is applied to a financial scene. The method comprises the following steps: acquiring user feature data and question video data of a question object; obtaining answer audio data made by a response object to the question video data; generating preliminary answer expression information based on the answer audio data; based on the answer audio data, the user feature data and the question video data, performing expression correction on the preliminary answer expression information to obtain target answer expression information; and generating a digital portrait response video based on the target answer expression information and the answer audio data. According to the embodiment of the invention, the expression accuracy of the digital portrait can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology and is applied to the financial scenario. In particular, it relates to a method, device, equipment and storage medium for generating responses based on digital human images. Background Art

[0002] A digital human image is a virtual character image that can be used in scenarios for interacting with users. For example, when a user consults an insurance staff member online, the insurance staff member makes a response, and a corresponding digital human animation is generated based on the audio content of the response by the insurance staff member, and the generated digital human animation is used to provide an answer to the user. However, the facial expressions of the human animation generated in the above scenario are not accurate, and the user experience is poor. Therefore, how to improve the facial expression accuracy of digital human images has become an urgent problem to be solved. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for generating responses based on digital human images, aiming to improve the facial expression accuracy of digital human images.

[0004] To achieve the above object, a first aspect of the embodiments of the present application proposes a method for generating responses based on digital human images, the method comprising:

[0005] Obtaining user feature data and question video data of the questioner;

[0006] Obtaining answer audio data made by the responder to the question video data;

[0007] Generating preliminary answer expression information based on the answer audio data;

[0008] Performing expression correction on the preliminary answer expression information based on the answer audio data, the user feature data and the question video data to obtain target answer expression information;

[0009] Generating a digital human response video based on the target answer expression information and the answer audio data.

[0010] In some embodiments, the user feature data includes insurance purchase record information, insurance browsing record information, and user identity attribute information, the question video data includes question video frames and question audio data, and the performing expression correction on the preliminary answer expression information based on the answer audio data, the user feature data and the question video data to obtain target answer expression information includes:

[0011] Performing insurance purchase intention recognition based on the insurance purchase record information and the insurance browsing record information to obtain insurance purchase intention information;

[0012] Construct a user portrait based on the user identity attribute information to obtain user portrait information;

[0013] Conduct a demand analysis on the question object based on the question audio data to obtain question information;

[0014] Conduct an expression recognition on the question object based on the question video frame to obtain expression information;

[0015] Based on the answer audio data, the insurance intention information, the user portrait information, the question information, and the expression information, perform expression correction on the preliminary answer expression information to obtain the target answer expression information.

[0016] In some embodiments, the performing expression correction on the preliminary answer expression information based on the answer audio data, the insurance intention information, the user portrait information, the question information, and the expression information to obtain the target answer expression information includes:

[0017] Perform text conversion on the answer audio data to obtain text data, and perform text division on the text data to obtain at least one text paragraph;

[0018] Through an attention prediction model, based on the insurance intention information, the user portrait information, the question information, and the expression information, determine the attention of the question object to each text paragraph;

[0019] Through a preset expression mapping table, based on each attention, determine the expression correction value of each text paragraph;

[0020] Based on each expression correction value, perform expression correction on the preliminary answer expression information to obtain the target answer expression information.

[0021] In some embodiments, the answer audio data includes at least two original audio segments, and the generating preliminary answer expression information based on the answer audio data includes:

[0022] For each original audio segment, through an expression generation model, a voice feature extraction model, a sliding window, and a sliding step, perform expression information generation on the original audio segment to generate segment expression information;

[0023] Perform expression information splicing based on the segment expression information of each original audio segment to obtain the original answer expression information.

[0024] In some embodiments, the performing expression information generation on the original audio segment through an expression generation model, a voice feature extraction model, a sliding window, and a sliding step to generate segment expression information includes:

[0025] Obtain the cache status of the audio slice buffer to obtain cache status information;

[0026] If the cache status information indicates that the audio slice buffer caches cached audio slices, splice the original audio segment according to the cached audio slices to obtain a target audio segment, and clear the audio slice buffer; wherein, the cached audio slice is a slice of the previous audio segment, the previous audio segment is one of the original audio segments, the timestamp of the previous audio segment is less than the timestamp of the original audio segment, and the timestamp of the previous audio segment is adjacent to the timestamp of the original audio segment;

[0027] Perform audio slicing on the target audio segment through the audio slice buffer, the sliding window, and the sliding step to obtain target audio slices;

[0028] Generate segment voice feature information based on the target audio slices through the voice feature extraction model;

[0029] Generate the segment expression information based on the segment voice feature information through the expression generation model.

[0030] In some embodiments, the performing audio slicing on the target audio segment through the audio slice buffer, the sliding window, and the sliding step to obtain target audio slices includes:

[0031] Perform audio slicing on the target audio segment based on the sliding window and the sliding step to obtain a set of candidate audio slices, wherein the set of candidate audio slices includes at least one candidate audio slice;

[0032] Extract the audio slice with the largest timestamp from the set of candidate audio slices to obtain a selected audio slice;

[0033] Update the set of candidate audio slices based on the selected audio slice through the sliding window;

[0034] Use each candidate audio slice in the updated set of candidate audio slices as the target audio slice.

[0035] In some embodiments, the updating the set of candidate audio slices based on the selected audio slice through the sliding window includes:

[0036] If the length of the selected audio slice is less than the length of the sliding window, and the timestamp of the original audio segment corresponding to the target audio segment is the largest among the timestamps of all the original audio segments, zero-padding is performed on the selected audio slice to obtain an updated audio slice, and the updated audio slice is added to the candidate audio slice set;

[0037] If the length of the selected audio slice of the original audio segment is equal to the sliding window, the selected audio slice is added to the candidate audio slice set.

[0038] To achieve the above object, a second aspect of the embodiments of the present application provides a response generation device based on a digital portrait, the device includes:

[0039] A first acquisition module, configured to acquire user feature data and question video data of a questioner;

[0040] A second acquisition module, configured to acquire answer audio data made by a responder to the question video data;

[0041] An expression generation module, configured to generate preliminary answer expression information based on the answer audio data;

[0042] An expression correction module, configured to correct the preliminary answer expression information based on the answer audio data, the user feature data, and the question video data to obtain target answer expression information;

[0043] A video generation module, configured to generate a digital portrait response video based on the target answer expression information and the answer audio data.

[0044] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0045] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0046] A method, device, equipment and storage medium for generating responses based on digital portraits proposed in this application obtain user feature data, question video data of the questioner, and answer audio data of the responder, generate preliminary answer expression information in combination with the answer audio data, and correct the preliminary answer expression information based on the user feature data, question video data and answer audio data to obtain target answer expression information that better conforms to the voice semantic expression. Finally, a digital portrait response video is generated based on the target answer expression information and the answer audio data, thereby improving the expression accuracy of the digital portrait, making it more natural and in line with the actual context during the expression process, and further optimizing the user experience during the interaction process, overcoming the problem of poor user experience caused by inaccurate expressions in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of a method for generating responses based on digital portraits provided by an embodiment of this application;

[0048] Figure 2 is Figure 1 a flowchart of step S103 in

[0049] Figure 3 is Figure 2 a flowchart of step S201 in

[0050] Figure 4 is Figure 3 a flowchart of step S303 in

[0051] Figure 5 is Figure 4 a flowchart of step S403 in

[0052] Figure 6 is Figure 1 a flowchart of step S104 in

[0053] Figure 7 is Figure 6 a flowchart of step S605 in

[0054] Figure 8 is a schematic structural diagram of a device for generating responses based on digital portraits provided by an embodiment of this application;

[0055] Figure 9 is a schematic hardware structure diagram of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] In order to make the objectives, technical solutions and advantages of this application clearer, the following further elaborates on this application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0057] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different module division from that in the device or a different order from that in the flowchart. Terms such as "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some terms involved in this application are analyzed as follows:

[0060] Sliding Window: It refers to a windowing processing method used in data processing, which is commonly used in fields such as signal processing, computer vision, natural language processing, and time series analysis. The sliding window sets a window of a fixed size and slides on the data sequence at a certain step length, and each time it intercepts the data within the window range for processing. This method can improve the locality of data processing, reduce computational redundancy, and achieve efficient information extraction in continuous data processing scenarios. For example, in the field of speech signal processing, the sliding window is used for speech feature extraction, dividing audio data into small segments and analyzing them step by step; in the field of computer vision, the sliding window can be used for object detection, gradually scanning image regions to identify target objects; in time series analysis, the sliding window can be used to calculate the moving average, smooth the data trend, or perform anomaly detection. The selection of the size and step length of the sliding window will directly affect the processing effect, and usually needs to be optimized and adjusted according to the specific application scenario to balance computational efficiency and information integrity.

[0061] A digital human image is a virtual human image that can be used in scenarios where interaction with users is involved. For example, when a user consults an insurance staff member online, the insurance staff member makes a reply, and a corresponding digital human animation is generated based on the audio content of the insurance staff member's reply, and the generated digital human animation is used to provide an answer to the user. However, the facial expressions of the human animation generated in the above scenario are not accurate enough, and the user experience is poor. Therefore, how to improve the facial expression accuracy of digital human images has become an urgent problem to be solved.

[0062] Based on this, the embodiments of this application provide a method, device, equipment and storage medium for generating responses based on digital human images, aiming to improve the facial expression accuracy of digital human images.

[0063] A method and device, equipment, and storage medium for response generation based on digital portraits provided by the embodiments of the present application will be specifically described through the following embodiments. First, the method for response generation based on digital portraits in the embodiments of the present application will be described.

[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machine simulations of digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0065] The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0066] The method for response generation based on digital portraits provided by the embodiments of the present application relates to the field of artificial intelligence technology and is applied to the financial scenario. The method for response generation based on digital portraits provided by the embodiments of the present application can be applied to a terminal, a server, or software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for response generation based on digital portraits, etc., but is not limited to the above forms.

[0067] This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] It should be noted that in each specific embodiment of this application, when it comes to performing relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of this application embodiment will be obtained.

[0069] Figure 1 is an optional flowchart of the response generation method based on digital portraits provided by the embodiments of this application, Figure 1 The method in may include but is not limited to steps S101 to S105.

[0070] Step S101, obtain the user feature data and the question video data of the questioner;

[0071] Step S102, obtain the answer audio data made by the responder to the question video data;

[0072] Step S103, generate preliminary answer expression information based on the answer audio data;

[0073] Step S104, perform expression correction on the preliminary answer expression information based on the answer audio data, user feature data, and question video data to obtain the target answer expression information;

[0074] Step S105, generate a digital portrait response video based on the target answer expression information and the answer audio data.

[0075] Steps S101 to S105 illustrated in the embodiments of the present application, by obtaining user characteristic data of the questioner, question video data, and answer audio data of the responder, generating preliminary answer expression information in combination with the answer audio data, and correcting the preliminary answer expression information based on the user characteristic data, question video data, and answer audio data to obtain target answer expression information that better conforms to the speech semantics expression, and finally generating a digital human portrait response video based on the target answer expression information and the answer audio data, thereby achieving the improvement of the expression accuracy of the digital human portrait, making it more natural and more in line with the actual context during the expression process, and further optimizing the user experience during the interaction process, overcoming the problem of poor user experience caused by inaccurate expressions in the prior art.

[0076] In step S101 of some embodiments, the questioner is a user who consults questions online with an insurance customer service. The user can conduct real-time interactions through intelligent terminals such as smartphones, tablets, computers, etc. The user characteristic data includes insurance purchase record information, insurance browsing record information, user identity attribute information, etc. The user data characteristic data can be queried through a database, or a form filled in by the user can be obtained before consulting questions online. The question video data refers to the video data recorded by the user during the consultation process. This data includes question video frames and question audio data, which can provide a reference for subsequent expression correction. The question video data can be recorded in real time through the camera of the user device and uploaded to the server for storage and processing through a data transmission module.

[0077] In step S102 of some embodiments, the responder is an online customer service staff of an insurance company or an intelligent customer service system based on a knowledge base. The human customer service can obtain the user's question in real time through a headset microphone device and answer it in voice; the intelligent customer service system can automatically generate a voice answer through text processing and speech synthesis technologies (such as TTS, Text-to-Speech). The answer audio data refers to the voice answer made by the responder in response to the question video data. This data includes speech features such as speech content, intonation, pause, and rhythm to ensure the naturalness and coherence of the answer. The method of obtaining the answer audio data can include real-time voice recording, that is, the human customer service answers in real time through a microphone and transmits the audio data to the system for storage and processing; or, in the intelligent customer service scenario, after the system analyzes the user's question, it matches the corresponding answer from the knowledge base and uses speech synthesis technology to generate an audio answer for subsequent digital human portrait expression generation and synchronization processing.

[0078] Please refer to Figure 2 , in some embodiments, the answer audio data includes at least two original audio segments, and step S103 may include but is not limited to steps S201 to S202:

[0079] Step S201: For each original audio segment, use an expression generation model, a speech feature extraction model, a sliding window, and a sliding step size to generate expression information for the original audio segment, and generate segment expression information.

[0080] Step S202: Based on the segment expression information of each original audio segment, perform expression information splicing to obtain the original response expression information.

[0081] Steps S201 to S202 shown in the embodiments of the present application use an expression generation model, a speech feature extraction model, and combine the sliding window and sliding step size technology to generate expression information for each original audio segment one by one, so that the expression information can accurately correspond to the instantaneous features of the audio, ensuring the fineness of the expression change. Then, based on the segment expression information of each segment, splicing is performed to form the complete original response expression information, thus avoiding the problems of sudden changes and incoherence between expression information, making the expression performance of the digital portrait more natural and smooth. In this way, the expression synchronization effect of the digital portrait driven by speech is effectively improved, and the expression detail performance ability is improved, thereby enhancing the user's interaction experience.

[0082] Please refer to Figure 3 , in some embodiments, step S201 may include but is not limited to steps S301 to S305:

[0083] Step S301: Obtain the cache status of the audio slice buffer to obtain cache status information.

[0084] Step S302: If the cache status information indicates that the audio slice buffer caches cached audio slices, according to the cached audio slices, splice the original audio segment to obtain a target audio segment, and empty the audio slice buffer; wherein, the cached audio slice is a slice of the previous audio segment, the previous audio segment is one of the original audio segments, the timestamp of the previous audio segment is less than the timestamp of the original audio segment, and the timestamp of the previous audio segment is adjacent to the timestamp of the original audio segment.

[0085] Step S303: Use the audio slice buffer, a sliding window, and a sliding step size to slice the target audio segment to obtain target audio slices.

[0086] Step S304: Through the speech feature extraction model, generate segment speech feature information based on the target audio slices.

[0087] Step S305: Through the expression generation model, generate segment expression information based on the segment speech feature information.

[0088] Steps S301 to S305 illustrated in the embodiments of the present application improve the expression coherence of digital portraits and the accuracy of voice driving by optimizing the audio slice caching mechanism, audio splicing processing, and expression generation process. Specifically, first, by obtaining the caching status of the audio slice cache and performing splicing processing when there are cached audio slices, the target audio segment can make full use of the voice information of adjacent timestamps, reduce the impact of audio mutations on expression generation, and at the same time clear the cache to ensure the real-time nature of data processing. Then, based on the audio slice cache, sliding window, and sliding step, the target audio segment is sliced into audio slices, so that the audio signal can be processed more smoothly in the time dimension. Finally, through the voice feature extraction model and expression generation model respectively, voice feature analysis is performed on the target audio slices, and corresponding expression information is generated, so that the expression changes of the digital portrait are more accurate and conform to the voice semantic expression. In this way, the naturalness and coherence of the expressions of digital portraits are effectively improved, and problems such as sudden changes in expression information, unstable audio processing, and expression synchronization deviation in the prior art are overcome, thereby enhancing the user's interaction experience.

[0089] In step S301 of some embodiments, the audio slice cache is a data cache module for temporarily storing audio slices. This cache can store the slice data of the previous audio segment to support subsequent audio splicing processing. The cache status information refers to the data reflecting the current storage status of the audio slice cache, including whether the cache is empty and the timestamp information of the cached audio slices.

[0090] In step S302 of some embodiments, when the cache status information indicates that there are cached audio slices in the audio slice cache, the cached audio slices need to be spliced with the current original audio segment to form a target audio segment. Specifically, since the cached audio slices are from the previous audio segment, and the timestamp of the previous audio segment is less than that of the current original audio segment, and the two timestamps are adjacent, that is, the slices in the cached audio slices must be the slices left over from the previous audio. As for what circumstances will leave slices, it will be elaborated in detail below. After splicing, to avoid reusing cached data, the audio slice cache needs to be cleared to store the audio slices at the next moment, so as to ensure the real-time nature and accuracy of audio processing.

[0091] Please refer to Figure 4 , in some embodiments, step S303 may include but is not limited to steps S401 to S404:

[0092] Step S401, slice the target audio segment based on the sliding window and sliding step to obtain a set of candidate audio slices, where the set of candidate audio slices includes at least one candidate audio slice;

[0093] Step S402: Extract the audio slice with the largest timestamp from the candidate audio slice set to obtain the selected audio slice;

[0094] Step S403: Update the candidate audio slice set based on the selected audio slice through a sliding window;

[0095] Step S404: Use each candidate audio slice in the updated candidate audio slice set as the target audio slice.

[0096] Steps S401 to S404 illustrated in the embodiments of the present application slice the target audio segment by using a sliding window and a sliding step length to generate a candidate audio slice set, then select the audio slice with the largest timestamp from the candidate audio slice set, then update the candidate audio slice set through the sliding window mechanism, and finally use each candidate audio slice in the updated candidate audio slice set as the new target audio segment, so as to merge the slices left over from the previous audio into the current audio.

[0097] In step S401 of some embodiments, the sliding window is used to define the current processing range of the target audio segment, and the sliding step length is used to control the moving speed of the window, so as to ensure that the audio segment can be segmented with a certain overlap ratio. The audio slicing process includes segmenting the target audio segment according to the set sliding window length and controlling the interval between slices with the sliding step length, so that there is a certain degree of overlap between adjacent slices. Candidate audio slices refer to the audio data segments obtained through the above slicing process. Each audio slice contains voice information of a certain time length and retains the corresponding timestamp information. The candidate audio slice set refers to a set composed of multiple candidate audio slices, where each slice is derived from the target audio segment and is used for subsequent speech feature extraction and expression generation.

[0098] In step S402 of some embodiments, the selected audio slice refers to the audio slice with the largest timestamp extracted from the candidate audio slice set. The timestamp refers to the time mark of the audio slice in the target audio segment and is used to represent the time position of the audio slice relative to the original audio.

[0099] Please refer to Figure 5 , in some embodiments, step S403 includes but is not limited to steps S501 to S502:

[0100] Step S501: If the length of the selected audio slice is less than the length of the sliding window and the timestamp of the original audio segment corresponding to the target audio segment is the largest among the timestamps of all original audio segments, perform zero-padding on the selected audio slice to obtain an updated audio slice, and add the updated audio slice to the candidate audio slice set;

[0101] Step S502: If the length of the selected audio slice of the original audio segment is equal to the sliding window, add the selected audio slice to the candidate audio slice set.

[0102] Steps S501 to S502 illustrated in the embodiments of the present application, in the case where the length of the selected audio slice is less than the length of the sliding window, make up for the missing part by zero-padding, so that the updated audio slice meets the requirements of the sliding window and is added to the candidate audio slice set, to reduce the problem of missing speech information caused by insufficient slice length. And in the case where the length of the selected audio slice is equal to the sliding window, directly add the selected audio slice to the candidate audio slice set. In this way, the processing method of audio slices is effectively optimized, the integrity of audio data in the time dimension is improved, and more stable and continuous data input is provided for subsequent speech feature extraction and expression generation, thereby enhancing the expression synchronization effect and naturalness of digital portraits.

[0103] In step S501 of some embodiments, when the length of the selected audio slice is less than the length of the sliding window, zero-padding operation can be used to supplement the insufficient audio data, so that the length of the updated audio slice reaches the preset standard of the sliding window. The zero-padding method can use silent data filling to avoid generating additional audio signal interference and maintain the alignment accuracy of audio data on the time axis. In addition, this step also ensures the integrity of the target audio segment in terms of timestamp, that is, only when the timestamp of the original audio segment corresponding to the target audio segment is the largest among the timestamps of all original audio segments, filling is performed to avoid audio information redundancy caused by unnecessary data expansion.

[0104] In step S502 of some embodiments, when the length of the selected audio slice is equal to the length of the sliding window, no additional data filling process is required, and the audio slice can be directly added to the candidate audio slice set. Since the audio slice already meets the preset standard of the sliding window at this time, no further adjustment is required, thereby reducing the additional computational overhead and ensuring the efficiency and real-time performance of audio data processing.

[0105] After step S520 in some embodiments, it further includes: if the length of the selected audio slice is less than the length of the sliding window, and the timestamp of the original audio segment corresponding to the target audio segment is not the largest among the timestamps of each original audio segment, cache the selected audio slice in the audio slice buffer;

[0106] It should be noted that zero-padding is performed on the selected audio slice when the timestamp of the original audio segment corresponding to the target audio segment is the largest among the timestamps of each original audio segment.

[0107] If the length of the selected audio slice is less than the length of the sliding window, and the timestamp of the original audio segment corresponding to the target audio segment is not the largest among the timestamps of all original audio segments, cache the selected audio slice in the audio slice buffer.

[0108] If the length of the selected audio slice of the original audio segment is equal to the sliding window, add the selected audio slice to the candidate audio slice set.

[0109] This is because when audio processing performs feature extraction in units of sliding windows, the audio length of each window must be strictly aligned with the feature length after passing through the model to ensure audio-visual synchronization. If the length of the selected audio slice is less than the length of the sliding window and its timestamp is already the maximum among all segments, it means that the end of a sentence or a piece of speech has been reached and there is no subsequent audio to supplement the current window. Therefore, padding the length with zero filling at this time will not disrupt the speech continuity, satisfying both the necessary condition of feature and audio data length alignment and avoiding the insertion of additional silent segments, thus preventing local feature disorders caused by silence. At the same time, if the length of the selected audio slice is less than the length of the sliding window and the timestamp is not the maximum, it means that there is still available audio data in the future. Therefore, the slice is temporarily stored in the audio slice buffer for use in combination with the audio data processed next time. If the length of the selected audio slice is exactly equal to the length of the sliding window, it is directly added to the candidate audio slice set for further feature processing. This optimized design ensures the accuracy and smoothness of feature extraction during real-time audio processing, significantly improving the synchronization effect of speech and video in real-time digital human technology.

[0110] When implementing real-time digital human audio slice processing in the prior art, it is usually adopted to insert additional silent segments after a given extremely short audio (such as 400 ms) to make up the length of the sliding window to ensure the strict alignment of the audio data with the feature sequence output by the model (such as HuBERT). Although this method can meet the requirement of the duration consistency between the feature and the audio, it has obvious deficiencies: artificially inserting silent segments will disrupt the original continuity of the audio, resulting in disorders in the local features extracted by the model, thus having an adverse impact on subsequent lip synchronization and facial action generation. In addition, since the HuBERT model itself is more suitable for batch processing of longer audio, the existing method of performing frame-by-frame inference by supplementing silent segments piecemeal further exacerbates the instability of the model processing.

[0111] Therefore, in this embodiment, a large number of silent segments are no longer added artificially during each sliding window processing. Instead, audio segments that are not long enough for the sliding window and cannot be directly used for model inference are retained and superimposed on the next audio processing process for continued use. Zero padding is only used appropriately when the audio data is clearly at the end of a sentence and cannot be superimposed further backwards, which cleverly avoids the problem of artificial segmentation of audio data, maintains the continuity of the voice signal and the stability of the features, and further ensures the real-time and accurate synchronization of the digital human's expression and audio, achieving real-time, accurate and continuous audio and video effects that are difficult to achieve with existing technologies.

[0112] In step S304 of some embodiments, the speech feature extraction model is HuBERT (Hidden-Unit BERT), which is a self-supervised speech representation learning model that can extract high-dimensional speech features with semantic information by modeling the hidden units of the audio signal. The HuBERT model can learn speech features by clustering speech data, mask prediction, etc. without labeled data, so that it has strong adaptability and generalization ability in speech feature extraction tasks. The segment speech feature information refers to the speech feature data extracted based on the target audio slice, including but not limited to the spectral information of the speech, Mel cepstral coefficients (MFCC), speech fundamental frequency (F0), time-dependent information, and implicit speech semantic features. This feature information can be used in the subsequent expression generation model to ensure that the expression dynamics of the digital portrait can accurately reflect the speech content and emotional changes. The segment speech feature information obtained by the HuBERT speech feature extraction model can improve the accuracy of speech-driven expression generation, make the facial expression of the digital portrait more matched with the speech semantics, and thus optimize the user's interactive experience in the insurance customer service online question and answer scenario.

[0113] In step S305 of some embodiments, the expression generation model can be an expression mapping model based on deep learning, such as Wav2Lip, EMOCA (Emotional Capture and Animation) or an expression generation network based on GAN (Generative Adversarial Network). The model can derive corresponding facial muscle action parameters based on the input voice feature information, and generate expression information that conforms to voice semantics and emotional changes. Segment expression information refers to expression parameter data generated based on segment voice feature information, including but not limited to mouth shape changes, eyebrow movements, eye changes, cheek muscle adjustments, etc.

[0114] In step S202 of some embodiments, the original response expression information refers to the segment expression information generated based on each original audio segment, and the complete expression data sequence formed by splicing the expression information. This information is used to subsequently generate a digital human portrait expression animation synchronized with the response speech. The original response expression information includes data such as facial key point data, muscle movement parameters, and expression change sequences to ensure that the expressions of the digital human portrait can accurately reflect the speech content and emotional state. The way of splicing the expression information can be carried out by using time series alignment and feature fusion methods. First, in the time dimension, ensure that the timestamps of each segment expression information correspond to the original audio segments to ensure the temporal continuity of the expression sequence. Second, in terms of spatial features, through interpolation, smoothing filtering, or the dynamic adjustment mechanism of the deep learning model, make the transition between adjacent segment expression information more natural and avoid problems such as expression mutations or breaks. For example, the RNN (Recurrent Neural Network) or Transformer structure can be used to analyze the expression features between the front and back segments and adjust the expression parameters at the splicing boundary to make it smoothly transition. In addition, during the splicing process, a weight adjustment mechanism can be introduced to ensure that key expression changes (such as mouth opening and closing, eyebrow raising, and eye movement) can be consistent at the splicing points, so as to ensure that the finally generated original response expression information matches the response audio and make the expressions of the digital human portrait in the online insurance customer service Q&A scenario more natural and smooth.

[0115] Please refer to Figure 6 , in some embodiments, the user feature data includes insurance purchase record information, insurance browsing record information, and user identity attribute information, the question video data includes question video frames and question audio data, and step S104 includes but is not limited to steps S601 to S605:

[0116] Step S601, perform insurance purchase intention recognition based on the insurance purchase record information and the insurance browsing record information to obtain insurance purchase intention information;

[0117] Step S602, construct a user portrait based on the user identity attribute information to obtain user portrait information;

[0118] Step S603, perform demand analysis on the question object based on the question audio data to obtain question information;

[0119] Step S604, perform expression recognition on the question object based on the question video frames to obtain expression information;

[0120] Step S605, perform expression correction on the preliminary response expression information based on the answer audio data, the insurance purchase intention information, the user portrait information, the question information, and the expression information to obtain the target response expression information.

[0121] Steps S601 to S605 shown in the embodiment of the present application realize the recognition of insurance intention by combining the user's insurance record information and insurance browsing record information, so as to obtain the user's current concerns and potential needs for insurance products. On this basis, a user portrait is further constructed based on the user's identity attribute information, so as to provide a more comprehensive personalized user feature description, so that the system can optimize the expression feedback according to the characteristics of different user groups. Subsequently, the demand analysis is carried out through the question audio data, the core of the problem that the user is concerned about is accurately extracted, and the expression recognition is carried out based on the question video frame to identify the user's emotional state when asking questions, providing a basis for subsequent expression adjustment. Finally, by combining the answer audio data, insurance intention information, user portrait information, question information and expression information, the expression correction is carried out on the preliminary answer expression information to ensure that the expression of the digital portrait can match the answer content and adapt to the user's current emotions and demand changes. In this way, the expression accuracy and naturalness of the digital portrait are effectively improved, so that the customer service system can provide a more humane and personalized interactive experience, overcome the problems of dull expressions and lack of user intention matching in the prior art, and optimize the interactive quality and user experience of online Q&A of insurance customer service.

[0122] In step S601 of some embodiments, insurance record information refers to the insurance behavior data of the user on the insurance platform in the past, including information such as the type of insurance products purchased, insurance time, insurance amount, coverage period, and claim records. Insurance browsing record information refers to the browsing behavior of the user on the insurance platform, including the category of insurance products browsed, dwell time, click-through rate, search keywords, and viewed protection plans. Insurance intention information refers to the user's purchase tendency for a specific insurance product inferred based on insurance record information and insurance browsing record information, which may include the user's interest intensity in a certain type of insurance, the expected insurance amount, the main coverage content of concern, etc. Insurance intention recognition can be achieved in a variety of ways, such as a user behavior prediction model based on machine learning or deep learning, or a rule-based logical analysis method, combined with the user's browsing preferences, historical insurance behavior, and market trends, to predict the user's potential insurance intention.

[0123] In step S602 of some embodiments, the user identity attribute information refers to the basic information of the user, including but not limited to age, gender, occupation, income level, family status, health status, risk preference, etc. The user portrait information is a comprehensive description of user characteristics constructed based on the user identity attribute information, and usually includes the user's insurance demand preference, claim risk assessment, possible purchasing ability and behavior pattern. The way to construct the user portrait can be realized through data mining and machine learning models. For example, clustering analysis (such as K-means), collaborative filtering algorithm or deep neural network is used to classify user data, and combined with the user's historical behavior pattern to generate a personalized user portrait, so as to improve the pertinence of insurance purchase recommendation and customer service.

[0124] In step S603 of some embodiments, the question audio data refers to the consultation questions submitted by the user in the form of voice in the online Q&A scenario of the insurance customer service, including information such as voice content, intonation, speech rate, pause, etc. The question information is the core demand content of the user extracted based on the question audio data. For example, the user's specific questions about insurance terms, claim conditions, and payment methods. The demand analysis can be carried out through natural language processing (NLP) technology combined with a speech recognition (ASR) system. For example, semantic analysis is carried out using models such as BERT, TF-IDF or LSTM to extract keywords, sentiment tendency and context relationship from the question audio data, so as to accurately identify the category of questions concerned by the user and provide an optimization plan for the customer service response.

[0125] In step S604 of some embodiments, the question video frame refers to the user's facial image frame captured by the camera during the user's consultation process, including information such as the user's facial expression, eye expression, mouth shape, etc. The expression information is the current emotional state of the user recognized based on the question video frame, including expression categories such as pleasure, confusion, anxiety, anger, etc., and facial muscle change characteristics. The expression recognition can be realized through computer vision technology. For example, an expression recognition model based on a convolutional neural network (CNN) (such as Facial Action Coding System, FACS) or a deep learning framework (such as OpenFace, EmotionNet) is used to detect and classify the user's facial feature points to identify their current emotional state, so as to optimize the customer service response strategy, make the expression of the digital portrait more in line with the user's psychological expectation, and improve the interaction experience.

[0126] Please refer to Figure 7 , in some embodiments, step S605 may include but is not limited to steps S701 to S704:

[0127] Step S701, perform text conversion on the answer audio data to obtain text data, and perform text segmentation on the text data to obtain at least one text paragraph;

[0128] Step S702, determining the attention of the question subject to each text paragraph through the attention prediction model based on the insurance intention information, user portrait information, question information and expression information;

[0129] Step S703, determining the expression correction value of each text paragraph based on each attention degree through a preset expression mapping table;

[0130] Step S704, based on each expression correction value, the preliminary answer expression information is corrected to obtain the target answer expression information.

[0131] In the steps S701 to S704 shown in the embodiment of the present application, the answer audio data is first converted into text to obtain text data, and the text data is divided to generate at least one text paragraph, so that the answer content can be analyzed in a fragmented form. Then, based on the insurance intention information, user portrait information, question information and expression information, the attention prediction model is used to analyze the attention degree of the question object to each text paragraph to identify the user's key focus content in the conversation, thereby providing a basis for expression optimization. Next, based on the attention degree of each text paragraph, the expression correction value is determined from the preset expression mapping table, so that the expression adjustment can enhance or weaken the content of interest to the user, ensuring the rationality of the expression change. Finally, based on each expression correction value, the preliminary answer expression information is corrected so that the target answer expression information finally generated can match the focus of the answer content and the user's emotional feedback. In this way, the expression expressiveness of the digital portrait is effectively improved, so that it can not only reflect the voice rhythm, but also be personalized in combination with the user's focus, overcoming the problems of dull expression and lack of pertinence in the prior art, thereby optimizing the user interaction experience in the online question and answer scenario of insurance customer service.

[0132] In step S701 of some embodiments, the text data refers to the text content obtained by performing speech recognition on the response audio data, usually containing complete customer service response information, which may involve insurance terms, insurance purchase suggestions, claim settlement processes, etc. The text conversion can be achieved through automatic speech recognition (ASR) technology, such as based on deep learning models like DeepSpeech, Whisper, Wav2Vec2.0, etc., to convert the audio signal into text information, and perform grammar correction and punctuation completion on the converted text. A text paragraph refers to several parts into which the converted text data is split according to logical or semantic relationships. Each text paragraph usually corresponds to a complete semantic unit, such as a solution step, an explanation of an insurance policy, or a specific response to the user's question. Possible ways of text division include rule-based methods (such as splitting by punctuation marks, keywords, or grammatical structures), or machine learning-based text segmentation methods (such as BERT, TextRank), to ensure that the divided text paragraphs conform to semantic logic.

[0133] In step S702 of some embodiments, the attention prediction model is a deep learning model based on the attention mechanism (Attention Mechanism), such as Transformer, BERT, or LSTM-Attention structure. This model can analyze the attention level of the questioner to each text paragraph based on the user's insurance purchase intention information, user profile information, question information, and expression information. Attention refers to the interest or importance weight of the user in a specific text paragraph during the interaction process, usually calculated based on the user's historical behaviors such as browsing records, real-time expression reactions such as confusion, concentration, and question content such as keyword matching, and finally generate an attention score for each text paragraph for subsequent expression optimization.

[0134] In step S703 of some embodiments, the preset expression mapping table refers to a mapping table constructed according to the correspondence between attention information and predefined expressions. For example, when the user has a high attention level to a certain paragraph, the digital portrait can appropriately adjust the expression to enhance the communication effect, such as raising the eyebrows, widening the eyes, increasing mouth movements, etc.; while when the attention level is low, the amplitude of expression changes can be reduced to keep the expression relatively stable. The expression correction value refers to the adjustment parameter determined from the preset expression mapping table based on the attention score, and is independently adjusted based on multi-dimensional expression parameters such as mouth shape, eyebrows, and eye expressions. For example, for eyebrows, the expression levels are divided based on a numerical range (such as 0.0 - 0.3 corresponding to calm, 0.3 - 0.6 corresponding to concentration, 0.6 - 1.0 corresponding to emphasis, etc.) to optimize the expression presentation effect of the digital portrait.

[0135] In step S704 of some embodiments, the target response expression information refers to the final expression information obtained by optimizing the preliminary response expression information in combination with the expression correction value, making the facial expression of the digital human more in line with the user's focus and the emotional expression of the response content. Possible ways of expression correction include: rule-based adjustment (such as directly modifying expression parameters according to a preset expression mapping table), dynamic adjustment based on machine learning (such as using GAN or RNN to generate a continuous expression transition sequence), and adaptive optimization based on reinforcement learning (such as continuously adjusting expression model parameters according to historical interaction feedback). Through these optimization methods, the expression of the digital human can more accurately match the speech content and user attention, making the interaction more natural and enhancing the user's immersion and understanding efficiency.

[0136] In step S105 of some embodiments, generating a digital human response video based on the target response expression information and the response audio data includes expression animation generation, audio synchronization processing, and video synthesis. The target response expression information includes facial parameters such as mouth shape, eye expression, and eyebrows to match the semantics and emotions of the response speech. The response audio data is optimized customer service speech to ensure clarity and coherence. The digital human response video can be generated by technologies such as Wav2Lip, GAN, or 3D Blendshape. First, use the target expression information to drive the facial animation, then match the mouth shape with the speech rhythm, and finally synthesize the video. This method improves the naturalness of the expression and the speech synchronization, providing a more realistic and smooth interaction experience for the digital human in the insurance customer service scenario.

[0137] Please refer to Figure 8 , the embodiments of the present application also provide a response generation device based on a digital human, which can implement the above-mentioned response generation method based on a digital human. The device includes:

[0138] A first acquisition module 801, configured to acquire user feature data and question video data of the questioner;

[0139] A second acquisition module 802, configured to acquire response audio data made by the responder to the question video data;

[0140] An expression generation module 803, configured to generate preliminary response expression information based on the response audio data;

[0141] An expression correction module 804, configured to perform expression correction on the preliminary response expression information based on the response audio data, user feature data, and question video data to obtain target response expression information;

[0142] A video generation module 805, configured to generate a digital human response video based on the target response expression information and the response audio data.

[0143] The specific implementation of the digital portrait-based response generation device is basically the same as the specific embodiments of the above digital portrait-based response generation method, and will not be elaborated here.

[0144] An embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above digital portrait-based response generation method. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0145] Please refer to Figure 9 , Figure 9 which illustrates the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0146] A processor 901, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0147] A memory 902, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 is called to execute the digital portrait-based response generation method of the embodiments of the present application;

[0148] An input / output interface 903, which is used to implement information input and output;

[0149] A communication interface 904, which is used to implement communication interaction between this device and other devices. It can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0150] A bus 905, which transmits information between various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0151] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other inside the device through the bus 905.

[0152] An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating responses based on digital portraits is implemented.

[0153] As a non-transitory computer-readable storage medium, a memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] The method for generating responses based on digital portraits, the device for generating responses based on digital portraits, the device, and the storage medium provided by the embodiments of the present application obtain the user characteristic data of the questioner, the question video data, and the answer audio data of the responder, generate preliminary answer expression information in combination with the answer audio data, and correct the preliminary answer expression information based on the user characteristic data, the question video data, and the answer audio data to obtain target answer expression information that better conforms to the voice semantic expression. Finally, a digital portrait response video is generated based on the target answer expression information and the answer audio data, thereby improving the expression accuracy of the digital portrait, making it more natural and more in line with the actual context during the expression process, and further optimizing the user experience during the interaction process, overcoming the problem of poor user experience caused by inaccurate expressions in the prior art.

[0155] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0156] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine some steps, or different steps.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or a suitable combination thereof.

[0159] As used in the specification of this application and the above drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data may be interchanged under appropriate circumstances so that the embodiments of this application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0160] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one)" or a similar expression thereof refers to any combination of these items, including any combination of single items or plural items. For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0162] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0165] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A response generation method based on digital portraits, characterized in that The method includes: Obtaining user characteristic data and question video data of the questioner; Obtaining answer audio data made by the responder to the question video data; Generating preliminary answer expression information based on the answer audio data; Performing expression correction on the preliminary answer expression information based on the answer audio data, the user characteristic data, and the question video data to obtain target answer expression information; Generating a digital human portrait response video based on the target answer expression information and the answer audio data.

2. The method according to claim 1, characterized in that, The user characteristic data includes insurance purchase record information, insurance browsing record information, and user identity attribute information. The question video data includes question video frames and question audio data. The performing expression correction on the preliminary answer expression information based on the answer audio data, the user characteristic data, and the question video data to obtain target answer expression information includes: Performing insurance purchase intention recognition based on the insurance purchase record information and the insurance browsing record information to obtain insurance purchase intention information; Constructing a user portrait based on the user identity attribute information to obtain user portrait information; Performing demand analysis on the questioner based on the question audio data to obtain question information; Performing expression recognition on the questioner based on the question video frames to obtain expression information; Performing expression correction on the preliminary answer expression information based on the answer audio data, the insurance purchase intention information, the user portrait information, the question information, and the expression information to obtain the target answer expression information.

3. The method according to claim 2, wherein The performing expression correction on the preliminary answer expression information based on the answer audio data, the insurance purchase intention information, the user portrait information, the question information, and the expression information to obtain the target answer expression information includes: Performing text conversion on the answer audio data to obtain text data, and partitioning the text data to obtain at least one text paragraph; Determining the attention degree of the questioner to each text paragraph based on the insurance purchase intention information, the user portrait information, the question information, and the expression information through an attention degree prediction model; Determining an expression correction value for each text paragraph based on each attention degree through a preset expression mapping table; Performing expression correction on the preliminary answer expression information based on each expression correction value to obtain the target answer expression information.

4. The method according to any one of claims 1 to 3, characterized in that, The answer audio data includes at least two original audio segments. The generating preliminary answer expression information based on the answer audio data includes: For each original audio segment, generating segment expression information by generating an expression model, a voice feature extraction model, a sliding window, and a sliding step length for the original audio segment; Performing expression information splicing based on the segment expression information of each original audio segment to obtain original answer expression information.

5. The method according to claim 4, wherein The generating segment expression information by generating an expression model, a voice feature extraction model, a sliding window, and a sliding step length for the original audio segment includes: Obtaining the cache state of an audio slice buffer to obtain cache state information; If the cache status information indicates that the audio slice buffer caches cached audio slices, perform audio splicing on the original audio segment according to the cached audio slices to obtain a target audio segment, and clear the audio slice buffer; wherein, the cached audio slices are slices of the previous audio segment, the previous audio segment is one of the original audio segments, the timestamp of the previous audio segment is less than the timestamp of the original audio segment, and the timestamp of the previous audio segment is adjacent to the timestamp of the original audio segment. Perform audio slicing on the target audio segment through the audio slice buffer, the sliding window, and the sliding step to obtain target audio slices. Generate segment voice feature information based on the target audio slices through the voice feature extraction model. Generate the segment expression information based on the segment voice feature information through the expression generation model.

6. The method according to claim 5, wherein The performing audio slicing on the target audio segment through the audio slice buffer, the sliding window, and the sliding step to obtain target audio slices includes: Perform audio slicing on the target audio segment based on the sliding window and the sliding step to obtain a set of candidate audio slices, where the set of candidate audio slices includes at least one candidate audio slice. Extract the audio slice with the largest timestamp from the set of candidate audio slices to obtain a selected audio slice. Update the set of candidate audio slices based on the selected audio slice through the sliding window. Use each candidate audio slice in the updated set of candidate audio slices as the target audio slice.

7. The method according to claim 6, wherein The updating the set of candidate audio slices based on the selected audio slice through the sliding window includes: If the length of the selected audio slice is less than the length of the sliding window, and the timestamp of the original audio segment corresponding to the target audio segment is the largest among the timestamps of all the original audio segments, perform zero-padding on the selected audio slice to obtain an updated audio slice, and add the updated audio slice to the set of candidate audio slices. If the length of the selected audio slice of the original audio segment is equal to the sliding window, add the selected audio slice to the set of candidate audio slices.

8. A response generation device based on digital portraits, characterized in that, The device includes: A first acquisition module, configured to acquire user feature data and question video data of a question object. A second acquisition module, configured to acquire answer audio data made by an answer object to the question video data. An expression generation module, configured to generate preliminary answer expression information based on the answer audio data. An expression correction module, configured to perform expression correction on the preliminary answer expression information based on the answer audio data, the user feature data, and the question video data to obtain target answer expression information. A video generation module, configured to generate a digital human portrait answer video based on the target answer expression information and the answer audio data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method for generating a response based on a digital portrait according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method for generating a response based on a digital portrait according to any one of claims 1 to 7 is implemented.

Images