3D intelligent virtual image interaction method and system based on video depth

By building an intelligent question-and-answer library and obtaining in-depth information of virtual human videos, the problem that virtual digital humans cannot interact in 3D display devices is solved, and the vivid and natural interaction between virtual digital humans in 3D space is realized, which improves the user experience.

CN119090004APending Publication Date: 2024-12-06CHINA CITIC BANK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411023529.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art cannot realize 3D interaction of virtual digital people in 3D display devices. Virtual digital people can only be presented on a plane, and cannot fully reflect the three-dimensional sense and interact in 3D space.

Method used

By collecting Q&A content, building an intelligent Q&A library, obtaining the in-depth information of virtual human videos, and binding the resource ID with Q&A content and deep information, output virtual human videos containing deep information, and using 3D engine rendering or deep learning algorithms to obtain virtual In-depth information of human videos realizes 3D presentation of virtual human videos.

Benefits of technology

It improves the vividness and naturalness of virtual digital human interaction, makes the interaction between users and virtual digital humans more vivid and natural, while maintaining the real-time and efficiency of the interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090004A_ABST
    Figure CN119090004A_ABST
Patent Text Reader

Abstract

The invention relates to a 3D intelligent virtual image interaction method and system based on video depth, and the method comprises the steps: collecting question and answer contents, and constructing an intelligent question and answer library which can be learned based on a large language model; obtaining depth information of a virtual human video, wherein the depth information is a depth map of each frame corresponding to the virtual human video; binding and storing the resource ID, the question and answer content and depth information of the virtual human video corresponding to the question and answer content; and obtaining a virtual human video and a depth information URL according to the resource ID, obtaining resources according to the URL, and outputting answer content and the virtual human video containing the depth information. The virtual digital human video is converted from 2D to 3D by generating the depth information, so that the interaction between the user and the virtual digital human is more vivid and natural, and the real-time performance and efficiency during interaction are not influenced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a 3D intelligent virtual image interaction method and system based on video depth. Background Art

[0002] An existing invention patent (CN113992929A) discloses a virtual digital human interaction method, system, device and computer program product. The basic principle is to assign an associated digital human video ID to the preset answer based on the robot intelligent question-answering engine so that when the user asks a question, the digital human video can be played while displaying the answer to the question, thereby realizing the interaction of the digital human.

[0003] This invention only supports interaction with virtual digital humans in the form of two-dimensional video. Although it can have a good display effect on a flat electronic display device without moving the viewing angle, since the content is only a two-dimensional video, parallax cannot be generated through displacement. Regardless of whether the virtual digital human is produced by 3D modeling or driven by a real person, this method presents a two-dimensional interaction in 2D form and cannot be presented in 3D form in 3D display devices such as VR headsets.

[0004] Although the existing virtual digital people have exquisite images, the objects that end users can interact with are only immovable perspectives. The virtual digital people are fixed on a plane and cannot fully reflect the three-dimensional sense of the virtual digital people. Moreover, they can only be displayed on a plane in a 3D display device and cannot interact in 3D space. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention ensures low latency and low computing resource consumption through preset answers and pre-made videos, while at the same time, improves the presentation effect of the virtual digital human based on pre-made videos from being limited to a two-dimensional space, thereby solving the problem that 2D content cannot provide 3D interaction.

[0006] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0007] A 3D intelligent virtual image interaction method based on video depth, characterized by comprising:

[0008] S1. Collect question and answer content and build an intelligent question and answer library, which can be learned based on a large language model;

[0009] S2. Obtaining depth information of the virtual human video, wherein the depth information is a depth map of each frame corresponding to the virtual human video;

[0010] S3. Binding and storing the resource ID with the question and answer content and the depth information of the virtual human video corresponding to the question and answer content;

[0011] S4. Obtain the virtual human video and depth information URL according to the resource ID, obtain the resource according to the URL, and output the answer content and the virtual human video containing the depth information.

[0012] Furthermore, before obtaining the depth information of the virtual human video, the method further includes:

[0013] Create a virtual human video corresponding to each question and answer content.

[0014] Furthermore, the step of producing a virtual human video corresponding to each question and answer content includes:

[0015] Generate virtual human video through facial expressions and body movements; or

[0016] Artificially create virtual human videos.

[0017] Furthermore, the obtaining of the depth information of the virtual human video includes:

[0018] When the virtual human video is obtained by 3D engine rendering, the depth information of the virtual human video is calculated by the 3D engine;

[0019] When the virtual human video is not obtained by 3D engine rendering, the depth information of the virtual human video is obtained by a deep learning algorithm.

[0020] Furthermore, the obtaining of the depth information of the virtual human video by a deep learning algorithm includes:

[0021] Collecting video data and label data, wherein the label data includes depth information of each pixel corresponding to the video data;

[0022] Preprocessing the video data and the label data to obtain a preprocessed data set;

[0023] Using the preprocessed data set to train the model, the model parameters are optimized by minimizing the loss function between the predicted depth and the true depth information;

[0024] Use the trained model to estimate the depth information of the input virtual human video and obtain the corresponding label data.

[0025] Furthermore, the using the preprocessed data set to train the model includes:

[0026] Introduce the time dimension into the model and add a spatial attention mechanism.

[0027] The present invention also relates to a 3D intelligent virtual image interaction system based on video depth, characterized in that it comprises:

[0028] An intelligent question-answering module is used to collect question-answering content and build an intelligent question-answering library, which can be learned based on a large language model;

[0029] A depth information acquisition module, used to acquire depth information of a virtual human video, wherein the depth information is a depth map of each frame corresponding to the virtual human video;

[0030] An information binding module, used to bind and store the resource ID with the question and answer content and the depth information of the virtual human video corresponding to the question and answer content;

[0031] The resource output module is used to obtain the virtual human video and the depth information URL according to the resource ID, obtain the resource according to the URL, and output the answer content and the virtual human video containing the depth information.

[0032] The present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0033] The present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0034] The memory is used to store program instructions;

[0035] The processor is used to execute the above method by calling program instructions.

[0036] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.

[0037] The beneficial effects of the present invention are:

[0038] The 3D intelligent virtual image interaction method and system based on video depth described in the present invention improves the presentation effect of the virtual digital human based on pre-made videos, which is limited to the plane, to the 3D space, making the interaction between the user and the virtual digital human more vivid and natural, and easier to generate emotional connection. While achieving a more vivid presentation effect, it does not affect the real-time and efficiency of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0040] Figure 1 A schematic diagram of a 3D intelligent virtual image interaction method based on video depth according to the present invention;

[0041] Figure 2A schematic diagram of the structure of a 3D intelligent virtual image interaction system based on video depth according to the present invention;

[0042] Figure 3 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. It should be understood that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments, and these embodiments are only used to illustrate the present application and are not used to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0044] like Figure 1 As shown, the embodiment of the present invention discloses a 3D intelligent virtual image interaction method based on video depth, and the steps of the process include:

[0045] Step S1. Collect question and answer content and build an intelligent question and answer database, which can be learned based on a large language model.

[0046] In an embodiment of the present invention, question and answer content can be collected, sorted, all intelligent question and answer content can be obtained, and an intelligent question and answer library can be constructed to provide text information for the next step.

[0047] Preferably, learning can be performed based on a large language model to expand the intelligent question-answering library and improve the question-answering capabilities of the intelligent virtual image in various fields. This can be achieved specifically through the following steps:

[0048] (1) Collect multi-source data and pre-process the data. Multi-source data can be obtained from the external Internet or internal knowledge base;

[0049] (2) Use professional domain data to fine-tune the large language model to make it perform better in finance or specific fields. The fine-tuning techniques that can be used include Fine-tuning;

[0050] (3) Evaluate and tune the fine-tuned model. The model can be tuned by adjusting model parameters;

[0051] (4) Use reinforcement learning with human feedback (RLHF) to update the intelligent question-answering database based on user feedback and improve the quality of question-answering.

[0052] Step S2: Obtain the depth information of the virtual human video, the depth information is the depth map of each frame corresponding to the virtual human video. The depth map is a grayscale image that encodes the distance between each pixel of the 2D image and the observation point, and the distance is expressed in grayscale.

[0053] In the embodiment of the present invention, before step S2 of acquiring the depth information of the virtual human video, the method further includes:

[0054] Create a virtual human video corresponding to each question and answer content.

[0055] Specifically, a virtual human video corresponding to each question and answer content is produced, including:

[0056] Generate virtual human video through facial expressions and body movements; or

[0057] Artificially create virtual human videos.

[0058] In an embodiment of the present invention, a method for obtaining depth information of a virtual human video includes:

[0059] When the virtual human video is obtained by 3D engine rendering, the depth information of the virtual human video is calculated by the 3D engine. For a video rendered by a 3D engine, the depth can be calculated by the parallax information between the left and right cameras. When rendering a video, the depth information of the video can be saved in a depth channel, so the information of the depth channel can be directly extracted as the depth information of the video.

[0060] When the virtual human video is not obtained by 3D engine rendering, the depth information of the virtual human video is obtained by a deep learning algorithm.

[0061] Preferably, the depth information of the virtual human video is obtained by a deep learning algorithm, including:

[0062] Collecting video data and label data, wherein the label data includes depth information of each pixel corresponding to the video data;

[0063] Preprocessing the video data and the label data to obtain a preprocessed data set;

[0064] Using the preprocessed data set to train the model, the model parameters are optimized by minimizing the loss function between the predicted depth and the true depth information;

[0065] Use the trained model to estimate the depth information of the input virtual human video and obtain the corresponding label data.

[0066] In the embodiment of the present invention, the video data may be acquired through a stereo camera, a laser radar or other equipment, so that the depth information calculated thereby is more accurate, and the data is more real and usable.

[0067] In an embodiment of the present invention, a convolutional neural network (CNN) may be selected for training.

[0068] When estimating the depth information of a video, the video can be converted into a continuous sequence of images. In the virtual human video of the embodiment of the present invention, when the virtual human moves, the depth information between two adjacent frames of its limb movements has a certain correlation (this correlation will be affected by the frame rate, and when the frame rate is high, the correlation may be higher), while when the virtual human moves, the depth information between two adjacent frames of its face may not change much. At the same time, the relative depth information of the virtual human's limb sites in the same frame may change greatly, while the relative depth information of the virtual human's facial sites in the same frame does not change much.

[0069] Preferably, for limb areas with action correlation, the time dimension can be introduced into the convolutional neural network (CNN) to better capture and process areas with large changes in depth information. For facial areas, a spatial attention mechanism can be introduced to select and focus on facial depth information in video frames. In an embodiment of the present invention, the priority of facial area processing can be appropriately reduced, resources can be reasonably allocated, and processing efficiency can be improved.

[0070] Preferably, the model parameters can be adjusted by minimizing the loss function to improve the model prediction accuracy. Common loss functions include: mean square error, which can be used for regression problems to calculate the average of the squares of the difference between the predicted value and the true value; cross entropy loss, which can be used for classification problems to evaluate the difference between the actual output and the expected output; absolute error, which can be used for regression problems to calculate the average of the absolute values ​​of the difference between the predicted value and the true value. In an embodiment of the present invention, the model parameters can be optimized by minimizing the mean square error between the predicted depth and the true depth information.

[0071] Step S3: Bind and store the resource ID, the question and answer content, and the depth information of the virtual human video corresponding to the question and answer content.

[0072] In the embodiment of the present invention, there is a unique corresponding relationship between the resource ID, the question and answer content, and the depth information of the virtual human video corresponding to the question and answer content.

[0073] Step S4. Obtain the virtual human video and depth information URL according to the resource ID, obtain the resource according to the URL, and output the answer content and the virtual human video containing the depth information.

[0074] In the embodiment of the present invention, during the interaction with the virtual human, not only the answer content is output, but also the virtual human video containing depth information is output according to the resource ID.

[0075] In the embodiment of the present invention, both the resource ID and the URL are stored in a database for easy management.

[0076] like Figure 2 As shown, the embodiment of the present invention discloses a 3D intelligent virtual image interaction system based on video depth, and its structure includes:

[0077] An intelligent question-answering module is used to collect question-answering content and build an intelligent question-answering library, which can be learned based on a large language model;

[0078] A depth information acquisition module, used to acquire depth information of a virtual human video, wherein the depth information is a depth map of each frame corresponding to the virtual human video;

[0079] An information binding module, used to bind and store the resource ID with the question and answer content and the depth information of the virtual human video corresponding to the question and answer content;

[0080] The resource output module is used to obtain the virtual human video and the depth information URL according to the resource ID, obtain the resource according to the URL, and output the answer content and the virtual human video containing the depth information.

[0081] In an embodiment of the present invention, the input request of the intelligent question-answering module is the question text, and the output response is the answer text and resource ID. The present invention is compatible with any form of text question-answering robot, and the principle of the text question-answering robot system is not described in detail here. The present invention needs to form a text question-answering robot based on the collected question-answering content. In the preparation stage, the present invention produces a digital human video and video depth information for each section of the answer content, and each section of the answer content has a corresponding virtual human resource. In the intelligent question-answering module, the main function is consistent with the text question-answering robot, the difference is that the corresponding virtual digital human resource ID is returned based on the reply text to achieve 3D presentation.

[0082] By using this system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.

[0083] In an embodiment of the present invention, an interaction method is provided based on the system architecture, and the system can run on the client. The text or voice input function realizes the input when the user interacts with the virtual digital human. Ask a question text request to the intelligent question and answer module and obtain the answer text and resource ID. The resource output module is called on the client to query the resource address corresponding to the resource ID, and the video and depth information resources are downloaded according to the resource address. On the basis of the plane video, the depth information is reflected to the video through geometric displacement deformation on the client side, and the video is converted into 3D stereo and presented. When the user is in a non-input state, the presented virtual digital human is in an idle state, and the client plays the idle state digital human video and depth information.

[0084] Taking the financial service scenario as an example, in real life, account managers need to answer customer questions and provide help to customers at all times. In order to solve the problem of insufficient staff of account managers, intelligent question-and-answer robots and customer questions and answers can be introduced in some online scenarios. Although the intelligent question-and-answer robot has a lot of question-and-answer knowledge, it is easy to give online customers a sense of mechanicalness and unnaturalness. In such a scenario, a virtual digital human image can be selected as the image of the interactive object instead of the traditional intelligent question-and-answer robot. In order to ensure the image presentation effect of the virtual digital human in the online operating environment of most customers (such as PC, mobile terminal), the virtual human interaction will be presented in the form of a video stream. By using the present invention, even if the video itself is 2D content, it can be changed to 3D when presented and provide more possibilities for interaction, such as interaction with the mouse.

[0085] Additionally, an embodiment of the present invention further provides an interaction method, including:

[0086] The user asks a question to the virtual person, which is then input into the intelligent question-answering module for processing and then the answer text and resource ID are returned;

[0087] Find the corresponding resource address according to the resource ID, and download the video and depth information according to the resource address;

[0088] Based on the two-dimensional video, the depth information is reflected to the video through geometric displacement and deformation, turning the video into a 3D stereoscopic presentation.

[0089] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0090] The embodiment of the present invention further provides an electronic device for executing the above method, as a device for implementing the method, and its structure is as follows: Figure 3As shown, the electronic device 300 includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one. It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the specific structure of the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may also include a display screen 305 for displaying images, or receiving user operation instructions when necessary.

[0091] The processor 301 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 304 may include a receiver and a transmitter. The transceiver 304 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0092] Processor 301 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0093] The processor 301 may also include one or more processing units, for example, the processor 301 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 300. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 301 for storing instructions and data. In some embodiments, the memory in the processor 301 is a high-speed cache memory. The memory may store instructions or data that the processor 301 has just used or circulated. If the processor 301 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 301 is reduced, and the efficiency of the system is improved.

[0094] The processor 301 can run the method provided in the embodiment of the present application. The processor 301 can include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute the method provided in the embodiment of the present application, for example, part of the algorithm in the method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain faster processing efficiency.

[0095] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or may include a high-speed random access memory, or may include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0097] Optionally, the memory 303 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the method provided in any embodiment of the present application.

[0098] The memory 303 can be used to store computer executable program codes, which include instructions. The processor 301 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the memory 303. The memory 303 may include a program storage area and a data storage area. Among them, the program storage area may store the code of the operating system, application program, etc. The data storage area may store data created during the use of the electronic device 300 (such as images, videos, etc. collected by the camera application), etc.

[0099] The memory 303 may also store one or more computer programs corresponding to the method provided in the embodiment of the present application. The one or more computer programs are stored in the above-mentioned memory 303 and are configured to be executed by the one or more processors 301. The one or more computer programs include instructions, and the above-mentioned instructions can be used to execute each step in the above-mentioned corresponding embodiment.

[0100] Of course, the code of the method provided in the embodiment of the present application can also be stored in an external memory. In this case, the processor 301 can run the code of the method stored in the external memory through the external memory interface, and the processor 301 can control the running process.

[0101] The display screen 305 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 305, where N is a positive integer greater than 1. The display screen 305 can be used to display information input by a user or information provided to a user and various graphical user interfaces (GUI). For example, the display screen 305 can display photos, videos, web pages, or files.

[0102] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0103] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0105] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0107] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A 3D intelligent virtual image interaction method based on video depth, characterized in that: include: S1. Collect question and answer content and build an intelligent question and answer library, which can be learned based on a large language model; S2. Obtaining depth information of the virtual human video, wherein the depth information is a depth map of each frame corresponding to the virtual human video; S3. Binding and storing the resource ID with the question and answer content and the depth information of the virtual human video corresponding to the question and answer content; S4. Obtain the virtual human video and depth information URL according to the resource ID, obtain the resource according to the URL, and output the answer content and the virtual human video containing the depth information.

2. The method according to claim 1, characterized in that Before obtaining the depth information of the virtual human video, the method further includes: Create a virtual human video corresponding to each question and answer content.

3. The method according to claim 2, characterized in that The step of producing a virtual human video corresponding to each question and answer content includes: Generate virtual human video through facial expressions and body movements; or Artificially create virtual human videos.

4. The method according to claim 1, characterized in that The obtaining of depth information of the virtual human video includes: When the virtual human video is obtained by 3D engine rendering, the depth information of the virtual human video is calculated by the 3D engine; When the virtual human video is not obtained by 3D engine rendering, the depth information of the virtual human video is obtained by a deep learning algorithm.

5. The method according to claim 4, characterized in that The method of obtaining the depth information of the virtual human video by using a deep learning algorithm includes: Collecting video data and label data, wherein the label data includes depth information of each pixel corresponding to the video data; Preprocessing the video data and the label data to obtain a preprocessed data set; Using the preprocessed data set to train the model, the model parameters are optimized by minimizing the loss function between the predicted depth and the true depth information; Use the trained model to estimate the depth information of the input virtual human video and obtain the corresponding label data.

6. The method according to claim 5, characterized in that The method of using the preprocessed data set to train a model includes: Introduce the time dimension into the model and add a spatial attention mechanism.

7. A 3D intelligent virtual image interaction system based on video depth, characterized in that: include: An intelligent question-answering module is used to collect question-answering content and build an intelligent question-answering library, which can be learned based on a large language model; A depth information acquisition module, used to acquire depth information of a virtual human video, wherein the depth information is a depth map of each frame corresponding to the virtual human video; An information binding module, used to bind and store the resource ID with the question and answer content and the depth information of the virtual human video corresponding to the question and answer content; The resource output module is used to obtain the virtual human video and the depth information URL according to the resource ID, obtain the resource according to the URL, and output the answer content and the virtual human video containing the depth information.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store program instructions; The processor is configured to execute the method according to any one of claims 1 to 6 by calling program instructions.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.