Live interactive service method and system based on virtual image

Abstract

The present invention provides a live interactive service method based on a virtual image, pre-processing interactive image data; the interactive image data includes a foreground image, a background image and audio; obtaining first interactive data and second interactive data; the first interactive data includes real-person audio, video and text data obtained by the server; the second interactive data includes text, audio and video interactive request data initiated by the requesting end; generating a virtual image based on the first interactive data; generating interactive graphics and interactive text based on the second interactive data; word segmentation processing of the interactive text, synthesizing interactive audio according to the phonemes corresponding to the interactive text and importing it into the audio data; generating an index data set based on interactive graphics, interactive audio and interactive text; calling and generating index actions, interactive graphics, interactive text and interactive audio based on the virtual image from the index data set and feeding them back to the foreground image. The present invention effectively and quickly improves the processing capabilities of audio and video products.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a live interactive service method based on a virtual image. Background Art

[0002] The production of existing interactive programs mainly depends on the broadcasting level of real anchors. It is difficult to avoid the production and broadcast of program errors or low-quality video programs due to factors such as the ups and downs of the physiological or psychological conditions of real anchors. It is difficult to achieve the production level of high-quality interactive information service products with "0" verbal errors. The production and processing capabilities of existing programs are completely dependent on manual production. In today's era of rapid information dissemination, it is impossible to effectively and quickly improve the processing capabilities of interactive audio and video products.

[0003] In view of this, there is an urgent need to propose a live interactive service method based on virtual images. Summary of the invention

[0004] To this end, the present invention provides a live interactive service method based on a virtual image, which effectively and quickly improves the processing capabilities of interactive audio and video products.

[0005] The above object of the present invention is achieved through the following solutions:

[0006] Preprocessing interactive image data; the interactive image data includes a foreground image, a background image and audio;

[0007] Acquire first interaction data and second interaction data; the first interaction data includes real-person audio, video and text data acquired by the server; the second interaction data includes text, audio and video interaction request data initiated by the requesting end;

[0008] generating a virtual image based on the first interaction data;

[0009] generating an interactive graphic and an interactive text based on the second interactive data;

[0010] Process the interactive text through word segmentation, synthesize the interactive audio according to the phonemes corresponding to the interactive text and import it into the audio data; generate an index data set based on interactive graphics, interactive audio and interactive text;

[0011] According to the second interactive data, an index action based on the virtual image, an interactive graphic, an interactive text and an interactive audio are called from the index data set to be fed back to the foreground image.

[0012] Further, the feedback includes first feedback and second feedback;

[0013] The first feedback specifically includes calling the interactive image, interactive audio and index action matching the second interactive data from the index data set and replacing them in the foreground image;

[0014] If the first feedback is not executed within the preset time, the second feedback is executed;

[0015] The server generates real-time interaction with real people based on the interactive text that has been processed by word segmentation, and then updates the virtual image in real time and directly feeds it back to the foreground image.

[0016] Furthermore, the interactive data also includes third interactive data, and the third interactive data is pre-recorded real-person action, video and audio;

[0017] and the pre-recording time of the third interactive data is earlier than the acquisition time of the first interactive data;

[0018] The third interaction data is used to generate a pre-index data set;

[0019] The interactive graphics, interactive audio and interactive text in the first interactive data and the second interactive data are added to the pre-indexed data set to obtain an indexed data set.

[0020] Further, when obtaining the first interaction data and the third interaction data, a voice data set based on the audio data is established for synthesizing the interaction audio;

[0021] The process of establishing the speech data set is specifically as follows:

[0022] Collect audio data from multiple different broadcast contents and extract interactive vocabulary;

[0023] Generate phoneme units and voiceprint waveforms corresponding to interactive vocabulary;

[0024] An index corresponding to interactive vocabulary, phoneme units, and voiceprint waveforms is established to obtain a speech dataset.

[0025] Furthermore, when creating a virtual image, images of the real person are collected from multiple camera angles at the same time, and then a virtual three-dimensional model of the real person is created.

[0026] The present invention also provides a live interactive service system based on a virtual image, comprising:

[0027] A first acquisition module, acquiring first interaction data from a server;

[0028] A second collection module obtains second interaction data from the server;

[0029] A synthesis module receives the second interactive data and generates an interactive text according to a pre-stored interactive vocabulary library;

[0030] A material generation module stores multiple sets of materials including image data and audio data, generates interactive audio after receiving interactive text, and calls foreground images and background images;

[0031] An interactive image synthesis module receives interactive text and generates interactive graphics;

[0032] The fusion module aligns the foreground image, background image and audio track on the time axis, and obtains the virtual anchor interaction data after fusion.

[0033] Furthermore, it also includes a storage module, which includes a local storage unit and a cloud storage unit;

[0034] Receive the virtual anchor interaction data from the fusion module, and perform local storage, cloud storage, backup and call of the virtual anchor interaction data as needed.

[0035] An electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a live interactive service method based on a virtual image is implemented.

[0036] A computer-readable storage medium stores a computer program, wherein the program, when executed by a processor, implements a live interactive service method based on a virtual image.

[0037] The above technical solution of the present invention has the following advantages compared with the prior art:

[0038] (1) It can achieve a realistic and lifelike interactive program anchor broadcasting effect, and avoid program errors or low-quality video programs caused by human factors such as slips of the tongue and poor broadcasting status.

[0039] (2) Improve broadcast efficiency. Without the preparation phase before the live broadcast, a lot of daily program recording time can be saved; the response to live broadcast programs such as emergency warnings and on-site reports is more timely.

[0040] (3) Improve the efficiency of video content production. It can produce a large number of video programs in large quantities, for a long time, and continuously, greatly improving the processing capabilities of audio and video products. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a method provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of module connection of the device provided by an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of an electronic device provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0044] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and systems can be simplified for display.

[0045] The following description and accompanying drawings fully illustrate specific embodiments of the present invention so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The examples represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. The scope of the embodiments of the present invention includes the entire scope of the claims, and all available equivalents of the claims. In this article, each embodiment may be represented individually or generally by the term "invention", which is only for convenience, and if more than one invention is disclosed in fact, it is not intended to automatically limit the scope of the application to any single invention or inventive concept. In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method or electronic device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or electronic device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method or electronic device including the elements. The various embodiments herein are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the methods, products, etc. disclosed in the embodiments, since they correspond to the method part disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0046] This embodiment provides a live interactive service method based on a virtual image. Figure 1 As shown, the following steps are included:

[0047] Preprocessing interactive image data; the interactive image data includes a foreground image, a background image and audio;

[0048] Acquire first interaction data and second interaction data; the first interaction data includes real-person audio, video and text data acquired by the server; the second interaction data includes text, audio and video interaction request data initiated by the requesting end;

[0049] generating a virtual image based on the first interaction data;

[0050] generating an interactive graphic and an interactive text based on the second interactive data;

[0051] Process the interactive text through word segmentation, synthesize the interactive audio according to the phonemes corresponding to the interactive text and import it into the audio data; generate an index data set based on interactive graphics, interactive audio and interactive text;

[0052] According to the second interactive data, an index action based on the virtual image, an interactive graphic, an interactive text and an interactive audio are called from the index data set to be fed back to the foreground image.

[0053] As a preferred method of this embodiment, in this embodiment, the server end is the real anchor and service party providing the service, the first interactive data is the real-time data collected from the real person, the requesting end is the audience requesting live interaction, and the second interactive data is the text, barrage, voice and graphic feedback initiated by the audience.

[0054] Further, the feedback includes first feedback and second feedback;

[0055] The first feedback specifically includes calling the interactive image, interactive audio and index action matching the second interactive data from the index data set and replacing them in the foreground image;

[0056] If the first feedback is not executed within the preset time, the second feedback is executed;

[0057] The server generates real-time interaction with real people based on the interactive text that has been processed by word segmentation, and then updates the virtual image in real time and directly feeds it back to the foreground image.

[0058] Furthermore, the interactive data also includes third interactive data, and the third interactive data is pre-recorded real-person action, video and audio;

[0059] and the pre-recording time of the third interactive data is earlier than the acquisition time of the first interactive data;

[0060] The third interaction data is used to generate a pre-index data set;

[0061] The interactive graphics, interactive audio and interactive text in the first interactive data and the second interactive data are added to the pre-indexed data set to obtain an indexed data set.

[0062] Each time a live broadcast is conducted, after collecting the first interactive data of the real anchor, an index data set is generated based on the first interactive data and the second interactive data. The purpose of the index data set is to index the corresponding index actions of the virtual image and related graphics and audio data based on the preset segmentation vocabulary that has been extracted, thereby reducing the workload of the real anchor.

[0063] Since there is no index relationship when the live broadcast is performed for the first time, the third interactive data is recorded in advance and the index set is preset, and then the new index relationship is added to the index data set after each live broadcast.

[0064] Further, when obtaining the first interaction data and the third interaction data, a voice data set based on the audio data is established for synthesizing the interaction audio;

[0065] The process of establishing the speech data set is specifically as follows:

[0066] Collect audio data from multiple different broadcast contents and extract interactive vocabulary;

[0067] Generate phoneme units and voiceprint waveforms corresponding to interactive vocabulary;

[0068] An index corresponding to interactive vocabulary, phoneme units, and voiceprint waveforms is established to obtain a speech dataset.

[0069] Furthermore, when creating a virtual image, images of the real person are collected from multiple camera angles at the same time, and then a virtual three-dimensional model of the real person is created.

[0070] This embodiment also provides a live interactive service system based on a virtual image. Figure 2 As shown, including:

[0071] A first acquisition module, acquiring first interaction data from a server;

[0072] A second collection module obtains second interaction data from the server;

[0073] A synthesis module receives the second interactive data and generates an interactive text according to a pre-stored interactive vocabulary library;

[0074] A material generation module stores multiple sets of materials including image data and audio data, generates interactive audio after receiving interactive text, and calls foreground images and background images;

[0075] An interactive image synthesis module receives interactive text and generates interactive graphics;

[0076] The fusion module aligns the foreground image, background image and audio track on the time axis, and obtains the virtual anchor interaction data after fusion.

[0077] Furthermore, it also includes a storage module, which includes a local storage unit and a cloud storage unit;

[0078] Receive the virtual anchor interaction data from the fusion module, and perform local storage, cloud storage, backup and call of the virtual anchor interaction data as needed.

[0079] Combination Figure 3 As shown, the embodiment of the present disclosure provides an electronic device for a live interactive service based on a virtual image, including a processor (processor) 30 and a memory (memory) 31. Optionally, the electronic device may also include a communication interface (CommunicationInterface) 32 and a bus 33. Among them, the processor 30, the communication interface 32, and the memory 31 can communicate with each other through the bus 33. The communication interface 32 can be used for information transmission. The processor 30 can call the logic instructions in the memory 31 to execute the live interactive service method based on the virtual image of the above embodiment.

[0080] The embodiment of the present disclosure provides a storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above-mentioned live interactive service method based on a virtual image.

[0081] The above-mentioned storage medium may be a transient computer-readable storage medium or a non-transient computer-readable storage medium. Non-transient storage media include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, and may also be a transient storage medium.

[0082] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or electronic device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0083] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, systems and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0084] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A live interactive service method based on a virtual image, characterized in that: The steps include: Preprocessing interactive image data; the interactive image data includes a foreground image, a background image and audio; Acquire first interaction data and second interaction data; the first interaction data includes real-person audio, video and text data acquired by the server; The second interactive data includes text, audio and video interactive request data initiated by the requesting end; generating a virtual image based on the first interaction data; generating an interactive graphic and an interactive text based on the second interactive data; Process the interactive text by word segmentation, synthesize the interactive audio according to the phonemes corresponding to the interactive text and import it into the audio data; Generate index datasets based on interactive graphics, interactive audio, and interactive text; Invoke the index action, interactive graphics, interactive text and interactive audio based on the virtual image from the index data set according to the second interactive data and feed them back to the foreground image; the feedback includes the first feedback and the second feedback; The first feedback specifically includes calling the interactive image, interactive audio and index action matching the second interactive data from the index data set and replacing them in the foreground image; If the first feedback is not executed within the preset time, the second feedback is executed; The server generates a real-person interaction based on the segmented interactive text, and then updates the virtual image in real time and directly feeds it back to the foreground image; The interactive data also includes third interactive data, which is pre-recorded real-person action, video and audio; and the pre-recording time of the third interactive data is earlier than the acquisition time of the first interactive data; The third interaction data is used to generate a pre-index data set; The interactive graphics, interactive audio and interactive text in the first interactive data and the second interactive data are added to the pre-indexed data set to obtain an indexed data set; When acquiring the first interaction data and the third interaction data, establishing a voice data set based on the audio data for synthesizing the interaction audio; The process of establishing the speech data set is specifically as follows: Collect audio data from multiple different broadcast contents and extract interactive vocabulary; Generate phoneme units and voiceprint waveforms corresponding to interactive vocabulary; An index corresponding to interactive vocabulary, phoneme units, and voiceprint waveforms is established to obtain a speech dataset.

2. The live interactive service method based on virtual images according to claim 1, characterized in that: When creating a virtual image, images of the real person are collected from multiple camera angles at the same time, and then a virtual three-dimensional model of the real person is created.

3. A live interactive service system based on a virtual image, suitable for implementing any of the methods described in claims 1-2, characterized in that: include: A first acquisition module, acquiring first interaction data from a server; A second collection module obtains second interaction data from the server; A synthesis module receives the second interactive data and generates an interactive text according to a pre-stored interactive vocabulary library; A material generation module stores multiple sets of materials including image data and audio data, generates interactive audio after receiving interactive text, and calls foreground images and background images; An interactive image synthesis module receives interactive text and generates interactive graphics; The fusion module aligns the foreground image, background image and audio track on the time axis, and obtains the virtual anchor interaction data after fusion.

4. The live interactive service system based on virtual images according to claim 3, characterized in that: It also includes a storage module, which includes a local storage unit and a cloud storage unit; receives the virtual anchor interaction data from the fusion module, and performs local storage, cloud storage, backup and call of the virtual anchor interaction data according to needs.

5. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images