Live broadcast control method, apparatus and device, and storage medium

Abstract

The invention provides a live broadcast control method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of computers, and in particular to the fields of artificial intelligence, information streams, digital humans, smart e-commerce and the like. According to the specific implementation scheme, under the condition that live broadcasting of a real person live broadcasting room is closed, a digital person live broadcasting room is created based on information of the real person live broadcasting room; and performing live broadcast based on the digital human live broadcast room. According to the embodiment of the invention, by intelligently creating the digital human live broadcast room and rendering the digital human live broadcast room, the digital human live broadcast room can be quickly started, so that the manual configuration time required for manually switching to the digital human live broadcast room during live broadcast is shortened.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, information flow, digital human, and smart e-commerce. Background Art

[0002] With the development of AI (Artificial Intelligence), digital human live broadcast has become a potential new live broadcast method and has gradually been applied and developed in multiple industries.

[0003] Digital human live broadcast technology is an innovative way of interaction. A complete digital human live broadcast room can not only attract users' attention, but also save labor costs and improve work efficiency. Summary of the invention

[0004] The present disclosure provides a live broadcast control method, apparatus, device and storage medium.

[0005] According to one aspect of the present disclosure, a live broadcast control method is provided, comprising:

[0006] When the live broadcast room of a real person is closed, a digital human live broadcast room is created based on the information of the live broadcast room of the real person;

[0007] Live broadcast based on the digital human live broadcast room.

[0008] According to another aspect of the present disclosure, a live broadcast control device is provided, comprising:

[0009] A creation module, used to create a digital human live broadcast room based on the information of the real person live broadcast room when the live broadcast is closed;

[0010] The live broadcast module is used for live broadcast based on the digital human live broadcast room.

[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0017] In the disclosed embodiment, by intelligently creating a digital human live broadcast room and rendering it, it is possible to quickly start broadcasting the digital human live broadcast room, thereby shortening the manual configuration time required to manually switch to the digital human live broadcast room when the real person live broadcast is off, and providing a richer and more personalized live broadcast experience.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 is a flowchart of a live broadcast control method according to an embodiment of the present disclosure;

[0021] Figure 2 It is a schematic diagram of the process of creating a digital human live broadcast room according to an embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram of a process of generating image elements according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram of a process of obtaining a target template based on a diffusion model according to an embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of a diffusion model iteration process according to an embodiment of the present disclosure;

[0025] Figure 6 is a timing diagram of switching from a live broadcast room of a real person to a live broadcast room of a digital person according to an embodiment of the present disclosure;

[0026] Figure 7 is a timing diagram of switching from a digital human live broadcast room to a real person live broadcast room according to an embodiment of the present disclosure;

[0027] Figure 8 is a structural schematic diagram of a live broadcast control device according to an embodiment of the present disclosure;

[0028] Fig. 9 It is a block diagram of an electronic device used to implement the live broadcast control method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] With the widespread application of digital humans in fields including but not limited to entertainment, education, and commercial marketing, most live broadcasters will use digital humans for live broadcasts to break the limitations of region and time and provide viewers with a richer and more novel live broadcast experience. However, how to effectively use digital humans for live broadcasts and improve live broadcast efficiency still needs to be improved.

[0032] In view of this, the embodiments of the present disclosure provide a live broadcast control method. The method can intelligently and automatically create a digital human live broadcast room, render the digital human live broadcast room, and quickly start the digital human live broadcast room, so as to shorten the manual configuration time required to manually switch to the digital human live broadcast room when the real person live broadcast is off.

[0033] like Figure 1 FIG. 1 is a flow chart of a live broadcast control method proposed in an embodiment of the present disclosure, including:

[0034] S101, when the live broadcast room of the real person is closed, a digital human live broadcast room is created based on the information of the live broadcast room of the real person.

[0035] That is, the suspension of real-person live broadcast is a trigger signal, which is used to trigger the creation of a digital human live broadcast room to achieve seamless switching between real-person live broadcast and digital human live broadcast.

[0036] During implementation, a virtual live broadcast room can be created based on digital human technology to continue the needs and / or style of real-life live broadcasts, so that live broadcasts can continue without human supervision. Digital human live broadcast is an emerging form of live broadcast, which uses virtual characters (digital humans) generated by artificial intelligence to conduct live broadcast activities. Digital humans can be completely fictional characters or images created based on real-life models.

[0037] S102, live broadcast based on the digital human live broadcast room.

[0038] As the name suggests, live broadcasting based on digital human live broadcasting rooms refers to live broadcasting activities using highly anthropomorphic and interactive virtual images created by artificial intelligence technology. When using digital human live broadcasting rooms for live broadcasting, you can refer to the live broadcast content and live broadcast format of real-person live broadcast rooms for live broadcasting.

[0039] In the disclosed embodiment, when the live broadcast room of a real person is closed, the automatic creation of a digital human live broadcast room is triggered and the live broadcast is carried out. The live broadcast demand can be responded to immediately, without waiting for the preparation time of the real anchor, and without the need for the real anchor to perform cumbersome configuration of the digital human live broadcast room. While saving the relevant costs of the real anchor, the startup speed and overall efficiency of the live broadcast can be greatly improved. In addition, the digital human is not restricted by time, place and physiological conditions, and can meet the viewing needs of different audience groups, increasing the flexibility and coverage of the live broadcast. Live broadcast based on the digital human live broadcast room can be quickly adjusted and configured according to the different live broadcast content and scenes of the real live broadcast room, meeting the diversified needs of the audience and providing a richer and more personalized live broadcast experience.

[0040] Setting up a digital human live broadcast room is a complex and meticulous process, involving configuration and customization of multiple aspects. For example, designing a unique image for the digital human, writing professional scripts for the digital human, selecting a suitable voice for the digital human, and decorating the digital human live broadcast room. In order to maximize the efficiency of creating a digital human live broadcast room, in the disclosed embodiment, the second live broadcast element of the digital human live broadcast room can be generated in advance during the live broadcast of the real anchor, based on the first live broadcast element of the real live broadcast room that is more important or takes a long time to generate. When the live broadcast of the real live broadcast room is closed, such as Figure 2 As shown, the creation of a digital human live studio can be accelerated based on the following steps:

[0041] S201, obtaining a second live broadcast element generated in advance based on a first live broadcast element in a live broadcast room of a real person.

[0042] In the live broadcast process, there are many first live broadcast elements, which are the basis for building live broadcast content. The first live broadcast elements mainly include but are not limited to:

[0043] Live broadcast content, such as the text explained by the anchor, the products on display, the performances performed, etc.;

[0044] Activity information, such as promotions, etc.

[0045] Host image: such as the host's appearance, voice, live broadcast habits, etc.

[0046] Interactive elements: such as the way and content of interaction with the audience.

[0047] Live broadcasts of real people can be collected and transmitted by live broadcast platforms, recorded and saved in real time, so as to parse the first live broadcast elements from the screen recording, and provide basic data for the subsequent creation of digital human live broadcast rooms.

[0048] The information of the product may be dynamically updated, such as optimizing activities, putting products on and taking products off the shelves. In the embodiment of the present disclosure, an access interface for the product information may be provided so that the product information can be accessed in real time.

[0049] During implementation, in order to improve the generation of a digital human live broadcast room, the first live broadcast element obtained may be analyzed and processed to generate a second live broadcast element. The second live broadcast element includes at least one of the following:

[0050] (1) Image elements in the digital human live studio;

[0051] Image elements are visual elements generated for the digital human live broadcast room based on the first live broadcast element of the real person live broadcast room. For example, obtain the product information of the historical real person live broadcast room, and generate the digital human live broadcast room widget according to the discount, promotion and event information. Generate the digital human live broadcast room patch according to the product name and price. Generate the digital human live broadcast room background image according to the product details page.

[0052] Among them, the digital human live broadcast room widget refers to the functional icons on the live broadcast room interface. In the digital human live broadcast room, the widget may include high-definition pictures of the goods, preferential information of the goods, promotional information of the goods, etc., so as to construct a richer live broadcast scene.

[0053] Digital Human Live Studio stickers refer to images or text information superimposed on the video screen of the live studio, which are used to display brand information, such as product name, product price, etc. In the Digital Human Live Studio, stickers can be dynamic and updated as the live content changes, such as real-time adjustment of the price of the product being promoted, the countdown of the event, etc., to improve marketing effects, attract the attention of the audience, and increase interaction.

[0054] The background image of the digital human live broadcast room refers to the background picture of the digital human live broadcast room, which sets the scene and atmosphere for the live broadcast content. In the digital human live broadcast room, the background image can be very rich and varied, from static images to dynamic videos, and even interactive virtual environments. A good background image can not only attract the audience's vision, but also enhance the emotional expression and information transmission of the live broadcast content.

[0055] In the disclosed embodiment, by using image elements such as digital human live broadcast room widgets, digital human live broadcast patches, and digital human live broadcast background images, the visual effects of the digital human live broadcast room can be greatly enriched, attracting the audience's attention and making the live broadcast content more vivid and interesting.

[0056] (2) Digital Human;

[0057] Digital human, i.e. virtual anchor in the digital human live studio, is a virtual human with human form and behavior simulated by artificial intelligence. During implementation, video clips of the real live studio can be intercepted, and the digital human image of the digital human live studio can be generated based on the anchor's face by detecting clear faces in the video clips. If no clear face can be detected in the video clip, a high-quality face is selected from the public image library based on the anchor's gender as the digital human image.

[0058] In addition, popular live broadcasters can be screened based on the traffic, sales, etc. of historical live broadcasts to build digital human broadcasters.

[0059] When the live broadcast room is shut down and a seamless switch to the digital human live broadcast is required, in order to continue the live broadcast experience, the image, movements, and expressions of the digital human live broadcast can be created as much as possible based on the last live broadcaster before the live broadcast room is shut down to imitate the real broadcaster.

[0060] (3) Digital human voice;

[0061] The voice of the digital human is generated through sound synthesis technology, which can simulate sounds of different timbres and emotions, making the expression of the digital human richer and more realistic. During implementation, the voice clips of the anchor in the live broadcast room can be intercepted, and the corresponding digital human voice can be generated based on the anchor's voice information.

[0062] (4) Digital human live broadcast script;

[0063] The script of the digital human live broadcast room is the script or dialogue content that the digital human follows during the live broadcast process. It determines the theme, content and process of the digital human live broadcast and is the basis for ensuring the smooth progress of the live broadcast. During implementation, the product information of the real person live broadcast room and the host's oral information can be obtained. Based on the ASR (Automatic Speech Recognition) technology, the host's oral information is converted into text, combined with the detailed information of the product, and multiple sets of scripts that conform to the host's style are generated through the large model.

[0064] (5) Question and answer information for the Digital Human live broadcast room.

[0065] That is, the questions and answers used for interaction and warm-up before or during the live broadcast. It can help digital humans interact with the audience, increase the live broadcast room's activity and the audience's sense of participation. For example, based on the product information of the real person's live broadcast room and the question and answer information of the historical live broadcast room users, the big model is used to generate the question and answer pairs of the digital human's live broadcast room for the warm-up question and answer of the digital human's live broadcast room.

[0066] In the disclosed embodiments, the image elements of the digital human live broadcast room can create different live broadcast atmospheres, such as festival atmosphere, event atmosphere, etc., making it easier for the audience to immerse themselves in the live broadcast scene and improve the interactivity and participation of the live broadcast. As the core of the digital human live broadcast room, the digital human can interact with the audience, display products, explain product features and answer audience questions. Generating the digital human voice through artificial intelligence based on the voice of the real anchor helps to enhance the audience's viewing experience and make information transmission more effective. The digital human live broadcast room script can ensure that the live broadcast content is rich and smooth, while maximizing the interest of the audience. The warm-up question and answer information in the digital human live broadcast room can increase the liveliness of the live broadcast room. In summary, these elements work together to enable the digital human live broadcast room to provide a novel, interactive and efficient experience, while reducing costs and improving work efficiency.

[0067] S202, creating a digital human live broadcast room based on the second live broadcast element.

[0068] Based on the generated second live broadcast element, the generated second live broadcast element is imported into the digital human live broadcast room, and various parameters of the digital human live broadcast room are configured on the live broadcast platform, such as the live broadcast room name, cover, label, etc.

[0069] During implementation, multiple live broadcast room widgets can be generated in advance, and the corresponding live broadcast room widgets can be dynamically called for display as the live broadcast progresses. Since the digital human live broadcast room widget usually contains product discount information, when the discount information in the live broadcast room widget is invalid, it is necessary to take effective response strategies in time to avoid misleading the audience. During implementation, when the second live broadcast element includes the digital human live broadcast room widget, the validity of the discount information in the digital human live broadcast room widget can be detected; when the discount information is invalid, stop outputting the digital human live broadcast room widget to the digital human live broadcast room.

[0070] That is, during the live broadcast of the digital human live broadcast room, it is necessary to check the validity of the preferential activities displayed in the widget regularly or in real time according to the product information recorded in the database. For example, during the live broadcast of the digital human, the operator can modify the product information in the database at any time according to the needs. During the live broadcast of the digital human, the products with preferential activities to be displayed can be verified based on the database to check whether the preferential activities are valid. Once it is found that a certain discount has expired or invalidated due to other reasons, it can be blocked immediately to optimize the corresponding live broadcast widget and not display it to the audience. For example, directly delete the links and data related to the widget, or set the widget to transparent when rendering to avoid the audience seeing outdated preferential information and causing live broadcast failure. In this way, the content of the digital human live broadcast room is always in line with expectations, and the audience's viewing experience is also improved. In addition, the correct information processing mechanism not only helps to build the trust of the audience, but also may have a positive impact on key performance indicators such as the sales conversion rate of the live broadcast room.

[0071] In the disclosed embodiment, the second live broadcast element of the digital human live broadcast room is generated by the first live broadcast element of the real person live broadcast room, and the artificial intelligence technology is used for simulation to ensure the consistency of the element information of the digital human live broadcast room and the real person live broadcast room. Based on the second live broadcast element, the digital human live broadcast room is created, which can improve the efficiency of live broadcast and reduce the cost of live broadcast.

[0072] In the embodiment of the present disclosure, the specific implementation steps of generating an image element based on the first live broadcast element of the live broadcast room are as follows: Figure 3 As shown, including the following:

[0073] S301, obtaining a target image and a template to be processed, wherein the template to be processed is a template in a template set corresponding to a target image class required by the target image.

[0074] The target image is an existing product image, and the template set is a set of different templates. These templates correspond to different image classes or scene requirements. The target image is the basis for subsequent processing, and its content and features will determine the selection of templates in subsequent steps. For example, if the target image is a tourist scene image, a landscape template may be selected as the template to be processed. In addition, it is also possible to determine whether it is a holiday based on the current time, and select a template that can render the corresponding holiday theme or holiday activity during the holiday.

[0075] S302 : Under the guidance of the target image, the template to be processed is adjusted based on the diffusion model to obtain a target template that is visually adapted to the target image.

[0076] The diffusion model is a generative model that is trained by gradually adding noise to the image and gradually removing noise, so that the obtained diffusion model can be used to adjust the image to generate high-quality and diverse images. Under the guidance of the target image, the diffusion model will iteratively adjust the template to be processed, so as to continuously adjust the template so that the template can better set off the target image and visually assist the target image in conveying information, and can also help the audience better understand and pay attention to the generated image elements.

[0077] S303, synthesizing the target image into the target template to obtain an image-like element.

[0078] After obtaining the target template that matches the target image, the target image is synthesized into the template. The resulting image-like element is a perfect visual combination of the target image and the target template. It not only retains the content and features of the target image, but also incorporates the style and layout of the target template.

[0079] In the disclosed embodiment, by adjusting the target template through the diffusion model, the target template can be visually made closer to the target image, thereby improving the quality and appreciation of the generated image elements. The target image is synthesized into a visually adapted template, so that the image and the background can be more naturally integrated, reducing the sense of disharmony and improving the quality of the generated digital human live studio.

[0080] In the embodiment of the present disclosure, under the guidance of the target image, based on Figure 4 The method shown uses the diffusion model to iteratively adjust the template to be processed to obtain the target template, including:

[0081] S401, superimposing the first Gaussian noise into the target image to obtain an initial guide image; and superimposing the second Gaussian noise into the template to be processed to obtain an intermediate template.

[0082] like Figure 5 As shown, by adding the first Gaussian noise to the target image, the target image has a certain randomness, and the initial guide image is obtained. Similarly, by adding the second Gaussian noise to the template to be processed, the template and the target image are at the same noise level, so as to facilitate the subsequent noise reduction process to optimize the template.

[0083] S402, merging the initial guide image into the target position of the intermediate template to obtain a first composite image.

[0084] By combining the content of the target image and the layout of the template to be processed, the target image (initial guide image) with Gaussian noise added is merged into the specified position of the intermediate template. During implementation, the merge can be based on a mask. The mask image corresponds to the template and is used to define the position of the target image in the template.

[0085] S403, using a diffusion model to gradually reduce noise on the first synthetic image to obtain a target template.

[0086] That is, the first synthetic image is iterated through the diffusion model to gradually repair the entire template while maintaining the original style and important information of the template.

[0087] During implementation, the diffusion model is used to reduce the noise of the first synthetic image step by step, and the process of obtaining the target template is as follows: Figure 5 As shown, the following steps are included:

[0088] Step A1, predicting the prediction noise corresponding to the current time step based on the diffusion model.

[0089] The time step represents the number of stages from the completely noisy state to the completely data state during the diffusion process. During the iteration of the diffusion model, the current image state is predicted for noise at each time step. Noise prediction is based on the model's understanding of the image noise pattern and the rules learned from the training data. The predicted noise can estimate the amount of noise in the current first synthesized image, providing a basis for the next step of the repair template.

[0090] During implementation, in order to ensure that the target image does not change, and to make the target template obtained by denoising keep consistent with the main content conveyed by the original template image as much as possible when fusing the target image. Therefore, during implementation, the time step adopted by the inverse process of the diffusion model is limited and can be a relatively small value. In addition, the embodiment of the present disclosure also adds the first Gaussian noise to the target image for initialization to avoid the problem of adding pure Gaussian noise at the initial time step causing unsatisfactory effects. For example, it can avoid the situation where the target image and the template to be processed do not retain any original information due to pure Gaussian noise.

[0091] Step A2: denoising the first synthesized image based on the predicted noise to obtain a second synthesized image.

[0092] Based on the predicted noise estimated in step A1, each pixel in the first synthetic image is adjusted to achieve the effect of fine-tuning the first synthetic image, so that the template part of the first synthetic image after noise reduction is more harmonious with the target image.

[0093] Step A3, segmenting the image portion of the target image from the second synthetic image to obtain a first intermediate guide image, and segmenting the template portion to be processed to update the intermediate template.

[0094] like Figure 5 As shown, the second composite image is segmented, and the part of the template to be processed is also extracted to obtain a new intermediate template. At the same time, the part of the target image is extracted to form a first intermediate guide image, so that in step A4, the third Gaussian noise is added to the first intermediate guide image to obtain a second intermediate guide image.

[0095] During implementation, the initial time step takes a larger value, and the value of the time step represents the noise level of the corresponding generated Gaussian noise. As the time step decreases, the noise level decreases, thereby gradually achieving fine-tuning of the template. Therefore, during implementation, the third Gaussian noise is related to the next time step of the current time step, and the noise level of the third Gaussian noise is less than the noise level of the third Gaussian noise of the previous time step.

[0096] Step A5, merge the second intermediate guide image into the target position of the updated intermediate template to obtain a new first synthetic image, and return to execute the step of predicting the predicted noise corresponding to the current time step based on the diffusion model until the current time step reaches the target time step, and obtain the updated intermediate template as the target template.

[0097] That is, merge the second intermediate guide image into the corresponding position in the updated intermediate template to form a new first synthetic image. And this image will be used as the basis for the next round of iteration. Repeat the above steps until the set number of time steps is reached. For example, if the initial time step T is set to 60, the template adjustment process is completed when the time step T=0 is executed. Among them, in each time step, the model will perform noise prediction, noise reduction, segmentation, noise addition and merging operations on the image. Finally, when the target time step is reached, the updated intermediate template obtained is the target template, which can ensure that the key features of the target image and the template to be processed are retained while having a lower noise level.

[0098] In summary, through the entire iterative process, the diffusion model can gradually reduce the noise in the image while maintaining the key features of the template to be processed, and finally obtain a high-quality target template.

[0099] In the disclosed embodiment, by adding Gaussian noise to the target image and the template to be processed, a certain degree of randomness can be introduced, so that a more natural and richer target template can be generated based on the diffusion model. The initial guide image is merged into the target position of the intermediate template to obtain a first synthetic image. The addition of the initial guide image can make the generated target template more consistent with the target image. Through the process of gradual noise reduction, the detailed features in the target template can be gradually optimized to obtain a high-quality target template. For example, when the color of the initial state of the template to be processed is very close to the color of the product in the target image, in order to highlight the product, the method can be used to increase the difference between the template color and the product color, so that the template and the product image are more harmonious while highlighting the theme of the product.

[0100] In the disclosed embodiment, in order to more effectively learn and generate high-quality samples, a deterministic denoising method proposed by DDIM (Denoising Diffusion Implicit Models) is adopted. By modifying the sampling strategy, the same high-quality data can be generated in fewer steps, thereby improving the generation speed.

[0101] During implementation, the DDIM diffusion model is used to gradually reduce the noise of the first synthetic image, and obtaining the target template can also be achieved based on the following steps:

[0102] Step B1, determine the step size of the time step, and determine the current time step based on the step size.

[0103] The time step size determines the speed and quality of the diffusion process. Generally speaking, the smaller the step size, the more refined the diffusion process, but the more computational effort; the larger the step size, the coarser the diffusion process, but the less computational effort. Therefore, before starting the diffusion process, a time step variable needs to be initialized, for example, in each iteration cycle, the value of the current time step is updated according to the preset step size.

[0104] For example, if the previous time step is t1 and the step size is dt, the current time step is (t1-dt).

[0105] Step B2: predicting noise information in the first synthetic image based on the current time step in the diffusion model.

[0106] At each current time step that needs to be processed, the current first synthetic image is input into the diffusion model, and the diffusion model predicts the noise information in the first synthetic image based on the current time step.

[0107] Step B3: denoise the first composite image based on the noise information to obtain a third composite image.

[0108] The first synthesized image is subjected to denoising using the predicted noise information. The denoised image is the third synthesized image, which has a visual representation more suitable for the target image than the first synthesized image.

[0109] Step B4, segmenting the image portion of the target image from the third synthetic image to obtain a third intermediate guide image, and segmenting the template portion to be processed to update the intermediate template.

[0110] The target image part and the to-be-processed template part are segmented from the third composite image, and the segmented to-be-processed template part is used to update the intermediate template to prepare for the next step of merging.

[0111] Step B5, adding a fourth Gaussian noise to the third intermediate guide image to obtain a fourth intermediate guide image; the fourth Gaussian noise is related to the next time step of the current time step, and the noise level of the third Gaussian noise is less than the noise level of the third Gaussian noise in the previous time step;

[0112] Step B6, merge the third intermediate guide image into the target position of the updated intermediate template to obtain a new first composite image, and return to the step of determining the step size of the time step until the current time step reaches the target time step, and obtain the updated intermediate template as the target template.

[0113] The third intermediate guide image with the fourth Gaussian noise added is merged into the target position of the updated intermediate template. The above steps are repeated until the current time step reaches the target time step, and the updated intermediate template is finally obtained as the target template.

[0114] In the disclosed embodiment, the DDIM diffusion model is used to generate the target template, which can generate a high-quality target template in fewer iterative steps, thereby speeding up the efficiency of generating the second live broadcast element of the digital human live broadcast room and saving processing resources.

[0115] In the disclosed embodiment, in order to ensure that the target template obtained based on the diffusion model has a better adaptation effect with the target image, the step size of the time step in the diffusion model can be dynamically adjusted based on the time step.

[0116] Dynamic adjustment of the time step size refers to dynamically adjusting the time step size according to the current iteration state or specific rules during the iteration process of the diffusion model to achieve the purpose of optimizing the diffusion process. For example, a larger step size can be used in the early stage of the diffusion process to quickly remove noise, and the step size can be reduced in the later stage to fine-tune the image details. For example, all time steps can be divided into multiple time step intervals, and different time step intervals correspond to different step sizes. The larger the value of the time step interval, the larger the corresponding step size. In this way, the repair template can be gradually and finely adapted to the target image.

[0117] In some embodiments, the new first synthetic image obtained in step B6 can also be evaluated during each iteration, such as by evaluating the aesthetics, clarity, content relevance, and adaptability between the target image and the template through a neural network model to obtain a final evaluation result based on different dimensions. Then, the step size can be dynamically adjusted according to the final evaluation result. For example, if the model feedback shows that the final evaluation result is not ideal, the step size can be increased to speed up the process. If the final evaluation result is not bad, the step size can be reduced to achieve refined adjustment.

[0118] The specific adjustment method can be determined based on actual conditions, and the embodiments of the present disclosure are not limited to this.

[0119] In the disclosed embodiment, by dynamically adjusting the step size of the time step, the diffusion model used to process the synthetic image can adopt different step sizes at different iteration stages, which not only ensures the processing efficiency but also improves the processing accuracy at the key stage to optimize the quality of the synthetic image.

[0120] In the embodiment of the present disclosure, when the live broadcast room of a real person is started again, the live broadcast room identifier of the target object is detected. When the live broadcast room identifier indicates that the digital human live broadcast room is broadcasting live, the digital human live broadcast room is stopped and switched to the live broadcast room of the real person for live broadcast.

[0121] In the disclosed embodiment, by detecting the target object's live broadcast room identifier, when the system detects that the real person's live broadcast room is started again, the broadcast of the digital human live broadcast room will be suspended and automatically switched to the real person's live broadcast room. This can provide better flexibility for switching between real person live broadcast and digital human live broadcast in the live broadcast platform, and the live broadcast plan can be adjusted according to the real-time situation to ensure the provision of higher quality and more diverse live broadcast content.

[0122] In summary, in the disclosed embodiments, the live broadcast of real people and the live broadcast of digital humans can be switched seamlessly, which can save the operation of manually configuring and generating the live broadcast room of digital humans, thus improving the live broadcast efficiency.

[0123] The process of switching from live broadcast to digital live broadcast is as follows: Figure 6 As shown:

[0124] In S601, when the live broadcast host turns off the broadcast, the live broadcast turn-off signal is triggered. In S602, the live broadcast center receives the live broadcast turn-off signal, obtains the stream address of the live broadcast room, and sends the stream address to the digital human live broadcast (i.e., the digital human live broadcast management platform). In S603, based on the acquired stream address of the live broadcast room, key information such as live broadcast content, host image, interactive content, etc. is extracted from the audio and video data contained in the stream address, and the live broadcast room is intelligently created by analyzing and processing it. In S604, the live broadcast room disconnects the stream address. In S605, the digital human live broadcast room is pushed.

[0125] Correspondingly, when the live broadcaster opens the live broadcast room again, the process of switching from the digital human live broadcast to the live broadcast of the real person is as follows: Figure 7 As shown:

[0126] In S701, when the live broadcast host starts to enter the live broadcast room for live broadcast, the live broadcast start signal is triggered. In S702, the live broadcast center receives the live broadcast start signal and determines the current live broadcast status by detecting the live broadcast room identifier of the target object. In S703, when the current live broadcast room is the digital human live broadcast room and the live broadcast room is ready. In S704, the digital human live broadcast room stream is cut off by the stream address. In S705, the live broadcast room stream of the real person is pushed to the address stream of the digital human live broadcast room.

[0127] Based on the same technical concept, the embodiment of the present disclosure also provides a live broadcast control device 800, such as Figure 8 As shown, including:

[0128] The creation module 801 is used to create a digital human live broadcast room based on the information of the real person live broadcast room when the live broadcast is closed;

[0129] The live broadcast module 802 is used for live broadcast based on the digital human live broadcast room.

[0130] In some embodiments, wherein creating a module comprises:

[0131] An acquisition unit, used for acquiring a second live broadcast element generated in advance based on the first live broadcast element in a live broadcast room of a real person;

[0132] The live broadcast unit is used to create a digital human live broadcast room based on the second live broadcast element.

[0133] In some embodiments, the second live broadcast element includes at least one of the following:

[0134] The image elements, digital humans, digital human voices, digital human live broadcast room scripts, and digital human live broadcast room warm-up question and answer information.

[0135] In some embodiments, the image elements of the digital human live broadcast room include at least one of the following:

[0136] Digital human live broadcast room widgets, Digital human live broadcast room patches, and Digital human live broadcast room background images.

[0137] In some embodiments, the invention further comprises:

[0138] An acquisition module is used to acquire a target image and a template to be processed; the template to be processed is a template in the template set corresponding to the target image class required by the target image;

[0139] A processing module, used for adjusting the template to be processed based on the diffusion model under the guidance of the target image to obtain a target template that is visually adapted to the target image;

[0140] The synthesis module is used to synthesize the target image into the target template to obtain the image-like element.

[0141] In some embodiments, the processing module includes:

[0142] A superposition unit, used for superimposing the first Gaussian noise into the target image to obtain an initial guide image; and superimposing the second Gaussian noise into the template to be processed to obtain an intermediate template;

[0143] a merging unit, configured to merge the initial guide image into a target position of the intermediate template to obtain a first composite image;

[0144] The denoising unit is used to gradually denoise the first synthetic image using a diffusion model to obtain a target template.

[0145] In some embodiments, the noise reduction unit is specifically configured to:

[0146] Predict the prediction noise corresponding to the current time step based on the diffusion model;

[0147] Denoising the first synthesized image based on the predicted noise to obtain a second synthesized image;

[0148] Segmenting an image portion of the target image from the second synthetic image to obtain a first intermediate guide image, and segmenting a to-be-processed template portion to update the intermediate template;

[0149] adding a third Gaussian noise to the first intermediate guide image to obtain a second intermediate guide image;

[0150] The second intermediate guide image is merged into the target position of the updated intermediate template to obtain a new first synthetic image, and the step of predicting the predicted noise corresponding to the current time step based on the diffusion model is returned to be executed until the current time step reaches the target time step, and the updated intermediate template is obtained as the target template.

[0151] In some embodiments, the noise reduction unit is specifically configured to:

[0152] Determine the time step size;

[0153] Determine the current time step based on the step size;

[0154] predicting noise information in the first synthetic image based on the current time step in the diffusion model;

[0155] Denoising the first synthesized image based on the noise information to obtain a third synthesized image;

[0156] Segmenting an image portion of the target image from the third synthetic image to obtain a third intermediate guide image, and segmenting a to-be-processed template portion to update the intermediate template;

[0157] adding a fourth Gaussian noise to the third intermediate guide image to obtain a fourth intermediate guide image;

[0158] The third intermediate guide image is merged into the target position of the updated intermediate template to obtain a new first composite image, and the step of determining the step size of the time step is returned to be executed until the current time step reaches the target time step, and the updated intermediate template is obtained as the target template.

[0159] In some embodiments, the step size of the time steps is dynamically adjusted based on the time steps.

[0160] In some embodiments, the live broadcast unit is specifically used to:

[0161] In the case where the second live broadcast element includes a digital human live broadcast room widget, detecting the validity of the preferential information in the digital human live broadcast room widget;

[0162] When the discount information becomes invalid, stop outputting the Digital Human Live Room widget to the Digital Human Live Room.

[0163] In some embodiments, a switching module is further included, which is used to:

[0164] When the live broadcast room of a real person is started again, the live broadcast room logo of the target object is detected;

[0165] When the live broadcast room logo indicates that the digital human live broadcast room is broadcasting live, stop broadcasting the digital human live broadcast room;

[0166] Switch to the live broadcast room for live broadcast.

[0167] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0168] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0170] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0171] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0172] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0173] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the live broadcast control method. For example, in some embodiments, the live broadcast control method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the live broadcast control method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the live broadcast control method in any other appropriate manner (e.g., by means of firmware).

[0174] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0176] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0178] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0179] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0180] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0181] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A live broadcast control method, comprising: When the live broadcast room of a real person is closed, a digital human live broadcast room is created based on the information of the live broadcast room of the real person; Live broadcast is carried out based on the digital human live broadcast room.

2. The method according to claim 1, wherein: The step of creating a digital human live broadcast room based on the information of the real person live broadcast room includes: Acquire a second live broadcast element generated in advance based on the first live broadcast element of the live broadcast room; Based on the second live broadcast element, the digital human live broadcast room is created.

3. The method according to claim 2, wherein: The second live broadcast element includes at least one of the following: The image elements, digital humans, digital human voices, digital human live broadcast room scripts, and digital human live broadcast room warm-up question and answer information.

4. The method according to claim 3, wherein: The image elements of the digital human live broadcast room include at least one of the following: Digital human live broadcast room widgets, Digital human live broadcast room patches, and Digital human live broadcast room background images.

5. The method according to claim 3, wherein: Based on the first live broadcast element of the live broadcast room, generating the image element includes: Acquire a target image and a template to be processed; the template to be processed is a template in a template set corresponding to a target image class required by the target image; Under the guidance of the target image, adjusting the template to be processed based on a diffusion model to obtain a target template that is visually adapted to the target image; The target image is synthesized into the target template to obtain the image-like element.

6. The method according to claim 5, wherein: The step of adjusting the template to be processed based on a diffusion model under the guidance of the target image to obtain a target template that is visually adapted to the target image includes: superimposing the first Gaussian noise into the target image to obtain an initial guide image; and, Superimposing the second Gaussian noise onto the template to be processed to obtain an intermediate template; Merging the initial guide image into the target position of the intermediate template to obtain a first composite image; The first synthetic image is gradually denoised using a diffusion model to obtain the target template.

7. The method according to claim 6, wherein: The step of gradually reducing the noise of the first synthetic image by using a diffusion model to obtain the target template includes: Predicting the prediction noise corresponding to the current time step based on the diffusion model; Performing noise reduction on the first synthesized image based on the predicted noise to obtain a second synthesized image; Segmenting the image portion of the target image from the second synthetic image to obtain a first intermediate guide image, and segmenting the to-be-processed template portion to update the intermediate template; adding a third Gaussian noise to the first intermediate guide image to obtain a second intermediate guide image; The second intermediate guide image is merged into the target position of the updated intermediate template to obtain a new first synthetic image, and the step of predicting the predicted noise corresponding to the current time step based on the diffusion model is returned to be executed until the current time step reaches the target time step, and the updated intermediate template is obtained as the target template.

8. The method according to claim 6, wherein: The step of gradually reducing the noise of the first synthetic image by using a diffusion model to obtain the target template includes: Determine the time step size; determining a current time step based on the step size; predicting noise information in the first synthetic image based on the current time step in the diffusion model; Performing noise reduction on the first synthesized image based on the noise information to obtain a third synthesized image; Segmenting the image portion of the target image from the third synthetic image to obtain a third intermediate guide image, and segmenting the to-be-processed template portion to update the intermediate template; adding a fourth Gaussian noise to the third intermediate guide image to obtain a fourth intermediate guide image; The third intermediate guide image is merged into the target position of the updated intermediate template to obtain a new first composite image, and the step of determining the step size of the time step is returned to be executed until the current time step reaches the target time step, and the updated intermediate template is obtained as the target template.

9. The method according to claim 8, wherein: The step size of the time step is dynamically adjusted based on the time step.

10. The method according to claim 2, wherein: The step of creating the digital human live broadcast room based on the second live broadcast element includes: In the case where the second live broadcast element includes a digital human live broadcast room widget, detecting the validity of the preferential information in the digital human live broadcast room widget; When the preferential information becomes invalid, the output of the digital human live broadcast room widget to the digital human live broadcast room is stopped.

11. The method according to any one of claims 1 to 10, further comprising: When the live broadcast room of a real person is started again, the live broadcast room logo of the target object is detected; When the live broadcast room identifier indicates that the digital human live broadcast room is broadcasting live, stopping the broadcast of the digital human live broadcast room; Switch to the live broadcast room for live broadcast.

12. A live broadcast control device, comprising: A creation module, used to create a digital human live broadcast room based on the information of the real person live broadcast room when the live broadcast is closed; A live broadcast module is used for live broadcasting based on the digital human live broadcast room.

13. The device according to claim 12, wherein: The creation module includes: An acquisition unit, used for acquiring a second live broadcast element generated in advance based on the first live broadcast element of the live broadcast room; A live broadcast unit is used to create the digital human live broadcast room based on the second live broadcast element.

14. The device according to claim 13, wherein: The second live broadcast element includes at least one of the following: The image elements, digital humans, digital human voices, digital human live broadcast room scripts, and digital human live broadcast room warm-up question and answer information.

15. The device according to claim 14, wherein: The image elements of the digital human live broadcast room include at least one of the following: Digital human live broadcast room widgets, Digital human live broadcast room patches, and Digital human live broadcast room background images.

16. The device according to claim 14, wherein: Also includes: An acquisition module is used to acquire a target image and a template to be processed; The template to be processed is a template in the template set corresponding to the target image class required by the target image; A processing module, used for adjusting the template to be processed based on a diffusion model under the guidance of the target image to obtain a target template that is visually adapted to the target image; A synthesis module is used to synthesize the target image into the target template to obtain the image-like element.

17. The device according to claim 16, wherein: The processing module comprises: A superposition unit, configured to superimpose the first Gaussian noise onto the target image to obtain an initial guide image; and superimpose the second Gaussian noise onto the template to be processed to obtain an intermediate template; a merging unit, configured to merge the initial guide image into a target position of the intermediate template to obtain a first composite image; The denoising unit is used to gradually denoise the first synthetic image using a diffusion model to obtain the target template.

18. The device according to claim 17, wherein: The noise reduction unit is specifically used for: Predicting the prediction noise corresponding to the current time step based on the diffusion model; Performing noise reduction on the first synthesized image based on the predicted noise to obtain a second synthesized image; Segmenting the image portion of the target image from the second synthetic image to obtain a first intermediate guide image, and segmenting the to-be-processed template portion to update the intermediate template; adding a third Gaussian noise to the first intermediate guide image to obtain a second intermediate guide image; The second intermediate guide image is merged into the target position of the updated intermediate template to obtain a new first synthetic image, and the step of predicting the predicted noise corresponding to the current time step based on the diffusion model is returned to be executed until the current time step reaches the target time step, and the updated intermediate template is obtained as the target template.

19. The device according to claim 17, wherein: The noise reduction unit is specifically used for: Determine the time step size; determining a current time step based on the step size; predicting noise information in the first synthetic image based on the current time step in the diffusion model; Performing noise reduction on the first synthesized image based on the noise information to obtain a third synthesized image; Segmenting the image portion of the target image from the third synthetic image to obtain a third intermediate guide image, and segmenting the to-be-processed template portion to update the intermediate template; adding a fourth Gaussian noise to the third intermediate guide image to obtain a fourth intermediate guide image; The third intermediate guide image is merged into the target position of the updated intermediate template to obtain a new first composite image, and the step of determining the step size of the time step is returned to be executed until the current time step reaches the target time step, and the updated intermediate template is obtained as the target template.

20. The device according to claim 19, wherein The step size of the time step is dynamically adjusted based on the time step.

21. The device according to claim 13, wherein: The live broadcast unit is specifically used for: In the case where the second live broadcast element includes a digital human live broadcast room widget, detecting the validity of the preferential information in the digital human live broadcast room widget; When the preferential information becomes invalid, the output of the digital human live broadcast room widget to the digital human live broadcast room is stopped.

22. The apparatus according to any one of claims 12 to 21, further comprising a switching module, configured to: When the live broadcast room of a real person is started again, the live broadcast room logo of the target object is detected; When the live broadcast room identifier indicates that the digital human live broadcast room is broadcasting live, stopping the broadcast of the digital human live broadcast room; Switch to the live broadcast room for live broadcast.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images