Generation method of live broadcast room scene, electronic equipment and computer readable storage medium
Through the combination of image generation model and transparent channel technology, the seamless integration of scene background and prospects in the live broadcast room is achieved, solving the problems of poor generation effect and high cost, and improving the uniqueness and efficiency of scenes in the live broadcast room.
Patent Information
- Application Number
- CN202411018598.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-12
AI Technical Summary
The live broadcast room has poor scene generation effect, and the prior art is difficult to achieve seamless fusion between background images and foreground object images, resulting in homogeneity of generation styles and high cost.
By obtaining input information, using image generation model processing and generation rules, combining transparent channel technology to fuse background images and object images to achieve seamless fusion of background and foreground.
It improves the generation effect of live broadcast scenes, solves the problems of poor generation effect and high cost, and realizes the natural integration of background and prospects.
Smart Images

Figure CN118945438A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, specifically to technical fields such as image recognition and image fusion, and more specifically to a method for generating a live broadcast room scene, an electronic device, and a computer-readable storage medium. Background Art
[0002] At present, with the vigorous development of the live broadcast industry, the visual requirements for live broadcast scenes are getting higher and higher.
[0003] In the related art, live broadcasting scenes are often generated according to preset templates, which causes the styles of the generated live broadcasting scenes to be homogenized and lack uniqueness.
[0004] In order to generate personalized live broadcast scenes, the corresponding live broadcast scenes are often generated by selecting personalized customized designs. In personalized customized designs, it is necessary to frequently communicate with designers about the requirements of the live broadcast scenes to ensure the generation effect of the live broadcast scenes, which will cause the generation speed of the live broadcast scenes to be too slow and the generation cost of the live broadcast scenes to be too high.
[0005] Therefore, no matter which of the above generation methods is adopted, there is a risk that it is difficult to generate the required live broadcast scene according to the requirements of the live broadcast scene, thereby leading to a technical problem of poor generation effect of the live broadcast scene.
[0006] With respect to the technical problem of poor generation effect of the above-mentioned live broadcast room scene, no effective solution has been proposed so far. Summary of the invention
[0007] The embodiments of the present application provide a method for generating a live broadcast room scene, an electronic device, and a computer-readable storage medium to at least solve the technical problem of poor generation effect of the live broadcast room scene.
[0008] According to one aspect of an embodiment of the present application, a method for generating a live studio scene is provided. The method may include: obtaining input information, wherein the input information includes a generation rule for a live studio scene to be generated; processing the input information using an image generation model to obtain a background image that satisfies the generation rule, wherein the image generation model is obtained by training the generation model using background image samples of the live studio scene, and the background image is used to represent the background of the live studio scene to be generated; obtaining an object image to be displayed on the background image, wherein the object image includes a transparent channel, and the image content of the object image includes at least one object to be displayed; using the transparent channel, fusing the object image and the background image to obtain the live studio scene.
[0009] According to another aspect of the embodiment of the present application, a method for generating a network live studio scene is provided. The method may include: obtaining input information, wherein the input information includes a generation rule for a network live studio scene to be generated; processing the input information using an image generation model to obtain a background image that satisfies the generation rule, wherein the image generation model is obtained by training the generation model using background image samples of the network live studio scene, and the background image is used to represent the background of the network live studio scene to be generated; obtaining a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed; using the transparent channel, the product image and the background image are merged to obtain the network live studio scene.
[0010] According to another aspect of the embodiment of the present application, a method for generating a live studio scene is provided. The method may include: in response to an input operation on an operation interface, displaying input information on the operation interface, wherein the input information includes a generation rule for a live studio scene to be generated; in response to a scene generation operation on the operation interface, displaying a live studio scene that satisfies the input information on the operation interface, wherein the live studio scene is obtained by fusing an object image and a background image using a transparent channel in the object image, the image content of the object image includes at least one object to be displayed, the background image satisfies the generation rule, and is obtained by processing the input information using an image generation model, the image generation model is obtained by training the generation model using background image samples of the live studio scene, and the background image is used to represent the background of the generated live studio scene.
[0011] According to another aspect of the embodiment of the present application, a method for generating a live studio scene is provided. The method may include: displaying input information on a presentation screen of a virtual device, wherein the input information includes generation rules for a live studio scene to be generated; processing the input information using an image generation model of the virtual device to obtain a background image that satisfies the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live studio scene, and the background image is used to represent the background of the live studio scene to be generated; driving the virtual device to obtain an object image to be displayed on the background image, and using a transparent channel in the object image to fuse the object image and the background image to obtain a live studio scene, wherein the image content of the object image includes at least one object to be displayed; and driving the virtual device to display the live studio scene.
[0012] According to another aspect of an embodiment of the present application, an electronic device is further provided. The electronic device may include a memory and a processor: the memory stores an executable program; the processor is used to run the executable program, and when the executable program is run by the processor, any one of the above methods is implemented.
[0013] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.
[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which implement the methods in various embodiments of the present application when executed by a processor.
[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.
[0016] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.
[0017] In an embodiment of the present application, when generating a live studio scene, input information can be obtained, that is, the generation rules of the live studio scene to be generated can be obtained. By processing the obtained input information using an image generation model, a background image that meets the generation rules can be obtained, and an object image to be displayed on the background image can be obtained. By using a transparent channel, the obtained object image and the background image are fused to obtain the live studio scene. Since in the generation process of the live studio scene, not only the input information is processed to obtain a background image that meets the generation rules, but also the object image to be displayed on the background image and the background image are fused to generate the live studio scene, the purpose of achieving seamless fusion between the background image and the foreground object image is achieved, thereby achieving the technical effect of improving the generation effect of the live studio scene, and then solving the technical problem of poor generation effect of the live studio scene.
[0018] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 It is a schematic diagram of an application scenario of a method for generating a live broadcast room scene according to an embodiment of the present application;
[0021] Figure 2 is a flow chart of a method for generating a live broadcast room scene according to an embodiment of the present application;
[0022] Figure 3 It is a flow chart of a method for generating a network live broadcast room scene according to an embodiment of the present application;
[0023] Figure 4 is a flowchart of another method for generating a live studio scene according to an embodiment of the present application;
[0024] Figure 5 is a flowchart of another method for generating a live broadcast room scene according to an embodiment of the present application;
[0025] Figure 6 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating a live studio scene according to an embodiment of the present application;
[0026] Figure 7 It is a structural block diagram of a computing environment of a method for generating a live broadcast room scene according to an embodiment of the present application;
[0027] Figure 8 is a schematic diagram of a preview effect of a live broadcast room scene according to an embodiment of the present application;
[0028] FIG9( a ) is a flow chart of a method for generating decoration diffusion of a live broadcast room based on controllable layout and image background removal according to an embodiment of the present application;
[0029] FIG9( b ) is a schematic diagram of a network live broadcast room decoration diffusion generation model based on controllable layout and image background removal according to an embodiment of the present application;
[0030] Fig.10 is a schematic diagram of a device for generating a live broadcast room scene according to an embodiment of the present application;
[0031] Fig.11 It is a schematic diagram of a device for generating a network live broadcast room scene according to an embodiment of the present application;
[0032] Fig.12 is a schematic diagram of another device for generating a live broadcast room scene according to an embodiment of the present application;
[0033] Fig.13 is a schematic diagram of another device for generating a live broadcast room scene according to an embodiment of the present application;
[0034] Fig.14 It is a block diagram of an electronic device for a method for generating a live studio scene according to an embodiment of the present application;
[0035] Fig.15It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0038] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:
[0039] The Stable Diffusion Model can be one of the text graph models and can be used to generate high-quality and realistic images. The text graph model can be used for image generation. For example, a user can use natural language to describe the image to be generated by the user. The text graph model can be used to generate the corresponding image according to the user's needs.
[0040] Controllable Layout Generation can be used to allow users to control and generate specific layout designs by specifying parameters or sketches.
[0041] Image Matting is an image processing technology that can be used to separate objects of interest from the original image.
[0042] Patches are a type of image synthesis technology. Patches can be used to overlay small graphics or images in live videos. The overlaid small graphics or images can be used to display product-related information.
[0043] According to an embodiment of the present application, a method for generating a live broadcast room scene is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0044] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the method for generating the live broadcast room scene provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the image generation method is deployed in the server 10, which can be a cloud. The server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here may include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client devices 20 together constitute the client relative to the server 10. On the graphical user interface of the client device 20, an operation interface for displaying the target image can be deployed, and the operation interface can be an operation interface of an e-commerce platform. The client device 20 can interact with the user through the graphical user interface to implement the call of the method for generating the live broadcast room scene, and then implement the method for generating the live broadcast room scene provided in the embodiment of the present application.
[0045] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: when generating a live broadcast room scene, a user can use a client device such as a mobile phone, a desktop computer, a laptop computer, etc. to communicate data with the server through a network. After receiving the generation instruction sent by the user through the client device, the server can execute step S102 to obtain input information, or execute step S104 to process the input information using an image generation model to obtain a background image that meets the generation rules, and can also execute step S106 to obtain an object image to be displayed on the background image, and execute step S108 to fuse the object image and the background image using a transparent channel to obtain a live broadcast room scene. In the above process, after the server receives the display instruction sent by the user through the client device, the live broadcast room scene can be sent to the client device through the server. It should be noted that when the operating resources of the client device can meet the deployment and operating conditions of the method for generating the live broadcast room scene, the embodiment of the present application can be performed in the client device.
[0046] Under the above operating environment, this application provides Figure 2 The method for generating the live broadcast room scene shown. Figure 2 is a flow chart of a method for generating a live broadcast room scene according to an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps:
[0047] Step S202, obtaining input information.
[0048] In the technical solution provided in the above step S202 of the present application, the above input information may include generation rules of the live broadcast room scene to be generated, and the generation rules may be used to represent the design intent of the live broadcast room scene to be generated.
[0049] In this embodiment, the input information may be information input by the user including generation rules of the live broadcast room scene to be generated. The input information may also include: the original sketch of the live broadcast room scene to be generated, text description information and image input information. The original sketch of the live broadcast room scene to be generated may be used to draw the basic shape of the live broadcast room scene to be generated and mark the key points of the live broadcast room scene to be generated. The text description information may be used to indicate the spatial layout requirements and style requirements of the live broadcast room scene to be generated. The image input information may be used to indicate the template image of the live broadcast room scene to be generated. The template image may be an image selected by the user or directly uploaded from the local end to the cloud. This is only for example and is not specifically limited.
[0050] In this embodiment, the design intention of the live broadcast room scene to be generated may at least include: the spatial layout, functional elements and style elements of the live broadcast room scene to be generated. For example, the spatial layout of the live broadcast room scene to be generated may include at least one of the following layout methods: a method of arranging the objects to be displayed in the center, a method of arranging the objects to be displayed on both sides, and a method of alternating the objects to be displayed, etc. The functional elements of the live broadcast room scene to be generated may be, but are not limited to, the following elements: interactive functional elements, shelf functional elements and shelf functional elements, etc. The style elements of the live broadcast room scene to be generated may be, but are not limited to, the following elements: electronic product style elements, toiletries style elements, household product style elements and travel style elements, etc., which are only used as examples and are not specifically limited.
[0051] This embodiment obtains input information. Optionally, by inputting the original sketch of the live studio scene to be generated, the global text description of the live studio scene to be generated, and the reference image of the live studio scene to be generated in the client, the above input information can be obtained, that is, the generation rules of the live studio scene to be generated can be obtained, thereby determining the design intent of the live studio scene to be generated.
[0052] It should be noted that the above generation rules can be general design rules applicable to all users, or can be design rules formulated according to the needs of specific users. The above generation rules can be used to guide the specific process, method and result of generating a live broadcast room scene, which is only for example and not limited to this.
[0053] Step S204: Process the input information using the image generation model to obtain a background image that meets the generation rules.
[0054] In the technical solution provided in the above step S204 of the present application, the above image generation model can be obtained by training the generation model using background image samples of the live broadcast room scene. The above background image can be used to represent the background of the live broadcast room scene to be generated.
[0055] In this embodiment, the generation model can be an image synthesis model based on controllable layout generation (Open-Set Grounded Text-to-Image Generation, referred to as GLIGEN), or an instance-level controlled text-to-image diffusion model (Instance Diffusion), and the image generation model can be a trained GLIGEN model or a trained Instance Diffusion model. For example, by using background image samples of the live studio scene, the GLIGEN model or the Instance Diffusion model is trained to obtain an image generation model, which is only an example and is not specifically limited.
[0056] After acquiring the input information, this embodiment can use the image generation model to process the input information to obtain a background image that meets the generation rules. Optionally, based on the acquired input information, the acquired input information is input into the image generation model, and then the image generation model is used to perform semantic processing on the input information input therein, so as to obtain the spatial layout information and style information of the live broadcast room scene to be generated. Based on the obtained spatial layout information and style information of the live broadcast room scene to be generated, a background image that meets the above generation rules can be generated.
[0057] For example, if the above input information is a prompt word input by the user, the prompt word may include: generation rules, text description information and image input information of the live broadcast room scene to be generated. The prompt word input by the user is input into the image generation model, and then the image generation model is used to perform semantic processing on the generation rules, text description information and image input information of the prompt word input therein, so as to obtain the spatial layout information and style information of the live broadcast room scene to be generated. Based on the obtained spatial layout information and style information of the live broadcast room scene to be generated, a background image that meets the above generation rules can be generated. This is only for example and is not specifically limited.
[0058] It should be noted that the above method for obtaining a background image that satisfies the generation rules is only an example and is not specifically limited here. As long as the image generation model can be used to process the input information to obtain a background image that satisfies the generation rules, the process and method are within the protection scope of the embodiments of the present application.
[0059] Step S206, obtaining the object image to be displayed on the background image.
[0060] In the technical solution provided in the above step S206 of the present application, the above object image may include a transparent channel, and the image content of the above object image may include at least one object to be displayed. The above transparent channel may be used to represent the channel of transparency information of each area in the object image. The above transparent channel may allow the transparency of each area of the fused live studio scene to be controlled when the object image is fused with the background image. In other words, the above transparent channel may be used to define which area of the fused live studio scene is transparent, which area is opaque, and which area is translucent.
[0061] In this embodiment, the above-mentioned objects to be displayed may be commodities to be displayed. For example, the commodities to be displayed may at least include the following commodities: clothing products, beauty products, electronic products, toiletries, and performance products, etc., which are only examples and are not specifically limited.
[0062] In this embodiment, after the input information is processed by the image generation model to obtain the background image satisfying the generation rule, the object image to be displayed on the background image can be obtained. Optionally, on the basis of obtaining the background image satisfying the generation rule, a patch image of the object to be displayed and / or an image of the object to be displayed to be displayed on the background image can be obtained, and a background removal operation is performed on the obtained patch image of the object to be displayed and / or the image of the object to be displayed, so as to obtain the object image to be displayed on the background image.
[0063] For example, based on the background image that satisfies the generation rules, a patch image of the beauty product to be displayed on the background image and / or an image of the beauty product to be displayed can be obtained, and a background removal operation is performed on the obtained patch image of the beauty product to be displayed and / or the image of the beauty product to be displayed, so that an image of the beauty product to be displayed on the background image can be obtained. This is only an example and is not specifically limited to this.
[0064] Step S208: using a transparent channel, fuse the object image and the background image to obtain a live broadcast room scene.
[0065] In the technical solution provided in the above step S208 of the present application, after obtaining the object image to be displayed on the background image, the above object image and the background image are merged using a transparent channel to obtain a live broadcast room scene. Optionally, based on the object image to be displayed, the object image to be displayed is converted into an object image that matches the appearance attribute information of the background image according to the appearance attribute information of the background image. The background image that meets the generation rules is corrected, and the converted object image and the corrected background image are merged using the transparent channel in the object image to obtain a live broadcast room scene, thereby achieving seamless and natural fusion between the object image and the background image, thereby eliminating the need for complex re-editing of the merged live broadcast room scene, thereby enhancing the harmony between the decoration elements and the live broadcast content.
[0066] Through the above steps S202 to S208 of the present application, when generating the live studio scene, input information can be obtained, that is, the generation rules of the live studio scene to be generated can be obtained. By processing the obtained input information using the image generation model, a background image that meets the generation rules can be obtained, and an object image to be displayed on the background image can be obtained. By using a transparent channel, the obtained object image and the background image are fused to obtain the live studio scene. Since in the generation process of the live studio scene, not only the input information is processed to obtain the background image that meets the generation rules, but also the object image to be displayed on the background image and the background image are fused to generate the live studio scene, the purpose of achieving seamless fusion between the background image and the foreground object image is achieved, thereby achieving the technical effect of improving the generation effect of the live studio scene, and then solving the technical problem of poor generation effect of the live studio scene.
[0067] The above method of this embodiment is further introduced below.
[0068] As an optional implementation, step S204, using the image generation model to process the input information to obtain a background image that satisfies the generation rules, includes: using the image generation model to process the input information to obtain layout information that satisfies the generation rules; using the image generation model to generate a background image according to the layout information.
[0069] In this embodiment, the above layout information can be used to represent the layout structure of the live broadcast room scene to be generated. For example, the above layout information can be spatial layout information, and the spatial layout information can be used to represent at least one of the following layout methods: a method of centering the object to be displayed, a method of arranging the objects to be displayed on both sides, and a method of alternating the objects to be displayed, etc. This is only for example and is not specifically limited.
[0070] In this embodiment, after the input information is acquired, the input information can be processed using the image generation model to obtain layout information that satisfies the generation rules. Optionally, based on the acquired input information, the acquired input information is input into the image generation model, and then the image generation model is used to perform semantic processing on the input information input therein, so as to obtain layout information that satisfies the above generation rules.
[0071] In this embodiment, after obtaining the layout information, the background image can be generated according to the layout information using the image generation model. Optionally, on the basis of obtaining the layout information satisfying the generation rule, the layout information is input into the image generation model, and then the image generation model is used to perform image generation on the layout information input therein, so as to obtain the background image satisfying the generation rule.
[0072] As an optional implementation, step S204, using the image generation model to process the input information to obtain a background image that meets the generation rules, includes: expanding the input information; using the image generation model to process the expanded input information to obtain a background image that meets the generation rules.
[0073] In this embodiment, the amount of information in the input information after expansion may be greater than the amount of information in the input information before expansion.
[0074] In this embodiment, after the input information is obtained, the input information is expanded. Optionally, based on the input information obtained, the obtained input information is input into the text generation model for feature analysis, and the analyzed input information is expanded in the text generation model, because the amount of information in the expanded input information can be greater than the amount of information in the input information before the expansion, thereby achieving the purpose of obtaining richer scene description information.
[0075] In this embodiment, after the input information is expanded, the expanded input information is processed using the image generation model to obtain a background image that meets the generation rules. Optionally, based on the expanded input information, the expanded input information is input into the image generation model, and then the image generation model is used to perform semantic processing on the expanded input information input therein to obtain a background image that meets the generation rules.
[0076] It should be noted that the above method for obtaining a background image that satisfies the generation rules is only an example and is not specifically limited here. As long as the image generation model can be used to process the input information to obtain a background image that satisfies the generation rules, the process and method are within the protection scope of the embodiments of the present application.
[0077] As an optional implementation, expanding the input information includes: expanding the text description information in the input information using a text generation model.
[0078] In this embodiment, the text generation model can be obtained by training a large model, wherein the large model can be a large language model (LLM). The text description information can be used to describe the generation rules of the live broadcast room scene to be generated through text.
[0079] In this embodiment, after the input information is acquired, the text description information in the input information is expanded using the text generation model. Optionally, based on the acquired input information, the acquired input information is input into the text generation model for feature analysis, and the text description information in the input information can be obtained, and the text description information in the input information is expanded in the text generation model. Since the amount of information in the expanded input information can be greater than the amount of information in the input information before the expansion, the purpose of obtaining a richer scene description is achieved.
[0080] As an optional implementation, the expanded input information is processed using an image generation model to obtain a background image that meets the generation rules, including: using the expanded text description information to guide the image generation model to process the image input information in the input information to obtain a background image that meets the generation rules.
[0081] In this embodiment, the image input information may include a generation rule of a live studio scene to be generated represented by an image, wherein the generation rule of the live studio scene to be generated may be used to define the position and layout of the objects to be displayed in the live studio.
[0082] In this embodiment, after the input information is expanded, the expanded text description information is used to guide the image generation model to process the image input information in the input information to obtain a background image that meets the generation rules. Optionally, based on the expanded text description information, the expanded text description information is used to perform feature analysis on the image input information in the input information in the image generation model to obtain image features corresponding to the image input information. The obtained image features are input into the image generation model for image generation to obtain a background image that meets the generation rules.
[0083] As an optional implementation, obtaining an object image to be displayed on a background image includes: obtaining an initial object image to be displayed on the background image; and performing image background removal processing on the initial object image to obtain the object image.
[0084] In this embodiment, the initial object image may include a patch image of the object to be displayed and / or an image of the object to be displayed uploaded by the client. The patch image of the object to be displayed may be used to represent a patch of the commodity to be displayed, and the image of the object to be displayed may be used to represent an image of the commodity to be displayed uploaded by the user through the client.
[0085] In this embodiment, the initial object image may include a patch image of the electronic product to be displayed and / or an image of the electronic product to be displayed uploaded by the client, which is only for illustration and not for specific limitation.
[0086] In this embodiment, after the input information is processed using the image generation model to obtain a background image that satisfies the generation rules, the initial object image to be displayed on the background image can be obtained. Optionally, on the basis of obtaining the background image that satisfies the generation rules, this embodiment can obtain a patch image of the object to be displayed and / or an image of the object to be displayed on the background image. For example, a patch image of toiletries to be displayed on the background image and / or an image of toiletries can be obtained, which is only for illustration and not for specific limitation.
[0087] In this embodiment, after the initial object image is acquired, the initial object image is subjected to image background removal processing to obtain the object image. Optionally, based on the acquisition of the initial object image, this embodiment uses layer diffusion technology to perform image background removal processing on the acquired patch image of the object to be displayed and / or the image of the object to be displayed, so as to obtain the object image to be displayed on the background image. For example, the acquired patch image of the viewing product and / or the image of the viewing product is subjected to image background removal processing to obtain the image of the viewing product to be displayed on the background image, which is only for illustration and is not specifically limited.
[0088] As an optional implementation, performing image background removal processing on the initial object image to obtain the object image includes: determining a transparent image layer of the initial object image; and performing diffusion processing on the transparent image layer to obtain the object image.
[0089] In this embodiment, after obtaining the initial object image to be displayed on the background image, the transparent image layer of the initial object image is determined. Optionally, based on the initial object image, this embodiment uses LayerDiffusion to generate a transparent channel for the obtained initial object image, so as to obtain the transparent image layer of the initial object image.
[0090] In this embodiment, after the transparent image layer is determined, the transparent image layer is diffused to obtain the object image. Optionally, based on the transparent image layer, this embodiment performs layer diffusion processing on the obtained transparent image layer to obtain the object image to be displayed on the background image.
[0091] As an optional implementation, the method further includes: determining appearance attribute information of the background image; clustering the object images according to the appearance attribute information of the background image to obtain object images whose appearance attribute information matches the appearance attribute information of the background image.
[0092] In this embodiment, the appearance attribute information can be used to represent the background image style of the live broadcast room. For example, the appearance attribute information can include at least the following information: color tone information, texture information, style information, etc. The color tone information can include: cold color tone information and warm color tone information, etc. The texture information can include: texture information of the live broadcast room background and texture information of the live broadcast room foreground, etc. The style information can include: simple style information, European style information, rural style information, store style information, business style information, etc. This is only for example and not for specific limitation.
[0093] In this embodiment, the appearance attribute information of the background image is determined. Optionally, this embodiment performs feature extraction on the appearance attribute in the background image to obtain the appearance attribute information of the background image. For example, the hue information, texture information and style information of the background image can be obtained, which is only for illustration and not specifically limited.
[0094] In this embodiment, after the appearance attribute information is determined, the above-mentioned object images are clustered according to the appearance attribute information of the background image, and object images whose appearance attribute information matches the appearance attribute information of the background image can be obtained. Optionally, this embodiment, based on the determination of the appearance attribute information, clusters the object images to be displayed on the background image according to the determined appearance attribute information, and obtains object images to be displayed whose appearance attribute information matches the appearance attribute information of the background image. For example, according to the determined appearance attribute information, clustering is performed on the images of electronic products to be displayed on the background image, and images of electronic products to be displayed whose appearance attribute information matches the appearance attribute information of the background image can be obtained. This is only for illustration and is not specifically limited.
[0095] As an optional implementation, the method further includes: adjusting the object image whose appearance attribute information matches the appearance attribute information of the background image.
[0096] In this embodiment, the quality of the object image after the adjustment may be higher than the quality of the object image whose appearance attribute information matches the appearance attribute information of the background image. The adjustment operation may include at least one of the following operations: color correction operation, contrast optimization operation and detail enhancement (e.g., brightness correction and clarity adjustment) operation.
[0097] In this embodiment, the object image whose appearance attribute information matches the appearance attribute information of the background image is adjusted. Optionally, based on obtaining the object image to be displayed whose appearance attribute information matches the appearance attribute information of the background image, this embodiment performs at least one of the following adjustment operations on the object image: color correction operation, contrast optimization operation, and detail enhancement operation, etc., so that the quality of the adjusted object image can be higher than the quality of the object image whose appearance attribute information matches the appearance attribute information of the background image.
[0098] As an optional implementation, the method also includes: using an image correction model to correct the background image; using a transparent channel to fuse the object image and the background image to obtain a live broadcast room scene, including: using a transparent channel to fuse the adjusted object image and the corrected background image to obtain a live broadcast room scene.
[0099] In this embodiment, the image correction model can be obtained by training a diffusion model, wherein the diffusion model can be implemented based on a stable diffusion refinement model, and the stable diffusion refinement model can be a stable-diffusion-xl-refiner-1.0 model, which is only used as an example and is not specifically limited.
[0100] In this embodiment, the background image is corrected using the image correction model. Optionally, based on obtaining the background image that satisfies the generation rule, this embodiment inputs the obtained background image that satisfies the generation rule into the image correction model for image correction, and a corrected background image can be obtained.
[0101] In this embodiment, the adjusted object image and the corrected background image are fused using a transparent channel to obtain a live studio scene. Optionally, on the basis of obtaining the corrected background image, this embodiment uses the transparent channel included in the object image to perform image fusion on the adjusted object image and the corrected background image to obtain the live studio scene. For example, the adjusted object image is used as the foreground of the live studio scene, and the corrected background image is used as the background of the live studio scene, and the adjusted object image and the corrected background image are fused to obtain the live studio scene.
[0102] As an optional implementation, the method also includes: obtaining feedback information of the live broadcast room scene; based on the feedback information, determining the scene elements to be adjusted in the live broadcast room scene; and adjusting the scene elements in the live broadcast room scene according to the feedback information.
[0103] In this embodiment, the feedback information can be used to indicate layout suggestions and style suggestions for the live broadcast room scene, etc. The scene elements can be used to indicate layout elements and style elements to be adjusted in the live broadcast room scene, etc.
[0104] In this embodiment, feedback information of the live broadcast room scene is obtained. Optionally, this embodiment collects relevant data of the live broadcast room scene in real time, and filters the collected relevant data of the live broadcast room scene to obtain feedback information of the live broadcast room scene, for example, layout suggestions and style suggestions for the live broadcast room scene can be obtained, which is only for example and not specifically limited.
[0105] In this embodiment, after obtaining feedback information of the live broadcast scene, based on the feedback information, the scene elements to be adjusted are determined in the live broadcast scene. Optionally, based on obtaining feedback information of the live broadcast scene, this embodiment can determine the scene elements to be adjusted in the live broadcast scene according to the obtained feedback information. For example, the layout elements and style elements to be adjusted can be determined in the live broadcast scene. This is only for example description and is not specifically limited.
[0106] In this embodiment, after determining the scene elements to be adjusted, the scene elements in the live studio scene are adjusted according to the feedback information. Optionally, based on determining the scene elements to be adjusted, this embodiment adjusts the scene elements in the live studio scene according to the feedback information obtained for the live studio scene, for example, adjusting the layout elements and style elements in the live studio scene, thereby achieving the purpose of real-time editing of the live studio scene.
[0107] The present application also provides a method for generating a network live broadcast room scene. Figure 3 is a flow chart of a method for generating a network live broadcast room scene according to an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps:
[0108] Step S302, obtaining input information.
[0109] In the technical solution provided in the above step S302 of the present application, the above input information may include generation rules for the online live broadcast room scene to be generated, and the generation rules may be used to represent the design intent of the online live broadcast room scene to be generated.
[0110] In this embodiment, the input information may be information input by the user including the generation rules of the live broadcast room scene to be generated. The input information may also include: the original sketch of the live broadcast room scene to be generated, text description information and image input information. The original sketch of the live broadcast room scene to be generated may be used to draw the basic shape of the live broadcast room scene to be generated and mark the key points of the live broadcast room scene to be generated. The text description information may be used to indicate the spatial layout requirements and style requirements of the live broadcast room scene to be generated. The image input information may be used to indicate the template image of the live broadcast room scene to be generated. The template image may be an image selected by the user or directly uploaded from the local end to the cloud. This is only for example and is not specifically limited.
[0111] In this embodiment, the design intention of the live broadcast room scene to be generated may at least include: the spatial layout, functional elements and style elements of the live broadcast room scene to be generated. For example, the spatial layout of the live broadcast room scene to be generated may include at least one of the following layout methods: a method of arranging the objects to be displayed in the center, a method of arranging the objects to be displayed on both sides, and a method of alternating the objects to be displayed, etc. The functional elements of the live broadcast room scene to be generated may be, but are not limited to, the following elements: interactive functional elements, shelf functional elements and shelf functional elements, etc. The style elements of the live broadcast room scene to be generated may be, but are not limited to, the following elements: electronic product style elements, toiletries style elements, household product style elements and travel style elements, etc., which are only used as examples and are not specifically limited.
[0112] This embodiment obtains input information. Optionally, by inputting the original sketch of the network live broadcast room scene to be generated, the global text description of the network live broadcast room scene to be generated, and the reference image of the network live broadcast room scene to be generated in the client, the above input information can be obtained, that is, the generation rule of the network live broadcast room scene to be generated can be obtained, thereby determining the design intent of the network live broadcast room scene to be generated.
[0113] Step S304: Process the input information using the image generation model to obtain a background image that meets the generation rules.
[0114] In the technical solution provided in the above step S304 of the present application, the above image generation model can be obtained by training the generation model using background image samples of the network live broadcast room scene. The above background image can be used to represent the background of the network live broadcast room scene to be generated.
[0115] After acquiring the input information, this embodiment uses the image generation model to process the input information to obtain a background image that meets the generation rules. Optionally, based on the acquired input information, the acquired input information is input into the image generation model, and then the image generation model is used to perform semantic processing on the input information input therein, so as to obtain the spatial layout information and style information of the network live broadcast room scene to be generated. Based on the obtained spatial layout information and style information of the network live broadcast room scene to be generated, a background image that meets the generation rules can be generated.
[0116] It should be noted that the above method for obtaining a background image that satisfies the generation rules is only an example and is not specifically limited here. As long as the image generation model can be used to process the input information to obtain a background image that satisfies the generation rules, the process and method are within the protection scope of the embodiments of the present application.
[0117] Step S306, obtaining a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed.
[0118] In the technical solution provided in the above step S306 of the present application, the above product image may include a transparent channel, and the image content of the above product image may include at least one product to be displayed.
[0119] In this embodiment, the above-mentioned products to be displayed may be commodities to be displayed. For example, the commodities to be displayed may at least include the following commodities: clothing products, beauty products, electronic products, toiletries, and performance products, etc., which are only exemplified here and are not specifically limited.
[0120] This embodiment processes the input information using the image generation model to obtain a background image that satisfies the generation rules, and then obtains a product image to be displayed on the background image. Optionally, on the basis of obtaining the background image that satisfies the generation rules, a patch image of the product to be displayed and / or an image of the product to be displayed to be displayed on the background image can be obtained, and a background removal operation is performed on the obtained patch image of the product to be displayed and / or the image of the product to be displayed, so as to obtain a product image to be displayed on the background image.
[0121] For example, based on the background image that satisfies the generation rules, a patch image of the beauty product to be displayed on the background image and / or an image of the beauty product to be displayed can be obtained, and a background removal operation is performed on the obtained patch image of the beauty product to be displayed and / or the image of the beauty product to be displayed, so that an image of the beauty product to be displayed on the background image can be obtained. This is only an example and is not specifically limited to this.
[0122] Step S308, using a transparent channel, the product image and the background image are merged to obtain a live broadcast room scene.
[0123] In the technical solution provided in the above step S308 of the present application, after obtaining the product image to be displayed on the background image, the product image and the background image are fused using a transparent channel to obtain a network live broadcast room scene. Optionally, based on the acquisition of the product image to be displayed, the product image to be displayed is converted into a product image that matches the appearance attribute information of the background image according to the appearance attribute information of the background image. The background image that meets the generation rules is corrected, and the converted product image and the corrected background image are fused using the transparent channel in the product image, so that a network live broadcast room scene can be obtained, thereby achieving a seamless and natural fusion between the object image and the background image, thereby eliminating the need for complex re-editing of the fused live broadcast room scene, thereby enhancing the harmony between the decoration elements and the live broadcast content.
[0124] Through the above steps S302 to S308 of the present application, when generating a live broadcast room scene, input information can be obtained, that is, the generation rules of the live broadcast room scene to be generated can be obtained. By processing the obtained input information using the image generation model, a background image that meets the generation rules can be obtained, and a product image to be displayed on the background image can be obtained. By using a transparent channel, the obtained product image and the background image are fused to obtain a live broadcast room scene. Since in the generation process of the live broadcast room scene, not only the input information is processed to obtain a background image that meets the generation rules, but also the product image to be displayed on the background image and the background image are fused to generate the live broadcast room scene, the purpose of seamless fusion between the background image and the foreground object image is achieved, thereby achieving the technical effect of improving the generation effect of the live broadcast room scene, and then solving the technical problem of poor generation effect of the live broadcast room scene.
[0125] The embodiment of the present application also provides a method for generating a live broadcast room scene. Figure 4 is a flow chart of another method for generating a live broadcast room scene according to an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps:
[0126] Step S402: In response to an input operation on the operation interface, the input information is displayed on the operation interface.
[0127] In the technical solution provided in the above step S402 of the present application, the above input information may include generation rules of the live broadcast room scene to be generated, and the above input information may be used to describe the design intent of the live broadcast room scene to be generated.
[0128] In this embodiment, if information is input on the operation interface, the input information can be displayed on the operation interface in response to the input operation on the operation interface, that is, the generation rules of the live broadcast room scene to be generated can be displayed on the operation interface.
[0129] Step S404, in response to the scene generation operation on the operation interface, a live broadcast room scene satisfying the input information is displayed on the operation interface.
[0130] In the technical solution provided in the above step S404 of the present application, the above-mentioned live studio scene can be obtained by fusing the object image and the background image using a transparent channel in the object image. The image content of the above-mentioned object image may include at least one object to be displayed. The above-mentioned background image can meet the generation rule and can be obtained by processing the input information using an image generation model. The above-mentioned image generation model can be obtained by training the generation model using background image samples of the live studio scene. The above-mentioned background image can be used to represent the background of the generated live studio scene.
[0131] In this embodiment, after the input information is displayed on the operation interface in response to the input operation on the operation interface, the live broadcast room scene that meets the input information is displayed on the operation interface in response to the scene generation operation on the operation interface. Optionally, if the scene generation is clicked on the operation interface in this embodiment, the acquired input information is input into the image generation model in response to the scene generation operation on the operation interface, and then the input information input therein is semantically processed by using the image generation model, so that the spatial layout information and style information of the live broadcast room scene to be generated can be obtained. According to the obtained spatial layout information and style information of the live broadcast room scene to be generated, a background image that meets the generation rules can be generated, and an object image to be displayed on the background image can be obtained. According to the appearance attribute information of the background image, the object image to be displayed is converted into an object image that matches the appearance attribute information of the background image. The background image that meets the generation rules is corrected, and the converted object image and the corrected background image are image-fused by using the transparent channel in the object image, so that the live broadcast room scene that meets the input information can be obtained, and the live broadcast room scene that meets the input information is displayed on the operation interface.
[0132] Through the above steps S402 to S404 of the present application, when displaying the live studio scene, if information is input on the operation interface, the input information input can be displayed on the operation interface in response to the input operation on the operation interface, that is, the generation rules of the live studio scene to be generated can be displayed on the operation interface. If the scene generation is clicked on the operation interface, the live studio scene that meets the input information can be displayed on the operation interface in response to the scene generation operation on the operation interface. Since in the display process of the live studio scene, not only the input information is processed to obtain the background image that meets the generation rules, but also the object image to be displayed on the background image and the background image are fused to generate the live studio scene, the purpose of achieving seamless fusion between the background image and the foreground object image is achieved, thereby achieving the technical effect of improving the generation effect of the live studio scene, and then solving the technical problem of poor generation effect of the live studio scene.
[0133] The embodiment of the present application also provides a method for generating a live broadcast room scene. Figure 5 is a flow chart of another method for generating a live broadcast room scene according to an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps:
[0134] Step S502: display input information on the presentation screen of the virtual device, wherein the input information includes generation rules for the live broadcast room scene to be generated.
[0135] In the technical solution provided in the above step S502 of the present application, the above virtual device may be a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device, which is only used as an example here without specific limitation.
[0136] In this embodiment, the input information is displayed on the presentation screen of the virtual device. Optionally, in this embodiment, the input information can be obtained by inputting the original sketch of the live broadcast room scene to be generated, the global text description of the live broadcast room scene to be generated, and the reference image of the live broadcast room scene to be generated in the client, thereby displaying the input information on the presentation screen of the virtual device.
[0137] Step S504, using the image generation model of the virtual device to process the input information to obtain a background image that meets the generation rules, wherein the image generation model is trained using background image samples of the live broadcast scene, and the background image is used to represent the background of the live broadcast scene to be generated.
[0138] In the technical solution provided in the above step S504 of the present application, after the input information is displayed on the presentation screen of the virtual device, the input information is processed using the image generation model of the virtual device to obtain a background image that meets the generation rules.
[0139] In this embodiment, based on the input information obtained, the input information is input into the image generation model of the virtual device, and then the image generation model of the virtual device is used to perform semantic processing on the input information input therein, so as to obtain the spatial layout information and style information of the live studio scene to be generated. According to the obtained spatial layout information and style information of the live studio scene to be generated, a background image that meets the generation rules can be generated.
[0140] Step S506, driving the virtual device to obtain the object image to be displayed on the background image, and using the transparent channel in the object image to fuse the object image and the background image to obtain a live broadcast room scene, wherein the image content of the object image includes at least one object to be displayed.
[0141] In the technical solution provided in the above step S506 of the present application, after the input information is processed by the image generation model of the virtual device to obtain the background image that meets the generation rules, the virtual device is driven to obtain the object image to be displayed on the background image, and the object image and the background image are fused by using the transparent channel in the object image to obtain the live broadcast room scene.
[0142] In this embodiment, on the basis of obtaining a background image that satisfies the generation rules, the driving virtual device can obtain a patch image of an object to be displayed and / or an image of an object to be displayed on the background image, and the driving virtual device can perform a background removal operation on the obtained patch image of the object to be displayed and / or the image of the object to be displayed, so as to obtain an object image to be displayed on the background image. The driving virtual device converts the object image to be displayed into an object image that matches the appearance attribute information of the background image according to the appearance attribute information of the background image. The driving virtual device performs a correction operation on the background image that satisfies the generation rules, and the driving virtual device uses the transparent channel in the object image to perform image fusion on the converted object image and the corrected background image, so as to obtain a live broadcast room scene, thereby achieving a seamless and natural fusion between the object image and the background image, so that there is no need to perform complex re-editing on the fused live broadcast room scene, thereby improving the harmony between the decoration elements and the live broadcast content.
[0143] Step S508, driving the virtual device to display the live broadcast room scene.
[0144] In the technical solution provided in the above step S508 of the present application, after driving the virtual device to obtain the object image to be displayed on the background image, and using the transparent channel in the object image to fuse the object image and the background image to obtain the live broadcast room scene, the driving virtual device can display the obtained live broadcast room scene.
[0145] Through the above steps S502 to S508 of the present application, when displaying the live studio scene, the input information can be displayed on the presentation screen of the virtual device. The input information is processed using the image generation model of the virtual device to obtain a background image that meets the generation rules. The virtual device can be driven to obtain the object image to be displayed on the background image, and the object image and the background image are fused using the transparent channel in the object image to obtain the live studio scene, and the virtual device can be driven to display the live studio scene. Since in the display process of the live studio scene, not only the input information is processed to obtain the background image that meets the generation rules, but also the object image to be displayed on the background image and the background image are fused to generate the live studio scene, thereby achieving the purpose of achieving seamless fusion between the background image and the foreground object image, thereby achieving the technical effect of improving the generation effect of the live studio scene, and then solving the technical problem of poor generation effect of the live studio scene.
[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, for example, the data for verification, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0147] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the image generation method of each embodiment of the present application.
[0149] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 6 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating a live studio scene is shown. Figure 6 As shown, the computer terminal 60 (or mobile device) may include one or more (602a, 602b, ..., 602n are used to illustrate) processors 602 (the processor 602 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 704 for storing data, and a transmission module 606 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 6 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown.
[0150] It should be noted that the one or more processors 602 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 60 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0151] The memory 604 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods in the embodiments of the present application. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, that is, the image generation method in the above-mentioned embodiments is realized. The memory 604 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 604 may further include a memory remotely arranged relative to the processor 602, and these remote memories may be connected to the computer terminal 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0152] The transmission device 606 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 60. In one example, the transmission device 606 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 606 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0153] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 60 (or mobile device).
[0154] Figure 6 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the above-mentioned computer terminal 60 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Figure 6 The block diagram shows the use of the above Figure 6 The computer terminal 60 (or mobile device) is shown as an embodiment of a computing node in the computing environment 701. Figure 7 A structural block diagram of a computing environment is shown, such as Figure 7As shown, computing environment 701 includes multiple computing nodes (such as servers) running on a distributed network (shown as 710-1, 710-2, ..., in the figure). The computing nodes all contain local processing and memory resources, and end users 702 can remotely run applications or store data in computing environment 701. Applications can be provided as multiple services 720-1, 720-2, 720-3 and 720-4 in computing environment 701, representing services "A", "D", "E" and "H" respectively.
[0155] The end user 702 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 702's provision and / or request can be provided to the entry gateway 730. The entry gateway 730 may include a corresponding agent to handle the provision and / or request for services (one or more services provided in the computing environment 701).
[0156] Services are provided or deployed based on various virtualization technologies supported by the computing environment 701. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.
[0157] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 7 As shown, service 720-2 can be equipped with one or more Pods 740-1, 740-2, ..., 740-N (collectively referred to as Pods). Pods can include a proxy 745 and one or more containers 742-1, 742-2, ..., 742-M (collectively referred to as containers). One or more containers in the Pod process requests related to one or more corresponding functions of the service, and the proxy 745 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.
[0158] During operation, executing a user request from the end user 702 may require calling one or more services in the computing environment 701, and executing one or more functions of a service may require calling one or more functions of another service. Figure 7As shown, service "A" 720-1 receives a user request from end user 702 from ingress gateway 730, service "A" 720-1 may call service "D" 720-2, and service "D" 720-2 may request service "E" 720-3 to perform one or more functions.
[0159] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.
[0160] At present, with the vigorous development of the live broadcast industry, the visual requirements for live broadcast scenes are getting higher and higher.
[0161] In the related art, the live broadcast room scene is often generated according to the preset template, which will cause the style of the generated live broadcast room scene to be homogenized and lack uniqueness. In order to generate a personalized live broadcast room scene, a personalized customized design is often selected to generate the corresponding live broadcast room scene. In the personalized customized design, it is necessary to frequently communicate with the designer about the requirements of the live broadcast room scene to ensure the generation effect of the live broadcast room scene, which will cause the generation speed of the live broadcast room scene to be too slow and the generation cost of the live broadcast room scene to be too high.
[0162] Therefore, no matter which of the above generation methods is adopted, there is a risk that it is difficult to generate the required live broadcast scene according to the requirements of the live broadcast scene, thereby leading to a technical problem of poor generation effect of the live broadcast scene.
[0163] The present application provides a method for generating a live broadcast room scene, which solves the technical problem of poor generation effect of the live broadcast room scene. Different from the traditional solution, the method cannot achieve seamless fusion between the background image and the foreground object image, and solves the technical problem of poor generation effect of the live broadcast room scene.
[0164] In an embodiment of the present application, the design intention is expressed by inputting information (for example, simple sketches and text description information, etc.), and the image generation model is used to automatically understand these intentions, generate a background image, and generate an object image with a transparent channel, thereby achieving the purpose of seamless fusion between the background image and the foreground object image, thereby achieving the technical effect of improving the generation effect of the live broadcast room scene, and further solving the technical problem of poor generation effect of the live broadcast room scene.
[0165] In this embodiment, Figure 8 is a schematic diagram of a preview effect of a live broadcast room scene according to an embodiment of the present application, such as Figure 8As shown, on the operation interface of the mobile device 801, by clicking the "Preview" button, the live broadcast scene displayed on its operation interface can be previewed, and by clicking the "Apply" button, the previewed live broadcast scene can be applied to the live broadcast, and by clicking the "Return" button, one can return to the editing interface of the live broadcast scene.
[0166] The above method of this embodiment is further introduced below.
[0167] In this embodiment, in order to solve the technical problem of poor generation effect of the live broadcast room scene, a method for generating diffuse decoration of a live broadcast room based on controllable layout and image background removal can be proposed. Figure 9 (a) is a flow chart of a method for generating diffuse decoration of a live broadcast room based on controllable layout and image background removal according to an embodiment of the present application. As shown in Figure 9 (a), the method may include the following steps:
[0168] Step S902, parse the user input information and build the live broadcast room scene.
[0169] In the technical solution provided in the above step S902 of the present application, the position and layout of the objects to be displayed in the live broadcast room can be defined through intuitive sketch input, for example, drawing the basic shape of the live broadcast room scene or marking the key points of the live broadcast room scene. Using image recognition technology (for example, a pipeline algorithm (ram-grounded-sam) that automatically segments objects through text prompts), objects can be automatically identified and extracted from pictures uploaded by users, and detailed design sketches can be constructed. In addition, users can also provide global text description information to further indicate the construction of the live broadcast room scene layout.
[0170] Step S904, expanding and optimizing the semantics of the text description.
[0171] In the technical solution provided in the above step S904 of the present application, after parsing the user input information and constructing the live broadcast room scene, the semantics of the text description is expanded and optimized.
[0172] In this embodiment, LLM is used to perform in-depth analysis on the global text description information input by the user, thereby expanding the global text description information into richer scene description information, thereby enhancing the details and expressiveness of the text description, providing more accurate and rich text prompt information (prompt) for image generation, and ensuring that the generated image can accurately reflect the user's design intent.
[0173] Step S906, generating a high-resolution live broadcast room background image.
[0174] In the technical solution provided in the above step S906 of the present application, after the semantics of the text description is expanded and optimized, a high-resolution live broadcast room background is generated.
[0175] In this embodiment, the expanded global text description information is combined with the user sketch and input into a diffusion model with controllable layout such as GLIGEN or InstanceDiffusion. These models can generate a high-resolution and consistent live studio background image based on the user's input information. Subsequently, the generated image is refined and corrected using the stable-diffusion-xl-refiner-1.0 model, thereby achieving the purpose of improving the image quality.
[0176] Step S908, generating a transparent channel patch, and integrating the generated transparent channel patch with the background image of the live broadcast room.
[0177] In the technical solution provided in the above step S908 of the present application, after generating a high-resolution live broadcast room background, a transparent channel patch is generated, and the generated transparent channel patch is merged with the live broadcast room background image.
[0178] In this embodiment, the LayerDiffusion technology is used to generate a transparent channel for the generated product patch, and the product image uploaded by the user is automatically deducted to remove the original background in the product image and generate a white background image. Then, according to the style of the generated live broadcast room background image, the algorithm matches the live broadcast room background, and based on the clustering algorithm, a style product patch that matches the tone, texture and style is generated, thereby achieving the purpose of ensuring the harmonious unity of the decoration card and the overall live broadcast room design. After the patch design is completed, the image rendering engineering stage is entered. By applying image processing technologies such as color correction, contrast optimization and detail enhancement, the visual quality of the decoration card can be improved, thereby achieving the purpose of ensuring that the decoration card is integrated with the live broadcast room background in terms of color, brightness and clarity, and enhancing the attractiveness and professionalism of the overall visual effect. Finally, through image fusion technology, the processed product patch and the product image with the original background removed can be integrated into the background image of the live broadcast room, thereby completing the artificial intelligence (AI) automated decoration process of the entire live broadcast room.
[0179] Step S910, real-time interaction and dynamic adjustment of the decoration elements of the live broadcast room.
[0180] In the technical solution provided in the above step S910 of the present application, after generating a transparent channel patch and fusing the generated transparent channel patch with the background image of the live broadcast room, the decoration elements of the live broadcast room are interacted and dynamically adjusted in real time.
[0181] In this embodiment, based on the decorated live broadcast room scene, real-time interaction and dynamic adjustment of decoration elements can be supported, allowing anchors and users to instantly modify the decoration layout and elements of the live broadcast room based on the feedback and needs of the live broadcast, thereby achieving the purpose of improving the flexibility and participation of the live broadcast decoration, and providing the audience with a richer and more personalized viewing experience.
[0182] In this embodiment, the above method can be implemented by a network live broadcast room decoration diffusion generation model based on controllable layout and image background removal. For example, Figure 9(b) is a schematic diagram of a network live broadcast room decoration diffusion generation model based on controllable layout and image background removal according to an embodiment of the present application. As shown in Figure 9(b), the model may include: user input 911, a large language model 912, a design sketch 913, a pipeline 914, a refined corrected image 915, a product patch 916, a product image 917 and a fused live broadcast room scene 918.
[0183] In this embodiment, after the user inputs the following types of information: original sketch (optional input type 2), text description information (optional input type 1) and image input information (optional input type 3), the large language model 912 (e.g., LLM model) is used to perform in-depth analysis on the input text description information, thereby expanding the text description information into richer scene description information, thereby enhancing the details and expressiveness of the text description, providing a more precise and rich prompt for image generation, and ensuring that the generated image can accurately reflect the user's design intent.
[0184] In this embodiment, after the user inputs the following types of information: original sketch, text description information and image input information, the position and layout of the objects to be displayed in the live studio can be defined through the input sketch, for example, drawing the basic shape of the live studio scene or marking the key points of the live studio scene. Using the ram-grounded-sam pipeline, objects can be automatically identified and extracted from the pictures uploaded by the user, and a detailed design sketch can be constructed.
[0185] In this embodiment, the expanded scene description information and the constructed design sketch are combined and input into the diffusion model of the controllable layout for synthesis, so as to obtain the background image of the live broadcast room scene. Subsequently, the stable-diffusion-xl-refiner-1.0 model is used to refine and correct the background image of the live broadcast room scene, thereby achieving the purpose of improving the image quality.
[0186] In this embodiment, a transparent channel is generated for the generated product patch, and the product image uploaded by the user is automatically cut out to remove the original background in the product image and generate a white background image. Then, according to the style of the generated live broadcast room background image, the algorithm matches the live broadcast room background, and based on the clustering algorithm, generates a style product patch that matches the tone, texture and style. After the patch design is completed, the image rendering engineering stage is entered. By applying image processing technologies such as color correction, contrast optimization and detail enhancement, the visual quality of the decoration card can be improved. Finally, through image fusion technology, the processed product patch and the product image with the original background removed can be integrated into the background image of the live broadcast room, thereby completing the AI automated decoration process of the entire live broadcast room.
[0187] In an embodiment of the present application, after parsing the user input information and constructing the live broadcast room scene, the semantics of the text description are expanded and optimized to generate a high-resolution live broadcast room background image and a transparent channel patch, and the generated transparent channel patch and the product image with the original background removed are merged with the live broadcast room background image. In addition, the decoration elements of the live broadcast room are interacted and dynamically adjusted in real time, thereby achieving the purpose of seamless fusion between the background image and the foreground object image, thereby achieving the technical effect of improving the generation effect of the live broadcast room scene, and further solving the technical problem of poor generation effect of the live broadcast room scene.
[0188] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 2 The live broadcast room scene generation method shown is a live broadcast room scene generation device.
[0189] Fig.10 is a schematic diagram of a device for generating a live broadcast room scene according to an embodiment of the present application, such as Fig.10 As shown, the device 1000 for generating the live broadcast room scene may include: a first acquisition unit 1002 , a first analysis unit 1004 , a second acquisition unit 1006 and a first fusion unit 1008 .
[0190] The first acquisition unit 1002 is used to acquire input information, wherein the input information includes a generation rule for a live broadcast room scene to be generated.
[0191] The first analysis unit 1004 is used to process the input information using an image generation model to obtain a background image that meets the generation rules, wherein the image generation model is trained using background image samples of a live broadcast scene, and the background image is used to represent the background of the live broadcast scene to be generated.
[0192] The second acquisition unit 1006 is configured to acquire an object image to be displayed on the background image, wherein the object image includes a transparent channel, and the image content of the object image includes at least one object to be displayed.
[0193] The first fusion unit 1008 is used to fuse the object image and the background image using a transparent channel to obtain a live broadcast room scene.
[0194] It should be noted that the first acquisition unit 1002, the first analysis unit 1004, the second acquisition unit 1006 and the first fusion unit 1008 correspond to steps S202 to S208 in Example 2, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 602) and processed by one or more processors (e.g., processors 602a, 602b..., 602n), and the above-mentioned units can also be run in the computer terminal 60 provided in Example 2 as part of the device.
[0195] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 3 The method for generating a network live broadcast room scene shown is a device for generating a network live broadcast room scene.
[0196] Fig.11 is a schematic diagram of a device for generating a network live broadcast room scene according to an embodiment of the present application, such as Fig.11 As shown, the device 1100 for generating the online live broadcast room scene may include: a second acquisition unit 1102, a second analysis unit 1104, a third acquisition unit 1106 and a second fusion unit 1108.
[0197] The second acquisition unit 1102 is used to acquire input information, wherein the input information includes generation rules for the network live broadcast room scene to be generated.
[0198] The second analysis unit 1104 is used to process the input information using an image generation model to obtain a background image that meets the generation rules, wherein the image generation model is trained using background image samples of a live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated.
[0199] The third acquisition unit 1106 is configured to acquire a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed.
[0200] The second fusion unit 1108 is used to use a transparent channel to fuse the product image and the background image to obtain a live broadcast room scene.
[0201] It should be noted that the second acquisition unit 1102, the second analysis unit 1104, the third acquisition unit 1106 and the second fusion unit 1108 correspond to steps S302 to S308 in Example 2, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 602) and processed by one or more processors (e.g., processors 602a, 602b..., 602n), and the above-mentioned units can also be run in the computer terminal 60 provided in Example 2 as part of the device.
[0202] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 4 The live broadcast room scene generation method shown is a live broadcast room scene generation device.
[0203] Fig.12 is a schematic diagram of another device for generating a live broadcast room scene according to an embodiment of the present application, such as Fig.12 As shown, the device 1200 for generating the live broadcast room scene may include: a first display unit 1202 and a second display unit 1204.
[0204] The first display unit 1202 is used to respond to input operations on the operation interface and display input information on the operation interface, wherein the input information includes generation rules for the live broadcast room scene to be generated.
[0205] The second display unit 1204 is used to respond to the scene generation operation acting on the operation interface, and display the live broadcast room scene that meets the input information on the operation interface, wherein the live broadcast room scene is obtained by utilizing the transparent channel in the object image to fuse the object image and the background image, the image content of the object image includes at least one object to be displayed, the background image satisfies the generation rule, and is obtained by processing the input information using the image generation model, the image generation model is obtained by training the generation model using the background image samples of the live broadcast room scene, and the background image is used to represent the background of the generated live broadcast room scene.
[0206] It should be noted that the first display unit 1202 and the second display unit 1204 correspond to steps S402 to S404 in Embodiment 2, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 2. It should be noted that the above units may be hardware components or software components stored in a memory (e.g., memory 602) and processed by one or more processors (e.g., processors 602a, 602b..., 602n), and the above units may also be part of the device and run in the computer terminal 60 provided in Embodiment 2.
[0207] Fig.13 is a schematic diagram of another device for generating a live broadcast room scene according to an embodiment of the present application, such as Fig.13 As shown, the device 1300 for generating the live broadcast room scene may include: a first display unit 1302 , a third analysis unit 1304 , a fourth acquisition unit 1306 and a second display unit 1308 .
[0208] The first display unit 1302 is used to display input information on the presentation screen of the virtual device, wherein the input information includes generation rules of the live broadcast room scene to be generated.
[0209] The third analysis unit 1304 is used to process the input information using the image generation model of the virtual device to obtain a background image that meets the generation rules, wherein the image generation model is trained using background image samples of the live broadcast scene, and the background image is used to represent the background of the live broadcast scene to be generated.
[0210] The fourth acquisition unit 1306 is used to drive the virtual device to acquire the object image to be displayed on the background image, and use the transparent channel in the object image to merge the object image and the background image to obtain a live broadcast room scene, wherein the image content of the object image includes at least one object to be displayed.
[0211] The second display unit 1308 is used to drive the virtual device to display the live broadcast room scene.
[0212] It should be noted that the first display unit 1302, the third analysis unit 1304, the fourth acquisition unit 1306 and the second display unit 1308 correspond to steps S502 to S508 in Embodiment 2, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 2. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 602) and processed by one or more processors (e.g., processors 602a, 602b..., 602n), and the above units can also be run in the computer terminal 60 provided in Embodiment 2 as part of the device.
[0213] In an embodiment of the present application, when generating a live studio scene, input information can be obtained, that is, the generation rules of the live studio scene to be generated can be obtained. By processing the obtained input information using an image generation model, a background image that meets the generation rules can be obtained, and an object image to be displayed on the background image can be obtained. By using a transparent channel, the obtained object image and the background image are fused to obtain the live studio scene. Since in the generation process of the live studio scene, not only the input information is processed to obtain a background image that meets the generation rules, but also the object image to be displayed on the background image and the background image are fused to generate the live studio scene, the purpose of achieving seamless fusion between the background image and the foreground object image is achieved, thereby achieving the technical effect of improving the generation effect of the live studio scene, and then solving the technical problem of poor generation effect of the live studio scene.
[0214] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.
[0215] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.
[0216] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.
[0217] In this embodiment, the computer terminal can execute the program code in the method.
[0218] Optionally, Fig.14 is a block diagram of an electronic device for a method for generating a live studio scene according to an embodiment of the present application. Fig.14 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1402, a memory 1404, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.
[0219] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal B via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0220] The processor can call the information and application programs stored in the memory to perform the following steps: obtaining input information, wherein the input information includes generation rules for a live broadcast room scene to be generated; using an image generation model to process the input information to obtain a background image that satisfies the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; obtaining an object image to be displayed on the background image, wherein the object image includes a transparent channel, and the image content of the object image includes at least one object to be displayed; using the transparent channel, the object image and the background image are merged to obtain the live broadcast room scene.
[0221] Optionally, the processor may also execute the following program code: using an image generation model to process input information to obtain layout information that satisfies generation rules, wherein the layout information is used to represent the layout structure of the live broadcast room scene to be generated; and using the image generation model to generate a background image according to the layout information.
[0222] Optionally, the processor may also execute program code of the following steps: expanding the input information, wherein the amount of information of the expanded input information is greater than the amount of information of the input information before the expansion; and processing the expanded input information using an image generation model to obtain a background image that satisfies the generation rules.
[0223] Optionally, the processor may also execute the following program code: using a text generation model to expand the text description information in the input information, wherein the text generation model is obtained by training a large model, and the text description information is used to describe the generation rules of the live broadcast room scene to be generated through text.
[0224] Optionally, the processor may also execute the following program code: using the expanded text description information to guide the image generation model to process the image input information in the input information to obtain a background image that satisfies the generation rules, wherein the image input information includes the generation rules of the live broadcast room scene to be generated represented by the image.
[0225] Optionally, the processor may also execute the program code of the following steps: obtaining an initial object image to be displayed on a background image, wherein the initial object image includes a patch image of the object to be displayed and / or an image of the object to be displayed uploaded by a client; performing image background removal processing on the initial object image to obtain an object image.
[0226] Optionally, the processor may also execute program codes of the following steps: determining a transparent image layer of an initial object image; and performing diffusion processing on the transparent image layer to obtain an object image.
[0227] Optionally, the processor may also execute program code of the following steps: determining appearance attribute information of the background image; clustering the object image according to the appearance attribute information of the background image to obtain the object image whose appearance attribute information matches the appearance attribute information of the background image.
[0228] Optionally, the processor may also execute the program code of the following steps: adjusting the object image whose appearance attribute information matches the appearance attribute information of the background image, wherein the quality of the adjusted object image is higher than the quality of the object image whose appearance attribute information matches the appearance attribute information of the background image.
[0229] Optionally, the processor may also execute the program code of the following steps: correcting the background image using an image correction model, wherein the image correction model is obtained by training a diffusion model; and fusing the adjusted object image and the corrected background image using a transparent channel to obtain a live broadcast room scene.
[0230] Optionally, the processor may also execute program code of the following steps: obtaining feedback information of the live broadcast room scene; determining scene elements to be adjusted in the live broadcast room scene based on the feedback information; and adjusting scene elements in the live broadcast room scene according to the feedback information.
[0231] Optionally, the processor can call the information and application programs stored in the memory to perform the following steps: obtain input information, wherein the input information includes generation rules for the online live broadcast room scene to be generated; use the image generation model to process the input information to obtain a background image that meets the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the online live broadcast room scene, and the background image is used to represent the background of the online live broadcast room scene to be generated; obtain a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed; use the transparent channel to fuse the product image and the background image to obtain the online live broadcast room scene.
[0232] Optionally, the processor can call the information and application programs stored in the memory to perform the following steps: responding to an input operation on the operation interface, and displaying the input information on the operation interface, wherein the input information includes a generation rule for a live broadcast room scene to be generated; responding to a scene generation operation on the operation interface, and displaying a live broadcast room scene that satisfies the input information on the operation interface, wherein the live broadcast room scene is obtained by utilizing a transparent channel in the object image to fuse an object image and a background image, the image content of the object image includes at least one object to be displayed, the background image satisfies the generation rule, and is obtained by processing the input information using an image generation model, the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the generated live broadcast room scene.
[0233] Optionally, the processor can call the information and application programs stored in the memory to perform the following steps: display input information on the presentation screen of the virtual device, wherein the input information includes generation rules for the live broadcast room scene to be generated; use the image generation model of the virtual device to process the input information to obtain a background image that meets the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; drive the virtual device to obtain an object image to be displayed on the background image, and use a transparent channel in the object image to fuse the object image and the background image to obtain the live broadcast room scene, wherein the image content of the object image includes at least one object to be displayed; drive the virtual device to display the live broadcast room scene.
[0234] By using an embodiment of the present application, a method for generating a live studio scene is provided. When generating the live studio scene, input information can be obtained, that is, the generation rules of the live studio scene to be generated can be obtained. By processing the obtained input information using an image generation model, a background image that meets the generation rules can be obtained, and an object image to be displayed on the background image can be obtained. By using a transparent channel, the obtained object image and the background image are fused to obtain the live studio scene. Since in the generation process of the live studio scene, not only the input information is processed to obtain the background image that meets the generation rules, but also the object image to be displayed on the background image and the background image are fused to generate the live studio scene, the purpose of achieving seamless fusion between the background image and the foreground object image is achieved, thereby achieving the technical effect of improving the generation effect of the live studio scene, and then solving the technical problem of poor generation effect of the live studio scene.
[0235] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the computer terminal B may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, referred to as MID), a PAD, or other terminal devices. Figure 6 It does not limit the structure of the computer terminal B. For example, the computer terminal B may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations are shown.
[0236] Optionally, Fig.15 is a structural block diagram of a computer terminal according to an embodiment of the present application, such as Fig.15 As shown, the computer terminal B may include: one or more (only one is shown in the figure) processors 1502 , a memory 1504 , and a coupler 1506 .
[0237] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating the live broadcast room scene in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for generating the live broadcast room scene. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal B via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0238] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain input information, wherein the input information includes generation rules for the live broadcast room scene to be generated; use the image generation model to process the input information to obtain a background image that meets the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; obtain an object image to be displayed on the background image, wherein the object image includes a transparent channel, and the image content of the object image includes at least one object to be displayed; use the transparent channel to fuse the object image and the background image to obtain the live broadcast room scene.
[0239] Optionally, the processor may also execute the following program code: using an image generation model to process input information to obtain layout information that satisfies generation rules, wherein the layout information is used to represent the layout structure of the live broadcast room scene to be generated; and using the image generation model to generate a background image according to the layout information.
[0240] Optionally, the processor may also execute program code of the following steps: expanding the input information, wherein the amount of information of the expanded input information is greater than the amount of information of the input information before the expansion; and processing the expanded input information using an image generation model to obtain a background image that satisfies the generation rules.
[0241] Optionally, the processor may also execute the following program code: using a text generation model to expand the text description information in the input information, wherein the text generation model is obtained by training a large model, and the text description information is used to describe the generation rules of the live broadcast room scene to be generated through text.
[0242] Optionally, the processor may also execute the following program code: using the expanded text description information to guide the image generation model to process the image input information in the input information to obtain a background image that satisfies the generation rules, wherein the image input information includes the generation rules of the live broadcast room scene to be generated represented by the image.
[0243] Optionally, the processor may also execute the program code of the following steps: obtaining an initial object image to be displayed on a background image, wherein the initial object image includes a patch image of the object to be displayed and / or an image of the object to be displayed uploaded by a client; performing image background removal processing on the initial object image to obtain an object image.
[0244] Optionally, the processor may also execute program codes of the following steps: determining a transparent image layer of an initial object image; and performing diffusion processing on the transparent image layer to obtain an object image.
[0245] Optionally, the processor may also execute program code of the following steps: determining appearance attribute information of the background image; clustering the object image according to the appearance attribute information of the background image to obtain the object image whose appearance attribute information matches the appearance attribute information of the background image.
[0246] Optionally, the processor may also execute the program code of the following steps: adjusting the object image whose appearance attribute information matches the appearance attribute information of the background image, wherein the quality of the adjusted object image is higher than the quality of the object image whose appearance attribute information matches the appearance attribute information of the background image.
[0247] Optionally, the processor may also execute the program code of the following steps: correcting the background image using an image correction model, wherein the image correction model is obtained by training a diffusion model; and fusing the adjusted object image and the corrected background image using a transparent channel to obtain a live broadcast room scene.
[0248] Optionally, the processor may also execute program code of the following steps: obtaining feedback information of the live broadcast room scene; determining scene elements to be adjusted in the live broadcast room scene based on the feedback information; and adjusting scene elements in the live broadcast room scene according to the feedback information.
[0249] Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain input information, wherein the input information includes generation rules for the online live broadcast room scene to be generated; use the image generation model to process the input information to obtain a background image that meets the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the online live broadcast room scene, and the background image is used to represent the background of the online live broadcast room scene to be generated; obtain a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed; use the transparent channel to fuse the product image and the background image to obtain the online live broadcast room scene.
[0250] Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: responding to the input operation on the operation interface, displaying the input information on the operation interface, wherein the input information includes the generation rules of the live broadcast room scene to be generated; responding to the scene generation operation on the operation interface, displaying the live broadcast room scene that meets the input information on the operation interface, wherein the live broadcast room scene is obtained by utilizing the transparent channel in the object image to fuse the object image and the background image, the image content of the object image includes at least one object to be displayed, the background image satisfies the generation rules, and is obtained by processing the input information using an image generation model, the image generation model is obtained by training the generation model using the background image samples of the live broadcast room scene, and the background image is used to represent the background of the generated live broadcast room scene.
[0251] Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: display input information on the presentation screen of the virtual device, wherein the input information includes generation rules for the live broadcast room scene to be generated; use the image generation model of the virtual device to process the input information to obtain a background image that meets the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; drive the virtual device to obtain an object image to be displayed on the background image, and use the transparent channel in the object image to fuse the object image and the background image to obtain the live broadcast room scene, wherein the image content of the object image includes at least one object to be displayed; drive the virtual device to display the live broadcast room scene.
[0252] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0253] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.
[0254] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.
[0255] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: obtaining input information, wherein the input information includes generation rules for a live broadcast room scene to be generated; processing the input information using an image generation model to obtain a background image that satisfies the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; obtaining an object image to be displayed on the background image, wherein the object image includes a transparent channel, and the image content of the object image includes at least one object to be displayed; using the transparent channel, the object image and the background image are merged to obtain the live broadcast room scene.
[0256] Optionally, the computer-readable storage medium may also execute program code of the following steps: using an image generation model to process input information to obtain layout information that satisfies generation rules, wherein the layout information is used to represent the layout structure of the live broadcast room scene to be generated; and using the image generation model to generate a background image according to the layout information.
[0257] Optionally, the computer-readable storage medium may also execute program code for the following steps: expanding the input information, wherein the amount of information of the expanded input information is greater than the amount of information of the input information before the expansion; and processing the expanded input information using an image generation model to obtain a background image that satisfies the generation rules.
[0258] Optionally, the computer-readable storage medium may also execute the program code of the following steps: using a text generation model to expand the text description information in the input information, wherein the text generation model is obtained by training a large model, and the text description information is used to describe the generation rules of the live broadcast room scene to be generated through text.
[0259] Optionally, the computer-readable storage medium can also execute the program code of the following steps: using the expanded text description information to guide the image generation model to process the image input information in the input information to obtain a background image that meets the generation rules, wherein the image input information includes the generation rules of the live broadcast room scene to be generated represented by the image.
[0260] Optionally, the computer-readable storage medium may also execute program code for the following steps: obtaining an initial object image to be displayed on a background image, wherein the initial object image includes a patch image of the object to be displayed and / or an image of the object to be displayed uploaded by a client; performing image background removal processing on the initial object image to obtain an object image.
[0261] Optionally, the computer-readable storage medium may also execute program codes for the following steps: determining a transparent image layer of an initial object image; and performing diffusion processing on the transparent image layer to obtain an object image.
[0262] Optionally, the computer-readable storage medium may also execute program code for the following steps: determining appearance attribute information of a background image; clustering object images according to the appearance attribute information of the background image to obtain object images whose appearance attribute information matches the appearance attribute information of the background image.
[0263] Optionally, the computer-readable storage medium may also execute program code for the following steps: adjusting an object image whose appearance attribute information matches the appearance attribute information of a background image, wherein the quality of the adjusted object image is higher than the quality of an object image whose appearance attribute information matches the appearance attribute information of the background image.
[0264] Optionally, the computer-readable storage medium may also execute program codes of the following steps: correcting the background image using an image correction model, wherein the image correction model is obtained by training a diffusion model; and fusing the adjusted object image and the corrected background image using a transparent channel to obtain a live broadcast room scene.
[0265] Optionally, the computer-readable storage medium may also execute program code of the following steps: obtaining feedback information of the live broadcast room scene; determining scene elements to be adjusted in the live broadcast room scene based on the feedback information; and adjusting scene elements in the live broadcast room scene according to the feedback information.
[0266] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining input information, wherein the input information includes generation rules for a webcast room scene to be generated; processing the input information using an image generation model to obtain a background image that satisfies the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the webcast room scene, and the background image is used to represent the background of the webcast room scene to be generated; obtaining a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed; using the transparent channel, the product image and the background image are merged to obtain the webcast room scene.
[0267] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: in response to an input operation on an operation interface, displaying input information on the operation interface, wherein the input information includes generation rules for a live broadcast room scene to be generated; in response to a scene generation operation on the operation interface, displaying a live broadcast room scene that satisfies the input information on the operation interface, wherein the live broadcast room scene is obtained by utilizing a transparent channel in the object image to fuse an object image and a background image, the image content of the object image includes at least one object to be displayed, the background image satisfies the generation rules, and is obtained by processing the input information using an image generation model, the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the generated live broadcast room scene.
[0268] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: displaying input information on a presentation screen of a virtual device, wherein the input information includes generation rules for a live broadcast room scene to be generated; processing the input information using an image generation model of the virtual device to obtain a background image that satisfies the generation rules, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; driving the virtual device to obtain an object image to be displayed on the background image, and using a transparent channel in the object image to fuse the object image and the background image to obtain a live broadcast room scene, wherein the image content of the object image includes at least one object to be displayed; driving the virtual device to display the live broadcast room scene.
[0269] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include computer instructions, and the computer instructions implement the method provided by the above embodiment when executed by a processor.
[0270] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.
[0271] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.
[0272] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0273] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0274] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0275] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0276] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0277] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, referred to as Read-Only Memory), random access memory (RAM, referred to as Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0278] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for generating a live broadcast room scene, characterized in that: include: Acquire input information, wherein the input information includes a generation rule of a live broadcast room scene to be generated; The input information is processed using an image generation model to obtain a background image that satisfies the generation rule, wherein the image generation model is obtained by training the generation model using background image samples of a live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; Acquire an object image to be displayed on the background image, wherein the object image includes a transparent channel, and image content of the object image includes at least one object to be displayed; The object image and the background image are fused using the transparent channel to obtain the live broadcast room scene.
2. The method according to claim 1, characterized in that The input information is processed using an image generation model to obtain a background image that satisfies the generation rule, including: Processing the input information using an image generation model to obtain layout information that satisfies the generation rule, wherein the layout information is used to represent the layout structure of the live broadcast room scene to be generated; The background image is generated according to the layout information using an image generation model.
3. The method according to claim 1, characterized in that The input information is processed using an image generation model to obtain a background image that satisfies the generation rule, including: Expanding the input information, wherein the amount of information in the expanded input information is greater than the amount of information in the input information before the expansion; The expanded input information is processed using the image generation model to obtain the background image that meets the generation rule.
4. The method according to claim 3, characterized in that The input information is expanded to include: The text description information in the input information is expanded using a text generation model, wherein the text generation model is obtained by training a large model, and the text description information is used to describe the generation rules of the live broadcast room scene to be generated through text.
5. The method according to claim 4, characterized in that The expanded input information is processed using the image generation model to obtain the background image that satisfies the generation rule, including: The expanded text description information is used to guide the image generation model to process the image input information in the input information to obtain the background image that meets the generation rules, wherein the image input information includes the generation rules of the live broadcast room scene to be generated represented by an image.
6. The method according to claim 1, characterized in that Acquiring an object image to be displayed on the background image, comprising: Acquire an initial object image to be displayed on the background image, wherein the initial object image includes a patch image of the object to be displayed and / or an image of the object to be displayed uploaded by a client; The initial object image is subjected to image background removal processing to obtain the object image.
7. The method according to claim 6, characterized in that Performing image background removal processing on the initial object image to obtain the object image includes: determining a transparent image layer of the initial object image; The transparent image layer is subjected to diffusion processing to obtain the object image.
8. The method according to claim 6, characterized in that The method further comprises: Determining appearance attribute information of the background image; According to the appearance attribute information of the background image, clustering processing is performed on the object image to obtain the object image whose appearance attribute information matches the appearance attribute information of the background image.
9. The method according to claim 8, characterized in that The method further comprises: The object image whose appearance attribute information matches the appearance attribute information of the background image is adjusted, wherein the quality of the adjusted object image is higher than the quality of the object image whose appearance attribute information matches the appearance attribute information of the background image.
10. The method according to claim 9, characterized in that The method further comprises: Correcting the background image using an image correction model, wherein the image correction model is obtained by training a diffusion model; Using the transparent channel to fuse the object image and the background image to obtain the live studio scene includes: using the transparent channel to fuse the adjusted object image and the corrected background image to obtain the live studio scene.
11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Obtaining feedback information of the live broadcast room scene; Based on the feedback information, determining a scene element to be adjusted in the live broadcast room scene; According to the feedback information, the scene elements in the live broadcast room scene are adjusted.
12. A method for generating a network live broadcast room scene, characterized in that: include: Acquire input information, wherein the input information includes a generation rule for a network live broadcast room scene to be generated; The input information is processed using an image generation model to obtain a background image that satisfies the generation rule, wherein the image generation model is obtained by training the generation model using background image samples of a live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; Acquire a product image to be displayed on the background image, wherein the product image includes a transparent channel, and the image content of the product image includes at least one product to be displayed; The product image and the background image are fused using the transparent channel to obtain the live broadcast room scene.
13. A method for generating a live broadcast room scene, characterized in that: include: In response to an input operation on the operation interface, the input information is displayed on the operation interface, wherein the input information includes a generation rule of a live broadcast room scene to be generated; In response to a scene generation operation performed on the operation interface, a live broadcast room scene that meets the input information is displayed on the operation interface, wherein the live broadcast room scene is obtained by utilizing a transparent channel in the object image to fuse the object image and the background image, the image content of the object image includes at least one object to be displayed, the background image satisfies the generation rule, and is obtained by processing the input information using an image generation model, the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the generated live broadcast room scene.
14. A method for generating a live broadcast room scene, characterized in that: include: Displaying input information on a presentation screen of a virtual device, wherein the input information includes a generation rule of a live broadcast room scene to be generated; The input information is processed using the image generation model of the virtual device to obtain a background image that meets the generation rule, wherein the image generation model is obtained by training the generation model using background image samples of the live broadcast room scene, and the background image is used to represent the background of the live broadcast room scene to be generated; The virtual device is driven to obtain an object image to be displayed on the background image, and the object image and the background image are merged by using a transparent channel in the object image to obtain the live broadcast room scene, wherein the image content of the object image includes at least one object to be displayed; The virtual device is driven to display the live broadcast room scene.
15. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 14 when running.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 14.
17. A computer program product, characterized in that The method comprises computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.
Citation Information
Cited By
Image generation method and device, electronic equipment, storage medium and program
CN120912725A
