Virtual scene generation method, image processing method and electronic equipment

Abstract

The invention discloses a virtual scene generation method, an image processing method and electronic equipment, and relates to the field of image processing and virtual scene generation. The method comprises the steps that first text data is obtained, and the first text data at least comprises a first text used for describing at least one article and a second text used for describing the environment where the at least one article is located; based on the first text data, a first scene image and environment texture information corresponding to the environment are generated, and the first scene image is used for representing a scene formed by at least one article and the environment; based on the first scene image, determining a three-dimensional model and article texture information corresponding to the at least one article; and rendering the three-dimensional model based on the article texture information, and rendering a preset model based on the environment texture information to generate a virtual scene corresponding to the first text data. The technical problem that in the related technology, the manufacturing cost of the propaganda material of the article is high is solved.

Description

Technical Field

[0001] The present application relates to the fields of image processing and virtual scene generation, and in particular, to a virtual scene generation method, an image processing method and an electronic device. Background Art

[0002] At present, for merchants, it is very time-consuming, labor-intensive and costly to produce promotional materials for a product. In addition, generally a merchant has a large number of products. If promotional materials need to be shot for different products, a large investment is required.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method for generating a virtual scene, an image processing method, and an electronic device, so as to at least solve the technical problem of high production cost of promotional materials for items in the related art.

[0005] According to one aspect of an embodiment of the present application, a method for generating a virtual scene is provided, comprising: obtaining first text data, wherein the first text data at least comprises: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located; based on the first text data, generating a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, determining a three-dimensional model and object texture information corresponding to at least one object; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data.

[0006] According to another aspect of an embodiment of the present application, a method for generating a virtual scene is also provided, including: obtaining first text data, wherein the first text data at least includes: a first text for describing a plurality of objects, and a second text for describing an environment in which the plurality of objects are located, the plurality of objects including goods and a storage table on which the goods are placed; based on the first text data, generating a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize a scene composed of the plurality of objects and the environment; based on the first scene image, determining a three-dimensional model and object texture information corresponding to the plurality of objects; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data.

[0007] According to another aspect of an embodiment of the present application, a method for generating a virtual scene is also provided, comprising: in response to an input instruction acting on an operation interface, displaying first text data on the operation interface, wherein the first text data includes at least: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located; in response to a virtual scene generation instruction acting on the operation interface, displaying a virtual scene corresponding to the first text data on the operation interface, wherein the virtual scene is generated by rendering a three-dimensional model corresponding to at least one object based on object texture information corresponding to the at least one object, and rendering a preset model based on environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on a first scene image, the first scene image and the environmental texture information are generated based on the first text data, and the first scene image is used to characterize a scene consisting of at least one object and the environment.

[0008] According to another aspect of an embodiment of the present application, a method for generating a virtual scene is also provided, including: displaying first text data on a presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the first text data includes at least: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located; based on the first text data, generating a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, determining a three-dimensional model and object texture information corresponding to at least one object; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data; and driving the VR device or AR device to render and display the virtual scene.

[0009] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: obtaining first text data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the first text data, and the first text data includes at least: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generating a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, determining a three-dimensional model and object texture information corresponding to at least one object; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data; outputting the virtual scene by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the virtual scene.

[0010] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the methods in the above embodiments when running.

[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the methods in the above embodiments.

[0012] In an embodiment of the present application, first text data is obtained, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to at least one object are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information, to generate a virtual scene corresponding to the first text data, thereby improving the efficiency of generating the virtual scene. It is easy to notice that the first scene image and the environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image, the three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, thereby obtaining the virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for objects in related technologies.

[0013] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 is a schematic diagram of a hardware environment of a virtual reality device according to a method for generating a virtual scene in an embodiment of the present application;

[0016] Figure 2It is a structural block diagram of a computing environment of a method for generating a virtual scene according to an embodiment of the present application;

[0017] Figure 3 is a flowchart of a method for generating a virtual scene according to Embodiment 1 of the present application;

[0018] Figure 4 is a schematic diagram of a first scene image according to an embodiment of the present application;

[0019] Figure 5 is a schematic diagram of environmental texture information according to an embodiment of the present application;

[0020] Figure 6 is a schematic diagram of environmental texture information according to an embodiment of the present application;

[0021] Figure 7 is a schematic diagram of texture information of an object according to an embodiment of the present application;

[0022] Figure 8 is a schematic diagram of a virtual scene according to an embodiment of the present application;

[0023] Fig. 9 is a schematic diagram of another virtual scene according to an embodiment of the present application;

[0024] Fig.10 is a flow chart of a method for generating a virtual scene according to an embodiment of the present application;

[0025] Fig.11 is a flow chart of stylizing a video according to an embodiment of the present application;

[0026] Fig.12 is a flowchart of a method for generating a virtual scene according to Embodiment 2 of the present application;

[0027] Fig.13 is a flowchart of a method for generating a virtual scene according to Embodiment 3 of the present application;

[0028] Fig.14 is a flowchart of a method for generating a virtual scene according to Embodiment 4 of the present application;

[0029] Fig.15 is a flowchart of an image processing method according to Embodiment 5 of the present application;

[0030] Fig.16 is a schematic diagram of a virtual scene generation device according to Example 6 of the present application;

[0031] Fig.17 is a schematic diagram of a virtual scene generation device according to Embodiment 7 of the present application;

[0032] Fig.18 is a schematic diagram of a virtual scene generation device according to Example 8 of the present application;

[0033] Fig.19 is a schematic diagram of a virtual scene generation device according to Example 9 of the present application;

[0034] Fig. 20 is a schematic diagram of an image processing device according to Embodiment 10 of the present application;

[0035] Fig.21 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0039] Text-generated images refer to the ability to use AI models to generate images based on text descriptions;

[0040] Image-generated text means that AI models can be used to generate descriptive text based on images;

[0041] Scene parsing refers to identifying all objects in an image and annotating them using segmentation masks or detection boxes.

[0042] At present, generative artificial intelligence (AIGC) generally generates the atmosphere background image of the product through text, and then manually combines the product image and the background image to obtain the final effect image. This process is inefficient; direct image generation (Midjourney) is mainly through the method of text-generated images. At that time, this method was difficult to retain the main characteristics and detailed features of the product itself. The entire image was randomly generated based on the text.

[0043] In order to solve the above problems, the present application can place the three-dimensional model of the object into a three-dimensional scene, and then output a two-dimensional image through rendering. Therefore, the present application can strictly retain the object's own properties and detailed characteristics, and restore the authenticity of the product to the greatest extent.

[0044] Currently, there are three main approaches to solving the problem in the process of video stylization: (1) stylizing individual image frames and then stitching the stylized images into a video in sequence; (2) stylizing individual image frames, but using the relationship between video frames (such as optical flow) to constrain the relationship between frames; (3) stylizing a few key frames and then using the relationship between frames to transfer the style of the key frames to the entire video.

[0045] The above three ideas have their own advantages and disadvantages. (1) Since the stylized videos are spliced ​​together, the continuity between frames cannot be guaranteed, which will cause the generated video to have very obvious visual jitter. (2) Although there are constraints between frames, the video jitter will be much better, but since the stylization is done on a single image frame, it is very time-consuming. (3) Using certain key frames can save time, and the video also optimizes the jitter problem by using the relationship between frames, but the overall style has been changed, the main object is not protected, and the color is inconsistent with the original video.

[0046] This application designs a video stylization method that combines a diffusion model and a video keyframe-based method, while taking into account time consumption, video stability, and the application of cutting-edge image-to-image generation technology. This method can not only achieve video-to-video stylization generation, but also achieve a certain degree of visual stability, and can also keep the main object and the overall color consistent with the original video, and has good practicality.

[0047] Example 1

[0048] According to an embodiment of the present application, a method for generating a virtual scene is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Figure 1 Schematic diagram of the hardware environment of a virtual reality device according to a method for generating a virtual scene in an embodiment of the present application. Figure 1 As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 through a network. The above-mentioned virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc., and the above-mentioned terminal 104 is not limited to PCs, mobile phones, tablet computers, etc. The server 102 can be a server corresponding to a media file operator, and the above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

[0050] Optionally, the virtual reality device 104 of this embodiment includes: a memory, a processor, and a transmission device. The memory is used to store an application, which can be used to execute: obtaining first text data, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; generating a first scene image and environmental texture information corresponding to the environment based on the first text data, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; determining a three-dimensional model and object texture information corresponding to at least one object based on the first scene image; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, and generating a virtual scene corresponding to the first text data, thereby solving the technical problem of high production cost of promotional materials for objects in the related art.

[0051] The terminal of this embodiment can be used to display first text data on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; generate a first scene image and environmental texture information corresponding to the environment based on the first text data; determine a three-dimensional model and object texture information corresponding to at least one object based on the first scene image; render the three-dimensional model based on the object texture information and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data, and send the virtual scene to the virtual reality device 104, which displays the virtual scene at the target delivery position after receiving it.

[0052] Optionally, the virtual reality device 104 of this embodiment has an eye-tracking HMD (Head Mount Display) head display and an eye-tracking module that have the same functions as those in the above embodiment, that is, the screen in the HMD head display is used to display real-time images, and the eye-tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyeballs. The terminal of this embodiment obtains the user's position information and movement information in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view direction in the virtual three-dimensional space.

[0053] Figure 1 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the above-mentioned AR / VR device (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Figure 2 The block diagram shows the use of the above Figure 1 The AR / VR device (or mobile device) shown is an embodiment of a computing node in the computing environment 201. Figure 2 is a structural block diagram of a computing environment of a method for generating a virtual scene according to an embodiment of the present application, such as Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the figure). Different computing nodes contain local processing and memory resources, and end users 202 can remotely run applications or store data in computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in computing environment 201, representing services "A", "D", "E" and "H" respectively.

[0054] The end user 202 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 202's provision and / or request can be provided to the entry gateway 230. The entry gateway 230 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 201).

[0055] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0056] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Pods can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.

[0057] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. Figure 2 As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may request service "E" 220-3 to perform one or more functions.

[0058] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0059] Under the above operating environment, this application provides Figure 3 It should be noted that the method for generating a virtual scene in this embodiment can be Figure 1The illustrated embodiment is executed by a mobile terminal. Figure 3 is a flow chart of a method for generating a virtual scene according to Embodiment 1 of the present application. Figure 3 As shown, the method may include the following steps:

[0060] Step S302: Acquire first text data.

[0061] The first text data at least includes: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located.

[0062] The at least one item mentioned above may be an item to be promoted by the merchant, and the type of the item is not limited here.

[0063] The first text data mentioned above may be text data input by a user.

[0064] The first text mentioned above can be used to describe the characteristics of the item itself. For example, the first text can be the type of item, the color of the item, the user of the item, the material of the item, etc., and no limitation is made here.

[0065] The above-mentioned second text can be used to describe the characteristics of the environment in which the item is located, for example, the usage scenario of the item, the placement scenario of the item, etc., without any limitation.

[0066] In an optional embodiment, the user can input first text data on an operation interface, which can be an interface for the user to interact with a computer program. The operation interface can facilitate the user to operate the computer program on the interface by providing images or text. The operation interface can include a text data display area and a virtual scene display area.

[0067] In another optional embodiment, the user can first click on the text input control on the operation interface and input the first text data based on the text input control. The first text data input by the user can be displayed on the operation interface. After the user confirms that the first text data is correct, the subsequent rendering process can be executed according to the first text data, and a virtual scene corresponding to the first text data can be generated, which can be displayed in the virtual scene display area.

[0068] Step S304: Generate a first scene image and environment texture information corresponding to the environment based on the first text data.

[0069] The first scene image is used to represent a scene composed of at least one object and an environment.

[0070] The above-mentioned environmental texture information is mainly used to enhance the realism and immersion of the virtual scene, wherein the environmental texture information can be used to represent data of environmental conditions and texture features of the reflected light on the surface of a simulated object.

[0071] The first scene image may display the objects and scenes described by the first text data.

[0072] Figure 4 is a schematic diagram of a first scene image according to an embodiment of the present application, the first text data can be commercial photography, using natural light, a handbag, placed on a table near a vase, with a lake as the background, and a shallow depth of field, such as Figure 4 As shown, a first scene image may be generated according to first text data.

[0073] Figure 5 is a schematic diagram of environmental texture information according to an embodiment of the present application. The first text data may be based on commercial photography, natural light, a lake background, and a shallow depth of field, such as Figure 5 As shown, it may be environment texture information generated according to the first text data.

[0074] Step S306: determining a three-dimensional model and object texture information corresponding to at least one object based on the first scene image.

[0075] The above-mentioned object texture information can be used to represent the texture properties of the object, such as the material of the object, the smoothness of the object, etc.

[0076] The above-mentioned three-dimensional model can be a model used to represent the shape of an object, wherein the object and the three-dimensional model can be different. The object can be any real object, and the three-dimensional model can be a virtual product of a merchant, and both can be of the same type.

[0077] In an optional embodiment, the first scene image may be subjected to scene analysis to obtain a three-dimensional model and object texture information corresponding to at least one object contained in the first scene image. Optionally, at least one object and background in the first scene image may be identified and classified, thereby completing the scene analysis process for the first scene image.

[0078] Step S308: Rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information, to generate a virtual scene corresponding to the first text data.

[0079] The above-mentioned preset model may be a model for representing the environment form.

[0080] In an optional embodiment, a renderer may be used to render the three-dimensional model based on the object texture information, and a renderer may be used to render the preset model based on the environment texture information to generate a virtual scene corresponding to the first text data. The renderer may be a unified renderer (Unity).

[0081] In another optional embodiment, the three-dimensional model of the object may be rendered using the object texture information so as to retain the object's own properties and detail characteristics.

[0082] The present application provides a process-based method for generating object-based scenes based on text. The method can be based on the current mainstream text-to-image and image-to-text model methods, and combined with the user's product design, to quickly generate user-satisfied product-based videos based on the product 3D model, greatly reducing the operating costs of merchants.

[0083] Through the above steps, first text data is obtained, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to at least one object are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information, to generate a virtual scene corresponding to the first text data, thereby improving the efficiency of generating virtual scenes. It is easy to notice that the first scene image and the environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image, the three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, thereby obtaining the virtual scene corresponding to the first text data. Human beings only need to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for objects in related technologies.

[0084] In the above embodiments of the present application, a first scene image and environmental texture information corresponding to the environment are generated based on first text data, including at least one of the following: generating a first scene image and environmental texture information based on the first text data using a first generation model; generating a first scene image based on the first text data using a first generation model, and de-analyzing the semantics of the first scene image to generate environmental texture information.

[0085] The first generation model mentioned above may be a text-image model. The text-image model may be used to represent a model for generating images based on text information.

[0086] In an optional embodiment, the first generation model may be directly used to generate the first scene image and the environment texture information based on the first text data.

[0087] In another optional embodiment, a first generation model can be used to generate a first scene image based on first text data, and then semantics for describing the scene in the first scene image can be generated based on the first scene image according to a graph-to-text model. The semantics can be de-parsed to generate environmental texture information. By generating the first scene image through the first text data and then de-parsed the semantics in the first scene image, richer information can be obtained, increasing the complexity of the environmental texture information, thereby restoring the realism of the environmental texture information to the greatest extent.

[0088] In the above embodiment of the present application, the semantics of the first scene image are reversely parsed to generate environmental texture information, including: using the second generation model to generate second text data based on the first scene image, wherein the second text data is used to represent the text describing the first scene image; and using the first generation model to generate environmental texture information based on the second text data.

[0089] The second generation model mentioned above may be a picture-to-text model. The picture-to-text model may be used to represent a model for generating text information based on an image.

[0090] In an optional embodiment, when the first text data contains less information, a first scene image can be generated based on the first text data using a first generation model, and then a second text data can be generated based on the first scene image using a second generation model. In this way, the text data can be enriched by first generating an image and then parsing the image, thereby obtaining second text data with more information, so that the environmental texture information generated based on the second text data is more accurate and more in line with the real scene.

[0091] In the above embodiment of the present application, based on the first scene image, determining the three-dimensional model and object texture information corresponding to at least one object includes: identifying at least one object in the first scene image to obtain a segmentation result of at least one object; based on the segmentation result, determining the three-dimensional model from at least one preset model; based on the segmentation result and the first scene image, generating the object texture information.

[0092] In an optional embodiment, at least one object in the first scene image can be identified to obtain a segmentation result of the at least one object. Optionally, the segmentation result can be an instance segmentation map of the at least one object. The instance segmentation map can provide boundary information of the at least one object in the first scene image, thereby distinguishing the instance corresponding to the at least one object in the first scene image.

[0093] The at least one preset model may be a pre-set three-dimensional component. Different three-dimensional components correspond to different forms.

[0094] The at least one preset model mentioned above may exist in a component library, which may contain three-dimensional components of various forms, such as a cuboid, a sphere, etc., which is not limited here, and the preset models stored in the material library may be pre-made according to the actual application scenario.

[0095] The above segmentation results can be used to represent the classification results of the type, shape, and size of at least one object.

[0096] In an optional embodiment, a three-dimensional model that matches the shape and size of at least one object can be determined from at least one preset model according to the segmentation result, so that the three-dimensional model is used as the model corresponding to the object. The object texture information can be determined based on the shape of the object in the segmentation result and the texture attribute of the object in the first scene image.

[0097] In the above embodiment of the present application, based on the segmentation result, a three-dimensional model is determined from at least one preset model, including: performing depth estimation on the first scene image based on the segmentation result to obtain a depth estimation result; using the depth estimation result to restore the scene of at least one object to obtain bounding box information corresponding to the at least one object; and obtaining a preset model that successfully matches the bounding box information from at least one preset model to obtain a three-dimensional model.

[0098] The above-mentioned bounding box information is used to represent the bounding box or bounding box information of the object, which can be a rectangle or a cube that completely surrounds the object. The bounding box can be used to represent the position, size and direction of the object.

[0099] In an optional embodiment, depth estimation can be performed on the first scene image based on the segmentation result. Optionally, the distance or depth information of objects in the image can be estimated through computer vision technology. Depth estimation can help the computer understand the distance relationship of objects in the image, thereby achieving the purpose of scene restoration of at least one object.

[0100] Furthermore, after performing scene restoration on at least one object, the bounding box information of the at least one object in the scene can be determined, and the bounding box information can be matched with at least one preset model. If the preset model successfully matches the bounding box information, the preset model can be determined as the three-dimensional model of the at least one object.

[0101] In another optional embodiment, if the similarity between the two is greater than a preset value, it means that the bounding box information matches the preset model successfully; or, if the bounding box information of the preset model is the same as the bounding box information, it means that the bounding box information matches the preset model successfully. The matching method between the preset model and the bounding box information is not specifically limited here.

[0102] Figure 6 is a schematic diagram of environmental texture information according to an embodiment of the present application, such as Figure 6 As shown, a first scene image can be acquired first, and then at least one object in the first scene image can be identified to obtain a segmentation result of the at least one object. Then, depth estimation can be performed on the first scene image based on the segmentation result to obtain a depth estimation result. The depth estimation result can be used to restore the scene of at least one object to obtain bounding box information corresponding to the at least one object. A preset model that successfully matches the bounding box information can be obtained from at least one preset model to obtain a three-dimensional model.

[0103] In the above embodiment of the present application, object texture information is generated based on the segmentation result and the first scene image, including: determining attribute information of at least one object based on the segmentation result and the first scene image; generating initial texture information corresponding to at least one object based on the attribute information of at least one object; and amplifying the initial texture information to obtain object texture information.

[0104] The above-mentioned attribute information can be used to indicate the category of at least one object, for example, fruit, tree, stone, etc. The attribute information of at least one object is not specifically limited here.

[0105] In an optional embodiment, the type of at least one object can be determined based on the segmentation result and the first scene image, and attribute information of at least one object can be determined based on the type. Initial texture information corresponding to at least one object can be determined based on the attribute information of at least one object. For example, if the attribute of the object is tree, the initial texture information corresponding to the object may be wood grain; if the attribute of the object is stone, the initial texture information corresponding to the object may be marble grain.

[0106] In another optional embodiment, after obtaining the initial texture information corresponding to the object, the initial texture information can be amplified to obtain high-resolution object texture information, so that the high-resolution object texture information can be used to render the object later to improve the rendering precision.

[0107] In the above embodiment of the present application, the initial texture information is amplified to obtain the object texture information, including: using a super-resolution model to amplify the initial texture information to obtain the object texture information.

[0108] The above-mentioned super-resolution model can be Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN), which is only used as an example here. Other super-resolution models can also be used to amplify the initial texture information. The role of the super-resolution model is to amplify a low-resolution image into a high-resolution image. The high-resolution image obtained in this way is also very clear when applied to high-definition scene images, and will not be blurred. For example, a low-resolution image (512*512) can be enlarged to a high-resolution image (2048*2048).

[0109] In an optional embodiment, the super-resolution model can be used to amplify the initial texture information to obtain the object texture information.

[0110] Figure 7 is a schematic diagram of texture information of an object according to an embodiment of the present application, such as Figure 7 What is shown may be the texture information of the magnified item, and the texture corresponding to the item may be generated according to the description information of the item. For example, the description information may be that wood is smooth and bright yellow, and the wood grain may be displayed under the list of generated textures; the description information may be that marble is white, bright, smooth and clean, and the marble grain may be displayed under the list of generated textures.

[0111] Figure 8 is a schematic diagram of a virtual scene according to an embodiment of the present application, such as Figure 8 As shown, the left side may be a first scene image generated according to the first text data, and the right side may be a virtual scene corresponding to the first text data obtained by rendering the three-dimensional model and the preset model according to the first scene image and the environmental texture information. The object in the first scene image may be a bag and the background may be indoors. The preset model is rendered according to the object texture information obtained from the first scene image, and the preset model is rendered based on the environmental texture information to obtain a virtual scene containing a bag and with an indoor background.

[0112] Fig. 9 is a schematic diagram of another virtual scene according to an embodiment of the present application, such as Fig. 9As shown, the left side may be a first scene image generated according to the first text data, and the right side may be a virtual scene corresponding to the first text data obtained by rendering the three-dimensional model and the preset model according to the first scene image and the environmental texture information. The object in the first scene image may be a bag and the background may be outdoors. The preset model is rendered according to the object texture information obtained from the first scene image, and the preset model is rendered based on the environmental texture information to obtain a virtual scene containing a bag and with an outdoor background.

[0113] Fig.10 is a flow chart of a method for generating a virtual scene according to an embodiment of the present application, such as Fig.10 As shown, the method includes:

[0114] Step S1001, the user inputs first text data;

[0115] Step S1002, generating a first scene image based on first text data;

[0116] Step S1003, reverse-analyzing the semantics of the first scene image to generate environment texture information;

[0117] Optionally, the second generation model is used to generate second text data based on the first scene image, wherein the second text data is used to represent text describing the first scene image; and the first generation model is used to generate environmental texture information based on the second text data.

[0118] Step S1004, identifying at least one object in the first scene image, obtaining a segmentation result of at least one object, and executing steps S1005 and S1008;

[0119] Step S1005, determining attribute information of at least one object based on the segmentation result and the first scene image;

[0120] Step S1006, generating initial texture information corresponding to at least one object based on the attribute information of at least one object;

[0121] Step S1007, using the super-resolution model to amplify the initial texture information to obtain the texture information of the object;

[0122] Step S1008, performing depth estimation on the first scene image based on the segmentation result to obtain a depth estimation result;

[0123] Step S1009, using the depth estimation result to restore the scene of at least one object, obtain the bounding box information corresponding to the at least one object, obtain a preset model that successfully matches the bounding box information from at least one preset model, and obtain a three-dimensional model.

[0124] Step S1010, rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information, to obtain a virtual scene corresponding to the first text data.

[0125] This application proposes a process method that can quickly generate object scene graphs based on text, and can improve the effect of the generated virtual scene through scene analysis, scene restoration, text-to-graph, and graph-to-text. The image effect generated by this method is of high quality and usable.

[0126] In the above embodiment of the present application, a first scene image is generated based on first text data, including: generating a second scene image based on second text data using a first generation model; stylizing the second scene image to obtain a stylized image; restoring at least one object in the second scene image to the stylized image to obtain a restored image; and restoring the color of the second scene image to the restored image to obtain the first scene image.

[0127] The above-mentioned stylization refers to converting an image into an image with a specific artistic style. This conversion can be achieved by changing the image's tone, color distribution, texture, and details.

[0128] In an optional embodiment, a second scene image can be generated based on the second text data using the first generation model, and the second scene image can be stylized to obtain a stylized image. In the process of stylizing the second scene image, in order to avoid changing the style of at least one object, at least one object in the second scene image can be obtained by clipping, and the at least one object can be restored to the stylized image to obtain a restored image. In order to maintain color consistency in the restored image, after obtaining the restored image, the color of the second scene image can be restored to the restored image to obtain the first scene image.

[0129] In the above embodiment of the present application, at least one object in the second scene image is restored to the stylized image to obtain a restored image, including: segmenting the second scene image to obtain an image of at least one object and a segmentation result; based on the segmentation result of the at least one object, overlaying the image of the at least one object on the stylized image to obtain a restored image.

[0130] In an optional embodiment, the second scene image may be segmented to obtain an image of at least one object and a segmentation result, that is, the image of at least one object in the second scene image may be cut out based on the segmentation result, and the at least one image may be overlaid on the stylized image to obtain a restored image.

[0131] In the above embodiment of the present application, the color of the second scene image is restored to the restored image to obtain the first scene image, including: obtaining a first channel value of the color channel of the second scene image, and obtaining a second channel value of the color channel of the restored image; replacing the second channel value with the first channel value to obtain the first scene image.

[0132] The first channel value mentioned above may be a color channel (UV channel) of the second scene image.

[0133] In an optional embodiment, a first channel value of a color channel of the second scene image may be obtained, and a second channel value of a color channel of the restored image may be obtained. Since the color in the restored image obtained after stylizing the second scene image has been changed, the second channel value in the restored image needs to be replaced with the first channel value, so that the color channel in the first scene image is the first channel value in the second scene image.

[0134] In the above embodiment of the present application, the second scene image is stylized to obtain a stylized image, including: sampling a target video containing the second scene image to obtain a sampled image, wherein the target video is generated based on the second text data using a first generation model; and stylizing the second scene image based on the sampled image using a diffusion model to obtain a stylized image.

[0135] The above-mentioned diffusion model may be a mathematical model used to describe the diffusion process of a substance in space.

[0136] In an optional embodiment, the target video containing the second scene image can be sampled to obtain a sampled image. Optionally, the target video can be frame extracted to obtain multiple video frames, and key frames can be extracted from the multiple video frames. The key frames can be used as the sampled image.

[0137] In another optional embodiment, a diffusion-related method, such as the current mainstream controllable control net method, is applied to the key frames to transform an image into another image with a different style, thereby obtaining a stylized image.

[0138] Fig.11 is a flow chart of stylizing a video according to an embodiment of the present application, such as Fig.11 As shown, the method includes:

[0139] Step S1101, sampling a target video containing a second scene image to obtain a sampled image;

[0140] Step S1102, performing stylization processing on the second scene image based on the sampled image using a diffusion model to obtain a stylized image;

[0141] Optionally, the target video may be transformed according to a single stylized image to obtain a style-transformed video.

[0142] Step S1103, segmenting the second scene image to obtain an image of at least one object and a segmentation result;

[0143] Step S1104, based on the segmentation result of the at least one object, overlaying the image of the at least one object onto the stylized image to obtain a restored image;

[0144] In a video scene, an image of at least one object may be overlaid onto the style-transferred video to obtain a restored video.

[0145] Step S1105 , restoring the color of the second scene image to the restored image to obtain the first scene image.

[0146] The color of the second scene image in the target video can be restored to the restored video to obtain a final video containing the first scene image.

[0147] Through the above steps, the image of at least one object in the second scene image can be protected from being stylized, and the original characteristics of the object can be retained. Moreover, since the above steps process the entire video instead of processing it by splicing, the produced video has less jitter and the overall quality of the video is higher.

[0148] In this application, a video stylization method combining a diffusion model and a video keyframe-based method is designed under the premise of taking into account time consumption, video stability, and applying cutting-edge image-to-image generation technology. This method can not only achieve video-to-video stylization generation, but also achieve a certain visual stability, and can also keep the main object and the overall color consistent with the original video, and has good practicality.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0150] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0152] Example 2

[0153] According to an embodiment of the present application, a method for generating a virtual scene is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0154] Fig.12 is a flowchart of a method for generating a virtual scene according to Embodiment 2 of the present application, such as Fig.12 As shown, the method comprises the following steps:

[0155] Step S1202: Acquire first text data.

[0156] The first text data at least includes: a first text for describing a plurality of objects, and a second text for describing an environment in which the plurality of objects are located, wherein the plurality of objects include commodities and a storage table on which the commodities are placed.

[0157] Step S1206: Generate a first scene image and environment texture information corresponding to the environment based on the first text data.

[0158] The first scene image is used to represent a scene consisting of multiple objects and an environment.

[0159] Step S1208: determining three-dimensional models and object texture information corresponding to the plurality of objects based on the first scene image.

[0160] Step S1210: Rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information, to generate a virtual scene corresponding to the first text data.

[0161] In an e-commerce scenario, the first text is used to describe multiple items, which may include goods to be sold by the store and a display stand on which the goods are placed. The display stand may be a table, a display stand, etc., which is not limited here. The second text may be a description of the environment in which the multiple items are located, which may be indoors, outdoors, etc. A first scene image consisting of multiple items and the environment and environmental texture information corresponding to the environment may be generated based on the first text data; a three-dimensional model and item texture information corresponding to the multiple items may be determined based on the first scene image. The three-dimensional model may be a three-dimensional model of the goods to be sold and a three-dimensional model of the display stand. The item texture information may be the texture information of the goods to be sold and the texture information of the display stand. The three-dimensional model of the goods to be sold may be rendered based on the texture information of the goods to be sold. The three-dimensional model of the display stand may be rendered based on the texture information of the display stand, and a preset model used to represent the environment may be rendered based on the environmental texture information to generate a virtual scene corresponding to the first text data.

[0162] Through the above steps, first text data is obtained, wherein the first text data at least includes: a first text for describing a plurality of objects, and a second text for describing an environment in which the plurality of objects are located, wherein the plurality of objects include commodities and a shelf on which the commodities are placed; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to represent a scene composed of the plurality of objects and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to the plurality of objects are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information, to generate a virtual scene corresponding to the first text data, and to achieve In order to improve the efficiency of generating virtual scenes, it is easy to notice that the first scene image and environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image. The three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, so as to obtain the virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production cost of promotional materials for objects in related technologies.

[0163] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0164] Example 3

[0165] According to an embodiment of the present application, a method for generating a virtual scene is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0166] Fig.13 is a flow chart of a method for generating a virtual scene according to Embodiment 3 of the present application, such as Fig.13 As shown, the method comprises the following steps:

[0167] Step S1302: In response to an input instruction on the operation interface, display first text data on the operation interface.

[0168] The first text data at least includes: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located.

[0169] The above-mentioned operation interface can be used to display the first text data. The above-mentioned first text data can be displayed by performing relevant touch operations on the display interface.

[0170] Step S1304 , in response to the virtual scene generation instruction acting on the operation interface, the virtual scene corresponding to the first text data is displayed on the operation interface.

[0171] The virtual scene is generated by rendering a three-dimensional model corresponding to at least one object based on object texture information corresponding to the at least one object, and rendering a preset model based on environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on a first scene image, the first scene image and the environmental texture information are generated based on first text data, and the first scene image is used to represent a scene composed of at least one object and the environment.

[0172] The virtual scene generation instruction may be an instruction generated by touching a related control on the operation interface when a virtual scene needs to be generated according to the first text data. The virtual scene may be displayed on the operation interface according to the virtual scene generation.

[0173] Through the above steps, in response to the input instruction acting on the operation interface, first text data is displayed on the operation interface, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; in response to the virtual scene generation instruction acting on the operation interface, a virtual scene corresponding to the first text data is displayed on the operation interface, wherein the virtual scene is generated by rendering a three-dimensional model corresponding to at least one object based on object texture information corresponding to the at least one object, and rendering a preset model based on environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on the first scene image, and the first scene image and the environmental texture information are generated based on the first text data. The first scene image is used to characterize a scene composed of at least one object and an environment, thereby improving the efficiency of generating virtual scenes. It is easy to notice that the first scene image and environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image. The three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, so as to obtain the virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for objects in related technologies.

[0174] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0175] Example 4

[0176] According to an embodiment of the present application, a method for generating a virtual scene in a virtual reality scene such as a virtual reality VR device, an augmented reality AR device, etc. is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0177] Optionally, in this embodiment, the virtual scene generation method can be applied to a hardware environment composed of a server and a virtual reality device. The virtual scene is displayed on a presentation screen of a virtual reality VR device or an augmented reality AR device. The server can be a server corresponding to a media file operator. The network includes but is not limited to: a wide area network, a metropolitan area network or a local area network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc.

[0178] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory is used to store an application, which can be used to execute: displaying first text data on a presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; generating a first scene image and environmental texture information corresponding to the environment based on the first text data; wherein the first scene image is used to represent a scene composed of at least one object and the environment; determining a three-dimensional model and object texture information corresponding to at least one object based on the first scene image; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data; and driving the VR device or AR device to render and display the virtual scene.

[0179] Fig.14 is a flowchart of a method for generating a virtual scene according to Embodiment 4 of the present application, such as Fig.14 As shown, the method comprises the following steps:

[0180] Step S1402: display the first text data on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device.

[0181] The first text data at least includes: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located.

[0182] Step S1404: Generate a first scene image and environment texture information corresponding to the environment based on the first text data.

[0183] The first scene image is used to represent a scene composed of at least one object and an environment.

[0184] Step S1406: determining a three-dimensional model and object texture information corresponding to at least one object based on the first scene image.

[0185] Step S1408: Rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information, to generate a virtual scene corresponding to the first text data.

[0186] Step S1410: driving the VR device or AR device to render and display the virtual scene.

[0187] Through the above steps, the first text data is displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to represent the scene composed of at least one object and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to at least one object are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information to generate a virtual scene corresponding to the first text data; and the VR device is driven Or AR devices render and display virtual scenes, thereby improving the efficiency of generating virtual scenes. It is easy to notice that the first scene image and environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image. The three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, so as to obtain the virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for objects in related technologies.

[0188] It should be noted that the method for generating a virtual scene in the VR device or AR device in the embodiment may include: Fig.14 The method of the illustrated embodiment is used to achieve the purpose of driving a VR device or an AR device to display a virtual scene.

[0189] Optionally, the processor of this embodiment can call the application stored in the memory to execute the above steps through a transmission device. The transmission device can receive media files sent by the server through the network, and can also be used for data transmission between the processor and the memory.

[0190] Optionally, in a virtual reality device, a head-mounted display with eye tracking is provided, wherein the screen in the HMD is used to display the displayed video images, the eye tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyes, the tracking system is used to track the user's position information and motion information in the real three-dimensional space, and the computing processing unit is used to obtain the user's real-time position and motion information from the tracking system, and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view direction in the virtual three-dimensional space, etc.

[0191] In an embodiment of the present application, a virtual reality device may be connected to a terminal, and the terminal and the server are connected through a network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc. The terminal is not limited to a PC, a mobile phone, a tablet computer, etc. The server may be a server corresponding to a media file operator, and the network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

[0192] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0193] Example 5

[0194] According to an embodiment of the present application, an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0195] Fig.15 is a flowchart of an image processing method according to Embodiment 5 of the present application, such as Fig.15 As shown, the method comprises the following steps:

[0196] Step S1502: Acquire first text data by calling a first interface.

[0197] The first interface includes a first parameter, a parameter value of the first parameter is first text data, and the first text data includes at least: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located.

[0198] The first interface in the above steps may be an interface for data interaction between the cloud server and the client. The client may pass the first text data into the interface function as the first parameter of the interface function to upload the first text data to the cloud server.

[0199] Step S1504: Generate a first scene image and environment texture information corresponding to the environment based on the first text data.

[0200] The first scene image is used to represent a scene composed of at least one object and an environment.

[0201] Step S1506: Determine a three-dimensional model and object texture information corresponding to at least one object based on the first scene image.

[0202] Step S1508: Rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information, to generate a virtual scene corresponding to the first text data.

[0203] Step S1510: output the virtual scene by calling the second interface.

[0204] The second interface includes a second parameter, and a parameter value of the second parameter is a virtual scene.

[0205] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the virtual scene into the interface function as the second parameter of the interface function to achieve the purpose of sending the virtual scene to the client.

[0206] Through the above steps, the first text data is obtained by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the first text data, and the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to at least one object are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information to generate a virtual scene corresponding to the first text data; the virtual scene is output by calling the second interface Scene, wherein the second interface includes a second parameter, and the parameter value of the second parameter is a virtual scene, thereby improving the efficiency of generating virtual scenes. It is easy to notice that the first scene image and environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image. The three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, so as to obtain the virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for objects in related technologies.

[0207] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0208] Example 6

[0209] According to an embodiment of the present application, a virtual scene generation device for implementing the above virtual scene generation method is also provided. Fig.16 is a schematic diagram of a virtual scene generation device according to Example 6 of the present application, such as Fig.16 As shown, the device 1600 includes: an acquisition module 1602 , a generation module 1604 , a determination module 1606 , and a rendering module 1608 .

[0210] Among them, the acquisition module is used to acquire first text data, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; the generation module is used to generate a first scene image and environmental texture information corresponding to the environment based on the first text data, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; the determination module is used to determine the three-dimensional model and object texture information corresponding to at least one object based on the first scene image; the rendering module is used to render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data.

[0211] It should be noted that the acquisition module 1602, the generation module 1604, the determination module 1606, and the rendering module 1608 correspond to steps S302 to S308 in Embodiment 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above modules may also be part of the device and may be run in the computer terminal 10 provided in Embodiment 1.

[0212] In the above embodiments of the present application, the generation module is also used to generate a first scene image and environmental texture information based on the first text data using the first generation model; the generation module is also used to generate a first scene image based on the first text data using the first generation model, and de-parse the semantics of the first scene image to generate environmental texture information.

[0213] In the above embodiment of the present application, the generation module is also used to generate second text data based on the first scene image using the second generation model, wherein the second text data is used to represent the text describing the first scene image; and generate environmental texture information based on the second text data using the first generation model.

[0214] In the above embodiment of the present application, the determination module is also used to identify at least one object in the first scene image to obtain a segmentation result of at least one object; based on the segmentation result, determine a three-dimensional model from at least one preset model; based on the segmentation result and the first scene image, generate object texture information.

[0215] In the above embodiment of the present application, the determination module is also used to perform depth estimation on the first scene image based on the segmentation result to obtain a depth estimation result; use the depth estimation result to restore the scene of at least one object to obtain bounding box information corresponding to at least one object; obtain a preset model that successfully matches the bounding box information from at least one preset model to obtain a three-dimensional model.

[0216] In the above embodiment of the present application, the generation module is also used to determine the attribute information of at least one object based on the segmentation result and the first scene image; generate initial texture information corresponding to at least one object based on the attribute information of at least one object; and amplify the initial texture information to obtain the object texture information.

[0217] In the above embodiments of the present application, the generation module is also used to amplify the initial texture information using the super-resolution model to obtain the texture information of the object.

[0218] In the above embodiment of the present application, the generation module is also used to generate a second scene image based on the second text data using the first generation model; stylize the second scene image to obtain a stylized image; restore at least one object in the second scene image to the stylized image to obtain a restored image; and restore the color of the second scene image to the restored image to obtain the first scene image.

[0219] In the above embodiment of the present application, the generation module is also used to segment the second scene image to obtain an image of at least one object and a segmentation result; based on the segmentation result of at least one object, the image of at least one object is overlaid on the stylized image to obtain a restored image.

[0220] In the above embodiment of the present application, the generation module is also used to obtain the first channel value of the color channel of the second scene image, and obtain the second channel value of the color channel of the restored image; replace the second channel value with the first channel value to obtain the first scene image.

[0221] In the above embodiment of the present application, the generation module is also used to sample the target video containing the second scene image to obtain a sampled image, wherein the target video is generated based on the second text data using the first generation model; and the second scene image is stylized based on the sampled image using the diffusion model to obtain a stylized image.

[0222] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0223] Example 7

[0224] According to an embodiment of the present application, a virtual scene generation device for implementing the above virtual scene generation method is also provided. Fig.17 is a schematic diagram of a virtual scene generation device according to Example 7 of the present application, such as Fig.17 As shown, the device 1700 includes: an acquisition module 1702 , a generation module 1704 , a determination module 1706 , and a rendering module 1708 .

[0225] Among them, the acquisition module is used to acquire first text data, wherein the first text data at least includes: a first text for describing multiple objects, and a second text for describing the environment in which the multiple objects are located, the multiple objects include goods and a storage table on which the goods are placed; the generation module is used to generate a first scene image and environmental texture information corresponding to the environment based on the first text data, wherein the first scene image is used to characterize the scene composed of multiple objects and the environment; the determination module is used to determine the three-dimensional model and object texture information corresponding to the multiple objects based on the first scene image; the rendering module is used to render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data.

[0226] It should be noted that the acquisition module 1702, the generation module 1704, the determination module 1706, and the rendering module 1708 correspond to steps S1002 to S1008 in Embodiment 2, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above modules may also be part of the device and may be run in the computer terminal 10 provided in Embodiment 1.

[0227] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0228] Example 8

[0229] According to an embodiment of the present application, a virtual scene generation device for implementing the above virtual scene generation method is also provided. Fig.18 is a schematic diagram of a virtual scene generation device according to Example 8 of the present application, such as Fig.18 As shown, the device 1800 includes: an acquisition module 1802 and a display module 1804.

[0230] Among them, the acquisition module is used to respond to the input instruction acting on the operation interface, and display the first text data on the operation interface, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; the display module is used to respond to the virtual scene generation instruction acting on the operation interface, and display the virtual scene corresponding to the first text data on the operation interface, wherein the virtual scene is generated by rendering a three-dimensional model corresponding to at least one object based on the object texture information corresponding to the at least one object, and rendering a preset model based on the environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on the first scene image, the first scene image and the environmental texture information are generated based on the first text data, and the first scene image is used to represent the scene composed of at least one object and the environment.

[0231] It should be noted that the acquisition module 1802 and the display module 1804 correspond to steps S1102 to S1104 in Embodiment 3, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above modules may also be part of the device and may be run in the computer terminal 10 provided in Embodiment 1.

[0232] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0233] Example 9

[0234] According to an embodiment of the present application, a virtual scene generation device for implementing the above virtual scene generation method is also provided. Fig.19 is a schematic diagram of a virtual scene generation device according to Example 9 of the present application, such as Fig.19 As shown, the device 1900 includes: a display module 1902 , a generation module 1904 , a determination module 1906 , a rendering module 1908 , and a driving module 1910 .

[0235] Among them, the display module is used to display first text data on the presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; the generation module is used to generate a first scene image and environmental texture information corresponding to the environment based on the first text data, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; the determination module is used to determine the three-dimensional model and object texture information corresponding to at least one object based on the first scene image; the rendering module is used to render the three-dimensional model based on the object texture information, and to render the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data; the driving module is used to drive the VR device or AR device to render and display the virtual scene.

[0236] It should be noted that the above-mentioned display module 1902, generation module 1904, determination module 1906, rendering module 1908, and driving module 1910 correspond to steps S1202 to S1210 in Example 4, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0237] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0238] Example 10

[0239] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig. 20 is a schematic diagram of an image processing device according to Embodiment 10 of the present application, such as Fig. 20 As shown, the device 2000 includes: a calling module 2002 , a generating module 2004 , a determining module 2008 , a rendering module 2010 , and an output module 2012 .

[0240] Among them, the calling module is used to obtain first text data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first text data, and the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; the generating module is used to generate a first scene image and environmental texture information corresponding to the environment based on the first text data, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; the determining module is used to determine the three-dimensional model and object texture information corresponding to at least one object based on the first scene image; the rendering module is used to render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data; the output module is used to output the virtual scene by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the virtual scene.

[0241] It should be noted that the above-mentioned calling module 2002, generating module 2004, determining module 2008, rendering module 2010, and output module 2012 correspond to steps S1302 to S1312 in Embodiment 5, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Embodiment 1.

[0242] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0243] Embodiment 11

[0244] The embodiment of the present application may provide an electronic device, which may be an AR / VR device, and the AR / VR device may be any AR / VR device in an AR / VR device group. Optionally, in this embodiment, the AR / VR device may also be replaced by a terminal device such as a mobile terminal.

[0245] Optionally, in this embodiment, the AR / VR device may be a program code for the following steps in the method for generating a virtual scene: obtaining first text data, wherein the first text data includes at least: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generating a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, determining a three-dimensional model and object texture information corresponding to at least one object; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data.

[0246] Optionally, Fig.21 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.21 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0247] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating the virtual scene in the embodiment of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for generating the virtual scene. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0248] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain first text data, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generate a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, determine a three-dimensional model and object texture information corresponding to at least one object; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data.

[0249] Optionally, the processor may also execute program code of the following steps: using a first generation model to generate a first scene image and environmental texture information based on first text data; using a first generation model to generate a first scene image based on first text data, and de-analyzing the semantics of the first scene image to generate environmental texture information.

[0250] Optionally, the processor may also execute program code of the following steps: generating second text data based on the first scene image using the second generation model, wherein the second text data is used to represent text describing the first scene image; generating environmental texture information based on the second text data using the first generation model.

[0251] Optionally, the processor may also execute program code of the following steps: identifying at least one object in the first scene image to obtain a segmentation result of at least one object; based on the segmentation result, determining a three-dimensional model from at least one preset model; and generating object texture information based on the segmentation result and the first scene image.

[0252] Optionally, the processor may also execute program code for the following steps: performing depth estimation on the first scene image based on the segmentation result to obtain a depth estimation result; performing scene restoration on at least one object using the depth estimation result to obtain bounding box information corresponding to at least one object; obtaining a preset model that successfully matches the bounding box information from at least one preset model to obtain a three-dimensional model.

[0253] Optionally, the processor may also execute program code for the following steps: determining attribute information of at least one object based on the segmentation result and the first scene image; generating initial texture information corresponding to at least one object based on the attribute information of at least one object; and amplifying the initial texture information to obtain object texture information.

[0254] Optionally, the processor may also execute program code of the following steps: amplifying initial texture information using a super-resolution model to obtain object texture information.

[0255] Optionally, the processor may also execute program code for the following steps: generating a second scene image based on the second text data using the first generation model; stylizing the second scene image to obtain a stylized image; restoring at least one object in the second scene image to the stylized image to obtain a restored image; restoring the color of the second scene image to the restored image to obtain the first scene image.

[0256] Optionally, the processor may also execute program code of the following steps: segmenting the second scene image to obtain an image of at least one object and a segmentation result; and based on the segmentation result of the at least one object, overlaying the image of the at least one object onto the stylized image to obtain a restored image.

[0257] Optionally, the processor may also execute the following program code: obtaining a first channel value of a color channel of the second scene image, and obtaining a second channel value of a color channel of the restored image; and replacing the second channel value with the first channel value to obtain the first scene image.

[0258] Optionally, the processor may also execute the program code of the following steps: sampling a target video including a second scene image to obtain a sampled image, wherein the target video is generated based on the second text data using a first generation model; and stylizing the second scene image based on the sampled image using a diffusion model to obtain a stylized image.

[0259] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain first text data, wherein the first text data at least includes: a first text for describing multiple objects, and a second text for describing the environment in which the multiple objects are located, and the multiple objects include goods and a storage table on which the goods are placed; based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to represent the scene composed of the multiple objects and the environment; based on the first scene image, determine the three-dimensional models and object texture information corresponding to the multiple objects; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data.

[0260] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to an input instruction acting on the operation interface, display first text data on the operation interface, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; in response to a virtual scene generation instruction acting on the operation interface, display a virtual scene corresponding to the first text data on the operation interface, wherein the virtual scene is generated by rendering a three-dimensional model corresponding to at least one object based on object texture information corresponding to the at least one object, and rendering a preset model based on environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on a first scene image, the first scene image and the environmental texture information are generated based on the first text data, and the first scene image is used to represent a scene composed of at least one object and the environment.

[0261] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: display first text data on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, determine a three-dimensional model and object texture information corresponding to at least one object; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data; drive the VR device or AR device to render and display the virtual scene.

[0262] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain first text data by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first text data, and the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, determine a three-dimensional model and object texture information corresponding to at least one object; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data; output the virtual scene by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the virtual scene.

[0263] According to an embodiment of the present application, first text data is obtained, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to at least one object are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information, to generate a virtual scene corresponding to the first text data, thereby improving the efficiency of generating virtual scenes. It is easy to notice that the first scene image and the environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image, the three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, thereby obtaining a virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for objects in related technologies.

[0264] It can be understood by those skilled in the art that Fig.21 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.21 The structure of the electronic device is not limited. For example, the computer terminal A may also include Fig.21 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.21 Different configurations are shown.

[0265] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0266] Example 12

[0267] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method for generating a virtual scene provided in the above embodiment 1.

[0268] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be located in any computer terminal in the AR / VR device terminal group in the AR / VR device network, or in any mobile terminal in the mobile terminal group.

[0269] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining first text data, wherein the first text data includes at least: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generating a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, determining a three-dimensional model and object texture information corresponding to at least one object; rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environmental texture information, to generate a virtual scene corresponding to the first text data.

[0270] Optionally, Fig.21 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.21 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0271] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating the virtual scene in the embodiment of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for generating the virtual scene. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0272] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain first text data, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generate a first scene image, and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, determine a three-dimensional model and object texture information corresponding to at least one object; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data.

[0273] Optionally, the processor may also execute program code of the following steps: using a first generation model to generate a first scene image and environmental texture information based on first text data; using a first generation model to generate a first scene image based on first text data, and de-analyzing the semantics of the first scene image to generate environmental texture information.

[0274] Optionally, the processor may also execute program code of the following steps: generating second text data based on the first scene image using the second generation model, wherein the second text data is used to represent text describing the first scene image; generating environmental texture information based on the second text data using the first generation model.

[0275] Optionally, the processor may also execute program code of the following steps: identifying at least one object in the first scene image to obtain a segmentation result of at least one object; based on the segmentation result, determining a three-dimensional model from at least one preset model; and generating object texture information based on the segmentation result and the first scene image.

[0276] Optionally, the processor may also execute program code for the following steps: performing depth estimation on the first scene image based on the segmentation result to obtain a depth estimation result; performing scene restoration on at least one object using the depth estimation result to obtain bounding box information corresponding to at least one object; obtaining a preset model that successfully matches the bounding box information from at least one preset model to obtain a three-dimensional model.

[0277] Optionally, the processor may also execute program code for the following steps: determining attribute information of at least one object based on the segmentation result and the first scene image; generating initial texture information corresponding to at least one object based on the attribute information of at least one object; and amplifying the initial texture information to obtain object texture information.

[0278] Optionally, the processor may also execute program code of the following steps: amplifying initial texture information using a super-resolution model to obtain object texture information.

[0279] Optionally, the processor may also execute program code for the following steps: generating a second scene image based on the second text data using the first generation model; stylizing the second scene image to obtain a stylized image; restoring at least one object in the second scene image to the stylized image to obtain a restored image; restoring the color of the second scene image to the restored image to obtain the first scene image.

[0280] Optionally, the processor may also execute program code of the following steps: segmenting the second scene image to obtain an image of at least one object and a segmentation result; and based on the segmentation result of the at least one object, overlaying the image of the at least one object onto the stylized image to obtain a restored image.

[0281] Optionally, the processor may also execute the following program code: obtaining a first channel value of a color channel of the second scene image, and obtaining a second channel value of a color channel of the restored image; and replacing the second channel value with the first channel value to obtain the first scene image.

[0282] Optionally, the processor may also execute the program code of the following steps: sampling a target video including a second scene image to obtain a sampled image, wherein the target video is generated based on the second text data using a first generation model; and stylizing the second scene image based on the sampled image using a diffusion model to obtain a stylized image.

[0283] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain first text data, wherein the first text data at least includes: a first text for describing multiple objects, and a second text for describing the environment in which the multiple objects are located, and the multiple objects include goods and a storage table on which the goods are placed; based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to represent the scene composed of the multiple objects and the environment; based on the first scene image, determine the three-dimensional models and object texture information corresponding to the multiple objects; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data.

[0284] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to an input instruction acting on the operation interface, display first text data on the operation interface, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; in response to a virtual scene generation instruction acting on the operation interface, display a virtual scene corresponding to the first text data on the operation interface, wherein the virtual scene is generated by rendering a three-dimensional model corresponding to at least one object based on object texture information corresponding to the at least one object, and rendering a preset model based on environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on a first scene image, the first scene image and the environmental texture information are generated based on the first text data, and the first scene image is used to represent a scene composed of at least one object and the environment.

[0285] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: display first text data on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, determine a three-dimensional model and object texture information corresponding to at least one object; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data; drive the VR device or AR device to render and display the virtual scene.

[0286] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain first text data by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first text data, and the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to characterize the scene composed of at least one object and the environment; based on the first scene image, determine a three-dimensional model and object texture information corresponding to at least one object; render the three-dimensional model based on the object texture information, and render the preset model based on the environmental texture information to generate a virtual scene corresponding to the first text data; output the virtual scene by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the virtual scene.

[0287] According to an embodiment of the present application, first text data is obtained, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing the environment in which the at least one object is located; based on the first text data, a first scene image and environmental texture information corresponding to the environment are generated, wherein the first scene image is used to characterize a scene composed of at least one object and the environment; based on the first scene image, a three-dimensional model and object texture information corresponding to at least one object are determined; the three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environmental texture information, to generate a virtual scene corresponding to the first text data, thereby improving the efficiency of generating virtual scenes. It is easy to notice that the first scene image and the environmental texture information can be directly generated according to the first text data associated with the object, and the object texture information and the three-dimensional model can be determined according to the first scene image, the three-dimensional model of the object can be rendered according to the object texture information, and the preset model of the environment can be rendered based on the environmental texture information, thereby obtaining a virtual scene corresponding to the first text data. Manual labor only needs to provide the above-mentioned first text data to generate the virtual scene corresponding to the first text data, which can greatly reduce labor costs and improve the effect of generating virtual scenes, thereby solving the technical problem of high production costs of promotional materials for commodities in related technologies.

[0288] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0289] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0290] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0291] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0292] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0293] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0294] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a virtual scene, characterized in that: include: Acquire first text data, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located; Based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to represent a scene formed by the at least one object and the environment; Determining, based on the first scene image, a three-dimensional model and object texture information corresponding to the at least one object; The three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environment texture information to generate a virtual scene corresponding to the first text data.

2. The method according to claim 1, characterized in that Generating a first scene image and environment texture information corresponding to the environment based on the first text data includes at least one of the following: Generate the first scene image and the environment texture information based on the first text data using a first generation model; The first scene image is generated based on the first text data using a first generation model, and the semantics of the first scene image is reversely parsed to generate the environmental texture information.

3. The method according to claim 2, characterized in that De-analyzing the semantics of the first scene image to generate the environment texture information includes: Generate second text data based on the first scene image using a second generation model, wherein the second text data is used to represent text describing the first scene image; The environment texture information is generated based on the second text data using the first generation model.

4. The method according to claim 1, characterized in that: Determining, based on the first scene image, a three-dimensional model and object texture information corresponding to the at least one object, including: identifying the at least one object in the first scene image to obtain a segmentation result of the at least one object; Based on the segmentation result, determining the three-dimensional model from at least one preset model; The object texture information is generated based on the segmentation result and the first scene image.

5. The method according to claim 4, characterized in that Based on the segmentation result, determining the three-dimensional model from at least one preset model includes: Performing depth estimation on the first scene image based on the segmentation result to obtain a depth estimation result; Performing scene restoration on the at least one object using the depth estimation result to obtain bounding box information corresponding to the at least one object; A preset model that successfully matches the bounding box information is acquired from the at least one preset model to obtain the three-dimensional model.

6. The method according to claim 4, characterized in that Generating the object texture information based on the segmentation result and the first scene image includes: Determining attribute information of the at least one object based on the segmentation result and the first scene image; Based on the attribute information of the at least one object, generating initial texture information corresponding to the at least one object; The initial texture information is amplified to obtain the object texture information.

7. The method according to claim 1, characterized in that Generating a first scene image based on the first text data includes: Generate a second scene image based on the second text data using the first generation model; Stylizing the second scene image to obtain a stylized image; Restoring the at least one object in the second scene image onto the stylized image to obtain a restored image; The color of the second scene image is restored to the restored image to obtain the first scene image.

8. The method according to claim 7, characterized in that Restoring the at least one object in the second scene image to the stylized image to obtain a restored image includes: Segmenting the second scene image to obtain an image of the at least one object and a segmentation result; Based on the segmentation result of the at least one object, the image of the at least one object is overlaid on the stylized image to obtain the restored image.

9. The method according to claim 7, characterized in that: Restoring the color of the second scene image to the restored image to obtain the first scene image includes: Acquire a first channel value of a color channel of the second scene image, and acquire a second channel value of a color channel of the restored image; The second channel value is replaced by the first channel value to obtain the first scene image.

10. A method for generating a virtual scene, characterized in that: include: Acquire first text data, wherein the first text data at least includes: a first text for describing a plurality of objects, and a second text for describing an environment in which the plurality of objects are located, wherein the plurality of objects include commodities and a storage table on which the commodities are placed; Based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to represent a scene formed by the multiple objects and the environment; Based on the first scene image, determining three-dimensional models and object texture information corresponding to the plurality of objects; The three-dimensional model is rendered based on the object texture information, and the preset model is rendered based on the environment texture information to generate a virtual scene corresponding to the first text data.

11. A method for generating a virtual scene, characterized in that: include: In response to an input instruction acting on an operation interface, first text data is displayed on the operation interface, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located; In response to a virtual scene generation instruction acting on the operation interface, a virtual scene corresponding to the first text data is displayed on the operation interface, wherein the virtual scene is generated by rendering a three-dimensional model corresponding to the at least one object based on the object texture information corresponding to the at least one object, and rendering a preset model based on the environmental texture information corresponding to the environment, the object texture information and the three-dimensional model are determined based on a first scene image, the first scene image and the environmental texture information are generated based on the first text data, and the first scene image is used to represent a scene constituted by the at least one object and the environment.

12. A method for generating a virtual scene, characterized in that: include: Displaying first text data on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the first text data at least includes: a first text for describing at least one object, and a second text for describing an environment in which the at least one object is located; Based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to represent a scene formed by the at least one object and the environment; Determining, based on the first scene image, a three-dimensional model and object texture information corresponding to the at least one object; Rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information to generate a virtual scene corresponding to the first text data; The VR device or the AR device is driven to render and display the virtual scene.

13. An image processing method, characterized in that: include: Acquire first text data by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the first text data, and the first text data includes at least: a first text for describing at least one object, and a second text for describing an environment where the at least one object is located; Based on the first text data, generate a first scene image and environmental texture information corresponding to the environment, wherein the first scene image is used to represent a scene formed by the at least one object and the environment; Determining, based on the first scene image, a three-dimensional model and object texture information corresponding to the at least one object; Rendering the three-dimensional model based on the object texture information, and rendering the preset model based on the environment texture information to generate a virtual scene corresponding to the first text data; The virtual scene is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the virtual scene.

14. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 13 when running.

Images