Article display method and device, equipment, medium and product

Abstract

The invention discloses an article display method, device and equipment, a medium and a product, and relates to the technical field of universe. The method comprises the steps of obtaining a to-be-processed image of a target object at any view angle and a description text corresponding to the target object; according to the to-be-processed image, the description text, a pre-trained text graph model and a preset picture decoder, obtaining a single-view-angle picture corresponding to the to-be-processed image; generating a three-dimensional grid model according to the single-view-angle picture; based on a preset texture generation model and a description text, performing texture addition on the three-dimensional grid model to obtain a target three-dimensional model of the target object; and in response to a viewing operation of any virtual avatar in a preset meta universe exhibition hall on the target article, rendering and displaying the target three-dimensional model. According to the technical scheme, the generation efficiency of the three-dimensional model of the article is improved, and an exhibition mode of the universe exhibition hall is provided.

Description

Technical Field

[0001] This application relates to the field of metaverse technology, and more particularly to the field of image processing and 3D model construction technology, specifically to a method, apparatus, device, medium, and product for displaying objects. Background Technology

[0002] With societal progress and the continuous development of internet-related technologies, more and more users are using various terminals such as computers or mobile phones to search for and browse information online. Online shopping, due to its convenience and efficiency, has been widely praised and favored by users. However, many products are not intuitively presented through webpage text descriptions and image displays alone, which can easily mislead users. Therefore, creating 3D models of products has become a key research focus for professionals in related fields.

[0003] Currently, the mainstream method for technicians to use software for manual 3D modeling is to build 3D models manually. However, manually building 3D models consumes a lot of manpower, and collecting product details is too costly and inefficient. Furthermore, simply displaying the product through a webpage cannot allow users to understand its 3D state from a 2D interface. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and product for displaying objects, to improve the efficiency of generating three-dimensional models of objects, and to provide an exhibition mode of a metaverse exhibition hall.

[0005] According to one aspect of this application, a method for displaying an item is provided, comprising:

[0006] Acquire images of the target item from any perspective, along with the corresponding descriptive text.

[0007] Based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder, a single-view image corresponding to the image to be processed is obtained.

[0008] Generate a 3D mesh model from a single-view image;

[0009] Based on the preset texture generation model and descriptive text, textures are added to the 3D mesh model to obtain the target 3D model of the target object.

[0010] In response to any virtual avatar in the preset metaverse exhibition hall viewing the target item, render and display the target's 3D model.

[0011] According to another aspect of this application, an article display device is provided, comprising:

[0012] The image and text acquisition module acquires an image of the target item from any perspective, as well as the corresponding descriptive text of the target item.

[0013] The single-view determination module obtains the single-view image corresponding to the image to be processed based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder.

[0014] The mesh model generation module generates a 3D mesh model based on a single-view image.

[0015] The target model generation module adds textures to the 3D mesh model based on a preset texture generation model and descriptive text to obtain the target 3D model of the target item.

[0016] The Metaverse Display Module responds to any virtual avatar in the preset Metaverse Exhibition Hall viewing the target item by rendering and displaying the target's 3D model.

[0017] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the article display method according to any embodiment of this application.

[0021] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the article display method described in any embodiment of this application.

[0022] According to another aspect of this application, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the article display method according to any embodiment of this application.

[0023] In the technical solution of this application embodiment, based on the image to be processed and the descriptive text of the target item, a single-view image corresponding to the image to be processed is obtained based on a pre-trained text-based image model and a preset image decoder. This adds information related to the appearance and material properties of the target item to the ordinary image, thereby improving the accuracy of the generated model. Based on the single-view image, a three-dimensional mesh model is generated, which can quickly generate the basic framework of the model, greatly improving the speed and efficiency of model generation before display, and helping to improve the efficiency of display. Based on the preset texture generation model and descriptive text, textures are added to the three-dimensional mesh model to obtain the target three-dimensional model of the target item. Textures can be generated for the three-dimensional mesh based on the descriptive text, which is more in line with the original appearance of the target item and improves the accuracy of the generated model. Finally, the generated model is rendered and displayed through the Metaverse Exhibition Hall, providing a display channel within the Metaverse and providing users with a reliable way to obtain information about the target item.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of an item display method according to Embodiment 1 of this application;

[0027] Figure 2 This is a flowchart of an article display method applicable to Embodiment 2 of this application;

[0028] Figure 3 This is a structural schematic diagram of an item display device according to Embodiment 3 of this application;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the item display method of the embodiments of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This application provides a flowchart of an item display method according to Embodiment 1. This embodiment is applicable to situations where items are displayed to users through a metaverse. This method can be executed by an item display device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110. Obtain an image of the target item from any perspective, as well as the corresponding descriptive text of the target item.

[0035] The target item can be any item that needs to be displayed. In this embodiment, taking the metaverse exhibition hall of a bank providing users with digital space as an example, the bank's physical items such as goods and gifts are displayed in three dimensions to provide users with remote viewing, browsing and purchasing services.

[0036] The image to be processed can be a photograph taken from any perspective of the target object, or a frame from a video, etc. The method of acquiring the image to be processed is not limited in this embodiment. The descriptive text corresponding to the target object can be a combination of text describing the various basic attributes and characteristics of the target object itself, such as appearance parameters (length, width, height, etc.), material information, shape, etc. The specific content of the descriptive text is not limited in this embodiment.

[0037] For example, if a bank wants to display its precious metal products in a pre-built metaverse showroom, it first needs to photograph the precious metal products and provide a product description.

[0038] S120. Based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder, obtain a single-view image corresponding to the image to be processed.

[0039] The text-to-image model can be a pre-trained machine learning image processing model trained on a large training set of text and images, enabling it to convert text into image descriptions. The image decoder is a tool that decodes the image descriptions to generate an image. This application does not limit the training process of the text-to-image model or the selection of the image decoder.

[0040] A single-view image can be an image based on the image to be processed, sharing the same viewpoint, and obtained by generating a text-based image with the assistance of descriptive text. In other words, the single-view image contains the appearance features of the target object included in the descriptive text. Understandingly, the descriptive text can first be converted into text features. The text-based image model uses the image to be processed as its base, taking both the text features and the image as input. The model outputs an image encoding corresponding to the single-view image, which contains text features. This image encoding is then input into an image decoder for decoding, yielding the single-view image.

[0041] Of course, the embodiments of this application do not limit the method of transforming text features into descriptions.

[0042] S130. Generate a 3D mesh model based on a single-view image.

[0043] In this context, a 3D mesh model serves as the foundation for simulating a 3D solid model. It's understood that in the actual construction of a 3D model, a mesh model must first be built, and a simulated 3D model resembling the solid object is generated based on the 3D framework composed of the mesh. In existing technologies, the mesh generation process relies heavily on manual labor, resulting in low efficiency. In this embodiment, a 3D network model is generated based on a single-view image. For example, a pre-trained multi-view expansion model and a graph-generated 3D model are used to expand the single-view image, and a 3D mesh is generated based on the expanded image. Of course, this embodiment does not limit the training and use of the multi-view expansion model and the graph-generated 3D model.

[0044] S140. Based on the preset texture generation model and description text, add texture to the three-dimensional mesh model to obtain the target three-dimensional model of the target item.

[0045] The texture generation model is used to generate textures on the surface of a 3D mesh. The descriptive text contains information about the appearance and material of the target object, which helps the texture generation model add textures to the 3D mesh model that match the target object. Based on the surface texture of the target object described in the descriptive text, textures are added to the 3D mesh model to obtain the target 3D model, which is a simulation model that can be used for display and has the same appearance as the target object.

[0046] Of course, the training and use process of the texture generation model is not limited in this embodiment.

[0047] S150: In response to any virtual avatar in the preset metaverse exhibition hall viewing the target item, render and display the target's 3D model.

[0048] The Metaverse Showroom is a digital space built using Metaverse technology for product display and transactions. Users can connect to the Metaverse Showroom through different terminal devices, enter the showroom as their own virtual avatar, browse the 3D models of the displayed products, and make purchases. Therefore, when any user's virtual avatar selects to view a target item in the Metaverse Showroom, the target 3D model corresponding to the parent item generated in the previous steps is rendered and displayed to the user.

[0049] For example, bank staff take photos of precious metal products that need to be displayed online, and describe the appearance, material, and other information of the products in text. The descriptive text and photos are put into a text-based image model, which outputs an image encoding containing the descriptive text. The image is then decoded into a single-view image by an image decoder. A three-dimensional mesh frame is generated based on this single-view image, and textures are generated on the three-dimensional mesh frame using the descriptive text. This results in a simulated target three-dimensional model that users can view in the Metaverse exhibition hall.

[0050] In the technical solution of this application embodiment, based on the image to be processed and the descriptive text of the target item, a single-view image corresponding to the image to be processed is obtained based on a pre-trained text-based image model and a preset image decoder. This adds information related to the appearance and material properties of the target item to the ordinary image, thereby improving the accuracy of the generated model. Based on the single-view image, a three-dimensional mesh model is generated, which can quickly generate the basic framework of the model, greatly improving the speed and efficiency of model generation before display, and helping to improve the efficiency of display. Based on the preset texture generation model and descriptive text, textures are added to the three-dimensional mesh model to obtain the target three-dimensional model of the target item. Textures can be generated for the three-dimensional mesh based on the descriptive text, which is more in line with the original appearance of the target item and improves the accuracy of the generated model. Finally, the generated model is rendered and displayed through the Metaverse Exhibition Hall, providing a display channel within the Metaverse and providing users with a reliable way to obtain information about the target item.

[0051] Example 2

[0052] Figure 2 This is a flowchart illustrating an item display method according to Embodiment 2 of this application. This embodiment further refines the generation operation of the three-dimensional network model based on the foregoing embodiments. Figure 2 As shown, the method includes:

[0053] S210. Obtain an image of the target item from any perspective, as well as the corresponding descriptive text of the target item.

[0054] S220. Based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder, obtain a single-view image corresponding to the image to be processed.

[0055] S230. Input the single-view image into the preset multi-view diffusion model to obtain six images from different perspectives to be merged, which expand the single-view image into multiple views.

[0056] The multi-view expansion model can be any machine learning model that expands a single view into multiple perspectives. Using a multi-view expansion model, a single-view image is used as a base, and images from other perspectives besides that single view are expanded, typically resulting in six images from different perspectives as the images to be merged. These six different perspectives are generally the front, back, left, right, top, and bottom of the target object. If a single-view image matches any one of these six perspectives, it can be used as one of the images to be merged; conversely, if a single-view image does not match any of the six perspectives, it can be directly expanded to obtain six images from different perspectives, which will then be used as the images to be merged. Of course, this application embodiment can employ any multi-view expansion model from related technologies, and this application embodiment does not limit the type of multi-view expansion model, nor the training and usage process.

[0057] S240. Merge and stitch the images to be merged into a single image, and input the single image into a preset image processing model to obtain the image feature vector sequence corresponding to the target item.

[0058] As the name suggests, a single image can be a single image, meaning it's a composite image created by stitching together six different perspective images generated in the preceding steps. The stitching can be done according to pre-set positions, arranging the images from different perspectives at predetermined locations. The stitched single image is then input into an image processing model, which outputs an image feature vector sequence. This image feature vector sequence can be the 3D information of the target object obtained from analyzing images from multiple different perspectives. In essence, by stitching six images from different perspectives into a single image at predetermined positions, the image processing model analyzes the spatial relationships between the target object at different angles from these perspectives, thereby outputting a vector sequence carrying this spatial relationship information, i.e., the image feature vector sequence.

[0059] Of course, the image processing model can adopt any of the related technologies. The embodiments of this application do not limit the type of the model, as well as the training and usage process.

[0060] S250. Input the image feature vector sequence into the preset three-dimensional reconstruction model to obtain a three-dimensional mesh model.

[0061] Among them, the three-dimensional reconstruction model can be a data processing model generated by the basic architecture of the three-dimensional model based on image features. Since the image feature vector sequence contains the spatial relationship between the various structures of the target object, a basic three-dimensional architecture model is output based on the image feature vector sequence. The architecture is built by a mesh, which is the three-dimensional mesh model.

[0062] Of course, the three-dimensional reconstruction model can be any of the related technologies. The embodiments of this application do not limit the type of the model, as well as the training and usage process.

[0063] S260. Based on the preset texture generation model and descriptive text, add texture to the 3D mesh model to obtain the target 3D model of the target item.

[0064] S270: In response to any virtual avatar in the preset metaverse exhibition hall viewing the target item, render and display the target's 3D model.

[0065] In addition, the Metaverse Exhibition Hall supports purchasing transactions. Users can participate in the purchase of any product displayed in the Metaverse Exhibition Hall through their own virtual avatars, and online purchases will trigger offline logistics and delivery.

[0066] In the technical solution of this application embodiment, a text-derived image is generated based on the text features of the image to be processed and the descriptive text, adding more appearance detail information to the single-view image. This is then expanded using a multi-view expansion model, resulting in six different view images that are stitched together into a single image for image processing. This produces an image feature vector sequence containing the spatial relationships between the various structures of the target object, and a target mesh model is generated based on this sequence to form a frame structure. This process enables multi-view expansion based on a single-view image and utilizes the model to analyze the spatial relationships between different structural positions of the target object in different views, thereby facilitating the rapid and efficient generation of a 3D mesh and improving the efficiency of 3D model generation.

[0067] In one optional implementation, the step S220 of obtaining a single-view image corresponding to the image to be processed based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder may include: inputting the descriptive text into a preset text encoder so that the text encoder outputs text features corresponding to the descriptive text; inputting the text features and the image to be processed into the text-to-image model to obtain an image information matrix with semantic information corresponding to the descriptive text; and inputting the image information matrix into the image decoder to obtain a single-view image.

[0068] The text encoder can be a tool that converts text into encoding. The text features can be the encoded information output by the text encoder, which stores the content of the target item described in the descriptive text. In this embodiment, the descriptive text is encoded by the text encoder to obtain the corresponding text features. These text features contain semantic information describing the appearance of the target item, and exist in encoded form.

[0069] The image information matrix can be the output of a text-based image model, which integrates semantic information from text features and image information from the image to be processed. In this embodiment, the text features encoded by the text encoder and the image to be processed are input together into a pre-trained text-based image model, so that the model outputs an image information matrix. This image information matrix includes image information from the image to be processed, as well as semantic information corresponding to the descriptive text. In other words, the image information matrix includes not only information from the initial photograph of the product, but also descriptive information recording the product's appearance. Understandably, such an image information matrix can help to quickly and efficiently simulate the product's appearance. Decoding the image information matrix using an image decoder yields a single-view image. Of course, this single-view image differs from the initial image to be processed because it incorporates more appearance details based on the semantic information of the descriptive text.

[0070] In this embodiment, by performing text-to-image modeling operations on the text features corresponding to the descriptive text and the image to be processed, an image information matrix that simultaneously possesses image information and semantic information is generated. This provides a more reliable basis for the subsequent generation of 3D models and helps to improve the efficiency of 3D model generation and display.

[0071] In one optional implementation, the step of inputting a single image into a preset image processing model in S240 to obtain an image feature vector sequence corresponding to the target item may include: dividing the single image into image blocks of a preset size; determining the distance dependency between each image block based on the self-attention mechanism in the image processing model; and generating an image feature vector sequence based on the distance dependency.

[0072] The image contains six different viewpoints, but processing these viewpoints within a single image improves efficiency and accuracy in analyzing spatial relationships. First, the image is segmented into image patches of a preset size. The relationships between these patches are analyzed using a self-attention mechanism within the image processing model, yielding distance dependencies between them. These distance dependencies represent the spatial distances between parts of a target object displayed in one patch and parts displayed in other patches. This distance dependency helps the model determine the spatial positional relationships between parts displayed in different patches. Therefore, this distance dependency is converted into an image feature vector sequence for output.

[0073] In the above implementation, the distance dependency between image patches is determined based on the self-attention mechanism of the image processing model, which indirectly determines the spatial positional relationship between different parts of the target object. Therefore, converting this information into feature vector output helps to construct the three-dimensional model based on these feature vectors and improves the efficiency of model construction.

[0074] In one optional implementation, the step S260, which involves adding textures to a 3D mesh model based on a preset texture generation model and descriptive text to obtain a target 3D model of the target item, may include: generating texture maps for each mesh in the 3D mesh model in the texture generation model according to the descriptive text; and fusing the texture maps and the 3D mesh model to obtain the target 3D model.

[0075] The texture generation model can be any tool that generates texture information for a 3D model based on its appearance description. Since the descriptive text describes the appearance, material, and other information of the target object, this descriptive text is input into the texture generation model so that the model can generate a texture map based on this descriptive text. In fact, this texture map can be a 3D stereoscopic texture of the outer surface of the 3D mesh model. By attaching this texture map to the previously generated 3D mesh model in space, the target 3D model can be obtained, which is the model that needs to be rendered and displayed in the end.

[0076] In the above embodiments, generating texture maps according to the description text can make the appearance of the final generated target 3D model more closely match the appearance and material of the target object, thereby improving the visual appeal of the generated 3D model.

[0077] In a further optional implementation, generating texture maps for each mesh in the 3D mesh model in the texture generation model based on the description text may include: extracting depth images of at least two views of the 3D mesh model; using the description text as semantic guidance information, converting the depth images into texture maps in the texture generation model.

[0078] In this embodiment, the depth image can be an image representing the depth information of a 3D mesh model acquired from different perspectives. That is, each pixel of the 3D mesh in the depth image contains depth information. This depth information can determine the coordinate position of each pixel on the 3D mesh in 3D space during rendering. Depth images from different perspectives can determine the coordinate position of each point on the 3D mesh in 3D space from different angles. Preferably, depth images from four or more perspectives are selected for processing. Of course, the acquisition of the depth image can be based on the coordinate system information in the modeling space of the 3D mesh model, and this embodiment does not limit this.

[0079] Semantic guidance information can be used as input to the texture generation model to assist in generating texture details. This involves inputting descriptive text as semantic guidance information into the texture generation model, enabling the model to understand the appearance and material properties of the target object and generate textures for its surface. Simultaneously, the depth image is also one of the inputs to this texture generation model. Based on the depth image, texture information is generated according to the spatial location of different pixels, converting the depth image into a texture map. This gives the 3D mesh an external texture, thus forming the target 3D model.

[0080] In the above embodiments, texture maps are generated based on the depth image of the three-dimensional mesh model so that the texture maps fit the structure of the mesh, thereby ensuring that the generated three-dimensional model is more vivid and reasonable, and ensuring the visual appeal of the generated target three-dimensional model.

[0081] Example 3

[0082] Figure 3 This is a structural schematic diagram of an item display device provided in Embodiment 3 of this application. Figure 3 As shown, the device 300 includes:

[0083] The image and text acquisition module 310 acquires an image of the target item from any perspective, as well as the descriptive text corresponding to the target item.

[0084] The single-view determination module 320 obtains the single-view image corresponding to the image to be processed based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder.

[0085] Mesh model generation module 330 generates a 3D mesh model based on a single-view image;

[0086] The target model generation module 340 adds texture to the 3D mesh model based on the preset texture generation model and description text to obtain the target 3D model of the target item.

[0087] The Metaverse Display Module 350 responds to any virtual avatar in the preset Metaverse Exhibition Hall viewing the target item by rendering and displaying the target's 3D model.

[0088] In the technical solution of this application embodiment, based on the image to be processed and the descriptive text of the target item, a single-view image corresponding to the image to be processed is obtained based on a pre-trained text-based image model and a preset image decoder. This adds information related to the appearance and material properties of the target item to the ordinary image, thereby improving the accuracy of the generated model. Based on the single-view image, a three-dimensional mesh model is generated, which can quickly generate the basic framework of the model, greatly improving the speed and efficiency of model generation before display, and helping to improve the efficiency of display. Based on the preset texture generation model and descriptive text, textures are added to the three-dimensional mesh model to obtain the target three-dimensional model of the target item. Textures can be generated for the three-dimensional mesh based on the descriptive text, which is more in line with the original appearance of the target item and improves the accuracy of the generated model. Finally, the generated model is rendered and displayed through the Metaverse Exhibition Hall, providing a display channel within the Metaverse and providing users with a reliable way to obtain information about the target item.

[0089] In one alternative implementation, the single-viewpoint determination module 320 may include:

[0090] The text feature output unit is used to input the descriptive text into a preset text encoder so that the text encoder outputs the text features corresponding to the descriptive text.

[0091] The image matrix generation unit is used to input text features and the image to be processed into the text-generated image model to obtain an image information matrix that describes the semantic information corresponding to the text.

[0092] The single-viewpoint generation unit is used to input the image information matrix into the image decoder to obtain a single-viewpoint image.

[0093] In one alternative implementation, the mesh model generation module 330 may include:

[0094] The image to be merged determination unit is used to input a single-view image into a preset multi-view diffusion model to obtain six images from different perspectives to be merged, which expand the single-view image into multiple views.

[0095] The feature vector sequence determination unit is used to merge and stitch together each image to be merged into a single image, and input the single image into a preset image processing model to obtain the image feature vector sequence corresponding to the target item;

[0096] The three-dimensional mesh model determination unit is used to input the image feature vector sequence into the preset three-dimensional reconstruction model to obtain the three-dimensional mesh model.

[0097] In one optional implementation, the feature vector sequence determination unit may include:

[0098] The image block segmentation subunit is used to segment a single image into image blocks of a preset size;

[0099] The dependency determination subunit is used to determine the distance dependencies between image patches based on the self-attention mechanism in the image processing model;

[0100] The feature vector sequence generation subunit is used to generate image feature vector sequences based on distance dependencies.

[0101] In one alternative implementation, the target model generation module 340 may include:

[0102] The texture mapping generation unit is used to generate texture maps for each mesh in the 3D mesh model based on the description text.

[0103] The target model determination unit is used to fuse texture maps and 3D network models to obtain the target 3D model.

[0104] In one alternative implementation, the texture mapping generation unit may include:

[0105] A depth image extraction subunit is used to extract depth images of at least two viewpoints of a 3D mesh model;

[0106] The texture generation subunit is used to convert depth images into texture maps in the texture generation model, using descriptive text as semantic guidance information.

[0107] The item display device provided in this application embodiment can execute the item display method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each item display method.

[0108] Example 4

[0109] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0110] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0111] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the item display method.

[0113] In some embodiments, the item display method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the item display method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the item display method by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0120] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the item display method provided in any embodiment of this application. This program product shares the same inventive concept as the item display methods disclosed in the embodiments of this application, and therefore will not be described in detail here.

[0121] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for displaying items, characterized in that, include: Acquire an image of the target item from any viewpoint, and the corresponding descriptive text of the target item; Based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder, a single-view image corresponding to the image to be processed is obtained; Generate a 3D mesh model based on the single-view image; Based on the preset texture generation model and the description text, textures are added to the three-dimensional mesh model to obtain the target three-dimensional model of the target item; In response to any virtual avatar in the preset metaverse exhibition hall viewing the target item, the target 3D model is rendered and displayed.

2. The method according to claim 1, characterized in that, The step of obtaining a single-view image corresponding to the image to be processed based on the image to be processed, the descriptive text, the pre-trained text-to-image model, and the preset image decoder includes: The descriptive text is input into a preset text encoder so that the text encoder outputs the text features corresponding to the descriptive text; The text features and the image to be processed are input into the text-to-image model to obtain an image information matrix with semantic information corresponding to the descriptive text; The image information matrix is ​​input into the image decoder to obtain the single-view image.

3. The method according to claim 1, characterized in that, The step of generating a 3D mesh model based on the single-view image includes: The single-view image is input into a preset multi-view diffusion model to obtain six images from different perspectives to be merged, which are multi-view expanded from the single-view image. The images to be merged are combined and stitched into a single image, and the single image is input into a preset image processing model to obtain the image feature vector sequence corresponding to the target item. The image feature vector sequence is input into a preset three-dimensional reconstruction model to obtain the three-dimensional mesh model.

4. The method according to claim 3, characterized in that, The step of inputting the single image into a preset image processing model to obtain the image feature vector sequence corresponding to the target item includes: The single image is divided into image blocks of a preset size; Based on the self-attention mechanism in the image processing model, the distance dependency between each image patch is determined; The image feature vector sequence is generated based on the distance dependency.

5. The method according to claim 1, characterized in that, The process of adding textures to the 3D mesh model based on a preset texture generation model and the descriptive text to obtain the target 3D model of the target item includes: Based on the description text, a texture map for each mesh in the three-dimensional mesh model is generated in the texture generation model; The texture map and the 3D network model are fused to obtain the target 3D model.

6. The method according to claim 5, characterized in that, The step of generating texture maps for each mesh in the 3D mesh model based on the description text includes: Extract depth images from at least two viewpoints of the 3D mesh model; The descriptive text is used as semantic guidance information in the texture generation model to convert the depth image into the texture map.

7. An item display device, characterized in that, include: The image text acquisition module is used to acquire an image of the target item from any perspective, as well as the descriptive text corresponding to the target item. A single-view determination module is used to obtain a single-view image corresponding to the image to be processed based on the image to be processed, the descriptive text, a pre-trained text-to-image model, and a preset image decoder. The mesh model generation module is used to generate a three-dimensional mesh model based on the single-view image. The target model generation module is used to add textures to the three-dimensional mesh model based on a preset texture generation model and the description text to obtain the target three-dimensional model of the target item. The Metaverse Display Module is used to render and display the target 3D model in response to any virtual avatar's viewing operation of the target item in the preset Metaverse Exhibition Hall.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the article display method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the article display method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the item display method according to any one of claims 1-6.

Images