Lightweight panorama generation method and device based on AIGC

Abstract

The invention provides a light-weight panorama generation method and device based on AIGC. The method comprises the steps that an initial instruction input by a user is received, and a 360-degree panorama is generated; the 360-degree panoramic image is split into six face images of a hexahedron structure; extracting visual elements of each surface in the hexahedral graph, generating corresponding natural language descriptions, and splicing the natural language descriptions into a complete description of a panorama scene; calculating the semantic similarity between the initial instruction of the user and the complete description of the panorama scene; identifying important elements of the six face images of the hexahedron structure and positions of the important elements in the images, and generating anchor point data containing labels and coordinates; constructing a CSS3D cube structure and loading images of all surfaces; adding an interactive component in the corresponding image area according to the anchor point data; and binding a preset or self-defined interaction event for the interactive component. According to the invention, rapid and low-cost generation of the panorama is realized, and efficient display and diversified interaction are achieved in the VR interaction scene, so that the user experience is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of panoramic image generation, and particularly relates to a lightweight panoramic image generation method and system based on AIGC. BACKGROUND

[0002] In recent years, artificial intelligence generated content (AIGC) technology has made significant breakthroughs in the fields of image generation, 3D modeling, and virtual scene construction. Core algorithms such as diffusion models and generative adversarial networks have achieved end-to-end generation capabilities from text descriptions to high-quality 2D / 3D scenes. For example, Skybox AI can quickly generate panoramic image base materials, while algorithms such as NeRF (Neural Radiance Field) further support the automatic reconstruction of 3D spatial structures. At the same time, the rise of multi-modal large language models such as GPT and Claude provides a new paradigm for natural language-driven scene design and interaction logic generation. These developments have laid the algorithmic foundation for the deep integration of AIGC and VR technology, making it possible to automate the generation chain of "text → panoramic scene → interaction logic".

[0003] In existing technologies, the generation and interaction design of VR panoramic images highly depend on manual processes, which have significant bottlenecks such as low generation efficiency, rigid interaction design, and limited creative iteration. Low generation efficiency is reflected in the fact that professional-level panoramic image production relies on multi-lens camera arrays or manual 3D modeling, which is costly and limited by physical conditions (such as light and spatial complexity). Post-processing requires manual repair of distortions and defects using tools such as Photoshop and Maya, resulting in long production times and high costs for a single scene. Rigid interaction design is reflected in the fact that the triggering logic of existing VR interaction scenes relies on developer pre-set rules, and element recognition relies on manual annotation (such as hotspots and operable objects), leading to low interaction response accuracy and poor adaptability, making it difficult to support real-time updates of dynamic scenes. Limited creative iteration is reflected in the fact that during the process from concept design to scene implementation, manual solutions cannot quickly generate multiple versions of prototypes, and there is a lack of intelligent creative assistance tools, leading to the solidification of design inspiration and the slow response to user individualized needs.

[0004] Although AIGC technology has made some progress in image generation, how to efficiently adapt AIGC-generated panoramic images to VR interaction scenes to achieve lightweight and convenient panoramic image generation and interaction remains a key problem to be solved, which is of great significance for promoting the popularization and development of VR technology in more fields. SUMMARY

[0005] In order to realize the rapid and low-cost generation of panoramic pictures, achieve efficient display and diversified interaction in the VR interaction scene, and significantly improve the user experience, effectively reduce the technical threshold and cost investment of panoramic picture generation and VR interaction, the present application provides a lightweight panoramic picture generation method and system based on AIGC.

[0006] To solve the above technical problems, the present application provides the following technical solutions: A lightweight panoramic picture generation method based on AIGC, comprising the following steps: Step 1, receiving the initial instruction input by the user, and generating a 360° panoramic picture based on the initial instruction; Step 2, splitting the 360° panoramic picture into six face pictures of a hexahedral structure; Step 3, extracting the visual elements of each face in the hexahedral picture, generating the corresponding natural language description, and splicing into the complete description of the panoramic picture scene; Step 4, calculating the semantic similarity between the user's initial instruction and the complete description of the panoramic picture scene, and if it is lower than the set threshold, re-generating the panoramic picture; Step 5, identifying the important elements of the six face pictures of the hexahedral structure and their positions in the image, and generating anchor point data containing labels and coordinates; Step 6, constructing a CSS3D cube structure based on the six face pictures of the hexahedral structure, loading each face image to display the 360° panoramic scene; Step 7, adding interactive components to the corresponding image area according to the anchor point data; binding the interactive event of the interactive components.

[0007] Further, the initial instruction includes a natural language instruction or a sketch description.

[0008] Further, the step 1 is realized by using the AIGC technology based on the diffusion model, specifically, the Skybox AI generates a panoramic picture according to the user input.

[0009] Further, the step 2 divides the 360° panoramic picture into images in the front, back, left, right, up and down directions through spatial mapping and projection transformation, and converts them into a format compatible with CSS3D.

[0010] Further, the step 3 uses a multi-modal large language model to analyze the hexahedral picture and output structured description text of each face.

[0011] Further, the step 4 uses a text embedding model to convert the text into a vector, and calculates the similarity between the user's instruction and the complete description of the panoramic picture scene through cosine similarity.

[0012] Further, the step 5 adopts a multi-modal large model to identify key elements in the image and output a list of (label, coordinate) tuples.

[0013] Further, the interactive component includes at least one of a button, a hotspot icon, or a highlighted area.

[0014] Further, the interaction event includes at least one of clicking a pop-up information box, page jumping, voice playing, and loading product details, and supports user-defined event logic through JavaScript.

[0015] In another aspect, the present application provides a lightweight panoramic image generation device based on AIGC, comprising: An input and generation module is configured to receive an initial instruction input by a user and generate a 360° panoramic image based on the initial instruction; An image splitting module is configured to split the 360° panoramic image into six face images of a hexahedral structure; An image element extraction module is configured to extract visual elements of each face of the hexahedral image, generate corresponding natural language descriptions, splice them into a complete description of the panoramic image scene, and perform a similarity judgment; An image anchor point acquisition module is configured to identify important elements of the six face images of the hexahedral structure and their positions in the image, and generate anchor point data containing labels and coordinates; A front-end display module is configured to construct a CSS3D cube structure based on the six face images of the hexahedral structure, load each face image to display a 360° panoramic scene; An interaction feedback module is configured to add an interactive component to a corresponding image area according to the anchor point data, and bind a preset or customized interaction event to the interactive component; A storage module is configured to store the panoramic image, anchor point data, interactive component, and event template.

[0016] Compared with the prior art, the present application has the following beneficial effects: High-efficiency generation of 360° panoramic pictures: With the help of AIGC technology, users can quickly obtain 360° panoramic pictures through simple natural language input, without relying on professional equipment and complex modeling processes, greatly improving the generation efficiency of 360° panoramic pictures and significantly reducing costs. Compared with traditional shooting methods, it saves the cost of equipment purchase and shooting time; compared with modeling software production, it reduces manpower and time investment. Strong compatibility: The format of the panoramic picture after splitting is perfectly compatible with CSS3D, which can be quickly displayed on the front end without additional conversion and adaptation work, improving the compatibility and universality of the system. Whether on a PC browser or various mobile VR devices, 360° panoramic pictures can be smoothly displayed, expanding the application range. Precise interaction: Through multi-modal large model recognition to generate interactive anchor point data, it can accurately identify interactive elements in 360° panoramic pictures, realize various interactive logic, and greatly improve user experience. Users can naturally interact with predefined or customized interactive elements in 360° panoramic pictures. Integrated solution: The interactive device integrates the generation, display, and interaction of 360° panoramic pictures, forming a complete solution for easy deployment and application. Whether it is a small or large enterprise, it can easily integrate the device to quickly build its own 360° panoramic picture interaction system. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0018] Figure 1 The method flowchart of the present application; Figure 2 The panoramic picture and hexahedron picture area correspondence of the present application; Figure 3 The device module interaction diagram of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] Embodiment 1 The embodiments of the present application will be further described below with reference to the drawings.

[0021] Please refer to Figure 1 The AIGC-based lightweight 360° panoramic picture generation method provided by the present application comprises the following steps, Step 1, receiving the initial instruction input by the user, and generating a 360° panoramic picture based on the initial instruction; the initial instruction includes a natural language instruction or a sketch description, using SkyboxAI to receive the text requirement input by the user or the content description of the sketch, guiding the large model to generate a 360° panoramic picture meeting the requirements of the instruction.

[0022] Step 2, splitting the 360° panoramic picture into six pictures of hexahedral structure; based on space mapping and projection transformation, the generated 360° panoramic picture is automatically split into six pictures of hexahedral structure, corresponding to the front, back, left, right, up and down six directions respectively. In the six pictures, +X is right, -X is left, +Y is up, -Y is down, +Z is front, and -Z is back, corresponding to the six corresponding areas of the 360° panoramic picture.

[0023] Step 3, extracting the visual elements of each face in the hexahedral picture, generating the corresponding natural language description, and splicing into the complete description of the panoramic picture scene; using a multi-modal large model (such as GPT-4o, Doubao), the generated hexahedral picture is described in natural language, which requires to give the main elements in the picture. For the six faces, each generates data in the form of (the nth face, descriptive text). Finally, all the descriptive texts describing the hexahedral picture are spliced to form a complete description of the 360° panoramic picture scene.

[0024] Step 4, calculating the semantic similarity between the user's initial instruction and the complete description of the panoramic picture scene, and if it is lower than the set threshold, the panoramic picture is regenerated; using a text embedding model text-embedding-3-large to generate a vector v user , a vector v cube is generated for the complete description of the 360° panoramic picture scene, and the cosine similarity is used to calculate the semantic similarity between the two vectors. A threshold is set, for example, 0.80 or above is determined as "meeting the requirements". If it does not meet the requirements, return to step one to re-execute.

[0025] Step 5: Identify important elements in the six face maps of the hexahedral structure and their positions in the image, generate anchor point data containing labels and coordinates; use multi-modal large models (such as GPT-4o, Baobao) to identify important element anchor point information contained in each face of the hexahedral map, and generate a list like [(Label1, Coordinate1), (Label2, Coordinate2),...]. Among them, the label is used to clearly identify the name of the element, which is convenient for users and the system to identify and operate; the coordinate information is determined by image recognition and positioning algorithm to determine the specific position of the element in the 360° panoramic map.

[0026] Step 6: Based on the six face maps of the hexahedral structure, build a CSS3D cube structure, load each face image to show the 360° panoramic scene; use CSS3D technology to build a cube structure for displaying panoramic maps on the front end, accurately load the six split face maps to different faces of the cube, and thus build a complete and realistic panoramic display effect.

[0027] Step 7: Add interactive components to the corresponding image area according to the anchor point data; bind the interactive events of the interactive components. According to the anchor point data generated by AI, add interactive components to the corresponding image area, and the optional preset interactive components are: buttons, hot icon or set highlight area.

[0028] The system predefines multiple interactive events for the convenience of users, including clicking to pop up an information box, jumping to a page, playing a voice introduction, loading product details (suitable for e-commerce scenarios), and other interactive events. At the same time, the system supports users to customize events using Javascript language and bind them to interactive components.

[0029] This embodiment takes the user's wish to generate a 360° panoramic map of the Eiffel Tower in Paris as an example, and the specific steps are as follows: Step 1: The user inputs the natural language instruction "generate a 360° panoramic map of the Eiffel Tower in Paris, with streets, crowds and greenery around" through the input module. Use Skybox AI to receive user input text requirements or content descriptions of sketches to guide the large model to generate a 360° panoramic map that meets the requirements of the instruction. After receiving the instruction, Skybox AI uses the diffusion model to generate the corresponding 360° panoramic map according to the training data and algorithm. During the generation process, the model will learn relevant features from massive image data based on elements such as "Eiffel Tower in Paris", "streets", "crowds" and "greenery" to generate a 360° panoramic map containing these elements.

[0030] Step 2: The image splitting module automatically splits the generated 360° panorama into a hexahedral graph in a programmed manner and converts it into a format compatible with CSS3D. The split hexahedral graph has six faces corresponding to the front, back, left, right, up, and down directions, respectively. user Step 3: For the hexahedral graph, important elements such as the "Eiffel Tower," "street," "crowd," and "greenery" are extracted. The user's initial description is converted into a vector v cube using the text-embedding-3-large model. The similarity between the user's description and the complete description of the 360° panorama is calculated using cosine similarity.

[0031] Step 4: Analyze the hexahedral graph to determine the positions of these elements in the image. Use a multi-modal large model (such as GPT-4o or Baidu) to identify the important anchor point information contained in each face of the hexahedral graph. Generate a list containing labels and coordinates. In the example, [(Eiffel Tower, (0.5, 0, 0)), (greenery, (0, 0.5, 0)),...] is extracted.

[0032] Step 5: The front-end display module uses CSS3D technology to build a cubic structure and loads the split hexahedral map to different faces of the cube to display the 360° panorama. CSS3D technology utilizes the browser's graphics processing capabilities to efficiently render three-dimensional scenes, providing users with a realistic Paris Eiffel Tower panorama. Step 6: According to the anchor point data, add interactive components such as transparent buttons or hot icons to the corresponding image area. Add a transparent button to the Eiffel Tower image area.

[0033] Step 7: Add interactive events to the interactive components at the anchor points. Bind the "click to view the history of the Eiffel Tower" interactive event. When the user clicks on the button, the corresponding interactive operation is triggered.

[0034] Please refer to Figure 2 for the area correspondence between the 360° panorama and the regular hexahedral graph. For the 360° panorama generated by Skybox AI, we will divide it into -X, X, -Y, Y, -Z, and Z regions, corresponding to the six faces of the regular hexahedral graph, and use a Python script to perform the segmentation. Figure 3

[0035] Example 2 Please refer to Figure 3 ​The base AIGC lightweight 360° panoramic map generation and interaction device provided by the application includes an input and generation module N1, an image splitting module N2, an image element extraction module N3, an image anchor point acquisition module N4, a front-end display module N5, an interaction feedback module N6, and a storage module N7. The input and generation module N1 is responsible for supporting step 1, obtaining user input, and guiding the Skybox AI to generate a 360° panoramic map meeting the requirements. In the embodiment, the input and generation module can receive the user input natural language instruction "generate a 360° panoramic map of the Eiffel Tower in Paris, surrounded by streets, crowds, and green plants" and guide the Skybox AI to generate a 360° panoramic map.

[0036] The image splitting module N2 is responsible for supporting step 2, based on spatial mapping and projection transformation, automatically splitting the generated 360° panoramic map into hexahedral face corresponding images. In the embodiment, the image splitting module can split the 360° panoramic map generation structure of step 1 according to user requirements to obtain a hexahedral image.

[0037] The image element extraction module N3 is responsible for supporting steps 3 and 4, using a multi-modal large model (such as GPT-4o, Doubaobao), performing natural language description on the generated hexahedral image, and requiring it to describe the main elements in the picture. Integrate each face description to form a complete description of the 360° panoramic map scene. In addition, use the text embedding model text-embedding-3-large to generate vectors for the user's initial description and the complete description of the 360° panoramic map scene, calculate the similarity between the two descriptions through cosine similarity, and judge the relationship with the set threshold. If the similarity exceeds the threshold, it is considered that the generated result meets the user's requirements, and if it is lower than the threshold, it is required to be regenerated. In the embodiment, the image element extraction module can extract element descriptions such as "Eiffel Tower", "street", etc. from the 360° panoramic map, integrate them into a complete description of the 360° panoramic map, and then convert them into text vectors together with the user's requirement "generate a 360° panoramic map of the Eiffel Tower in Paris, surrounded by streets, crowds, and green plants", calculate the cosine similarity, and finally compare it with the threshold. In the embodiment, it is assumed that the similarity between the user's requirement and the complete description of the 360° panoramic map exceeds the threshold.

[0038] The image anchor point acquisition module N4 supports step 5 by using a multimodal large model (such as GPT-4o or Doubao) to identify the anchor point information of important elements contained in each face of the hexahedral image, generating a list in the form of [(label 1, coordinate 1), (label 2, coordinate 2), ...]. The labels clearly identify the names of the elements, facilitating identification and operation by users and the system; the coordinate information determines the specific location of the element in the panoramic image through image recognition and positioning algorithms. In this embodiment, the image anchor point acquisition module can acquire the labels and coordinates of core elements such as "Eiffel Tower," "street," and "crowd."

[0039] The front-end display module N5 supports step 6, which uses CSS3D technology to build a page on the front end for displaying a 360° panoramic image. In this embodiment, the front-end display module can create an interactive front-end page based on the generated hexahedral diagram of the Eiffel Tower in Paris using CSS3D technology.

[0040] The interactive feedback module N6 supports steps 7 and 8, creating interactive elements on the front-end display page corresponding to the hexahedral image area based on the anchor point data generated by the multimodal large model, and supporting users to directly use preset or custom interactive events. In this embodiment, the interactive feedback module can add interactive elements such as transparent buttons or hotspot icons to core elements with labels and coordinates, such as "Eiffel Tower," "street," and "crowd," and add interactive events to these interactive elements.

[0041] The storage module stores generated panoramic images, anchor point data, interactive elements, and front-end interactive component templates, ensuring data storage and fast access for convenient subsequent retrieval and management. In this embodiment, the storage module provides predefined interactive components and events, and stores user-generated process information.

[0042] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0043] It should be understood that the above description of the preferred embodiments is quite detailed, but this should not be construed as limiting the scope of protection of this invention. It is neither necessary nor possible to exhaustively describe all possible implementations. Those skilled in the art, guided by this invention, can make substitutions or modifications without departing from the scope of the claims, all of which fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for generating a lightweight panoramic image based on AIGC, characterized in that, The method comprises the following steps: Step 1: receiving an initial instruction input by a user and generating a 360° panoramic image based on the initial instruction; Step 2: splitting the 360° panoramic image into six face images of a hexahedral structure; Step 3: extracting visual elements of each face of the hexahedral image, generating corresponding natural language descriptions, and splicing into a complete description of the panoramic image scene; Step 4: calculating the semantic similarity between the initial instruction of the user and the complete description of the panoramic image scene, and if it is lower than a set threshold, re-generating the panoramic image; Step 5: identifying important elements of the six face images of the hexahedral structure and their positions in the images, and generating anchor point data containing labels and coordinates; Step 6: constructing a CSS3D cube structure based on the six face images of the hexahedral structure, loading each face image to display a 360° panoramic scene; Step 7: adding an interactive component to the corresponding image area according to the anchor point data; binding a preset or customized interaction event to the interactive component. 2.The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The initial instruction includes a natural language instruction or a sketch description. 3.The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The step 1 is implemented by using an AIGC technology based on a diffusion model, specifically, a panoramic image is generated by Skybox AI according to user input.

4. The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The step 2 divides the 360° panoramic image into images in front, back, left, right, up and down directions through spatial mapping and projection transformation, and converts them into a format compatible with CSS3D.

5. The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The step 3 analyzes the hexahedral image by using a multi-modal large language model, and outputs structured description texts of each face.

6. The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The step 4 converts texts into vectors by using a text embedding model, and calculates the similarity between the user instruction and the complete description of the panoramic image scene by using cosine similarity.

7. The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The step 5 identifies key elements in the image by using a multi-modal large model, and outputs a tuple list of (label, coordinate). 8.The method of claim 1, wherein, The interactive component includes at least one of a button, a hot icon or a highlighted area.

9. The AIGC-based lightweight panoramic map generation method of claim 1, wherein, The interaction event includes at least one of clicking to pop up an information box, page jumping, voice playing and product detail loading, and supports user-defined event logic through JavaScript. 10.A device for generating a lightweight panoramic map based on AIGC, characterized in that, It comprises: An input and generation module for receiving an initial instruction input by a user and generating a 360° panoramic image based on the initial instruction; An image splitting module for splitting the 360° panoramic image into six face images of a hexahedral structure; An image element extraction module for extracting visual elements of each face of the hexahedral image, generating corresponding natural language descriptions, splicing into a complete description of the panoramic image scene, and performing similarity judgment; An image anchor point acquisition module for identifying important elements of the six face images of the hexahedral structure and their positions in the images, and generating anchor point data containing labels and coordinates; A front-end display module for constructing a CSS3D cube structure based on the six face images of the hexahedral structure, loading each face image to display a 360° panoramic scene; An interaction feedback module for adding an interactive component to the corresponding image area according to the anchor point data; binding a preset or customized interaction event to the interactive component; A storage module for storing panoramic images, anchor point data, interactive components and event templates. The AIGC-based lightweight panoramic map generation device is used to execute the steps in the AIGC-based lightweight panoramic map generation method according to any one of claims 1-9. The AIGC-based lightweight panoramic map generation device is used to execute the steps in the AIGC-based lightweight panoramic map generation method according to any one of claims 1-9.

Images