Bonsai generation method and device based on virtual environment interaction, equipment and medium

Abstract

The embodiment of the invention provides a bonsai generation method and device based on virtual environment interaction, equipment and a medium, and belongs to the technical field of virtual reality. The method comprises the following steps: in response to a bonsai design request, carrying out data acquisition on an actual scene to obtain space information corresponding to the actual scene; generating a virtual scene based on the spatial information, wherein an interface corresponding to the virtual scene comprises a plurality of bonsai design elements; determining a target bonsai design element in the plurality of bonsai design elements; generating a bonsai model in the virtual scene based on the target bonsai design element; and in response to an actual bonsai generation request, projecting the bonsai model in an actual scene to obtain a target bonsai design. According to the embodiment of the invention, the interaction capability can be enhanced, the manual operation steps are reduced, and the working efficiency of a bonsai designer can be improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a bonsai generation method and device, equipment and medium based on virtual environment interaction. Background Art

[0002] Bonsai design requires manual operation and field observation by the designer, and repeated debugging by the designer during the design process to make a precise bonsai design that fully expresses creativity. In the process of creating bonsai design, not only a lot of manpower and material resources are consumed, but also the efficiency is often low. Although there are some bonsai design tools on the market that can help designers, these design tools have a low degree of intelligence and lack the ability to interact with designers, resulting in high difficulty in use. Improper use will increase the workload of designers, thereby further reducing the work efficiency of designers. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a bonsai generation method and device, equipment and medium based on virtual environment interaction, which can enhance the interactive capability and reduce the manual operation steps, thereby helping to improve the work efficiency of bonsai designers.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application proposes a bonsai generation method based on virtual environment interaction, the method comprising: In response to the bonsai design request, data collection is performed on the actual scene to obtain spatial information corresponding to the actual scene; Generate a virtual scene based on the spatial information, wherein the interface corresponding to the virtual scene includes a plurality of bonsai design elements; determining a target bonsai design element among the plurality of bonsai design elements; Based on the target bonsai design element, generating a bonsai model in the virtual scene; In response to a request for generating an actual bonsai, the bonsai model is projected into the actual scene to obtain a target bonsai design.

[0005] In some embodiments, in response to the bonsai design request, collecting data on the actual scene to obtain spatial information corresponding to the actual scene includes: In response to the bonsai design request, calibrate the sensor and set the scanning mode of the sensor according to the target material, wherein the sensor is used to collect data of the actual scene; Scan the target material using the calibrated sensor, and obtain the spatial information by feedback after scanning; The spatial information is converted into point cloud data, and a three-dimensional model corresponding to the target material is generated based on the point cloud data.

[0006] In some embodiments, the use of a calibrated sensor to scan the target material and the spatial information obtained by feedback after scanning includes: According to the type of the sensor, calling the corresponding candidate interface; In response to the selection operation of the candidate interface, the target material is scanned by using the sensor, and the scanning result is acquired and processed through the selected target interface, the scanning result includes the spatial information, and the target interface belongs to one of the candidate interfaces.

[0007] In some embodiments, the use of a calibrated sensor to scan the target material and the spatial information obtained by feedback after scanning includes: When the sensor is a lidar sensor, the lidar sensor is used to emit laser light, and then depth information is determined by measuring the time of laser emission, and the depth information is used as the scanning result.

[0008] In some embodiments, the use of a calibrated sensor to scan the target material and the spatial information obtained by feedback after scanning includes: When the type of the sensor is a structured light sensor, using the structured light sensor to project light corresponding to a preset light pattern onto a target object in the actual scene; According to the change of light on the target object, the three-dimensional shape information of the target object is determined, and the three-dimensional shape information is used as the scanning result.

[0009] In some embodiments, generating a bonsai model in the virtual scene based on the target bonsai design element includes: The step of obtaining a two-dimensional texture map and a three-dimensional bonsai model selected by a user in response to the operation on the bonsai design element; Converting the three-dimensional bonsai model into a two-dimensional plane, and determining a mapping area on the two-dimensional plane according to the geometric structure of the three-dimensional bonsai model; Mapping the two-dimensional texture map to the mapping area, and generating a three-dimensional bonsai model after texture mapping; Based on preset lighting simulation conditions, the texture-mapped three-dimensional bonsai model is rendered to generate a bonsai model in the virtual scene.

[0010] In some embodiments, the method further comprises the step of generating the plurality of bonsai design elements; The step of generating the plurality of bonsai design elements comprises: Perform feature extraction on the acquired bonsai image to obtain multiple bonsai features; Classifying each of the bonsai features into a target feature group to obtain an updated feature group; When a new bonsai image is acquired, a step of extracting features from the acquired bonsai image to obtain a plurality of bonsai features is performed, or the target feature group is divided into a plurality of sub-groups, and each bonsai feature in the new bonsai image is classified into a target sub-group in the plurality of sub-groups, and the bonsai features of each of the feature groups or each of the sub-groups are all bonsai design elements.

[0011] To achieve the above purpose, another aspect of the embodiment of the present application provides a bonsai generation device based on virtual environment interaction, the device comprising: A collection module, for collecting data of the actual scene in response to a bonsai design request, so as to obtain spatial information corresponding to the actual scene; A first generating module, configured to generate a virtual scene based on the spatial information, wherein an interface corresponding to the virtual scene includes a plurality of bonsai design elements; A selection module, configured to determine a target bonsai design element from among the plurality of bonsai design elements; A second generating module, configured to generate a bonsai model in the virtual scene based on the target bonsai design element; The projection module is used for projecting the bonsai model into the actual scene in response to the actual bonsai generation request to obtain the target bonsai design.

[0012] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned bonsai generation method based on virtual environment interaction is implemented.

[0013] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned bonsai generation method based on virtual environment interaction is implemented.

[0014] The embodiments of the present application include at least the following beneficial effects: The technical solution of this application can more intelligently understand the needs of designers and adapt to the actual environment through data collection and virtual scene generation, and users can directly operate bonsai design elements in the virtual scene. These operations can be responded to quickly, enhancing the interaction between the tool and the designer, and by updating the design in real time, the bonsai model is projected in the actual scene, and the virtual design is transformed into a practical application to ensure the practicality and feasibility of the design. It can be seen that the more interactive and easier-to-operate operation process in the technical solution of this application makes it easier for designers to use and reduces the learning cost. Through the automated and intelligent design process, the amount of work that designers need to do manually is reduced, which is conducive to improving the work efficiency of bonsai designers. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a bonsai generation method based on virtual environment interaction; Figure 2 is a flow chart of step S101 in a bonsai generation method based on virtual environment interaction; Figure 3 It is a flowchart of a bonsai generation method based on virtual environment interaction, which uses sensors for scanning when applied in a specific scene; Figure 4 It is a flowchart of a bonsai generation method based on virtual environment interaction, which uses a laser radar sensor to scan when applied in a specific scene; Figure 5 is a flow chart of step S103 in a bonsai generation method based on virtual environment interaction; Figure 6 It is a process architecture timing diagram of mapping a two-dimensional texture map to a three-dimensional bonsai model when a bonsai generation method based on virtual environment interaction is applied in a specific scene; Figure 7 It is a schematic diagram of the relationship between various bonsai design elements when the bonsai generation method based on virtual environment interaction is applied in a specific scene; Figure 8 It is a schematic diagram of the architecture of the calling tool when the bonsai generation method based on virtual environment interaction is applied to Apple Vision Pro; Fig. 9 It is a structural schematic diagram of a bonsai generation device based on virtual environment interaction; Fig.10 It is a schematic diagram of the hardware structure of an electronic device. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0017] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0018] Apple Vision Pro is Apple’s first head-mounted “spatial computing” display device, which provides users with a new interactive experience by combining virtual content with the real world.

[0019] VisionOS is the AR glasses system released by Apple at the 2023 Apple Worldwide Developers Conference on June 6, 2023. Apple Vision Pro will be equipped with this system for the first time.

[0020] Stable Diffusion is a text-to-image generative model that is able to generate photo-realistic images from any text input.

[0021] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0022] VR (Virtual Reality) is a technology that uses computer technology to simulate the environment and create an immersive experience that makes users feel as if they are in the scene.

[0023] AR (Augmented Reality) is a technology that superimposes virtual information or objects onto the real world, allowing users to see these virtual elements while seeing the real world.

[0024] MR (Mixed Reality) is a technology that combines virtual reality (VR) and augmented reality (AR). It can superimpose virtual elements on the real world and allow users to interact with these virtual elements in real time.

[0025] As VR and AR technologies develop, researchers are beginning to explore the application of these technologies in bonsai design. VR technology can provide a fully immersive virtual environment, but the interaction with the real world is limited. AR technology allows virtual elements to be combined with the real world, providing more flexibility and practicality. MR technology is a combination of VR and AR technology, which allows users to interact with virtual elements in the real world while maintaining a sense of the real environment. This technology provides new possibilities for bonsai design, allowing designers to intuitively preview and modify designs in a real environment.

[0026] The launch of Apple Vision Pro represents an important milestone in MR technology. It provides advanced display technology, precise eye tracking, gesture recognition and spatial audio system, providing designers with a powerful tool to achieve more efficient and intuitive bonsai design. And with the development of artificial intelligence and machine learning technologies such as AIGC and LLM, new skills are provided for bonsai design. Through AI-assisted processing of massive image data and user personalized customization needs, automatic generation of design plans, optimization of plant layout and simulation of plant growth can be achieved.

[0027] Although there are some bonsai design tools on the market that can help designers, these design tools have a low level of intelligence and lack the ability to interact with designers, which makes them difficult to use. Improper use will increase the designer's workload, thereby further reducing the designer's work efficiency.

[0028] In order to solve the problems of low efficiency, lack of precision and limited user participation in the traditional bonsai design and production process, the main purpose of the embodiments of the present application is to propose a bonsai generation method and device, equipment and medium based on virtual environment interaction, which can enhance the interactive capability and reduce the manual operation steps, thereby helping to improve the work efficiency of bonsai designers.

[0029] Figure 1 is a flowchart of a bonsai generation method based on virtual environment interaction provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0030] Step S101, in response to a bonsai design request, data collection is performed on an actual scene to obtain spatial information corresponding to the actual scene.

[0031] Step S102, generating a virtual scene based on the spatial information, wherein the interface corresponding to the virtual scene includes a plurality of bonsai design elements.

[0032] Step S103, determining a target bonsai design element from among a plurality of bonsai design elements.

[0033] Step S104: generating a bonsai model in the virtual scene based on the target bonsai design elements.

[0034] Step S105 , in response to the actual bonsai generation request, projecting the bonsai model into the actual scene to obtain a target bonsai design.

[0035] Steps S101 to S104 shown in the embodiment of the present application can more intelligently understand the needs of designers and adapt to the actual environment through data collection and virtual scene generation, and users can directly operate bonsai design elements in the virtual scene, and these operations can be responded to quickly, which enhances the interaction between the tool and the designer, and through real-time design updates, the bonsai model is projected in the actual scene, and the virtual design is transformed into a practical application, ensuring the practicality and feasibility of the design. It can be seen that the operation process with stronger interactive capabilities and easier operation in the technical solution of the present application makes it easier for designers to use, reduces learning costs, and reduces the workload that designers need to complete manually through automated and intelligent design processes, which is conducive to improving the work efficiency of bonsai designers.

[0036] In step S101 of some embodiments, the actual scene can be scanned by a 3D scanner to obtain accurate spatial information, or the scene can be photographed and the spatial information can be analyzed and extracted using an image recognition algorithm. Augmented reality (AR) technology can also be used to assist in collecting spatial information through the user's movement in the scene and changes in perspective.

[0037] In step S102 of some embodiments, the method of generating a virtual scene can be to use professional 3D modeling software such as Autodesk Maya or 3ds Max to manually create a virtual scene and bonsai design elements according to spatial information, or to use a game engine such as Unity or Unreal Engine to quickly generate a virtual scene according to spatial information and integrate interactive design elements. When generating a virtual scene, an interface corresponding to the virtual scene is generated. The interface can be a graphical user interface. The graphical user interface allows users to interact with the virtual scene through controls such as buttons, sliders, and drop-down menus, select and operate bonsai design elements. The bonsai design elements include existing bonsai models, and the color and shape of the bonsai can be customized. Optionally, the virtual scene corresponding interface can also include multi-level menus and toolbars, providing rich operation options, including adding, deleting, and modifying bonsai design elements.

[0038] In step S103 of some embodiments, in the interface of the virtual scene, the user can select the target bonsai design element by interactive means such as clicking and dragging. Optionally, the target bonsai design element is determined by intelligent recommendation, that is, the target bonsai design element is recommended by machine learning algorithm according to the user's historical selection and preference, or the most suitable bonsai design element is automatically determined according to spatial information and physical rules by physical simulation technology.

[0039] In step S104 of some embodiments, the method of generating a bonsai model in a virtual scene based on the target bonsai design element includes generating a three-dimensional bonsai model with precise length, height and width in the virtual scene according to the size parameters in the target bonsai design element, or searching for an existing bonsai model with the target bonsai design element in a database and using it as the generated bonsai model.

[0040] In step S105 of some embodiments, the bonsai model can be projected into the actual scene using AR technology, and the user can view the design effect on site. Holographic projection technology can also be used to create a more three-dimensional and realistic bonsai model display. The bonsai model can also be displayed on a display screen, allowing the user to interact and adjust through the touch screen.

[0041] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203: Step S201, in response to a bonsai design request, calibrate the sensor and set the scanning mode of the sensor according to the target material, wherein the sensor is used to collect data of the actual scene.

[0042] Step S202: Scan the target material using the calibrated sensor, and feed back the spatial information obtained after the scanning.

[0043] Step S203: convert the spatial information into point cloud data, and generate a three-dimensional model corresponding to the target material based on the point cloud data.

[0044] It is understandable that, in the steps S201 to S203 shown in the embodiment of the present application, the present embodiment can accurately scan the target material and obtain detailed spatial information by calibrating the sensor and setting a specific scanning mode. This calibration process ensures that perception sensors of different modes generate consistent sensor data, which are consistent with each other when perceiving the environment. The spatial information obtained from the sensor can be converted into point cloud data, and a three-dimensional model corresponding to the target material is generated based on these point cloud data. This process involves the conversion of data collected from the actual scene to a digital model, which improves the accuracy and authenticity of the model. By converting spatial information into point cloud data, various data processing techniques can be used to process and analyze the point cloud, such as noise reduction, feature extraction, etc., to generate a more accurate three-dimensional model. The automated point cloud to three-dimensional model conversion process reduces manual intervention and improves work efficiency. At the same time, the combination of AI algorithms can further optimize the model generation process, realize intelligent design and modeling, and quickly convert the original scan data into a three-dimensional proposal, improve the efficiency of the workflow, and make the process from design to implementation faster and more accurate.

[0045] See also Figure 3 , this embodiment is explained by combining specific application examples: like Figure 3 As shown, first, the scanning device is calibrated. The device calibration includes two parallel sub-steps: pupil distance calibration: calibrate the pupil distance of the device to ensure that the device can accurately measure and process the distance between the eyes; face size measurement: measure the size of the face to provide necessary reference data for subsequent scanning. Then, after selecting the scanning mode, perform the scan. The scanning step includes two parallel sub-steps: LiDAR emission: use the laser radar (LiDAR) technology to emit a laser beam and obtain depth information by measuring the time it takes for the laser to reflect back; structured light camera projection and processing: use a structured light camera to project a specific light pattern and process the reflected light to obtain the three-dimensional information of the object. Then, the three-dimensional shape of the object is obtained using the depth information and reflection time data. The obtained three-dimensional model is optimized to improve the accuracy and quality of the model, and to verify whether the optimized three-dimensional model meets the design requirements and standards. Finally, the final three-dimensional model data is stored and archived for subsequent use and retrieval.

[0046] In this embodiment, the built-in sensor of Apple Vision Pro is used for calibration, including pupil distance calibration and facial size measurement. The required material of the bonsai is used as the target material, and the appropriate scanning mode and parameters are selected to set the material or the bonsai as a whole. The three-dimensional scan is performed using the LiDAR sensor and structured light camera of Vision Pro. Then, the collected data is processed by the processor of Vision Pro, converted into point cloud data, and meshed to form a preliminary three-dimensional model. Finally, the accuracy and completeness of the model are verified to ensure that every detail of the bonsai is accurately captured, and the scanned data and model are stored and archived for use on the design platform.

[0047] Optionally, the preliminary three-dimensional model is optimized, and the optimization measures include texture mapping and lighting effect adjustment.

[0048] In some embodiments, the specific implementation of step S202 may be: According to the type of sensor, the corresponding candidate interface is called; in response to the selection operation of the candidate interface, the target material is scanned using the sensor, and the scanning result is obtained and processed through the selected target interface. The scanning result includes spatial information, and the target interface is one of the candidate interfaces.

[0049] It is understandable that the main function of the interface is to provide adaptation for different types of sensors to ensure that sensor data can be correctly read and processed. A sensor may correspond to one or more interfaces, and users can select the most suitable interface based on specific application scenarios and requirements. The system will predefine a set of candidate interfaces based on sensor type and user requirements, and users can select a suitable interface from a set of candidate interfaces. Optionally, the sensor type is automatically identified by the sensor model, serial number or communication protocol, and one or more interfaces are called as candidate interfaces. Then a user interface is provided to list all candidate interfaces, allowing users to manually select based on the actual connected sensor.

[0050] Optionally, when the user completes the interface selection operation, the corresponding driver is triggered to drive the sensor, or the callback function is automatically executed to respond to the user operation, and a predefined configuration file is used to map the sensor type to the corresponding interface. When the sensor is connected, the configuration file is automatically loaded to call the interface.

[0051] Optionally, when scanning the target material, a timer is set and the target material is scanned using a sensor at preset time intervals, for example, a light signal is emitted to the target material, and a light signal reflected from the target material is received as a scanning result, or a sound wave signal is emitted to the target material, and a sound wave signal reflected from the target material is received as a scanning result.

[0052] Optionally, the sensor driver custom development interface provided by Apollo and other platforms is used to read and process the data collected by the sensor by implementing specific base class methods, and modeling is performed based on the collected data to achieve a three-dimensional digital description of spatial information.

[0053] Optionally, the interfaces available to users include: Scan Mode Configuration Interface (setScanMode interface): can enable high-density, fast-scanning LiDAR working mode.

[0054] Custom Scan Area Specification Interface (defineScanRegion interface): allows dynamic capture of objects required for bonsai modeling with limited range and limited features, such as containers, plant size, substrate, etc.

[0055] Scan Data Post-Processing Interface (processScanData interface): used to perform post-scan data processing, such as data smoothing, hole filling, feature extraction, etc.

[0056] Scan Data Export Interface (exportScanData interface): can export scan data into different formats, such as STL, OBJ, PLY, etc., for use in design platform software.

[0057] User Feedback Loop Interface (feedbackLoop interface): It can evaluate the scan results by creating a user feedback mechanism and automatically adjust the scan strategy based on the feedback.

[0058] Historical Scan Data Comparison Interface (compareHistoricalScans interface): This interface can compare current scan data with historical data and is used to monitor changes in objects or environments over time.

[0059] API for Automated Scan Planning (autoScanPlanner interface): Provides an automated scan planning interface that allows the system to automatically perform scan tasks based on preset conditions.

[0060] In step S202 of some embodiments, when the sensor type is a lidar sensor, the lidar sensor is used to emit laser, and then the depth information is determined by measuring the time of laser emission, and the depth information is used as the scanning result.

[0061] See also Figure 4 , this embodiment is explained by combining specific application examples: like Figure 4 As shown in the figure, first, the LiDAR (i.e., laser radar sensor) hardware is configured and initialized to ensure that the device is in the correct working state. Next, the system sets the network listening and input modes to prepare for receiving data. This includes configuring network parameters so that the LiDAR device can communicate. Optionally, according to specific needs, custom functions or processing logic can be added by configuring interfaces to process LiDAR data to meet specific application needs. Define the interface for interacting with the LiDAR device, which includes interfaces for data reading and processing to ensure that data can be smoothly transmitted from the LiDAR device to the processing system. For example, ReadScanCallback: This is a callback function for data reading. It receives and processes the raw scan data transmitted in real time, which is the raw data captured by the LiDAR device. Optionally, after the data is collected by the LiDAR device, the collected data can also be preprocessed. The preprocessing includes several sub-steps: Filtering: Filter the raw LiDAR data to remove unnecessary information or noise. Denoising: Remove noise from the data to improve the accuracy of the data. Time synchronization: Ensure that the timestamps of the data are synchronized, which is very important for multi-sensor fusion or time series analysis. Finally, convert the data collected by the LiDAR device into other formats for further analysis and use. This may include converting point cloud data into 3D meshes or OBJ files.

[0062] In step S202 of some embodiments, when the sensor type is a structured light sensor, the structured light sensor is used to project light corresponding to a preset light pattern onto a target object in an actual scene; based on changes in light on the target object, the three-dimensional shape information of the target object is determined, and the three-dimensional shape information is used as a scanning result.

[0063] See also Figure 5 In some embodiments, step S104 may include but is not limited to steps S301 to S303: Step S301, in response to the operation on the bonsai design element, obtaining the two-dimensional texture map and the three-dimensional bonsai model selected by the user.

[0064] Step S302: convert the three-dimensional bonsai model into a two-dimensional plane, and determine a mapping area on the two-dimensional plane according to the geometric structure of the three-dimensional bonsai model.

[0065] Step S303, mapping the two-dimensional texture map to the mapping area, and generating a three-dimensional bonsai model after texture mapping.

[0066] Step S304, based on the preset illumination simulation conditions, the three-dimensional bonsai model after texture mapping is rendered to generate a bonsai model in the virtual scene.

[0067] Optionally, the user can select appropriate texture materials according to the requirements of the bonsai design, which can be high-resolution pictures or specially designed patterns.

[0068] In some embodiments, the three-dimensional model is analyzed to determine the mapping area on the two-dimensional plane according to the geometric structure of the three-dimensional bonsai model, and the model is UV unfolded to generate a two-dimensional plane to prepare for texture mapping. The UV unfolding should minimize the distortion and stretching of the texture. The texture is edited and adjusted with image editing software to ensure that the texture is correctly aligned and displayed on the two-dimensional plane, and then the edited texture is applied to the two-dimensional plane of the model, which can be achieved by the texture mapping function in the 3D modeling software. Further, the material properties of the three-dimensional bonsai model after texture mapping, such as reflectivity, transparency, roughness, etc., can be adjusted to enhance the realism of the model, and some details of the model can also be enhanced, such as adding wear, stains or other special effects to make the model more realistic. Subsequently, the lighting conditions in the virtual environment are set first to simulate the effects of different light sources on the texture to ensure the realistic effect of the texture under different lighting, and then the rendering test is performed in the 3D modeling software to observe whether the texture and lighting effects meet the design expectations.

[0069] Optionally, textures and materials are optimized to suit the needs of real-time rendering, ensuring smooth display on devices such as Apple Vision Pro.

[0070] Optionally, test users' interaction with the textured model via gestures and eye tracking to ensure the interaction is natural and intuitive.

[0071] Optionally, evaluate the impact of texture mapping and lighting effects on system performance and make adjustments as necessary to achieve the best balance of performance and visuals.

[0072] In some embodiments, the process of mapping a two-dimensional texture to a three-dimensional model can be implemented on visionOS by using RealityKit and SwiftUI. Combining the APIs of RealityKit, SwiftUI, and visionOS can create an immersive 3D experience in spatial computing applications and effectively map two-dimensional texture maps to three-dimensional models. In addition to applying the above APIs, the following APIs can also be applied to achieve this: Optionally, developers can convert 2D vector content into 3D models through the MeshResource API: Use MeshResource.ShapeExtrusionOptions to specify the resolution and extrusion depth of the shape to achieve the conversion of 2D graphics to 3D meshes; use the LowLevelTexture API to quickly update texture resources and implement dynamic textures; and use the HoverEffectComponentAPI to add visual effects during user interaction, such as highlight or spotlight effects, to enhance the user's hovering experience.

[0073] Optionally, use appropriate resolution and geometry thickness to avoid flickering artifacts caused by thin geometry with high contrast.

[0074] It is understandable that the present embodiment can achieve the following technical effects: by mapping the two-dimensional texture map onto the three-dimensional bonsai model, the seamless combination of the two-dimensional texture and the three-dimensional geometric structure is achieved, and the visual authenticity and detail performance of the model are enhanced. Specifically, by determining the mapping area according to the geometric structure of the three-dimensional bonsai model on the two-dimensional plane, it is ensured that the texture map is correctly mapped to the corresponding part of the model, and the distortion and deformation of the texture are avoided. Through texture mapping, the visual effect of the three-dimensional bonsai model is enhanced, making it more realistic and closer to the bonsai in the real world. In addition, the three-dimensional bonsai model after texture mapping is rendered based on the preset lighting simulation conditions, and a bonsai model with realistic light and shadow effects can be generated in a virtual scene. In the above process, the user can see the effect of the texture map and bonsai model selected by the user in the virtual scene in real time, providing an interactive bonsai design experience, and through the automated texture mapping and rendering process, the need for designers to manually adjust the texture and lighting is reduced, and the design efficiency is improved.

[0075] See also Figure 6 In a specific application example, the process of mapping a two-dimensional texture map to a three-dimensional bonsai model can be achieved in the following way: In a specific application example, developers use the MeshResource API to process the data of a 3D model. This API may allow developers to access and manipulate the mesh resources of a 3D model. Developers can also use MeshResource.ShapeExtrusionOptions to specify the resolution and extrusion depth of the shape, which involves the generation or modification of a 3D model. Through the LowLevelTexture API, developers can perform low-level operations on texture resources, such as directly processing texture data. The HoverEffectComponent component may be used to add visual effects when the user interacts, such as visual feedback when the user hovers over a 3D model. Use components such as the HoverEffectComponent to add visual effects when the user interacts. In this solution, the RealityKit, SwiftUI, and visionOS APIs are combined to create an immersive 3D experience in spatial computing applications, effectively mapping two-dimensional texture maps to three-dimensional models.

[0076] In some embodiments, the bonsai generation method based on virtual environment interaction further includes: Generate multiple bonsai design elements. The specific method of generating multiple bonsai design elements includes the following steps: Feature extraction is performed on the acquired bonsai image to obtain multiple bonsai features; each bonsai feature is classified into a target feature group to obtain an updated feature group; when a new bonsai image is acquired, the step of extracting features from the acquired bonsai image to obtain multiple bonsai features is performed, or the target feature group is divided into multiple sub-groups, and each bonsai feature in the new bonsai image is classified into a target sub-group in the multiple sub-groups, and the bonsai features of each feature group or each sub-group are all bonsai design elements.

[0077] It can be understood that the bonsai features of each feature group or sub-group in this embodiment are used as design elements, which enrich the material library available to designers and improve the diversity and personalization of designs. When a new bonsai image is acquired, feature extraction can be performed quickly to ensure timely processing and analysis of new images. Through automated feature extraction and classification, designers can be assisted in intelligent design, reduce repetitive work, and improve design efficiency. In addition, accurate feature extraction and classification help improve the quality of the design and ensure the consistency of design elements with actual bonsai features.

[0078] See also Figure 7Specifically, the bonsai features obtained by feature extraction of bonsai images can first be classified into plant, soil, rock, decoration, container, light, environment and other feature groups. Then, the plant feature group can be divided into two subgroups: plant species A and B. The decoration feature group can be divided into two subgroups: human statues and animal statues. The container feature group can be divided into two subgroups: ceramic pots and stone pots. The light feature group can be divided into two subgroups: natural light and artificial light. The environment feature group can be divided into two subgroups: indoor and outdoor. For details, please refer to Table 1.

[0079] Table 1 - Bonsai design elements

[0080] Below, the present application also provides a specific application example to provide a detailed introduction and usage description of the solution of the above embodiment: See also Figure 8 In this embodiment, a new VisionOS project is created using Xcode, and the basic parameters and target devices of the project are set. ARKit is used for environmental perception to obtain the three-dimensional structure of the surrounding environment; ARKit's plane detection, object recognition, and scene reconstruction functions are used to provide accurate spatial information for the overlay of virtual content. 3D models and scenes are created using 3D modeling software (such as Maya, Blender) or Reality Composer, and 3D content is imported into the VisionOS project to ensure that the size and proportion of the model match the real world. RealityKit framework is used for 3D rendering and animation, and RealityKit's physics engine and lighting model are used to enhance the realism of virtual content. SwiftUI is used to design the user interface, including menus, buttons, and preview windows, to ensure that the interface elements are clearly visible on the high-resolution display of Apple Vision Pro. ARKit's hand tracking function is used to achieve accurate gesture recognition and control. By designing interactive methods for gesture control and eye tracking, users can interact with virtual content naturally. Metal is used for efficient real-time rendering to ensure the smoothness and realism of virtual content. According to the performance characteristics of Apple Vision Pro, the rendering settings are optimized to balance visual effects and performance.

[0081] During use, first, the user wears Apple Vision Pro and starts the bonsai design app, which collects the user's initial preferences and workspace information through sensors; then, after the user selects plants, tools, and design styles through the virtual interface, the AI ​​design assistance system provides personalized design suggestions based on the user's selection. Then, the user interacts with the virtual plants and tools through gestures and voice to create designs, and the design effects are displayed through real-time rendering technology, so that the user can see the changes instantly; finally, after the user completes the design, the virtual design is turned into an actual bonsai by projecting the completed three-dimensional bonsai model into the actual scene. The user can also choose to save or export the design as a production guide, and can also take pictures and share their results through the app.

[0082] See also Fig. 9 The embodiment of the present application further provides a bonsai generation device based on virtual environment interaction, which can implement the above-mentioned bonsai generation method based on virtual environment interaction, and the device includes: The collection module 401 is used to collect data of the actual scene in response to the bonsai design request to obtain spatial information corresponding to the actual scene.

[0083] The first generating module 402 is used to generate a virtual scene based on the spatial information, and the interface corresponding to the virtual scene includes a plurality of bonsai design elements.

[0084] The selection module 403 is used to determine a target bonsai design element from the plurality of bonsai design elements.

[0085] The second generating module 404 is used to generate a bonsai model in the virtual scene based on the target bonsai design element.

[0086] The projection module 405 is used to project the bonsai model into the actual scene in response to the actual bonsai generation request to obtain the target bonsai design.

[0087] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0088] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned bonsai generation method based on virtual environment interaction when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0089] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0090] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 501 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiment of the present application.

[0091] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the bonsai generation method based on virtual environment interaction in the embodiment of this application; The input / output interface 503 is used to implement information input and output.

[0092] The communication interface 504 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0093] The bus 505 transmits information between various components of the device (eg, the processor 501 , the memory 502 , the input / output interface 503 , and the communication interface 504 ).

[0094] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0095] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the bonsai generation method based on virtual environment interaction is implemented.

[0096] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0097] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The bonsai generation method, device, equipment and medium based on virtual environment interaction provided in the embodiments of the present application collect data on the actual scene in response to the bonsai design request and obtain the spatial information corresponding to the actual scene, which enables the technical solution of the present application to more accurately understand and adapt to the actual environment and provide accurate spatial reference for subsequent designs. A virtual scene is generated based on the collected spatial information, which allows designers to design in a simulated environment, thus reducing dependence on the actual physical space and improving the flexibility and efficiency of the design. Multiple bonsai design elements are called for users to choose from. By providing a wealth of design elements, designers can choose according to their own creativity and needs, which increases the personalization and diversity of the design.

[0099] Based on the target bonsai design elements, a bonsai model is generated in the virtual scene, which reflects the interactive ability of the tool. Designers can adjust and optimize the design through intuitive operations, making the design process more intuitive and user-friendly. In response to the actual bonsai generation request, the bonsai model is projected in the actual scene to obtain the target bonsai design, thereby transforming the virtual design into a practical application, ensuring the practicality and feasibility of the design.

[0100] It can be seen that the technical solution of this application can more intelligently understand the needs of designers and adapt to the actual environment through data collection and virtual scene generation, and users can directly operate bonsai design elements in the virtual scene, and these operations can be responded to quickly, which enhances the interaction between the tool and the designer, and through real-time design updates, the bonsai model is projected in the actual scene, and the virtual design is transformed into a practical application to ensure the practicality and feasibility of the design. It can be seen that the more interactive and easier-to-operate operation process in the technical solution of this application makes it easier for designers to use, reduces learning costs, and reduces the workload that designers need to complete manually through automated and intelligent design processes, which is conducive to improving the work efficiency of bonsai designers.

[0101] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0102] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0103] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0105] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0106] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0111] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A bonsai generation method based on virtual environment interaction, characterized in that: The method comprises: In response to the bonsai design request, data collection is performed on the actual scene to obtain spatial information corresponding to the actual scene; Generate a virtual scene based on the spatial information, wherein the interface corresponding to the virtual scene includes a plurality of bonsai design elements; determining a target bonsai design element among the plurality of bonsai design elements; Based on the target bonsai design element, generating a bonsai model in the virtual scene; In response to a request for generating an actual bonsai, the bonsai model is projected into the actual scene to obtain a target bonsai design.

2. The bonsai generation method based on virtual environment interaction according to claim 1, characterized in that: In response to the bonsai design request, data collection is performed on the actual scene to obtain spatial information corresponding to the actual scene, including: In response to the bonsai design request, calibrate the sensor and set the scanning mode of the sensor according to the target material, wherein the sensor is used to collect data of the actual scene; Scan the target material using the calibrated sensor, and obtain the spatial information by feedback after scanning; The spatial information is converted into point cloud data, and a three-dimensional model corresponding to the target material is generated based on the point cloud data.

3. The bonsai generation method based on virtual environment interaction according to claim 2, characterized in that: The target material is scanned by using the calibrated sensor, and the spatial information obtained by feedback after scanning includes: According to the type of the sensor, calling the corresponding candidate interface; In response to the selection operation of the candidate interface, the target material is scanned by using the sensor, and the scanning result is acquired and processed through the selected target interface, the scanning result includes the spatial information, and the target interface belongs to one of the candidate interfaces.

4. The bonsai generation method based on virtual environment interaction according to claim 3 is characterized in that: The target material is scanned by using the calibrated sensor, and the spatial information obtained by feedback after scanning includes: When the sensor is a lidar sensor, the lidar sensor is used to emit laser light, and then depth information is determined by measuring the time of laser emission, and the depth information is used as the scanning result.

5. The bonsai generation method based on virtual environment interaction according to claim 3 is characterized in that: The target material is scanned by using the calibrated sensor, and the spatial information obtained by feedback after scanning includes: When the type of the sensor is a structured light sensor, using the structured light sensor to project light corresponding to a preset light pattern onto a target object in the actual scene; According to the change of light on the target object, the three-dimensional shape information of the target object is determined, and the three-dimensional shape information is used as the scanning result.

6. The bonsai generation method based on virtual environment interaction according to claim 1, characterized in that: The step of generating a bonsai model in the virtual scene based on the target bonsai design element comprises: The step of obtaining a selected two-dimensional texture map and a three-dimensional bonsai model in response to the operation on the bonsai design element; Converting the three-dimensional bonsai model into a two-dimensional plane, and determining a mapping area on the two-dimensional plane according to the geometric structure of the three-dimensional bonsai model; Mapping the two-dimensional texture map to the mapping area, and generating a three-dimensional bonsai model after texture mapping; Based on preset lighting simulation conditions, the texture-mapped three-dimensional bonsai model is rendered to generate a bonsai model in the virtual scene.

7. The bonsai generation method based on virtual environment interaction according to claim 1, characterized in that: The method further comprises the steps of generating the plurality of bonsai design elements; The step of generating the plurality of bonsai design elements comprises: Perform feature extraction on the acquired bonsai image to obtain multiple bonsai features; Classifying each of the bonsai features into a target feature group to obtain an updated feature group; When a new bonsai image is acquired, a step of extracting features from the acquired bonsai image to obtain a plurality of bonsai features is performed, or the target feature group is divided into a plurality of sub-groups, and each bonsai feature in the new bonsai image is classified into a target sub-group in the plurality of sub-groups, and the bonsai features of each of the feature groups or each of the sub-groups are all bonsai design elements.

8. A bonsai generation device based on virtual environment interaction, characterized in that: The device comprises: A collection module, for collecting data of the actual scene in response to a bonsai design request, so as to obtain spatial information corresponding to the actual scene; A first generating module, configured to generate a virtual scene based on the spatial information, wherein an interface corresponding to the virtual scene includes a plurality of bonsai design elements; A selection module, for determining a target bonsai design element from among the plurality of bonsai design elements; A second generating module, configured to generate a bonsai model in the virtual scene based on the target bonsai design element; The projection module is used for projecting the bonsai model into the actual scene in response to the actual bonsai generation request to obtain the target bonsai design.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the bonsai generation method based on virtual environment interaction as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the bonsai generation method based on virtual environment interaction according to any one of claims 1 to 7 is implemented.

Images