Creative support method and system based on immersive VR exploration

Abstract

The invention discloses a creative support method and system based on immersive VR exploration, and the method comprises the steps: an initialization step: encoding a set of given design sample sets, defining a design space on the basis, and randomly selecting an initial design anchor point from the design space; a sampling step: sampling and decoding around the design anchor point to obtain a corresponding design suggestion, and displaying the design suggestion in a recommendation space; in the tracking step, the user wears VR glasses to roam in the recommendation space, sight line staying information of the user is automatically tracked through the VR glasses, and a new design anchor point is determined according to the sight line staying information; and a repeating step: repeatedly executing the sampling step and the tracking step until the user finds a satisfactory design work and quits. According to the highly immersive creative three-dimensional modeling method, more effective support is provided for the creation process of the user through passive input.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and particularly to a creative support method and system based on immersive VR exploration. Background Art

[0002] A Creativity Support Tool (CST) is a type of software tool or platform designed to assist people in creative thinking and design. Its goal is to reduce creative barriers, improve creative efficiency, and ultimately promote the generation of innovative outcomes. Creativity Support Tools have always been an appealing topic in the Human-Computer Interaction (HCI) community and have been explored in various application scenarios, such as brainstorming and 3D modeling. However, creativity is a complex and multi-faceted phenomenon. Although it has been claimed that Creativity Support Tools can provide promising opportunities for creative design and practice, there are also some debates about the limitations of these tools. The idea that using digital design tools may interfere with design creativity has been pointed out by some research. There is currently no consensus on the question of how CSTs can effectively support the creativity of their users. According to Ben Schneidermann's theory (Shneiderman B. Creativity support tools: accelerating discovery and innovation[J]. Communications of the ACM, 2007, 50(12): 20-32), a creativity support system must follow the principles of supporting exploratory search, providing complete operation history retention, allowing collaborative cooperation, and having good usability and robustness. In the field of graphics, creativity support systems for 3D modeling used to be a relatively popular area of exploration.The work of Siddhartha Chaudhuri et al. (Chaudhuri S, Koltun V. Data-driven suggestions for creativity support in 3D modeling[M] / / ACM SIGGRAPH Asia 2010 papers. 2010:1-10) uses similarity retrieval and shape matching techniques to retrieve models similar to the user's current model from the model library, and then identifies components that do not exist in the current model as suggestions to inspire design inspiration; the work of Xu K et al. (Xu K, Zhang H, Cohen-Or D, et al. Fit and diverse: Set evolution for inspiring 3d shape galleries[J]. ACM Transactions on Graphics(TOG), 2012, 31(4):1-10) defines a fitness function based on genetic algorithms and the user's selection to generate a set of models that meet the user's preferences and are creative; the work of Averkiou M et al. (Averkiou M, Kim V G, Zheng Y, et al. Shapesynth: Parameterizing model collections for coupled shape exploration and synthesis[C] / / Computer Graphics Forum. 2014, 33(2):125-134) realizes a parametric representation of the latent shape space by hierarchically embedding the model collection into the feature space, thus supporting the rapid exploration and synthesis of new shapes; the work of Yang Y L et al. (Yang Y L, Yang Y J, Pottmann H, et al. Shape space exploration of constrained meshes[J]. ACM Trans. Graph., 2011, 30(6):124) explores and navigates the high-dimensional shape space by approximating the shape feature space with tangent spaces and quadric parameterized surfaces. These works take inspiration inspiration and design recommendation as the entry points and design creative support tools.

[0003] With the emergence of various VR glasses and the popularity of the concept of the metaverse, we are gradually moving towards a virtual three-dimensional world. It is of great significance to provide powerful and easy-to-use tools to meet the needs of users at different levels to quickly create various three-dimensional assets. Creative activities such as three-dimensional modeling require creators to have strong innovation capabilities. Creativity is an important driving force in the design field, and virtual reality (VR) technology has shown great potential in stimulating creativity. However, the existing creativity support system (CST) has limitations in balancing divergent thinking and convergent thinking, and cannot meet the diverse needs of designers. However, the existing creativity support technology is mainly aimed at desktop environments. Existing studies have shown that virtual reality technology can help learn design knowledge, meet the requirements of design tasks, and improve innovative design capabilities by providing a creative journey. However, simply migrating existing technologies directly to a virtual reality environment can easily destroy the user's immersion, thereby affecting the stimulation of their creative inspiration. In addition, existing studies generally implement creativity support algorithms based on active input, which not only interrupts the original creative process, but also increases the learning cost of use. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a creative support method based on immersive VR exploration, comprising the following steps:

[0006] Initialization step: encode a given set of design sample sets, define a design space based on this, and randomly select an initial design anchor point from the design space;

[0007] A sampling step, sampling around the design anchor point, decoding the sampled objects to obtain corresponding design suggestions, and displaying the design suggestions at appropriate locations in a recommendation space; the recommendation space is a virtual scene;

[0008] Tracking step: the user puts on VR glasses to roam in the recommended space. The VR glasses automatically track the user's gaze information, and build a user preference model based on the gaze information to determine the new design anchor point;

[0009] Repeat the steps, repeatedly perform the sampling step and the tracking step, until the user gets a satisfactory idea and exits, and organize the design suggestions displayed in the latest recommendation space and output them as creative results.

[0010] Preferably, in the initialization step, the design sample is abstracted into a feature vector and forms a point in the design space, and the process is achieved by constructing a bijection of a three-dimensional model and the design space, including the following steps:

[0011] Based on symmetry, the three-dimensional model is abstracted into a directed bounding box;

[0012] For each directed bounding box, the points on the point cloud are symmetric along its three coordinate axes, and the number of symmetric points still on the corresponding grid of the point cloud is calculated, and this number is used as the symmetry index of the model relative to the directed bounding box; the candidate directed bounding box with the highest symmetry index is given priority, and when the symmetry indexes are the same, the one with the smallest volume is selected as the directed bounding box abstraction of the three-dimensional model;

[0013] Use the isometric feature mapping algorithm to embed the point cloud of the 3D model into a one-dimensional space and calculate its distribution;

[0014] The directed bounding box parameter vector and the low-dimensional embedding distribution of the 3D model are concatenated into a feature vector as an abstract representation of the 3D model.

[0015] The directed bounding box parameter vector and the distribution of the three-dimensional low-dimensional embedding are recalculated from the feature vector, and the cosine similarity between the low-dimensional embedding distribution and the low-dimensional embedding distribution of the existing three-dimensional models in the database is calculated. After finding the three-dimensional model components with the highest cosine similarity, these components are affine transformed according to the directed bounding box parameters to synthesize a complete three-dimensional model with reasonable semantics, which is added to the design space as a new feature vector.

[0016] Preferably, in the sampling step, sampling is performed on a sampling sphere with a radius r and a design anchor point as the center, and after decoding, n corresponding design suggestions are obtained and displayed at appropriate positions in the recommendation space.

[0017] Preferably, the sampling step further includes automatically synthesizing a new recommendation space, including the following steps:

[0018] Randomly select a template from the prepared recommended space templates. Each template has an entry channel and an exit channel, and the two channels can be seamlessly spliced;

[0019] Adjust the layout of the scenes in the recommended space while satisfying the accessibility display constraints;

[0020] The template is spliced ​​into the existing scene sequence to synthesize a new recommendation space.

[0021] Preferably, the template is spliced ​​into an existing scene sequence, and before the splicing, the positions of the design works displayed in the existing scenes are randomly disturbed and changed.

[0022] Preferably, the tracking step includes:

[0023] Collect the user's gaze information through VR glasses;

[0024] Analyze user preferences based on the line-of-sight stay information, so as to obtain a scalar field defined on the sampling sphere;

[0025] Determine an optimal traveling direction and the next anchor point through Monte Carlo sampling.

[0026] Preferably, the analysis of user preferences based on the line-of-sight stay information is achieved by calculating the attraction of an object to the user, expressed as:

[0027] Defined as:

[0028]

[0029] where A i represents the attraction of the object O i to the user, and d(.) represents the projection distance from the center c i of the object O i to the line of sight v.

[0030] The present invention also provides a creative support system based on immersive VR exploration, including:

[0031] An initialization module, which encodes a given set of design sample sets, and on this basis defines a design space, and randomly selects an initial design anchor point from the design space;

[0032] A sampling module, which samples around the design anchor point, decodes the sampled objects to obtain corresponding design suggestions, and displays the design suggestions at appropriate positions in the recommendation space; the recommendation space is a virtual scene;

[0033] A tracking module, where the user wears VR glasses and roams in the recommendation space, automatically tracks the line-of-sight stay information of the user through the VR glasses, and establishes a user preference model based on the line-of-sight stay information to determine a new design anchor point;

[0034] An output module, which repeats the operations of the sampling module and the tracking module until the user exits, and sorts out and outputs the design suggestions displayed in the latest recommendation space as creative results.

[0035] The present invention has the following beneficial effects:

[0036] (1) By constructing a bijective mapping between the three-dimensional model database and the feature space, the present invention can navigate, predict and synthesize the three-dimensional model that best meets the user's needs in the feature space according to the user's preference data;

[0037] (2) The present invention infers the user's interest in the presented recommendation through passive gaze tracking; based on the interest, a scalar field on a sphere centered at the current anchor point of the user in the design space is defined; and finally, the forward direction in the design space and the position of the next anchor point are determined through a Markov Monte Carlo process;

[0038] (3) The present invention designs a template-driven recommendation space synthesis method, which can continuously synthesize infinite recommendation spaces to meet the user's immersive roaming needs of any duration.

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A method step diagram of an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of user tracking according to an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a recommended space synthesis according to an embodiment of the present invention;

[0044] Figure 5 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] See also Figure 1 and Figure 2 The method step diagram and flow chart of an embodiment of the present invention are shown, which include the following steps:

[0046] S101, initialization step, encoding a given set of design sample sets, defining a design space based on the encoding, and randomly selecting an initial design anchor point from the design space;

[0047] S102, a sampling step, sampling is performed around the design anchor point, decoding the sampled objects to obtain corresponding design suggestions, and displaying the design suggestions at appropriate positions in a recommendation space; the recommendation space is a virtual scene;

[0048] S103, tracking step, the user wears VR glasses to roam in the recommended space, the VR glasses automatically track the user's gaze information, and establish a user preference model based on the gaze information to determine a new design anchor point;

[0049] S104, output step, repeatedly executing the sampling step and the tracking step until the user exits, and arranging and outputting the design suggestions displayed in the latest recommendation space as creative results.

[0050] Specifically, the design sample set is a collection of design works, which can be two-dimensional design works, such as flat design drawings, oil paintings, etc.; or three-dimensional, such as three-dimensional cars, vases, etc. The design space is an abstract space, in which a long feature vector is used to represent each design work. For example, the feature vector of a three-dimensional vase can include length, width, height, shape, style, component composition, etc. The design space is an abstract representation, and the high-dimensional feature vector corresponding to each design object is a point in the design space; the recommendation space is similar to a virtual exhibition hall, and each object corresponds to a point in the design space. During the user's roaming process, a roaming track will be formed in the design space. This track reflects the user's preference and evolution process for various design works during the roaming process.

[0051] In the embodiment of the present invention, the process in which a user puts on VR glasses to enter the system and starts looking for design inspiration is called a design journey. When a user starts a design journey, it starts from a random initial anchor point in the design space, samples are taken around it, and the sampled points are decoded by design to obtain the appropriate positions in the recommendation space; finally, the user roams in the recommendation space, and a new recommendation space is automatically synthesized to ensure that the user can roam continuously and smoothly and immerse himself in the inspiration brought by different designs; finally, during the user's roaming, the user's line of sight information is automatically tracked through VR glasses, and a user preference model is established based on this, and a new design anchor point is determined based on the preference. The whole process is repeated until the user exits.

[0052] Specifically, the embodiment of the present invention constructs a bijection between a three-dimensional model and a design space, including the following steps:

[0053] First, the three-dimensional model is abstracted as a directed bounding box based on symmetry. The faces on the convex hull of the model point cloud are traversed, and a set of candidate directed bounding boxes is constructed based on these faces. Then, for each directed bounding box, the points on the point cloud are symmetric along its three coordinate axes, and the number of its symmetric points that are still on the corresponding grid of the point cloud is calculated, which is used as the symmetry index of the model relative to the directed bounding box; the candidate directed bounding box with the highest symmetry index is given priority, and when the symmetry indexes are the same, the one with the smallest volume is selected as the directed bounding box abstraction of the three-dimensional model.

[0054] Then, the isometric feature mapping algorithm is used to embed the point cloud of the three-dimensional model into one-dimensional space and calculate its distribution.

[0055] Finally, the directed bounding box parameter vector and the low-dimensional embedding distribution of the 3D model are concatenated into a feature vector as an abstract representation of the 3D model. After completing the mapping from the 3D model to the feature space, the inverse mapping is also implemented: the directed bounding box parameter vector and the distribution of the 3D low-dimensional embedding are recalculated from the feature vector, and the cosine similarity between the low-dimensional embedding distribution and the low-dimensional embedding distribution of the existing 3D models in the database is calculated. After finding the 3D model components with the highest cosine similarity, these components are affine transformed according to the directed bounding box parameters, thereby synthesizing a complete 3D model with reasonable semantics.

[0056] Specifically, the extraction process of the directed bounding box is shown in Algorithm 1:

[0057]

[0058] For details, see Figure 3 As shown in Figure 1, when a user puts on glasses and starts a design journey, they start from a randomly generated position (anchor point) in the design space and sample on a sphere with a radius of r centered at the anchor point (as shown in Figure 1). Figure 3 After decoding, n corresponding design suggestions are obtained and displayed in the appropriate position of the recommendation space. When the user roams in the recommendation space, his / her sight information is collected to analyze his / her preferences, thereby obtaining a scalar field defined on the sampling sphere (such as Figure 3 Finally, the optimal direction of travel and the next anchor point (the position on the ball corresponding to the direction) are determined by Monte Carlo sampling. This process is repeated, and a corresponding trajectory is formed in the design space during the user's roaming (as shown in (b)). Figure 3 ).

[0059] Specifically, the recommended space is a scene similar to a virtual exhibition hall, which is used to display objects sampled from the design space. Users wear VR glasses to roam in the exhibition hall, and their sight information is recorded to model user preferences. i Attractiveness to users i Defined as:

[0060]

[0061] where d(.) measures the center of the object c i The projection distance to the line of sight v. Considering that each user's exploration time is uncertain, and in order to provide a better real experience, an automatic synthesis algorithm of the recommendation space is introduced to provide a theoretically infinite exploration space, so that users can always immerse themselves in it. Figure 4As shown in the figure, a template is randomly selected from the prepared recommended space templates. Each template has an entry and an exit channel, and the two channels can be seamlessly spliced. Then, the layout of the scene in the space is adjusted under the display constraints such as accessibility. (Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, Matthias Nieβner:

[0062] DiffuScene: Scene Graph Denoising Diffusion Probabilistic Model for Generative Indoor Scene Synthesis. CoRR abs / 2303.14207(2023)), and finally splice the template into the existing scene sequence. When the user roams the current scene, the next scene is synthesized immediately, and the cycle repeats until the user stops roaming. When the user roams in the recommendation space, he will see various recommended and displayed design works for his reference and inspiration, and finally he can start creating on a satisfactory work.

[0063] See also Figure 5 As shown, it is a system structure diagram of an embodiment of the present invention, including:

[0064] Initialization module 501 encodes a given set of design sample sets, defines a design space based on the encoding, and randomly selects an initial design anchor point from the design space;

[0065] The sampling module 502 performs sampling around the design anchor point, decodes the sampled objects, obtains corresponding design suggestions, and displays the design suggestions at appropriate positions in the recommendation space; the recommendation space is a virtual scene;

[0066] Tracking module 503: The user wears VR glasses and roams in the recommended space. The VR glasses automatically track the user's gaze information, and establish a user preference model based on the gaze information to determine a new design anchor point.

[0067] The output module 504 repeatedly executes the operations of the sampling module and the tracking module until the user exits, and organizes and outputs the design suggestions displayed in the latest recommendation space as creative results.

[0068] The creativity support system proposed in the embodiments of the present invention combines VR technology with CST, providing a brand-new design tool for designers and effectively solving this problem. By embedding the model into the feature space and using the preference prediction algorithm to accurately grasp the user's real-time preferences, the system achieves the balance between divergent thinking and convergent thinking, and can help designers gradually converge to the final design solution. The application of VR technology provides designers with an immersive experience, enabling them to freely explore different design possibilities in the virtual environment and intuitively feel the effects of design solutions. The heuristic suggestions of CST can help designers break through thinking limitations and discover new design ideas. The system can also provide personalized suggestions according to user preferences to meet the needs of different users, and will continuously optimize the suggestion algorithm based on the exploration trajectory and preferences of designers to provide suggestions that better meet user needs. With the continuous development of VR technology, this system is expected to become an important tool in the future design field, providing designers with a more efficient, convenient and personalized design experience.

[0069] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A creative support method based on immersive VR exploration, characterized in that, The following steps are involved: Initialization step: encode a given set of design sample sets, define a design space based on this, and randomly select an initial design anchor point from the design space; A sampling step, sampling around the design anchor point, decoding the sampled objects to obtain corresponding design suggestions, and displaying the design suggestions at appropriate locations in a recommendation space; the recommendation space is a virtual scene; Tracking step: the user puts on VR glasses to roam in the recommended space. The VR glasses automatically track the user's gaze information, and build a user preference model based on the gaze information to determine the new design anchor point; Repeat the steps, repeatedly perform the sampling step and the tracking step, until the user finds a satisfactory design work and exits.

2. The creative support method based on immersive VR exploration according to claim 1, wherein, In the initialization step, the design sample is abstracted into a feature vector and forms a point in the design space. The process is achieved by constructing a bijection of a three-dimensional model and the design space, and includes the following steps: Based on symmetry, the three-dimensional model is abstracted into a directed bounding box; For each directed bounding box, the points on the point cloud are symmetric along its three coordinate axes, and the number of symmetric points still on the corresponding grid of the point cloud is calculated, and this number is used as the symmetry index of the model relative to the directed bounding box; the candidate directed bounding box with the highest symmetry index is given priority, and when the symmetry indexes are the same, the one with the smallest volume is selected as the directed bounding box abstraction of the three-dimensional model; Use the isometric feature mapping algorithm to embed the point cloud of the 3D model into a one-dimensional space and calculate its distribution; The directed bounding box parameter vector and the low-dimensional embedding distribution of the 3D model are concatenated into a feature vector as an abstract representation of the 3D model. The directed bounding box parameter vector and the distribution of the three-dimensional low-dimensional embedding are recalculated from the feature vector, and the cosine similarity between the low-dimensional embedding distribution and the low-dimensional embedding distribution of the existing three-dimensional models in the database is calculated. After finding the three-dimensional model components with the highest cosine similarity, these components are affine transformed according to the directed bounding box parameters to synthesize a complete three-dimensional model with reasonable semantics, which is added to the design space as a new feature vector.

3. The creative support method based on immersive VR exploration according to claim 1, wherein, In the sampling step, sampling is performed on a sampling sphere with a radius of r and a design anchor point as the center, and n corresponding design suggestions are obtained after decoding and displayed at appropriate positions in the recommendation space.

4. The creative support method based on immersive VR exploration according to claim 3, wherein The sampling step also includes automatically synthesizing a new recommendation space, including the following steps: Randomly select a template from the prepared recommended space templates. Each template has an entry channel and an exit channel, and the two channels can be seamlessly spliced; Adjust the layout of the scenes in the recommended space while satisfying the accessibility display constraints; The template is spliced ​​into the existing scene sequence to synthesize a new recommendation space.

5. The creative support method based on immersive VR exploration according to claim 4, characterized in that The template is spliced ​​into the existing scene sequence, and before the splicing, the positions of the design works displayed in the existing scenes are randomly disturbed and changed.

6. The creative support method based on immersive VR exploration according to claim 3, characterized in that The tracking steps include: Collect the user's gaze information through VR glasses; Analyze user preferences based on gaze retention information to obtain a scalar field defined on the sampling sphere; Determine an optimal traveling direction and the next anchor point through Monte Carlo sampling.

7. The creative support method based on immersive VR exploration according to claim 6, wherein Analyze user preferences based on the line-of-sight stay information, which is achieved by calculating the attraction of an object to the user and is expressed as: Defined as: Among them, A i represents the attraction of the object O i to the user, and d(.) represents the center c i of the object O i to the projection distance of the line of sight v.

8. An innovative support system based on immersive VR exploration, characterized in that, Including: An initialization module that encodes a given set of design sample sets, defines a design space based on this, and randomly selects an initial design anchor point from the design space; A sampling module that samples around the design anchor point, decodes the sampled objects to obtain corresponding design suggestions, and displays the design suggestions at appropriate positions in the recommendation space; the recommendation space is a virtual scene; A tracking module that enables the user to roam in the recommendation space while wearing VR glasses, automatically tracks the user's line-of-sight stay information through the VR glasses, and establishes a user preference model based on the line-of-sight stay information to determine a new design anchor point; An output module that repeats the operations of the sampling module and the tracking module until the user exits, and organizes and outputs the design suggestions displayed in the latest recommendation space as creative results.

Images