Model rendering method and system based on mixed reality interaction creation

Abstract

The application provides a model rendering method and system based on mixed reality interactive creation, and relates to augmented reality technology, comprising: calculating an optimal rendering view angle based on the geometric structure of a three-dimensional model; obtaining a preset optimal viewing angle, and calculating a final rendering view angle by weighting the optimal viewing angle and the optimal rendering view angle; projecting the three-dimensional model based on the final rendering view angle to generate a baseline sketch map; obtaining an interactive coloring image with the baseline sketch map as a bottom map; identifying the interactive coloring image, extracting a regional image matched with the baseline sketch map, and performing geometric correction; extracting color features from the regional image, coloring the baseline sketch map based on the color features, and generating a color texture map; and rendering the three-dimensional model based on the color texture map. The application enables users to input creativity in a hand-drawing manner, automatically generates a high-quality map conforming to a target three-dimensional object, and finally renders and displays in real time in an augmented reality environment.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality technology, specifically relating to a model rendering method and system based on mixed reality interactive creation. Background Technology

[0002] In existing technologies, the creation of 3D content for augmented reality (AR) applications primarily relies on specialized 3D modeling and texture painting software, a complex and demanding process. Modification of model textures typically involves pre-made texture maps, which prevents users from making real-time, personalized dynamic changes based on specific images or colors in the real-world scene. While existing artificial intelligence (AI) image generation technologies can generate high-quality 2D images from text, these images lack 3D spatial attributes, making it difficult to directly and accurately map them onto the surface geometry of specific 3D objects. Summary of the Invention

[0003] The purpose of this invention is to provide a model rendering method and system based on mixed reality interactive creation, so as to improve the above-mentioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0004] Firstly, this application provides a model rendering method based on mixed reality interactive creation, including:

[0005] Obtain the 3D model and calculate the optimal rendering viewpoint based on the geometry of the 3D model;

[0006] Obtain the preset optimal viewing angle, and calculate the final rendering angle by weighting the optimal viewing angle and the optimal rendering angle.

[0007] Projecting the 3D model based on the final rendering perspective generates baseline line art textures;

[0008] Obtain an interactive coloring image with the baseline line art texture as the base image;

[0009] The interactive coloring image is identified, and the region image that matches the baseline line drawing is extracted and geometrically corrected.

[0010] Color features are extracted from the region image, and the baseline line drawing texture is colored based on the color features to generate a color texture map;

[0011] The 3D model is rendered based on color texture mapping.

[0012] Secondly, this application also provides a model rendering system based on mixed reality interactive creation, including:

[0013] The first module is used to acquire the 3D model and calculate the optimal rendering perspective based on the geometry of the 3D model.

[0014] The second module is used to obtain the preset best viewing angle, and to calculate the final rendering angle by weighting the best viewing angle and the best rendering angle.

[0015] The third module is used to project the 3D model based on the final rendering perspective and generate baseline texture maps.

[0016] The fourth module is used to obtain interactive coloring images with the baseline line art texture as the base image;

[0017] The fifth module is used to identify interactive coloring images, extract the region images that match the baseline line art texture, and perform geometric correction.

[0018] The sixth module is used to extract color features from the region image, color the baseline line drawing texture based on the color features, and generate a color texture map.

[0019] The seventh module is used to render the 3D model based on color texture maps.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention generates a basic texture template based on the structure of the 3D model itself, and then outputs the basic texture template to the user for creation. This allows users to input their ideas through the most natural hand-drawing method. Through AI intelligent understanding and enhancement, high-quality texture maps that conform to the surface geometry of the target 3D object are automatically generated, and the textures are perfectly applied back to the model. Finally, the images are rendered and displayed in real-time and accurately in an augmented reality environment. This method connects the entire chain from physical hand-drawing to model rendering to AR presentation, lowering the barrier to 3D creation and enhancing the immersiveness and creativity of the AR experience.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a model rendering method based on mixed reality interactive creation, as described in an embodiment of this application.

[0025] Figure 2 This is a diagram of a model rendering device based on mixed reality interactive creation, as shown in the embodiments of this application.

[0026] Symbol explanation: 800 - Model rendering device based on mixed reality interactive creation; 801 - Processor; 802 - Memory; 803 - Multimedia component; 804 - I / O interface; 805 - Communication component. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0029] Most research on sketch-based 3D modeling focuses on generating model structures, while research on converting hand-drawn brushstrokes into UV layouts that conform to the model is relatively limited. For the average user, there's a disconnect in the transformation path from 2D hand-drawn ideas to the personalized appearance of 3D AR objects, resulting in a disjointed user experience and low creation efficiency. In summary, current augmented reality digital content creation suffers from the following problems:

[0030] The model's appearance is fixed: the texture maps of the virtual model are pre-made, and users cannot make real-time, personalized dynamic changes based on specific images or colors in the real scene.

[0031] High barrier to entry: Creating high-quality personalized textures for 3D models requires professional software such as Photoshop and Substance Painter, which is a complex process that is difficult for ordinary users to participate in.

[0032] The technology chain is fragmented: existing technologies are in a "fragmented" state. AI painting tools (such as StableDiffusion) are good at generating 2D images from text, but the generated images lack three-dimensional spatial attributes and cannot be directly adapted to the surface of specific 3D models; sketch-based 3D modeling research focuses more on the generation of model geometry rather than the richness and artistry of textures; AR content platforms rely heavily on pre-built model libraries and lack the ability to generate real-time, personalized content.

[0033] The creative flow is not smooth: From a momentary inspiration in the real world (hand-drawn sketch) to the final presentation in the digital AR world, users need to go through multiple software and format conversions, which is a cumbersome process and makes it easy for creative ideas to be interrupted.

[0034] Example 1:

[0035] See Figure 1 To address the problems in the existing technology, this application provides a model rendering method based on mixed reality interactive creation, including steps S100, S200, S300, S400, S500, S600 and S700.

[0036] S100. Obtain the 3D model and calculate the optimal rendering viewpoint based on the geometry of the 3D model, including:

[0037] S110. Obtain the surface information of the 3D model, and calculate the area of ​​each surface and the normal vector of the surface.

[0038] Each model consists of several facets, including triangular and quadrilateral facets. For a triangular facet, two 3D vectors are obtained by subtracting any pair of its three vertices. The cross product of these two vectors yields the normal vector of the facet. The process is similar for quadrilateral facets. Two 3D vectors are obtained by subtracting any pair of its vertices. The cross product of these two vectors yields the normal vector of the quadrilateral facet. The facet normal vector is denoted as n.

[0039] Given the coordinates of the vertices of each facet, the area S of each facet can be calculated.

[0040] S120. The relative angle is constructed by calculating the difference between the normal vector of each facet and the optimal rendering view vector.

[0041] Let x be the vector of the optimal rendering viewpoint Y;

[0042] The relative angle α is obtained by calculating the angle between the normal vector n and the three-dimensional vector x using the dot product formula.

[0043] S130. Construct the objective function based on the relative angle and area of ​​each facet;

[0044] Suppose the 3D model has m faces, and the objective function is constructed as follows:

[0045] F(x) = α1×S1 + α2×S2 + …α i ×S i +…α m ×S m ;

[0046] Where F(x) is the objective function value, αi S represents the relative angle between the i-th facets. i The area of ​​the i-th facet is represented by m, which is the total number of faces in the 3D model.

[0047] S140. The optimal rendering view vector is obtained by minimizing the objective function.

[0048] When F(x) reaches its minimum value, the corresponding x is the optimal rendering view vector.

[0049] S200: Obtain the preset best viewing angle, and calculate the final rendering angle by weighting the best viewing angle and the best rendering angle.

[0050] The optimal viewing angle is the user's desired viewing angle, denoted as X;

[0051] The final rendering perspective V is calculated using the weighting function V=a×X+b×Y, where a and b are adjustable weight parameters.

[0052] S300: Projects the 3D model based on the final rendering perspective to generate baseline line art textures;

[0053] S310. Calculate the view occlusion degree based on the geometry of the 3D model;

[0054] The purpose of this step is to analyze whether the model has many (or large) complex occlusion areas, making it impossible to easily obtain its full appearance information from two angles; if the model does not have complex occlusion areas, baseline line art textures can be generated from two angles (e.g., front and back), and the two textures can be combined to fully display the model;

[0055] If there are complex occlusion areas, all facets need to be extracted and projected to ensure that the generated texture has no obvious blind spots and avoid affecting the user's viewing experience.

[0056] The occlusion of the viewing angle is calculated as follows:

[0057] S311. Calculate the dihedral angle between any two adjacent facets. If the dihedral angle is greater than a first preset value, mark it as an occlusion angle and mark the two facets connected by the occlusion angle as occlusion facets.

[0058] The dihedral angle is directional, ranging from 0-360°.

[0059] Traverse all adjacent face pairs and calculate the dihedral angle (angle between normal vectors) of each pair of adjacent face pairs; the first preset value is greater than 180°, and a dihedral angle between 180° and 360° indicates a concave structure, with the concavity being deeper the closer to 360°.

[0060] S312. Calculate the area ratio of all occluded surfaces to the total area of ​​the 3D model to obtain the occlusion area ratio;

[0061] The occlusion area ratio is used to reflect the proportion of potential occlusion areas on the surface of a 3D model under various viewing angles;

[0062] S313. By analyzing the connectivity of the occlusion angle, connected occlusion features are identified, including isolated occlusion pairs, occlusion chains, occlusion loops, and occlusion clusters.

[0063] Obtain the endpoint coordinates of the connecting edges of the occluded facets; by comparing the endpoint coordinates, identify whether there are connected edges, i.e., connected occlusion angles; when two facets form an occlusion angle but are not connected to other occlusion facets, they are isolated occlusion pairs, and the number of isolated occlusion pairs is recorded; when multiple occlusion facets are connected end-to-end to form a linear structure, they are occlusion chains, and the number and length of occlusion chains are recorded; when occlusion chains are closed end-to-end to form a loop structure, they are occlusion loops, and the number of occlusion loops is recorded; complex connection structures formed by multiple occlusion facets are occlusion clusters, and the number of occlusion clusters is recorded.

[0064] S314. Calculate the viewing angle occlusion degree based on the connectivity occlusion features and the occlusion area ratio;

[0065] First, calculate the occlusion complexity index:

[0066] Occlusion complexity = a × (number of isolated occlusion pairs) + b × (total length of occlusion chains) × (number of occlusion chains) + c × (number of occlusion loops) + d × (number of occlusion clusters);

[0067] a, b, c, and d are preset weights; among them, weights b and c are larger to ensure that when there is an occlusion chain or occlusion loop, the model tends to judge that the occlusion degree is high and it is difficult to display the whole picture from two perspectives.

[0068] Calculate the occlusion of the viewpoint:

[0069] Viewpoint occlusion = w1 × occlusion area ratio + w2 × occlusion complexity; where w1 and w2 are the weights of each item, and are preset values.

[0070] S320. Determine whether the occlusion of the viewing angle is less than a preset threshold.

[0071] S330. If the judgment result is yes, then the normal vector of the final rendering view is inverted to obtain the final rendering reverse view; the 3D model is projected and rendered based on the final rendering view and the final rendering reverse view.

[0072] If the occlusion is small, textures can be generated from two perspectives, and the final rendering perspective only needs to be calculated once, reducing the amount of computation.

[0073] S340. If the judgment result is negative, calculate the first rendering view based on the normal vector of the facet of the 3D model, and perform projection according to the first rendering view; delete the already projected facets and update the 3D model; recalculate the second rendering view based on the updated 3D model, and repeat the projection and facet deletion process until all facets are projected.

[0074] If the viewpoint occlusion is large, extract textures from multiple viewpoints to reduce blind spots. Therefore, first calculate a first rendering viewpoint using the same method as steps S200-S300, project from the first rendering viewpoint, and extract the first texture. Then delete the projected faces, and update to obtain a new model. Repeat the projection and face deletion process based on the new model. After all faces are projected, multiple textures (baseline line art textures) will be obtained. These textures can be combined to display the full appearance of the model.

[0075] The obtained baseline line drawing texture is processed by AI to remove color information, enhance line and grayscale features, and then output to the user.

[0076] S400: Obtain an interactive coloring image with the baseline line art texture as the base image;

[0077] After obtaining the black and white baseline drawing texture, users can then create their own coloring effects on it. Specifically, users can use software with drawing capabilities to display the baseline drawing texture on an electronic screen, where they can then process and create interactive coloring images.

[0078] As another implementation method, the baseline drawing texture can be printed as a paper document. Users can then process and create on the real paper baseline drawing texture, and then use a camera to capture real-world images. An image recognition algorithm can then be used to identify the baseline drawing texture to obtain an interactive coloring image.

[0079] S500: Recognizes interactive coloring images, extracts the region images that match the baseline line art texture, and performs geometric correction;

[0080] S510. Key points are identified from interactive coloring images and baseline line art textures using the local feature descriptor algorithm.

[0081] Local feature descriptor algorithms, such as SIFT, SURF, and ORB, detect key points in two images, including corners, edges, and texture-rich regions; record the position, scale, and orientation information of each key point; and calculate a feature descriptor for each key point.

[0082] S520. Perform similarity matching on the descriptors of key points, and filter out coarse matching pairs based on similarity.

[0083] Calculate the similarity between each keypoint descriptor in the interactive coloring image and all keypoint descriptors in the baseline texture using Euclidean or Hamming distance; construct coarse matching pairs based on the similarity.

[0084] S530. Based on coarse matching pairs, the homography matrix is ​​estimated using the Random Sampling Consensus Algorithm (RANSAC algorithm), and the set of interior points is obtained.

[0085] The RANSAC algorithm first randomly selects a small number of necessary data points to fit an initial homography matrix. Then, it uses this homography matrix to test all other points. Points that conform to the homography matrix are considered inliers, and those that do not are considered outliers. This process is repeated multiple times, and finally, the homography matrix with the most inliers is selected as the best model. All corresponding inliers are recorded.

[0086] S540. Obtain the region image for matching the baseline drawing texture based on the set of interior points, and transform the region image according to the homography matrix to obtain the standard coloring image.

[0087] By analyzing the spatial distribution of interior points in the interactive coloring image, the boundaries that match the outline of the baseline drawing are identified to obtain the region image; the pixels of the non-region image are masked.

[0088] The region image is actually derived directly from the baseline texture map. Therefore, the homography matrix can be directly used as the transformation matrix between the two. By transforming the region image, the standard color image can be obtained.

[0089] S600. Extract color features from the region image, color the baseline line drawing texture based on the color features, and generate a color texture map, including:

[0090] S610. Obtain the preset saturation threshold, and replace the pixel colors with saturation lower than the saturation threshold in the region image (standard coloring image) with transparent colors to obtain the corrected image; this step removes low-saturation pixels, such as pencil marks, paper background color, and insufficiently colored areas; the saturation threshold can be adjusted according to the painting characteristics of the standard coloring image, for example, different thresholds can be set for pencil painting, watercolor painting, and acrylic painting.

[0091] S620. Extract color features from the corrected image, wherein the color features include dominant hue, color distribution, and region coloring, including:

[0092] The color space of the corrected image is clustered to determine the number of main color clusters. The center color value of each color cluster is extracted to obtain multiple main colors. The proportion of each color cluster is analyzed and the saturation range and brightness range of the main colors are recorded. The presence of complementary or analogous colors among the main colors is analyzed and the hue difference is calculated to obtain the main color tone features.

[0093] By dividing different color block regions through color clustering, calculating the color similarity between adjacent color block regions, calculating the distance between the centers of adjacent color block regions, and calculating the area of ​​each color block region, the color distribution characteristics are obtained.

[0094] For each color region, calculate the color uniformity and smear density, and use a Gabor filter to calculate the brush stroke direction to obtain the region coloring features;

[0095] S630. Input the baseline line art texture and color features into the pre-trained coloring model to color the baseline line art texture;

[0096] Conditional Generative Adversarial Network (cGAN) can be used as the coloring model architecture, and it can be trained in advance with a large number of images; the baseline line drawing texture and color features are input into the trained coloring model, and the output is a color texture map;

[0097] S700: Render the 3D model based on the color texture map.

[0098] The newly generated color texture map is applied to the 3D model in real time to update the model's appearance.

[0099] By using AR tracking technology, the updated 3D model is accurately rendered and overlaid on the identified image area in the camera view, achieving an AR display that combines the virtual and real worlds.

[0100] The baseline textures in this application are entirely derived from the 3D model itself. Therefore, the final generated color texture maps can be seamlessly applied back to the model without any additional optimization. Users do not need to understand 3D modeling and UV unwrapping; they can directly obtain a virtual object with a complex 3D appearance simply by operating it like a simple coloring exercise. This achieves a "what you see is what you get" simplified creation experience, representing a significant breakthrough in both the democratization of technology and the art of creation.

[0101] Example 2:

[0102] This embodiment provides a model rendering system based on mixed reality interactive creation, including:

[0103] The first module is used to acquire the 3D model and calculate the optimal rendering perspective based on the geometry of the 3D model.

[0104] The second module is used to obtain the preset best viewing angle, and to calculate the final rendering angle by weighting the best viewing angle and the best rendering angle.

[0105] The third module is used to project the 3D model based on the final rendering perspective and generate baseline texture maps.

[0106] The fourth module is used to obtain interactive coloring images with the baseline line art texture as the base image;

[0107] The fifth module is used to identify interactive coloring images, extract the region images that match the baseline line art texture, and perform geometric correction.

[0108] The sixth module is used to extract color features from the region image, color the baseline line drawing texture based on the color features, and generate a color texture map.

[0109] The seventh module is used to render the 3D model based on color texture maps.

[0110] As an optional implementation, the first module includes:

[0111] The first unit is used to obtain the surface information of the 3D model and calculate the area of ​​each surface and the normal vector of the surface.

[0112] The second unit is used to construct the relative angle by calculating the difference between the normal vector of each facet and the optimal rendering view vector;

[0113] The third unit is used to construct the objective function based on the relative angle and area of ​​each facet;

[0114] The fourth unit is used to obtain the optimal rendering view vector by minimizing the objective function.

[0115] As an optional implementation, the third module includes:

[0116] The fifth unit is used to calculate the viewpoint occlusion based on the geometry of the 3D model;

[0117] The sixth unit is used to determine whether the occlusion of the viewing angle is less than a preset threshold.

[0118] The seventh unit is used to invert the normal vector of the final rendering view if the judgment result is yes, to obtain the final rendering reverse view; and to perform projection rendering on the 3D model based on the final rendering view and the final rendering reverse view.

[0119] The eighth unit is used to calculate the first rendering viewpoint based on the normal vectors of the facets of the 3D model if the judgment result is negative, and to project the facets according to the first rendering viewpoint; delete the projected facets and update the 3D model; recalculate the second rendering viewpoint based on the updated 3D model, and repeat the projection and facet deletion process until all facets are projected.

[0120] As an optional implementation, the fifth unit includes:

[0121] The first subunit is used to calculate the dihedral angle between every two adjacent facets. If the dihedral angle is greater than a first preset value, it is marked as an occlusion angle, and the two facets connected by the occlusion angle are marked as occlusion facets.

[0122] The second sub-unit is used to calculate the area ratio of all occluded patches to the total area of ​​the 3D model, thus obtaining the occlusion area ratio.

[0123] The third subunit is used to identify connected occlusion features by analyzing the connectivity of the occlusion angle. The connected occlusion features include isolated occlusion pairs, occlusion chains, occlusion loops, and occlusion clusters.

[0124] The fourth sub-unit is used to calculate the viewing angle occlusion degree based on the connectivity occlusion features and the occlusion area ratio.

[0125] Example 3:

[0126] Corresponding to the above method embodiments, this embodiment also provides a model rendering device based on mixed reality interactive creation. The model rendering device based on mixed reality interactive creation described below can be referred to in correspondence with the model rendering method based on mixed reality interactive creation described above.

[0127] Figure 2 This is a block diagram illustrating a model rendering device 800 based on mixed reality interactive creation, according to an exemplary embodiment. Figure 2As shown, the model rendering device 800 based on mixed reality interactive creation includes a processor 801 and a memory 802. The model rendering device 800 may also include one or more of the following: a multimedia component 803, an input / output (I / O) interface 804, and a communication component 805. The processor 801 controls the overall operation of the model rendering device 800 to complete all or part of the steps in the aforementioned model rendering method based on mixed reality interactive creation. The memory 802 stores various types of data to support the operation of the model rendering device 800. This data may include, for example, commands for any application or method operating on the model rendering device 800, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0128] Multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals.

[0129] The received audio signals can be further stored in memory 802 or transmitted via communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as keyboards, mice, buttons, etc. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the mixed reality interactive creation model rendering device 800 and other devices. Wireless communication includes, for example, Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0130] Example 4:

[0131] Corresponding to the above embodiment of the model rendering method based on mixed reality interactive creation, this embodiment also provides a readable storage medium. The readable storage medium described below can be referred to in correspondence with the model rendering method based on mixed reality interactive creation described above.

[0132] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described embodiment of the model rendering method based on mixed reality interactive creation.

[0133] Specifically, the readable storage medium can be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A model rendering method based on mixed reality interactive creation, characterized in that, include: Obtain the 3D model, and calculate the optimal rendering viewpoint based on the geometry of the 3D model, including: Obtain the surface information of the 3D model, and calculate the area and normal vector of each surface. The relative angle is constructed by calculating the difference between the normal vector of each facet and the optimal rendering view vector; Based on the relative angles and areas of each facet, construct the objective function: The optimal rendering view vector is obtained by minimizing the objective function. Obtain the preset optimal viewing angle, and calculate the final rendering angle by weighting the optimal viewing angle and the optimal rendering angle. Based on the final rendering perspective, the 3D model is projected to generate a baseline texture map, including: Calculate the viewpoint occlusion based on the geometry of the 3D model; Determine whether the occlusion level of the viewing angle is less than a preset threshold; If the judgment result is yes, then the normal vector of the final rendering view is inverted to obtain the final rendering reverse view; the 3D model is then projected and rendered based on the final rendering view and the final rendering reverse view. If the judgment result is negative, the first rendering viewpoint is calculated based on the normal vectors of the facets of the 3D model, and projection is performed according to the first rendering viewpoint; the projected facets are deleted, and the 3D model is updated; the second rendering viewpoint is recalculated based on the updated 3D model, and the projection and facet deletion processes are repeated until all facets are projected. Obtain an interactive coloring image with the baseline line art texture as the base image; The interactive coloring image is identified, and the region image that matches the baseline line drawing is extracted and geometrically corrected. Color features are extracted from the region image, and the baseline line drawing texture is colored based on the color features to generate a color texture map; The 3D model is rendered based on color texture mapping.

2. The model rendering method based on mixed reality interactive creation according to claim 1, characterized in that, The viewpoint occlusion is calculated based on the geometry of the 3D model, including: Calculate the dihedral angle between any two adjacent facets. If the dihedral angle is greater than a first preset value, mark it as an occlusion angle and mark the two facets connected by the occlusion angle as occlusion facets. Calculate the area ratio of all occluded surfaces to the total area of ​​the 3D model to obtain the occlusion area ratio; By analyzing the connectivity of the occlusion angle, connected occlusion features are identified, including isolated occlusion pairs, occlusion chains, occlusion loops, and occlusion clusters. The occlusion degree is calculated based on the connectivity occlusion features and the occlusion area ratio.

3. The model rendering method based on mixed reality interactive creation according to claim 1, characterized in that, The interactive coloring image is identified, and the region image that matches the baseline line art texture is extracted and geometrically corrected, including: Key points are identified from interactive coloring images and baseline textures using a local feature descriptor algorithm; Similarity matching is performed on the descriptors of key points, and coarse matching pairs are obtained by filtering based on similarity. Based on coarse matching pairs, the homography matrix is ​​estimated using a random sampling consensus algorithm, and the set of interior points is obtained. The region image for matching the baseline texture is obtained from the set of interior points. The region image is then transformed according to the homography matrix to obtain the standard coloring image.

4. The model rendering method based on mixed reality interactive creation according to claim 1, characterized in that, Extracting color features from the region image and coloring the baseline line art texture based on the color features includes: Obtain a preset saturation threshold, and replace the colors of pixels in the region image with transparent colors if the saturation is lower than the saturation threshold to obtain the corrected image; Color features are extracted from the corrected image, including dominant hue, color distribution, and region coloring. The baseline line art texture and color features are input into a pre-trained coloring model to color the baseline line art texture.

5. A model rendering system based on mixed reality interactive creation, characterized in that, include: The first module is used to acquire the 3D model and calculate the optimal rendering perspective based on the geometry of the 3D model. The first module includes: The first unit is used to obtain the surface information of the 3D model and calculate the area of ​​each surface and the normal vector of the surface. The second unit is used to construct the relative angle by calculating the difference between the normal vector of each facet and the optimal rendering view vector; The third unit is used to construct the objective function based on the relative angle and area of ​​each facet; The fourth unit is used to obtain the optimal rendering view vector by minimizing the objective function; The second module is used to obtain the preset best viewing angle, and to calculate the final rendering angle by weighting the best viewing angle and the best rendering angle. The third module is used to project the 3D model based on the final rendering perspective and generate a baseline texture map; the third module includes: The fifth unit is used to calculate the viewpoint occlusion based on the geometry of the 3D model; The sixth unit is used to determine whether the occlusion of the viewing angle is less than a preset threshold. The seventh unit is used to invert the normal vector of the final rendering view if the judgment result is yes, to obtain the final rendering reverse view; and to perform projection rendering on the 3D model based on the final rendering view and the final rendering reverse view. The eighth unit is used to calculate the first rendering viewpoint based on the normal vectors of the facets of the 3D model if the judgment result is negative, and to project the image based on the first rendering viewpoint; to delete the projected facets and update the 3D model; to recalculate the second rendering viewpoint based on the updated 3D model, and to repeat the projection and facet deletion process until all facets are projected. The fourth module is used to obtain interactive coloring images with the baseline line art texture as the base image; The fifth module is used to identify interactive coloring images, extract the region images that match the baseline line art texture, and perform geometric correction. The sixth module is used to extract color features from the region image, color the baseline line drawing texture based on the color features, and generate a color texture map. The seventh module is used to render the 3D model based on color texture maps.

6. The model rendering system based on mixed reality interactive creation according to claim 5, characterized in that, The fifth unit includes: The first subunit is used to calculate the dihedral angle between every two adjacent facets. If the dihedral angle is greater than a first preset value, it is marked as an occlusion angle, and the two facets connected by the occlusion angle are marked as occlusion facets. The second sub-unit is used to calculate the area ratio of all occluded patches to the total area of ​​the 3D model, thus obtaining the occlusion area ratio. The third subunit is used to identify connected occlusion features by analyzing the connectivity of the occlusion angle. The connected occlusion features include isolated occlusion pairs, occlusion chains, occlusion loops, and occlusion clusters. The fourth sub-unit is used to calculate the viewing angle occlusion degree based on the connectivity occlusion features and the occlusion area ratio.

Images