An AI and MR-based virtual three-dimensional scene drawing and building method

Abstract

The application belongs to the technical field of mixed reality, and relates to a virtual three-dimensional scene drawing and building method based on AI and MR, which comprises the following steps: dynamically constructing and maintaining a three-dimensional semantic map; when AI converts graffiti into a three-dimensional decoration, determining a semantic role and outputting a space constraint vector, so as to perform semantic matching search in the three-dimensional semantic map, and obtain a plurality of candidate poses; after collision pose elimination by interpenetration detection, the pose with the minimum deformation energy is selected as a recommended adsorption pose; after user confirmation, the cosine distance between the style embedding vector of the decoration and the average style vector of the existing objects in the scene is calculated, and if the cosine distance exceeds a threshold value, the decoration is presented in the MR space after tone migration. The application realizes automatic resolution of incremental space conflicts, intelligent matching of semantic association and cross-batch style consistency verification, solves the defects that the prior art cannot adapt to the process of gradually adding decorations, and improves the coordination and user experience of MR scene building.

Description

Technical Field

[0001] This invention belongs to the field of mixed reality technology and relates to a method for rendering and building virtual 3D scenes based on AI and MR. Background Technology

[0002] MR immersive AI painting tool is an interactive system that deeply integrates artificial intelligence painting generation capabilities with mixed reality environments. Users can create 3D doodles in real physical spaces using gestures or controllers. AI will transform the doodles into artistic 3D decorations in real time, and users can freely drag and drop and arrange decorations in the scene to build personalized virtual art spaces.

[0003] Existing MR scene layout technologies mainly fall into two categories. The first category utilizes spatial scanning data to generate a virtual scene framework and employs AI algorithms to optimize the overall layout of scene elements. The second category constructs and solves a layout optimization model to achieve real-time layout adjustments of virtual objects, such as the virtual object layout optimization method in augmented reality described in Chinese Patent Publication No. CN121458934A. Both approaches share the characteristic of focusing on the initial construction or one-time global layout of the scene, meaning that the positions of all objects are determined at the outset.

[0004] However, the usage mode of MR immersive AI painting tools is an incremental dressing-up process where users gradually add decorations in batches within an existing scene. Existing technology cannot adapt to this process, and its main drawback is: (1) Lack of real-time perception and automatic solution for spatial interweaving during incremental addition. When a user drags and drops a new decoration in a scene where several objects have already been placed, the existing solution only provides static collision detection and cannot perform incremental automatic adjustment of the interweaving and occlusion between the new object and the already placed objects, which causes the user to have to make repeated manual fine adjustments.

[0005] (2) Lack of semantic understanding of the functional attributes of physical space. Existing solutions can only identify the geometric features of a plane and cannot distinguish the essential differences in function between different types of physical areas. For example, a desktop is suitable for placing ornaments, while tree branches are suitable for hanging decorations. When a user hangs an AI-generated lantern under a tree branch, the system cannot automatically identify the tree branch as a hanging area, nor can it understand the spatial behavior attributes of the lantern as a hanging object. The user can only manually drag and position it based on their senses, which is cumbersome and yields unstable results.

[0006] (3) There is a lack of style consistency verification and reconciliation mechanism between different batches of AI-generated items. The costumes generated by users at different times may belong to different aesthetic styles. The existing technology does not perceive and reconcile these styles, resulting in a visual style fragmentation in the final scene and affecting the overall artistic expression. Summary of the Invention

[0007] In view of this, in order to solve the problems mentioned in the background technology, a method for virtual 3D scene rendering and construction based on AI and MR is proposed.

[0008] The objective of this invention can be achieved through the following technical solution: This invention provides a method for drawing and building a virtual three-dimensional scene based on AI and MR, including: collecting environmental data through MR sensors, identifying and assigning scores to physical areas that can be placed or hung, and dynamically constructing and maintaining a three-dimensional semantic map by combining the attributes, occupancy status and user change feedback of AI costumes. When the AI ​​converts graffiti into 3D costumes, it parses the object category, determines the semantic role as a hanging dependency, surface support, or surrounding companion, and outputs the corresponding spatial constraint vector. Based on the semantic role and spatial constraint vector of the costume currently dragged and dropped by the user, a semantic matching search is performed in the 3D semantic map to obtain multiple candidate poses; Spatial interleaving detection is performed on each candidate pose to eliminate conflicting poses. The deformation energy between the remaining candidate poses and the user's hand grasping posture is calculated, and the candidate pose with the smallest deformation energy is selected as the recommended adsorption pose. In response to the user's confirmation of the recommended snapping pose, the cosine distance between the style embedding vector of the costume and the average style vector of existing objects in the scene is calculated. If the cosine distance is greater than a preset distance threshold, the costume is rendered in MR space after color transfer processing.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention constructs a dynamically updatable 3D semantic map. Each time a user places or removes an AI-powered decoration, the semantic label, bounding box, and occupied area of ​​the object are synchronized to the map, and the area score is dynamically updated based on the user's actions of overturning the recommendation. At the same time, semantic matching search and spatial interleaving detection are performed during drag-and-drop, and bounding box intersection tests are performed on each candidate pose to eliminate conflicting poses. When the user gradually adds new decorations to the existing scene, the system can perceive and automatically avoid the interleaving and occlusion between new objects and already placed objects in real time, without requiring the user to repeatedly manually fine-tune the position.

[0010] (2) This invention analyzes the semantic role of each AI costume and outputs the corresponding spatial constraint vector. During the drag-and-drop process, candidate poses that meet the spatial constraints are retrieved in the three-dimensional semantic map based on the semantic role, and the recommended pose that is closest to the user's hand grasping posture is selected by deformation energy calculation. It can understand the semantic coexistence relationship between objects and automatically guide the costume to a spatial position that conforms to its semantic role, avoiding semantically absurd placement results.

[0011] (3) The present invention extracts the style embedding vector of each costume and calculates the cosine distance between it and the average style vector of existing objects in the scene. If it exceeds the preset distance threshold, the hue of the new costume is shrunk and mapped to the main hue of the scene, thereby achieving style unity while maintaining texture details and avoiding style fragmentation across batches. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the implementation steps of the method of the present invention; Figure 2 This is a logical diagram illustrating the dynamic construction and maintenance of a three-dimensional semantic map according to the present invention. Figure 3 This is a flowchart illustrating the execution of the tone migration process in this invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 As shown, the present invention provides a method for drawing and building a virtual 3D scene based on AI and MR, including: S1. Collecting environmental data through MR sensors, identifying and assigning scores to physical areas that can be placed or hung, and dynamically constructing and maintaining a 3D semantic map by combining the attributes, occupancy status and user change feedback of AI decorations.

[0016] Specifically, step S1 corresponds to the initialization process of the MR terminal performing spatial perception and regional classification scoring on the physical environment after the user enters the scene, as well as the continuous dynamic maintenance of the three-dimensional semantic map throughout the entire creation process.

[0017] In this embodiment, identifying and assigning scores to physical areas that can be placed or hung includes: The MR terminal synchronously acquires depth data and color images for each frame through an RGB-D camera; the depth data records the distance value from the corresponding physical surface point to the camera's optical center for each pixel, and the color image is a three-channel color image of red, green, and blue.

[0018] To perform planar detection and point cloud clustering on depth data, the following steps are specifically executed: Pixels whose distance values ​​exceed the range or return invalid values ​​in the depth data are marked as invalid points, and the rest are marked as valid depth pixels. Each valid depth pixel is back-projected to the camera coordinate system through the camera intrinsic parameter matrix to obtain a spatial point cloud.

[0019] For each point in the spatial point cloud, principal component analysis is performed on its nearest neighbor in the spatial neighborhood, and the eigenvector corresponding to the smallest eigenvalue is used as the local surface normal.

[0020] The point closest to the camera's optical center is selected from the unclassified points as the seed point. The seed point is then expanded to neighboring points under the condition that the angle between the normals is less than a preset angle threshold, forming a spatially continuous planar point cluster. The preset angle threshold is determined during the offline calibration stage by repeatedly detecting planes with different flatness at different distances, and is the maximum angle value that distinguishes between coplanar and non-coplanar points under each detection condition.

[0021] Perform overall plane fitting on the formed point cluster, and calculate the root mean square distance from all points in the point cluster to the fitting plane as the fitting residual. If the fitting residual is greater than a preset multiple of the standard deviation of depth measurement noise, the point cluster is split into sub-clusters and refitted. If the sub-cluster residual still exceeds the standard, the formed point cluster is discarded; otherwise, the corresponding planar region is retained.

[0022] The depth measurement noise standard deviation is read from the factory calibration parameters of the RGB-D camera. The determination process of the depth measurement noise standard deviation is as follows: During the factory calibration stage, the RGB-D camera is positioned facing a standard plane with known flatness, and multiple frames of depth data are continuously acquired at a specified distance. The depth value of the corresponding pixel area of ​​the standard plane in each frame is calculated to be different from the known true distance of the standard plane. The standard deviation of the resulting difference sequence is taken as the depth measurement noise standard deviation at the specified distance. The above determination is repeated for multiple distance levels, and the minimum value is selected as the depth measurement noise standard deviation used in this invention.

[0023] The preset multiple is determined during the offline calibration phase of the device in the following way: using an RGB-D camera, repeated depth measurements are performed on a standard plane with known flatness at different distances. Plane fitting is performed on the measured point cloud. The ratio distribution of the fitting residual to the standard deviation of the depth measurement noise is statistically analyzed. The third and fourth quartiles of the ratio distribution are rounded up as the preset multiple. If the standard plane does not cover the current ranging range in the factory calibration, the preset multiple corresponding to the nearest ranging range is used for interpolation estimation.

[0024] Repeat the seed point selection and region growth process until all spatial point clouds are classified or discarded, resulting in several spatially continuous planar regions.

[0025] Semantic segmentation is performed on the color image to obtain predefined scene semantic category labels for each pixel; specifically, the color image is input into a lightweight convolutional neural network pre-deployed on the terminal side, and the network is trained on an image dataset labeled with scene semantic category labels and then embedded in the terminal.

[0026] The point cloud of the planar region is projected onto the pixel plane of the color image. The semantic category labels of the pixels falling within the projection range of each planar region are statistically analyzed, and the semantic category with the highest statistical percentage is determined as the semantic category of the planar region.

[0027] If a planar region's semantic category belongs to the horizontal support semantic set, which includes surface categories such as ground, tabletop, countertop, bed surface, and chair surface that provide horizontal support in the physical world, then the planar region is determined to be a placeable region. An initial placeability score is calculated based on the area, flatness parameter, and angle of the normal relative to the gravity direction of the planar region. Here, the area refers to the two-dimensional projection convex hull area of ​​the point cloud of the planar region calculated by the Graham scan algorithm on its fitting plane; the flatness parameter is determined by the root mean square distance from each sampling point in the planar region to the fitting plane; and the angle of the normal is the angle between the normal of the region's fitting plane and the unit vector of the gravity direction.

[0028] The region point cloud is projected onto the gravity direction axis and a horizontal plane perpendicular to the gravity direction, respectively. The projection span in the gravity direction is taken as the distribution scale of the region point cloud in the gravity direction, and the maximum projection span in the horizontal plane is taken as the distribution scale of the region point cloud in the horizontal direction.

[0029] If a planar region's semantic category belongs to the vertically prominent semantic set, which includes structural categories such as tree trunks, columns, beams, and hooks that provide vertical or cantilever support in the physical world, and the distribution scale of the region's point cloud in the gravity direction is greater than its distribution scale in the horizontal direction, then the planar region is determined to be a hangable region.

[0030] The difference between the coordinates of the centroid of the planar region in the direction of gravity and the coordinates of the current ground reference plane in the direction of gravity is used as the vertical height of the planar region relative to the ground; the maximum span of the convex hull projected onto the horizontal plane perpendicular to the direction of gravity by the point cloud of the planar region is calculated as the horizontal extension length; the maximum distance of the point cloud of the planar region from the direction of the supporting trunk in the direction perpendicular to the normal of the fitted plane of the region is calculated as the protrusion depth.

[0031] The vertical height, horizontal extension length, and protrusion depth are linearly normalized using the maximum value of the corresponding values ​​of all hangable regions in the current frame as the denominator, and then the weighted arithmetic average is calculated to obtain the initial hangability score.

[0032] It should be noted that the calculation process for the initial placeability score is as follows: The area normalization value is obtained by performing a ratio operation with the maximum area of ​​all areas determined to be placeable in the current frame as the denominator and the area of ​​the planar region as the numerator.

[0033] The minimum value of the set of distances from each sampling point in the planar region to the fitted plane is selected as the numerator, and the flatness parameter is selected as the denominator. The flatness normalization value is obtained by ratio calculation. When the value of the flatness parameter is less than the standard deviation of the depth measurement noise, the flatness normalization value is directly set to 1.

[0034] The initial placement score is obtained by taking the weighted arithmetic average of the normalized area value, the normalized flatness value, and the cosine of the included angle of the normal.

[0035] Reference Figure 2 As shown, in this embodiment, the dynamic construction and maintenance of the three-dimensional semantic map includes: Using the identified placeable and hangable areas as map nodes, the semantic category label, functional score, bounding box spatial coordinates, and spatial adjacency relationships between nodes are used as node attributes to construct an initial 3D semantic map. The functional score includes an initial placeability score for placeable areas and an initial hangability score for hangable areas; the spatial adjacency relationships between nodes are determined based on whether the corresponding point clouds of two nodes have adjacent faces or intersecting boundaries in space. The specific determination process is as follows: Take the local surface normals of the point clouds corresponding to the two nodes, which have been obtained through principal component analysis in the above-mentioned plane detection stage. Then, use the absolute value of the vector dot product of the local surface normals of the two nodes with the inverse cosine function to obtain the angle between the normals of the fitted plane of the two nodes.

[0036] Using the centroid of the planar region as a point, construct the point normal equation of the fitting plane for two nodes by combining the local surface normal. Label one node as the first node and the other node as the second node. Substitute the three-dimensional coordinates of each point in the point cloud of the first node into the point normal equation of the fitting plane of the second node. The absolute value of the resulting algebraic distance is taken as the perpendicular distance from the first node to the fitting plane of the second node. Take the minimum value among all perpendicular distances as the point-to-surface shortest distance from the first node to the fitting plane of the second node. Similarly, calculate the point-to-surface shortest distance from the second node to the fitting plane of the first node.

[0037] If the angle between the normals of the fitting planes of two nodes is less than the adjacent angle threshold, and the shortest distance between the two points is less than the adjacent distance threshold, then the two nodes are determined to have an adjacent surface relationship. The adjacent angle threshold is taken as the upper quartile value obtained by statistically analyzing the angles between the normals of multiple sets of known coplanar physical surface samples in the offline stage (the sample size is not less than 100 sets). The adjacent distance threshold is taken as a preset multiple of the standard deviation of the depth measurement noise used to judge the fitting residual in the above-mentioned plane detection stage. The basis for reuse is that judging whether a node's point cloud is adjacent to the other's fitting plane is geometrically equivalent to judging whether each point in the point cloud can be attributed to the other's fitting plane, which is consistent with the residual logic of judging whether a point belongs to the fitting plane in the plane detection stage. Therefore, the same distance benchmark is used.

[0038] Project the point clouds of each of the two nodes onto a horizontal reference plane perpendicular to the direction of gravity in the world coordinate system to obtain two sets of two-dimensional projected point sets. Use the Graham scan algorithm to calculate the two-dimensional convex hull boundaries of the two sets of two-dimensional projected point sets respectively. If the overlapping area of ​​the two two-dimensional convex hull boundaries is greater than 0, or the shortest Euclidean distance between the two two-dimensional convex hull boundaries is less than the adjacency distance threshold, then it is determined that there is a boundary intersection relationship between the two nodes.

[0039] When two nodes satisfy at least one of the following conditions, namely, adjacent face relationship or boundary intersection relationship, it is determined that there is a spatial adjacency relationship between the two nodes, and the adjacent edge is recorded in the three-dimensional semantic map.

[0040] In response to the confirmation operation of the placement of a new costume, the semantic category label, bounding box, and spatial area occupied by the new costume are obtained. The spatial area is then spatially overlaid with the existing nodes intersecting in the map for analysis: For each existing node that intersects with the bounding box of the costume, the ratio of the intersection volume to the bounding box volume of the existing node is calculated as the occupancy ratio. The functional score of the occupied node is then reduced and updated according to the corresponding occupancy ratio. Specifically, the difference between 1 and the occupancy ratio is multiplied by the original functional score of the occupied node to obtain the updated functional score. The new costume is then registered as a new node in the map, and a semantic subordinate edge is established between it and the occupied node.

[0041] In response to the removal of a placed decoration, the corresponding node is deregistered from the map, all semantic dependent edges pointing to the occupied node are traversed from that node, the functional rating of the occupied node is restored to the rating value stored in the regional rating archive before the decoration was registered, and the corresponding semantic dependent edges are removed.

[0042] In response to the user's change of the recommended snap pose, the bounding boxes of all nodes in the 3D semantic map are traversed. The node containing the location coordinates of the original recommended snap pose and the node closest to that location coordinate is determined as the spatial region where the original recommended snap pose is located. Similarly, the spatial region where the user's final confirmed pose is located is determined.

[0043] Record the displacement path from the original recommended snap pose to the user's final confirmed pose, and calculate the Euclidean distance between the position coordinates of the two poses as the displacement path length.

[0044] The functional score of the spatial region where the original recommended adsorption pose is located is reduced by multiplying the displacement path length by the unit distance attenuation rate. The unit distance attenuation rate is the ratio of the arithmetic mean of the current functional scores of all nodes to the diagonal length of the scene bounding box. The physical meaning of this ratio is the score value corresponding to the average unit length of each node in the scene. Simultaneously, the functional score of the spatial region where the user's final confirmed pose is located is increased by the same magnitude as the reduction. The cumulative frequency of change operations is used as the confidence reference for the corresponding region's score update.

[0045] It should be noted that the confidence level serves the following purpose: when new outfits trigger semantic matching searches, the more frequently a region changes, the more its functional score will be multiplied by a decay confidence coefficient based on the equivalent score. The decay confidence coefficient is determined as follows: when the change frequency is 0 or 1, the decay confidence coefficient is 1; otherwise, the decay confidence coefficient is the reciprocal of the logarithm of the change frequency. The more frequent the changes, the lower the decay confidence coefficient, thus enabling the score update to have a self-correcting ability based on user behavior feedback.

[0046] S2. When the AI ​​converts the graffiti into a 3D costume, it parses the object category, determines the semantic role as a hanging dependency, surface support, or surrounding companion, and outputs the corresponding spatial constraint vector.

[0047] Specifically, when processing user doodles, the cloud-based AI generation service, in addition to generating 3D geometric meshes and surface textures, also outputs object category names simultaneously. The generation method is as follows: the stroke sequence data of the doodle is sent to a cloud-based AI-trained doodle classifier. The classifier outputs the category name string with the highest confidence. The string is sent to the MR terminal along with the asset attribute file. Typical category names include, but are not limited to: picture frame, sculpture, vase, lamp, bird, cloud, vine, bell, etc.

[0048] In this embodiment, the process of determining the semantic role includes: Using the semantic category tags of the 3D costume as an index, the system retrieves corresponding typical spatial existence form records from a pre-defined object semantic association knowledge base. These typical spatial existence forms include those suspended high up, placed on a plane, and surrounding or attached to objects.

[0049] The construction process of the object semantic association knowledge base is as follows: In the offline stage before method deployment, for decorative objects, the 3D spatial position parameters of objects relative to their supporting surface or attached object in actual scene instances are statistically analyzed from publicly available 3D scene datasets and scene image datasets. For each category, the spatial position parameters of all instances are clustered from bottom to top: initially, the parameter values ​​of each instance are treated as an independent cluster. The Euclidean distance between any two clusters is calculated, and the two closest clusters are merged. This process is repeated until the nearest distance between any two clusters is greater than the merging distance threshold. After clustering, the arithmetic mean of the parameters of all instances in the cluster with the largest sample size is taken as the typical spatial position parameter of the corresponding category and written into the object semantic association knowledge base. The merging distance threshold is the magnitude of the standard deviation vector of the spatial position parameters of all instances in the corresponding category.

[0050] Each record in the object semantic association knowledge base contains three fields: I. Typical spatial form field, which is an enumerated value, taking one of three values: hanging high, placed on a plane, or surrounding and attached to an object; II. Typical Suspension Offset Ratio field has a valid value only when the object is suspended at a high position in a typical space. The value is the statistical median value of the ratio of the distance from the suspension point to the center of the object to the vertical dimension of the object in the current category of object instances. III. The surrounding density field has a valid value only when the typical spatial form is surrounding the attached object. It is the median value of the ratio of the distance from the center of the object to the surface of the attached object to the radius of the outer sphere of the bounding box of the attached object in the current category of object instances.

[0051] After a successful match, the typical spatial existence form enumeration values ​​are processed separately: hanging high points to the semantic role of the hanging dependency, placed on a plane points to the surface support, and surrounding the dependent object points to the surrounding companion. The dependent target category of the record association is extracted from the object semantic association knowledge base.

[0052] It should be noted that if the category name is not found in the knowledge base, the MR terminal adopts a conservative strategy and defaults to determining the semantic role as a surface support, because surface placement is the most common form of decoration. The terminal then reports the current category name and the determination result to the cloud in the background for the cloud to expand the knowledge base in the future.

[0053] In this embodiment, the output process of the spatial constraint vector includes: The spatial location reference rule for determining the costume based on semantic roles is as follows: Hanging Dependency: Using the lower edge of the hangable area as the height reference benchmark, specifically referring to the minimum coordinate value of the hanging area's bounding box in the direction of gravity, the lower limit of the desired height range is determined by subtracting the vertical dimension of the decoration's bounding box from the coordinates of the lower edge of the hanging area. The vertical dimension of the decoration's bounding box is determined by taking the component parallel to the direction of gravity from the length, width, and height values ​​of the bounding box in the asset attribute file. The upper limit of the desired height range is obtained by subtracting the product of the vertical dimension and the adjustment factor from the coordinates of the lower edge of the hanging area. The adjustment factor is determined by the typical hanging offset ratio recorded by the decoration category in the object semantic association knowledge base. The spacing reference benchmark and desired spacing range fields are left blank.

[0054] Surface support: The height reference is taken from the surface height values ​​of all placeable region nodes in the 3D semantic map. Specifically, the surface height value refers to the coordinates of the centroid of the fitting plane of the placeable region in the direction of gravity. Both the lower and upper limits of the desired height interval are taken from the surface height values ​​themselves, used to compress the interval into a single point. The spacing reference and desired spacing range fields are left blank.

[0055] Surrounding Companion: The center of the bounding box of the matched object is used as the spacing reference point. The sum of the outer sphere radius of the bounding box of the object and the outer sphere radius of the decoration bounding box is used as the lower limit of the expected spacing. The magnification factor is determined by the surrounding density corresponding to the object category. The product of the lower limit of the expected spacing and the magnification factor is used as the upper limit of the expected spacing. The height reference benchmark and expected height range fields are left blank.

[0056] The desired height range and desired spacing range are combined with the reference baseline information to form a spatial constraint vector.

[0057] S3. Based on the semantic role and spatial constraint vector of the currently dragged and dropped costume, perform semantic matching search in the 3D semantic map to obtain multiple candidate poses.

[0058] Specifically, step S3 corresponds to the process of filtering target regions that meet semantic expectations and generating a set of candidate placement poses in the 3D semantic map based on the semantic role and spatial constraint vector of the costume when the MR terminal detects that the user performs a gesture of grabbing and dragging the costume. The drag-and-drop gesture trigger analysis process is as follows: The MR terminal outputs the three-dimensional position coordinates of the user's hand in the world coordinate system in real time. When the spatial distance between the three-dimensional position coordinates of the hand and the bounding box of a certain costume is less than the grasping distance limit, and the hand posture classifier determines that it is a clenched fist gesture, the drag-and-drop start event of the costume is triggered. The grasping distance limit is the statistical average value of the length of the finger joints in the user's palm, which is read from the human factors engineering parameters of the device.

[0059] In this embodiment, performing semantic matching search includes: Suspension Dependency: Read the lower and upper limits of the desired height range from the spatial constraint vector, traverse all suspendable region nodes in the 3D semantic map, and obtain the minimum coordinate value of the bounding box of each node in the gravity direction as the lower edge height; if the lower edge height is within the desired height range, the suspendable region is included in the candidate region set; if no suspendable region meets the condition after screening, all suspendable regions are directly included in the candidate region set as a fallback.

[0060] For each hangable region in the candidate region set, sort them in descending order according to the initial hangability score recorded in their node attributes. Calculate the arithmetic median of the initial hangability scores of all candidate regions. Regions with scores higher than the median are marked as priority regions, and regions with scores lower than or equal to the median are marked as candidate regions. Prioritize candidate pose generation within priority regions. If the number of candidate poses remaining after being eliminated by spatial interleaving detection in subsequent step S4 is insufficient to support effective selection of deformation energy (specifically, the number of candidate poses is 0 or only 1), then use regions in the candidate regions with the highest to lowest initial hangability scores to supplement candidate pose generation, and re-execute spatial interleaving detection until the number of remaining candidate poses is no less than 2.

[0061] If there are still fewer than two candidate poses after all priority and alternative regions have been activated, then the candidate poses are expanded from the current candidate region node to its directly adjacent (one-hop) node along the spatial adjacency edges recorded in the 3D semantic map. Adjacent nodes that meet the same height screening condition (the lower edge height of adjacent nodes is within the expected height range) are sorted in descending order according to the initial hangingability score and then supplemented to generate candidate poses. If the number of candidate poses remaining after one-hop expansion and spatial interleaving detection is still less than two, then the expansion continues to the next adjacent node until the requirements are met or there are no more adjacent nodes in the map that meet the height conditions for expansion.

[0062] The specific execution process for generating candidate poses for the hanging area is as follows: Determine the offset point directly below the lower edge of the hanging area. The specific offset is obtained by adding two indicators. One is half of the vertical dimension of the decoration's bounding box, and the other is the absolute value of the difference between the minimum coordinate value of the hanging area's bounding box in the gravity direction and the coordinate value of the centroid of the area fitting plane in the gravity direction. The former ensures that the decoration is located below the hanging area rather than embedded in it, while the latter compensates for the vertical thickness of the hanging area itself. The specific offset direction is the positive direction of gravity, and the obtained offset point is used as a candidate point.

[0063] Obtain the fitting plane normal vector of the hanging area, and take its reverse vector as the candidate orientation. The candidate orientation is the direction in which the front of the costume faces away from the hanging surface and towards the observable direction in space.

[0064] Candidate points and candidate orientations are combined into a candidate pose, thereby obtaining a set of candidate poses based on each suspendable region in the candidate region set.

[0065] Surface support: Read the desired height range (the upper and lower limits are the same, i.e., the target surface height value) from the spatial constraint vector, traverse all placeable area nodes in the 3D semantic map, obtain the surface height value of each node, and define the preset ratio of the vertical dimension of the decoration's bounding box as the height tolerance deviation. The preset ratio is determined during the equipment factory calibration stage by statistically analyzing the maximum distribution range of the average ground height deviation of similar objects on multiple typical placement planes for decorations of different semantic categories, and selecting the maximum deviation ratio covering 90% of the samples as the preset ratio for the corresponding category. If the absolute value of the difference between the surface height value and the desired height range is less than the height tolerance deviation, it is included in the candidate area set.

[0066] For each placeable region in the candidate region set, the same scoring and sorting method as the suspension dependencies described above is used (replacing the initial suspension score with the initial placeability score). Priority regions and candidate regions are divided by the median score. Priority regions are activated first. If, after spatial interleaving detection and elimination in step S4, fewer than two candidate poses remain, candidate regions are activated sequentially from highest to lowest placeability score to supplement candidate pose generation and re-detection until at least two candidate poses remain. If fewer than two remain after activating both priority and candidate regions, the system expands hop-by-hop towards adjacent nodes along spatial adjacency edges. Adjacent nodes that meet the height tolerance condition are sorted in descending order of initial placeability score to supplement candidate pose generation and detection until the quantity requirement is met or there are no more expandable adjacent nodes.

[0067] The specific execution flow for generating candidate poses for the placeable region is as follows: candidate points are generated on the fitting plane of the placeable region according to a mesh layout. The mesh spacing is taken as the length of the shortest side of the bounding rectangle of the decoration's bounding box projected onto the horizontal plane. The candidate points start from the centroid of the projected convex hull of the placeable region and expand outwards in a meshed manner until the candidate points exceed the boundary of the projected convex hull of the placeable region.

[0068] The candidate orientation of each candidate point is the opposite of the normal direction of the placeable area, to ensure that the bottom surface of the decoration is parallel to the normal of the supporting surface.

[0069] Similarly, a set of candidate poses is formed based on the candidate points and candidate orientations corresponding to each placeable region in the candidate region set.

[0070] Surrounding Companions: Read the dependent target category and expected spacing interval from the spatial constraint vector, and find one or more placed decoration nodes in the 3D semantic map whose semantic category label matches the dependent target category string, as the dependent objects.

[0071] Obtain the center coordinates of the bounding box of each attached object and the radius of the bounding box's circumscribed sphere. Using the center of the attached object's bounding box as the sphere's center, the lower limit of the desired spacing as the starting radius, the upper limit as the ending radius, and the diameter of the decoration's circumscribed sphere as the step size, generate multiple spherical layers. Sample on each sphere by stepping according to latitude and longitude angles. The longitude step size is 360° divided by the equal fraction of the longitude, and the latitude angle range is limited to -60° to +60°, so that the arc length between adjacent sampling points is not greater than the radius of the decoration's circumscribed sphere.

[0072] Each sampling point on each sphere is used as a candidate point, and the orientation of each candidate point is uniformly set to the direction vector from the candidate point to the center of the bounding box of the attached object, thus generating a set of candidate poses.

[0073] It should be noted that in the candidate poses generated by the above semantic roles, the three-dimensional position coordinates of the candidate points are directly used as the position components; the candidate orientation needs to have its direction vector converted into an orientation quaternion. Specifically, the candidate orientation direction vector is used as the target direction of the costume's forward axis in the world coordinate system, and the reverse vector of the gravity direction provided by the inertial measurement unit is used as the world upward reference. The candidate orientation direction vector is cross-producted with the world upward reference vector to obtain the right axis in the world coordinate system. Then, the candidate orientation direction vector is cross-producted with the right axis to obtain the corrected world upward axis. Thus, a three-axis orthogonal local rotation matrix is ​​constructed. The local rotation matrix is ​​converted into a unit quaternion, which is the orientation component of the corresponding candidate pose. The position component and the orientation component together constitute the complete rigid body transformation representation of the candidate pose.

[0074] S4. Perform spatial interleaving detection on each candidate pose to eliminate conflicting poses, calculate the deformation energy between the remaining candidate poses and the user's hand grasping posture, and select the candidate pose with the smallest deformation energy as the recommended adsorption pose.

[0075] In this embodiment, the step of performing spatial interleaving detection on candidate poses to eliminate conflicting poses includes: Construct a rigid body transformation matrix based on the position and orientation of the candidate pose. Apply the rigid body transformation matrix to the eight corner points of the axially aligned bounding box of the 3D costume in its own model coordinate system to obtain the coordinates of the eight corner points in the world coordinate system. Use the minimum and maximum values ​​of the eight corner points in each axis in the world coordinate system to determine the new spatial range of the 3D costume bounding box in the world coordinate system, which is used as the target bounding box.

[0076] Traverse all placed decoration nodes in the current 3D semantic map except for the currently dragged and dropped decoration itself, and perform spatial volume intersection calculations on the bounding box of each node and the target bounding box to obtain the intersection volume.

[0077] Calculate the arithmetic mean of the target bounding box volume and the bounding box volume of the currently placed decorations in the intersection test, and use the ratio of the minimum to the maximum value of the two bounding box volumes as the tolerance coefficient. The product of the arithmetic mean and the tolerance coefficient is determined as the intersection tolerance threshold.

[0078] When the intersection volume is greater than the intersection tolerance threshold, the candidate pose is determined to have a spatial conflict and is eliminated.

[0079] In this embodiment, calculating the deformation energy between the remaining candidate poses and the user's hand grasping posture includes: Obtain the position and orientation of the user's hand grasping posture, as well as the position and orientation of each remaining candidate posture.

[0080] Subtract the position component of the grasping posture from the position component of the remaining candidate pose to obtain the translation vector from the position of the grasping posture to the position of the remaining candidate pose.

[0081] Multiply the orientation quaternion of the candidate pose by the orientation conjugate quaternion of the grasping posture to obtain the relative rotation quaternion from the hand grasping posture to the candidate pose. Extract the real part of the relative rotation quaternion and calculate the minimum rotation angle using the following formula: ; In the formula, To obtain the minimum rotation angle, Let be the real part of the relative rotation quaternion; when When it is negative, take To ensure the minimum rotation angle is between 0 and If due to floating-point rounding... If the value is greater than 1, it is first truncated to 1 and then substituted into the inverse cosine for calculation; This formula is used to calculate the rotational deviation between the user's hand grasping posture and the candidate pose, so that the rotational cost (angle) can be converted into a dimension (arc length) that is consistent with the translational cost in the subsequent deformation energy calculation. It is the standard mathematical method for quaternion rotation calculation.

[0082] The translation and rotation weights are determined based on the semantic role of the 3D costume. The specific determination process is as follows: The translation vector magnitude of all candidate poses is used as the translation cost, and the arc length obtained by multiplying the minimum rotation angle by the radius of the circumscribed sphere of the costume bounding box is used as the rotation cost. The maximum value of the translation cost and the maximum value of the rotation cost are calculated for all candidate poses, and the ratio of the former to the latter is used as the scale normalization ratio.

[0083] If the semantic role is a dangling dependency, the translation weight is set to 1, and the rotation weight is the scale normalization ratio multiplied by the dangling sensitivity factor. The dangling sensitivity factor is obtained by: obtaining the fitting plane normal vectors of all candidate dangling regions selected by semantic matching search, calculating the angle between each normal vector and the mean of all normal vectors, taking the root mean square value of the angle as the normal dispersion, and then using the sum of the tangent of the normal dispersion and the constant 1 as the dangling sensitivity factor. The addend 1 corresponds to the baseline rotation weight when the normal dispersion is zero. When the orientation of all candidate dangling surfaces is completely consistent, the weight of the rotation cost is equal to the translation weight, and the rotation weight is not set to zero because the normal dispersion is zero.

[0084] If the semantic role is a surface support, the translation weight is set to 1, and the rotation weight is the scale normalization ratio multiplied by the support surface dependency factor. The support surface dependency factor is obtained by: obtaining the angle between the fitting plane normal vector of all candidate placement areas selected by semantic matching search and the gravity opposite direction vector, taking the arithmetic mean of all angles as the average tilt angle, and then taking the sine value of the average tilt angle as the support surface dependency factor.

[0085] If the semantic role is an encircling companion, set both the translation weight and rotation weight to 1.

[0086] By linearly weighting the translation vector magnitude and the displacement arc length converted from the minimum rotation angle, the deformation energy of each remaining candidate pose is obtained.

[0087] S5. In response to the user's confirmation of the recommended snapping pose, calculate the cosine distance between the style embedding vector of the costume and the average style vector of existing objects in the scene. If the cosine distance is greater than the preset distance threshold, perform tone transfer processing on the costume and present it in MR space.

[0088] In this embodiment, the style embedding vector of the costume includes the following extraction process: Convert the surface texture map of the 3D costume from the RGB color space to a device-independent color space (selected from CIEXYZ or CIE Lab). The conversion matrix is ​​stored on the terminal during the device's factory calibration.

[0089] For each pixel, construct a histogram for the three color channel values. Divide the value range of each channel into a fixed number of sub-intervals. Count the number of pixels falling into each sub-interval and normalize them to obtain three normalized vectors with a length equal to the number of sub-intervals. Then, concatenate them in sequence to form a color distribution vector.

[0090] The Sobel operator is used to calculate the first-order gradient magnitude in the horizontal and vertical directions of the brightness channel of the texture map. The maximum and minimum values ​​of the gradient magnitude of all pixels are counted. The interval from the minimum to the maximum value is divided into three sub-intervals according to the magnitude length, which are denoted as the low magnitude interval, the medium magnitude interval, and the high magnitude interval, respectively. The proportion of the number of pixels in each sub-interval to the total number of pixels is counted to obtain the texture change frequency feature vector of the three components.

[0091] By concatenating the color distribution vector and the texture change frequency feature vector, the style embedding vector of the costume is output.

[0092] Reference Figure 3 As shown, in this embodiment, the tone shifting process includes the following execution steps: The arithmetic mean of the style embedding vectors of all placed costumes in the current scene is calculated to obtain the average style vector. The average style vector is expressed in the form of histogram normalized values ​​for each color channel: the value range of each channel is divided into a predetermined number (e.g., 8 or 16) sub-intervals, and the normalized value of each sub-interval is the ratio of the number of pixels falling within the interval to the total number of pixels.

[0093] For each color channel, determine the sub-interval with the highest normalized value and use it as the starting point. Then, successively expand adjacent sub-intervals to the left and right. When the normalized value of an adjacent sub-interval is greater than or equal to the highest normalized value, the next normalized value is determined. When adjacent sub-intervals are included in the main color continuous interval, after the expansion terminates, the main color continuous interval of the corresponding color channel is obtained; the weighted average of the center values ​​of all sub-intervals in the main color continuous interval is calculated using the normalized value of each sub-interval as the weight, and the main color component of the corresponding color channel is obtained. The main color components of the three channels together constitute the scene main color vector.

[0094] Obtain the surface texture map of the costume, and convert the RGB values ​​of each pixel in the texture map to a specified color space. The specified color space refers to a color space that separates brightness, hue, and saturation, and can be selected from any one of the HSV, HSL, CIE Lab, or LCh color spaces. The conversion matrix or conversion formula is determined during the device's factory color calibration stage and can be directly called.

[0095] In a specified color space, while keeping the brightness and saturation components of each pixel unchanged, the hue components of each pixel are mapped towards the dominant hue range in a contracted manner. The degree of contraction is proportional to the amount of overflow when the cosine distance exceeds a preset distance threshold. The specific contraction mapping process is as follows: extract the hue component values ​​from the scene's dominant hue vector and preset the hue tolerance half-width (e.g., 15°) to define the dominant hue range. Traverse each pixel in the texture map. If the original hue value of the pixel is within the dominant hue range, no adjustment is made; otherwise, calculate the angle difference between the original hue value of each pixel and the nearest boundary of the dominant hue range. Specifically, when the original hue value of the pixel is less than the lower limit of the dominant hue range, the angle difference is obtained by subtracting the original hue value from the lower limit of the dominant hue range. The sign of the angle difference is positive, corresponding to the direction of subsequent hue increase processing. When the original hue value of the pixel is greater than the upper limit of the dominant hue range, the angle difference is obtained by subtracting the upper limit of the dominant hue range from the original hue value. The sign of the angle difference is negative, corresponding to the direction of subsequent hue decrease processing.

[0096] The numerator is the amount of cosine distance exceeding the preset distance threshold, and the denominator is the difference between the maximum possible distance (which is 2) and the preset distance threshold. The result of the ratio calculation is used as the degree of contraction.

[0097] Multiply the degree of contraction by the angle difference to obtain the specific contraction offset.

[0098] The pixel hue component value is added to the shrinkage offset to obtain the processed hue component.

[0099] The processed pixels are converted back from the specified color space to the original color space, and the surface texture of the 3D costume is updated to complete the tone transfer process.

[0100] It should be noted that if there are no costumes placed in the current scene, the average style vector is directly taken from the style embedding vector of the current costume itself, and the tone transfer processing step is skipped.

[0101] The cosine distance between the style embedding vector of the costume and the average style vector of existing objects in the scene is calculated as follows: calculate the dot product of the two vectors, and then calculate the magnitude of each of the two vectors respectively; use the dot product of the two vectors as the numerator and the product of the magnitudes of the two vectors as the denominator to perform a ratio operation, and subtract the quotient from 1 to obtain the cosine distance.

[0102] The process of obtaining the preset distance threshold is as follows: When the number of placed decorations is less than or equal to 2, at most one pair of cosine distances will be generated between any two decorations in the scene, and quartile operation cannot be performed. The preset distance threshold takes the initial default value, which is determined by statistically analyzing the cosine distances of the style embedding vectors of different categories of decoration pairs in the public 3D scene dataset during the offline stage and taking the median value. After rounding to one decimal place, it is written to the terminal flash memory, with a typical value of 0.5. When the number of placed decorations is greater than 2, the cosine distance between the style embedding vectors of every two placed decorations in the current scene is calculated. The obtained cosine distances are arranged in ascending order of value, and the cosine distance value located at the third quartile is taken as the preset distance threshold.

[0103] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A method for rendering and building a virtual 3D scene based on AI and MR, characterized in that, include: By collecting environmental data through MR sensors, identifying and assigning scores to physical areas that can be placed or hung, and combining the attributes, occupancy status, and user change feedback of AI decorations, a three-dimensional semantic map is dynamically constructed and maintained. When the AI ​​converts graffiti into 3D costumes, it parses the object category, determines the semantic role as a hanging dependency, surface support, or surrounding companion, and outputs the corresponding spatial constraint vector. Based on the semantic role and spatial constraint vector of the costume currently dragged and dropped by the user, a semantic matching search is performed in the 3D semantic map to obtain multiple candidate poses; Spatial interleaving detection is performed on each candidate pose to eliminate conflicting poses. The deformation energy between the remaining candidate poses and the user's hand grasping posture is calculated, and the candidate pose with the smallest deformation energy is selected as the recommended adsorption pose. In response to the user's confirmation of the recommended snapping pose, the cosine distance between the style embedding vector of the costume and the average style vector of existing objects in the scene is calculated. If the cosine distance is greater than a preset distance threshold, the costume is rendered in MR space after color transfer processing.

2. The method for rendering and building a virtual 3D scene based on AI and MR according to claim 1, characterized in that, The process of identifying and assigning scores to physical areas that can be placed or hung includes: The environmental data includes depth data and color images of the physical environment; Perform plane detection and point cloud clustering on depth data to obtain several spatially continuous planar regions, and perform semantic segmentation on color images to obtain predefined scene semantic category labels for each pixel; Project the point cloud of the planar region onto the pixel plane of the color image, count the semantic category labels of the pixels that fall within the projection range of each planar region, and determine the semantic category of the planar region as the semantic category of the region with the highest statistical percentage. If the semantic category of a certain planar region belongs to the horizontal bearing category semantic set, then the planar region is determined to be a placeable region, and the initial placeability score is calculated based on the area, flatness parameter and the angle between the normal and the gravity direction of the planar region. If a planar region's semantic category belongs to the vertical protrusion semantic set, and the distribution scale of the region's point cloud in the gravity direction is greater than its distribution scale in the horizontal direction, then the planar region is determined to be a hangable region, and an initial hangability score is calculated based on the region's vertical height relative to the ground, horizontal extension length, and protrusion depth.

3. The method for rendering and building a virtual 3D scene based on AI and MR according to claim 2, characterized in that, The dynamic construction and maintenance of the 3D semantic map includes: Using the identified placeable and hangable areas as map nodes, and using the region semantic category, functional score, bounding box spatial coordinates, and spatial adjacency relationship between nodes as node attributes, an initial three-dimensional semantic map is constructed; the functional score includes the initial placeability score of the placeable area and the initial hangability score of the hangable area. In response to different operations, the following map maintenance measures will be implemented: Placement of new cosmetic items: Obtain the semantic category, bounding box and spatial area occupied by the new cosmetic item, perform spatial overlay analysis on the spatial area and the existing nodes that intersect in the map, determine the occupied nodes and the occupancy ratio, update the functional score of the occupied nodes according to the occupancy ratio, register the new cosmetic item as a new node to the map, and establish a semantic subordinate edge between it and the occupied nodes. Remove placed decorations: Unregister the corresponding node and restore the function rating of the released area; User changes recommended adsorption pose: Record the displacement path between the original recommended adsorption pose and the user's final confirmed pose. Decrease the functional score of the spatial region corresponding to the original recommended adsorption pose according to the length of the displacement path, and increase the functional score of the spatial region corresponding to the final confirmed pose by the same amount. Use the frequency of changes as a confidence reference for score updates.

4. The method for rendering and building a virtual 3D scene based on AI and MR according to claim 1, characterized in that, The determination of semantic roles includes: Using the semantic category tags of the three-dimensional costumes as an index, the corresponding typical spatial existence form records are retrieved from the preset object semantic association knowledge base; the typical spatial existence forms include hanging at a high place, placed on a plane, and surrounding and attached to objects; Among them, "suspended at a high place" refers to the semantic role of the suspended dependency, "placed on a plane" refers to the surface support, "surrounded by the dependent object" refers to the surrounding companion, and the dependent target category of the record association is extracted from the object semantic association knowledge base.

5. A method for rendering and building a virtual 3D scene based on AI and MR according to claim 2, characterized in that, The spatial constraint vector is output according to the following process: The spatial location reference rule for determining the costume based on semantic roles is as follows: Hanging dependency: The lower edge of the hanging area is used as the height reference benchmark. The lower limit of the expected height range is determined by the vertical dimension of the decoration enclosure box. The upper limit of the expected height range is determined by the product of the vertical dimension and the adjustment factor. The adjustment factor is determined by the typical hanging offset ratio of the decoration category recorded in the object semantic association knowledge base. The spacing reference benchmark and expected spacing range fields are left blank. Surface support: Using the surface of the placement area as the height reference, the desired height range is compressed into the height value of the surface, and the spacing reference and desired spacing range fields are left blank; Surrounding objects: The center of the bounding box of the matched object is used as the spacing reference point. The sum of the outer sphere radius of the bounding box of the object and the outer sphere radius of the decoration bounding box is used as the lower limit of the expected spacing. The magnification factor is determined by the surrounding density corresponding to the object category. The product of the lower limit of the expected spacing and the magnification factor is used as the upper limit of the expected spacing. The height reference benchmark and expected height range fields are left blank. The desired height range and desired spacing range are combined with the reference baseline information to form a spatial constraint vector.

6. The method for rendering and building a virtual 3D scene based on AI and MR according to claim 5, characterized in that, The execution of semantic matching search includes: Suspension Dependency: Screen suspendable areas with heights within the desired height range and sort them in descending order of initial suspensionability score. Generate candidate points by shifting along the direction of gravity below each suspendable area in turn. Determine the candidate orientation based on the surface normal direction of the suspendable area to obtain a set of candidate poses. Surface support: Select placeable areas with heights within the desired height range and sort them in descending order of initial placeability score. Generate multiple candidate points by sampling the surface of each placeable area according to the grid. Use the normal direction of the placeable area as the orientation of the candidate pose to obtain a set of candidate poses. Surrounding objects: Find placed decorations in the 3D semantic map that match the semantic category label and are attached to the target category. Using the center of the bounding box as a reference, generate multiple candidate points by angle sampling on a sphere that meets the desired spacing range. Point the orientation of each candidate pose to the center of the placed decoration to obtain a set of candidate poses.

7. The method for rendering and building a virtual 3D scene based on AI and MR according to claim 1, characterized in that, The step of performing spatial interleaving detection on candidate poses to eliminate conflicting poses includes: Mark the bounding box of the 3D costume as the target bounding box; For each candidate pose, the spatial volume intersection operation is performed between the target bounding box and the bounding boxes of all currently placed decorations to obtain the intersection volume; Calculate the arithmetic mean of the target bounding box volume and the bounding box volume of the currently placed decorations in the intersection test, and use the ratio of the minimum to the maximum value of the two bounding box volumes as the tolerance coefficient. The product of the arithmetic mean and the tolerance coefficient is determined as the intersection tolerance threshold. When the intersection volume is greater than the intersection tolerance threshold, the candidate pose is determined to have a spatial conflict and is eliminated.

8. The method for rendering and building a virtual 3D scene based on AI and MR according to claim 1, characterized in that, The calculation of the deformation energy between the remaining candidate poses and the user's hand grasping posture includes: Obtain the position and orientation of the user's hand grasping posture, as well as the position and orientation of each remaining candidate posture; Calculate the translation vector from the position of the grasping pose to the position of the remaining candidate poses, and the minimum rotation angle from the orientation of the grasping pose to the orientation of the candidate poses. Multiply the minimum rotation angle by the radius of the circumscribed sphere of the bounding box of the 3D costume to obtain the displacement arc length of the rotation cost; The translation and rotation weights are determined based on the semantic role of the 3D costume. The deformation energy of each remaining candidate pose is obtained by linearly weighting the translation vector magnitude and the displacement arc length converted from the minimum rotation angle.

9. A method for rendering and building a virtual 3D scene based on AI and MR according to claim 1, characterized in that, The style embedding vector of the costume includes the following extraction process: Perform color space conversion on the surface texture of the 3D costume to obtain device-independent color channel values; Construct a histogram for each color channel value and normalize it. Then, concatenate the normalized histogram vectors of each channel in sequence to obtain the color distribution vector. Gradient calculation is performed on the surface texture to obtain the texture change frequency feature vector; The style embedding vector is obtained by concatenating the color distribution vector with the texture change frequency feature vector.

10. A method for rendering and building a virtual 3D scene based on AI and MR according to claim 1, characterized in that, The tone shifting process includes the following execution steps: The dominant hue component is determined based on the peak range of the average style vector in the color distribution dimension. Convert the texture pixels of the costume from the original color space to the specified color space; In a specified color space, the brightness and saturation components of each pixel are kept unchanged, and the hue components of each pixel are shrunk and mapped to the main hue range. The degree of shrinkage is proportional to the amount of overflow when the cosine distance exceeds the preset distance threshold. The processed pixels are converted back from the specified color space to the original color space, and the surface texture of the 3D costume is updated to complete the tone transfer process.

Images