Space curve drawing and geometric body generation and interaction method based on mixed reality

Abstract

The present invention discloses a spatial curve drawing and geometric body generation and interaction method based on mixed reality, which belongs to the field of immersive human-computer interaction technology, including S1, detecting the user's hand posture through a spatial curve drawing module to complete the drawing of the spatial curve; S2, after completing the drawing of the spatial curve, completing the fitting of the geometric body through a geometric body fitting module, including the position, posture, size and related geometric parameters of the geometric body; S3, after completing the geometric body fitting, realizing the rendering, interaction and parameter adjustment of the geometric body through a geometric body generation and interaction module; with the spatial curve drawing and geometric body generation and interaction method based on mixed reality provided by the present invention, the user can use gesture control to draw and generate geometric bodies in a virtual space, and perform online real-time adjustment on the geometric parameters of the generated geometric bodies, which significantly improves the interactive experience and operation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of immersive human-computer interaction, and in particular to a method for drawing space curves and generating and interacting geometric bodies based on mixed reality. Background Art

[0002] As a fusion of virtual reality and augmented reality, mixed reality technology can seamlessly combine virtual objects with the real environment, allowing users to interact with virtual elements in real time in the real world, thereby achieving a more realistic and interactive experience. As the application of mixed reality in various fields gradually increases, the demand for geometric generation and interaction in immersive virtual space is becoming stronger.

[0003] Geometry is the basic unit that constitutes objects and plays a key role in various application scenarios. In mixed reality space, specific geometric bodies can be used for a variety of purposes, such as marking objects, creating virtual boundaries, realizing space planning and layout design, and prototyping three-dimensional objects. However, traditional geometric body generation methods mainly rely on predefined three-dimensional models or geometric primitives, and often require users to perform tedious operations in the background. Therefore, this method has obvious limitations in dynamic and highly interactive environments. Not only is the operation efficiency low, but it is also difficult to meet the requirements of real-time and flexibility. For example, in a mixed reality environment, users need to generate corresponding geometric bodies based on real-time interactions and actions, and adjust their position, posture and size. This requires the system to not only integrate the interactive interface and respond quickly to user input, but also have the ability to handle various changes in real time. Compared with the traditional predefined model method, the geometric body generation method based on spatial curve drawing can more naturally and intuitively simulate the drawing and construction process in the real world, thereby significantly improving user experience and creation efficiency.

[0004] At present, mixed reality technology is mainly limited to the visualization of virtual objects, and the real-time generation and dynamic adjustment of virtual objects have not yet been realized. The existing technology lacks an effective solution for generating geometric bodies based on real-time user interaction in the mixed reality space, which cannot meet the actual needs of instant creation and adjustment, and greatly limits the flexibility and efficiency of creation. This technical gap restricts the application and development of mixed reality technology in more complex and dynamic scenes. Therefore, it is urgent to develop an innovative technical solution that enables users to achieve real-time drawing, generation and interaction of geometric bodies in a mixed reality environment through natural and intuitive gesture operations, thereby meeting the urgent need for efficient and flexible creation tools. Summary of the invention

[0005] The purpose of the present invention is to provide a spatial curve drawing and geometric body generation and interaction method based on mixed reality to solve the problems existing in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides a method for drawing space curves and generating and interacting geometric bodies based on mixed reality, comprising the following steps:

[0007] S1. Detect the user's hand posture through the space curve drawing module to complete the drawing of the space curve;

[0008] S2. After the drawing of the space curve is completed, the geometric body fitting is completed through the geometric body fitting module, including the position, posture, size and related geometric parameters of the geometric body;

[0009] S3. After completing the geometry fitting, the geometry generation and interaction module is used to realize the rendering, interaction and parameter adjustment of the geometry.

[0010] Preferably, the space curve drawing module in step S1 includes:

[0011] Finger joint recognition: capture the user's hand through mixed reality glasses and identify the position of each joint of the user's hand in real time, including the wrist, palm, fingertips, interphalangeal joints, metacarpophalangeal joints and palmocarpal joints;

[0012] Customized function menu, called out and closed through gesture recognition. When it is recognized that the user's left hand is open with the palm facing the eyes, the function menu is called out; otherwise, the function menu is automatically closed. The function menu includes drawing function, erasing function and locking function, and the function menu is used to assist users to switch freely between functions.

[0013] Preferably, the function menu realizes its various functions through gesture control, specifically:

[0014] A. Drawing function: When the user clicks the drawing function button in the function menu, a virtual green ball appears at the tip of the right index finger, indicating that the drawing state has been entered. The green ball continuously tracks the position of the right index finger, and the system continuously tracks the positions of the right index finger and thumb, and calculates the Euclidean distance between the two. The formula is:

[0015] ;

[0016] in, is the coordinates of the right thumb tip, is the coordinate of the right index fingertip, and is determined by the Euclidean distance If yes, it means starting to draw the curve. The system visualizes the motion trajectory of the right index fingertip as a three-dimensional space curve in real time. Specifically, the current position of the right index fingertip is added as a new point to the LineRenderer component of Unity3D, and the position of the fingertip is continuously tracked. When the distance between the fingertip position and the previous point is greater than the set threshold (5 mm), the system collects the fingertip position again and adds it to the LineRenderer component as a new point. When the two fingers are open, the system ends the drawing of the current curve. The user can draw a new curve by pinching the two fingers together again, or end the drawing state by clicking the drawing function button again.

[0017] B. Erase function: When the user clicks the erase function button, a virtual red ball appears at the tip of the right index finger, indicating that the erase state has been entered. The red ball continues to track the position of the right index finger. The system will continue to track the positions of the right index finger and thumb, and calculate the Euclidean distance between the two. According to the Euclidean distance, when the two fingers are pinched together and the distance to the drawn line curve is less than the set threshold (2 cm), the system will destroy the corresponding LineRenderer component to erase the line. Click the erase function button again to end the erase state. In addition, the erase function is also applicable to the generated geometric bodies, and its operating principle is the same as that of erasing lines.

[0018] C. Lock function: After the user clicks the lock function button, the system enters the lock state. At this time, other function buttons are hidden, and the interactive components of the completed curves and generated geometric bodies are closed. The user cannot select or operate objects through gestures. After clicking the lock function button again, the system ends the lock state and restores the display and interactive functions of all function buttons and gesture interactive components. This function can avoid accidental touches;

[0019] There is a mutually exclusive relationship between the three functions. When the user clicks a function button to activate the corresponding state, the system will automatically end the other function states to ensure that the functional modules do not interfere with each other, improving the smoothness of user operations and the response efficiency of the system. For example, when the user clicks the erase function button in the drawing function state, the system will automatically end the drawing function state and enter the erase function state, and vice versa.

[0020] Preferably, the geometric body fitting module in step S2 supports the drawing, fitting, generation and interaction of 11 geometric bodies, including line segment class, plane class with boundary, body class and parameterized surface class, wherein the line segment class includes straight line segment and curve segment; the plane class with boundary includes circular plane, triangular plane and quadrilateral plane; the body class includes ellipsoid, cuboid and cylinder; the parameterized surface class includes funnel, bowl and ring, specifically including:

[0021] Geometry selection menu: Users can select the geometry they want to generate by clicking a button or dragging a slider, and the name of the selected geometry will be highlighted. This menu is called out and closed based on gesture recognition. When the system recognizes that the user's left hand is clenched into a fist with the palm facing the eyes, the function menu is called out; otherwise, the function menu is automatically closed.

[0022] Fitting of geometric parameters.

[0023] Preferably, the fitting of geometric parameters specifically includes:

[0024] A. Basic feature drawing. When generating a specific geometric body, the user draws the basic features of the geometric body to complete the curve drawing. For example, if the user wants to generate a cuboid, he only needs to draw the four sides of two opposite faces respectively; if he wants to generate a cylinder, he only needs to draw two opposite circles;

[0025] B. Fitting function activation: In the non-drawing state, the user activates the fitting function through gestures. According to the Euclidean distance between the specified joints, when the user's left thumb tip and index finger tip are pinched together, a coroutine is started to monitor the position of the left thumb tip and index finger tip. At this time, when the system detects that the two fingers are opened in a short time (0.2 seconds), the fitting function is activated; otherwise, the coroutine is closed;

[0026] C. Data preparation: After completing the activation gesture of the fitting function, the system reads all the points in the drawn curve and stores them in the Vector of MathNet. <double>Type, after the point set is saved, the system will clear all drawn curves;

[0027] D. Different fitting algorithms are used to process different types of geometric bodies.

[0028] Preferably, different fitting algorithms are used for processing different types of geometric bodies, including the following types:

[0029] a. Straight line segment: Determine the main direction of the straight line by constructing the covariance matrix and performing eigenvalue decomposition. Then, calculate the projection of all points in this direction, and determine the starting point, end point and direction of the straight line segment by identifying the two farthest projection points;

[0030] b. Curve segment: First, generate uniformly distributed B-spline node vectors, and then use the De Boor algorithm to perform uniform sampling on the B-spline curve to generate a dense curve point set; by calculating the cumulative length between the sampling points and performing linear interpolation according to the preset interval distance, finally obtain a smooth and equally spaced fitting point set;

[0031] c. Plane with boundary: The principal component analysis (PCA) method is used to calculate the normal vector and centroid of the plane, and the three-dimensional point set is projected onto the fitting plane to obtain a two-dimensional point set. The QuickHull algorithm is used to calculate the convex hull of the projected point set to determine the outer boundary, and the polygon with the largest area (a dodecagon is used to approximate a circle) is identified in the convex hull point set as the final fitting result. The algorithm output includes the coordinates of the polygon vertices and their rotation information;

[0032] d. Ellipsoid: Construct a system of linear equations and solve the coordinates of the center of the sphere and the radius parameters by the least squares method;

[0033] e. Cuboid: The input point set is divided into two groups for clustering based on the connection distance threshold (4 cm) between adjacent points. The depth direction is determined by calculating the centroid of the two groups of point sets. The large point set is fitted with a rectangle to obtain four vertices, and the complete eight vertices are generated in combination with the depth information.

[0034] f. Cylinder: Clustering method is used to group points. The center position, axis and height are determined by calculating the centroid of two groups of point sets. The larger point set is fitted with a circular surface based on the least squares method to obtain the radius of the cylinder.

[0035] g. Funnel and bowl shapes: First, identify the lowest point in the point set as the bottom vertex, select the point farthest from the bottom as the top vertex, and substitute the vertex into the two-dimensional curve equation. For the funnel shape: Bowl shape: , and get the corresponding surface parameters; rotate the two-dimensional curve to get a three-dimensional surface, where the funnel shape: Bowl shape: ;in, , , They represent the coordinates of points on the curve or surface in three-dimensional space, express The offset ratio is used to control the scaling of the surface. Indicates the offset of the curve, which is used to control the starting point of the curve and the position of the surface. Represents the shape parameter of the curve, which is used to control the curvature of the curve. Represents the scaling parameter of the curve, which is used to control the scaling of the surface;

[0036] h. Ring: Perform least squares fitting on the point sets of the large circle and the small circle respectively, obtain the center, radius and normal vector of the two circles, and obtain the three-dimensional surface equation of the ring: ,in, is the great circle radius, is the radius of the small circle.

[0037] Preferably, the geometry generation and interaction module in step S3 includes the following functions:

[0038] Geometry rendering: read the fitting point set and use a custom method to generate the mesh of the geometry;

[0039] Geometric interaction: Based on the generated mesh, a collider is generated for the geometric body, and the ObjectManipulator component in MRTK3 is added to realize the natural interaction between the user and the geometric body. The user can directly grab, move, rotate and scale the geometric body through gestures. At the same time, the ConstraintManager component in MRTK3 is used to perform customized restriction management on the interactive behavior of the geometric body, for example, the geometric body is only allowed to move and rotate in the specified direction. When the user's hand touches the geometric body, the boundary of the geometric body will light up, showing the boundary of the current geometric body; when the hand leaves, the border will gradually disappear;

[0040] Posture indication: Each geometric object is equipped with an arrow or coordinate system to describe its posture. When the user's hand touches the geometric object, the arrow or coordinate system appears, and when the hand is no longer touching, the arrow or coordinate system disappears;

[0041] After each movement or rotation of a geometry, the system starts a coroutine to automatically determine and adjust the alignment of the geometry's y-axis with the world coordinate system;

[0042] Make custom adjustments to geometry parameters.

[0043] Preferably, the specific steps of geometric body rendering are:

[0044] Vertex generation: By calculating the x and z coordinates of the geometric body at different height levels (y values) of the local coordinate system, multiple sets of vertices at different heights are obtained;

[0045] Face construction: Triangular faces are formed between vertices at adjacent heights to describe the Mesh, and MeshFilter and MeshRenderer components are added to achieve complete rendering and visualization of the geometry.

[0046] Preferably, after each movement or rotation of the geometric body, the system will start a coroutine to automatically determine and adjust the alignment of the y-axis of the geometric body with the world coordinate system. The specific steps are:

[0047] Angle calculation: calculate the angle between the y-axis of the geometric body and the three axes of the world coordinate system;

[0048] Alignment judgment: When the angle is less than the preset threshold (5°), the system aligns the y-axis of the geometry to the corresponding axis of the world coordinate system through smooth rotation.

[0049] Preferably, the customized adjustment of geometric parameters specifically includes:

[0050] A. Triangular plane and quadrilateral plane

[0051] Interactive node generation: Generate interactive balls at the vertex positions of geometric bodies;

[0052] Gesture interaction: When the user's hand touches the geometric body, the vertex ball is visualized and the interactive component is activated. The user can dynamically adjust the vertex position by grabbing the ball and moving it on the triangular or quadrilateral plane;

[0053] Real-time fitting: When you move the vertex ball, the geometry generation and interaction module will activate the corresponding geometry fitting algorithm to calculate, update and render the mesh of the geometry in real time;

[0054] B. Ellipsoid, cuboid and cylinder

[0055] Posture coordinate system: For volume-type geometries, the geometry generation and interaction module configures the coordinate axes used to describe its local coordinate system;

[0056] Independent axis scaling: Users can independently scale the geometry on the xyz axes in the local coordinate system (i.e., posture) by grabbing and moving the coordinate axes of the local coordinate system (i.e., posture); this independent axis scaling method allows users to accurately adjust the size of the geometry in all directions.

[0057] C. Parametric Surface

[0058] For different types of parametric surfaces, the geometry generation and interaction module will generate corresponding interactive elements;

[0059] Dynamic parameter adjustment: Users grab and move interactive elements through gestures, and the geometry generation and interaction module will automatically update the corresponding surface parameters to achieve dynamic adjustment of parametric surfaces. At the same time, during the parameter adjustment process, the system will recalculate the mesh data of the geometry in real time and render the updated surface instantly.

[0060] Therefore, the present invention adopts the above-mentioned spatial curve drawing and geometric body generation and interaction method based on mixed reality, which has the following beneficial effects:

[0061] (1) Through the set spatial curve drawing module, based on hand tracking and gesture recognition, users can freely use their fingers to draw and erase curves in a three-dimensional virtual space;

[0062] (2) By calculating the distance between the current finger joint position and the last stored point, the algorithm determines whether to store a new point. This not only significantly improves the efficiency of the algorithm and reduces the computational burden, but also provides better input data for the processing of subsequent modules.

[0063] (3) In the geometry generation and interaction module, the geometric body fitting requirements in specific scenarios are achieved through the combined application of different mathematical methods;

[0064] (4) In the geometry generation and interaction module, by using the basic components of the MRTK3 open source framework (including ObjectManipulator, ConstraintManager, and BoundsControl), and expanding and optimizing functions, a richer interactive experience can be achieved.

[0065] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of the spatial curve drawing and geometric body generation and interaction method based on mixed reality of the present invention;

[0067] Figure 2 A schematic diagram of a customized function menu for an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of the principle of drawing functions for an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of a geometric body menu selection according to an embodiment of the present invention;

[0070] Figure 5 Draw a schematic diagram for the basic features of the embodiment of the present invention;

[0071] Figure 6 Schematic diagram of activation of a fitting function according to an embodiment of the present invention, wherein (a) is before fitting, and (b) is activation of the fitting function and generation of a geometric body;

[0072] Figure 7 A schematic diagram of generating corresponding interactive elements for the parameterized surface according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] See also Figure 1 , a spatial curve drawing and geometric body generation and interaction method based on mixed reality, comprising the following steps:

[0075] S1. Detect the user's hand posture through the space curve drawing module to complete the drawing of the space curve; the space curve drawing module includes:

[0076] Finger joint recognition: capture the user's hand through mixed reality glasses and identify the position of each joint of the user's hand in real time, including the wrist, palm, fingertips, interphalangeal joints, metacarpophalangeal joints and palmocarpal joints;

[0077] Customize function menus, such as Figure 2 As shown, calling out and closing are performed through gesture recognition. When it is recognized that the five fingers of the user's left hand are spread out and the palm is facing the eyes, the function menu is called out; otherwise, the function menu is automatically closed. The function menu includes a drawing function, an erase function and a lock function, and the function menu is used to assist the user to switch freely between functions.

[0078] The function menu provides a variety of function modules designed to achieve geometric drawing, erasing and locking operations in a mixed reality environment through gesture control, including:

[0079] A. Drawing function: When the user clicks the drawing function button in the function menu, a virtual green ball appears at the tip of the right index finger, indicating that the drawing state has been entered. The green ball continuously tracks the position of the right index finger, and the system continuously tracks the positions of the right index finger and thumb, and calculates the Euclidean distance between the two. The formula is:

[0080] ;

[0081] in, is the coordinates of the right thumb tip, is the coordinate of the right index fingertip, and is determined by the Euclidean distance If yes, it means starting to draw the curve. The system visualizes the motion trajectory of the right index fingertip as a three-dimensional space curve in real time, such as Figure 3 As shown, specifically: the current position of the right index fingertip is added as a new point to the LineRenderer component of Unity3D, and the position of the fingertip is continuously tracked. When the distance between the fingertip position and the previous point is greater than the set threshold (5 mm), the system collects the fingertip position again and adds it to the LineRenderer component as a new point; when the two fingers are spread apart, the system ends the drawing of the current curve, and the user can draw a new curve by pinching the two fingers together again, or end the drawing state by clicking the drawing function button again;

[0082] B. Erase function: When the user clicks the erase function button, a virtual red ball appears at the tip of the right index finger, indicating that the erase state has been entered. The red ball continues to track the position of the right index finger. The system will continue to track the positions of the right index finger and thumb, and calculate the Euclidean distance between the two. According to the Euclidean distance, when the two fingers are pinched together and the distance to the drawn line curve is less than the set threshold (2 cm), the system will destroy the corresponding LineRenderer component to erase the line. Click the erase function button again to end the erase state. In addition, the erase function is also applicable to the generated geometric bodies, and its operating principle is the same as that of erasing lines.

[0083] C. Lock function: After the user clicks the lock function button, the system enters the lock state. At this time, other function buttons are hidden, and the interactive components of the completed curves and generated geometric bodies are closed. The user cannot select or operate objects through gestures. After clicking the lock function button again, the system ends the lock state and restores the display and interactive functions of all function buttons and gesture interactive components. This function can avoid accidental touches;

[0084] There is a mutually exclusive relationship between the three functions. When the user clicks a function button to activate the corresponding state, the system will automatically end the other function states to ensure that the functional modules do not interfere with each other, improving the smoothness of user operations and the response efficiency of the system. For example, when the user clicks the erase function button in the drawing function state, the system will automatically end the drawing function state and enter the erase function state, and vice versa.

[0085] S2. After the drawing of the space curve is completed, the geometric body fitting is completed through the geometric body fitting module, including the position, posture, size and related geometric parameters of the geometric body;

[0086] The geometry fitting module supports the drawing, fitting, generation and interaction of 11 types of geometric bodies, including line segment class, plane class with boundary, body class and parametric surface class. Among them, the line segment class includes straight line segment and curve segment; the plane class with boundary includes circular plane, triangular plane and quadrilateral plane; the body class includes ellipsoid, cuboid and cylinder; the parametric surface class includes funnel, bowl and ring, including:

[0087] Geometry selection menu: Figure 4 As shown, the user selects the geometry to be generated by clicking a button or dragging a slider, and the name of the selected geometry will be highlighted. The menu is called out and closed based on gesture recognition. When the system recognizes that the user's left hand is clenched into a fist with the palm facing the eyes, the function menu is called out; otherwise, the function menu is automatically closed.

[0088] Parameters of the fitted geometry, including:

[0089] (1) Basic feature drawing

[0090] When generating a specific geometric body, the user draws the basic features of the geometric body to complete the curve drawing. For example, if the user wants to generate a rectangular parallelepiped, he only needs to draw the four sides of two opposite faces respectively. Figure 5 As shown; if you want to generate a cylinder, just draw two opposite circles;

[0091] (2) Fitting function activation

[0092] In the non-drawing state, the user activates the fitting function through gestures. According to the Euclidean distance between the specified joints, when the user's left thumb tip and index finger tip are pinched together, a coroutine is started to monitor the position of the left thumb tip and index finger tip. At this time, when the system detects that the two fingers are opened in a short time (0.2 seconds), the fitting function is activated; otherwise, the coroutine is closed; Figure 6 shown.

[0093] (3) Data preparation

[0094] After completing the activation gesture of the fitting function, the system reads all the points in the drawn curve and stores them in MathNet's Vector <double>Type, after the point set is saved, the system will clear all drawn curves;

[0095] (4) Geometry fitting algorithm

[0096] Different fitting algorithms are used to process different types of geometric bodies, including the following types:

[0097] a. Straight line segment: Determine the main direction of the straight line by constructing the covariance matrix and performing eigenvalue decomposition. Then, calculate the projection of all points in this direction, and determine the starting point, end point and direction of the straight line segment by identifying the two farthest projection points;

[0098] b. Curve segment: First, generate uniformly distributed B-spline node vectors, and then use the De Boor algorithm to perform uniform sampling on the B-spline curve to generate a dense curve point set; by calculating the cumulative length between the sampling points and performing linear interpolation according to the preset interval distance, finally obtain a smooth and equally spaced fitting point set;

[0099] c. Plane with boundary: The principal component analysis (PCA) method is used to calculate the normal vector and centroid of the plane, and the three-dimensional point set is projected onto the fitting plane to obtain a two-dimensional point set. The QuickHull algorithm is used to calculate the convex hull of the projected point set to determine the outer boundary, and the polygon with the largest area (a dodecagon is used to approximate a circle) is identified in the convex hull point set as the final fitting result. The algorithm output includes the coordinates of the polygon vertices and their rotation information;

[0100] d. Ellipsoid: Construct a system of linear equations and solve the coordinates of the center of the sphere and the radius parameters by the least squares method;

[0101] e. Cuboid: The input point set is divided into two groups for clustering based on the connection distance threshold (4 cm) between adjacent points. The depth direction is determined by calculating the centroid of the two groups of point sets. The large point set is fitted with a rectangle to obtain four vertices, and the complete eight vertices are generated in combination with the depth information.

[0102] f. Cylinder: Clustering method is used to group points. The center position, axis and height are determined by calculating the centroid of two groups of point sets. The larger point set is fitted with a circular surface based on the least squares method to obtain the radius of the cylinder.

[0103] g. Funnel and bowl shapes: First, identify the lowest point in the point set as the bottom vertex, select the point farthest from the bottom as the top vertex, and substitute the vertex into the two-dimensional curve equation. For the funnel shape: Bowl shape: , and get the corresponding surface parameters; rotate the two-dimensional curve to get a three-dimensional surface, where the funnel shape: Bowl shape: ;in, , , They represent the coordinates of points on the curve or surface in three-dimensional space, express The offset ratio is used to control the scaling of the surface. Indicates the offset of the curve, which is used to control the starting point of the curve and the position of the surface. Represents the shape parameter of the curve, which is used to control the curvature of the curve. Represents the scaling parameter of the curve, which is used to control the scaling of the surface;

[0104] h. Ring: Perform least squares fitting on the point sets of the large circle and the small circle respectively, obtain the center, radius and normal vector of the two circles, and obtain the three-dimensional surface equation of the ring: ,in, is the great circle radius, is the radius of the small circle.

[0105] S3. After completing the geometry fitting, the geometry generation and interaction module is used to realize the rendering, interaction and parameter adjustment of the geometry, including the following functions:

[0106] 1. Geometry rendering: Read the fitting point set and use a custom method to generate the mesh of the geometry. The specific steps include:

[0107] Vertex generation: By calculating the x and z coordinates of the geometric body at different height levels (y values) of the local coordinate system, multiple sets of vertices at different heights are obtained;

[0108] Face construction: Triangular faces are formed between vertices at adjacent heights to describe the Mesh, and MeshFilter and MeshRenderer components are added to achieve complete rendering and visualization of the geometry.

[0109] 2. Geometric interaction: Based on the generated mesh, a collider is generated for the geometric body, and the ObjectManipulator component in MRTK3 is added to achieve natural interaction between the user and the geometric body. The user can directly grab, move, rotate and scale the geometric body through gestures. At the same time, the ConstraintManager component in MRTK3 is used to perform customized restriction management on the interactive behavior of the geometric body, for example, the geometric body is only allowed to move and rotate in the specified direction. Using the BoundsControl component in MRTK3, when the user's hand touches the geometric body, the boundary of the geometric body will light up to show the boundary of the current geometric body; when the hand leaves, the border will gradually disappear;

[0110] 3. Posture indication: Each geometric body is equipped with an arrow or coordinate system to describe its posture. When the user's hand touches the geometric body, the arrow or coordinate system appears, and when the hand is no longer touching, the arrow or coordinate system disappears;

[0111] 4. After each movement or rotation of the geometry, the system will start a coroutine to automatically determine and adjust the alignment of the geometry's y-axis with the world coordinate system; the specific steps are:

[0112] Angle calculation: calculate the angle between the y-axis of the geometric body and the three axes of the world coordinate system;

[0113] Alignment judgment: When the angle is less than the preset threshold (5°), the system aligns the y-axis of the geometry to the corresponding axis of the world coordinate system through smooth rotation.

[0114] 5. Customize and adjust geometry parameters: Although the ObjectManipulator component provides basic geometry scaling functions, there are limitations in some cases. For example, when scaling a triangular plane, the ObjectManipulator component can only achieve proportional scaling, that is, changing the length of the three sides, but not the ratio between the three sides. The geometry generation and interaction module has been optimized for this problem, including:

[0115] A. Triangular plane and quadrilateral plane

[0116] Interactive node generation: Generate interactive balls at the vertex positions of geometric bodies;

[0117] Gesture interaction: When the user's hand touches the geometric body, the vertex ball is visualized and the interactive component is activated. The user can dynamically adjust the vertex position by grabbing the ball and moving it on the triangular or quadrilateral plane;

[0118] Real-time fitting: When you move the vertex ball, the geometry generation and interaction module will activate the corresponding geometry fitting algorithm to calculate, update and render the mesh of the geometry in real time;

[0119] B. Ellipsoid, cuboid and cylinder

[0120] Posture coordinate system: For volume-type geometries, the geometry generation and interaction module configures the coordinate axes used to describe its local coordinate system;

[0121] Independent axis scaling: Users can independently scale the geometry on the xyz axes in the local coordinate system (i.e., posture) by grabbing and moving the coordinate axes of the local coordinate system (i.e., posture); this independent axis scaling method allows users to accurately adjust the size of the geometry in all directions.

[0122] C. Parametric Surface

[0123] For different types of parametric surfaces, the geometry generation and interaction module will generate corresponding interaction elements; for example, for a funnel-shaped surface, three spheres and an asymptote will be generated, such as Figure 7 shown.

[0124] Dynamic parameter adjustment: Users grab and move interactive elements through gestures, and the geometry generation and interaction module will automatically update the corresponding surface parameters to achieve dynamic adjustment of parametric surfaces. At the same time, during the parameter adjustment process, the system will recalculate the mesh data of the geometry in real time and render the updated surface instantly.

[0125] Therefore, the present invention adopts the above-mentioned spatial curve drawing and geometric body generation and interaction method based on mixed reality. Users can use gesture control to draw and generate geometric bodies in virtual space, and make online real-time adjustments to the geometric parameters of the generated geometric bodies, which significantly improves the interactive experience and operation accuracy.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.< / double> < / double>

Claims

1. A spatial curve drawing and geometric body generation and interaction method based on mixed reality, characterized in that: The following steps are involved: S1. Detect the user's hand posture through the space curve drawing module to complete the drawing of the space curve; S2. After the drawing of the space curve is completed, the geometric body fitting is completed through the geometric body fitting module, including the position, posture, size and related geometric parameters of the geometric body; S3. After completing the geometric body fitting, the geometric body rendering, interaction and parameter adjustment are realized through the geometric body generation and interaction module; The geometry generation and interaction module includes the following features: Geometry rendering: read the fitting point set and use a custom method to generate the mesh of the geometry; Geometric interaction: Based on the generated mesh, a collider is generated for the geometric body, and the ObjectManipulator component in MRTK3 is added to realize the natural interaction between the user and the geometric body. The user can directly grab, move, rotate and scale the geometric body through gestures. At the same time, the ConstraintManager component in MRTK3 is used to perform customized restriction management on the interactive behavior of the geometric body. When the user's hand touches the geometric body, the boundary of the geometric body will light up to show the boundary of the current geometric body; when the hand leaves, the border will gradually disappear; Posture indication: Each geometric object is equipped with an arrow or coordinate system to describe its posture. When the user's hand touches the geometric object, the arrow or coordinate system appears, and when the hand no longer touches it, the arrow or coordinate system disappears; After each movement or rotation of a geometry, the system starts a coroutine to automatically determine and adjust the alignment of the geometry's y-axis with the world coordinate system; Customize and adjust geometry parameters; Custom adjustments to geometry parameters, including: A. Triangular plane and quadrilateral plane Interactive node generation: Generate interactive balls at the vertex positions of geometric bodies; Gesture interaction: When the user's hand touches the geometric body, the vertex ball is visualized and the interactive component is activated. The user can dynamically adjust the vertex position by grabbing the ball and moving it on the triangular or quadrilateral plane; Real-time fitting: When you move the vertex ball, the geometry generation and interaction module will activate the corresponding geometry fitting algorithm to calculate, update and render the mesh of the geometry in real time; B. Ellipsoid, cuboid and cylinder Posture coordinate system: For volume-type geometries, the geometry generation and interaction module configures the coordinate axes used to describe its local coordinate system; Independent axial scaling: Users can independently scale the geometry on the xyz axes in the local coordinate system by grabbing and moving the coordinate axes of the local coordinate system. C. Parametric Surface For different types of parametric surfaces, the geometry generation and interaction module will generate corresponding interactive elements; Dynamic parameter adjustment: Users grab and move interactive elements through gestures, and the geometry generation and interaction module will automatically update the corresponding surface parameters to achieve dynamic adjustment of parametric surfaces. At the same time, during the parameter adjustment process, the system will recalculate the mesh data of the geometry in real time and render the updated surface instantly.

2. The method for drawing spatial curves and generating and interacting geometric bodies based on mixed reality according to claim 1, characterized in that: The space curve drawing module in step S1 includes: Finger joint recognition: capture the user's hand through mixed reality glasses and identify the position of each joint of the user's hand in real time; Customize the function menu, call out and close it through gesture recognition. When it is recognized that the user's left hand has five fingers spread out and the palm is facing the eyes, the function menu is called out; otherwise, the function menu is automatically closed. The function menu includes drawing function, erasing function and locking function.

3. The method for drawing spatial curves and generating and interacting geometric bodies based on mixed reality according to claim 2, characterized in that: The function menu realizes its various functions through gesture control, specifically: A. Drawing function: When the user clicks the drawing function button in the function menu, a virtual green ball appears at the tip of the right index finger, indicating that the drawing state has been entered. The green ball continuously tracks the position of the right index finger, and the system continuously tracks the positions of the right index finger and thumb, and calculates the Euclidean distance between the two. The formula is: ； in, is the coordinates of the right thumb tip, is the coordinate of the right index fingertip, and is determined by the Euclidean distance If yes, it means that the curve drawing starts. The system visualizes the motion trajectory of the right index fingertip as a three-dimensional space curve in real time. Specifically, the current position of the right index fingertip is added as a new point to the LineRenderer component of Unity3D, and the position of the fingertip is continuously tracked. When the distance between the fingertip position and the previous point is greater than the set threshold, the system collects the fingertip position again and adds it to the LineRenderer component as a new point. When the two fingers are opened, the system ends the drawing of the current curve. The user draws a new curve by pinching the two fingers together again, and clicks the drawing function button again to end the drawing state. B. Erase function: When the user clicks the erase function button, a virtual red ball appears at the tip of the right index finger, indicating that the erase state has been entered. The red ball continues to track the position of the right index finger. The system will continue to track the positions of the right index finger and thumb, and calculate the Euclidean distance between the two. According to the Euclidean distance, when the two fingers are pinched together and the line curve drawn by the distance is less than the set threshold, the system will destroy the corresponding LineRenderer component to erase the line. Click the erase function button again to end the erase state. C. Lock function: After the user clicks the lock function button, the system enters the lock state. At this time, other function buttons are hidden, and the interactive components of the completed curves and generated geometric bodies are closed. The user cannot select or operate objects through gestures. After clicking the lock function button again, the system ends the lock state and restores the display and interactive functions of all function buttons and gesture interactive components; There is a mutually exclusive relationship between the three functions. When the user clicks a function button to activate the corresponding state, the system will automatically end the other function states.

4. The method for drawing spatial curves and generating and interacting geometric bodies based on mixed reality according to claim 1, characterized in that: The geometric body fitting module in step S2 supports the drawing, fitting, generation and interaction of 11 geometric bodies, including line segment class, plane class with boundary, body class and parameterized surface class, among which the line segment class includes straight line segment and curve segment; the plane class with boundary includes circular plane, triangular plane and quadrilateral plane; the body class includes ellipsoid, cuboid and cylinder; the parameterized surface class includes funnel, bowl and ring, specifically including: Geometry selection menu: Users click a button or drag a slider to select the geometry they want to generate, and the name of the selected geometry will be highlighted. This menu is called out and closed based on gesture recognition. When the system recognizes that the user's left hand is clenched into a fist with the palm facing the eyes, the menu is called out; otherwise, the menu is automatically closed. Fitting of geometric parameters.

5. The method for drawing spatial curves and generating and interacting geometric bodies based on mixed reality according to claim 4, characterized in that: The fitting of geometric parameters specifically includes: A. Basic feature drawing: when generating a geometric body, the user draws the basic features of the geometric body to complete the curve drawing; B. Fitting function activation: In the non-drawing state, the user activates the fitting function through gestures. According to the Euclidean distance between the specified joints, when the user's left thumb tip and index finger tip are pinched together, a coroutine is started to monitor the position of the left thumb tip and index finger tip. At this time, when the system detects that the two fingers are opened in a short time, the fitting function is activated; otherwise, the coroutine is closed; C. Data preparation: After completing the activation gesture of the fitting function, the system reads all the points in the drawn curve and stores them in the Vector of MathNet. <double>Type, after the point set is saved, the system will clear all drawn curves;< / double> D. Different fitting algorithms are used to process different types of geometric bodies.

6. The method for drawing spatial curves and generating and interacting geometric bodies based on mixed reality according to claim 5, characterized in that: Different fitting algorithms are used to process different types of geometric bodies, including the following types: a. Straight line segment: The main direction of the straight line is determined by constructing the covariance matrix and performing eigenvalue decomposition. Then, the projection of all points in this direction is calculated, and the starting point, end point and direction of the straight line segment are determined by identifying the two projection points at the far end; b. Curve segment: First, generate uniformly distributed B-spline node vectors, and then use the De Boor algorithm to perform uniform sampling on the B-spline curve to generate a dense curve point set; by calculating the cumulative length between the sampling points and performing linear interpolation according to the preset interval distance, finally obtain a smooth and equally spaced fitting point set; c. Plane with boundary: The principal component analysis method is used to calculate the normal vector and centroid of the plane, and the three-dimensional point set is projected onto the fitting plane to obtain a two-dimensional point set. The QuickHull algorithm is used to calculate the convex hull of the projected point set to determine the outer boundary, and a polygon with a large area is identified in the convex hull point set as the final fitting result. The algorithm output includes the coordinates of the polygon vertices and their rotation information; d. Ellipsoid: Construct a system of linear equations and solve the coordinates of the center of the sphere and the radius parameters by the least squares method; e. Cuboid: The input point set is divided into two groups for clustering based on the connection distance threshold between adjacent points. The depth direction is determined by calculating the centroid of the two groups of point sets. The large point set is fitted with a rectangle to obtain four vertices, and the complete eight vertices are generated in combination with the depth information. f. Cylinder: Clustering method is used to group points. The center position, axis and height are determined by calculating the centroid of two groups of point sets. The circular surface fitting based on the least squares method is performed on the large point set to obtain the radius of the cylinder. g. Funnel and bowl shapes: First, identify the lowest point in the point set as the bottom vertex, select the point farthest from the bottom as the top vertex, and substitute the vertex into the two-dimensional curve equation. For the funnel shape: Bowl shape: , and get the corresponding surface parameters; rotate the two-dimensional curve to get a three-dimensional surface, where the funnel shape: Bowl shape: ;in, , , They represent the coordinates of points on the curve or surface in three-dimensional space, express The offset ratio is used to control the scaling of the surface. Indicates the offset of the curve, which is used to control the starting point of the curve and the position of the surface. Represents the shape parameter of the curve, which is used to control the curvature of the curve. Represents the scaling parameter of the curve, which is used to control the scaling of the surface; h. Ring: Perform least squares fitting on the point sets of the large circle and the small circle respectively, obtain the center, radius and normal vector of the two circles, and obtain the three-dimensional surface equation of the ring: ,in, is the great circle radius, is the radius of the small circle.

7. The method for drawing spatial curves and generating and interacting geometric bodies based on mixed reality according to claim 1, characterized in that: The specific steps of geometry rendering are: Vertex generation: By calculating the x and z coordinates of the geometric body at different height levels of the local coordinate system, multiple sets of vertices at different heights are obtained; Face construction: Triangle faces are formed between vertices at adjacent heights, and MeshFilter and MeshRenderer components are added to achieve complete rendering and visualization of the geometry.

8. The method for drawing space curves and generating and interacting geometric bodies based on mixed reality according to claim 1, characterized in that: After each movement or rotation of a geometry, the system starts a coroutine to automatically determine and adjust the alignment of the geometry's y-axis with the world coordinate system. The specific steps are as follows: Angle calculation: calculate the angle between the y-axis of the geometric body and the three axes of the world coordinate system; Alignment judgment: When the angle is less than the preset threshold, the system aligns the y-axis of the geometry to the corresponding axis of the world coordinate system through smooth rotation.

Images