Free body modeling method, system and equipment based on gesture and eye movement signal, medium and product

Abstract

The invention discloses a free body modeling method, system and device based on gesture and eye movement signals, a medium and a product, and relates to the field of intelligent interactive modeling. The method comprises the following steps: constructing a three-dimensional model; sampling points on the three-dimensional model are obtained, and control points of the three-dimensional model are automatically arranged based on a least square method; establishing a function mapping relation between the control points and the sampling points; acquiring a user gesture and a user eye movement signal; zooming and rotation of the three-dimensional model are completed based on the visual angle zooming gesture and the visual angle rotation gesture; after triggering eye movement capture by adopting the triggering gesture, determining a to-be-controlled point based on a user eye movement signal; and after the to-be-controlled point is controlled based on the deformation control gesture, deformation of the three-dimensional model is completed according to the function mapping relation so as to realize free body modeling. According to the invention, the interaction mode of the user in the modeling process is closer to the mode of'eye player and hand control 'in daily life.

Description

Technical Field

[0001] The present application relates to the field of intelligent interactive modeling, and in particular to a free-form modeling method, system, device, medium and product based on gesture and eye movement signals. Background Art

[0002] In recent years, with the introduction of the concept of intelligent design, new technical requirements and challenges have been put forward for human-computer interaction technology in the product design process. Traditional interactive tools such as mouse and keyboard have great limitations in terms of naturalness and friendliness of use, and it is difficult to meet the development needs of human-computer interaction technology in intelligent design, making it difficult for users to express their innovative thinking freely and naturally.

[0003] Gestures are an important carrier for humans to express non-verbal intention perception. With the continuous development of science and technology in recent years, research on gesture recognition technology has received widespread attention. In addition, eye activity is an important way for humans to search and communicate. At present, research on gesture recognition mainly focuses on improving gesture recognition accuracy and processing efficiency, but there are few human-computer interaction modeling methods for user hand-eye coordination, which cannot make the user's interaction method in the modeling process closer to the "eye-selection and hand-control" mode in daily life, where "eye selection" refers to selection through the eyes and "hand control" refers to control through hand movements. Summary of the invention

[0004] The purpose of this application is to provide a free-form modeling method, system, device, medium and product based on gestures and eye movement signals, which can make the user's interaction method in the modeling process closer to the "eye-control" mode in daily life.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a free-form modeling method based on gesture and eye movement signals, comprising:

[0007] Build 3D models;

[0008] Acquire sampling points on the three-dimensional model, and automatically arrange control points of the three-dimensional model based on the least square method; the distribution of the arranged control points is adaptively matched to the shape of the three-dimensional model;

[0009] Establishing a function mapping relationship between the control point and the sampling point;

[0010] Acquire user gestures and user eye movement signals; the user gestures at least include perspective zoom gestures, perspective rotation gestures, trigger gestures and deformation control gestures;

[0011] Complete scaling and rotation of the three-dimensional model based on the perspective scaling gesture and the perspective rotation gesture;

[0012] After the trigger gesture is used to trigger eye movement capture, a point to be controlled is determined based on the user's eye movement signal;

[0013] After the points to be controlled are controlled based on the deformation control gesture, the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling; the points to be controlled are one or more control points of the three-dimensional model.

[0014] Optionally, construct a three-dimensional model, including:

[0015] Get the initial model;

[0016] Discretizing the initial model;

[0017] The discretized initial model is imported into the VTK software system to obtain the three-dimensional model.

[0018] Optionally, in the VTK software system, obtaining sampling points on the three-dimensional model, and automatically arranging control points of the three-dimensional model based on a least squares method, includes:

[0019] Acquire dimension data of the three-dimensional model; the dimension data includes: length data, width data and height data;

[0020] Generate a minimum bounding box of the three-dimensional model based on the size data of the three-dimensional model;

[0021] Performing mesh division in three dimensions of the three-dimensional model respectively, dividing the minimum bounding box into a plurality of sub-hexahedrons;

[0022] Screening out a sub-hexahedron containing a three-dimensional model entity, and using the center of gravity of the contained three-dimensional model entity as the sampling point;

[0023] Fitting the tangent plane of each sampling point using the least square method to obtain an initial vector;

[0024] Normalizing the initial vector to obtain a normal vector corresponding to each sampling point;

[0025] The normal vector is redirected, and the sampling point is translated along the direction of the redirected normal vector to obtain the control point.

[0026] Optionally, redirecting the normal vector comprises:

[0027] Determine the product value of the vector from the center of gravity of the three-dimensional model entity to the meshing point on the surface of the sub-hexahedron and the normal vector;

[0028] The direction of the normal vector is determined based on the positive or negative value of the product value, and the redirected normal vector is obtained.

[0029] Optionally, after the trigger gesture is used to trigger eye movement capture, determining a point to be controlled based on the user eye movement signal includes:

[0030] Performing Kalman filtering on the eye movement signal to obtain the local coordinates of the gaze point;

[0031] When the change speed of the gaze point is less than the speed threshold, the coordinate data of the gaze point is collected according to the number of frames;

[0032] When the number of acquired frames exceeds a frame number threshold, an average value of the coordinate data of the gaze point is determined to obtain a local coordinate of the gaze point relative to a screen; the screen is a medium for displaying a three-dimensional model;

[0033] Determine a neighborhood of the control point based on the center position and radius of the control point, and determine whether the local coordinates of the gaze point relative to the screen are located within one of the neighborhoods;

[0034] When the local coordinates of the gaze point relative to the screen are located within one of the neighborhoods, a control point in the three-dimensional model corresponding to the neighborhood is used as a central control point;

[0035] When the local coordinates of the gaze point relative to the screen are within the plurality of neighborhoods, a control point corresponding to the neighborhood closest to the local coordinates of the gaze point is used as a central control point;

[0036] Determine the distances between all control points except the central control point in the three-dimensional model and the central control point;

[0037] Arrange all control points in the three-dimensional model except the central control point in ascending order according to the distance to obtain a control point sequence;

[0038] The central control point is used as the point to be controlled, or the first M control points in the control point sequence and the central control point are used as the point to be controlled.

[0039] Optionally, after controlling the point to be controlled based on the deformation control gesture, completing the deformation of the three-dimensional model according to the function mapping relationship to achieve free-form modeling, including:

[0040] Determining a total distance between adjacent fingertips in the deformation control gesture;

[0041] When the total distance is greater than a threshold, the first M control points in the control point sequence and the central control point are used as the points to be controlled, and the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling;

[0042] When the total distance is less than or equal to a threshold, the central control point is used as the point to be controlled, and the three-dimensional model is deformed according to the function mapping relationship to achieve free-form modeling.

[0043] In a second aspect, the present application provides a free-form modeling system based on gesture and eye movement signals, comprising: a server, a VTK software system, an eye tracker, and a gesture acquisition device;

[0044] The VTK software system is implanted in the server; the eye tracker and the gesture acquisition device are both connected to the server;

[0045] The eye tracker is used to collect user eye movement signals; the gesture collection device is used to collect user gestures; the server is used to implement the above-mentioned free form modeling method based on gestures and eye movement signals based on the user eye movement signals and user gestures, so as to complete the construction, scaling, rotation and deformation of the three-dimensional model in the VTK software system to realize free form modeling.

[0046] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-mentioned free-form modeling methods based on gestures and eye movement signals.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned free-form modeling methods based on gestures and eye movement signals.

[0048] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned free-form modeling methods based on gestures and eye movement signals.

[0049] According to the specific embodiments provided in this application, this application has the following technical effects:

[0050] The present application provides a free-form modeling method, system, device, medium and product based on gestures and eye movement signals. The scaling and rotation of a three-dimensional model can be completed by using a perspective scaling gesture and a perspective rotation gesture based on the user's gestures; and after the control point is controlled by a deformation control gesture based on the user's gesture, the deformation of the three-dimensional model is completed according to a function mapping relationship to realize free-form modeling, so that the user's interaction method in the modeling process can be closer to the "eye-gesture control" mode in daily life. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 This is a diagram of the application environment of a free-form modeling method based on gestures and eye movement signals in one embodiment of the present application.

[0053] Figure 2 A flowchart of a free-form modeling method based on gesture and eye movement signals is provided in accordance with an embodiment of the present application.

[0054] Figure 3 A schematic diagram of a free-form modeling process provided in another embodiment of the present application.

[0055] Figure 4 A schematic diagram of completing the scaling of a three-dimensional model using gesture instruction 1 provided in an embodiment of the present application.

[0056] Figure 5 A schematic diagram of using gesture instruction 2 to complete the rotation of a three-dimensional model provided in an embodiment of the present application.

[0057] Figure 6 A schematic diagram of a visual transformation process provided in another embodiment of the present application.

[0058] Figure 7 A schematic diagram of gesture selection for a single point to be controlled and multiple points to be controlled provided in an embodiment of the present application.

[0059] Figure 8 A schematic diagram of using gesture command 3 to complete the dragging of a three-dimensional model provided in an embodiment of the present application.

[0060] Fig. 9 A flow chart of three-dimensional model deformation control using multiple points to be controlled is provided in one embodiment of the present application.

[0061] Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0064] The free-form modeling method based on gesture and eye movement signals provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the user gesture and the user eye movement signal to the server 104. After receiving the user gesture and the user eye movement signal, the server 104 completes the scaling and rotation of the three-dimensional model based on the perspective scaling gesture and the perspective rotation gesture in the user gesture. After the eye movement capture is triggered by the trigger gesture in the user gesture, the control point to be controlled is determined based on the user eye movement signal. After the control point to be controlled is controlled based on the deformation control gesture in the user gesture, the deformation of the three-dimensional model is completed according to the function mapping relationship to realize free form modeling. Before free form modeling, the server also needs to build a three-dimensional model, obtain the sampling points on the three-dimensional model, and automatically arrange the control points of the three-dimensional model based on the least squares method. The distribution of the control points after arrangement is adaptively fitted with the shape of the three-dimensional model. Then, a function mapping relationship between the control points and the sampling points is established. The server 104 can feedback the obtained free form modeling results to the terminal 102. In addition, in some embodiments, the free-form modeling method based on gestures and eye movement signals can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform free-form modeling on the user gestures and the user eye movement signals, or the server 104 can obtain the user gestures and the user eye movement signals from the data storage system and perform free-form modeling on the user gestures and the user eye movement signals.

[0065] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0066] In an exemplary embodiment, Figure 2As shown, a free-form modeling method based on gesture and eye movement signals is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 200 to 206. Among them:

[0067] In a first aspect, the present application provides a free-form modeling method based on gesture and eye movement signals, comprising:

[0068] Step 200: Construct a three-dimensional model.

[0069] Step 201: Acquire sampling points on the three-dimensional model, and automatically arrange the control points of the three-dimensional model based on the least square method. The distribution of the arranged control points is adaptively matched to the shape of the three-dimensional model.

[0070] Step 202: Establish a function mapping relationship between control points and sampling points.

[0071] Step 203: Obtain user gestures and user eye movement signals. User gestures include at least perspective zoom gestures, perspective rotation gestures, trigger gestures, and deformation control gestures.

[0072] Step 204: completing the scaling and rotation of the three-dimensional model based on the perspective scaling gesture and the perspective rotation gesture.

[0073] Step 205: After the eye movement capture is triggered by the trigger gesture, the point to be controlled is determined based on the user's eye movement signal.

[0074] Step 206: After the points to be controlled are controlled based on the deformation control gesture, the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling. The points to be controlled are one or more control points of the three-dimensional model.

[0075] Implementing the above steps 200 to 206 can make the user's interaction method in the modeling process closer to the "eye-control" mode in daily life.

[0076] In another exemplary embodiment of the present application, before performing the free operation deformation of the geometric model, the model needs to be discretized. Based on this, the implementation process of the above step 200 of the present application may include:

[0077] Step 01: Get the initial model. For example, import the 3D model in common formats such as STP or IGS into the 3D modeling software SolidWorks.

[0078] Step 02: Discretize the initial model. For example, convert the 3D model in common formats such as STP or IGS imported into the 3D modeling software SolidWorks into an OBJ file. The OBJ file is a standard 3D model format, and the discretization of the 3D model can be completed through format conversion.

[0079] Step 03: Import the discretized initial model into the VTK (Visualization Toolkit) software system to obtain a three-dimensional model. In this embodiment, the VTK software system provides a free deformation modeling environment.

[0080] In another exemplary embodiment of the present application, in the VTK software system, sampling points on the three-dimensional model are obtained, and control points of the three-dimensional model are automatically arranged based on the least squares method. Based on this, the implementation process of the above step 201 can be described as:

[0081] Step 11: Obtain the dimension data of the three-dimensional model. The dimension data includes: length data, width data and height data.

[0082] Step 12: Generate a minimum bounding box of the three-dimensional model based on the size data of the three-dimensional model. For example, the minimum bounding box of the three-dimensional model is generated using the length, width, and height data of the three-dimensional model (ie, the object to be deformed).

[0083] Step 13: Perform mesh division in the three dimensions of the 3D model to divide the minimum bounding box into multiple sub-hexahedrons. For example, divide the bounding box into K sub-hexahedrons by K equal divisions in the three dimensions. 3 A hexahedron.

[0084] Step 14: Filter out the sub-hexahedrons containing the 3D model entity, and use the centroid of the contained 3D model entity as the sampling point. For example, the sub-hexahedrons are classified according to the situation of the internal surrounding model entity, and the sub-hexahedrons that only partially contain the model entity are filtered out for surface feature sampling. In this case, the centroid of the internal model entity is taken as the sampling point.

[0085] Step 15: Use the least squares method to fit the tangent plane of each sampling point to obtain the initial vector. For example, by collecting data from the neighboring points of the sampling point, converting it into eigenvalue decomposition of the covariance matrix, and finding the eigenvector corresponding to the minimum eigenvalue, the normal vector at the sampling point is obtained. Specifically:

[0086] Assume the tangent plane equation of the sampling point is:

[0087] Ax+By+Cz+D=0.

[0088] Where A, B, and C represent the components of the normal vector of the tangent plane on the x, y, and z axes, respectively, and D represents the vertical distance from the origin to the tangent plane.

[0089] Calculate the l nearest neighbor points closest to the sampling point, and the l nearest neighbor points are recorded as set L:

[0090] L = {(x i ,y i ,z i )|i=1,2,...,l}. (x i ,y i ,z i ) are the coordinates of the neighboring points.

[0091] Find the centroid of the set of neighboring points, and use this centroid as the reference point to represent the coordinates of all neighboring points, which is:

[0092]

[0093] In the formula, (x' i ,y' i ,z' i ) are the coordinates of the neighboring points after re-expression.

[0094] Fitting the tangent plane using the least squares method solves the following minimum problem:

[0095]

[0096] In the formula, Q represents the solution to the minimum problem, and min represents the minimum value.

[0097] By taking the derivative and setting it to zero, we get the following system of equations:

[0098]

[0099] Pair Matrix Perform eigenvalue decomposition, and the eigenvector corresponding to the minimum eigenvalue obtained is the solution to the minimum problem.

[0100] Step 16: Normalize the initial vector to obtain the normal vector corresponding to each sampling point. Since the vector (A, B, C) is a normal vector of the plane Ax+By+Cz+D=0, the unit normal vector of the tangent plane at the sampling point is obtained by normalizing the vector, and this unit normal vector can be used as the normal vector at the sampling point.

[0101] Step 17, redirect the normal vector, and translate the sampling point along the redirected normal vector direction to obtain the control point. Since the normal vector direction is still uncertain at this time, the normal vector must be redirected in this step to ensure that the normal vector points to the outside of the three-dimensional model. For example, determine the product value of the vector of the sub-hexahedron surface mesh division point and the normal vector of the center of gravity of the three-dimensional model entity. Determine the direction of the normal vector based on the positive or negative value of the product value to obtain the redirected normal vector.

[0102] In another exemplary embodiment of the present application, after completing the control point layout, a function mapping relationship between the control points and the sampling points on the three-dimensional model is established. When the user (i.e., the operator) moves the control point, the minimum bounding box (the control volume in this case) is deformed. According to the mapping relationship, the geometric object (i.e., the three-dimensional model) placed inside the control volume will also have a corresponding trend change, thereby achieving the desired deformation effect. Figure 3 As shown, the overall deformation process from control point layout to dragging the control points to cause the object to deform can be obtained.

[0103] In another exemplary embodiment of the present application, the operation of the modeling perspective can be implemented based on a gesture interaction interface. In this embodiment, corresponding to the above step 204, it mainly includes two perspective transformation operation instructions: the first is perspective scaling. This operation instruction only changes the size of the model in the field of view, and does not change the observation angle. The second is perspective rotation. This operation instruction only changes the observation angle of the model, and does not change the size of the model in the field of view. Both of these operation instructions are relatively common operations in CAD (Computer Aided Design) operation modeling. Among them, the perspective transformation process is as follows: Figure 6 The specific correspondence between the user gestures and the instructions for implementing the above operation instructions is as follows Figure 4 and Figure 5 shown.

[0104] For example, the implementation process of the above two perspective transformation operation instructions can be described as follows:

[0105] (1) Zooming the viewing angle uses Figure 4 The dynamic gesture instruction 1 shown, when recognizing that the two-hand gesture is open, collects the initial distance between the two hands, and changes the distance between the three-dimensional model and the camera according to the change in the distance between the two hands, thereby achieving the effect of zooming the viewing angle. The calculation formula for the camera position is:

[0106]

[0107] In the formula, A n is the change in the distance between the two hands, A n-1 is the initial distance between the two hands, L0 is the initial distance between the palms of the two hands, Ln is the distance between the palms of the two hands, x0, y0, and z0 are the three-dimensional coordinates of the camera position in the initial gesture VTK software system, and x n ,y n 、z n are the three-dimensional coordinates of the camera position in the VTK software system after the gesture starts. k1 is the scaling parameter, which can be adjusted according to user habits, and the default value is 1.2. After each gesture switch, L0 is re-collected, and x0, y0, and z0 are re-acquired.

[0108] Among them, the camera is a function that comes with the VTK software system, so we will not go into details about the camera here.

[0109] (2) The rotation command is used as follows Figure 5 The dynamic gesture instruction 2 shown in the figure, when recognizing that the left hand gesture is open and the right hand gesture is clenched, collects the initial position of the left hand palm coordinates, and rotates the perspective camera of the 3D model according to the subsequent position change to achieve the effect of rotating the perspective. The camera rotates X degrees clockwise along the vertical axis and Y degrees clockwise along the horizontal axis. The X and Y calculation formulas are:

[0110] X,Y=-k2·(X n -X0),-k2·(Y n -Y0).

[0111] Where X0 and Y0 are the x-axis and y-axis components of the initial palm center coordinates. n , Y n The x and y axes are used to collect the palm center coordinates. k2 is a rotation parameter that can be adjusted according to user habits. The default value is 0.01.

[0112] In another exemplary embodiment of the present application, in the free-form modeling process based on control points, the operation instructions used are mainly divided into model operation instructions and viewpoint operation instructions. Model operation instructions are used to change the appearance of the model and change the model parameters. In this embodiment, they are control point selection, switching single / multiple control points, and dragging control points. Viewpoint operation instructions are to change the viewing angle of the model without changing the appearance and parameters of the model, such as the zoom view and rotation view control provided in the above embodiments. Among them, gesture actions are mainly responsible for movement-related operations, such as dragging, scaling, rotating and other operations, because of their high flexibility. A very critical feature in eye movement interaction is the gaze point, and the activation area and control point selection can be changed by changing the position of the gaze point.

[0113] In actual application, the central control point to be edited is selected through the user's sight (i.e., the user's eye movement signal). The range of hand opening is collected to determine whether to select a single control point or multiple control points around the central control point for deformation control. Based on this, the implementation process of the above steps 205 and 206 can be described as:

[0114] Step 21: Perform Kalman filtering on the eye movement signal to obtain the local coordinates of the gaze point. For example, the eye movement capture is triggered by a gesture to start collecting eye movement signals. The eye movement signal is Kalman filtered to obtain the local coordinates P of the gaze point. P is the local coordinate of the pixel point of the gaze point relative to the entire screen, and the x and y coordinates are both in the range of [-1, 1].

[0115] Step 22: When the change speed of the gaze point is less than the speed threshold, the coordinate data of the gaze point is collected according to the number of frames. For example, when the change speed of the eye movement gaze point is less than the speed threshold v1, the collection starts.

[0116] Step 23: When the number of collected frames exceeds the frame number threshold, the average value of the coordinate data of the gaze point is determined to obtain the local coordinates of the gaze point relative to the screen. For example, when the collected data exceeds the frame number threshold H, the x and y coordinates are averaged to obtain the local coordinates E of the gaze point relative to the screen.

[0117] Step 24: Determine the neighborhood of the control point based on the center position and radius of the control point, and determine whether the local coordinates of the gaze point relative to the screen are located in a neighborhood. When the local coordinates of the gaze point relative to the screen are located in a neighborhood, the control point corresponding to the local coordinates of the gaze point in the three-dimensional model is used as the central control point. When the local coordinates of the gaze point relative to the screen are located in multiple neighborhoods, the control point corresponding to the neighborhood closest to the local coordinates of the gaze point in the three-dimensional model is used as the central control point.

[0118] For example, the square with a side length of R at point P is set as the neighborhood S of point P (that is, the gaze point corresponding to the local coordinate P). If the coordinate value of the local coordinate E is within the neighborhood S of point P, it is considered that point P is selected as the central control point. If there are multiple control points P that meet the requirements at the same time, the one closest to the screen is selected as the central control point.

[0119] During step 24, in actual application, because the center of the control point has a Cartesian coordinate value in the coordinate system of the VTK software system, and the eye gaze coordinates are two-dimensional coordinates relative to the screen, the coordinates of the center of the control point need to be converted to the coordinate system. Using the coordinate system inversion function in the VTK software system, the Cartesian coordinate value of the control point in the world coordinate system can be converted to the two-dimensional coordinate of the VTK field of view coordinate system to obtain point P.

[0120] The gaze point coordinates are converted into local two-dimensional coordinates in the VTK software system remodeling window according to the screen size, modeling window size and other conditions to obtain the local coordinates E.

[0121] Step 25: Determine the distances between all control points except the central control point in the three-dimensional model and the central control point.

[0122] Step 26: Arrange all control points in the three-dimensional model except the central control point in ascending order according to distance to obtain a control point sequence.

[0123] Step 27: The central control point is used as the point to be controlled, or the first M (eg, 4) control points in the control point sequence and the central control point are used as the point to be controlled. When the central control point is switched, the range of the multi-point control points also changes accordingly.

[0124] For example, after the range of single / multi-point control is determined in step 27, switching between single / multi-point control is mainly implemented based on dynamic gesture instructions. In daily life, people often use a wide open palm to represent big and more, and use grabbing and pinching gestures to represent small and less. Based on this habit, the dynamic gesture instructions set are as follows: Figure 7 The opening range of the left hand is described by the sum S of the distances Sn between the fingertips of adjacent fingers. When the total distance S of the fingertips is greater than the threshold m, it is determined that multiple control points are selected, otherwise, a single control point is selected. Figure 7 Part (a) and Figure 7 Part (b) can obtain the corresponding relationship between the selection gestures of single control point and multiple control points and the number of control points.

[0125] Step 28: Determine the total distance between the fingertips of adjacent fingers in the deformation control gesture. When the total distance is greater than the threshold, the first M control points in the control point sequence and the center control point are used as the control points to be controlled, and the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling. When the total distance is less than or equal to the threshold, the center control point is used as the control point to be controlled, and the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling.

[0126] After the range of the control points is determined based on step 27, the three-dimensional model can be modified and deformed using gesture commands. For example, multi-control point manipulation is performed by Figure 8 The gesture command 3 shown is implemented to achieve the effect of dragging the control point to deform the model. The dragging gesture makes the operation more in line with the requirements. Among them:

[0127] (1) Drag the control point to achieve Figure 8 The gesture command 3 shown in the figure, when the gesture command is recognized, the initial palm position is collected, and the control point displacement ΔP is changed according to the change of the hand position. x,y,z , to achieve deformation. ΔPx,y,z The calculation formula is:

[0128]

[0129] Where k3 is the drag coefficient, which proportionally reduces the actual displacement of the gesture and the displacement of the control point in the window. The default value is 2.5.

[0130] (2) The multi-point deformation logic adopts a polling method. Every time a gesture completes a movement and the control point moves, it will traverse each control point being dragged, and input the displacement change of each control point being moved into the FFD algorithm. When the deformation calculation caused by all control points is completed, the multi-point deformation is completed. In actual operation, the model sampling rate can be reduced or the computer performance can be improved to improve the hand tracking of the gesture drag model deformation. For example, the process of deformation control using n control points is as follows: Fig. 9 shown.

[0131] Based on the above description, the free-form modeling method based on gesture and eye movement signals provided by the present application is based on radial basis functions, envelops the three-dimensional model (referred to as model) within the control points, establishes a function mapping relationship between the coordinates of each point on the model and the coordinates of the control points, and realizes the control of the model deformation by moving the control points. The control points are adaptively arranged according to the shape and size of the three-dimensional model. The free-form deformation environment is connected with the hand-eye collaborative interaction method through a program interface to realize three operations: (1) The user can realize operations such as zooming in, zooming out and rotating the modeling perspective based on gesture recognition. (2) Select the center control point to be edited by the eyes. By collecting the amplitude of the hand opening, it is determined whether to select a single control point or multiple surrounding control points. (3) Move the control point through gesture operation to control the three-dimensional model to produce free deformation. By introducing two new interaction methods, gesture and eye movement, the present application realizes the control point selection based on hand-eye fusion, the control point movement operation based on gesture, and the modeling perspective adjustment function in the free deformation environment, and drives the modeling module to complete the free-form modeling process.

[0132] Based on the same inventive concept, the embodiment of the present application also provides a free-form modeling system based on gestures and eye movement signals for implementing the free-form modeling method based on gestures and eye movement signals involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the free-form modeling system based on gestures and eye movement signals provided below can refer to the limitations of the free-form modeling method based on gestures and eye movement signals above, and will not be repeated here.

[0133] In an exemplary embodiment, a free-form modeling system based on gestures and eye movement signals is provided, including: a server, a VTK software system, an eye tracker, and a gesture acquisition device.

[0134] The VTK software system is embedded in the server. The eye tracker and gesture acquisition device are connected to the server.

[0135] The eye tracker is used to collect user eye movement signals. The gesture collection device is used to collect user gestures. The server is used to implement the above-mentioned free-form modeling method based on gestures and eye movement signals based on user eye movement signals and user gestures, so as to complete the construction, scaling, rotation and deformation of the three-dimensional model in the VTK software system and realize free-form modeling.

[0136] As an optional implementation, the gesture acquisition device may be a LEAP MOTION sensor, and the eye tracker may be a Tobii Spectrum 150 eye tracker.

[0137] In summary, the devices that the interaction process of known designs mainly relies on are mice and keyboards, which greatly limit the designer's work scope and design methods. However, this application mainly uses gesture acquisition equipment and eye trackers to complete the collection of user hand-eye information, and uses computer programs to process it to complete the capture of gesture movements and eye gaze points. The operation mode of gesture and eye movement fusion is combined with the free model deformation system to realize the eyes to find and activate control points, and gesture movements are used to change the viewing angle of the model, change the number of control points and manipulate deformation, realizing the "eye-selection and control" operation mode. The introduction of multi-point control has accelerated the efficiency of modifying the model. The setting of gestures conforms to the electronic device operation habits of most users, and the eye gaze load is small. The modeling interaction method proposed in this application can provide users with a new free modeling mode for product concept design. The interaction process is intuitive and friendly, and the product deformation transition is smooth, which can give full play to the designer's innovative design ability.

[0138] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store free-form modeling data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a free-form modeling method based on gestures and eye movement signals is implemented.

[0139] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0140] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0141] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0145] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A free-form modeling method based on gesture and eye movement signals, characterized in that: The free-form modeling method based on gesture and eye movement signals includes: Build 3D models; Acquire sampling points on the three-dimensional model, and automatically arrange control points of the three-dimensional model based on the least square method; the distribution of the arranged control points is adaptively matched to the shape of the three-dimensional model; Establishing a function mapping relationship between the control point and the sampling point; Acquire user gestures and user eye movement signals; the user gestures at least include perspective zoom gestures, perspective rotation gestures, trigger gestures and deformation control gestures; Complete scaling and rotation of the three-dimensional model based on the perspective scaling gesture and the perspective rotation gesture; After the trigger gesture is used to trigger eye movement capture, a point to be controlled is determined based on the user's eye movement signal; After the points to be controlled are controlled based on the deformation control gesture, the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling; the points to be controlled are one or more control points of the three-dimensional model.

2. The free-form modeling method based on gesture and eye movement signals according to claim 1, characterized in that: Construct 3D models, including: Get the initial model; Discretizing the initial model; The discretized initial model is imported into the VTK software system to obtain the three-dimensional model.

3. The free-form modeling method based on gesture and eye movement signals according to claim 1, characterized in that: In the VTK software system, sampling points on the three-dimensional model are obtained, and control points of the three-dimensional model are automatically arranged based on the least square method, including: Acquire dimension data of the three-dimensional model; the dimension data includes: length data, width data and height data; Generate a minimum bounding box of the three-dimensional model based on the size data of the three-dimensional model; Performing mesh division in three dimensions of the three-dimensional model respectively, dividing the minimum bounding box into a plurality of sub-hexahedrons; Screening out a sub-hexahedron containing a three-dimensional model entity, and using the center of gravity of the contained three-dimensional model entity as the sampling point; Fitting the tangent plane of each sampling point using the least square method to obtain an initial vector; Normalizing the initial vector to obtain a normal vector corresponding to each sampling point; The normal vector is redirected, and the sampling point is translated along the direction of the redirected normal vector to obtain the control point.

4. The free-form modeling method based on gesture and eye movement signals according to claim 3, characterized in that: Redirecting the normal vector includes: Determine the product value of the vector from the center of gravity of the three-dimensional model entity to the meshing point on the surface of the sub-hexahedron and the normal vector; The direction of the normal vector is determined based on the positive or negative value of the product value, and the redirected normal vector is obtained.

5. The free-form modeling method based on gesture and eye movement signals according to claim 1, characterized in that: After the trigger gesture is used to trigger eye movement capture, the point to be controlled is determined based on the user's eye movement signal, including: Performing Kalman filtering on the eye movement signal to obtain the local coordinates of the gaze point; When the change speed of the gaze point is less than the speed threshold, the coordinate data of the gaze point is collected according to the number of frames; When the number of acquired frames exceeds a frame number threshold, an average value of the coordinate data of the gaze point is determined to obtain a local coordinate of the gaze point relative to a screen; the screen is a medium for displaying a three-dimensional model; Determine a neighborhood of the control point based on the center position and radius of the control point, and determine whether the local coordinates of the gaze point relative to the screen are located within one of the neighborhoods; When the local coordinates of the gaze point relative to the screen are located within one of the neighborhoods, a control point in the three-dimensional model corresponding to the neighborhood is used as a central control point; When the local coordinates of the gaze point relative to the screen are within the plurality of neighborhoods, a control point corresponding to the neighborhood closest to the local coordinates of the gaze point is used as a central control point; Determine the distances between all control points except the central control point in the three-dimensional model and the central control point; Arrange all control points in the three-dimensional model except the central control point in ascending order according to the distance to obtain a control point sequence; The central control point is used as the point to be controlled, or the first M control points in the control point sequence and the central control point are used as the point to be controlled.

6. The free-form modeling method based on gesture and eye movement signals according to claim 5, characterized in that: After the point to be controlled is controlled based on the deformation control gesture, the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling, including: Determining a total distance between adjacent fingertips in the deformation control gesture; When the total distance is greater than a threshold, the first M control points in the control point sequence and the central control point are used as the points to be controlled, and the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling; When the total distance is less than or equal to a threshold value, the central control point is used as the point to be controlled, and the deformation of the three-dimensional model is completed according to the function mapping relationship to achieve free-form modeling.

7. A free-form modeling system based on gesture and eye movement signals, characterized in that: The free-form modeling system based on gesture and eye movement signals comprises: a server, a VTK software system, an eye tracker and a gesture acquisition device; The VTK software system is implanted in the server; the eye tracker and the gesture acquisition device are both connected to the server; The eye tracker is used to collect user eye movement signals; the gesture collection device is used to collect user gestures; the server is used to implement the free form modeling method based on gestures and eye movement signals as described in any one of claims 1 to 6 based on the user eye movement signals and user gestures, so as to complete the construction, scaling, rotation and deformation of the three-dimensional model in the VTK software system to realize free form modeling.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the free-form modeling method based on gesture and eye movement signals as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the free-form modeling method based on gesture and eye movement signals described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the free-form modeling method based on gesture and eye movement signals described in any one of claims 1 to 6 is implemented.

Images