System and method for three-dimensional graphical user interface with wide applicability

Abstract

A system and method with a three-dimensional graphical user interface. The three-dimensional graphical user interface provides an enhanced user interactive experience, which is achieved by the user controlling the movement of an object of any size with a number of independent degrees of freedom and using sufficient degrees of freedom to represent the movement. During operation, the three-dimensional graphical user interface obtains position data in the form of absolute addresses and translation and rotation motion vectors from a special three-dimensional navigator using color detection and motion detection methods. The three-dimensional navigator converts the color indicator data for use by the three-dimensional graphical user interface. During the conversion process, the three-dimensional navigator uses a matrix to represent the six degrees of freedom of the object's movement.

Description

Technical Field

[0001] The present invention relates to a three-dimensional graphical user interface (3D GUI), which may also be referred to as a 3D graphical interface, a 3D human-machine interface or a 3D window, and which may be used on a computer, an electronic display, a control system or an electro-mechanical system. The 3D graphical user interface of the present invention provides an absolute address, linear and non-linear motion vectors to a 3D object, so that the 3D object can move with at least three independent degrees of freedom according to the principle of physical kinematics; and the present invention may use a graphic rendering device to present the object or the motion to a user for viewing. Background Art

[0002] It is known that in general, a so-called graphical user interface (GUI) refers to a software module that is embedded in an electronic system or a computer. Or more specifically, the software module can be embedded in an operating system of an electronic system or a computer, or embedded in one or more network servers. For a graphical user interface, its ultimate goal is to make its users feel that they can engage with some graphical features displayed in a display device; for example, these graphical features are icons, menu bars, title bars, or ribbons, etc.

[0003] A graphical user interface (GUI) can not only provide the above-mentioned features to a user, but also provide the user with some functions that are not related to graphical output; for example, the function is related to audio playback, speech recognition, fingerprint reading, intelligent agents, robotic manipulation, etc.; or, the function is related to some advanced techniques of analysis; for example, the function is related to machine learning; or, the function is related to neural networks; or, the function is related to an automated device turning on or off an electronic device; or, the function is related to investigating the user's habits and desires.

[0004] A well-designed GUI is one that engages its users relatively easily and creates intuitive and / or direct interactions. In the past few decades, computer GUIs have been two-dimensional (2D). For example, icons and cursors on computers are all two-dimensional. When the era of 3D digital graphics comes, a demand for a user-engaging type of 3D GUI corresponding to the 3D digital graphics is born. The 3D GUI is a 3D GUI that can take some new functions into consideration; for example, the new functions at least include moving a character in a 3D cartoon in an intuitive, direct, real-time, and intelligent manner; or, the GUI can control a robot in an intuitive, direct, real-time, and intelligent manner according to a user's instruction.

[0005] In the past, many known technologies have proposed some methods for improving the functions of graphical user interfaces, which methods can improve the design of a graphical user interface; or, these methods can make a graphical user interface have more versatile functions, but none of the above efforts can provide the functions provided by the present invention. For example, Ullman (US 9,405,430) disclosed a graphical user interface, which includes a function selection tree (menu tree), which can reduce the distance a cursor needs to move during the selection process. Anzures (US 8,736,561) disclosed a method for adjusting the content (content) or meaning (context) of a graphical object. Tseng (US 8,954,887) disclosed a graphical user interface, when a touch screen is pressed by its user for more than a period of time, its graphical user interface will pop up a new window.

[0006] Kushman (US 9,189,254) disclosed an automated tool that can interact with multiple users on a website through a graphical interface for each user. Fostall (US 9,690,446) disclosed profiles of finger gestures that are detected by a touch screen panel, and the finger gestures make the graphical user interface of the touch screen panel more intuitive to use. Matthews (US 8,527,896) disclosed a neural network system that provides a time-domain-to-frequency-domain converter that can convert the input signals between the time domain and the frequency domain before the neural network system extracts features, thereby reducing the workload of the signal processor of the neural network system.

[0007] Figure 1D What is shown is a conventional two-dimensional (2D) graphic display device (115); the two-dimensional graphic display device may be a computer monitor. Figure 1D Also shown is a graphical user interface (105), which is applied to a two-dimensional graphic display (115) as described above. It is worth noting that the graphical user interface (105) is also two-dimensional; therefore, Figure 1D As further shown, the graphic features, such as icons (108), located on the graphical user interface (105) are also in two-dimensional format. Based on the above-mentioned correspondence, a Figure 1A The conventional navigation device (101) shown therein, such as a mouse, provides a motion vector in a two-dimensional format. Regarding this phenomenon, Figure 1C There is further illustration.

[0008] like Figure 1C and 1DAs shown, when the computer system (112) is in operation, a user moves a navigator (101) on a two-dimensional reference plane (104) to achieve the purpose of moving the cursor (111); the two-dimensional reference plane (104) can be a mouse pad or a desktop. When the mouse (101) moves along the two-dimensional reference surface (104), its image detector (102) will capture a series of images related to the two-dimensional reference surface (104) and compare them. Then the mouse (101) transmits some relative motion vectors to an electronic system (112), or to one or a group of servers (Servers), which are connected by a network, which can be an Internet network (internet), or can be indicated as a connection method with a function similar to the above-mentioned Internet.

[0009] when Figure 1D When the computer (112) shown in FIG. 1 receives the aforementioned motion vector data, Figure 1B The cursor (111) in the two-dimensional graphical user interface (105) shown here will make corresponding movements. Figure 1C Further explain the steps in detail; such as Figure 1C As shown, when a mouse (101) moves on a mouse pad or a desktop (104) with a two-dimensional motion vector (Δu, Δv), a cursor (111) located in a two-dimensional graphical user interface (105) will generate a corresponding motion vector (Δx, Δy).

[0010] In the present invention, a three-dimensional graphical user interface can be Figure 2A As shown in (207), when a conventional two-dimensional navigator (101) is used in a three-dimensional graphical user interface (3D GUI) such as (207), it will face several challenges: First, when the data of a two-dimensional mouse is to be converted into a three-dimensional format to facilitate the subsequent use of the three-dimensional graphical user interface, the matrix (matrix, array, tensor, etc.) transformation process required will consume a lot of energy and time of the central processing unit (CPU) or graphics processing unit (GPU). Second, and this point may be more important than the first point mentioned above, the conventional two-dimensional mouse (101) cannot provide angular displacement data for a three-dimensional graphical user interface; finally, the conventional two-dimensional navigator (101) lacks a widely used method (a comprehensive means) that can provide depth values. Summary of the invention

[0011] The object of the present invention is to provide a system and method for a three-dimensional graphical user interface (3D GUI) with pervasive, comprehensive and fully integrated functions.

[0012] US10,254,855, which has the same applicant as the present invention, discloses a high resolution and high sensitivity three-dimensional navigator. The technology that has not been disclosed in this invention is how a three-dimensional graphical user interface should use the robot's kinematics, stereoscopic perspective, artificial intelligence, and how to give the object six degrees of freedom when processing an object as small as a pixel, so that a computer can interact closely with a user.

[0013] Therefore, according to one aspect of the present invention, a system is provided, comprising at least:

[0014] A memory and at least one processor, wherein the processor is coupled to the memory of a computer, a display system, an electronic system or an electromechanical system; a display device is provided on the system, and a three-dimensional graphical user interface (3D GUI) is presented on the display device.

[0015] The 3D graphical user interface is capable of moving a 3D object in a 3D space represented by the 3D graphical user interface with a motion having at least three degrees of freedom, and the motion has the characteristics of a linear motion vector and / or a non-linear motion vector.

[0016] The linear motion vector and the non-linear motion vector represent translational and rotational motions, respectively. The linear motion vector and the non-linear motion vector are generated from a single gestural motion of a navigational device on a reference surface. When generating the linear motion vector and the non-linear motion vector, the system does not use other input devices or motion detection devices.

[0017] In some embodiments, after the object is moved by the navigator that measures an absolute position from the reference surface, the positional accuracy of the object remains stable during the movement and is not affected by accumulated errors of position.

[0018] In some other embodiments, the three-dimensional space includes a predetermined vector field, which can be combined with the linear motion vector and / or the nonlinear motion vector provided by the navigator to form a new motion vector of the object, wherein the magnitude and directionality of the new motion vector are different from another motion vector of the object formed when the navigator is not used.

[0019] In other implementations, the three-dimensional space represented by the three-dimensional graphical user interface includes multiple two-dimensional planes, wherein each of the two-dimensional planes represents a specific application or function.

[0020] In some implementations, activating or de-activating the two-dimensional plane requires providing a non-linear motion vector for an object or cursor.

[0021] In other embodiments, where the primary properties of an object accessible by the system are represented by a matrix of data rather than a single value defining a mathematical point, the object is a cursor, a graphical entity, or a physical object. The matrix is ​​shown in formula (8) in the embodiment.

[0022] In other embodiments, the system further includes at least some independent software modules, which are stored in the memory of the system in a layered architecture, partially or completely. Some of the software modules are independently (separately) focused on perspectives and / or robotic motions (robotickinematics), and the software modules can operate separately (separately) or collaboratively (collaboratively).

[0023] In other embodiments, the three-dimensional graphical user interface uses an operating system loaded into the memory of the system, and enables a user to interact with objects displayed in the three-dimensional graphical user interface, or with objects controlled by the computer, the electronic system, or the electromechanical system, through the linear motion vector and the non-linear motion vector, wherein the linear motion vector and the non-linear motion vector can be assigned or generated on a graphical entity, and the graphical entity can be as small as a pixel, or as small as the smallest drawing unit of the graphical user interface, such as a voxel.

[0024] Therefore, according to another aspect of the present invention, a system is provided, the system at least comprising:

[0025] A memory and at least one processor coupled to the memory of a computer, a display system, an electronic system, or an electromechanical system; the system is capable of enhancing the user's experience of engaging and interacting with a 3D scenery, the 3D scenery being presented on the 3D display device, the 3D scenery having a plurality of 3D vector graphics.

[0026] The system is capable of adjusting the perspective angle of a 2.5D coordinate system embedded in a 3D graphical rendering device.

[0027] The stereoscopic viewing angle is the angle of intersection between the X-axis and the Z-axis of the 2.5D coordinate system, or the angle between the X-axis and the Y-axis of the 2.5D coordinate system.

[0028] The system enables an operator to visually classify multiple graphical vectors and / or some motion vectors into several different classes, wherein the graphical vectors are entities used to construct the three-dimensional vector graphics, and each of the classes represents a combination of special feature graphics, which are related to the three-dimensional vector graphics and are different from the feature graphics of other classes.

[0029] In some embodiments, the visual classification reaches an optimal condition when the system delivers the stereoscopic view to a specific perspective angle via a starting three-dimensional position of a given world space camera and then a series of translation and rotation motion vectors.

[0030] In some other embodiments, the system is capable of interacting with the user by manipulating the three-dimensional scene, wherein some of the graphical objects displayed in the three-dimensional scene are represented by some three-dimensional nodes (such as vertices), and the three-dimensional nodes have the characteristics of a three-dimensional address and a three-dimensional motion vector, wherein each three-dimensional address has components of X, Y, and Z, or at least one three-dimensional motion vector has components of ΔX, ΔY, and ΔZ; and / or α, β, and γ; wherein the nodes are classified into several categories according to the interaction of the cursor with the nodes.

[0031] Therefore, according to another aspect of the present invention, a method for engaging a user with a plurality of objects is provided, wherein the method is executed on a computer, wherein the plurality of objects are located in a three-dimensional graphical user interface in a display system, and the method comprises at least the following steps:

[0032] A three-dimensional navigator is provided, the three-dimensional navigator being capable of communicating with the three-dimensional graphical user interface, the three-dimensional navigator being movable on a reference surface.

[0033] Alternatively, a control device having linear motion vector and / or non-linear motion vector characteristics is used to communicate with the three-dimensional graphical user interface, the three-dimensional navigator or the control device moving along a two-dimensional vector lying on a surface having at least three colors (tinted by tri-coloration) or at least three different wavelength ranges.

[0034] The two-dimensional vector is transformed into a three-dimensional vector, or the two-dimensional vector is transformed into a vector with a higher dimension than two dimensions, wherein the transformation is performed on the hue value of an area of ​​the surface dyed with the three colors, and the image of the area is captured by a color-sensitive image sensor located in the three-dimensional navigator or the control device.

[0035] Therefore, according to another aspect of the present invention, a system is provided, which can control a robot connected to the system, and the system at least comprises:

[0036] A memory and at least one processor, wherein the processor is coupled to the memory of a computer, a display system, an electronic system or an electromechanical system; a display device is provided on the system, and a three-dimensional graphical user interface (3D GUI) is presented on the display device.

[0037] A navigator is capable of controlling the position and movement of the robot in real time.

[0038] The position and motion of the robot have more than three degrees of freedom, and both the position and motion of the robot can be controlled by the system in an in-situ manner using the navigator.

[0039] The 3D GUI is capable of monitoring more than three of the degrees of freedom, and the motion is capable of being modified by the 3D GUI.

[0040] In some implementations, the robot is a structure with any shape, or a structure with joints.

[0041] In some implementations, the structure having an arbitrary shape is a cartoon character, or other graphical structure.

[0042] In some other implementations, the robot is a movable device other than the three-dimensional graphical user interface (3D GUI).

[0043] In other embodiments, where the robot is invisible but still controllable by a user, the robot functions as a perspective camera that views the view presented by the 3D GUI from a specific position and / or perspective angle specified by the robot.

[0044] In other embodiments, if the system is configured such that the robot is invisible but still controllable by a user, the robot performs one of the following functions:

[0045] (a) Execute the signing procedure.

[0046] (b) Execute the function of the drawing program.

[0047] (c) Execute the cartoon function in the animation.

[0048] (d) Execute the actions of characters in animated video games.

[0049] (e) Procedures for performing three-dimensional imaging diagnosis.

[0050] The robot performing one of the above functions can be regarded as a finger, hand, waist, elbow, arm, leg, head or torso of the operator's body.

[0051] Therefore, according to another aspect of the present invention, a computer system is provided, the computer system at least comprising:

[0052] Display device.

[0053] Memory.

[0054] A system or at least one processor, wherein the system or the at least one processor is coupled to the memory of a computer, a display system, an electronic system or an electromechanical system; the system also includes one or more display servers.

[0055] A three-dimensional graphical user interface (3D GUI), which is projected onto the display device, and which at least includes a layered configuration of software modules, wherein the layered configuration of software modules is stored in the memory of the computer system, and the layered configuration of software modules can interact with the processor and be executed by the processor.

[0056] The three-dimensional graphical user interface can establish some areas on the display device, which are called windows; in the windows, some graphical objects can be displayed in a three-dimensional coordinate system; wherein the graphical objects can be represented by some absolute positions located in the three-dimensional coordinate system.

[0057] The graphical object can move following the translation and rotation in the three-dimensional kinematics of the object. The total degree of freedom of the three-dimensional motion of the graphical object (including at least translation and rotation) is at least three, and the accuracy of the position of the graphical object will not be lost due to the movement.

[0058] A three-dimensional navigator, wherein the three-dimensional navigator can select whether to use an application interface (API) to interact with the three-dimensional graphical user interface (3D GUI), and the three-dimensional navigator can provide an absolute position in the three-dimensional coordinate system for the graphical objects displayed in the three-dimensional graphical user interface (3D GUI).

[0059] By executing the software module stored in the memory of the computer system, the graphical object is moved between the two absolute positions with at least three degrees of freedom in the display device; during the movement, the movement of the graphical object follows the translational movement and rotational movement in kinematics.

[0060] In some embodiments, the software module includes some instructions readable and executable by the processor, and the instructions are used to generate some robot movements, or generate some graphical actions according to the laws of three-dimensional kinematics and perspective.

[0061] In some other implementations, the software module is further managed by a display server in operation so that the functions of the software module are appropriately displayed in the window.

[0062] In other embodiments, the software modules include some process modules that independently control cursor movement, perspective manipulation, robotics, and machine learning programs.

[0063] Therefore, according to another aspect of the present invention, a method for simultaneously interacting with multiple three-dimensional objects in a three-dimensional graphical user interface (3D GUI) is provided, the method being executed on a computer, the method being caused by a pre-set action of a three-dimensional navigator, the method being used to activate a mechanism, the mechanism being used to select or / and deselect an event state, and characterized in that the pre-set action at least comprises the following steps:

[0064] When the user moves the three-dimensional navigator on a two-dimensional reference surface, by placing a finger on the upper surface of the three-dimensional navigator, the three-dimensional cursor located in the three-dimensional graphical user interface (3D GUI) moves on the two-dimensional plane in a corresponding manner, and the tilting condition of the two-dimensional plane is controlled according to the position of the finger on the surface of the three-dimensional navigator.

[0065] At the beginning of the action, the user presses a button on the three-dimensional navigator or presses the mechanism on the three-dimensional navigator to determine the position of the first endpoint, and the three-dimensional navigator is located on a two-dimensional reference surface; continues to press the button or keeps the mechanism in a pressed state until the cursor has moved a certain distance along the two-dimensional plane and reaches the second endpoint. Using the first endpoint and the second endpoint as a diagonal line, the three-dimensional graphical user interface (3D GUI) establishes a volume in the three-dimensional space, and the volume enables the user to select or / and deselect the objects enclosed by the volume.

[0066] In some implementations, a detector is provided on the surface of the navigator, and the detector is capable of converting the two-dimensional movement and / or position of the finger on the surface into a three-dimensional movement and / or position signal.

[0067] Therefore, according to another aspect of the present invention, a method for controlling / adjusting the motion of a robot in an in-situ manner is provided, wherein the method is executed on a computer and comprises at least the following steps:

[0068] Multiply some matrices, where each matrix has dimension no less than three.

[0069] Using a maneuvering device that can provide three-dimensional translation and rotation motion vectors simultaneously, selected joints or movable parts on the robot change their position or movement, while other unselected joints or parts on the robot continue to undergo unchanged pre-set movements.

[0070] In some implementations, the movement method of the robot is a three-dimensional spiral movement (3D spiral).

[0071] In some other embodiments, the robot's movement method includes at least forward kinematics and inverse kinematics, and at least one movement generated by the forward kinematics and / or inverse kinematics method is generated by the moving device.

[0072] In other embodiments, the method of robot movement can be used to drive the movement of interactive characters in animated video games, cartoons, medical images, or movies.

[0073] Therefore, according to another aspect of the present invention, there is provided an image rendering method executed on a computer, the method creating a digital impressionistic graphical rendering effect on a computer, an electronic control system or an electromechanical system, the method comprising at least the following steps:

[0074] A three-dimensional graphical object is generated in a three-dimensional graphical user interface (3D GUI), wherein the three-dimensional graphical object has some basic properties, one of which is that the three-dimensional graphical object has volume; wherein the three-dimensional graphical object is constructed by a number of vertices connected to each other, wherein the vertices define the external shape of the three-dimensional graphical object formed by a number of facets.

[0075] In the memory of the computer, the electronic control system or the electromechanical system, a feature vector space is created, wherein the feature vector space is used to represent the three-dimensional graphical object, wherein each vertex is associated with a multidimensional feature vector space, and wherein each vertex has a normal vector perpendicular to a small plane corresponding to the vertex.

[0076] In the memory of the computer, the electronic control system or the electromechanical system, the three-dimensional graphical user interface uses some distinct margins to separate (i.e., classify) some of the multi-dimensional feature vectors, aligns the normal vectors and the small planes in several directions, so that the vertices and the small planes defined by the vertices are classified in a way that is perceivable and recognizable to a viewer.

[0077] In some embodiments, the method uses a three-dimensional navigator that moves in a controllable manner on a colored or tinted two-dimensional reference plane to access selected vertices, and at the same time, changes some illumination intensities of a system of illumination inside the three-dimensional navigator by touching a surface element of the three-dimensional navigator, so that the direction of the normal vector of the accessed vertex changes, while also changing the visual characteristics of the graphical object near the vertex.

[0078] Therefore, according to another aspect of the present invention, a three-dimensional image rendering method executed on a computer is provided, the method comprising at least the following steps:

[0079] Some three-dimensional graphical vectors are presented in a manner that refers to at least one vanishing point.

[0080] The position of one of the vanishing points can be manipulated using artificial intelligence technology.

[0081] Some of the three-dimensional graphics vectors tracked by the method are grouped, wherein each group forms a margin with respect to the other groups, the margin being configured to be recognized by a user of the method.

[0082] When the limit reaches different values, the method generates some graphical rendering effects, or supports some different levels of interaction between the user and the method.

[0083] In some implementations, the artificial intelligence technology at least includes a support vector machine (SVM) or a neural network.

[0084] In some other implementations, the artificial intelligence technology at least includes an artificial method that can be controlled by a user.

[0085] Therefore, according to another aspect of the present invention, a non-transitory computer readable storage medium is provided, wherein the medium has some instructions. When the instructions are executed by a processor, the processor presents a three-dimensional graphical user interface (3D GUI) on a display device and performs some operations, wherein the operations at least include:

[0086] An object in three-dimensional space presented by the 3D GUI is moved with motion having at least three independent degrees of freedom, the motion being characterized by linear motion vectors and / or non-linear motion vectors.

[0087] Wherein, the linear motion vector and / or the nonlinear motion vector represent a translation motion vector and a rotation motion vector respectively, and the linear motion vector and / or the nonlinear motion vector are generated by a single posture movement of a navigator located on a reference surface, and when generating the linear motion vector and / or the nonlinear motion vector, the navigator does not need to apply other inputs or other motion detection devices.

[0088] The beneficial effects of the present invention are as follows: In order to solve the shortcomings of the known technologies mentioned above, one purpose of the present invention is to provide a three-dimensional graphical user interface (3D GUI) with wide usability ("pervasive", comprehensive and fully integrated, that is, it is widely functional and fully integrated), which enables a computer, electronic control system or electromechanical system to control the movement of an object by using sufficient degrees of freedom, thereby enhancing the user's engagement experience; it is worth noting that the above-mentioned control can be implemented on objects of any size; for example, the object can be as small as a pixel, or the object can be as large as a network composed of multiple computers. The reader should note that in the present invention document, from here on, all disclosures about the three-dimensional graphical user interface (3D GUI) can be used Figure 2A The three-dimensional graphical user interface (207) shown in the figure is used as a representative. At the same time, in order to simplify the description, the above-mentioned three-dimensional graphical user interface (207) is sometimes indicated as "the three-dimensional graphical user interface of the present invention", or "the present three-dimensional graphical user interface", or "the present three-dimensional graphical interface".

[0089] In order to achieve the above objectives, a 3D graphical user interface must provide an absolute address of a 3D object, linear and nonlinear motion vectors, so that its users can get an experience of engaging directly with a 3D object, which makes the user feel that the engagement and interaction between the user and the 3D object is extraordinary and transparent; the engagement and interaction is so transparent and extraordinary that the user no longer feels that he is engaging and interacting with a 3D object by using a graphical user interface. In addition, when a computer, electronic control system or electromechanical system provides a high resolution and high sensitivity 3D navigator as disclosed in U.S. Patent No. US10,254,855 to the 3D graphical user interface of the present invention, the 3D graphical user interface of the present invention can provide the user with the most complete capabilities and advantages; the functions of the above high resolution and high sensitivity 3D navigator will be further disclosed in the following text (for example, Section 1 of the embodiment).

[0090] It is noteworthy that the high-resolution and high-sensitivity 3D navigator can provide an absolute address to an object; the position of the object provided by the high-resolution and high-sensitivity 3D navigator can maintain consistent accuracy throughout the movement; in contrast, the position information provided by the previous navigator to the object is obtained through a series of successive approximations, which will cause the accuracy of the motion information detected by a previous navigator to continue to decline with the length of its use. The reason why the 3D navigator of the present invention can accurately detect the above motion information is that it moves on a specially colored or tinted reference surface. To further illustrate, the 3D GUI of the present invention can provide a 2.5D coordinate system (a 2D system with a separate rotational axis), and the 2.5D coordinate system can help users understand the 3D scenery by providing them with interaction with the 3D scenery; more specifically, the 3D GUI of the present invention is a rendering method using 3D vector graphics.

[0091] The 3D graphical user interface of the present invention can move a perspective camera / world space camera with linear and nonlinear motion vectors having six degrees of freedom; by moving the world space camera, the present invention achieves the purpose of controlling a perspective angle (in the present invention, "world space camera" and "stereo camera" have the same meaning and function). The 3D graphical user interface of the present invention can classify some 3D graphical vectors and / or some 3D motion vectors by manipulating the stereo angle; here, the reader should pay special attention to that the 3D graphical vectors indicated by the present invention are basic graphical entities used to construct the aforementioned 3D vector graphics or 3D motion vectors, and the 3D graphical vectors can be used to indicate the levels of user engagement of the 3D vector graphics or the 3D motion vectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1A ,B, C, and D illustrate some elements related to a conventional two-dimensional graphical user interface (2D GUI), which uses a two-dimensional navigator to move a cursor;

[0093] Figure 2A , B, C, and D illustrate some elements related to the three-dimensional graphical user interface of the present invention. The three-dimensional graphical user interface of the present invention uses a special three-dimensional navigator, and the three-dimensional navigator provides a three-dimensional motion vector of an object, and the three-dimensional motion vector has six degrees of freedom (DOF);

[0094] Figure 3A Depicted is a typical motion vector integration process that can derive the position of a relative motion detector, which is a conventional navigator;

[0095] Figure 3B A typical process of deriving a motion vector is shown, which can derive a motion vector of an absolute address detector, the absolute address detector being a three-dimensional navigator (202) disclosed in the present invention, the three-dimensional navigator being placed on a colored reference surface (205T);

[0096] Figure 4A, B, D, and E illustrate some of the most advanced motion detectors of the present time; for example, the most advanced motion detectors of the present time are a depth detection camera (401), a virtual reality goggles (405), a touch panel (402), and the three-dimensional navigator (202) disclosed in the present invention, etc.;

[0097] Figure 5A and 5B illustrates a special method used by a three-dimensional navigator (202) disclosed in the present invention, the method being related to specifying a rotational plan of a three-dimensional cursor, the method enabling the three-dimensional navigator disclosed in the present invention to stably select a three-dimensional object in a three-dimensional space;

[0098] Fig. 6A The illustration shows a multi-layered software in a 3D GUI on a windowing system. In the 3D GUI, there is a specific software layer between an operating system kernel and an input device, which is designed to control a user's viewing experience. Some vendors in the GUI market segment (e.g., Awesome) are also listed.

[0099] Figure 6B An application interface (API) is shown, which bridges different types of input devices to the three-dimensional graphical user interface disclosed in the present invention;

[0100] Fig. 7A What is illustrated is a robot that can be directly controlled by the three-dimensional graphical user interface disclosed in the present invention;

[0101] Figure 7B The depicted figure is an anthropomorphic robot (Humanoid), which uses 25 degrees of freedom to produce a variety of different movements;

[0102] Fig. 8A What is depicted is a cartoon character that uses a simplified anthropomorphic robot to produce different actions;

[0103] Figure 8B The image is from Fig. 8A The robot structure is transformed from the cartoon character in the movie. The structure of the robot is similar to Figure 7B Compared with the robot structure of cartoon characters in the film, fewer joints are used;

[0104] Fig. 9 What is depicted is a signature that is actually the result of actions generated by many neural network signals in the hand;

[0105] Fig. 10A , B, C, D, E, F, and G illustrate a patch on a polygon network, the shape of which can be achieved by manipulating the translation of its corner points and the rotation of its vertices;

[0106] Fig. 10H , I, and J illustrate that the three-dimensional graphical user interface of the present invention makes the scene presented by the three-dimensional graphical user interface of the present invention more comprehensible to the observer by aligning some graphical vectors in the three-dimensional scene;

[0107] Fig.11A And 11B shows how a rotating plane abc changes the gesture of an object;

[0108] Fig. 11C 11D shows a special case in which the illumination condition of the three-dimensional navigator disclosed in the present invention changes at the same time when performing a revolution and rotation motion (for example, the intensity of a light source in the illumination gradually dims), resulting in the generation of a spiral.

[0109] Fig.11E The illustration shows how a user of the three-dimensional navigator disclosed in the present invention can change the posture of an object by moving his or her finger to different positions on the posture detection plane (203); for example, the posture refers to the tilt of the object;

[0110] Fig.12 Depicted are the position and motion vectors of a robot gripper, the effective length of which can be changed by changing the length of the tool it holds;

[0111] Fig.13A 13B is a diagram to illustrate that the three-dimensional graphical user interface of the present invention can be compared with conventional 2D windows (e.g., WORD) closely cooperates; the basic functions of the cursors in these two figures prove that the three-dimensional graphical user interface of the present invention has a wide range of three-dimensional usability (pervasive 3D);

[0112] Fig.14 The illustrated embodiment is the hardware environment of the three-dimensional graphical user interface of the present invention; and

[0113] Fig.15 What is shown are two groups of objects composed of graphic vectors (one group is represented by black dots ·, and the other group is represented by asterisks ★). After being analyzed by the artificial intelligence classification (classifying) program (610) disclosed in the present invention, they are divided into two categories by a margin (1502), wherein the first category is (1501L) and the second category is (1501R); wherein the method used by the classification program at least includes support vector machine (SVM, 616), convolutional neural network (CNN), etc., and the size and direction of these graphic vectors can be controlled by the stereoscopic viewing angle module (607). DETAILED DESCRIPTION

[0114] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0115] Before the present invention is described in detail, it should be noted that similar elements are denoted by the same reference numerals in the following description.

[0116] The present invention claims the following three U.S. priority rights: (1) U.S. application number 16 / 056,752, filed on August 7, 2018; (2) U.S. application number 16 / 164,928, filed on October 19, 2018; (3) U.S. application number 16 / 456,009, filed on June 28, 2019.

[0117] The present invention is related to the following U.S. inventions or Chinese applications: (1) U.S. Patent No. US 9,720,525, filed on May 29, 2012; (2) U.S. Application No. 13 / 834,085, filed on March 15, 2013; (3) U.S. Patent No. US9,733,727, filed on October 15, 2013; (4) U.S. Patent No. US10,254,855, Application No. 14 / 294,369, filed on June 3, 2014; (5) U.S. Patent No. US9,703,396, filed on July 12, 2013; (6) Chinese Patent Application No. 201410673274.7, filed on November 19, 2014.

[0118] As described above, the present invention discloses a three-dimensional graphical user interface (3D GUI); Figure 2A The 3D graphical user interface shown in (207) is used on an electronic system, for example, a computer. Such a device (an electronic system having (207)) provides an absolute address, linear and non-linear motion vectors to a 3D object, giving its user an experience of engaging and interacting directly with the 3D object. For example, Figure 2A The cartoon elf Genie (204) in the three-dimensional graphical user interface (207) receives different movement directions at the position (x', y', z') (i.e., changes from n" to n'"), which enables a flying carpet (201) controlled by the elf Genie (204) to make corresponding movements according to different vectors n and n' of the rotation plane (Planeabc) (the details will be discussed in the following sections). The three-dimensional graphical user interface disclosed in the present invention not only provides a method for constructing and manipulating a three-dimensional structure (i.e., the elf 204 or a three-dimensional landscape), but also provides a complete method for manipulating a three-dimensional object such as a robot with many degrees of freedom.

[0119] After years of technological development, the known graphical user interface (GUI) has evolved and now can provide many forms of outlooks. The outlook here refers to a general form of visual presentation. At the same time, the known graphical user interface also provides some special features, which at least include hovering icons and the function of cursor quickly grabbing objects (snapping). If we make a general judgment based on the above-mentioned outlooks, a graphical user interface can be roughly divided into two-dimensional (2D), 2.5-dimensional (2.5D), or three-dimensional (3D) according to the degrees of freedom (DOF) it gives to the objects it presents (here 2.5D means that in addition to the ability to indicate the coordinate position of an object in a 2D format, it also has a rotational axis that is independent of the other two coordinate axes used in the 2D format; 2.5D hereinafter means 2D format with separate rotational axis capability).

[0120] However, from the perspective of how to enhance the user's engagement, interaction, and learning experience, the effectiveness of a graphical user interface is not just a function of the overall appearance it provides, but also includes more sophisticated and complex effects produced by the combination of software and hardware engineering.

[0121] For example, when a GUI displays an icon as if it is hovering, its observer may think that the GUI has true 3D rendering capability; but when the observer finds that the GUI cannot move the hovering icon in three dimensions, in fact, the ultimate capability and value of a GUI is determined by the full interaction between the GUI and its users, rather than by some superficial phenomena.

[0122] Cursor snapping is another example; it was originally designed to compensate for the lack of positioning accuracy of older navigators, a problem common to all relative motion detectors. However, when a modern GUI was equipped with a touch-sensitive feature (note that such a touch-sensitive GUI is still a known technology), it has the ability to use absolute addresses to approach icons. As a result, cursor snapping has become a less important feature, and the problem of cursor positioning accuracy has become a question that only the historically interested know about, or more specifically, a problem that is only affected by the older navigators ( Figure 1A (101)).

[0123] In recent years, the growth of the entire graphical user interface industry has slowed down. Occasionally, when a new input device is introduced to the market, the performance of the graphical user interface using the new input device can be improved to a certain extent, but in the past, there have been few dramatic changes that have made a clear mark in the graphical user interface industry. As the electronics industry has gradually entered the era of more complete three-dimensional graphics in recent years, many advocates have hailed an upcoming revolution in the electronics industry. In line with these expectations, a number of new gimmicky devices have also appeared on the market, such as glasses with embedded depth cameras, virtual reality headsets or goggles, etc.

[0124] Unfortunately, until now, the electronic industrial revolution expected by many of the above advocates has not occurred. In this invention, we point out that before the electronic industrial revolution expected by many of the above advocates arrives, some basic problems must be solved first. For example, how a three-dimensional graphical user interface can interact with a user in a broad and practical way is one of the above basic problems. The purpose of this invention is to provide a three-dimensional graphical user interface that can truly interact with the user. Through this three-dimensional graphical user interface, the present invention completes the evolution of graphical user interface technology.

[0125] Section 1. Obtaining the absolute address directly from a navigator is an effective way to avoid cumulative positioning errors

[0126] Most computer users have had the experience that as the capabilities of a computer's operating system's peripherals improve, the performance of the operating system's graphical user interface also gets better; but few have a deeper understanding of this phenomenon, and that understanding uses basic physics to explain exactly how a graphical user interface can fully interact with a user.

[0127] Since the mouse was first introduced to the computer industry, it has been a relative motion detection device. Such a relative motion detection mouse does not know its absolute address on a reference surface; due to this limitation, an operator must move the mouse step by step on a reference surface, like crawling, in order to drive the cursor on the screen to its final destination.

[0128] This process may not be as accurate as many people expect. In order to make the cursor movement process more efficient, Microsoft Windows operating system (such as: Windows 10, etc.) added a new function in later versions, which enables a cursor to be automatically placed at certain pre-specified positions (the function of the cursor quickly grabbing objects, snapping). This function reduces the tiny steps when a cursor crawls on the graphical user interface, thereby greatly reducing the energy and time consumption of the graphical user interface.

[0129] However, this does not mean that an efficient graphical user interface can simply obtain input data from a complex navigator and then use it. Such a complex navigator can be an electromagnetic tracker, a depth camera, an accelerometer, or a laser associated navigational device. After analyzing it to the end, we will understand that the reasons that make the three-dimensional graphical user interface of the present invention more interactive (more engaging) with the user than the known technology include at least the basic positioning method of the present invention and its method of moving objects (basic means of positioning and maneuvering); in the present invention, the above-mentioned basic positioning method adopts absolute addressing; the above-mentioned method of moving objects refers to providing translational motion vector and rotational motion vector simultaneously for one object or multiple objects.

[0130] In order to achieve the above-mentioned purpose of being more interactive with the user, the three-dimensional graphical user interface of the present invention is combined with a special three-dimensional navigator, that is, Figure 2B (202), cooperation, the special three-dimensional navigator is disclosed in U.S. Patent No. US10,254,855, the present invention is fully integrated with the case, and the entire invention of the case is incorporated as a reference. It is precisely because of the close cooperation between the three-dimensional graphical user interface of the present invention and the three-dimensional navigator that a computer or an electronic system becomes a computer or electronic system that truly interacts with the user. Although other navigators can also be used in combination with the three-dimensional graphical user interface of the present invention, the performance they obtain is usually better than the above-mentioned three-dimensional graphical user interface of the present invention and the above-mentioned special three-dimensional navigator (that is, Figure 2B The efficiency achieved through close cooperation with 202) is poor.

[0131] Therefore, one aspect of the 3D graphical user interface of the present invention is that when the 3D graphical user interface of the present invention is combined with the high resolution, high sensitivity 3D navigator ( Figure 2BWhen used in combination with 202), the 3D graphical user interface of the present invention can provide an absolute address for an object. As a result, when the high-resolution, high-sensitivity 3D navigator is on a specially colored or tinted reference surface (e.g. Figure 2D When moving on the (205T) in the high resolution, high sensitivity three-dimensional navigator, the cursor or object corresponding to the high resolution, high sensitivity three-dimensional navigator can maintain its positioning accuracy without encountering the problem of continuous decline in positioning accuracy. With the function of maintaining positioning accuracy, the aforementioned cursor snapping function of the object is almost no longer needed.

[0132] Section 2. Treating an object as a real object rather than a mathematical point is an important function of the three-dimensional graphical user interface of the present invention

[0133] There is a further aspect of the 3D GUI of the present invention, which refers to the ability of the 3D GUI of the present invention to simultaneously provide an object with translational (or linear) and rotational (or non-linear) motion vectors. Since the known technology treats all objects as mathematical points, the known technology, including the most advanced GUIs today, lacks a widely used method that can directly combine the translational and rotational motion vectors of an object; readers should note that in mathematics, we call a point a point; and in painting, we call a dot in a pattern a dot; in the present invention, these two terms (mathematical point, painting point; point, dot) can be used interchangeably; if there is any special distinction between them in any case, we will remind you.

[0134] Readers of the present invention should know without any reminder that in the real world, all objects perform translation and rotation at the same time when they move. The reason is simple. Real objects are not just some mathematical points. They also have physical extensions. Therefore, when a graphical user interface intends to provide a vivid, fully interactive experience to its users, it must provide a method to simultaneously control the above two movements (translation and rotation) of a real object. This problem has been hidden by the previous two-dimensional user graphical interface (2D GUI) in the past.

[0135] In a conventional two-dimensional user graphical interface, the position of an object is indicated by two parameters in a two-dimensional coordinate system; for example, the two parameters are (X, Y). Correspondingly, the motion vector of a two-dimensional object is also indicated by a set of two-dimensional parameters; for example, we can use the change in the two parameters in the aforementioned two-dimensional coordinate system (ΔX, ΔY) to indicate the motion vector of a two-dimensional object. Therefore, when a user uses a conventional two-dimensional navigator (for example, a mouse) to move an object in a two-dimensional user graphical interface, he or she will still encounter some technical challenges when encountering certain complex motion requirements, but in general simple motion requirements, the user will feel that the degree of freedom of the object is roughly equivalent to that of the conventional two-dimensional navigator.

[0136] When three-dimensional graphical user interfaces become more and more common (a situation introduced by the three-dimensional graphical user interface of the present invention), the "coincidence" mentioned above that occurred in previous two-dimensional user graphical interfaces, that is, the degrees of freedom of the position space are the same as the degrees of freedom of the motion vector, will no longer exist in the three-dimensional graphical user interface. In short, a three-dimensional object will not only translate, but also rotate, and most of the time, a three-dimensional object will perform both of the above movements at the same time. From a mathematical point of view, in a three-dimensional coordinate system, a translation motion has three degrees of freedom (that is, ΔX, ΔY, and ΔZ); in a three-dimensional coordinate system, a rotation motion also has three degrees of freedom (for example, Figure 4B α, β, and γ in .

[0137] When the above two types of motion are combined, a 3D GUI must be able to provide 6 degrees of freedom to an object in order to move the object completely freely in a 3D space. Similarly, a 3D navigator that is intended to be used by the 3D GUI must also be able to provide 6 degrees of freedom to the 3D GUI in order to achieve the purpose of enabling it to move an object completely freely in a 3D space. The known technologies in the past have no way to meet these two technical requirements.

[0138] In a traditional 2D graphical user interface, a tiny object can be represented by a very small drawing entity, pixel, or voxel. When a drawing entity is as small as a pixel or voxel, the corresponding object is treated as a mathematical point or a graphical dot by the traditional graphical user interface; it is worth noting that a mathematical point has no physical extension (for example, volume) or directionality. Because traditional 2D graphical user interfaces adopted the above-mentioned limitations and regarded them as their basic design rules, functions related to rotational motion have been almost abandoned by traditional 2D graphical user interfaces. If a situation requires a traditional 2D graphical user interface to rotate an object, then, for example Figure 1B As shown, the previous 3D GUI must use a very clumsy and indirect method to perform the above rotation. Figure 1B In a conventional two-dimensional graphical user interface (105), the movement of the cursor (111) is indicated by a translational motion vector (Δx, Δy); please note that the movement of the cursor (111) does not have a rotational motion component.

[0139] When an application occasionally requires a user to provide a rotational motion vector for an object larger than a pixel (for example, the object is a cross 113), previous 2D graphical user interfaces do not have a direct way to do this; previous 2D graphical user interfaces must go through some indirect procedures to complete the above rotation work, for example, previous 2D graphical user interfaces must rely on dragging a point (114) a distance that is separated from the rotation axis point (pivot point), P(x, y); then, by using linear algebra, previous 2D graphical user interfaces can derive a rotational angle.

[0140] From the above disclosure, we can see that moving an object in a graphical user interface using any method may result in the graphical user interface deriving some unpredictable results of the rotational motion vector. For example, when we do not know the distance between the aforementioned rotation axis point P and the dragging point (114), we cannot predict the result of the aforementioned rotational motion vector. Now we can understand that using different methods to move an object will result in different combinations of translational motion vectors and rotational motion vectors.

[0141] From the above disclosed content, we can see that in a realistic three-dimensional graphical user interface such as the present invention, since most objects moving in the three-dimensional world move with translation motion vectors and rotation motion vectors at the same time, the above situation (the user cannot predict the result of the rotation motion vector) is only a small part of many situations.

[0142] If a GUI designer does not have a good knowledge of the basic laws of physics (for example, almost all 3D objects have rotational motion), it is easy for such a GUI designer to fall into the trap of the old 2D GUIs, which forgo the ability to rotate objects as small as a pixel. In the following paragraphs, we will present an example that shows how the method and process of cursor movement can profoundly affect the performance of a 3D GUI that uses rotational motion and cursor directionality (and also uses translational motion).

[0143] Section 3. Using a 3D GUI to interact with multiple 3D objects at the same time

[0144] Please refer to Figure 1B and 1C The above-described process of rotating an object (e.g., rotating the crosshair (113)) has an unfortunate feature: during the rotation of the object (113) (e.g., dragging a point (114)), the cursor (111) itself does not rotate. In next-generation electronic devices, the above-described feature does not meet the requirements of a three-dimensional graphical user interface with a high degree of engagement with the user.

[0145] Today, it is common for a user to interact with multiple objects at the same time. Figure 1A As shown, when a conventional two-dimensional graphical user interface is to interact with multiple objects at the same time, the operator of the conventional two-dimensional graphical user interface will press the mouse button (e.g., 103) and then drag a certain distance in a slantwise direction on a mouse pad or desktop (104); in this way, multiple icons (i.e., (108), (109, and (110)) will be surrounded by a rectangular block (107) drawn with dotted lines. From the geometric relationship (encirclement) between the block and the icon described above, it is shown that the several icons or objects surrounded by the block (107), i.e., (108), (109), and (110), have been already controlled by the electronic system, such as Figure 1D(112) in the figure is selected and prepared for the subsequent process. Compared with the graphical user interface invented decades ago without the function of dragging a cursor, the above-mentioned function of using a two-dimensional graphical user interface to interact with multiple two-dimensional objects at the same time represents a higher level of interaction performance with the user.

[0146] When a device engineer attempts to extend the capabilities of a two-dimensional graphical user interface to a three-dimensional graphical user interface, he or she must develop a method or process that enables the three-dimensional graphical user interface to specify a three-dimensional volume in space so that the three-dimensional graphical user interface can interact with a group of three-dimensional objects surrounded by the three-dimensional volume. In this regard, we would like such a process to be similar to the process used by the previous two-dimensional graphical user interface, that is, we would like to be able to use a reference surface such as Figure 2D (205) in the example above, dragging a special 3D navigator (i.e. Figure 2B (202) in the figure) to achieve the purpose of moving a three-dimensional cursor in a three-dimensional graphical user interface. Regarding this purpose, Figure 5A and 5B Its solution is shown.

[0147] Please note carefully, if Figure 5A As shown, in order for an operator to designate a cuboid (for example, a box (501A)) in three-dimensional space, the operator must drag the cursor (504A) along the diagonal line (AB) of the cuboid (501A). Figure 2A As shown, according to the basic design rules of the three-dimensional graphical user interface of the present invention, the above-mentioned cursor dragging process needs to specify a specific rotation plane (that is, Figure 5A Plane a 1 b 1 c 1 , the present invention will refer to this plane as the rotation plane from here on), this plane is for the cursor ( Figure 5A The (504A) in it is used to crawl on it.

[0148] When we use the navigator (202) of the present invention, the work of specifying the rotation plane is handled by the posture detection mechanism (203) (sometimes we also call it appearance detection mechanism). Known technologies, such as mice, do not have any mechanism specifically responsible for handling the above work (that is, a mechanism specifically used to tilt the posture of a cursor). Figure 5A as well as Figure 5BIt is shown that when we choose different rotation planes, this will result in different cuboids (that is, the shapes and sizes of (501A) and (501B) are different); the difference in shape and size of the two cuboids ((501A) and (501B)) can be easily seen from their diagonals (that is, AB and AC). Specifically, the rotation plane a 1 b 1 c 1 It is generated by the diagonal AB, which is used to indicate the rectangular parallelepiped (501A); the rotation plane a 2 b 2 c 2 is generated by the diagonal line AC, which is used to indicate the cuboid (501B); when we compare the two cuboids ((501A) and (501B)), we will understand that only the cuboid (501B) can completely surround the car (502); for example, the wheel (503) is surrounded by the cuboid (501B). On the other hand, by the rotation plane a 1 b 1 c 1 The resulting cuboid (ie 501A) can only enclose a portion of the racing car (501) (note that the wheel (503) is not enclosed by the cuboid (501A)).

[0149] If a GUI design engineer is not aware of the basic characteristics of the 3D GUI of the present invention, the above-mentioned capability that facilitates user interaction, namely the ability to drag a 3D navigator (202) along a specific rotation plane in a 3D space, will be inadvertently lost.

[0150] In fact, the above problem is a special problem of most navigators and graphical user interfaces on the market today. Following this commonality, we can see today that many graphical sketching programs rely on two devices or programs to complete more than one of the above tasks; for example, the task requires the combination of two separate and independent actions to complete the above task, one of which is related to a rotational knob and the other is related to a traditional two-dimensional navigator.

[0151] The basic product design rules for these types of devices or processes insist that a three-dimensional object be always treated as a mathematical point, regardless of its size (e.g., the three-dimensional object can be as small as a pixel, or as large as the entire screen, etc.).

[0152] Due to the lack of understanding of the basic physical properties of real three-dimensional objects, the known technology delegates the task of translating an object and the task of rotating an object to different navigators or programs (for example, using two separate devices or programs to perform the above tasks).

[0153] Once the process of moving a three-dimensional object (we can also regard the cursor (504) as a three-dimensional object) must be divided into many intermediate steps or distributed to many different devices to complete, the dragging process of the three-dimensional navigator of the present invention is almost eliminated; therefore, when a user uses such a known graphical user interface, the degree of interaction between the user and the graphical user interface, as well as the user's creativity, are immediately reduced. The known graphical user interface technology lacks the support of a true three-dimensional navigator. This makes the known three-dimensional graphical user interface just a "quasi-3D" graphical user interface, that is, the known graphical user interface technology does not have any effective and direct method that can move a three-dimensional object in a widely feasible way (about "quasi-3D" in Section 5).

[0154] In order to provide a 3D object with some special motion vectors that are difficult to generate with an optical mouse, the most advanced "quasi 3D" graphical user interfaces must restrict certain aspects of the movement of a 3D object to make it reachable or use other degrees of freedom it needs.

[0155] Now we can understand more clearly why even the best graphical user interfaces known in the art do not have enough degrees of freedom to describe the three-dimensional movement of an object. For example, many professional drawing programs (such as: 3DS PowerPoint, etc.) still use the known 2D mouse to move a 3D object or cursor, giving it some kind of "3D-like" GUI functionality.

[0156] In an exemplary process of using the above professional drawing program to edit the outline of a three-dimensional object, an operator can use a method with less than 6 degrees of freedom to modify the edges or facets of a three-dimensional object. When using this method, the professional drawing program will keep the other edges or facets unchanged; but this ad hoc divisional process of the degrees of freedom of an object's movement is almost unacceptable when encountering some applications that require high performance; such applications include at least animated video games, medical robots, or robot simulators; these applications do not leave time for a user to split an action into many intermediate actions.

[0157] There are many other disciplines that require an operator to use complex techniques and they encounter the same obstacles when they use an improvised method to divide the degrees of freedom of the object's movement to modify the characteristics of a 3D object, such as edges or facets, etc.

[0158] Section 4. A navigator that provides an absolute address can eliminate the accumulated errors of the relative motion detected by the navigator

[0159] In Section 1, we have briefly explained the previous 2D navigators (e.g. Figure 1A (101) in FIG. 1 can only provide relative motion vectors. Today, many navigators available on the market belong to the type of detecting relative motions.

[0160] Is such a navigator powerful enough to meet the needs of today's 3D graphical user interfaces? A user of the 3D graphical user interface of the present invention sometimes relies on manually pushing an object to complete many highly skilled tasks. For example, a digital drawing artist may need to depict a graphical object or an anatomical design with high positional accuracy. This kind of work would prefer an operator to use absolute addresses. Faced with this situation, most relative motion detectors on the market today are outdated (insufficient applicability). In order to help readers easily understand the shortcomings of relative motion detectors in this regard, the following paragraphs explain its several basic problems.

[0161] When an operator uses a conventional two-dimensional navigator ((101) in FIG. 1) to move a cursor or object on a screen, the positioning accuracy of the navigator (mouse) is often affected by unexpected errors caused by the density and shape of microscopic dust particles or surface defects on the reference surface that change with location. As a result, when a mouse is moved on the reference surface (104), the mouse ( Figure 1A The motion vector of the previous navigator derived from (101) in the figure will fluctuate continuously. Figure 3A Inside ) is composed of many small steps (for example, Figure 3A Inside as well as ), the above-mentioned inaccurate positioning problem becomes more serious.

[0162] That is, when an operator moves on any reference surface ( Figure 1A Drag a previous navigator ( Figure 1A 101) after a certain distance, this process is equivalent to requiring a graphical user interface (GUI) that receives the motion vector data of the previous navigator (mouse) Figure 1B (105)) to integrate (accumulate) the above-mentioned small steps (that is, Figure 3A Inside as well as ), and as a result of such a procedure, the errors embedded in the above small steps are also accumulated.

[0163] Due to the inherent problem of relative motion detectors (i.e., errors will be accumulated), when a relative motion navigator ( Figure 1A 101) to move objects or cursors in the past two-dimensional graphical user interface ( Figure 1B (105) in it, where it places the cursor on the screen ( Figure 1B After the (111) in the figure moves a certain distance, it will inevitably gradually lose its positioning accuracy.

[0164] Using some makeshift methods or equipment (for example, increasing the frame rate of a navigator's captured image) to improve the accuracy of the aforementioned relative motion can temporarily solve the problem; however, the aforementioned error accumulation situation has never left. The ultimate solution to this problem is to use absolute addresses, which is why some high-resolution and high-sensitivity touch display screens have become more and more popular in recent years.

[0165] Figure 3A and Figure 3B What is shown is a comparison of the performance of the two navigators, that is, Figure 3A and Figure 3B is to compare a relative motion detector (for example, Figure 1A ) and an absolute address detector (for example, Figure 2B The performance difference of the three-dimensional navigator (202) of the present invention is shown in FIG. Figure 3A As shown, when a graphical user interface uses a conventional relative motion detector (eg, moves on a reference surface (104)), the corresponding motion vector generated is can be expressed as:

[0166]

[0167] Here as well as Represents some intermediate relative motion vectors (ie the small steps mentioned above) generated by the navigator (101).

[0168] As explained before, in each of the above-mentioned intermediate relative motion vectors (that is, etc.), there is an error caused by measurement (that is, Error bx ), the error caused by this measurement is mainly caused by dust particles or defects of different density or contour shape on the reference surface (104). Therefore, when the relative motion detector (101) completes its journey and arrives at the destination B', the error sum of its final motion vector (that is, Tolerance_relative_motion) is an accumulated value (summation), which can be expressed by the following formula (2):

[0169]

[0170] As shown in formula (2), the value of Tolerance_relative_motion is obtained by taking the error of each small step (that is, taking all Error bx; x = 1 to n). It is worth noting that the series in formula (2) is a divergent series (that is, theoretically n can be a user-defined value; as the value of n is selected, the value of Tolerance_relative_motion also increases accordingly); therefore, when an operator moves on the reference surface ( Figure 1A (104)) Move the mouse shell step by step ( Figure 1A When (101) in the figure, a corresponding measurement error will be continuously added to the final value of the above Tolerance_relative_motion. bx is always a positive number. When n reaches a certain value, the above Tolerance_relative_motion will be too large to be considered a normal value. Once this happens, it means that the navigator ( Figure 1A (101)) exceeds the error tolerance (the error is too large), the corresponding cursor movement program, or even some applications related to the navigator, must be terminated (usually this is controlled by a graphical user interface), and a cursor must be reset to its last position data obtained through relative motion. This fundamental shortcoming makes it impossible for the most advanced relative motion detectors to continuously and stably provide relative motion vectors to a user so that the user can outline an object with a long line (many artistic sketches or sketches belong to this category).

[0171] In the next generation of 3D GUIs (e.g., the 3D GUI of the present invention), there are many applications that require an input device to continuously and stably provide position data; a traditional navigator can run thousands of frames per second, but the positioning accuracy may occasionally go out of control, and such a traditional navigator may not be acceptable to a professional worker. For example, when a professional digital artist uses a professional drawing art program such as MAYA or 3DS MAX to draw a cartoon character, he or she does not want the positioning accuracy of the professional drawing art program to continuously decline when he or she moves the mouse on a reference surface.

[0172] Using touch screen products (for example, WACOM, which relies on a unique stylus to poke the surface of a display panel that contains a grid made of fine transparent conductors) can temporarily alleviate the severity of the above problems, but the degrees of freedom of such touch screen products are far less than 6. Therefore, despite the limited degrees of freedom of touch screen products, the unique characteristics mentioned above make today's touch screen products more useful than other products when drawing static pictures. Since the above-mentioned problem (the degree of freedom of the touch screen is far less than 6) has not really disappeared, when a graphic artist uses a touch screen product for a while, the graphic artist may feel uncomfortable, the reason is that the degrees of freedom allowed by the touch screen are very limited.

[0173] Figure 3B What is shown is a motion detector (that is, a three-dimensional navigator disclosed in the present invention) that can measure absolute addresses from a colored or dyed reference surface (205T). Figure 3B As shown, when the three-dimensional navigator (202) disclosed in the present invention moves on a reference surface (205T) tinted by a means of tri-coloration (please note that in the present invention, we indicate a colored or tinted reference surface as (205T); the same surface, but without color or tinted, is indicated as (205) to distinguish it), using the method taught by U.S. Patent No. US10,254,855, the three-dimensional navigator (202) can convert the color index data it measures into three-dimensional positional data (3D positional data, that is, X, Y, Z). Therefore, the final motion vector of the three-dimensional navigator (202) is It can be expressed by the following formula:

[0174]

[0175] Here (X C , Y C , Z C ) and (X C’ , Y C’ , Z C’ ) represents a vector The absolute addresses of the two endpoints, C and C'. Figure 3B As shown, the measurement The error generated can be obtained by accumulating the above measurements The error generated by the absolute address of the two endpoints (that is, C and C') is expressed as Error C and Error C’ (Note that this measurement The process will only cause two errors). When a GUI is calculating the final displacement data When , its Tolerance_absolute_address value can be expressed as:

[0176] Tolerance_absolute_address=Error C +Error C' (4)

[0177] Here the reader needs to think carefully, Tolerance_absolute_address will not be affected by the intermediate steps mentioned above. Now we understand that although some known optical mice claim that they can increase the frame rate of captured images to a terrifying level of thousands of frames per second, due to the inherent problems mentioned above, the effectiveness of these known technologies is still limited.

[0178] Occasionally, when a beautiful stroke in a drawing (e.g., a cartoon) requires a user to generate a long series of relative motion vectors, previous navigators are prone to failure. The positioning error of a motion detector using absolute addresses does not deteriorate with the number of measurements (e.g., the frame rate of the captured images used to measure the address). Here we need to reiterate that the phenomenon of non-degradation of the accuracy of the measured address does not occur in devices that measure relative motion vectors; it is this basic reason that explains why we can foresee that operators of artistic drawing and the robotics industry, etc., will prefer to use the three-dimensional navigator (202) of the present invention for the performance of their professional work and connect the navigator to the three-dimensional graphical user interface of the present invention.

[0179] If we use the previously disclosed knowledge to analyze the characteristics of some navigators on the market today, we can come to the conclusion that a high-resolution and high-sensitivity touch panel can be considered as a device that provides absolute addresses in the XY plane (for example, WACOM products, etc.); on the other hand, such touch panels still suffer from a basic limitation, which is that such touch panels cannot provide 6 degrees of freedom to an object located in a three-dimensional graphical user interface. The same situation occurs in most notebooks using touch pads (for example, touch pads provided by Synaptics Corp.), that is, such computers using touch pads cannot easily provide 6 degrees of freedom to an object located in a three-dimensional graphical user interface. A game console controller (game console) that uses several buttons to specify a specific movement of an object (for example, the specific movement is a translation movement) is roughly regarded as a relative motion detector.

[0180] Finally, a depth camera is a hybrid product, that is, it can provide Z (depth) values ​​in the form of absolute addresses, but the motion vectors provided by the RGB image detector embedded in the depth camera are relative (ΔX, ΔY). Other devices, such as smartphones with 3D touch force sensitive functions, are also facing the same type of situation. That is, for example, a touch panel embedded in a 3D touch-sensitive smartphone may be able to detect an absolute address on an XY plane (that is, the touch panel) by measuring capacitance, but its touch pressure detection mechanism may use a layer of piezoelectric film, and the piezoelectric film detects a relative motion (that is, ΔZ). When we merge the above two mechanisms into one system, such a merged system may not necessarily lead to a meaningful result, which is the ability to move an object in a true 3D way.

[0181] A gyro-sensor embedded in a smartphone can add the function of detecting rotation to the smartphone; however, the gyro is also a relative motion detector after all, and for a smartphone circuit board, the gyro and the display panel are two devices with completely different properties soldered on the same circuit board. Therefore, we can see that the known navigation devices cannot provide translation and rotation motion vectors for a three-dimensional object at the same time, and many navigation devices do not provide rotation motion vectors at all.

[0182] When the 3D graphical user interface of the present invention uses the above-mentioned conventional navigator and controls the movement of a 3D object in a real-time manner, some movements of the 3D object, especially some rotational movements, need to be restricted. Figure 2B When cooperating with (202) in the present invention, the three-dimensional graphical user interface can simultaneously provide a three-dimensional object with a translation motion vector, a rotation motion vector, and an absolute address; the complete specifications of the three-dimensional navigator are disclosed in U.S. Patent No. US10,254,855.

[0183] Section 5. Basic advantages of a 3D GUI that provides 6 degrees of freedom

[0184] From the above we can understand that the performance of a GUI is deeply affected by the basic capabilities of the navigator it uses. Since the known technology lacks an effective way to solve the above problem (that is, to provide an "intrinsic" rotation motion vector for a 3D object, regardless of its size), although the most advanced GUIs today claim to have the function of providing 3D cursor movement, these GUIs are still just some "quasi-3D" GUIs.

[0185] When we use the word "quasi" to describe a GUI, we mean that the GUI can still present a static object in three dimensions (that is, the object has a three-dimensional address, depth value, and it has volume), but when a three-dimensional object needs to move with a considerable number of degrees of freedom (for example, translation and rotation), the "quasi 3D" GUI does not have a widely used method to allow its users to interact with a three-dimensional object. As a result of this basic limitation, the performance of a "quasi 3D" GUI is quite limited.

[0186] A layman seeing a graphical user interface that can provide three-dimensional addresses might incorrectly infer that any navigator that can provide an address with three independent parameters to a three-dimensional object must satisfy a sufficient condition that enables the navigator to move the object freely in three-dimensional space.

[0187] In fact, providing a 3D object with three independent parameters is only the necessary condition (that is, the minimum condition, minimal) to provide an object with three-dimensional properties. In the past, many different navigators were developed driven by the above-mentioned wrong expectations. Such navigators all attempted to move a 3D object with three parameters (for example, (Δx, Δy, Δz), which is a relative motion vector; or (X, Y, Z), which is an absolute address).

[0188] Today, we can find many navigators on the market that belong to the above types. For example, a virtual reality (VR) headset (headset, Figure 4E 405) uses (Δx, Δy, Δz) to move the scenery or objects in its display device; a 3D printer uses (Δx, Δy, Δz) to inform its printer nozzle to control the position and amount of plastic sprayed onto the model; an electromagnetic motion tracker uses (Δx, Δy, Δz) to specify the position of an object in an animated video game; a three-dimensional touch display uses (Δx, Δy, Δz) to move an object; an animated video game console provides a specific value according to the set function of the button pressed (such as A, B, X, Y), for example, the specific value is (Δx, Δy, Δz), and so on.

[0189] In the physical world, the motion of a three-dimensional object almost always has more than three degrees of freedom (a complete three-dimensional motion should have six or more degrees of freedom). In this sense, when the above devices encounter the need to manipulate the position of an object in three-dimensional space in a comprehensive manner, they are short of several degrees of freedom.

[0190] To mask this problem (i.e. the lack of freedom provided by previous navigators), some "quasi-3D" GUIs use proprietary software packages that restrict a user's freedom to one degree at a time, allowing the user to use other degrees of freedom. Obviously, this degree of freedom exchange method is a workaround.

[0191] In summary, although the movement of a real object includes elements of translation and rotation, the known technologies (e.g., virtual reality head mounted devices, depth cameras, animated electronic game consoles, electromagnetic wave trackers, etc.) do not have a practical way to distinguish the translation and rotation of an object, or to easily calculate the rotational motion vector of the object; therefore, the rotational motion vector is actually ignored by the known technologies, and the data sent to an electronic system by the known technologies (i.e., the previous navigators) are always regarded as translational motion vectors. Now we have the second basic reason why the present invention calls this type of graphical user interface a "quasi-3D" graphical user interface.

[0192] The adjective "quasi" here means that the appearance of objects in a "quasi-3D" GUI may be very similar to the appearance of objects in a true 3D GUI, but once it comes to user interaction, the above-mentioned "quasi-3D" GUI lacks effective methods to generate complex movements. Therefore, the above-mentioned "quasi-3D" GUI can only generate some "rudimentary" degrees of freedom to allow a user to move a 3D object.

[0193] This problem of lack of sufficient usable degrees of freedom may not be very obvious when the specified action of an object does not have much "dynamics" (that is, in this case, some degrees of freedom remain unused), but when an animated video game or a cartoon character is very active (that is, the actions of the animated video game or the cartoon character are quite vivid and varied), the problem of lack of sufficient usable degrees of freedom becomes serious.

[0194] In the past, some graphical user interfaces have assigned the task of providing translational motion vectors and rotational motion vectors for an object to different devices or programs; in this case, a graphical user interface that works with two different devices or programs has no way to provide a widely available, clear, and direct way to integrate the two different motions into one.

[0195] The result is that although the rendering of a static electronic graphic artwork can be very beautiful (for example, the shading of today's most advanced cartoons can be quite realistic), the dynamics of many objects' movements are still quite stiff and mechanical. This is due to the graphics programs used in these most advanced cartoons (for example, 3DS or This is because the graphical user interface of the editor (such as ) does not provide enough degrees of freedom (that is, 6 or more) for the objects being edited.

[0196] In the above-mentioned drawing programs, a user can select special icons or instructions that are used to specify some specific rotational or translational movement of an object at a time. For example, if an operator presses (that is, selects) an icon that is designated to provide rotational movement, he or she can rotate a three-dimensional object while rotating the three-dimensional object without allowing the three-dimensional object to perform other movements (e.g., translation) at the same time.

[0197] When a cartoon character is drawn using the above method, through such a process of limiting the degrees of freedom, the cartoon character must limit one or more degrees of freedom at a time during its drawing process (this process can be done by selecting some specific icons) so that it can use the required degrees of freedom. By limiting a degree of freedom here and another degree of freedom there in drawing a drawing, a drawing artist can still complete the drawing process in a way that each single stroke has less than 6 degrees of freedom, but after a long drawing process, the behavior or style of the entire cartoon character may become quite mechanical and unrealistic, and the root cause mentioned above is not easy for an art layman to understand.

[0198] When a person is just drawing one or two strokes on a screen, the above symptoms may not be obvious. However, when a cartoon character has been built up by many strokes (each of which suffers from the problem of insufficient freedom to a greater or lesser extent), the above symptoms will become a noticeable impact on the final quality of a drawing artwork. So, we understand that the creativity of an electronic drawing artist will be limited by the limited freedom of the previous graphical user interface (the limited freedom of the previous graphical user interface is affected by the performance of the navigator).

[0199] Here we have an example: If one compares a graph drawn with a state-of-the-art drawing program (e.g. 3DS He or she can easily recognize the facial structure of a princess drawn by a drawing program (such as the ratio of cheek width to skull height, etc.) from the facial structure of a princess drawn by a previous cartoon artist. From the perspective of aesthetics, the facial structure of a princess drawn by a drawing program is difficult to meet some golden rules, which are used to describe beautiful appearance (for example, the ratio of cheek width to skull height, etc.); We often encounter a situation: when a cartoon character drawn or edited with a polygonal network (such as Fig. 10A)When a graphic artist wants to implement the above so-called golden rule to a cartoon character he or she has in mind, the polygon network itself becomes the highest technical hurdle that the graphic artist must overcome.

[0200] So while today’s digital cartoon characters can be interesting because of their quirky outlines, or their renderings can be incredibly realistic beyond the norm, the power of today’s most advanced graphics programs cannot be compared to the cartoons of the past when it comes to examining the aesthetics upon which the core value of most works of art rests.

[0201] In order to infuse a work of art with the sensibility of aesthetics, when we move the three-dimensional navigator ( Figure 2B 202 in the book, we need to coordinate the nerve cells of an artist's fingers, waist, elbow, arm, etc. We all have the following experience: every time humans invent a new technology (for example, digital painting), the new technology may bring some new conveniences (for example, amazing rendering (shading) effects, etc.); however, it is also a natural result that the new technology may also bring some side effects (for example, the movements of digitally painted characters are more mechanical than those of hand-painted cartoons in the past), so these side effects have to wait for newer technologies to overcome them.

[0202] An important purpose of the present invention is to bring more degrees of freedom to a graphical user interface so as to overcome the side effects mentioned above (the side effect is that when a graphical user interface needs to depict the complete three-dimensional movement of a digital drawing character, its degrees of freedom are not sufficient to meet this need), so as to enable the tremendous power of two neural networks to cooperate with each other, one of which refers to the human body and the other refers to the computer; by coordinating the tremendous power of the two neural networks, new forms of computing technology can be fostered, thereby enabling the entire electronics industry to continue to grow and stride into the era of artificial intelligence (AI).

[0203] From the above evaluation, we can understand that the present invention is a valuable asset to the electronics industry, and we can expect that the electronics industry will be strongly supported by the AI ​​(artificial intelligence) industry in the future. In many AI-related functions, the present invention's 3D graphical user interface plays a key role. This is because the present invention's 3D graphical user interface seamlessly bridges two technical fields, the first of which interprets each action as a linear or translational motion (for example, the previous 2D graphical user interface, Figure 1B (105)); wherein the second technical field is to interpret almost every action as nonlinear motion or rotational motion (for example, the three-dimensional graphical user interface of the present invention, Figure 2A (207)).

[0204] If one uses computer technology terms to describe the characteristics of the two technical fields mentioned above, the first technical field mentioned above is related to a computer operating simply according to Boolean logic, while the second technical field mentioned above is more related to the functioning of neural networks (that is, the decisions made by a computer are not entirely based on predetermined logical relationships, but are made based on the observation of some recurrent phenomena).

[0205] As explained above, traditional cartoon artists do not suffer from the inherent problems encountered by digital cartoon art; a traditional cartoon artist can use his hands to draw a character with a full set of (6) degrees of freedom. Therefore, when we transfer a work from a piece of paper to a computer, the navigation method or device we prefer to use will be the three-dimensional graphical user interface (3D GUI) of the present invention. Figure 2A When certain properties of an object (e.g., some complex motion, heat conduction, flow rate, etc.) are expressed by many degrees of freedom (including some degrees of freedom that are non-kinematic), the interaction between a group of such objects can be very complex. In this case, the matrix operation shown in formula (8) becomes an effective method for describing the properties of an object with many (e.g., >= 3) degrees of freedom.

[0206] The 3D graphical user interface of the present invention can not only bring convenience to the digital cartoon industry, but also bring convenience to other industries. In the future, anyone who relies on finesse to perform work can rely on the 3D graphical user interface of the present invention to complete their mission (for example, surgery simulation trainer, animation electronic game console, 3D mechanical drawing, etc.). Finally, we can foresee that most computer operating systems in the future will adopt the 3D graphical user interface of the present invention. At that time, even a home user can benefit from the contribution of the 3D graphical user interface of the present invention to some daily work.

[0207] For example, the 3D graphical user interface of the present invention can provide some 3D sketching features for a presentation program, or the 3D graphical user interface of the present invention can provide a special 3D cursor so that a user can more easily select a certain 3D icon, or the 3D graphical user interface of the present invention can support an application environment that can combine multiple 3D windows (for example, Fig. 13B In summary, the 3D graphical user interface (207) of the present invention represents the beginning of a true 3D operating system that is more interactive, smarter, and easier to engage with users than the most advanced operating system technology of the time.

[0208] At this point we can clearly understand that when more members of the electronics industry enter the field of three-dimensional drawing, a new three-dimensional graphical user interface must be developed, which can provide an object with sufficient degrees of freedom. To achieve this goal, a graphical user interface designer must clearly answer a question to himself or herself to determine whether he or she really understands the core value of the three-dimensional graphical user interface (207) of the present invention.

[0209] The answer to this problem is that the 3D GUI (207) of the present invention can provide 6 degrees of freedom to some motion vectors at the same time. What is generally overlooked is that a motion vector with 6 degrees of freedom also requires the coordination of a person's fingers or hands to move it freely. In order to cover up this problem, some manufacturers have worked hard to develop some gimmick devices in the past. For example, Microsoft and Apple A depth camera (e.g., Kinect) is developed to facilitate the measurement of the distance of a tracked object (e.g., the distance between an infrared spot on an observer's hand and the depth camera (401); the infrared spot is Figure 4A as shown by point F in the figure).

[0210] Measuring the relative distances of the specks in the non-visible wavelength range is an algebraic method that does not lead to any valid measurement of the rotational displacement of the specks. Figure 4A This is a depth camera (401) of this type. Some manufacturers of another type (e.g., Facebook) Samsung etc.) provide a virtual reality head-mounted device to the observer ( Figure 4E (405)).

[0211] This type of virtual reality head mounted device (405) uses special detectors (i.e., accelerometers; global positioning system GPS, or methods with equivalent functions to the above detectors) to detect the relative motion vector of the head mounted device body (i.e., Figure 4E Therefore, during operation, a relative displacement or movement (i.e. +ΔX, -ΔX, +ΔY, -ΔY, +ΔZ, -ΔZ) can be generated, which can be used by the virtual reality head mounted device (405) to manipulate some scenes or objects presented to the observer.

[0212] Please note that the information provided by the depth camera (401) or the virtual reality head mounted device (405) cannot construct a complete set of data that indicates the position and / or movement of a three-dimensional object using six degrees of freedom. In order to provide a complete set of data to indicate the position and / or movement of a three-dimensional object, a high-quality three-dimensional graphical user interface (such as the three-dimensional graphical user interface of the present invention) must provide the following data:

[0213] (1) Three-dimensional position data, preferably provided in the form of absolute addresses (e.g., X, Y, Z);

[0214] (2) Providing three-dimensional translation motion vectors and rotation motion vectors simultaneously, for example, Figure 4B (ΔX, ΔY, ΔZ) and (α, β, γ); in some known technologies, for example, in the aviation industry, the above-mentioned rotational motions are called pitch, yaw, and roll, that is, pitching or leaning back, turning left and right, and rolling movements).

[0215] (3) Select whether to use or not use the function that provides spin motion data ( Figure 2A Θ' in this function is usually related to the higher order terms (HOT) of the motion of an object. Regarding the measurement of the higher order terms of the motion of an object, US Patent No. 9,733,727 discloses. As disclosed in US Patent No. 9,733,727, (Col 22, line 1 through 7): "human hands, particularly fingers, are capable of many gestures. When a human hand moves, the present device will sense a motion that may not necessarily be solely translational. There are many factors capable of generating the second or even the third terms on the right-hand side of Eqs. (5) and (6)". There are many factors capable of generating the second or even the third terms on the right-hand side of Eqs. (5) and (6).

[0216]

[0217] Here (Δx, Δy) represents the displacement of a tracked object, which is a function of time (that is, from t 0 The time elapsed from t to t).

[0218] As further disclosed in U.S. Pat. No. 9,733,727, (Col 22, lines 7 through 16):

[0219] "Together, these terms provide the contributions of delicate motionvectors of the hand gestures such as jittering, waving, rotation, and stroking, that the conventional art does not have the comprehensive knowledge to utilize. Being able to detect and manipulate the respective signals (ie, variation in the pixel values) in the non-linear regime will make the nextgeneration object navigation described herein and the operating systems, and the electronic devices that use it, much more interactive with their operators."

[0220] The reader should carefully consider that the third and fourth terms on the right side of the above formulas (5) and (6) indicate the "higher order terms" (HOTs) of the motion of an object that can be moved by the 3D graphical user interface of the present invention. The first term on the right side of formulas (5) and (6) indicates linear motion, which is actually the translation motion vector.

[0221] As for the second term on the right side of formulas (5) and (6), it indicates the spin motion vector, which is actually the rotational motion of an object (that is, Figure 2A A rotational motion can be generated by the relative distance between two objects, which is similar to the rotation of the earth around the sun; or a rotational motion can be generated by the course or movement of an object itself, for example, the earth rotates around an axis; the second case described above is called a spinning movement.

[0222] In order to extract HOT data stably, we would like to use an accurate method to measure the position (that is, the absolute address); to achieve this goal, the navigator (202) disclosed in the present invention captures images from a colored or tinted reference surface (tinted reference surface 205T).

[0223] In U.S. Pat. No. 9,733,727, it is explained that the direction of the rotational motion (i.e., the second term on the right side of equations (5) and (6)) is perpendicular to the direction of the translational motion vector (i.e., orthogonal), that is, each single HOT term in the above equations (5) and (6) is orthogonal to its immediate predecessor. Depending on the sensitivity and resolution of the navigator (202) and the degree of coloring of the reference surface, the entire HOT can actually be extended, so that the navigator (202) can have more than 6 degrees of freedom. For the sake of simplicity, the subsequent discussion of the present invention focuses on the motion with 6 degrees of freedom.

[0224] (4) A neural network on a human body, which controls an input device (e.g., the three-dimensional navigator 202 of the present invention) with a high degree of transparency, and the control function is based on the graphical rendering effect of the three-dimensional graphical user interface of the present invention.

[0225] Previous graphical user interfaces did not incorporate the above understanding into their design principles, let alone have its advantages. Regarding the above point (4), we will discuss its value in Section 9. This section currently focuses on the first three points. Figure 4A As shown, when an operator rotates his or her hand with a small angle σ (e.g., σ < 3°), today's most advanced depth cameras can only record the depth value of the tracking point F, and since this point F has a rotation angle of a small angle σ, the depth value D is susceptible to certain error values ​​(e.g., the depth value becomes D±d).

[0226] This situation is similar to that disclosed in US 9,733,727. As previously disclosed in the present invention, known navigators treat rotational motion as a source of noise. Since this situation is a historical problem in the pattern recognition industry, such a symptom that occurred in previous navigators may also occur in other image detectors, for example, the image detector is a depth camera used in the above-mentioned ( Figure 4A 401 in the above).

[0227] To be clear, the “error” mentioned above is not a real error; it is a rotational motion vector superimposed on the translational motion vector D. The image detector ( Figure 4A There is no effective way to directly measure the above rotation angle σ in (411) and (412); the rotation displacement (for example, σ) is treated as noise by the image detector of the aforementioned depth camera. The 3D navigator used in the 3D graphical user interface of the present invention fully incorporates the basic design of US 9,733,727, so the operator of the present invention can stably provide a rotation angle to the object during the process of moving the object.

[0228] If the depth camera (401) wants to solve the above problem by itself (that is, measure the rotational motion of a single infrared spot F), it is obvious that the calculation process involved will be very complicated (that is, derive the rotational motion vector of the aforementioned infrared spot F). What's worse, the performance of different image detectors is not the same; to this day, the depth camera industry has no effective way to deal with this problem. Finally, a development engineer will find that the algorithm used by a depth camera (401) still uses a simple algebraic method to calculate the rotation angle, which is no different from the method used by previous two-dimensional graphical user interfaces to rotate their objects (for example, rotating Figure 1B The cross in (113).

[0229] Figure 4B Shown is a touch panel (402) that interacts with a finger at point G to generate a touching event. According to the teachings of known navigation devices, the relative movement between point G and a predefined pivot point can be used to determine the rotational movement of an object (408).

[0230] It is worth noting that such methods (such as Figure 4B The method shown in the figure is neither intuitive nor complete. For example, in order to rotate the cylinder (408) by an angle, different touch panel manufacturers require the user to use different gestures to move his or her fingers or hands. From a basic principle, such as Figure 4BAs shown, the root cause of the above problem is related to the degrees of freedom of a graphical user interface; a touch panel (402) has two degrees of freedom, namely the X and Y axes; but the rotational movement of a three-dimensional object (408) has three degrees of freedom, that is, Figure 4B α, β, and γ in it.

[0231] Therefore, if a person attempts to rotate a three-dimensional object (408) using a two-dimensional touch panel such as described above (i.e. (402)), several degrees of freedom are lacking. When the situation comes to using a two-dimensional touch panel to provide positioning data for an object located in a two-dimensional graphical user interface, the above-mentioned problem is temporarily alleviated because the degrees of freedom of the above-mentioned two-dimensional touch panel match the degrees of freedom of the above-mentioned two-dimensional graphical user interface. We have explained that what happens to a depth camera is similar to what happens to a known navigator (i.e., the rotational motion vector of a point-like object cannot be easily measured). When a person waves his or her hand (406) in front of a depth camera (401), spots with many wavelengths falling in the invisible wavelength range (e.g., infrared IR) are projected onto his or her hand from the light source (410).

[0232] These spots are usually formed by infrared lasers; when the laser is about to leave the depth camera mentioned above and enter the surrounding environment, the laser must pass through several microscopic slits, and the intensity of these spots (for example, the intensity of the first-order and second-order interference fringes, etc.) will change with the relative distance between the hand (406) and the depth camera (401). Using the principle of slit interference, one can understand that the intensity of the above-mentioned spots will change with the depth value. Therefore, when an operator waves his or her hand, the depth camera (401) can recognize (using an infrared image detector (411)) some changes in the depth information of the spots projected onto the operator's hand (that is, Figure 4A D) It is worth noting that the spots themselves do not carry any information about their rotation.

[0233] If one wants to calculate the rotation angle of an object (e.g., hand (406)), he or she must compare the relative displacement of some objects in an image (e.g., using an RGB image detector (412) to capture some images of the operator's hand, etc.). The above methods usually require linear algebra, or some methods with equivalent effects to linear algebra. As can be imagined, the above-described process of calculating the rotation angle of an object is very cumbersome and time-consuming, and the calculated results are often easily affected by many sources of noise.

[0234] In U.S. Pat. No. 9,733,727, the rotational motion vector of a navigator body is calculated by adding a pseudo circle to an image captured from an image of a reference surface; this method is far more accurate than the known technology. In U.S. Pat. No. 10,254,855, the tilt of an object in a three-dimensional graphical user interface is controlled by adjusting the shading conditions of a reference surface (e.g., changing the illumination intensity of the reference surface (205T)). Therefore, as disclosed in U.S. Pat. No. 10,254,855, the three-dimensional graphical user interface of the present invention can simultaneously provide a translational motion vector, a rotational motion vector, and an absolute address to an object; when the above three parameters are used together, the three parameters provide a three-dimensional object with a complete set of 6-degree-of-freedom motions.

[0235] Section 6. Operations provided by the three-dimensional graphical user interface of the present invention

[0236] Figure 2D Depicted is a three-dimensional navigator (202) that can work closely and beneficially with the three-dimensional graphical user interface (207) of the present invention. Figure 2D As shown, when an operator swings his or her finger on the posture detection mechanism (203), the posture of a three-dimensional object (e.g., flying carpet 201) located in the three-dimensional graphical user interface (207) of the present invention can be directly adjusted by the operator (e.g., the change of the posture can be determined by the normal vector from become to indicate).

[0237] To further illustrate, when an operator swings his or her finger on the gesture detection mechanism (203), the entire mouse body (202) moves on a colored or dyed reference surface (205T), and the three-dimensional navigator (202) is able to provide absolute addresses (x, y, z), translation motion vectors (T), and tilting motion vectors, which can be regarded as a first type of "rotational" motion vector (e.g., a change in the normal vector).

[0238] In addition to the first type of rotational motion vector, the three-dimensional navigator (202) of the present invention can also provide a spin motion vector; this spin motion vector can be regarded as the second type of rotational motion vector. From a mathematical point of view, adjusting the tilt posture of a three-dimensional object is the easiest type of rotational motion to perform for an object, because the adjustment only requires adjusting the normal vector of the three-dimensional object.

[0239] According to formula (12), changing the normal vector of a 3D object (i.e. tilting) requires three parameters n: 1 , n 2 , or n 3 If an operator wants to make a three-dimensional object spin, the three-dimensional navigator (202) of the present invention must provide a second rotation motion vector. According to formula (12), a parameter ω is required to make a three-dimensional object spin.

[0240] like Figure 2A As shown, a process of spinning a three-dimensional object (e.g., 201) by an angle θ' in the three-dimensional graphical user interface (207) of the present invention requires rotating the mouse body (202) by an angle ω on a reference surface (205). However, the process of adjusting the tilt of a three-dimensional object involves adjusting the intersection of the plane abc with the three axes X, Y, and Z, that is, the values ​​of a, b, and c; the present invention hereafter refers to the plane abc as the plane of rotation.

[0241] It is noteworthy that when the object (201) is spinning, its rotation axis (i.e., the unit normal vector n) does not change. This means that the two rotational motions described above are orthogonal to each other. In computer vision technology, the above two processes (i.e., adjusting a, b, and c; spinning an object at an angle θ) are both classified as rotation procedures; the above parameters can be combined to form the parameters R of the matrix R in formula (8) 11 ~R 33; We will discuss R in the following paragraphs 11 ~R 33 parameter.

[0242] We can also use different methods to explain the above situation. Some people divide the movement of an object into linear (translation) and nonlinear (rotation) movement. In US 9,733,727, we further disclose a method for extracting a higher-order term of a movement from a navigator (that is, HOT); when such a function is implemented on the three-dimensional graphical user interface (207) of the present invention, the total degree of freedom of an object can exceed 6; in this case, it is not appropriate to call the graphical user interface (207) of the present invention a complete three-dimensional (exactly 3D) or a graphical user interface with 6 degrees of freedom (6DOF). More precisely, the device of the present invention is a truly high-quality graphical user interface (highquality GUI), which can describe some complex movements, and the degrees of freedom of these complex movements can exceed 6.

[0243] In order to meet the challenge of using linear and nonlinear motion vectors to describe the motion of a 3D object, the 3D GUI of the present invention provides the following functions:

[0244] 1. An application interface (API) that simultaneously provides linear (i.e., translational) motion vectors and nonlinear (e.g., rotational) motion vectors for a 3D object;

[0245] 2. A navigator, which is a tool that can directly provide a 3D object with a 3D absolute address (that is, Figure 2A In this way, some unnecessary 2D to 3D matrix transformation process can be avoided or reduced;

[0246] 3. A complete and clear method for editing or modifying two modes of motion vectors (i.e., translation and rotation motion vectors); the method is based on the input data of the three-dimensional navigator (202) described above, so that a user can arrange or modify a series of movements of the above-mentioned objects in an intuitive and very effective manner;

[0247] 4. An intelligent method for processing multiple three-dimensional objects according to the function of an intelligent neural network (for example, a series of matrix operations, when executing the matrix operations, the values ​​of the elements of the matrices can be manipulated by the three-dimensional navigator of the present invention); the intelligent method for processing three-dimensional objects is to give the function of the above-mentioned intelligent neural network to a three-dimensional graphical user interface module in the form of a software suite, and the intelligent method for processing three-dimensional objects can make the interaction between the three-dimensional graphical user interface and its user better than the known technology, and at the same time, the intelligent method for processing three-dimensional objects can effectively reduce the computing load of its central processing unit (CPU) or graphics processing unit (GPU).

[0248] Fig. 6A The illustrated diagram is a typical graphical user interface, which is software with a layered architecture, and the graphical user interface software is operated on hardware (620). The hardware (620) is further illustrated in Fig.14 .like Fig. 6A As shown, a graphical user interface is a plurality of layers of software that exist between an input device (601) and a kernel (605) of an operating system (e.g., Windows, Linux, OS, Android); it should be noted here that Microsoft Corp. calls the entire operating system including the kernel (605) and the graphical user interface (207) WINDOWS.

[0249] According to the general definition of a graphical user interface, a window (note that lowercase English words are used here to help readers distinguish it from Microsoft's WINDOWS) refers to an area on the screen (for example, (207)) that is specifically allocated for a specific application; for example, Fig. 13B (1303), (1304), and (1306)); a window manager (window manager, (604)) is a system software that is used to control the placement and appearance of windows in a windowing system in a graphical user interface (e.g., (207)). Typical window managers include stacking, tiling, dynamic, or composite.

[0250] For more detailed information about the characteristics of graphical user interfaces, readers can search for "Graphical User Interface" on Wikipedia to obtain relevant information. It is worth noting that although the known technology tends to implement the functions of the above-mentioned layered architecture software in software (for example, Fig. 6A (602), (603, and (604) in FIG. 1 ), this phenomenon does not exclude the possibility that the next generation of 3D GUI (207) will incorporate some of the above software layers (e.g., support vector machine (616); neural network (610), etc.) Figure 6B The possibility of implementing the internal processing module shown in the figure on hardware (e.g., special function IC, ASIC). Now please refer specifically to Fig.14 , hardware (620; also Fig. 6A 620) is a computer, a display system, an electromechanical system, or a computing device according to the general purpose of the present invention. The computing device generally includes a central processing unit (CPU) (1402); a main memory (1404); a memory interface (Memory I / O) (1410); a bridge (Bridge) (1412); an input / output device (1406A) (1406B); and a cache memory (1414) that communicates with the central processing unit (CPU) (1402). The central processing unit (CPU) (1402) can be any kind of logic circuit; the central processing unit (CPU) (1402) can respond to and process some instructions received from the main memory (1404), and the central processing unit (CPU) 1402 can also read or write data to the main memory (1404). The main memory (1404) can include one or more memory chips that can store data, and the central processing unit (CPU) (1402) can access data from any location in the main memory (1404).

[0251] The graphical user interface of the present invention generally uses input / output devices (1406) as its display device. For example, the input / output device (1406) is an electronic display screen. Fig. 6A The input device (601) is usually equivalent to another Fig.14 The input / output devices, for example, (1406A) and (1406B) are devices that can interact with the main memory (1402). Figure 2AWhat is shown is how the appearance of an object in a three-dimensional graphical user interface (207) of the present invention is affected by the two rotational movements described above. It is worth noting that the object (201) in this graphical user interface has been deliberately depicted as a cartoon object, that is, a flying carpet; by depicting it in this way, the observer receives a message inferred from the environment in which he is located, that is, the flying carpet (201) can fly, and a flying movement represents that the object (201) being focused is located in a three-dimensional space.

[0252] That means that the gesture of a flying object is not just a special appearance that gives people a sense of appearance (for example, this special appearance is a two-dimensional icon (2D icon); a 2.5D icon (2.5D icon); or a three-dimensional icon (3D icon); more precisely, it is related to the movement in three-dimensional space. As shown in Figure 2, the object (201), that is, the flying carpet mentioned here, not only has an absolute address (x, y, z) in a three-dimensional coordinate system (expressed by X, Y, and Z axes), but also has a special "tilting gesture". The tilting gesture can be represented by a special plane that intersects the X, Y, and Z axes at points a, b, and c respectively (as we specified before, we call this plane plane abc, or plane of rotation abc). Formula (7) expresses Figure 2A The mathematical formula for the rotation plane abc.

[0253]

[0254] According to the basic design rules of the 3D graphical user interface of the present invention, a motion vector is provided for a 3D object (for example, a flying carpet (201)), involving the following scenarios:

[0255] (1) When an operator (such as Figure 2D ) When the three-dimensional navigator (202) of the present invention moves a motion vector S on the reference surface (205) (note that, in fact, the reference surface (205) is equivalent to the rotation plane abc), the flying carpet (201) will move (like crawling) on ​​the rotation plane abc by a translation vector corresponding to the motion vector S;

[0256] (2) Providing a rotational motion to the 3D object involves a different scenario:

[0257] (ii) From the perspective of analytic geometry, this situation (that is, changing the inclination of the rotation plane abc) is equivalent to changing the normal vector of the rotation plane abc (for example, from become ). If an observer places his viewpoint on the flying carpet (201) and examines it carefully, the above normal vector changes from become The changes can be verified from the changes in the appearance of the flying carpet.

[0258] (iii) When a point (e.g., a pixel or voxel) is moved by a displacement that is a combination of a translation and a rotation, the parameters of which can be represented by R and T in equation (8), the point reaches a new position, which we denote by P'.

[0259]

[0260] As explained in formula (8), the rotation displacement R is not a single-valued object, but is represented by a (3×3) matrix, that is,

[0261]

[0262] Similarly, the translation displacement data T is not a single parameter data, it is a matrix containing three parameters, which is expressed as

[0263]

[0264] Using formula (8), the above two displacements (that is, the translation of an object and the movement of the posture; they can also be called motion vectors) can be combined into one displacement; that is, the final motion vector of an object is the result of the addition (superposition) of the translation motion vector and the rotation motion vector. It is worth noting that such a combined motion vector can be applied to an object as small as the size of a pixel (or voxel, if we want to emphasize its volumetric nature); this is one of the most basic functions of the three-dimensional graphical user interface (207) of the present invention, and one of the basic differences between the three-dimensional graphical user interface (207) of the present invention and the previous "quasi-three-dimensional" (quasi 3D) graphical user interface.

[0265] The above rotation matrix R can be converted into different formats. As shown in formula (11), the rotation matrix R can be regarded as the combination of three rotation matrices (that is, three consecutive matrices multiplied together), where each matrix is ​​used to specify the rotational movement around an axis (that is, the X axis, the Y axis, and the Z axis). Therefore, as shown in formula (11), the rotational displacement of a point in a three-dimensional coordinate system can be represented by three rotation angles, that is, φ, θ, as expressed in formula (11), φ, θ are used to replace R in formula (8) 11 ~R 33 , we can rewrite the rotation matrix R; and vice versa.

[0266]

[0267] As an alternative, the rotation matrix R can be described as in formula (12), where the parameter n 1 , n 2 , and n 3 Represents an axis of rotation The unit vector of the (pivot axis), ω represents the rotation around this axis. The rotation (e.g., spinning) angle of .

[0268]

[0269] In actual operation, according to different applications, the three-dimensional graphical user interface of the present invention can select formula (11) or (12), or other forms of rotation matrix R (for example, quaternion, q uaternion, etc.) for use, the results will be the same. For example, formula (11) requires three angle data ( φ, θ) to describe a rotation action; as for formula (8), it only needs an angle data to describe a rotation action

[0270] Depending on different applications, different forms of rotation matrices R will require different parameters to indicate a rotational motion. Therefore, the three-dimensional graphical user interface (207) of the present invention provides an application interface (API) to allow the three-dimensional graphical user interface (207) of the present invention to provide the parameters required for related calculations according to different situations. Figure 6B What is shown is such an API (609). It is worth noting that the API (609) receives data from an input device (601), and this input device (601) is not necessarily the three-dimensional navigator (202) of the present invention. In actual operation, since other navigators can also be accepted by the API (609), these different types of navigators can use the API (609) of the present invention as a portal, and then the data they input will be processed by different internal processing modules (e.g., internal process modules, 615). Such internal processing modules at least include modules designed specifically for cursor movement (606), stereoscopic painting (607), and robots (608).

[0271] For example, in the animation electronic game industry, the most commonly used rotation matrix R format is quaternion. In that case, the 3D graphical user interface (207) of the present invention will send a signal from the above API (609) to the 3D navigator (202), requesting it to send the parameters representing the quaternion Q, that is, p, q, r, and s; its formula (13) is:

[0272] Q=p+qi+rj+sk (13)

[0273] Here, p, q, r, and s are real numbers; i, j, and k are the basic units of quaternions.

[0274] In the known art, moving a cursor or object using translational motion vectors and rotational motion vectors requires an operator to use different navigators (e.g., using a rotary knob and a translational motion navigator, etc.) to complete the task. When the movement of a cursor or object is composed of many intermediate steps of navigators, the entire process of moving the cursor or object becomes very complicated. When a computer, electronic system, or electromechanical system uses the three-dimensional graphical user interface (207) of the present invention, a movement combining translational motion and rotational motion as shown in formula (8) can be provided by a gesture of a single hand (that is, a movement of a hand, which movement of the hand at least includes the translational movement of the palm and the swinging movement of the fingers). The above-mentioned special ability enables a user to interact directly with the three-dimensional graphical user interface of the present invention.

[0275] Now let's take a closer look at previous 2D GUIs (e.g. Figure 1B 105) in the basic principles (rudiments). Figure 1B As shown, when a cursor is presented in a specific shape (e.g., arrow (111)) on a conventional display device, the cursor is aimed at a point P (the point P is on the cross (113)) to represent where the cursor is focused (i.e., x, y).

[0276] For both the observer and the computer (112), all the cursor cares about is the point P; it is noteworthy that the point P has no further geometric or directional meaning. For example, rotating the arrow cursor (111) 45 degrees around the point P will not have any effect on the point P (a mathematical point / dot has no rotational meaning). Due to the lack of rotational properties, a single rotation of the point P will not rotate the cross object (113) at all. Since the invention of the computer mouse, the above method of specifying addresses or movements in a graphical user interface (that is, only translational motion vectors, no rotational motion vectors) has been widely accepted in the electronics industry. Take the cross (113) as an example. Since the geometric properties of the cross (113) have degenerated into a mathematical point, previous navigators (e.g., (101)) have no way to rotate the cross (113) simply by using a translational motion vector to interact with the point P.

[0277] When a user attempts to rotate the cross (113), he or she must specify another point on the cross (e.g., point (114)); by dragging the point (114) with a translation motion vector, the conventional graphical user interface knows how much the cross (113) needs to be rotated.

[0278] Therefore, in order to perform the work of rotating the cross (113), the known technology usually has to rely on linear algebra to derive the new position of an object; this explains why the previous method of rotating objects in the graphical user interface is not a direct method. Affected by the inherent shortcomings mentioned above, today's consumers will encounter some troubles when purchasing navigators from the mass-market market. Regardless of whether the previous navigator mentioned above is a narrow wooden board, a branch, a glove, a stylus, a pair of glasses, or a rotating knob, etc., these previous navigators face the same situation: the graphical user interface connected to these previous navigators still treats the object as one or a group of mathematical points, and therefore, the method used to manipulate the movement of the object is still not a direct method. In short, as explained above in the present invention, due to the limitations of mathematical definitions, a point object itself will not have any characteristics related to rotation; the entire graphical user interface has been limited by the overly simplified product design guidelines mentioned above for decades.

[0279] Section 7. The three-dimensional graphical user interface of the present invention embedded with robot kinematics functions

[0280] As the use of robots becomes more common, the methods used by robots to calculate position and motion (e.g., algorithms or software) have become quite outdated. The reason is that there is no effective way to control the position or motion of a robot in real time.

[0281] For some applications that require in-situ monitoring / modifying of robotic programs, the three-dimensional graphical user interface (207) of the present invention becomes a time-saving and popular device. Fig. 7A Shown is a demonstration robot, for example, a robotic arm (a six-jointed A robot; the present invention will hereinafter refer to it as a robot (700)), the exemplary robot being able to benefit from the three-dimensional graphical user interface (207) of the present invention.

[0282] like Fig. 7A As shown, some joints or elbows of the robot (700) can be rotated by an angle (ie, θ 1 ,θ 2 ,θ 3 ,θ 4 ,θ 5 , and θ 6 ). When these six joints are Fig. 7A Connecting the shown method, the related matrix operation of each joint can be expressed as:

[0283]

[0284] Here C represents the cosine function, S represents the sine function, and L elbow1 is the length of the elbow connecting joint 1 and joint 2, L elbow2 is the length of the elbow connecting joint 3 and joint 4,

[0285]

[0286] When i = n, we get a T matrix, that is, T 0 n , the T 0 n The matrix provides address and rotation data to P end , that is, the robot (700) relative to the base coordinate system ( Fig. 7A It is worth noting that the matrix T in formula (15) 0i Parameter R 11 ~R 33 , X, Y, and Z can be directly applied to formula (8); this means that the three-dimensional graphical user interface of the present invention can directly control the movement of the robot (700). Expressed in another way, the above matrix T 0 i Parameter R 11 ~R 33 , X, Y, and Z can be converted into other forms; two of which are shown in formulas (11) and (12) respectively. The reader should note that when i is less than n, the above T 0 i The matrix represents the position and rotation vector of the internal joint, that is, 0 A 1 , 1 A 2 , 2 A 3 , 3 A 4 , 4 A 5 We should further pay special attention to that when we use the three-dimensional navigator (202) of the present invention, by considering the above-mentioned T matrix according to the following information, the three-dimensional graphical user interface (207) of the present invention can instill some physical meanings into the parameters of the above-mentioned T matrix.

[0287]

[0288] n represents the normal vector of the hand; if we use a parallel-jaw hand, n will be orthogonal to the robot's fingers; Fig.12 The direction of n is shown.

[0289] s represents the sliding vector of the hand. The direction of the sliding vector is toward the gripper. The so-called gripper is a simplified finger. s (direction / value) represents the opening and closing actions of the gripper. Fig.12 Displayed direction.

[0290] a represents the vector of the hand approaching. The direction of the hand approaching vector is toward the palm (i.e., 5 With z 5 The direction perpendicular to the plane of rotation formed by the axis; Fig.12 Displayed direction.

[0291] p represents the position vector of the hand, which is the position vector from the origin of the base coordinate system ( Fig. 7A as well as Fig.12 O) refers to the vector of the coordinate system to the center of the hand; Fig.12 Displayed direction.

[0292] Specifically,

[0293]

[0294] n x =C 1 [C 23 (C 4 C 5 C 6 -S 4 S 6 )-S 23 S 5 C 6 ]-S 1 [S 4 C 5 C 6 +C 4 S 6 ]=R 11 of Eq.(8);

[0295] n y =S 1 [C 23 (C 4 C 5 C 6 -S 4 S 6 )-S 23 S 5 C 6 ]-S 1 [S 4 C 5 C 6 +C 4 S 6 ]=R 21 of Eq.(8);

[0296] n z =-S 23 [(C 4 C 5 C 6 -S 4 S 6 )]-S 23 S 5 C 6 =R 31 of Eq.(8);

[0297] s x =C 1 [-C 23 (C 4 C 5 C 6 -S 4 S 6 )+S 23 S 5 C 6 ]-S 1 [-S 4 C 5 C 6 +C 4 S 6 ]=R 12 of Eq.(8);

[0298] s y =S 1 [-C 23 (C 4 C 5 C 6 +S 4 S 6 )+S 23 S 5 C 6 ]+C 1 [-S 4 C 5 C 6 +C 4 S 6 ]=R 22 of Eq.(8);

[0299] s z =S 23 [(C 4 C 5 C 6 +S 4 S 6 )]+S 23 S 5 C 6 =R 32 of Eq.(8);

[0300] a x =C 1 [(C 23 C 4 C 5 +S 23 C 5 )]+S 1 S 4 S 5 =R 13of Eq.(8);

[0301] a y =S 1 [(C 23 C 4 C 5 +S 23 C 5 )]+C 1 S 4 S 5 =R 23 of Eq.(8);

[0302] a z =-S 23 C 4 S 5 +C 23 C 5 =R 33 of Eq.(8);

[0303] p x =C 1 [d gripper (C 23 C 4 S 5 +S 23 C 5 )+S 23 d 4 +a 3 d 4 +a 3 C 23 +a 2 C 2 ]-S 1 (d gripper S 4 S 5 +d 2 )=absolute address parameter X of Eq.(8)

[0304] p y =S 1 [d gripper (C 23 C 4 S 5 +S 23 C 5 )+S 23 d 4 +a 3 d 4 +a 3 C 23 +a 2 C 2 ]+C 1 (dgripper S 4 S 5 +d 2 )=absolute address parameter Y of Eq.(8)

[0305] p z =d gripper (C 23 C 5 -S 23 C 4 S 5 )+C 23 d 4 -a 3 d 23 +a 2 S 2 +H=absolute addressparameter Zof Eq.(8) (17)

[0307] Fig.12 The image shown is a Fig. 7A The end point of the robot (end point; P end ) due to holding the tool (utensil or implement) to the vector extends for a distance in the direction of Fig.12 Here, the distance is indicated by the additional length of utensil. This extension action causes its end point to move to a new location, namely P' end .like Fig.12 As shown, the direction of the gripper can be determined by three orthogonal vectors as well as to express its characteristics.

[0308] We now understand that when we want to specify a position for the matrix T, we need to provide the rotation angles for each of its joints, that is, θ 1 ,θ 2 ,θ 3 , …, and θ 6 (For ease of explanation, the present invention will refer to the above angles as rotational angles from now on.) Until now, the prior art has not developed any effective method to derive these angles in a Cartesian coordinate system.

[0309] Whenever encountering such a situation, engineers must use indirect methods, such as iterative algorithms, to find the answer. To deal with this problem, the three-dimensional graphical user interface (207) of the present invention uses a Figure 2A The special three-dimensional navigator (202) shown in FIG. 1 is used to facilitate the adjustment of the rotation angle θ in real time. 1 ,θ 2 ,θ 3 , …, and θ 6 .

[0310] The above-mentioned function (adjusting the rotation angle in real-time) is very important to the robotics industry and the graphical user interface industry; many applications can be established; for example, an application is a special surgical operation; when performing the special surgical operation, a surgeon controls a robotized tool to perform the surgical operation, etc. The basic function of the three-dimensional graphical user interface (207) of the present invention is its ability to simultaneously control the absolute address, translation and rotation movement of a robot's joints (this function is represented by formula (8)).

[0311] In fact, from a broad perspective, any movement of a three-dimensional object that can be represented by formula (8) can use the three-dimensional graphical user interface (207) of the present invention to achieve an unprecedented user interaction experience. Some exemplary applications with such effects include at least:

[0312] (1) using a drawing program to modify the movement of a cartoon character, the drawing program using the three-dimensional graphical user interface (207) of the present invention;

[0313] (2) Drawing a mechanical part using the three-dimensional graphical user interface (207) of the present invention;

[0314] (3) Using the three-dimensional graphical user interface (207) of the present invention to control the actions of a character in an animated electronic game;

[0315] (4) Using the three-dimensional graphical user interface (207) of the present invention to manipulate a tool of a medical diagnostic instrument, etc.

[0316] The above robot manipulation process and the use of forward kinematics to determine a robot's endpoint (i.e., P endIn addition to forward kinematics, the three-dimensional graphical user interface (207) of the present invention can automatically adjust the position and angle of a robot's joints according to inverse kinematics.

[0317] In a Fig. 7A An exemplary six-joint During the motion process of the robot (700), the rotation angle of each joint can be automatically calculated as follows: the first three joints (i=1-3) have four possible solutions; based on each of the four possible solutions obtained for the first three joints, the next three joints (i=4-6) have two possible solutions. A typical method for calculating the position and / or angle of each joint is to project their three-dimensional vectors on x. i-1 -y i-1 On a plane (such as Fig. 7A shown).

[0318] Since the positions and / or rotation angles of joints 4 to 6 will not cause the two front elbows L elbow1 and L elbow2 To change the position of 0 i ) and (i = 4 to 6) to calculate its projection on x i-1 -y i-1 vector on the plane; therefore, the solution of the position and / or rotation angle of joints 4 to 6 is based on the solution of the first three joints, that is, the secondary matrix T 0 i To calculate.

[0319] Due to the aforementioned Fig. 7A There are many solutions for the joints of the robot (700) in the present invention, and artificial intelligence methods can be used to help the three-dimensional graphical user interface (207) of the present invention determine the above solutions more quickly and easily. In a typical navigation program with artificial intelligence functions, a program that uses artificial intelligence functions to process repetitive data processing can be implemented on the three-dimensional graphical user interface (207) of the present invention; the aforementioned data processing program with artificial intelligence functions can process a series of linear and nonlinear motion data (i.e., T in a timely manner) for a tracked object. 0 i , i-1 A i ,wait).

[0320] For example, in an imaginary animated video game, a cartoon character Peter Pan is fighting Captain Hook with a dagger. Combining the three-dimensional navigator (202) of the present invention with the three-dimensional graphical user interface (207) of the present invention enables an operator to freely move Peter Pan's dagger in a three-dimensional coordinate system (i.e., using the rotation matrix R and position data X, Y, Z in formula (8)); at the same time, by using the aforementioned robot kinematics, the three-dimensional graphical user interface (207) of the present invention can calculate the position of Peter Pan's dagger and the rotation vectors of some joints used by Peter Pan to control the dagger (i.e., using Fig.12 P' end , as well as to represent the position and motion vector of the dagger).

[0321] When an operator instructs Peter Pan to thrust the dagger inward and then swing it to deflect Captain Hook's sword, the entire process is similar to a fast robot interacting with a single program that combines some alternating forward kinematics and inverse kinematics. Occasionally, Peter Pan's combat process will require the assistance of artificial intelligence to speed up the solution process.

[0322] For example, in combat, the three-dimensional graphical user interface ( Figure 2A (207) in the figure) can occasionally pay attention to the relative distance between Peter Pan and Captain Hook; when a certain neural network condition is met, the three-dimensional graphical user interface of the present invention ( Figure 2A (207) in the figure will automatically adjust the relative distance and the related rotation angle. As explained above, the performance of a typical graphical user interface is determined by how efficiently and smoothly the graphical user interface interacts with the user. The three-dimensional graphical user interface ( Figure 2A The special functionality provided by (207) in the robot, namely robot kinematics, elevates a user's interactive experience to a level far beyond known technologies. In addition, when Peter Pan flies and swings in a three-dimensional space, this phenomenon also expresses that the three-dimensional space where Peter Pan is located can contain a predetermined vector field, which can be combined with a motion vector provided by a navigator to form a new motion vector for Peter Pan.

[0323] Figure 7B The figure shows a robot that imitates the human skeleton (Humanoid robot). Figure 7B This demonstration shows that a humanoid robot structure (701B) can use up to 25 degrees of freedom to represent the movements of its head, torso, pelvis, legs, and arms. When manipulating the humanoid robot structure (701B), the three-dimensional navigator ((202) in FIG. 2) of the present invention can provide 6 degrees of freedom at a time. At first glance at this situation, the reader may feel that the three-dimensional navigator ((202) in FIG. 2) of the present invention seems to be unable to manipulate all the joints of the humanoid robot structure at once (this would require 25 degrees of freedom at a time).

[0324] However, the three-dimensional graphical user interface ( Figure 2A (207) in the figure can adapt to this situation in a highly efficient manner. It is worth noting that usually an operator does not need to use 25 degrees of freedom at once. Therefore, the three-dimensional graphical user interface ( Figure 2A (207)) can select 6 degrees of freedom in a period of time to manipulate a specific posture of the anthropomorphic robot structure (701B), and then, by changing the selection of some different sections of the skeleton during the entire manipulation process, the last operator can still manipulate all 25 degrees of freedom during the entire manipulation process.

[0325] If a reader examines carefully Figure 7B , he or she will find that to move the head of the anthropomorphic robot structure (701B), 3 degrees of freedom are required (note, less than 6); to move an elbow of the anthropomorphic robot structure (701B), 1 degree of freedom is required (note, less than 6); to produce a movement in a region from the toes to the knees, 4 degrees of freedom are required (still less than 6), and so on. If a three-dimensional graphical interface ( Figure 2A The user of (207) in the present invention can cycle through the above-mentioned joint segment groups in a very fast manner, or use an intelligent program to determine which of the above-mentioned joint segments should be used, then the three-dimensional graphical user interface ( Figure 2A (207)) can still become a highly efficient and easy-to-use graphical user interface when only 6 degrees of freedom are used at a time. For example, when the three-dimensional graphical user interface of the present invention ( Figure 2A In the example of (207) above, when controlling the flying movement of a cartoon character Peter Pan, in this case, it is usually unnecessary to manipulate the posture of Peter Pan's legs; it is the rotational movement of Peter Pan's body and his two arms that has the greatest impact on his flying performance.

[0326] The anthropomorphic robot structure (701B) described above can still be modified to adapt to the different body structures of various cartoon characters (eg, dogs, monsters, beauties, etc.). Fig. 8A The illustrated figure is a cartoon character (800), whose pelvis and hip area are covered by his clothes (801); the corresponding humanoid robot structure (701B) is shown in FIG. Figure 8B .

[0327] Please note the pelvic area of ​​the simplified anthropomorphic robot structure (801B in FIG. 8 ); through such a simplification process, the degree of freedom of the entire cartoon character (800) can be greatly reduced, which makes the three-dimensional graphical user interface ( Figure 2A In actual operation, the complete anthropomorphic robot skeleton structure (701B) and the simplified anthropomorphic robot skeleton structure (801B) can be simultaneously displayed on the three-dimensional graphical user interface ( Figure 2A For example, an envisioned user creating a cartoon feature movie can use the three-dimensional graphical user interface of the present invention ( Figure 2A (207)) is used to control a simplified humanoid robot skeleton structure (801B) as a method of controlling a supporting role; as for controlling the actions of a main character, the related work can be handed over to a more complex humanoid robot structure (701B). Furthermore, when the problem comes to controlling facial expressions, the three-dimensional graphical user interface of the present invention adopts a more basic level, namely a polygon network.

[0328] Fig. 10A The polygon network of a human face is shown. Let us now analyze a patch (e.g., (1002)). Fig. 10B As shown, in order to modify the tile (1002), the three-dimensional graphical user interface of the present invention allows an operator to touch the four corners of the tile (i.e., A, B, C, and D) and the normal vector (1010) of the tile, which is also called a vertex (vertex (1010); alternatively, it can also be called a node).

[0329] Take corner point A as an example. When an operator clicks the corner point A with a mouse button and then drags the three-dimensional navigator (202) of the present invention on the reference surface (205T) for a certain distance, this action will cause the corner point A to generate a corresponding translation motion vector. Fig. 10CAs shown, by such a translation movement, and depending on the direction of dragging, the corner point can be moved to A', A", A'", or A"", etc.

[0330] Corresponding to the movement of the corner point, the shape and / or area of ​​the patch (1002) will also change. We should be careful to note that when the shape and / or area of ​​the patch (1002) changes with the movement of the corner point A, the position of the vertex (vertex 1010) in the patch (1002) also changes at the same time. Fig. 10B As shown in FIG. 10G , when an operator clicks the normal vector or vertex (1010) with a mouse button and then moves his or her finger on the posture detection mechanism ((203) in FIG. 2 ) on the three-dimensional navigator (202) of the present invention, he or she can adjust the inclination of the rotation plane (1002).

[0331] like Fig. 10D , 10E, and 10F, we can regard the above-mentioned process of adjusting the inclination of the rotating plane (1002) as a rotation action of the plane (1002) around the X, Y, or Z axis (i.e. Fig. 10D , 10E, and (1002'), (1002"), and (1002'") in 10F); this understanding also matches our previous explanation. As revealed by formula (12), adjusting the normal vector (that is, changing n 1 , n 2 , or n 3 The value of ) can lead to the rotation matrix R in R 11 ~R 33 changes, which means that the above process of adjusting the normal vector will result in the angle in formula (11) φ, and θ are changed. In a movie where users can interact with cartoon characters, a GUI that is more popular with users is a three-dimensional GUI that can simultaneously provide vertex translation motion vectors and rotation motion vectors for a patch in a polygonal network. The known technology requires the use of separate devices and / or programs to handle the above-mentioned tasks; when a GUI requires the use of multiple input devices and / or programs to control the movement of an object (the patch in the polygonal network mentioned above can be regarded as an object), its original purpose of adjusting the posture of a cartoon in real time has been frustrated.

[0332] The three-dimensional graphical user interface of the present invention ( Figure 2A(207)) can eliminate this problem. Therefore, the present invention provides an innovative processing method for manipulating the gestures / expressions of an object in a realistic manner.

[0333] Of course, there are many cases where a 3D object in a 3D GUI is not an anthropomorphic robot. In these cases, a flying carpet ( Figure 2A (201)) can be used to represent the three-dimensional graphical user interface of the present invention ( Figure 2A The basic value of (207)). Figure 2A As shown in FIG. 1 , when the intersection of the rotating plane and the coordinate axis changes (that is, when a, b, and c change), the flying carpet ( Figure 2A The appearance of (201)) will also change.

[0334] A particular change in appearance (i.e., "gesture") observed by an observer is in fact an illustration of the most important of many properties of a three-dimensional object. An oversimplified graphical user interface, like most of the ones available on the market today, would consider the above-mentioned flying carpet (201) as a mathematical point. Limited by the use of such simplifications, known graphical user interfaces discard the invaluable treasure of a three-dimensional object, which is related to the rotational movement and geometrical transformation of a three-dimensional object.

[0335] We now understand that when the electronics industry enters the 3D field, it will need a full-fledged 3D GUI that can provide 6 degrees of freedom (i.e. Figure 4B This phenomenon also further confirms a claim of the present invention, which states that: because the 3D GUI of the present invention can support its direct and intuitive interaction with the user with sufficient degrees of freedom, this feature represents that the 3D GUI of the present invention is a truly interactive 3D GUI.

[0336] Based on the above analysis, a rotation matrix used to control a robot is essentially the same as a rotation matrix used to control a 3D cursor and / or object in a 3D GUI. As shown in the flying carpet (201), most objects we encounter in the natural world are affected by both translational motion (i.e., changing from one X, Y, Z 3D address to another X', Y', Z' 3D address) and rotational motion (change of gesture).

[0337] When we expect the 3D graphics revolution to happen, the electronics industry will have a need to develop a new type of 3D graphical user interface (i.e., 207 disclosed in the present invention), which can simultaneously provide linear (translation) and nonlinear (rotation) motion vectors to a 3D object or a robotic device based on data generated by a posture movement of a navigator.

[0338] In the following paragraphs we will explain the basic function of stereography using the degrees of freedom of each point (i.e., pixel) in a display device. A user can use learning theorem to classify some graphical vectors used for drawing in a 3D GUI so that the GUI can more effectively present a 3D scene based on stereography techniques.

[0339] In this section, the 3D scenes we consider are static (i.e., the objects in the 3D scenes have no motion vectors); therefore, only graphical vectors will be discussed in this section. In section 8, we will show how an observer can interact with a dynamic 3D scene using a 3D graphical user interface (i.e., in such an interactive 3D scene, graphical vectors; motion vectors; and perspective angles; all participate in the interactive process). Learning principles are a powerful tool that can be used to classify graphical objects; they will be used in sections 7 and 8.

[0340] Before and Figures 10A to 10G In the relevant paragraphs, we have explained how the three-dimensional graphical user interface of the present invention provides each vertex of the polygon network (i.e. Fig. 10B (1010)) a 6-DOF motion to adjust the position and / or posture of a 3D object. Once the above editing procedure (i.e., the movement of vertices) is completed, the appearance of an object used as a demonstration will be like Fig. 10A Same, in Fig. 10A Here the polygonal network is a human face (1001).

[0341] It is worth noting that although the 3D GUI of the present invention can provide a vertex with a motion vector with 6 degrees of freedom, the finished product is a group of vertices, which may not have any special physical relationship with each other. As a result, the patches (such as Fig. 10B 1002) are like some unrelated leaves hanging randomly on a tree.

[0342] When a physicist represents a dimension of a feature vector by one degree of freedom per vertex (e.g., (1010)), then Fig. 10B The dimension of the feature vector space consisting of the feature vectors of all (e.g., n) vertices in a 3D GUI involves a total of n vertices in the 3D GUI. Therefore, when n is large, the number of vertices can be very large (that is, in an exemplary case, the total dimension of the vertex vector space is n×6).

[0343] In fact, in a modern digital graphics display device (e.g., LCD panel), the number of vertices in a polygonal network can be surprisingly high (e.g., tens of thousands). Therefore, a 3D object created in this way can become too complex to be easily understood by a human observer. For example, it is difficult for a human to easily infer any physical relationship between the leaves on a tree. Therefore, finding a way to reduce the dimensionality of the feature vectors is a must for next-generation high-quality 3D GUIs to be truly user-friendly.

[0344] As will be explained below, the solution to the above problem is to enable the 3D GUI of the present invention to use another type of vector, namely graphical vectors, which are different from motion vectors. The graphical vectors are related to drawing a 3D graphic entity (for example, a 3D vector graphic is constructed from a number of 3D graphic vectors). Depending on the application, an object in a 3D GUI can be constructed from a vector graphic (for example, Fig. 10H , or represented by a polygonal network (e.g., Fig. 10AStated differently, an object in a 3D GUI can be represented by several graphic vectors (e.g., Fig.10I GV 1 ); Alternatively, the three-dimensional graphical user interface can be composed of some vertex vectors (vertex vector, for example, Fig. 10B On a computer, two vectors coexist in the three-dimensional graphical user interface of the present invention:

[0345] (1) Vertex vectors (1010), i.e., normal vectors of tiles (e.g., (1002)) that form the outer surfaces of some graphical entities (thus, the directions of these vertex vectors may point in many different directions in a 3D GUI);

[0346] (2) Graphical vectors; for example, ), in fact each graphical vector is a three-dimensional vector that is projected onto a two-dimensional viewing plane (e.g., a flat-panel display). Therefore, the projection process maps some graphical features in a three-dimensional coordinate system to a 2.5D coordinate system.

[0347] Now referring to equation (8), we will notice that a point in a 3D GUI (for example, a point at x, y, z) is obtained by rotating the elements of the rotation matrix R (i.e., R 11 ~R 33 ) where its degrees of freedom are obtained. For example, the element R in the matrix R 12 Suitable as a parameter, this parameter can determine how much an object can move along the X axis when it moves a certain distance along the Y axis.

[0348] In the same way, the remaining elements in the rotation matrix R can determine other linear and nonlinear movements of the object along the X, Y, and Z axes. In short, the three-dimensional graphical user interface of the present invention can not only manipulate the positional values ​​of a point in a three-dimensional space, but also store the rotation matrix R of the point.

[0349] For an example, please refer to Figure 2A and 2B, and assuming that the light source ( Figure 2B 213) is set to a certain value so that R 11 : R 12 : R 13 =1:2:1, then, when the above object (for example, the object is located at a position, the position is represented by X, Y, and Z) moves along the X axis, its relative degrees of freedom (i.e. displacement) along the X, Y, and Z axes are 1:2:1.

[0350] In the same way, we can derive the degrees of freedom of the object along other axes. Therefore, by setting the illumination conditions of the light source (213) or by directly adjusting the values ​​of the elements of the matrix R, a three-dimensional graphical user interface can determine the degrees of freedom of each point in a three-dimensional space. In Section 9 (or more specifically, as Fig.11E ), an operator knows that by placing an index finger (1806) on the gesture detection mechanism ( Figure 2B 203), the three-dimensional navigator (202) of the present invention is able to determine one or a group of special lighting conditions for light sources S1, S2, and S3; thus, the degree of freedom of a specific point located in the three-dimensional graphical user interface of the present invention can be determined.

[0351] The prior art does not have such capability. Therefore, since the present invention can display the 3D GUI in a wide range of ways that are not affected by the size of the object (i.e., the object can be as small as one pixel or as large as Fig. 10H The church (1009) is generally large) to manipulate the freedom of a graphic vector, which makes the present invention (207) a technological breakthrough in the graphical user interface industry.

[0352] In actual operation, an index finger (1806) is placed on the gesture detection mechanism ( Figure 2B 203) is actually equivalent to indicating the normal vector of the tile (1002) in the three-dimensional graphical user interface (207) of the present invention; in the three-dimensional graphical user interface (207) of the present invention, each tile (1002) can be represented by a point; the above normal vectors are equivalent to the vertex vectors (1010) or node vectors of the polygon network (1001). When a group of graphic vectors (these graphic vectors can be derived from the above points) in the three-dimensional graphical user interface (207) of the present invention have similar rotation matrices, this situation represents that the above group of graphic vectors constructs a small plane (facet) in a three-dimensional space, and the small plane uses its normal vector to point to a specific direction in the three-dimensional space.

[0353] We can use the drawing method described above to depict Fig.10J The roof of the church (1009) in Fig.10J As shown, the roof (◇P 1 P 2 P 3 P 4 ) is surrounded by four graphic vectors, that is, the graphic vector as well as If an operator draws the above-mentioned three-dimensional navigator (202) by dragging the present invention on the reference surface (205), as well as Furthermore, during the dragging process, if the operator does not shake his or her index finger, all the rooftops (◇P 1 P 2 P 3 P 4 ) all have the same normal vector, that is,

[0354] Using the perspective sketching technique, we can transform the roof (◇P 1 P 2 P 3 P 4 )'s three-dimensional feeling stands out, and this is why Fig. 10H The vector graphics give the observer such a strong three-dimensional impact. Extending this technique, when an operator wiggles his or her index finger while dragging the three-dimensional navigator (202) of the present invention on the reference surface (205T), he or she actually creates (and uses) a new graphical expression method. In actual operation, the above-mentioned operator's action is equivalent to first giving a world space camera (world space camera; perspective camera) a starting three-dimensional position in the three-dimensional graphical user interface (207) of the present invention, and then using a series of translation and rotation motion vectors to make the stereoscopic viewing angle reach a specific angle, at which time the visual classification reaches an optimized condition.

[0355] We call this mode of expression "digital impressionism"; the term "digital impressionism" is borrowed from the term "impressionism"; the term "impressionism" was originally used by some artists, such as Claude Monet. The "impressionism" used by Claude Monet and his group refers to a painting technique that paints colored patches into the landscape to illustrate the fleeting effects of light, which are more likely to appear in an outdoor environment. When the three-dimensional graphical user interface (207) of the present invention produces an image presentation effect similar to "impressionism", we believe that calling "impressionism" the "digital impressionism" of the present invention is reasonable.

[0356] If a GUI designer has no way to specify the relationship between some graphic vectors in a three-dimensional scene, the dimensionality of the feature vector space established by these graphic vectors cannot be effectively reduced; as a result, viewers may not be able to easily and effectively understand the complex three-dimensional scene. In the past, whenever such a situation is encountered (for example, there are many objects in a three-dimensional scene, and the outlines of some of these objects are very complex), previous GUIs may attempt to use a quick-fix approach, which is to specify several distinctly different depth values ​​(i.e., position data that varies along the Z axis) for related objects in the three-dimensional scene. However, when a known GUI places many objects at different depth values, and the many objects are separated from each other in a continuous manner, a difficult situation arises (i.e., it becomes difficult to separate related objects in a scene into several distinctly different depth values).

[0357] We should remember that it was clearly explained in Section 2 that a real object has both size and orientation, but a mathematical point has neither of these properties. Moreover, a real object may be of any size; for example, some real objects can be as large as Fig. 10HSome real objects can be as small as a pixel, etc. That is, in a 2.5D coordinate system, a tangible object can be as large as a church (1009) in the sky, some ... Fig. 10B , C, D, E, and F), the freedom of the contours of these objects can still be cleverly handled for better visual effects.

[0358] When a graphic artist places many polygon networks in a three-dimensional scene and aligns some graphics vectors in some major directions (i.e., the direction of the vanishing lines) so that the graphics vectors are aimed at a common point (i.e., the vanishing point), the degrees of freedom of objects that appear in a 2.5D coordinate system corresponding to the three-dimensional scene can be reduced without distorting the outline of the above polygon networks.

[0359] Such a clever way of handling graphic vectors makes it easier for a viewer to understand a three-dimensional scene. Inspired by stereographic drawing techniques and artificial intelligence (i.e., learning theorem), the three-dimensional graphical user interface of the present invention allows a drawing artist to pick out some graphic vectors (e.g., contour lines of an object) and aim these graphic vectors at a specific vanishing point. According to the learning theorem, the above-mentioned procedure has the effect of classifying the above-mentioned graphic vectors into a special group. When a three-dimensional scene can be represented by some such groups as described above, the entire (classified) three-dimensional scene will look neat, and the relationship between the geographical locations of related objects in the three-dimensional scene will become easier for the viewer to understand.

[0360] Fig. 10HShown is a church (1009) that is depicted using the three-dimensional graphical user interface (207) of the present invention and in accordance with the method described above. We can notice that the outline of the church (1009) gives the viewer an extraordinary sensation of three-dimensionality; this effect cannot be easily matched even by the most advanced known technology today. Perspective sketching is a graphical rendering technique that was developed during the Renaissance era. When perspective sketching was still being developed, some old art masters discovered that by aligning the outlines of certain objects in certain cardinal directions and aiming the outlines at a point that the viewer may not see (i.e., the vanishing point), the entire picture would be instilled with an extraordinary three-dimensional feeling that could not be easily matched by other graphical rendering techniques. As Fig. 10H As shown, this artistic technique is so strong, yet so delicate, that an ordinary viewer can feel its impact without even recognizing the existence of the vanishing point.

[0361] To find these traces of alignment, Fig.10I As shown, we can extend the edge lines of certain objects (e.g., the roof of the church (1009), the eaves, etc.); by finding these extension lines, we can further find where they intersect with each other (i.e., Fig.10I The above technique may sound like a piece of art that has become a thing of the past for an experienced artist, but please note that those old masters did not realize that 3D drawing techniques have much to do with today's artificial intelligence technology, which is to classify some graphic vectors so that the viewer can better understand a picture (that is, to use machine learning to help humans learn some complex graphics).

[0362] An electronic system that understands this principle can explore the rich information buried in a large number of graphic vectors. Fig.10I, we reach an understanding: it is because we align some contours that a picture presented using a three-dimensional drawing technique looks so realistic. The above understanding is entirely related to reducing the degrees of freedom of the contours. However, the most advanced learning principles today only know that reducing the degrees of freedom of a large number of vectors helps a viewer learn the space constructed by these vectors. The most advanced learning principles today cannot explain why a picture presented by a three-dimensional drawing technique looks so realistic.

[0363] For centuries, aligning contours and aiming them at a vanishing point has been considered an artist's finesse. Fig.10I If you ask a scientist familiar with artificial intelligence, he will understand that the above process of aligning graphic vectors in a three-dimensional scene is related to classifying graphic vectors, but he may not understand why a vanishing point can have such a strong visual effect.

[0364] Fortunately, we know that the apparent degrees of freedom of objects represented in a 2.5D coordinate system change continuously as the relative distance of the object(s) from the vanishing point changes; and vanishing lines (e.g., VL 11 (1004), V.L. 12 (1005), V.L. 21 (1006), V.L. 22 (1007) etc.) is an auxiliary mechanism used in a graphic imaging device, and the auxiliary mechanism is used to clearly reveal the above properties. Therefore, the three-dimensional graphic user interface of the present invention is a bridge between science and art.

[0365] Raise Fig.10I For example. Fig.10I As shown, we can extend some of the roof, eaves, etc. of the church (1009) to the vanishing line VL 11 (1004), V.L. 12 (1005), V.L. 21 (1006), V.L. 22(1007), etc. When some vanishing lines converge to a vanishing point, for example, VP1 (1011) or VP2 (1012), anyone can recognize that the three-dimensional image presentation effect of the church (1009) is excellent without understanding the laws of physics. Examining the excellent three-dimensional image presentation effect mentioned above from a physical perspective, we know that the excellent three-dimensional image presentation effect mentioned above is related to reducing the dimension of the feature vector space (i.e., a space constructed by graphic vectors).

[0366] Please refer to Fig.10J In this stereoscopic image (which is a 2.5D coordinate system), we can find the church ( Fig.10I The roof line of (1009)) can be represented by the graphic vector to represent (that is, a 3D vector projected onto a 2.5D coordinate system; projection is an efficient way to reduce the dimensionality of a vector space constructed by many graphic vectors without causing unrealistic distortion effects).

[0367] To further explain, and Both graphics vectors are aiming at VP2 (1012); is a line representing the eaves. This arrangement can help a viewer recognize that the two graphic vectors above belong to the same object, namely the rhombus-shaped roof of the church (1009); the rhombus is represented by its four vertices P 1 , P 2 , P 3 , and P 4 From a physical point of view, the above program helps a user to classify some graphic vectors into some special categories. Using the same method, we can further recognize as well as All are aimed at the vanishing point VP2 (1012).

[0368] According to the learning principle (for example, support vector machine, support vector machine, SVM; for a detailed explanation of support vector machine, we can refer to: "Support Vector Network", Machine Learning, 20, 273-297 (1995), Corrina Cortes and Vladimir Vapnik); the following graphic vectors all point in a similar direction, that is, as well as Therefore, these graphic vectors can be classified into the first group of graphic vectors.

[0369] Furthermore, using the same technique, we can recognize that in the feature vector space associated with the church (1009), the image vector and can be classified into the second group; further, we can recognize that the graphic vector and can be classified into the third group; further, we can recognize that graphic vectors and can be classified into the fourth group; further, we can recognize graphic vectors and can be classified into the fifth group; further, we can recognize the graphic vector and can be classified into the sixth group. Therefore, by classifying the graphic vectors according to the three-dimensional drawing techniques, this method provides an unprecedented understanding to the viewers of the three-dimensional graphical user interface (207) of the present invention.

[0370] Theoretically, in a high-capability 3D drawing device such as the 3D graphical user interface (207) of the present invention, the number of the above-mentioned vanishing points can reach a certain level: the 3D graphical user interface (207) of the present invention can have as many vanishing points as a digital drawing artist wants. According to common sense, in a 3D drawing method or device, the more the number of the above-mentioned vanishing points or the more the number of vanishing lines, the stronger the 3D feeling (sensation of three-dimensionality) provided by the 3D drawing method or device, and this design rule (vanishing points help to classify graphic vectors, and therefore also help to classify 3D objects) has never been used by previous graphical user interfaces in the past.

[0371] On the other hand, the above design rules have a limitation: when the degree of freedom of graphic vectors in a three-dimensional drawing is reduced to a severely low level due to the use of too many vanishing points, some graphic vectors used to express characteristic graphics may be lost or distorted. When this happens, a graphic artist must make a compromise between further classifying graphic vectors and preserving them. Fig. 10H The window (1013) in is an example. Fig.10I disappeared. Fig.10IAs shown, when a 3D GUI classifies the roof and eaves of a church (1009), the window (1013) becomes a decorative feature that can be removed from the scene. When encountering such a situation, a drawing artist must make a decision whether to keep the window (1013), and this decision may vary greatly depending on the expectations of each application.

[0372] Support Vector Machine (SVM) is a machine learning process that is widely used in techniques such as pattern recognition, prediction, classification, and regression analysis. In many past implementation examples, SVM has been proven to perform better than other machine learning programs, making it a technology of considerable practical value in many fields of application. When we define SVM in a general form, we use a maximum margin classifier to explain the effectiveness of SVM. For a practical SVM, its decision function is a feature vector space (for example, Fig.15 X d(d>3) -Y d(d>3) ) is a hyperplane (hyperplane, 1510) in the image. When the aforementioned margin (margin, 1502) reaches a maximum value, the hyperplane (1510) divides a plurality of feature vectors into different classes.

[0373] When we first use an SVM, we need to train the SVM. In graphical vector space, one possible way to train an SVM is to provide a dataset where x i ∈R N and i ∈{-1,+1} (for the convenience of explanation, φ is temporarily regarded as a linear mapping function, and the nonlinear mapping method is not discussed for the time being), then the SVM method after the above education process can solve the following problems:

[0374]

[0375] Formula (18) is subject to the following two constraints:

[0376]

[0377] It is worth noting that due to x i ∈R N , w and b in formulas (19) and (20) represent the N There are two linear classifiers in the above. ξi represents the maximum error range allowed by the aforementioned SVM classification procedure. We can convert the aforementioned groups of As learning data, each x i are all input data, each y i are all output data, that is, the y i Indicates that x i Which group (classification, such as 1501L, 1501R) it belongs to. For the present invention, the aforementioned x i It is a graphical vector in a graphical user interface. According to the principles of artificial intelligence, a machine learning program is a one-to-one Data builds a model, and then, next time there is a new When data appears, the model can accurately output a predicted value, or within the allowable error range, that is, The “,” in formula (19) represents the dot product.

[0378] Section 8. Key components of the 3D navigator of the present invention and their application in the 3D graphical user interface of the present invention

[0379] This section 8 briefly explains Figure 2B The key components of the three-dimensional navigator (202) of the present invention are disclosed in full in U.S. Patent Application No. 14 / 294,369. Please refer to Figure 2B ,exist Figure 2BThere is a vertical cross-sectional view of the three-dimensional navigator (202) of the present invention at the lower left. As shown in this vertical cross-sectional view, the three-dimensional navigator (202) of the present invention has at least one empty hole (212), and the empty hole (212) has an opening facing the reference surface (205); inside the empty hole (212), there are three light sources (the three light sources are represented by a group 213) installed on the upper part of the empty hole, wherein each light source is set to emit light downward so that it can illuminate the reference surface (205); the wavelength range of the light emitted by each light source is different from each other.

[0380] The hole (212) also includes at least one color image detector (214), which is also placed above the hole and faces downward toward the reference surface (205). The color image detector (214) has many pixels, each of which (not shown) is only sensitive to light in a certain wavelength range of the three wavelength ranges. A posture detection mechanism (203) is also installed on the upper surface of the three-dimensional navigator (202) of the present invention; the posture detection mechanism (203) follows a method such as Figure 2D The event triggered by the finger (206) shown is responsible for adjusting (e.g., tilting the plane of a carpet) a three-dimensional object (e.g., Figure 2A The flying carpet (201) in the picture.

[0381] Fig.11E What is shown is an enlarged view of a gesture detection mechanism (203) located under a fingertip (1806). Fig.11E As shown, the above-mentioned posture detection mechanism (203) is a light diffusion surface ((203), for example, a Lambertian surface), and the light diffusion surface (203) is embedded in a certain position of a mouse shell (202), so that the light diffusion surface (203) can be touched by an operator's finger (1806). Below the light diffusion surface (203), there are three light sources (S1, S2, and S3) and an image detector (represented by a circle (1818)), and the three light sources (S1, S2, and S3) and the image detector (1818) are surrounded by a second hole (this second hole is not in the Fig.11E Please note that the second hole is Figure 2BThe holes (212) in the light diffusion surface (203) are not the same); when the three light sources (S1, S2, and S3) emit light of different colors or wavelengths at the same time, the intensities of the light of different colors or wavelengths reflected from the light diffusion surface (203) will change from one position to another.

[0382] The light reflected from the light diffusion surface (203) together constructs a special color gamut (color gamut, for example, CIE 1931RGB, etc.). It is worth noting that there are two image detectors on the three-dimensional navigator (202) of the present invention, one of which is facing upwards to capture the image of the inner surface of the light diffusion surface (203); the other image detector is facing downwards to capture the image of the reference surface (205) (this image detector is (214)).

[0383] We now refer to the light sources that illuminate the reference surface (205) downward as tinting light sources (i.e. (213)); meanwhile, we refer to the light sources that illuminate the light diffusing surface (203) upward as gesture adjusting light sources (i.e. S1, S2, and S3).

[0384] Fig.11E Illustrated are three gesture adjustment light sources (S1, S2, and S3) placed below the light diffusion surface (203); readers should note that in the previous paragraphs of the present invention, (203) is referred to as a gesture sensing feature.

[0385] When the finger (206) is moved to a place close to S1, the intensity of the reflected light detected by the image detector (1818) generally follows the following special relationship, that is, I R >>I G =I B According to the same method, when the finger (206) is moved to a place close to S2, the intensity of the reflected light detected by the image detector (1818) generally follows the following special relationship, that is, I G >>I R =I B Furthermore, when the finger (206) is moved to a place close to S3, the intensity of the reflected light detected by the image detector (1818) generally follows the following special relationship, that is, I B >>I G =I R .

[0386] Through analog to digital converting (ADC) circuit, the image detector (1818) converts the intensity value (I B , I G , I R ) is converted into a digital signal, and then the image detector (1818) transmits the above conversion result to an electronic system (e.g., 200) through a radio frequency channel (RF channel), a universal serial bus (USB), Bluetooth, etc. As explained above, the digital signal is suitable for describing the posture of a three-dimensional object. Therefore, when such posture data is received by the three-dimensional graphical user interface of the present invention, the posture data can be used to determine the values ​​of parameters a, b, and c of formula (7).

[0387] Through such procedures, Fig.11A As shown in FIG. 11B , the gestures of objects (1801A) and (1801B) appear to be different. Furthermore, based on the previous explanation of formula (7), the two gestures described above represent two different planes in a three-dimensional space (i.e., plane ab'c' and plane ab"c'). If we attempt to use other conventional pointing devices to produce similar effects (e.g., using a rotary knob or a navigator), the result will be no different from changing the address of a three-dimensional object, e.g., from (x', y', z') to (x", y", z") - the gestural information is completely lost. From the above analysis, we can see that gesture is a property embedded in all three-dimensional objects.

[0388] It is worth noting that in the three-dimensional graphical user interface (207) of the present invention, formula (7) is not the only formula that can be used to describe the posture of a three-dimensional object. In actual use, there are many other different mathematical formulas that can be used to describe the posture of a three-dimensional object. Figure 2A As an example, the flying carpet (201) can now be considered to be placed on an imaginary three-legged table chair (in fact, this imaginary table is plane abc); the length of each of the legs represents the intensity of the red, blue, and green light detected by the image detector (1818). When the finger (206) moves to different positions of the aforementioned light diffusion surface (203) (for example, close to S1, S2, or S3, etc.), the normal vector of the imaginary table surface also changes accordingly (for example, from Change to ).

[0389] The imaginary table mentioned above is just a fictitious component, which is proposed to make it easier for us to explain the tilt (or rotation) of the flying carpet (201). If we replace the imaginary table with a gust of wind, then Figure 2A As shown, the flying carpet (201) can now hover or sway in a three-dimensional space. For such situations, the mathematical formula used to describe the posture of the flying carpet (201) can be changed into other forms (that is, different from the form of formula (7)). In addition, the phenomenon that the flying carpet (201) can hover and sway in a three-dimensional space also expresses that the three-dimensional space can contain a predetermined vector field, which can be combined with a motion vector provided by a navigator to form a new motion vector of the flying carpet (201).

[0390] Furthermore, when the lighting conditions of the aforementioned light sources S1, S2, and S3 change automatically (for example, gradually dimming, etc.), or the shutter speed of the image detector (1818) covered by the aforementioned light diffusion surface (203) changes, or certain functions of the aforementioned three-dimensional graphical user interface change, then a special posture can be projected onto the aforementioned flying carpet (for example, the trajectory of the flying carpet (1801C) is a three-dimensional spiral (1818B); the three-dimensional spiral (1818B) is the result of the three-dimensional navigator (202) of the present invention making a circular motion on a colored or dyed reference surface (205T)).

[0391] We should carefully note that the three-dimensional navigator (202) of the present invention can determine whether the intensity of the aforementioned dyed light source (213) needs to refer to the posture to adjust the intensity of the light source (S1, S2, and S3). If the answer is "yes", the three-dimensional navigator (202) of the present invention can capture a tri-colored image (brightness, hue) from the reference surface ((205T), representing that the reference surface 205 is dyed with some tri-colored functions), and the dyeing condition of the reference surface (205T) can be manually adjusted through the finger (206).

[0392] Furthermore, in the aforementioned image detector (214), since each pixel has different sensitivities to different wavelengths of light (e.g., Bayer structure), three monochromatic sub-images can be extracted from a tri-colored image captured by the image detector (214), and the overall intensity of each sub-image represents the intensity of the reflected light from one of the three light sources S1, S2, and S3. Therefore, in addition to carrying information about the translational motion vector (e.g., the translational motion vector is caused by the relative displacement between the microscopic dust on the reference surface 205 and the image detector 214), in a tri-colored image as described above, it also carries information about the tilting condition (rotation) of an imaginary plane (i.e., plane abc); in the aforementioned tri-colored image, the intensity of each color of light (e.g., red, blue, and green) represents the intersection of the imaginary three-dimensional plane with the X, Y, and Z axes of a three-dimensional coordinate system.

[0393] Section 9. Obtaining absolute addresses from color gamut maps provides unprecedented convenience

[0394] In Section 1, we have explained that absolute addresses are an effective way for a navigator to avoid accumulating positioning errors. This section (Section 9) extends the discussion in that section to explain why a navigator that provides absolute addresses can help a user's hand movements become more transparent to the user's brain, and as a result, the navigator can be used to provide excellent teaching effects.

[0395] When a user uses a computer to create a visual effect that requires a high degree of artistic skill or craftsmanship, the neural networks located in the operator's hands, wrists, elbows, and arms must work smoothly with the operator's brain to enable the operator to complete the professional work.

[0396] At the beginning of a professional work, we may only have a vague image of the object to be completed. The subsequent process of fully depicting such a virtual object and transferring it to a medium (for example, a piece of paper, a touch-enabled display, or other similar device) is deeply influenced by the learning process of a person's neural network.

[0397] In order to shape a shape or define a movement trajectory in a more accurate and beautiful way, an artist must familiarize himself or herself with the gestures of his or her hands through repeated "previews", and this work can now be accomplished by moving (maneuvering) the three-dimensional navigator (202) of the present invention on a colored or dyed reference surface (205T). When such a working environment or method is established, the three-dimensional navigator (202) of the present invention can repeatedly measure absolute addresses, which is equivalent to artists moving their hands in an absolute coordinate system. In the above "preview" (that is, neural learning) process, the results of the artist's hand movements will be evaluated in a real-time manner by the three-dimensional graphical user interface of the present invention, so that the above process can be adjusted and gradually improved to a professional level.

[0398] In the past, this task would have been performed by an operator using a drawing tablet (a process equivalent to painting on a reference surface with an invisible absolute coordinate system); previous navigators played no role in this regard (in fact, not just previous navigation devices, but all relative motion detection devices, could not play any role in this regard).

[0399] The human neural network (for example, in a finger, hip, elbow, and arm) is an evolving structure; through a continuous learning process, a specific technique can be created. Fig. 9 The illustrated example is a person's signature (909). The strokes of the signature represent a unique combination of a person's neural network, and the strokes are unique to the person signing the signature.

[0400] In fact, this is one of the most important functions that nature has given to humans; no other creature in the world is capable of creating highly skilled works, such as painting, writing, playing music, weaving, etc. It is worth noting that in order to be able to implement a program stably, an input device that can provide absolute addresses is preferred.

[0401] Previous "quasi 3D" graphical user interfaces knew nothing of this. Absolute addresses and non-linear motion did not exist in previous navigators of the past few decades; as a result, the hand-to-brain transparency described above was sacrificed in previous graphical user interfaces, which may be one of several contributing reasons for the stagnation of the computer industry in recent years. The true innate creativity of the human neural network was sacrificed by the limitations of previous input devices. With the three-dimensional graphical user interface (207) of the present invention, we expect that the computer industry may regain growth momentum for at least another decade.

[0402] In order to make the three-dimensional navigator (202) of the present invention a navigator (mouse) capable of providing an absolute address, the reference surface (205) on which it moves must be dyed in a tri-coloration manner (e.g., red, blue, and green are added to the reference surface by chemical or / physical methods, that is, to produce a reference surface 205T as indicated by the present invention). On this colored or dyed reference surface 205T, the intensity of the reflected light of the three designated colors (e.g., red, blue, and green components) changes in a continuous manner along three directions (i.e., the X, Y, and Z axes of a 2.5D coordinate system imprinted into the reference surface 205T).

[0403] When an operator moves the mouse body (202) on this three-color reference surface (205T), the measured color index data (e.g., CIE 1931 RGB, etc.) is suitable as an ideal data source, which provides the three-dimensional graphical user interface (207) of the present invention with absolute addresses (e.g., X, Y, and Z) with unprecedented resolution and sensitivity.

[0404] We should carefully note that the color index obtained from the reference surface (205T) actually forms a 2.5D coordinate system; when the color image detector (214) captures a color image from this color gamut, the whole process facilitates a mapping process that maps the aforementioned 2.5D coordinate system to the pixel array of the aforementioned image detector (214).

[0405] A circuit (e.g., an analog-to-digital converter, etc.) is used to convert the color index value of each location into position data (e.g., Figure 2AThe color gamut diagram is a color gamut diagram of the present invention. ...

[0406] There are many ways to color a reference surface, such as illuminating it with light of different wavelengths, coloring it with ink or / and pigments. Whether to color a reference surface, and how to choose an appropriate method for coloring it, are questions that a GUI designer needs to answer when considering how sensitive or high-resolution his or her GUI should be.

[0407] In general, there is no fixed answer to this question; a GUI designer can make his or her own decision based on the final appearance of his or her application. This decision also has a lot to do with the basic design of a navigator (e.g., (202) and / or (205)). In less stringent cases, the reference surface (205) need not be colored using the methodology of tri-coloration described above; in this case any surface without a specific color pattern can be used.

[0408] According to this product design guideline, an ordinary wooden surface, or a marble surface, etc., can also achieve the purpose of use. Obviously, since no dye or pigment is used to dye the reference surface, the sensitivity and resolution of the above-mentioned three-dimensional navigator (202) are relatively rough compared to the sensitivity and resolution of the navigator using the aforementioned colored or dyed reference surface (205T).

[0409] U.S. Patent Application Serial No. 13 / 834,085 discloses an RGB mouse that does not use a colored or dyed mouse pad. It is worth noting that due to the lack of a colored or dyed mouse pad, all the data provided by the 13 / 834,085 patent are relative motion vectors rather than absolute address data. Therefore, this type of mouse is almost unable to support the learning process of human neural networks (for example, it is very difficult to sign using a conventional navigator).

[0410] However, compared with the performance of previous navigators (which only use monochromatic light), the resolution and sensitivity of RGB navigators, such as disclosed in patent 13 / 834,085, are still higher than the known technology. WORD, etc.), the functions of the above RGB navigator may be considered good enough to meet the needs of work.

[0411] In Section 2, we have explained that treating a graphical object as a real entity rather than a mathematical point is a major function of the three-dimensional graphical user interface (207) of the present invention; this function is related to providing an object with a complete set of (that is, 6 degrees of freedom) motion vectors. This complete set of motion vectors actually includes at least two types of rotational motion, namely gestural tilting and spinning (in the present invention, spinning refers to a motion intrinsic to an object, rather than a motion caused by the relative displacement between two objects).

[0412] In US 9,733,727, we disclosed a method for extracting a rotational motion vector from a "fuzzy" motion vector. Therefore, US 9,733,727 provides a powerful method for a graphical user interface to adjust the perspective angle of a 2.5D coordinate system. In the present invention, when we say 2.5D, we mean that it is a 2D coordinate system and also includes an independent rotation axis. When an object located in a 2.5D coordinate system receives an instruction from US application number US 9,733,727 to rotate, it can perform an autonomous action of spinning, or it can perform a relative displacement with other objects, that is, rotating. In either of the above two cases, a spinning motion, spinning, or a rotating motion, rotating, can represent a higher order term of the motion (HOT) of an object.

[0413] It is worth noting that the performance of the 2.5D navigator (202) described above is superior to that of the previous 2D navigator not because it has an additional 0.5 degree of freedom. In the previous paragraphs (e.g., Section 6), we have explained that providing a translation motion vector and a rotation motion vector to a three-dimensional object at the same time will make the three-dimensional graphical user interface (207) of the present invention a graphical user interface that interacts with the user. From a physical point of view, such a function is related to the "kinetics" of motion.

[0414] When the movement of an object is extremely active, with the support of the aforementioned posture detection mechanism (203) and the ability of the image detector (214) to detect the rotation angle ω, the three-dimensional navigator (202) of the present invention provides parameters such as a, b, c, and Θ to the three-dimensional graphical user interface (207) of the present invention, so that it can simultaneously provide translation and rotation (that is, tilting and spinning) vectors to a three-dimensional object.

[0415] Providing an absolute address directly from a navigator to a three-dimensional object can further improve the performance of the three-dimensional graphical user interface (207) of the present invention (that is, reduce the conversion process required by the original two-dimensional navigator). For this reason, in the future, people who need to use finesse to perform their work in professional work (for example, graphic artists, surgeons, etc.) may rely on the three-dimensional graphical user interface (207) of the present invention to improve their professional work performance.

[0416] Fig.11A , B, C, D, and E indicate that the three-dimensional graphical user interface (207) of the present invention manipulates the output data (e.g., the y position data of the flying carpet 201) of the three-dimensional navigator (202) of the present invention according to certain neural network functions embedded therein. In this exemplary case, an operator moves the three-dimensional navigator (202) on the above-mentioned colored or dyed reference surface (205T) in a circular manner (i.e. (1817A)), and while moving the three-dimensional navigator (202), three light sources of the light source (213) are used to represent the flying height (i.e., the flying height). Fig. 11C The intensity of the light on the y-axis in the figure will change continuously (for example, when the navigator (202) is moving, the light intensity ...). Fig.11A The light of the y value of the coordinate system gradually darkens according to the neural network function described above, and the trajectory of the three-dimensional position data converted from the color indicator data will be a bit like a three-dimensional spiral (3D spiral, (1817B)).

[0417] If we examine carefully Fig.11D , the colored or dyed reference surface (205T) is no different from a two-dimensional plane (that is, a plane without depth or thickness); all motion vectors generated by the operator's hand (e.g., Fig.11D V A , V B , VC , V D ) have a two-dimensional format (i.e., x, z). After the three-dimensional navigator (202) of the present invention projects the three different colors or wavelengths of light in the dyed light (213) onto the aforementioned colored or dyed reference surface (205T), a color gamut is formed on the reference surface (205T); thereafter, each geographical point of the colored or dyed reference surface (205T) can be represented by a special color index data (e.g., CIE 1931RGB).

[0418] Compared to tracking the traces of some microscopic dust, the three-dimensional navigator (202) of the present invention provides another function, that is, the function of measuring the absolute address through the method of measuring color (tint measurement). Corresponding to this function is the motion vector (that is, Fig.11D V A ',V B ',V C ', and V D ') have a three-dimensional format, and their values ​​are not affected by accumulated positional errors; this function is the next basic advantage of the three-dimensional graphical user interface (207) of the present invention compared with the mouse and other previous two-dimensional navigators, or all previous devices using relative displacement.

[0419] Extending the product design principle described above, the program of gradually dimming or enhancing an illumination light (213) described above can be completed manually by moving a finger (206) on the gesture detection mechanism (203); or, the program of gradually dimming or enhancing an illumination light (213) described above can be completed manually by starting a neural network function embedded in the three-dimensional graphical user interface (207) of the present invention; or, the program of gradually dimming or enhancing an illumination light (213) described above can be completed by selecting an icon to start an automatic program, and so on.

[0420] Section 10. Using the 3D GUI of the present invention in cooperation with 2D windows

[0421] This section demonstrates how the three-dimensional graphical user interface of the present invention can cooperate with a two-dimensional window (e.g., or a web page, etc.). A window in the known art is approximately a two-dimensional plane with a normal vector pointing in a fixed direction (e.g., a direction perpendicular to the surface of the display device as shown in (1307)); Fig.13A The actual usage shown above is briefly shown.

[0422] Another alternative is that the three-dimensional graphical user interface of the present invention can cooperate with multiple windows as described above (for example, Fig. 13B (1303) and (1304)), where the normal vectors of the two windows point to different directions. Fig. 13B What is shown is an actual usage scenario as described above (ie, multiple windows).

[0423] Please refer to Fig.13A We can notice that the layout of the aforementioned window (1306) is similar to that of the most advanced two-dimensional applications (e.g. WORD, etc.). However, the three-dimensional graphical user interface (207) of the present invention has some key functions that are different from those of the previous two-dimensional graphical user interface (e.g., (105)). First, according to the basic design rules of the three-dimensional graphical user interface (207) of the present invention, the above-mentioned window (1306) is actually a three-dimensional object located in a three-dimensional coordinate system (1302) (that is, the window is a plane, and each geographical location point on the plane has its own z value).

[0424] That is, each pixel in the aforementioned window (1306) can be represented by a specific three-dimensional address (X, Y, and Z). Second, the aforementioned window (1306) has a normal vector; in this case, the normal vector is represented by a symbol (1307) that points in an arbitrary direction relative to the normal vector of the display device (207). Third, in the above-mentioned three-dimensional graphical user interface (207) of the present invention, there is a world coordinate system (1302) in which Fig.13A In , the origin of the world coordinate system is represented by point V; in painting, the origin V represents the vanishing point of a stereoscopic drawing (that is, when an object is placed closer to the viewer, the object will appear larger, and vice versa).

[0425] Fourthly, the cursor (209) is also a three-dimensional object; for example, the cursor (209) can be placed Fig. 13BThe cursor (209) is a position between two windows (i.e., (1303) and (1304)) in the image processing system; moreover, the cursor (209) can be tilted (i.e., relative to the aforementioned world coordinate system (1302), it will have pitch, yaw, and roll, i.e., the angles of tilting, tilting, and rolling); moreover, it is aimed at a three-dimensional direction, which is represented by a dotted line (1305). In the present invention, the dotted line (1305) is referred to as the line of aiming. The fifth point is the menu bar and title bar (1301), or small devices (widegets; for example, scrolling bar, etc.; such small devices are not shown in the figure because of their wide variety. Fig.13A as well as Fig. 13B ), are also some three-dimensional objects (for example, the three-dimensional object is a thin plane, a plane in a slenderformation), and they also have their own normal vectors (for example, (1308)) in the three-dimensional coordinate system (1302) described above.

[0426] Putting the above properties together, the three-dimensional graphical user interface (207) of the present invention becomes like a three-dimensional manifold (or, 3D motif, a three-dimensional artistic theme); on this three-dimensional manifold, there are many silhouettes or objects, each of which represents a special plane, and the normal vectors, corner points, sizes, or depth values ​​of these planes are different from each other.

[0427] For example, a 3D motif provided by the 3D graphical user interface (207) of the present invention can form a 3D spiral; this is not easily achieved by the known technology in the past. It is worth noting that when the 3D cursor (209) is aimed at the window (1304), the 3D cursor (209) casts an intersection point P1304 (point of intersection) on the window (1304) through the aiming line (1309). When the above-mentioned 3D cursor (209) is pushed by the 3D navigator (202) of the present invention, the intersection point P 1304 will also move (for example, from P 1304 (x', y') moves to P" 1304 (x”, y”)).

[0428] Through such characteristics, P1304 It can be suitable as a cursor or indicator of the window (1304); when the indicator (1304) appears in the window (1304), the functions of ordinary two-dimensional windows, such as selecting several lines of text, drawing a circle, changing fonts, selecting icons, etc. in a WORD program, can be implemented in the three-dimensional graphical user interface (207) of the present invention. In other words, some libraries established for the previous two-dimensional graphical user interface, such as Adobe's Display PostScript, Apple's Quartz, etc., or other programs with functions similar to Display PostScript, these programs can be compatible with the three-dimensional graphical user interface (207) of the present invention.

[0429] This compatibility provides excellent convenience for a graphical user interface development engineer, mainly because a toolkit developed specifically for the three-dimensional graphical user interface of the present invention only requires very little modification (for example, setting the Z value in an XYZ coordinate to a constant value, such as 0, zero) to extend its usability to previous two-dimensional graphical user interfaces.

[0430] Note that there is an arrow-shaped two-dimensional cursor (1310) in the window (1304). As an auxiliary tool, this arrow-shaped two-dimensional cursor (1310) will always aim its tip at the dot cursor P. 1304 The arrow-shaped two-dimensional cursor (1310) is essentially a cursor of the same shape or function as before, that is, all it can do is allow the operator to more easily find the activated pixel area (that is, P 1304 (x', y')). However, the arrow-shaped two-dimensional cursor (1310) also has the inherent defects of all two-dimensional cursors.

[0431] like Figure 1A As explained above, a conventional two-dimensional cursor (111) cannot rotate the activated pixel region P(x, y) without reference to the position of other objects; similarly, the aforementioned two-dimensional cursor (1310) cannot rotate the pixel region P(x, y). 1304 (x', y') rotation. The three-dimensional graphical user interface (207) of the present invention can make up for the defects of the above-mentioned two-dimensional cursor through the basic functions of the three-dimensional cursor (209).

[0432] like Fig. 13B As shown, the tilt of the three-dimensional cursor (209) can be determined by the rotation plane abc 1304 To represent; we have learned from Section 3 that the rotation plane abc 1304The values ​​of a, b, and c in the parameter (1304) are determined by the attitude detection mechanism (203). Fig. 13B As shown, when we add the ability to detect the spinning of the mouse body (202), the entire set of cursor movement data of the three-dimensional cursor (209) can be expressed as (x, y, z, a, b, c, θ).

[0433] This means that the activated pixel region P(x, y) described above can be classified as a two-dimensional positioning data (x', y'), or a set of three-dimensional cursor movement data (x, y, z, a, b, c, θ), or any data derived from the above set of three-dimensional cursor movement data (e.g., first-order, second-order time differentials, etc.). In short, the activated pixel region P(x, y) described above is treated as an object rather than a mathematical point by the three-dimensional graphical user interface (207) of the present invention; this confirms the title of the present invention, that is, the three-dimensional graphical user interface (207) of the present invention is a three-dimensional graphical user interface with wide applicability. When we extend the above-mentioned properties of activating a pixel region P(x, y) to the entire plane (1304), this situation represents a special application of the three-dimensional graphical user interface (207) of the present invention, that is, activating a two-dimensional plane requires the three-dimensional graphical user interface (207) of the present invention to provide a non-linear motion vector to a cursor or object.

[0434] For a person skilled in the art, the sections disclosed in the present invention are used to illustrate but not limit the scope of use of the present invention. According to the technical content disclosed in the present invention and within the scope of the patent application, the method, procedure, material, structure, size, etc. of making and using a three-dimensional graphical user interface disclosed in the present invention can be improved and modified. The three-dimensional graphical user interface assigns linear and nonlinear motion vectors of three-dimensional objects with different degrees of freedom to its basic drawing units, such as pixels, voxels, or a whole system such as a robot that can move in a three-dimensional mode by color detection and shape detection of a colored or dyed reference surface.

[0435] However, the above is only an embodiment of the present invention and should not be used to limit the scope of the present invention. That is, all simple equivalent changes and modifications made according to the claims and description of the present invention are still within the scope of the present invention.

Claims

1. A three-dimensional graphical user interface system, characterized in that: The system at least comprises: A memory and at least one processor, the processor being coupled to the memory; a display device being provided on the system, and a three-dimensional graphical user interface being presented on the display device; The three-dimensional graphical user interface uses a widely used artificial intelligence function to process graphical data and object movement data; wherein the artificial intelligence function is a neural network capable of interpreting graphics, which performs a classification procedure based on the states of a plurality of combinations formed by a number of connected corner points / corner points; wherein the three-dimensional graphical user interface uses at least one of the connected corner point / corner point combinations to represent a three-dimensional vector graphic object; wherein at least one three-dimensional graphic vector embedded between the corner points / corner points is determined according to the three-dimensional absolute addresses of the corner points / corner points; wherein the three-dimensional graphical user interface is capable of moving at least one three-dimensional object in a three-dimensional space represented by the three-dimensional graphical user interface with more than three independent degrees of freedom, wherein the motion has the characteristics of a linear motion vector and / or a non-linear motion vector; The three-dimensional space can be used to locate the coordinate position by a 2.5D coordinate system, in which there is a rotation axis and an origin, wherein the angle between the X axis and the Z axis, or the angle between the X axis and the Y axis in the 2.5D coordinate system is used to define a stereoscopic viewing angle; The linear motion vector and the non-linear motion vector represent translation and rotational motion, respectively, and are generated from a single gesture motion of a three-dimensional navigator moving on a colored or dyed reference surface, wherein the three-dimensional navigator provides a plurality of absolute three-dimensional coordinate values ​​without using other motion detection devices, Wherein, the artificial intelligence function includes effectively reducing the at least one three-dimensional graphic vector or three-dimensional position vector from three dimensions to 2.5D by using a 2.5D coordinate system.

2. The system according to claim 1, characterized in that: After the object is moved by the three-dimensional navigator, the accuracy of the position of the object remains stable during the movement and is not affected by the cumulative error of positioning. The three-dimensional navigator measures the three-dimensional absolute address from the reference surface.

3. The system according to claim 1, characterized in that: The three-dimensional space includes a predetermined vector field, which can be combined with the linear motion vector and / or the nonlinear motion vector provided by the three-dimensional navigator to form a new motion vector of the object. The size and direction of the new motion vector are different from another motion vector of the object formed when the three-dimensional navigator is not used.

4. The system according to claim 1, characterized in that: The three-dimensional space represented by the three-dimensional graphical user interface includes multiple two-dimensional planes, each of which represents a special application or function, and at least one of the two-dimensional planes has its own 2.5D coordinate system.

5. The system according to claim 4, characterized in that: Activating or deactivating the two-dimensional plane requires providing non-linear motion vectors to objects or cursors, the at least one object being a cursor.

6. The system according to claim 1, characterized in that: The primary property of objects that can be accessed or touched by the system, be it cursors, graphical entities, or physical objects, is that they are represented by matrix data rather than by single values ​​defining mathematical points.

7. The system according to claim 1, characterized in that: The system also includes at least some independent software modules, which are stored in the memory of the system in part or in full in a layered architecture, some of which independently focus on stereogramming and / or use artificial intelligence functions to generate robot movements, and the software modules can operate separately or collaboratively.

8. The system according to claim 1, characterized in that: The three-dimensional graphical user interface uses an operating system loaded into the memory of the system, and enables a user to interact with objects displayed in the three-dimensional graphical user interface, or to interact with objects controlled by the system in an in-situ manner, through the linear motion vectors and the non-linear motion vectors. The linear motion vectors and the non-linear motion vectors can be assigned or generated on a graphic entity, and the graphic entity can be as small as a pixel, or as small as the voxel size of the smallest drawing unit of the graphical user interface.

9. A three-dimensional graphical user interface system, characterized in that: The system at least comprises: A memory and at least one processor, the processor being coupled to the memory; the system is capable of enhancing the user's experience of engaging and interacting with a three-dimensional scene, the three-dimensional scene being presented in a three-dimensional display device, the three-dimensional scene having a plurality of three-dimensional vector graphics; The system is capable of adjusting the stereoscopic viewing angle of a 2.5D coordinate system embedded in a three-dimensional graphics imaging device; The stereoscopic viewing angle is the angle between the X-axis and the Z-axis of the 2.5D coordinate system, or the angle between the X-axis and the Y-axis of the 2.5D coordinate system; The system can use artificial intelligence capabilities to allow the operator to visually classify multiple graphics vectors and / or some motion vectors into several different categories. wherein the artificial intelligence function is a neural network capable of interpreting graphics, which performs a classification procedure based on the states of multiple combinations formed by some connected corner points / corner points; wherein the three-dimensional graphical user interface uses at least one of the connected corner points / corner point combinations to represent a three-dimensional vector graphic object, The graphic vector is used to construct the three-dimensional vector graphic object, wherein each of the categories represents a combination of special feature graphics, the feature graphics are related to the three-dimensional vector graphics, and the feature graphics of each category are different from the feature graphics of other categories.

10. The system according to claim 9, characterized in that: When the system starts with a given world-space camera's three-dimensional position and then moves the world-space camera to a specific stereoscopic viewing angle with a series of translation and rotation motion vectors for a specific graphical polygonal network, the specific graphical polygonal network reaches visual classification optimization conditions.

11. The system according to claim 9, characterized in that: The system is capable of engaging and interacting with the user by manipulating the three-dimensional scene, wherein some of the graphical objects displayed in the three-dimensional scene are represented by some three-dimensional nodes, and the three-dimensional nodes have the characteristics of three-dimensional addresses and three-dimensional motion vectors, wherein each three-dimensional address has components of X, Y and Z, and at least one three-dimensional motion vector has components of three degrees of freedom of translational motion ΔX, ΔY and ΔZ, and / or three degrees of freedom of rotational motion α, β and γ; wherein the nodes are classified into several categories according to the interaction of the cursor with the nodes.

12. A method for user interaction with multiple objects, the method being executed on a computer, wherein the multiple objects are located in a three-dimensional graphical user interface in a display system, characterized in that: The method comprises at least the following steps: Providing a three-dimensional navigator, the three-dimensional navigator being capable of communicating with the three-dimensional graphical user interface, the three-dimensional navigator being movable on a reference surface, wherein the three-dimensional graphical user interface enables an operator to move at least one three-dimensional object in a three-dimensional space represented by the three-dimensional graphical user interface with a motion having at least three degrees of freedom, the motion having characteristics of a linear motion vector and / or a non-linear motion vector, the at least one three-dimensional object being represented by at least one three-dimensional graphic vector, wherein the at least one three-dimensional graphic vector has at least three degrees of freedom, and the at least one three-dimensional object having a volume in the three-dimensional space; Alternatively, a control device having linear motion vectors and / or nonlinear motion vector characteristics is used to communicate with the three-dimensional graphical user interface, the three-dimensional navigator or the control device moving along a two-dimensional vector lying on a surface having at least three colors or at least three different wavelength ranges; Converting the two-dimensional vector into a three-dimensional vector, or converting the two-dimensional vector into a vector with a dimension higher than two dimensions, wherein the conversion is performed on the brightness value of an area of ​​a surface dyed with three colors, and the image of the area is captured by a color image detector located in the three-dimensional navigator or the control device; and utilizing an artificial intelligence function to classify the three-dimensional graphic vector into a plurality of independent categories, wherein the artificial intelligence function is a neural network capable of interpreting graphics, which performs a classification procedure based on the states of multiple combinations formed by some connected corner points / corner points; wherein the three-dimensional graphical user interface uses at least one of the connected corner points / corner point combinations to represent a three-dimensional vector graphic object, Among them, the artificial intelligence function includes effectively reducing at least one three-dimensional graphic vector or three-dimensional position vector from three dimensions to 2.5D by using a 2.5D coordinate system.

13. A three-dimensional graphical user interface system, characterized in that: A robot connected to the system can be controlled, and the system at least comprises: A memory and at least one processor, the processor being coupled to the memory; a display device being provided on the system, and a three-dimensional graphical user interface being presented on the display device; The system has artificial intelligence capabilities; a navigator capable of controlling the position and movement of the robot in real time, Wherein, the position and movement of the robot have more than three degrees of freedom; wherein the position and movement of the components of the robot have more than three degrees of freedom, the movement of the components determining an action of the robot; the movement of the components is controlled by the system using the navigator in an in-situ manner; The three-dimensional graphical user interface is capable of monitoring more than three degrees of freedom of position and movement of the robot; wherein the movement of the robot can be modified by the three-dimensional graphical user interface (3D GUI) via the use of the navigator, wherein the artificial intelligence function is a neural network capable of interpreting graphics, which performs a classification procedure based on the states of multiple combinations formed by some connected corner points / corner points; wherein the three-dimensional graphical user interface uses at least one of the connected corner points / corner point combinations to represent a three-dimensional vector graphic object, Among them, the artificial intelligence function includes effectively reducing at least one three-dimensional graphic vector or three-dimensional position vector from three dimensions to 2.5D by using a 2.5D coordinate system.

14. The system according to claim 13, characterized in that: The robot is a structure having an arbitrary shape.

15. The system according to claim 14, characterized in that: The robot is a structure having joints.

16. The system according to claim 14, characterized in that: The structure with arbitrary shape is a cartoon character or other graphical structure.

17. The system according to claim 13, characterized in that: The robot is a movable device outside the three-dimensional graphical user interface.

18. The system according to claim 13, characterized in that: If the robot is invisible but still controllable by the user, the robot functions as a stereo camera, which sees the view presented by the three-dimensional graphical user interface from a specific position and / or stereoscopic perspective specified by the robot.

19. The system according to claim 13, characterized in that: If the system is configured such that the robot is invisible but still controllable by the user, the robot performs one of the following functions: (a) Execute the signing procedure; (b) execute the functions of the drawing program; (c) Execute the functions of the cartoon in the animation; (d) performing actions of characters in animated video games; (e) procedures for performing three-dimensional imaging diagnostics; The robot performing one of the above functions can be regarded as a finger, hand, wrist, elbow, arm, leg, head or torso of the operator's flesh.

20. A computer system with a three-dimensional graphical user interface, characterized in that: The computer system at least comprises: Display device; Memory; at least one processor, the at least one processor being coupled to the memory; the computer system further comprising one or more display servers; a three-dimensional graphical user interface, the three-dimensional graphical user interface being projected on the display device, the three-dimensional graphical user interface comprising at least a plurality of software modules in a layered structure, the software modules in the layered structure being stored in the memory of the computer system, and the software modules in the layered structure being capable of interacting with the processor and being executed by the processor; The software module includes at least one artificial intelligence module capable of interpreting graphics, the artificial intelligence module being capable of processing some data, the data being in the form of corner points or connecting lines of a graphical network; the at least one artificial intelligence module capable of interpreting graphics determines the relationship between connecting lines embedded between the corner points according to the three-dimensional absolute positions of the corner points and the graphic vectors associated with the corner points; the corner points of a part of the graphical network and a part of a plurality of graphic vectors of a three-dimensional coordinate system constitute characteristic graphics in the graphical network; Wherein, the three-dimensional graphical user interface can establish some areas on the display device, and the areas are called windows; in the windows, some graphical objects can be displayed in the three-dimensional coordinate system; wherein the graphical objects can be represented by some absolute positions located in the three-dimensional coordinate system, or, the graphical objects can be represented by some absolute positions located in a 2.5D coordinate system; The graphical object can move in accordance with the translational motion and rotational motion in the three-dimensional kinematics of the object, the total degree of freedom of the translational motion and the rotational motion of the three-dimensional motion of the graphical object is at least three, and the accuracy of the position of the graphical object will not be lost due to the movement; a three-dimensional navigator, wherein the three-dimensional navigator can use an application interface to interact with the three-dimensional graphical user interface through at least one protocol when operated, and the three-dimensional navigator can provide an absolute position in the three-dimensional coordinate system to the graphical object displayed in the three-dimensional graphical user interface; By executing the software module stored in the memory of the computer system, the graphical object is moved between the two absolute positions with at least three degrees of freedom in the display device; during the movement, the movement of the graphical object follows the translational movement and the rotational movement in kinematics, The artificial intelligence module includes effectively reducing at least one three-dimensional graphic vector or three-dimensional position vector from three dimensions to 2.5D by selectively using a 2.5D coordinate system.

21. The computer system according to claim 20, characterized in that: The software module contains some instructions that are readable and executable by the processor, and the instructions are used to generate some robot movements, or generate some graphical movements according to the laws of three-dimensional kinematics and stereography.

22. The computer system according to claim 20, characterized in that: The software module is operationally managed by the display server so that the functions of the software module are appropriately displayed in the window.

23. The computer system according to claim 20, characterized in that: The software modules include processing modules that independently control cursor movement, stereoscopic manipulation, robotics, and machine learning programs.

24. A method for simultaneously interacting with multiple three-dimensional objects in a three-dimensional graphical user interface, characterized in that: The method is executed on a computer, and is caused by a pre-set action of a three-dimensional navigator. The method is used to start a mechanism, and the mechanism is used to select or / and deselect an event state, and is characterized in that the pre-set action at least includes the following steps: When the user moves the three-dimensional navigator on the two-dimensional reference surface, by placing a finger on the upper surface of the three-dimensional navigator, the three-dimensional cursor located in the three-dimensional graphical user interface moves on the two-dimensional plane in a corresponding manner, and the tilt of the two-dimensional plane is controlled according to the position of the finger on the surface of the three-dimensional navigator; At the beginning of the action, the user presses a button on the three-dimensional navigator or presses the mechanism on the three-dimensional navigator to determine the position of the first endpoint, and the three-dimensional navigator is located on a two-dimensional reference surface; the button is continuously pressed or the mechanism is continuously kept in a pressed state until the cursor has moved a distance along the two-dimensional plane and reaches the second endpoint, using the first endpoint and the second endpoint as a diagonal line, and the three-dimensional graphical user interface (3D GUI) has artificial intelligence functions to establish a volume in three-dimensional space, and the volume enables the user to select or / and deselect the three-dimensional objects enclosed by the volume, Wherein, the artificial intelligence function is a neural network capable of interpreting graphics, which executes a classification procedure based on the states of multiple combinations formed by some mutually connected corner points / corner points; wherein, the three-dimensional graphical user interface uses at least one of the mutually connected corner points / corner point combinations to represent a three-dimensional vector graphic object.

25. The method according to claim 24, characterized in that: The surface of the navigator is provided with a detector, which can convert the two-dimensional movement and / or address of the finger on the surface into a three-dimensional movement and / or address signal.

26. A method for controlling and / or adjusting the motion of a robot in an in-situ manner, characterized in that: The method is executed on a computer and comprises at least the following steps: Multiply some matrices, where each matrix has a dimension of at least three; Using a mobile device capable of providing three-dimensional translation and rotational motion vectors simultaneously, so that the robot can change its position or motion in an instant manner; wherein the position and motion of the robot have more than three degrees of freedom, and the mobile device controls the position or motion of the robot in an in-situ manner; The translation and rotation motion vectors may be represented in a 3D format or a 2.5D format; the translation and rotation motion vectors are capable of changing the position / motion of a selected joint or a movable part of the robot; while other unselected joints or movable parts of the robot continue to undergo unchanged pre-set motions, wherein the selected joint or movable part is calculated by a three-dimensional graphical user interface with artificial intelligence capabilities, Wherein, the artificial intelligence function is a neural network capable of interpreting graphics, which executes a classification procedure based on the states of multiple combinations formed by some mutually connected corner points / corner points; wherein, the three-dimensional graphical user interface uses at least one of the mutually connected corner points / corner point combinations to represent a three-dimensional vector graphic object.

27. The method according to claim 26, characterized in that: The robot moves in a three-dimensional spiral motion.

28. The method according to claim 26, characterized in that: The robot's movement method at least includes forward kinematics and inverse kinematics methods, and at least one movement generated by the forward kinematics and / or inverse kinematics method is generated by the moving device.

29. The method according to claim 26, characterized in that: The method of robot movement can be used to drive the movement of interactive characters in animated video games, cartoon animations, medical images, or movies.

30. An image presentation method executed on a computer, characterized in that: The method is to create a digital impressionistic vector-graphic rendering effect on a computer, an electronic control system or an electromechanical system, and the method comprises at least the following steps: In a three-dimensional graphical user interface, at least one three-dimensional vector graphic object is generated by using an artificial intelligence function, wherein the three-dimensional vector graphic object has some basic properties, one of which is that the three-dimensional vector graphic object has volume; wherein the three-dimensional vector graphic object is constructed by a number of vertices connected to each other, wherein the vertices define the three-dimensional vector graphic object whose outer shape is formed by a number of small planes; In the memory of the computer, the electronic control system or the electromechanical system, a feature vector space is created, wherein the feature vector space is used to represent the three-dimensional graphical object, wherein each vertex is associated with a multi-dimensional feature vector space, and wherein each vertex has a normal vector perpendicular to a small plane corresponding to the vertex; The three-dimensional graphical user interface selectively uses an artificial intelligence function to manipulate the position of at least one vanishing point, and presents at least one three-dimensional graphic vector in a 2.5D coordinate system with reference to the at least one vanishing point. In the 2.5D coordinate system, there is a rotation axis. The 2.5D coordinate system is characterized in that: in the 2.5D coordinate system, the image presentation method can move the three-dimensional vector graphic object in the 2.5D coordinate system by translation or rotation in the 2.5D coordinate system by using the artificial intelligence function; classifying a plurality of three-dimensional graphic vectors into a number of independent categories; In the memory of the computer, the electronic control system or the electromechanical system, the three-dimensional graphical user interface uses the artificial intelligence function to separate some of the multi-dimensional feature vectors using some clear boundaries between each other; The normal vectors of at least two of the facets are aligned in two different directions so that the vertices and the facets defined by the vertices are categorized in a manner that is visible and recognizable to a viewer, Among them, the artificial intelligence function includes effectively reducing at least one three-dimensional graphic vector or three-dimensional position vector from three dimensions to 2.5D by using a 2.5D coordinate system.

31. The method according to claim 30, characterized in that: The method uses a three-dimensional navigator to move in a controllable manner on a colored or dyed two-dimensional reference plane so that it contacts a selected vertex of the three-dimensional vector graphic object. At the same time, by touching the surface elements of the three-dimensional navigator, some lighting intensities of the lighting system inside the three-dimensional navigator are changed, so that the direction of the normal vector of the contacted vertex changes, and the visual characteristics of the three-dimensional vector graphic object near the vertex are also changed; wherein, the operation of changing the normal vector of the contacted vertex can enhance the three-dimensional visual effect of the three-dimensional vector graphic object.

32. A three-dimensional image presentation method executed on a computer, characterized in that: The method comprises at least the following steps: Rendering a number of three-dimensional graphic vectors with reference to at least one vanishing point; The position of one of the vanishing points can be manipulated by the function of artificial intelligence; Grouping some of the three-dimensional graphic vectors rendered by the method, wherein each group forms a boundary with other groups, and the boundary is recognizable by a user; When the limit reaches different values, the method produces some image presentation effects, or supports some different levels of interaction between the user and the method. Among them, the function of the artificial intelligence is a neural network capable of interpreting graphics, which executes a classification procedure based on the states of multiple combinations formed by some mutually connected corner points / corner points; wherein, the three-dimensional graphical user interface uses at least one of the mutually connected corner points / corner point combinations to represent a three-dimensional vector graphic object.

33. The method according to claim 32, characterized in that: The technology adopted by the artificial intelligence function at least includes support vector machine or neural network.

34. The method according to claim 32, characterized in that: The technology used by the functions of the artificial intelligence at least includes artificial methods that can be controlled by the user.

35. A non-transitory computer-readable medium, characterized in that: The medium has some instructions, and when the instructions are executed by the processor, the processor presents a three-dimensional graphical user interface (3D GUI) on the display device, and the three-dimensional graphical user interface can widely provide at least one artificial intelligence function; wherein at least one of the artificial intelligence functions is a neural network capable of interpreting graphics, and the neural network capable of interpreting graphics performs a classification procedure based on the states of multiple combinations formed by some connected corner points / corner points; wherein the three-dimensional graphical user interface uses at least one of the connected corner points / corner points to represent a three-dimensional vector graphic object; and at the same time, the three-dimensional graphical user interface performs multiple operations, and the operations at least include: The three-dimensional vector graphic object is moved in a three-dimensional space by using the three-dimensional graphical user interface with a movement having more than three independent degrees of freedom, wherein the movement having more than three independent degrees of freedom is characterized in that: the movement is an instantaneous movement, and the instantaneous movement has linear, nonlinear, or both characteristics; wherein the linear motion vector and / or the nonlinear motion vector represent a translation motion vector and a rotation motion vector, respectively, and the linear motion vector and / or the nonlinear motion vector are generated by a single posture movement of a navigator located on a reference surface, and when generating the linear motion vector and / or the nonlinear motion vector, the navigator does not need to apply other inputs or other motion detection devices, Among them, the artificial intelligence function includes effectively reducing at least one three-dimensional graphic vector or three-dimensional position vector from three dimensions to 2.5D by using a 2.5D coordinate system.

Images