Hand control creation system and method

Abstract

Through the virtual and physical dimensions are connected in a method of actual double spacetime structure, also considered as eight-dimensional, thereby establishing a space where the virtual and the real are contrapuntal, making virtual object observable from multiple perspectives while creating, drawing or writing. An advanced real-time interactive system for virtual producing or smart manufacturing based on users. Cursor is able to produce and create virtual 3D objects by only hands, for usage including but not limited to medical physiological rehabilitation therapy, development of medical-related assistive technologies, education, 8D demonstrating, industrial manufacturing, virtual production, industry 5.0 development, high-dimension research, design, architecting, entertainment and so on. The system is customizable depending on users. Colour, cursor and trails are selective, customizable reset, reference, colour panel, file output and on / off functions.

Description

Hand Control Creation System and Method

[0001] This disclosure generally relates to developing fields including Cybernetic Control, Extended Reality technologies, such as Human Computer Interaction(HCI), Holographic Technology and so on; it particularly relates to an undeveloped field which I called High-dimensional Hand Control Systems.

[0002] The control method with control tools, such as mouse and keyboard devices show low efficiency in production or designing at the moment. For example, film production progress is changing due to new high-technologies. Forms a way of film making such as virtual production, which is depended on Extended Reality technologies. In this situation, a lack of advanced technology impedes virtual production’s efficiency.

[0003] Various aspects of the present disclosure relate to devices, systems, and methods for display and controlling. To individuals in related technology fields, the system is an advanced controlling system, a producing method, an interacting method, an interacting holography, a virtual sculpting method, a virtual shaping method, a human-based smart manufacturing method, a multi-spacetime controlling system, a vision method, obviously.

[0004] One example embodiment provides an Extended Reality producing method, a system for manufacture, user produce with interactive control; system directly forms in virtual space, which shapes by mapping and computing, mapping object is one or more persons, more persons include two-person, three-person, four-person, five-person, six-person and so on; for industrial manufacturing, system transmit virtual forms into signals for physical manufacturing process, which represents a second procedure, including but not limited to, guiding next step or steps sequence; for virtual production, system transmit virtual forms into signals for next step or next steps producing, including but not limited to, guiding virtual sequence, physical sequence or both sequences, such as additive manufacturing, virtual production and so on.

[0005] Another example embodiment provides an interactive hand tracking control method, by hand positions, hand points in three-dimensional space, user or users control by interacting with system, create input for mapping by left hand and right hand, finger position, gesture signs; feedback system for two hands in loop; output of hand tacks; further function settings, included but not limited to colour expressions, brush expressions.

[0006] Another example embodiment provides a real-time interactive three-dimensional display method, in this method, one person is one unit, persons are multiple units, the first person involved in specific field is prime unit, prime unit is dominating user in this method, user interact by real-time face tracking, this method provides limited multi-perspective observation.

[0007] Another example embodiment provides a multiple spacetime producing method, a multi-dimensional producing method, a multiple spacetime system, multiple spacetime is recognized as two four-dimensional performances and the connections between them, one is virtual dimension performance of spacetime, another is physical performance of spacetime, connections between the performances develops interactivity.

[0008] Another example embodiment provides a four-dimension(4D) application method on terminals and devices, in traditional applications show through display monitors, mostly perform in two-dimension or three-dimension(3D) as to space and time, this method is four-dimensional performance in applications, which still react in real-time.

[0009] An aspect of this disclosure is development of medical-related assistive technologies applied to individuals with disabilities or physiological conditions, patients recovering from injuries or surgeries can engage in gamified rehabilitation exercises that track progress and motivate recovery, included but not limited to musculoskeletar rehabilitation, rehabilitation of neurological disorders, increase patients’ motivation and interest during therapy, improvement of patients’ self-perceived effectiveness, which is an ideal sensorimotor control training; this aspect is an interactive virtual environment, which leads a change of attention in the patient.

[0010] Another example embodiment provides a technical method for technological products applied to physiological rehabilitation therapy, this product promotes a healthy and efficiently functioning neuromotor system for human body. This method operates without emitting radiation; this is a technical method for 3D printing products such as prosthetic manufacturing or custom prosthetic design.

[0011] Another example embodiment provides a technical method for assistive learning devices and products applied in education, included but not limited to real-time interactive control device for spatial drawing or spatial writing; a virtual, interactive and holographic interface; a real-time multi-dimensional display system; a highly detailed three-dimensional and observable display method.

[0012] Another aspect of this disclosure is a technical medium for architecture, three-dimensional printing and design, included but not limited to sculpting, shaping, colouring, modelling or any digital creation method for designing in related creative activities, included but not limited to architecture, design, art creating, media producing; a output method after real-time rendering, included but not limited to point cloud export, four-dimensional(spacetime) video export, 3D object export, image export.

[0013] Another aspect of this disclosure is in entertainment, an eight-dimension(8D) theoretical system in demonstrating activities, such as exhibitions, virtual exhibition halls, mall events, cultural tourism scenes and so on, eight-dimension theory is accorded to context of physics and philosophy; in the field of demonstration, user or users feel advanced, fantastic and extraordinary, this field develops a revolutionary eight-dimensional experience.

[0014] Advantageously, neither do users have to draw or sculpt with mouse and keyboard or similar devices while producing, nor do users have to create in restricted environment, due to complex controls and different perspectives in necessary production and crafting needs. This disclosure provides concepts to industrial 5.0 development method.

[0015] Further benefits and advantages of the present invention will become apparent after a careful reading of the detailed description with appropriate reference to the accompanying drawings.

[0016] There is no perfect holographic technology, which affects producing methods. Present smart manufacturing process is attached to Industry 4.0, which has defects in process part of human controlling; For users, Virtual Reality(VR), Augmented Reality(AR) and Mixed Reality( MR) devices has disadvantages of display and rely on control devices. Present Holographic Reality (HR) rely on display methods which has reached physical or visual limitations, such as traditional holograms, digital hologram display and so on; Extended Reality(XR) is a developing technology in production, medical-related assistive technologies applied to individuals with disabilities or physiological conditions and products for physiological rehabilitation therapy mostly use VR, AR or MR, which has physical or visual limitations; Traditional production process in low dimensional and low efficiently environment, for example, using mouse and keyboards to control and a screen as display monitor or as two-dimensional user interface(UI).

[0017] This invention is a human-based smart manufacturing process in Industry 5.0, it is considered as to initial smart manufacturing systems to Industry 4.0; This system is an advance simulation technology for interactive holography, also known as HR, this is a new method for digital holographic display; This invention is an XR technical use for development of medical-related assistive technologies and physiological rehabilitation therapy, which has less limitations but more convenience both physically and visually; This invention is a better producing method in Virtual Production, 3D Printing, Design, Architecture and so on; This method and system using in production is in higher dimensional and higher efficiently environment, control interactively with face, right and left hands through a three-dimensional( spatial) UI, which is operative.

[0018] This invention is able to be extensively implemented, the setup is at low cost. The invention has positive effects on high-dimensional research, human-based digitalization, Extended Reality, Holographic Reality and interaction methods. It may be referenced to Industry 5.0 development or human-based smart manufacturing development. It affects exploring new artistic directions or critiques after Post-modern art, especially as a technical medium. It makes less visual limitations in medical practice than with VR, AR and MR. Like other XR technologies, it promotes a healthy and efficiently functioning neuromotor system, which in its turn favors partial injury prevention.

[0019] Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings.Fig.1

[0020] is a perspective view of a system installation in accordance with one embodiment of the present invention.Fig.2

[0021] is a flow diagram illustrating main part of system for describing the system of.Fig.3

[0022] is an extended illustrating view of partial system as shown in, according to one embodiment.Fig.4

[0023] is a flow diagram illustrating second part of system for describing the system of, according to another embodiment.Fig.5

[0024] is a flow diagram illustrating assistive part system for the system of, in accordance with some embodiments.Fig.6

[0025] is an illustration of final part system for describing the system of.Fig.7

[0026] is an illustration of personalized custom system for the system of, in accordance with some embodiments.]Fig.8

[0027] is a perspective view of system environment structure method of the present invention.

[0028] Embodiments are described herein with reference to the figures where like reference numbers indicate identical or functionally similar elements. Also, in the figures, the left most digits of each reference number correspond to the figure in which the reference number is first used.

[0029] Modules can be processed on physical or virtual components depending on the installation, in preconditions of no obstructs to module, processor, or connection functions. The described operations and their associated modules may be employed in softwares, programs, firmwares, hardwares or any combinations thereof. All connections in description are one-directional.

[0030] In the following description, various aspects of the present invention will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will also be apparent to one skilled in the art that the present invention may be practiced without the specific details presented herein. Furthermore, well known features may be omitted or simplified in order not to obscure the present invention.

[0031] In some embodiments, hand cognition 204, hand cognition 301 and hand cognition 401 are the same module; cursor and trail three-dimensional visualization processor 212, cursor and trail three-dimensional visualization processor 305, cursor and trail three-dimensional visualization processor 408 and cursor and trail three-dimensional visualization processor 715 are the same module; cursor select 219, cursor select 308 cursor select 412 and cursor select 712 are the same module; render one 218, render one 309 and render one 601 are the same module ;chart diagram 307 is the same module as chart diagram 211; position two 207 and position two 501 are the same module; render two 217 and render two 602 are the same module; render three 216 and render three 603 are the same module; three-dimensional cursor select integration system 414 and three-dimensional cursor select integration system 604 are the same module; colour panel visualization system 411 and colour panel visualization system 605 are the same module; visual reset warning 504 and visual reset warning 606 are the same module; reference object 214 and reference object 706 are the same module; colour panel setting 409 and colour panel setting 714 are the same module; sensor input 201 and sensor input 719 are the same module.

[0032] is a perspective view of a system 100 installation in accordance with one embodiment of the present invention. As shown, system 100 is set by a combination of a computer 103, a flat display monitor or method 102, a user 104, a sensor 101. When sensor 101 condition is on, there is a main user 104 but multiple users can participate at the same time according to some embodiments. In one embodiment, sensor 101 scan user 104 and user’s face 107, one hand 105 and another hand 106 in preset settings. According to some embodiments, flat display monitor or method 102 installation is including but not limited to a screen monitor, a projector device, a 4D display screen, a hologram display screen and so on.

[0033] is a flow diagram 200 illustrating main part of system for describing the system 100 of. This diagram 200 is including a vision process and a hand process 202.

[0034] Sensor input 201 separately inputs face and hand data into face cognition 203 and hand cognition 204 modules. Inside hand process 202, hand cognition 204 transfers position performance into hand position 206 processing module. In hand position 206 module, position two 207 is for erasing already created objects and resetting chart diagram 211, this procedure is prior case of procedure, position one 209 and position three 210 are subsequent case of procedure. Position one 209 produces real colour data into diagram 211; position three 210 produces empty colour into diagram 211. Diagram 211 converts data into cursor and trail three-dimensional visualization processor 212 for next steps’ processing. Spatial position 205 is a spatial position data process, specifics show in. Cursor select 219 inputs cursor details into cursor and trail three-dimensional visualization processor 212 and converts into cursor data for chart diagram 211, which converts feedback into cursor and trail three-dimensional visualization processor 212. Face cognition 203 converts and transfers data into face data processor 213, which separately transfers processed face data into reference object 214, camera 215 and cursor and trail three-dimensional visualization processor 212. Reference object 214 transfers performance to render three 216. Cursor and trail three-dimensional visualization processor 212 converts data into visualized performance of trail into render two 217 and cursor into render one 218. Camera 215 adjust display angles of render three 216, render two 217 and render one 218.

[0035] is an extended illustrating view 300 of partial system as shown in, according to one embodiment, which extension is combination of spatial position 205 process and parts of vision process 200. Cursor and trail visualization process 302 is part of vision process 200. Connection 205 is the same procedure of modules and connections from hand cognition 301 to cursor and trail three-dimensional visualization processor 305 as shown in.

[0036] In this embodiment, hand and cognition 301 analyses hand 303 module, which selects spatial data into space position 304. Space position 304 transfer data into cursor and trail three-dimensional visualization processor 305. Colour select 306 and cursor select 308 separately input colour data and cursor data into cursor and trail three-dimensional visualization processor 305, cursor and trail three-dimensional visualization processor 305 separately converts visualized cursor into render one 309 and cursor data into chart diagram 307.

[0037] is a flow diagram 400 illustrating second part of system for describing the system of, according to another two embodiments. Hand 402 in these embodiments can be exchanged with hand 303 in previous embodiment of, at the same time, hand position 405 can be exchanged with hand position 206 of previous embodiment in, which both depending on the user’s control habits and considering.

[0038] In these embodiments, hand cognition 401 analyses and output data to hand 402 module, which separates and input spatial data into space position 403, and transfers position performance into hand position 405. Space position 403 inputs spatial data into two-dimensional visualization processor 404.

[0039] In one embodiment, if position one 406, spatial data is converted through two-dimensional visualization processor 404, hand position 405 controls colour cursor process. Colour panel setting 409 transfers colour data to colour panel visualization system 411 module through colour integration settings 410 processor; spatial data transfer from two-dimensional visualization processor 404 into colour integration settings 410, at the same time, colour integration settings 410 gives colour and colour cursor feedback to two-dimensional visualization processor 404 for further procedure. Two-dimensional visualization processor 404 transfers colour data into cursor select visualization processor 413 and cursor and trail three-dimensional visualization processor 408.

[0040] In another embodiment, if position two 407, hand position 405 controls two-dimensional visualization processor 404. At the same time, two-dimensional visualization processor 404 outputs spatial data into cursor select visualization processor 413 module after spatial and angular calculation process 415, two-dimensional visualization processor 404 outputs control data into cursor select 412 module for cursor selection, which further transfers into cursor select visualization processor 413. Cursor select visualization processor 413 outputs integrated data to three-dimensional cursor select integration system 414.

[0041] is a flow diagram 500 illustrating assistive part system for the system 100 of, in accordance with some embodiments. This embodiment is a reset process, position two 501 control calculate 502 module to setup presets for visual reset warning 504, at the same time, calculate 502 module setup presets for animation 503 input. Animation 503 inputs animation file into visual reset warning 504.

[0042] is an illustration of final part system 600 for describing the system 100 of. Basic render process 616 and additional render process 617 composites with a layer organization process 607 into display output 608 module.

[0043] is an illustration of personalized custom system 700 for the system 100 of, in accordance to some embodiments. One embodiment is a customizable process 701 for reference use, reference object input 705 module inputs customized object reference object 706, object is including but not limited to 3D objects, trails and 3D cursors.

[0044] Another embodiment is a customizable process 702 for visualization change of cursors and drawings. There are three preset object options in cursor options 707, object one 708 is a smooth ball; object two 709 is a rough star, object three 710 is a flat splash. Additionally, more cursors can be added to cursor options 707. Through three-dimensional cursor input 711 transfers cursor object into cursor select 712 module.

[0045] Another embodiment is a customizable process 703 for changing colour panels. Colour panel input 713 module inputs different preset colour panels into colour panel setting 714 module.

[0046] Another embodiment is a customizable process 704 for object output which is created during user 104 participation in the system 100. Cursor and trail three-dimensional visualization processor 715 sends object to file output 716 for output, this module can output files into including but not limited to TIF, TIFF, JPEG, JPG, BMP, EXR, PNG, DDS, MOV, MP4, H265, FBX formats.

[0047] Another embodiment is a sensor on / off customizable process 717. on / off button 718 controls sensor to be on or off by sending on or off commands to sensor input 719, the button is customizable depending on user’s habits and considering.

[0048] is a perspective aspect 800 of system environment structure method of the present invention. In this aspect, system 100 is a double spacetime structure based on continuous loops and real-time rendering, which is integrated by a virtual spacetime 801, a physical spacetime 802 and a portal 803. In this method 800, the portal 803 usually is set with a flat display monitor or method 102, eye 805 and hand 806 of the user 104 are physical performance with virtual object 804 in a loop of feedback and subjective inputs.

[0049] It is appreciated that although descriptions illustrate a number of steps according to at least one embodiment, the precise steps and / or order of steps may vary in different embodiments.

[0050] The description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure. First, the particular naming of the components and variables, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Also, the particular division of functionality between the various system components described herein is merely for purposes of example and is not mandatory.

[0051] Although the invention has been explained in relation to its preferred embodiment(s) as mentioned above, it is to be understood that many other possible modifications and variations can be made without departing from the scope of the present invention. It is, therefore, contemplated that the appended claim or claims will cover such modifications and variations that fall within the true scope of the invention.

[0052] This invention is a human-based smart manufacturing process in Industry 5.0, it is considered as to initial smart manufacturing systems to Industry 4.0. It may be referenced to Industry 5.0 development or human-based smart manufacturing development. It is a humanized producing control method using right and left hands instead of traditional controls using mouse and keyboards. A multiple perspective observable method in relevant to virtual manufacturing operations( Range: H= 75°, V= 65°; Field= 0.5~ 5.5m).

Claims

A system, comprising:a face system, able to create object from multiple perspectives in a range according to sensor installation;a display monitor or method, display monitor or method comprises for the one or more displays of the display monitor or method, forming a portal between virtual and physical dimensions;two hand systems, one hand process is for main visualization of creating object, another hand process is for changing visual settings or performance of the object;a colour panel system, to adjust colour inputs into object and change cursor colour visually, controlled by one hand;a cursor and drawing visualization system, wherein a main interact process to visualize creation of virtual objects through portal according to the display monitor or method;a cursor selection system, wherein an optional process to change performance of the cursor in at least three forms by another hand, which is operated visually;modules, all modules are functional for processing data all particular steps while running the system; andone or more input sensors, sensor or sensors are set in the system for receiving input data from the user, the input data comprising event data associated with actions taken by the user within the real-time environment.The system of claim 1, wherein the system creating an environment of integrated virtual and physical dimensions, simulating into one same dimension by spacetime.The system of claim 1, wherein the face system further comprising:a multiple perspective observation method to the virtual object created by user, available in both cases of close and wide observes.The system of claim 1, wherein the display monitor or method, the portal provides an indication of the detected one or more hand status or actions and one or more face status or actions with the one or more interactable elements.The system of claim 4, wherein the display monitor or method further comprising:a user interface, comprised by one or more interactable elements or layers, the elements or layers are combined by rendered performances, colour panel, cursor option panel and reset animation.The system of claim 1, wherein two hand systems, further comprising:one hand process controls cursor performance, trails performance, and reset performance;another hand process controls colour performance, colour panel performance, cursor select performance; andcustomizable of switching hands for controlling, depending on user’s considering.The system of claim 1, wherein the modules further comprising:computing units, configured to generate a real-time environment for a user or users, the environment including a plurality of control input data to be achieved by the user or users;processing units, configured to determine one or more intervention inputs in accordance with one or more triggering events based on the input data; andmodular customization system, with modular customization system makes the system customizable to users for more comfortable creation or production.The system of claim 7, wherein the modular customization system is a combination of several functional process comprising:customize reference input process;customize visualization change of cursor and drawings process;customize colour panel process;customize output file, one or more file and formats process; andcustomize on / off control button process.A system connections method, comprising:connections, to ensure all functional modules are connected in a specific way to make the system of claim 1 running in a double spacetime environment;a hierarchical connection method, prior and subsequent connections are able to function in orders; andbody connections, body connections integrate user or users into the system, human organs participate in multi-dimensional interactive loops.The system connections method of claim 9, wherein connections ensure functions further comprising:indicating, directions of data transfer are indicated in the system of claim 1.A double spacetime method, comprising:a double spacetime structure, this structure setups an eight-dimensional environment with some operations; andelements, participating to sustain the environment of double spacetime structure, makes interactions to the system of claim 1 possible.

Images