Human-computer interaction method and system based on three-dimensional coordinate measurement

Abstract

The present invention provides a solution of a human-computer interaction method and system based on three-dimensional coordinate measurement, and a plurality of solutions for applying same to solve technical problems. An operation body and / or an operator performs input and operation interaction by means of a virtual keyboard and a virtual mouse, or integrates a display picture space and carries out various participatory and interactive direct operations and experiences on object components and objects in virtual and real scenes. Virtual handwriting input, body movement posture recognition, and lip recognition interaction methods are further provided, and virtual mode control and interaction methods such as gesture recognition and virtual touch screen interaction are supported. Contact is not needed and there is no need to touch the screen, and mobility is achieved. The present invention is suitable for the two-dimensional conventional computing field, and particularly suitable for three-dimensional space computing fields (comprising VR, MR, AR, meta-universe, virtual simulation, software games, three-dimensional graphics, three-dimensional animation, and other three-dimensional scenes). Human-computer interaction operations in work and life of people can be greatly facilitated and optimized.

Description

A human-computer interaction method and system based on three-dimensional coordinate measurement Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method and system based on three-dimensional coordinate measurement. Background Art

[0002] Humans are intelligent creatures. The development of science and technology has produced numerous technologies and tools that enable, replace, and assist research, production, and daily life. The most notable include computer technology, artificial intelligence, biotechnology, and other technologies. The hardware and software, and the control and use of machines and equipment supported by these technologies and methods, constitute human-computer interaction, which is relevant to the present invention.

[0003] The most well-known and widely used human-computer interaction methods are the keyboard and mouse, which fully support the comprehensive, accurate, convenient, and arbitrary combination of input and operation of letters, numbers, symbols, commands, and cursors. Touchscreen control follows, mimicking the keyboard and mouse. Others include handwriting input, voice recognition, gesture recognition, and eye movement interaction. However, since these devices either cannot fully control the machine's commands or precisely control screen points like a keyboard and mouse, they are mostly single-functional, with shortcomings such as poor controllability, low accuracy, the need for switching, and sometimes the need to touch the screen. Therefore, they are usually used as supplements to the keyboard and mouse, or as simple replacements in certain situations and under specific conditions, and cannot completely replace the widespread use of the keyboard and mouse.

[0004] Compared to alphanumeric and cursor-based commands, data, and point-based interactions, another type of control interaction, action-based mapping and execution, requires motion capture. Existing methods include mechanical and inertial sensing, with newer sensor-attached methods emerging. However, current implementations exhibit issues such as inconsistency with real-world actions, lack of intuitiveness, and inconvenience. Alternatively, the need for attachment to the body can hinder operation and create a stiff, unnatural feel. These factors hinder the optimal state of device control, which is natural, intuitive, convenient, and satisfying.

[0005] Existing technologies also have virtual simulated touch screen interaction and gesture recognition, which do not require touching the physical display screen. However, they are limited by insufficient technical solutions, or the interactive command content is single, limited, and lacks precision. Some system configurations are relatively complex and large in size, and therefore have not yet been widely accepted and used.

[0006] The latest wearable 3D spatial computing devices, VR / MR / AR, achieve a fusion of virtual and real space, providing immersive 3D panoramic displays. Wearable spatial computing can be controlled through a combination of eye movements, gestures, and voice, as well as interactive interfaces such as flat interface identifiers, menus, or touchscreens within the display. However, the current results are not always satisfactory, and these technologies are not yet mature, remaining in the ongoing development, refinement, and even exploration stages.

[0007] Summary of the Invention

[0008] The human-computer interaction method of the present invention is based on the measurement of three-dimensional coordinates, the acquisition of three-dimensional spatial motion coordinates, and further related parameters such as trajectory, posture, and motion degree. It provides a method for simulating the construction of a model in a real physical space and performing actions on it, replacing the functions of the original physical keyboard and mouse with a virtual keyboard and mouse, and fully meeting the requirements of comprehensive, accurate, convenient, and arbitrary combination of input and operation of letters, numbers, symbol commands, and cursors.

[0009] It also provides various direct operations and experiences that are integrated into the display screen space scene for participation and interaction. It is to directly operate the operable objects and components in the virtual and real scene, or to map them to the execution device or to the character for execution, just like the operation and interaction of the rules, responses and phenomena in the real physical space.

[0010] And provide other ways of interaction, including: virtual handwriting input, body movement posture recognition, lip recognition interaction; support virtual touch screen, gesture recognition, eye movement interaction, and other related interaction methods, which can be selected for use in combination.

[0011] Input and operation are achieved through various virtual mode interaction methods of the present invention, without touching the screen or contact, and with the feature of mobility;

[0012] The main body of the interactive operation of the present invention is still to use the hands of humans, which is consistent with the normal natural behavior of humans.

[0013] The human-computer interaction of the present invention is based on three-dimensional coordinate measurement. The screen scene and the operated object can be in two-dimensional or three-dimensional form. Therefore, it is not only applicable to traditional two-dimensional computing fields such as desktop computer operation, but is also particularly suitable for the three-dimensional spatial characteristics of three-dimensional space computing types. It can fully and consistently support the human-computer interaction needs of various scenarios that integrate the screen space and merge virtual and real scenes.

[0014] The three-dimensional spatial characteristics of the present invention are particularly suitable for wearable and non-wearable three-dimensional spatial computing scenarios, including virtual reality, mixed reality, augmented reality, metaverse, virtual simulation, software games, three-dimensional graphics, three-dimensional animation, and other three-dimensional scene scenarios;

[0015] The present invention provides the elemental features of the spatial motion coordinate measurement of human-computer interaction, and further acquisition of related parameters such as trajectory, posture, and motion degree, and the characteristics of the virtual interaction mode, which do not require contact and have the characteristics of mobility within the measurement range. It not only provides the function of completely replacing the physical keyboard and mouse, but is also particularly suitable for flexible screens (rollable mobile terminals), minimally invasive surgery, etc., which are inconvenient to touch, contact, or have mobility requirements. In particular, it is consistent with the motion capture feature requirements of motion operation mapping execution applications and can directly call the mapping of parameters, thereby having more direct, consistent, intuitive, and natural performance, and is particularly suitable for control interaction in various motion operation mapping execution application scenarios;

[0016] In addition, the present invention also provides or supports a variety of other virtual interaction methods, which are rich, convenient, easy to implement, and can be used in combination. They can be used to coordinate, supplement, or simply replace certain specific conditions and occasions.

[0017] The present invention defines and provides a new paradigm for human-computer interaction. The application and implementation of the technology of the present invention can greatly facilitate and optimize human-computer interaction operations in people's work and life, and promote the application of advanced technologies.

[0018] The present invention includes the following parts:

[0019] 1. A human-computer interaction method based on three-dimensional coordinate measurement,

[0020] include:

[0021] (1) Virtual keyboard, virtual mouse, input, operation;

[0022] (2) integration into display space operations;

[0023] (3)Other methods.

[0024] The virtual keyboard and virtual mouse input and operation described in 1.(1) above include the following steps:

[0025] The virtual keyboard and virtual mouse are modeled by the computing device, set by the operator, configured as needed, and operated by the operating subject to provide and implement keyboard input and operation in any combination of letters, numbers, and symbols, as well as mouse operation functions;

[0026] Build:

[0027] S1. The operating area is defined as the area within the measurement range of the three-dimensional coordinate measuring device, in front of the operator, and in the surrounding space. Within this area, typically in front of the operator, a certain area is appropriately designated as the modeling operating area. The computing device typically creates a virtual model below this area to provide operational input, and the display is displayed using appropriately adjustable display parameters.

[0028] S2. For traditional two-dimensional computing types, where the display is two-dimensional (typically a flat surface, including curved surfaces), a virtual keyboard model is constructed within the modeling operation area, and the style is displayed simultaneously in the foreground and lower portion of the image scene displayed by the computing processing device, i.e., the lower portion of the display screen, with appropriate and adjustable conversion parameters. The corresponding displayed keyboard is mapped to each key of the model;

[0029] S3. For three-dimensional computing types, the display is a three-dimensional stereoscopic situation. For wearables, a panoramic display with a fusion of depth, virtuality and reality is displayed. A virtual keyboard model is constructed in the modeling operation area, and the model is appropriately positioned in front of the operator and displayed as the main body.

[0030] S4. For three-dimensional computing, the display is three-dimensional. For non-wearable devices, based on the limitations of display technology, if the full-view depth of field is a fusion of virtual and real, a virtual keyboard model is constructed within the modeling operation area as described in S3, and the model is positioned appropriately in front of the operator and displayed as a main body.

[0031] Alternatively, if the display is not a full-view depth-of-field virtual-real fusion display, a virtual keyboard model style may be constructed as described in S2 and simultaneously displayed in the foreground, lower part, or other required and appropriate position of the scene displayed in the three-dimensional space with appropriate and adjustable conversion parameters, and the corresponding displayed keyboard and the model keys are mapped to each other;

[0032] S5. In the modeling operation area described in S2, S3, and S4, the area beside the virtual keyboard model, such as the right or left, or up or down, is intended to be set as a virtual mouse operation area. The operator can be set to automatically enter the mouse operation area when the operator leaves the virtual keyboard model within a certain distance, or the entire modeling operation area can be set to a single virtual mouse function interaction operation state;

[0033] Select Settings to implement mouse operation function by gesture mouse recognition;

[0034] The second option (depending on the needs, not recommended) is to build a virtual mouse model to provide operations. For the S2 display situation (applicable to the S2 mode of S4), the displayed mouse and the model are also mapped and converted.

[0035] Hereinafter, when describing the display position and mapping conversion relationship of the virtual model throughout the present invention, for the S4 three-dimensional spatial computing type non-wearable type, based on the limitations of display technology, since they correspond to the same methods, the description of the S3 three-dimensional spatial computing type wearable type in the preceding paragraph, which falls under the panoramic depth of field virtual-reality fusion scenario, shall be the same as that of the S3 three-dimensional spatial computing type wearable type, and the description of the non-panoramic depth of field virtual-reality fusion scenario in the latter paragraph, which falls under the S2 two-dimensional traditional computing type, shall be the same as that of the S4 three-dimensional spatial computing type non-wearable type.

[0036] S6. Regarding the operator, in model mode, within the three-dimensional coordinate measurement range and the modeling operation area within the operation area, the operator performs corresponding mapping through three-dimensional coordinate measurement and displays the corresponding position in real time, marking the principal point (keystroke point, which can be multiple points) and / or the skeletal model. Depending on the situation and needs, the operator's display may be a video image, a virtual animation or image-based appearance, or a three-dimensional reconstruction. The simplest method is to display the outline of the operator or the principal point's related branches, marking the principal point, and / or displaying the skeletal model.

[0037] For gesture mice, only the display cursor needs to be recognized;

[0038] S7. In model mode, the operator enters the modeling operation area, positions himself / herself at an appropriate distance from the model, and then makes a ready gesture. The keyboard model automatically aligns with the operator at its first operating axis (at the wrists), assuming an initial posture ready for operation.

[0039] The virtual mouse model, after the operator enters the virtual mouse area and takes position, makes a ready gesture, and the system automatically locks and tracks, docks (combines and matches) the model and follows it. The sticking distance range is the entire mouse operation area, and the entire mouse operation area is the pad area;

[0040] set up:

[0041] The setting generally refers to adjusting the default or previously set parameter values;

[0042] S8. The size, position, angle, viewing angle, and display color of the proposed modeling operation area described in S1 may be adjusted by the operator as needed and convenient. Generally, for humans, the area is typically located directly in front of the operator so that the model and operation do not obstruct the operator's primary view of the displayed scene, and within the three-dimensional coordinate measurement detection range, for convenience and need.

[0043] For S2 two-dimensional traditional computing display (applicable to S4 and classified as S2 mode), it is generally located below the main line of sight of the screen in front of you. The usual desktop display is on the table top, at elbow height.

[0044] For S3 three-dimensional space calculation and stereoscopic display (applicable to S4 and classified as S3 mode), it can be anywhere in front of the operator that both hands can reach, usually between the shoulder and chest to the hip area that both hands can reach, according to convenience and need.

[0045] Set area status and range: select the modeling operation area as a single model operation area, or select the range setting within which the operator leaves the model area and enters other adjacent areas;

[0046] The organization content, layout style, configuration type, key size ratio, angle, and position of the virtual keyboard and virtual mouse (if selected) models described in S1, S2, S3, S4, and S5 are adjusted and set by the operator according to needs and convenience, adapted to ergonomic settings, and can be regionalized, three-dimensional, non-planar, etc., and suitable for the operator's operating subject characteristics;

[0047] Includes the option to set the specific response of each key; the angle is usually perpendicular to the line of sight (default);

[0048] The display color parameters include: edge brightness, letter symbol brightness, color, perspective (keyboard and disk), etc. The operator can select and set them according to the needs;

[0049] The mapping display of the model, for the S2 two-dimensional traditional computing display situation (applicable to S4 and classified as S2 mode) (omitted below), includes appropriate and adjustable conversion parameters, including: conversion ratio, conversion angle, and viewing angle, which are provided by the technical parameters of the system, display device, and three-dimensional coordinate measurement device and display requirements. The operator can adjust and select the settings as needed within the range provided by the system; generally, the conversion angle is usually converted to be parallel to the screen surface, and the viewing angle is the rear viewing direction (default);

[0050] For the S3 three-dimensional space calculation stereoscopic display of full depth of field and virtual-real fusion (the S4 method is applicable to the S3 method) (omitted below), the display is the model body, the mapping display is consistent with the model body, the ratio is 1:1 and cannot be adjusted, and the angle, viewing angle, and display color are the same as the model;

[0051] Choose to set two specific gestures to represent: ready operation and release operation;

[0052] Set the keyboard model adaptive parameters, docking distance: the distance range, docking position and spacing between the model and the operating body in the vertical docking within the modeling operation area;

[0053] For gesture mouse, choose to set three specific gestures and one action gesture, and define them as: cursor, left click, right click. You can choose cursor gesture or other gestures to define up and down flip actions to represent: up and down page flip;

[0054] For the virtual model mouse, you can choose to define the docked single gesture up and down flipping action definition to represent: up and down page turning;

[0055] Set the cursor's response ratio to the main action (the default is 1:1), including the response ratio of the up and down page movements.

[0056] Anti-shake misoperation control can be enabled;

[0057] Set the display area of ​​the operating subject mapping display;

[0058] The above parameters are valid for the entire modeling operation area;

[0059] The operating subject and the marked principal point and / or skeleton model are mapped and displayed at the corresponding position according to the display setting requirements of the area, including: scale, angle, viewing angle, brightness, color, perspective, and other parameters;

[0060] The conversion parameters in the modeling operation area, including scale, angle, and viewing angle, are consistent with the model conversion parameters described above. The display color parameters, including brightness, color, perspective, edge brightness, and other parameter values, can be adjusted by the operator as needed.

[0061] operate:

[0062] S9. In the modeling operation area within the operation area, on the virtual keyboard model and or virtual mouse model, the operator performs input and operation actions respectively;

[0063] For gesture mouse, it operates with three defined gestures and one action gesture: cursor, left click, right click, and page turning gesture;

[0064] response:

[0065] S10. The computing and processing device measures and processes the spatial position coordinates of the operating subject and operating action within the modeling operation area, primarily the three-dimensional coordinates of the principal point (keystroke point, which may be multiple points), extracts the coordinates, and makes a judgment: compares the coordinates with the key area of ​​each key of the virtual keyboard model constructed by the computing and processing device within the modeling operation area, i.e., the range of its coordinates; and determines whether the action "keystroke" is successful (including the keystroke action and the specific key) based on the degree of overlap between the three-dimensional coordinates measured by the principal point and the pre-built (pre-set library) parameter values ​​of the key area coordinates of the model; and executes (hardware and software) corresponding responses and operations;

[0066] For gesture mice, the coordinates of the main points and joint points are extracted, and the joint points and / or skeletal model structure patterns are compared with the pre-set library. Based on the degree of fit, the following judgments are made: identifying the cursor, and then mapping the position and movement of the cursor on the screen with the position and movement of the main point of the cursor gesture, identifying the left-click, right-click and page-turning gestures, and executing the corresponding responses of keystrokes and page-turning;

[0067] For the virtual model mouse, the position and movement of the virtual mouse model and the operating subject in the mouse operation area are mapped to the position and movement of the cursor in the mapping screen, and the left-click, right-click and page-turning actions of the mouse model are judged and recognized, and the corresponding keystroke and page-turning responses are executed;

[0068] S11. The virtual model, gestures, operating subjects and real-life scenes of the operating actions, together with the responses, are displayed in real time at the corresponding positions in the scene screen. The operator monitors the keyboard and mouse input operations by watching the screen; completing human-computer interaction;

[0069] The operator evaluates the operation situation, operation response, and operation results to further complete the operation task;

[0070] The main point generally refers to the keystroke point, which can be multiple points, including fingertips, contact points, etc.

[0071] The operator refers to the part or the whole that issues commands and monitors, and the operating subject refers to the part that completes the operation, including hands (including arms), robotic arms, etc.

[0072] The operation of integrating into the display space described in 1.(2) above includes the following steps:

[0073] Integration:

[0074] Integration refers to the operation subject or operator integrating into the space of the display screen of the computing and processing device, entering the virtual and real scene, operating as a member of it, and performing various direct operations and experiences on the objects and components of the display space scene in a participatory and interactive manner;

[0075] S1. The operating area is defined as the area within the measurement range of the three-dimensional coordinate measuring device, in front of the operator, and surrounding space. Typically, the entire operating area is selected as the single integrated operating area. Alternatively, the area within the operating area, outside the aforementioned modeling operating area, is selected as the integrated operating area. The operating subject can be configured to automatically enter the integrated operating area and functional state upon leaving the modeling operating area or within a certain distance of the virtual model (including any virtual mouse, provided it does not conflict with the virtual model).

[0076] S2. Within the three-dimensional coordinate measurement range and the operation area, the operating subject and / or the operator are mapped and integrated into the display screen space scene through three-dimensional coordinate measurement and displayed in real time, and the main point (contact point, which can be multiple points) and / or the skeletal model are marked at the corresponding position;

[0077] The fusion body and the entity (operation subject and / or operator) are in a mapping correspondence relationship. According to the application and use of the fusion operation, it is classified as S4(1) fusion direct operation, and the fusion body is displayed with appropriate and / or adjustable conversion parameters;

[0078] The S4(2) is integrated into the character operation, and the operation action and posture of the entity are mapped to the corresponding parts of the character; the integrated body is the character body, which is displayed by the design parameters of the character model;

[0079] For S4(1) direct integration operation, the integration can be selected according to the situation and needs: the integrated body display screen can appear in the form of video images, virtual animation or picture-fitting, or three-dimensional reconstruction display; the simplest way is to display the integrated body contour line, marked main points and or skeleton model;

[0080] set up:

[0081] The setting generally refers to adjusting the default or previously set parameter values;

[0082] S3. Set the regional status and range: select the entire proposed operation area as a single integrated operation area state, or select the modeling operation area within it, or outside the model, to integrate the operation area;

[0083] For S4(2) character integration operation, the integration mapping is based on the ontology display and structure of the character model. The design parameters except color parameters are not adjustable in the integration mapping.

[0084] For S4(1) integration direct operation, the appropriate and / or adjustable conversion parameters of the integrated body mapping display, including mapping conversion ratio, conversion angle, and mapping position, are provided by the conditional technical parameters and display requirements of the system, display device, and three-dimensional coordinate measurement device, and the operator can adjust and select the settings according to needs and convenience within the range provided by the system;

[0085] The mapping position refers to the initial and starting position of the entity and the merged body when the operation starts. Generally speaking, it is at the bottom and both sides of the main sight line that does not block the viewing of the displayed scene in front of you. You can choose it according to your convenience and needs:

[0086] The initial position of the entity is generally the appropriate and natural position within the detection range of the three-dimensional coordinate measurement, which is integrated into the operation area as described above: the entire operation area or the modeling operation area within it, or the part outside the model;

[0087] The entire operating area is usually selected as a single integrated operating area. Generally speaking, for a two-dimensional traditional computing display, it is usually located below the main line of sight of the screen in front of you. A typical desktop display is on the table top, at elbow height.

[0088] For three-dimensional computing and stereoscopic display, the display can be anywhere in front of the operator that is within reach of both hands, usually between the shoulder and chest and the hips, which can be reached by both hands, depending on convenience and need.

[0089] The starting position of the fusion body, for the two-dimensional display, is generally at an appropriate position near both ends of the lower edge of the front field of the display screen, with a conversion ratio and conversion angle to display the starting state fusion body;

[0090] For the three-dimensional display, according to the requirements of display technology limitations: full depth virtual and real fusion display, generally the fusion volume and the entity are overlapped, and the fusion volume is superimposed on the entity body (non-adjustable conversion);

[0091] Non-full-view depth-of-field fusion display is a fusion of the initial state and the volume by converting the scale and angle at an appropriate position on the lower edge of the foreground (similar to the case of 2D display);

[0092] When an operation interface pops up or an activation area appears, the interface or area usually occupies the main display surface. The fusion body usually starts at an appropriate position at the lower edge of the main display front field and is displayed with a solid projection conversion (similar to the case of non-full depth of field virtual and real fusion display); if it does not occupy the main display surface, the fusion body can follow the corresponding position of the lower edge of the interface or area and display the corresponding display - the projection conversion of the main display surface situation;

[0093] Set the area where the operator is integrated into the display;

[0094] The color parameters of the fusion body display include brightness, color, perspective, edge brightness, and other parameter values, which can be adjusted by the operator as needed;

[0095] Set the definition operations of object components, object activation confirmation, and exit confirmation of activation. The three defined gestures of the gesture mouse can be used to implement the operation. The gesture mouse is fully integrated into the operation area.

[0096] Set and adjust the activated area range and activate the color effect parameters of the display; set the main display area and position; for minor operations, it can be combined with the focus area selection method to enlarge the display for convenient use;

[0097] The activated display color effect parameters of the components, objects and areas include: brightness, color, perspective, edge brightness and other variable parameters, which can be adjusted by the operator as needed;

[0098] Select and set two specific gestures, defining the representations: ready operation and release operation; (can be the same as 1.(1))

[0099] Set the response ratio of the main action parameters of the operation (the default is 1:1), including: movement distance, rotation, angular change, etc.; can be used for incremental superposition control and anti-shake misoperation control; for precise situations, you can select settings to facilitate fine-tuning;

[0100] The set parameters are valid for the entire integration operation area;

[0101] The above-mentioned adjustment of setting parameters, adjustment of the fusion body position and / or the modeling operation area position can be performed dynamically in real time, which is better and more convenient;

[0102] operate:

[0103] S4. The hardware and software of the computing and processing device must be designed to support scenario-based operation. Based on the software's construction, the operator can directly manipulate operable physical components and objects, performing various participatory and interactive operations and experiences.

[0104] (1) Integrate direct operation, as a part of the display screen space, directly operate and interact with the operable objects and components in the virtual and real scenes; have the characteristics of remote control, remoteness: direct operation at far and near; the characteristics are: activation first, then operation; that is, first activate and confirm the target object to be controlled, and then apply the operation action mapping to it;

[0105] a. Direct application: directly applying the corresponding mapped operation action to the physical components and objects within the display screen area to perform operation and interaction;

[0106] b. Mapping response application: the data information of the control action is transmitted to the corresponding receiving port through network communication through data instructions, mapped to the response action of the corresponding execution device or machine, and replaced and extended by it to complete the action work - realized by mapping to the execution device or machine;

[0107] The operation starts from the initial position of the entity (operating subject and / or operator) and the starting position of the fusion body, and the corresponding mapping is carried out; when the operation is completed, the activation is exited, and the entity and the fusion body return to the original initial and starting position states and corresponding displays; the system is then ready to carry out a new round of activation and operation;

[0108] The activation confirmation and exit confirmation operations can be performed dynamically in real time and have the highest priority;

[0109] (2) Integrate character operation, map it to the character, and make it a member of the virtual and real space scene world, i.e., the role player, to carry out participatory operation and interactive operation; the actions and postures of the operating subject and the operator are mapped to the corresponding parts of the character; the characteristic is that the corresponding mapped operation action is issued from the perspective and position of the character, i.e., the execution action;

[0110] The character directly manipulates and interacts with the physical components and objects in the virtual and real world around him, completing the corresponding operation actions and control instructions given by the operator; the interaction follows the same general rules and experiences as in the real world;

[0111] The main display usually presents the visual field of view from the character's perspective, and can also display the operator's perspective (global picture); it can be switched in real time;

[0112] response:

[0113] S5. The computing and processing device, based on three-dimensional coordinate measurement technology and devices, measures and processes the position coordinates, movements, postures, etc. of the operating subject (which may be multiple parts) and / or the operator within the operating area, primarily the three-dimensional coordinates of the principal points (contact points, which may be multiple points) and / or joint points. The results are mapped to the corresponding parts of the integrated body and corresponding judgments are made: the results are compared with the position coordinates of the corresponding physical components and objects, and the pre-established (pre-established library) of the operation mode, and with the pre-established (pre-established library) of the degree of movement (speed, acceleration) and trajectory, the pattern of posture, and paradigm parameters. Based on the degree of coincidence or fit, corresponding judgments and identifications are made; and corresponding responses and operations are executed (hardware and software);

[0114] S6. The real scene of the operating subject and / or the operator and the operating action, together with the response, is displayed in the corresponding screen scene. The operator monitors the integration of the operation by watching the screen; completing the human-computer interaction function;

[0115] The main point generally refers to the contact point, which can be multiple points, including fingertips, joints, etc.

[0116] The other methods mentioned in 1.(3) above refer to the fact that the same system can also provide interactive methods including virtual handwriting input, body movement posture recognition, lip recognition, gesture recognition, virtual touch screen, eye movement interaction, etc., which are related to the position coordinates, trajectory, posture, etc. of the action for input and operation;

[0117] It can be set in the virtual mouse operation area described in S5 of the above invention content 1.(1) (replacing the mouse), or in an appropriate area outside the virtual keyboard model and / or virtual mouse model in the modeling operation area, or the entire modeling operation area can be selected as a single operation mode function area; or in an appropriate area outside the modeling operation area described in S1 of the above invention content 1.(2), or the entire operation area can be selected as a single operation area state; the range of settings provided is configured accordingly according to the corresponding requirements and needs of each interaction mode, and the operator can select according to the needs;

[0118] (1) Virtual handwriting input interaction,

[0119] S1. The computing and processing device measures and processes the three-dimensional coordinates of the principal point of the operator's action based on three-dimensional coordinate measurement technology and equipment, extracts the coordinates, tracks the trajectory, and determines: the structural pattern of the principal point (feature point) of the action trajectory is compared with a pre-built (pre-set) library, and an identification judgment is made based on the degree of fit; the hardware and software respond and operate accordingly to realize the input operation;

[0120] S2. The real scene of the operating subject and the operating action, together with the response, is displayed in the corresponding screen scene, and the operator monitors the handwriting input operation by watching the screen; completing human-computer interaction;

[0121] (2) Body gesture recognition and lip shape recognition interaction,

[0122] S1. The computing and processing device measures and processes the corresponding three-dimensional coordinates or posture parameters of the operator's body movements or lip shape based on three-dimensional coordinate measurement technology and equipment, extracts the coordinates of the principal points and joint points, and performs judgment and identification: compares the body posture joint points and / or the skeletal model structure pattern or the lip shape structure pattern with a pre-built (pre-set) library, and makes an identification judgment based on the degree of fit; executes the corresponding hardware and software responses and operations to realize interactive operation;

[0123] S2. The real scene of the operator and the operation action, together with the response, is displayed in the corresponding screen scene. The operator monitors the body movements or lip movements by watching the screen; completing human-computer interaction;

[0124] (3) Support virtual touch screen, gesture recognition, and eye movement interaction,

[0125] The support mentioned here means that the same system can also provide support for interactive use including virtual touch screen, gesture recognition, eye movement interaction, etc. Correspondingly, the software design needs to support it and be easily integrated into the system of the present invention;

[0126] aVirtual touch screen interaction,

[0127] The virtual touch screen interaction mode can be achieved by modifying the S2, S3, and S4 models in 1.(1) above as follows:

[0128] For the S2 two-dimensional traditional computing display scenario, a flat panel interface virtual touch screen model is constructed in the modeling operation area, and the style is simultaneously displayed on the display screen with appropriate and adjustable conversion parameters;

[0129] For the wearable S3 three-dimensional display scenario, a flat panel interface virtual touch screen model is constructed in the modeling operation area and displayed on the body;

[0130] For the S4 3D spatial display scenario, which is non-wearable, the situations are the same as those for S2 and S3 above.

[0131] The operator operates the virtual flat touch screen model, and the computing and processing device extracts the three-dimensional coordinates of the principal point (keystroke point, which can be multiple points), compares it with the coordinate area range of each icon and menu item in the library pre-set by the virtual touch screen model, and determines whether the "click" is successful based on the degree of overlap. The corresponding (hardware and software) response and operation are performed, and the principal point is marked at the corresponding position in the screen scene, and the corresponding position is displayed immediately together with the response, thereby completing the human-computer interaction;

[0132] b. Gesture recognition, eye movement interaction (existing methods),

[0133] The gesture recognition computing and processing device extracts the three-dimensional coordinates of the gesture body joint points, performs gesture processing, compares them with the structural patterns in the pre-set library, makes recognition judgments based on the degree of fit, and performs corresponding responses and displays to achieve interactive operations;

[0134] In the eye movement interaction, the computing and processing device extracts the posture parameters of the principal point pupil, performs projection processing, obtains the corresponding aiming position (eye pupil focus point) parameters, compares them with the coordinates of objects in the picture space and virtual and real scenes, makes judgment and extraction of the selected corresponding target body based on the degree of overlap, and performs corresponding response and display, thereby realizing interactive operation;

[0135] The judgment of the degree of coincidence or fit described in 1.(1), 1.(2) and 1.(3) above, as well as the adjustment of sensitivity and the adjustment of the amplitude of movement, are set by the system, and the operator can select and set the options provided by the system if any;

[0136] Input and operation are achieved through the various virtual mode interaction methods provided by the present invention, without the need for contact or touching the screen, and with mobility.

[0137] 2. A system for implementing the human-computer interaction method based on three-dimensional coordinate measurement described in Content 1 of the present invention,

[0138] Includes the following modules:

[0139] (1) Three-dimensional coordinate measurement module,

[0140] The three-dimensional coordinate measurement module is used to measure the spatial position coordinate parameters of the operating subject and / or the operator's operating actions in human-computer interaction. It is connected to the calculation processing module and its computing power and software support realize its functions. It measures the operating subject and / or the operator's virtual model and operating actions within the measurement range of the three-dimensional coordinate measurement device and the operating area, as well as the operating actions in areas outside the measurement range of the three-dimensional coordinate measurement device. The measured data is processed and calculated accordingly to obtain the spatial position coordinate parameters of the operating subject and / or the operator's operating actions, mainly the three-dimensional coordinates and posture parameters of the main points (keystroke points, contact points, focus points, etc.) and / or joint points, and further obtains the trajectory, posture, action degree and other related parameter values.

[0141] (2) Computational processing module,

[0142] The computing processing module is mainly constructed by at least one processor (including CPU, GPU, NPU, TPU, etc.) and the software running on it, which is used to process and calculate the parameter data information of other modules, make judgments, identify, execute responses, and run, etc.;

[0143] It includes: extracting the measurement data of the three-dimensional coordinate measurement module and making judgments: comparing with the regional coordinates, or patterns, or paradigm parameters in the pre-built (pre-set library), making corresponding judgments and identifications based on the degree of coincidence or fit, and executing (hardware, software) corresponding responses and operations;

[0144] The system is responsible for implementing the human-computer interaction method based on three-dimensional coordinate measurement described in the above invention content 1, establishing a proposed modeling operation area and an integrated operation area in the operation area, and constructing a virtual interaction mode (as the system default setting), including: a virtual keyboard, a virtual mouse interaction mode, or an integrated display space operation interaction mode, or other interaction modes;

[0145] Provides adjustment and selection of relevant parameters for each mode, including: operation area status and range, organizational layout style and display characteristic parameters of the constructed virtual keyboard, mouse (if selected) and other models, model display style conversion parameters, specific responses of each key that can be set, model docking distance range, gesture mouse, mapping response ratio and area parameters of the integrated operation action, incremental overlay and anti-shake misoperation control, starting position, ratio and angle settings of the integrated body mapping display, activation related settings, other corresponding settings, or adjustable parameters, etc., which can be set accordingly to adapt to ergonomics and needs, and for convenience; including the selection and adjustment settings of overlap or fit judgment sensitivity and movement amplitude;

[0146] (3) Display module,

[0147] Connected to the calculation and processing module, it is used to display the information needed to display the measurement, processing, calculation, judgment, recognition and response, operation and effect performed by the three-dimensional coordinate measurement module and the calculation and processing module for the operator to monitor and watch;

[0148] Including: display content and display requirements of the human-computer interaction method based on three-dimensional coordinate measurement according to the above-mentioned invention content 1, including the establishment area of ​​the interactive mode, various virtual models constructed, gesture mouse, integrated direct operation, character operation, or other methods, mapping and response of the operation subject and / or operator and operation action, main points, joint points, and other corresponding information that needs to be displayed;

[0149] Including: 2D traditional computing type requirement display, 3D space computing type requirement display, full-view depth virtual-real fusion display, non-full-view depth virtual-real fusion display;

[0150] The above modules jointly implement various virtual mode interaction modes and input operation functions of the human-computer interaction method of the present invention, thereby further completing the work tasks of the system;

[0151] The various virtual mode interaction methods provided by the system of the present invention realize input and operation without contact or touching the screen, and are mobile;

[0152] The three-dimensional coordinate measurement module is implemented based on three-dimensional coordinate measurement, including binocular stereo vision, structured light, TOF laser time-of-flight, or AI-based three-dimensional reconstruction methods.

[0153] The present invention is not limited to the selection of three-dimensional coordinate measurement technology and device. The present invention preferably uses binocular stereo vision technology and device, using two identical camera sensors with different viewpoints. The overlapping area of ​​the field of view is the measurement range and the appropriate operation area.

[0154] The three-dimensional coordinate measurement module preferably adopts a binocular stereo vision technology device, and the specific steps of the system are as follows:

[0155] S1. Three-dimensional coordinate measurement module, three-dimensional coordinate measurement is preferably two identical camera sensors with different viewpoints of a binocular stereo vision device, whose parameters and angles make it possible for the overlapping area of ​​its field of view to appropriately cover the operating area; if necessary, calibration is performed to obtain correction parameters for internal and external parameters;

[0156] S2. The calculation and processing module constructs various virtual interaction modes according to the human-computer interaction method based on three-dimensional coordinate measurement described in 1 of the present invention;

[0157] S3. According to the steps, the operator performs various virtual interaction mode parameters to adapt to ergonomics and needs, convenient settings;

[0158] S4. The operating subject performs input and operation actions on the virtual keyboard model and / or virtual mouse model;

[0159] For gesture mouse, it operates with three defined gestures and one action gesture: cursor, left click, right click, and page turning gesture;

[0160] Alternatively, the operation mode may be integrated into the display space, where the operating subject and / or operator is mapped and integrated into the display screen space scene to perform various direct operations and experiences on the operable objects and components in a participatory and interactive manner. The hardware and software of the system need to be designed to support the integrated operation mode, including the activation operation of the control target: activation first, then operation;

[0161] Alternatively, you can select settings and enter other modes of interaction based on the needs of the occasion, including: virtual handwriting input, body movement posture recognition, lip recognition interaction, support gesture recognition, virtual touch screen, eye movement interaction, etc., and use interactive methods related to action position coordinates, trajectory, posture, etc. for input and operation; the corresponding software needs to be designed to support it. Among them, gesture + eye movement interaction in wearable 3D scenes requires the addition of at least two binocular vision sensors;

[0162] S5. A three-dimensional coordinate measurement module processes the images from the sensors of the different viewpoint cameras at the same time, matches feature points such as principal points, and calculates (measures) the spatial position coordinate parameters of the operating subject and / or the operator and their actions based on triangulation methods and epipolar constraint relationship rules. This module primarily includes the three-dimensional coordinates and posture parameters of the principal points (keystroke points, contact points, focus points, etc.) and / or the joints of the operating subject's skeletal model, and further obtains relevant parameter values ​​such as trajectory, posture, and degree of movement.

[0163] S6. The calculation and processing module extracts the measurement data from the three-dimensional coordinate measurement module and compares it with the regional coordinates, patterns, or paradigm parameters stored in a pre-established library or table. Based on the degree of coincidence or fit, it makes a corresponding judgment and identification, and executes the corresponding response and operation (hardware or software);

[0164] S7. A display module displays information required to display the measurement, processing, calculation, identification, response, and operation of the three-dimensional coordinate measurement module and the calculation and processing module in accordance with the display content and display requirements of the human-computer interaction method based on three-dimensional coordinate measurement described in Summary 1 above;

[0165] The image is provided by two identical camera sensors at different viewpoints of a binocular stereo vision device, and one image is selected for correction and then displayed for a two-dimensional traditional computing display;

[0166] The operator monitors the input operation status by watching the operation scene and the corresponding display screen, thus realizing the human-computer interactive operation function;

[0167] The system follows the operator's operations and instructions to complete the operator's required operations, work, and other tasks;

[0168] The two identical cameras and sensors with different viewpoints can be distributed horizontally, symmetrically on the left and right, which is the preferred solution for ease of calculation. However, they can also be distributed vertically or in other distributions, or asymmetrically, and can be more than two sensors, or even multi-point, multi-unit (pair), distributed control arrangements, etc. Different distribution, arrangement, and number can be selected according to specific needs, and the calculation amount and settings will vary accordingly.

[0169] Similarly, according to the needs of the use environment, select the appropriate camera sensor, applicable frequency band, illumination and other technical parameters of the binocular stereo vision device;

[0170] Alternatively, various frequency band light spots and cursors may be used to mark or replace the main points and key points of the operating subject;

[0171] The preferred binocular stereo vision method for three-dimensional space calculation type has the following specific features based on the above:

[0172] a wearable solution,

[0173] (3) Display module,

[0174] The two-way binocular stereo vision image realizes the three-dimensional stereoscopic panoramic (immersive) display of full view, depth, virtual and real fusion;

[0175] This is to realize the three-dimensional space computing type wearable function of the display module of the human-computer interaction system of the present invention;

[0176] Including: the real scene and response of the operating subject and / or operator and the operating action is equal to the real-time display in the three-dimensional panoramic (immersive) picture display scene with full depth of field and virtual-real fusion;

[0177] bNon-wearable solution,

[0178] (1) Three-dimensional coordinate measurement module,

[0179] The layout and orientation of non-wearable three-dimensional coordinate measurement sensors are the same as the display orientation, facing inward toward the user or operator; they are usually arranged in multiple (pairs) and distributed, and the location and number are arranged according to the three-dimensional spatial detection range of the spatial domain;

[0180] This is to realize the three-dimensional coordinate measurement module function of the human-computer interaction system of the present invention;

[0181] (2) Computational processing module,

[0182] The processor and software support the implementation of the three-dimensional space calculation type configuration of the present invention and the preferred binocular stereo vision method, and better support AI integration and application;

[0183] This is to realize the non-wearable function of the three-dimensional space calculation type of the calculation processing module of the human-computer interaction system of the present invention;

[0184] This includes: supporting distributed control of multiple points and units (pairs) around the periphery, integrating and processing the overlapping edges of the detection area of ​​3D coordinate measurement sensors to form a continuous, integrated, and panoramic scene effect, and achieving coordinated and effective system control of distributed detection;

[0185] (3) Display module,

[0186] The two-way images of binocular stereo vision realize three-dimensional space stereo display;

[0187] This is to realize the non-wearable function of the three-dimensional space calculation type of the display module of the human-computer interaction system of the present invention;

[0188] Includes: non-wearable three-dimensional spatial stereoscopic image display, which may or may not be panoramic;

[0189] Due to the limitations of display technology, it may be a fusion of full depth of field and virtual reality, or it may not be a fusion of full depth of field and virtual reality. The virtual model and the fused body are displayed in real time using the corresponding methods and steps of the human-computer interaction method of the present invention described in Summary 1, respectively using the two types of mapping display relationships, as well as operation actions and responses;

[0190] The non-wearable computing processing module is usually a configuration with strong computing power, which can be a networked, distributed or even parallel processing, quantum computing device, etc., supporting the integration of AI and combined applications;

[0191] An integrated system that supports systematization, networking, and the mixing and joint use of a large number of execution devices and machines, as well as generalized computing devices and machines (smart devices);

[0192] The non-wearable three-dimensional display may include: holographic stereoscopic display technology, other naked-eye three-D display technologies, a simple method of wearing polarized glasses (not a screen), and other display technologies that will appear in the future.

[0193] 3. An application device scheme for applying the human-computer interaction method and system optimization mode of the present invention: a scroll-shaped flexible screen (mobile) terminal,

[0194] Flexible screens can be rolled up for easy movement, but currently there is no compatible, consistent, and convenient input and operation device. Physical keyboards are completely incompatible with portability and mobility, and flexible screens cannot be used as touch screens. The human-computer interaction method and system of the present invention provide various virtual interaction modes, eliminating the need for a physical keyboard, mouse, touch screen, or contact, and can completely resolve the structural technical barriers to flexible screen input and operation.

[0195] The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0196] (1) Scroll flexible screen and housing,

[0197] The housing contains a scroll (with elastic retraction), with two or two parts on the left and right sides, providing a rollable flexible screen that can be retracted and extended. It also contains hardware and software devices that support screen display and wireless communication, including smartphone (including processor) functions; it can independently complete mobile phone functions and screen display operations, and the screen area of ​​the housing (at least the front) is transparent;

[0198] (2) Three-dimensional coordinate measuring device,

[0199] Preferably, two identical camera sensors of the binocular stereo vision device are horizontally distributed at the upper front ends of the left and right shells, symmetrically on the left and right sides; or a sensor is added to the left or right shell symmetrically up and down to support a vertical screen;

[0200] (3) Support arm, cross bracket and clamping sleeve,

[0201] The support arm and cross bracket are used to support and hold up the flexible screen when it is unfolded. The card sleeve is used to lock the shell on the back and fix it to the front end of the support arm, and also serves as a data and power cable adapter.

[0202] The arm contains expansion storage, expansion battery and data, power cables and terminal sockets for expanding performance and connecting the shell and host firmware in wired mode; a multi-section structure can be used;

[0203] (4) Host equipment, stand and support,

[0204] The host includes a computer processor, software, storage, various ports, etc., providing more powerful computing performance and functions. It can be connected to the shell components by wire or wirelessly to work together and support the integrated application of AI.

[0205] The stand and bracket work together to support the arm and adjust the pitch angle and height or support the horizontal shell (when the screen is folded or unfolded). The stand contains data and power cables and terminal connectors, which serve as a wired connection between the host and the arm.

[0206] Steps:

[0207] S1. Activate the interaction mode of the human-computer interaction method of the present invention. Configure the operating area state and range, model, gesture mouse, conversion parameters, response ratio, area, docking, and other related settings and adjustable parameters to suit ergonomics, needs, and convenience. The operator then operates and works in the selected interaction mode, or continues to use the last selected setting or the system default setting.

[0208] Input and operation can be achieved through various virtual mode interaction methods provided by the present invention, without contact or touching the screen, and with mobility;

[0209] S2. When unfolded, it can fully provide all the functions of a smartphone and can work with the host wirelessly or with a wired connection. When held in hand, it can be input and operated with one hand.

[0210] S3. When unfolded, it supports the device horizontally or vertically, with a large screen, and usually works in wired mode. This is particularly suitable for unleashing the high performance of the host PC and for high-end scenarios requiring high-resolution, high-definition, and large screens.

[0211] S4. When the screen and housing are separated from the host, the host can be remotely controlled via a network connection.

[0212] All of the above can be conveniently input and operated using various virtual mode interactive methods of the present invention, thereby completing various tasks and tasks of the operator;

[0213] The flexible screen can be a two-dimensional display or a three-dimensional display;

[0214] This technical solution can also be applied with specific element changes, including:

[0215] The housing is fixed to the front end of the support arm by a card set, or the opposite back side of the housing can be fixed by a groove to clamp the front end of the support arm and perform line switching, or other equivalent functions;

[0216] The housing contains two reels arranged on the left and right, which can be symmetrical in shape, or the main and secondary parts can be single-axis rolled, with the ends being single-extension type with support rod structures;

[0217] The winding mechanism can be in the form of a reel or a folding method;

[0218] The device system configuration may also have other configurations and functional uses, including not requiring a separate host configuration, or focusing on an image screen (smart TV), or not requiring mobile phone communication functions, etc., and the configuration can be selected according to specific needs;

[0219] Other three-dimensional coordinate measurement technologies and devices or equivalent methods may be used.

[0220] 4. An application system solution using the human-computer interaction method and system optimization mode of the present invention: a micro-surgical robot system,

[0221] Early micro-surgical robotic systems were controlled through a console using a push-button handle coupled with foot pedals. The mechanical linkages used sensors that generated signals and transmitted them to a processing unit, resulting in a mechanically controlled main hand. Later generations developed and incorporated VR / AR headsets and 3D display technology. The surgeon controlled the system using a pair of hand controllers equipped with inertial sensors, resulting in a relatively compact and lightweight electronic sensing structure. The latest generation utilizes various sensors attached to the surgeon's headset, body, and especially shoulders, elbows, wrists, hands, and even fingers. These sensors detect the position of the surgeon's selected area, control the camera's position, achieve a 360-degree view of the surgical area, and control the robotic arm to follow the surgeon's movements, ensuring consistent movement.

[0222] The first two methods of motion capture are mechanical linkage capture and inertial sensor capture. Due to the kinematic space limitations of the machine and the instrument selection control (foot pedal) method, the operation is not intuitive, inconvenient and inflexible. The subsequent manual controller method has eliminated the foot pedal, which is a good improvement. The mobility and informationization of head-mounted 3D headsets have been significantly improved, providing a full range of three-dimensional stereoscopic imaging perspectives, and a higher amount of computer information can be used. However, the arm movement mechanism during manipulation (satisfying the performance of the detection device) is still inconsistent with the actual operation method and experience of real direct surgery, and the intuitiveness is still not good. The latest generation introduces a large number of sensors attached to body parts, making the field of view omnidirectional and the amount of information richer and more complete, which is more convenient for surgery. The control of robotic arms has also become more consistent and intuitive, reaching a new level. However, a large number of diverse sensors need to be arranged, attached and connected to the surgeon's body. These sensors need to be connected to collect data and transmit data and need to be powered. During the operation, the doctor's two arms need to constantly operate and move the body, which takes a long time and is very intense. Under such circumstances, the various sensors attached to the body may inevitably cause obstruction and drag, and may even fall off or give false alarms, which will affect or even cause errors and interrupt the operation; and there are problems such as the need for subsequent disinfection and sterilization; so its applicability, convenience, reliability and even safety are still affected.

[0223] The human-computer interaction method of the present invention is a virtual mode interaction feature, which does not require contact and has mobility. The spatial action coordinates and other element measurement features of the present invention have a more direct, consistent, intuitive and natural performance for action operation mapping execution applications, and are particularly suitable for application scenarios such as micro-surgical robots; it can completely and accurately map the action movement position and orientation of the operating subject's arm, head and / or operator, and is suitable for supporting three-dimensional spatial computing type wearable VR / MR / AR and non-wearable situations, and is integrated with intelligent, systematic and distributed control forms, and is conveniently integrated with AI medical assistance (preferably recommended), which can completely solve and avoid the problems of the current system. The technology is highly accurate, reliable, convenient, safe and guaranteed, and is fully capable of meeting medical requirements.

[0224] The basic components of the micro-surgical robot system include: computing and processing unit, interactive control unit, display unit, robot subsystem unit, and accessory units such as robotic surgical instruments and operating table.

[0225] Based on this, the technical features of this system solution are:

[0226] The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0227] (1) Interactive control unit,

[0228] The three-dimensional coordinate measurement is preferably carried out by applying a binocular stereo vision device with multiple points, multiple units (pairs), and distributed placement in the appropriate range and position of specific areas and designated workstations in the surgical environment space for distributed control;

[0229] include:

[0230] Mobile workstation: VR / MR / AR head-mounted mobile display, 3D coordinate measurement device configuration: horizontally distributed on both sides of the upper frame of the head display, symmetrical on the left and right, facing outward;

[0231] Fixed workstations: including operator workstations at the control console / room, surgeon workstations in the operating area, assistant or nurse workstations, and other detection locations;

[0232] The layout of the three-dimensional coordinate measuring device should be based on the optimal and effective location, area, and mounting conditions required for the surgical procedure and convenient three-dimensional measurement. The operator's station at the main console can be horizontally distributed on both sides of the upper frame of the 3D / 2D display (not a head-mounted display), symmetrically installed on the left and right sides, facing the operator, or can be separately or externally fixed with an independent stand or secured via various fastening devices. The surgeon's station in the operating area can be mounted on the operating table, ceiling, or other methods mentioned above. The selection of other locations is similar.

[0233] The 3D / 2D display screen should be configured according to the needs of the designated workstation and should be configured to match the layout of the binocular stereo vision device;

[0234] (2) Computational processing unit,

[0235] Computer processors and software support the application implementation of the present invention and the implementation of this system solution, including three-dimensional coordinate measurement, processing, judgment, recognition and response, 2D / 3D display (VR / MR / AR three-dimensional space calculation), robotic subsystems and binocular stereo vision including surgical positioning, registration, tracking, etc., support for incremental overlay control and anti-shake misoperation control, AI medical integration implementation support, etc.

[0236] Including directly calling the action space position coordinates and posture orientation parameters of the three-dimensional coordinate measurement unit to map to the execution action position and orientation of the character;

[0237] (3) Display unit,

[0238] Display the information that needs to be displayed by the three-dimensional coordinate measurement unit and the calculation processing unit accordingly;

[0239] Robot subsystem binocular stereo vision 2D / 3D display (optionally one channel of corrected 2D display or two channels of 3D display), or switch to other positions for binocular stereo vision display;

[0240] Including: display of surgical positioning, registration, tracking, etc. of the robot subsystem, display of surgical operation movement trajectory, position, orientation and response, and effect (movement and amount, direction angle value, etc.);

[0241] The configuration and functions of other units adopt existing technology or similar situations;

[0242] Steps:

[0243] S1. Activate the interaction mode of the human-computer interaction method of the present invention, and configure the operating area state and range, model, gesture mouse, conversion parameters, response ratio, area, docking, and other related settings and adjustable parameters to suit ergonomics, needs, and convenience. The operator then operates and works in the selected interaction mode, or directly uses the last parameter value or default settings.

[0244] The virtual mode interaction provided by the present invention realizes input and operation without contact and with mobility; it can completely map the movement position and orientation of the corresponding operating subject's arm and / or operator;

[0245] S2. During the preoperative planning and design phase, any of the virtual interaction methods provided by the present invention can be used, either individually or in combination.

[0246] The system supports the necessary organization and identification of the route design arrangement, description and other information of the plan, and arranges and sets the display characteristics of the physiological and pathological tissues and structures of the surgical area and parameters such as the viewing angle and orientation;

[0247] Setting response ratio parameters: The surgeon selects and sets the mapping response ratio (usually reduction) of the robot arm's movements based on the precision of the surgery, including parameters such as movement distance, rotation degree, angular change, etc.

[0248] Enable incremental overlay movement control and anti-shake misoperation control;

[0249] Set the area range and magnification ratio parameters: according to the precision of the surgery and the size of the area, select and set the focus area and display magnification ratio;

[0250] S3. During the operation execution phase, the role-integrated operation of the present invention is selected, which is the operation S4 described in 1.(2) of the above invention content 1. (2) and the role-integrated operation is selected and mapped to the role: the robot subsystem is executed;

[0251] S4. The operator operates using a binocular stereoscopic 3D coordinate measuring device at their workstation. The surgeon can operate by wearing a VR / MR / AR headset, or by sitting in front of a 3D display at the main console without a headset.

[0252] The surgeon can hold the scalpel mold in his hand to perform the corresponding action, or specify the main point as the corresponding point of the operation at the end of the gesture;

[0253] S5. The operator monitors the progress of the integrated surgery by viewing the 3D / 2D images displayed by the robot subsystem's binocular stereo vision; monitors the consistency of the master and slave arm's motion trajectories;

[0254] The robotic subsystem arm completely maps the total motion position and orientation of the operator's arm movements to perform the surgical operation;

[0255] Until the work and tasks of the operation are completed;

[0256] The arms of the robotic subsystem are usually multiple and multifunctional (including binocular stereo vision), and the selection is dynamic and can be selected and used at any time during the operation;

[0257] During surgery, if hands are inconvenient, selection and control can be performed through eye tracking, lip recognition or voice recognition;

[0258] The three-dimensional coordinate measuring device is configured with multiple points, multiple units (pairs), and distributed control, supporting simultaneous detection of multiple workstations and simultaneous operation by multiple people: doctors, assistants, and nurses can each choose to operate different robotic arms to jointly participate in, collaborate on, and cooperate in surgery (each position has a priority in the selection and control rights). The system of the present invention supports the simultaneous use of more robotic arms in the robotic subsystem to perform operations, and multiple robotic subsystems can also participate in the surgery together;

[0259] The doctor does not even need to be in the patient's presence, supporting remote surgery via an internet connection;

[0260] The system configuration and structure described are based on the current typical system application plan, but the elements can also be changed and applied: there can be other device node configurations and scales, and different system network topologies; the robot subsystem can also be an intelligent machine, equipment, and can be used in situations where it can move and execute autonomously; other three-dimensional coordinate measurement technologies and devices or equivalent methods can be used, and it can be applicable to the application needs of other industries, different characteristics and expertise requirements, and other functional uses.

[0261] 5. An application device solution for the human-computer interaction method and system optimization mode of the present invention: a smart phone, a tablet device terminal,

[0262] Currently, the most widely used mobile terminal devices, such as smartphones and tablet computers, are generally operated by touch screens for convenience. This not only requires touching the screen, but also the keyboard interface takes up screen display space, obscuring and affecting the content and viewing experience.

[0263] The human-computer interaction method of the present invention provides various virtual modes for interactive input and operation without touching the screen or contact, is portable, and has adjustable model and display conversion parameters and responses. This makes it ideal for human-computer interaction applications on tablet screen devices, ensuring consistent convenience.

[0264] The basic components of existing smartphones and tablet devices include: a computing processing unit, an interactive control unit, a display unit, a communication unit, and an interface accessory unit;

[0265] The technical features of this solution are:

[0266] The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0267] (1) Interactive control unit,

[0268] The three-dimensional coordinate measurement is preferably performed by using a binocular stereo vision device with at least one (pair) of sensors arranged horizontally on both ends of the upper frame of the front of the screen, symmetrically on the left and right, or symmetrically distributed up and down to support a horizontal screen, or three sensors on the left and right, or up and down;

[0269] (2) Computational processing unit,

[0270] The processor and software support the implementation of this technical solution, including the measurement, processing, judgment, recognition, response, and display of three-dimensional coordinates, and support for better AI integration and application, etc.

[0271] The configuration and functions of other units adopt existing technology or similar situations;

[0272] Steps:

[0273] S1. Enable the interactive mode of the human-computer interaction method of the present invention, enter the human-computer interaction interface state of the present invention, and present the last selected setting or the system default setting;

[0274] If the operator does not need to change, use it directly;

[0275] S2. Enter the settings interface and configure the operating area status and range, model, gesture mouse, conversion parameters, response ratio, area, docking, and other related settings and adjustable parameters according to the human-computer interaction method of the present invention to suit ergonomics, needs, and convenience. The operator then operates and works in the selected interactive mode.

[0276] S3. For devices with relatively small screens like smartphones, in order to reduce the screen space occupied by the model-type interaction mode (keyboard or touch screen, etc.), in step S8 of 1. (1) Setting the virtual keyboard, virtual mouse, input, and operation of the human-computer interaction method of the present invention in the above-mentioned Invention Summary 1, the following specific operations are set:

[0277] Regarding the appropriate and adjustable conversion parameters of the display style of the model described in S8, including conversion ratio, conversion angle, and viewing angle, the operator can adjust the conversion ratio as much as possible to reduce the display size, so that the space occupied by the display is as narrow as possible and can be recognized; while the construction size of the virtual model of the operation space can be relatively large, which is easy to operate and input and does not cause misoperation;

[0278] At the same time, a magnifying glass mechanism is adopted, that is, the corresponding model parts in a small area around the main point are magnified and displayed (the magnification is adjustable) to facilitate identification and observation;

[0279] The combination of the two reduces the screen space occupied by the model display;

[0280] The specific response of each key symbol can be jumped, amplified and popped up, and can be accompanied by different frequency sounds to facilitate operator monitoring;

[0281] This allows for interactive display with minimal space occupation and convenient interaction.

[0282] The human-computer interaction method of the present invention provides various virtual mode interaction methods to realize input and operation without contact or touching the screen;

[0283] The solution described in step S3 of display conversion for reducing screen space occupied by the virtual model can obviously be applied to any similar human-computer interaction input and operation scenarios to save screen space; it can also be applied to specific element changes; other three-dimensional coordinate measurement technologies and devices or equivalent methods can be used;

[0284] The solution is suitable for any flat, foldable, or flat-screen device, including human-computer interaction input and operation virtualization applications of various intelligent machines and devices.

[0285] 6. An application device scheme using the human-computer interaction method and system optimization mode of the present invention: a portable device for daily social interaction for deaf-mute people,

[0286] For deaf-mute people, the human-computer interaction method of the present invention can be used, including two main interaction modes, gesture recognition, body posture movement recognition, handwriting, manual and eye movement interaction, and even lip movement recognition, etc., making human-computer interaction convenient and easy;

[0287] However, for social interactions between people and ordinary people (who do not know how to use gestures for the deaf and mute), translation assistance is needed. A gesture corpus is established (with the help of an AI system) and stored in a portable device equipped with the interactive technical solution of the present invention for offline or online support (preferably with customized AI gesture recognition and multilingual translation). This is used to recognize the deaf and mute gestures, body movements, and handwriting of deaf and mute people in real time, and even convert the results of their manual + eye movement interaction or lip recognition into text display and voice broadcast presentation. The speech of ordinary people is recognized through voice, and handwriting, lip shape, and movement recognition are converted into text display presentation, so as to achieve continuous interactive communication. A portable device for text and voice display and broadcast with content information presented by both parties.

[0288] The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0289] (1) Hardware,

[0290] The portable device using the human-computer interaction method and system of the present invention is equipped with at least one (pair of) binocular stereo vision devices for three-dimensional coordinate measurement, a processor, a display, a sound pickup, a speaker, etc., supports wireless communication, networking and Bluetooth, and is equipped with a Bluetooth headset.

[0291] For two-way transmission and presentation, a device with two screens on both sides and a pair of three-dimensional coordinate measuring devices is selected;

[0292] (2) Software,

[0293] Support the implementation of this technical solution, including supporting the formation of interactive content information between the two parties and realizing the organization and presentation or two-way transmission and presentation, and supporting the switching of the presentation content selection screen;

[0294] Solution 1: preferably using the scroll flexible screen (mobile) terminal device solution of the present invention,

[0295] The scroll-shaped flexible screen (mobile) terminal of the present invention has a double-sided screen display structure on both the front and back sides when folded. The screen area of ​​the outer shell is made of transparent material, which is easy to stand up. The front side is equipped with at least one (pair) of three-dimensional coordinate measurement, preferably a binocular stereoscopic vision device;

[0296] When the social interaction is face-to-face, it is a two-way transmission and presentation, and both the front and back sides need to be equipped with a (pair of) three-dimensional coordinate measurement devices, preferably binocular stereo vision devices;

[0297] Steps:

[0298] S1. Activate this solution and the system automatically enters the interactive communication mode and interface. By default, gesture recognition and voice recognition are used as primary interaction methods, while other interaction methods, such as body gestures, handwriting, lip movements, and eye movements, are supplemented. These can be selected, configured, and used in real time.

[0299] S2. Social side-by-side mode, with both parties facing the same side of the flexible screen terminal (expandable); the information organization and size of the displayed screen can be set, or the last setting parameters or default settings can be used directly;

[0300] The same pair of three-dimensional coordinate measuring devices and voice pickup microphones respectively perform real-time recognition of deaf-mute gestures, body postures, and / or handwriting, lip shape, or eye movement interactions of deaf-mute people and convert the results into text and voice information for display and broadcast. They also perform voice recognition (pickup, speaker location) of ordinary people's speech and / or body postures, handwriting, lip shape, or eye movement interactions and convert them into text information for display. Both parties achieve continuous interactive communication by viewing the grouped text displayed on the screen, (ordinary people) and the voice broadcast information content.

[0301] S3. Face-to-face social mode: A flexible screen terminal is placed between two parties. Each screen displays the other party's information, or their own content is displayed together for monitoring. The organization and size of the information displayed on the screen can be set, or the last parameters or default settings can be used directly.

[0302] The three-dimensional coordinate measuring device (configured on both sides) respectively recognizes the deaf-mute person's deaf-mute gestures, body postures, and / or handwriting, lip shape or eye movement interactions in real time and converts the results into text and voice information, which is then transmitted to the other party's screen for display and broadcast (the side where the pickup microphone and speaker are located); the speech of ordinary people is recognized through voice, and / or body postures, handwriting, lip shape or eye movement recognition, which is converted into text information, which is then transmitted to the screen facing the deaf-mute person for display. Preferably, the text information presents the group content of both parties; both parties realize continuous interactive communication by viewing the information content presented by the screen text, (ordinary people) and voice broadcast;

[0303] It is best for ordinary people to pair S2 and S3 with wireless Bluetooth headsets to avoid using external speakers and microphones to pick up sound;

[0304] The front and back sides are each provided with a pair of binocular stereo vision devices for three-dimensional coordinate measurement, which can be horizontally symmetrically distributed or vertically symmetrically supported on a horizontal screen;

[0305] Solution 2: Preferably use the three-dimensional space computing type glasses device solution of the present invention,

[0306] a. The head-mounted AR / MR / VR of the present invention is a mobile head-mounted display similar to glasses, with binocular stereoscopic vision. The three-dimensional coordinate measurement is preferably performed by a binocular stereoscopic vision device configured with horizontal distribution on both sides of the upper frame of the head-mounted display, symmetrical on the left and right, and facing outwards;

[0307] Steps:

[0308] S1. Activate this solution and the system automatically enters the interactive communication mode and interface. By default, gesture recognition and voice recognition are used as primary interaction methods, while other interaction methods, such as body gestures, handwriting, lip movements, and eye movements, are supplemented. These can be selected, configured, and used in real time.

[0309] S2. A three-dimensional coordinate measuring device facing outward on the head-mounted display (HMD) performs real-time recognition of deaf-mute gestures, body gestures, handwriting, lip-reading, or eye-movement interactions (requiring the addition of a pair of binocular stereo vision), and converts the results into speech and / or text information for broadcast and / or display. It also performs speech recognition of ordinary people's speech, and / or body gestures, handwriting, lip-reading, or eye-movement recognition, and converts the results into text information for display on the mirror screen. The deaf-mute person views the on-screen text, while ordinary people communicate through voice broadcasting, enabling continuous interactive communication.

[0310] b. Based on the above, a further outward-facing mirror is configured with a transparent display screen, thereby forming a pair of transparent screens and display functions on both sides (the inner side is also a transparent display screen). There is an intermediate layer between the two sides that is normally transparent - the lens is transparent, and can be controlled to switch to opaque when interaction is required. When different contents are displayed on both sides, there is no crosstalk between the two sides. At the same time, it has a Bluetooth headset function (or is equipped with an outward-facing pickup microphone and speaker);

[0311] Steps:

[0312] S1. When this solution is enabled, the system automatically enters the interactive communication mode and interface. By default, gesture recognition and voice recognition are used as primary interaction methods, supplemented by other interaction methods such as body gestures, handwriting, lip movements, or eye movements. These can be selected, configured, and used in real time.

[0313] S2. The outward-facing three-dimensional coordinate measuring device of the head-mounted display (HMD) performs real-time recognition of the deaf-mute person's hand gestures, body postures, handwriting, lip-reading, or eye movement interactions (requiring the addition of a pair of binocular stereo vision), and converts the results into text and voice information for display on the external mirror screen and Bluetooth voice broadcast. The device also performs voice recognition of the speech of ordinary people, and converts the body postures, handwriting, lip-reading, or eye movement recognition into text information for display on the internal mirror screen for the deaf-mute person. Both parties achieve continuous interactive communication by viewing the text on the internal and external mirror screens, the information presented by the ordinary person, and the voice broadcast.

[0314] It is preferred that the text information presents the group contents of both parties for monitoring;

[0315] The portable device may be a split structure;

[0316] The front and back sides are transparent screens and the middle layer is usually transparent and can be controlled to be opaque when in use, which is the preferred solution. However, it is also possible that the two screens do not need to be transparent and in this case, or some middle layers do not need to be usually transparent, so that the wearer is worn when in use (interactive communication) and not worn when not in use;

[0317] Solution 3: Using the smartphone or tablet device terminal solution of the present invention,

[0318] The smartphone or tablet terminal application of the present invention is equipped with at least one (pair of) three-dimensional coordinate measurement, preferably a binocular stereo vision device, which is horizontally distributed towards both ends of the upper frame portion in front of the screen, symmetrically on the left and right, or symmetrically distributed up and down to support horizontal screen;

[0319] Install relevant software apps and support for the deaf-mute gesture corpus; pair a Bluetooth headset (recommended), or use the device's microphone and external speakers;

[0320] Steps:

[0321] S1. The smartphone or tablet device is held in front of both participants (shoulder-to-shoulder mode). Upon activation of this solution, the system automatically enters the interactive communication mode and interface. Deaf-mute gesture and voice recognition are the default primary interaction methods, with other interaction methods including body gestures, handwriting, lip movements, or eye movements as supplementary methods. These can be selected, configured, and used in real time.

[0322] S2. Set the organization and size of the information displayed on the screen, or use the last parameter or default settings directly;

[0323] The three-dimensional coordinate measuring device and the voice pickup microphone respectively perform real-time recognition of the deaf-mute person's deaf-mute gestures, body postures, and / or handwriting, lip shape or eye movement interactions, and convert the results into text and voice information for display and broadcast. They also perform voice recognition on the speech of ordinary people, and / or convert body postures, handwriting, lip shape or eye movement interactions into text information for display. Both parties achieve continuous interactive communication by viewing the grouped text displayed on the screen, the information content of (ordinary people) and the voice broadcast.

[0324] It is best for ordinary people to wear wireless Bluetooth headsets to avoid mutual interference of sounds;

[0325] The three solutions mentioned above can all be operated offline or online (preferably with customized AI deaf-mute gesture recognition, multilingual translation, etc.);

[0326] Among the three solutions mentioned above, it is recommended that ordinary people pair with wireless Bluetooth headsets to avoid using external speakers and microphones to pick up sound. This can also extend the range of the two, allowing for long-distance interaction between the two parties, even in different rooms (ordinary people only use voice), etc.

[0327] The headset in WiFi mode (customized version) uses the same principle, allowing both parties to use it at a greater distance;

[0328] Support remote interaction through networking (wireless, wired) connection, which requires an additional device. Ordinary people can use an ordinary smart device (monocular);

[0329] The device can be used to add, modify and maintain deaf-mute gesture language materials on a daily basis;

[0330] The speech recognition and text display function can be used by deaf people with normal throats and voices to correct their pronunciation in daily life.

[0331] 7. An application device scheme using the human-computer interaction method and system optimization mode of the present invention: a portable device for daily travel for blind people,

[0332] For blind people's human-computer interaction, the human-computer interaction method of the present invention can selectively use handwriting input interaction, gesture recognition, body posture movement recognition, lip movement recognition, combined with voice recognition input, to perform human-computer interaction control;

[0333] However, for outings, the help of guidance equipment is needed. The portable device equipped with the human-computer interaction technical solution of the present invention carries a database package corresponding to the navigation function (which can be established with the help of AI), and works in an offline or online support mode (preferably with customized AI recognition, guidance, and multi-language translation). With the support of software, it can prompt and help blind people to judge, identify, locate, and broadcast information about people, objects, roads, signs, traffic signals, etc. in real time through voice, vibration, etc., and issue path guidance, sign translation, obstacle prompts, danger warnings, etc., to help safe travel.

[0334] The system for implementing the human-computer interaction method of the present invention as described in claim 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0335] (1) Hardware,

[0336] The portable device using the human-computer interaction method and system of the present invention is equipped with at least one (pair of) binocular stereo vision devices for three-dimensional coordinate measurement, a processor, a sound pickup, a speaker, etc., supports wireless communication, networking and Bluetooth, and is equipped with dual stereo Bluetooth headsets and left and right vibration prompts;

[0337] It is recommended to use glasses with a binocular stereoscopic vision device on the upper frame: the two sides are horizontally distributed, symmetrical on the left and right, and facing outward. The left and right voice broadcast and vibration devices (which can be adjusted for front and back position and flipped up and down) are placed on the ear of the glasses frame;

[0338] (2) Software,

[0339] Support the implementation of this technical solution, record the offline database package of recognition, judgment, etc. required for the navigation function (which can be established with the help of AI), support online support work, and preferably have customized AI recognition, guidance, multi-language translation, etc.

[0340] Steps:

[0341] S1. Activate the portable device of this solution, and the system automatically enters the navigation working state, and works offline or online (customized AI recognition, guidance, multi-language translation, etc.);

[0342] S2. The device uses binocular stereoscopic vision to detect, locate, identify, process, and determine the three-dimensional coordinates of people, objects, roads, signs, and traffic signals encountered by the wearer in real time. The device then provides voice and / or vibration notifications, including real-time route guidance, sign translation, obstacle alerts, and hazard warnings, thereby assisting blind individuals in their travel safety.

[0343] The portable device may be a split structure;

[0344] Obviously, with slight modifications, adjustments, extensions and extensions to the implementation of the above technical solutions, and by extracting and strengthening the functions of identification and guidance, by strengthening the positioning, identification, target judgment, terrain, map matching, and integrating customized AI identification, judgment, and guidance, it can be easily transformed into a device solution for automatic positioning, identification, and navigation of surrounding objects by mobile devices, either online (recommended) or offline, and used for equipment service applications in situations such as autonomous navigation and automatic driving.

[0345] 8. An application device and system solution using the human-computer interaction method and system optimization mode of the present invention: virtual interactive driving and control,

[0346] Solution 1: Virtualization of other control and interaction devices,

[0347] Other control interactions refer to device control other than driving. Currently, other control interactions during driving mostly use touch screens with processors and voice interaction methods.

[0348] Touchscreen interaction typically involves a forward-facing screen located in the center of the instrument panel, ahead of the steering wheel. Due to the distance between the screen and the driver, operating the touchscreen while driving often requires the driver to lean forward and reach out to touch the screen. This significantly affects normal driving posture and road visibility, posing a safety risk.

[0349] Voice recognition technology overcomes the shortcomings of the previous paragraph; however, the driving environment, noise generated by the machine, and accents and dialects often make the accuracy, reliability, and sensitivity of voice recognition unsatisfactory and thus abandoned;

[0350] The human-computer interaction method of the present invention is a virtual mode. Input operations do not require touching the screen or contact, and are highly reliable. There is no need to lean over, which does not significantly affect the normal driving posture. Therefore, it can reach a higher level in terms of safety, accuracy, and convenience, and can fully meet the needs of control during driving.

[0351] This solution uses the virtual mode interaction method of the present invention to replace the existing method, specifically:

[0352] The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0353] (1) Hardware,

[0354] Three-dimensional coordinate measurement is preferably performed with a binocular stereo vision device, with at least one device (pair) positioned in front of the front driver or on both sides of the upper border of the screen. The device system includes a processor, supports wireless communication, networking, and Bluetooth, and is equipped with a Bluetooth headset. It is preferred to support AI integration.

[0355] (2) Software,

[0356] Support the implementation of this technical solution, and at the same time record the offline database package of recognition and judgment required for the navigation function, support online support work, and preferably have customized AI recognition, navigation, multi-language translation, etc.

[0357] Steps:

[0358] S1. Activate this solution application, and the system automatically enters the interactive mode control working state and menu interface of the human-computer interaction method of the present invention. By default, the virtual gesture mouse mode is the main interaction mode, and other interaction modes include body posture, lip shape, eye movement, handwriting or voice recognition as auxiliary modes, which can be selected, set and used in real time;

[0359] S2. The system uses a three-dimensional coordinate measuring device to identify the position of gesture mouse operations and key action instructions in accordance with the human-computer interaction method of the present invention, responds and operates the hardware and software accordingly, and optionally announces the progress content and results by voice.

[0360] S3. The operator monitors the operation and response by viewing the displayed screen or voice broadcast, and completes the operation task;

[0361] The virtual gesture mouse interaction method can also be replaced by the virtual touch screen interaction method supported by the present invention;

[0362] The configuration and arrangement described above are based on convenient implementation in combination with currently commonly used facilities, but other configurations and arrangement display forms are also possible. A transparent screen can also be used to display in a suitable area on the front glass; or head-mounted AR / MR or VR can be used for interaction and display, and the lenses of AR / MR or VR glasses are usually transparent (transparent screen). Operations are achieved through the virtual interaction method provided by the present invention, which does not require contact and has mobility.

[0363] Solution 2: Virtual interactive driving and control system,

[0364] The driving and control of this solution includes driving and control systems for vehicles in the air, on land, on water, underwater, and even in space and underground, with either manned or unmanned intelligent driving;

[0365] Taking land passenger vehicles as an example, current driving and control facilities include a steering wheel, speed lever, accelerator, brake pedal, clutch pedal, instrument panel, and various buttons. The front driver sits in a dedicated seat and controls the vehicle until the end of the journey. Because the driver's seat is crowded with these devices and instruments, a partition is created in the middle to prevent interference from neighboring passengers. As a result, regardless of the actual space in the vehicle or whether there are other passengers nearby, the driver cannot use the remaining space. They are essentially stuck in one seat and have to observe road conditions. Therefore, long-distance driving is a very tiring and arduous task for the driver.

[0366] On the other hand, autonomous driving technology has emerged, using sensors and machine-based decision-making to take over the driver's control. Because these systems are based on pre-designed strategies, they are generally effective under normal circumstances. However, complex road conditions, especially those involving unexpected or unforeseen events, can make it difficult to guarantee the correct judgment. This has occurred repeatedly, posing significant risks to both vehicles and drivers at highway speeds. Therefore, given current technological advancements, intervention is still necessary. Fully entrusting autonomous driving to the system is risky, especially at high speeds or in complex road conditions. Therefore, manual intervention is often preferred, and existing physical systems for human drivers will continue to be retained and used.

[0367] The human-machine interaction method of the present invention uses a virtual mode of interactive control, which does not require contact and is mobile. It can completely and consistently map the total movement and orientation of the operator's arm and / or the operator's movements, accurately executing the operation until the machine system completes the work or task. The application of the human-machine interaction method of the present invention is expected to replace existing physical facilities operated by humans, significantly expanding the scope of activities.

[0368] The virtual mode interactive control method of the human-computer interaction method of the present invention is applied to vehicle driving, supporting control in a standing position, supporting free movement and driving in the vehicle space, and even lying or recumbent driving, truly achieving free and comfortable driving; the system supports the better combination of automatic driving and AI intelligent driving technology, thus bringing real ease and freedom to the driver throughout the entire driving journey;

[0369] The virtual interactive driving and control system refers to the systematic control of driving and control in the form of a virtual interactive system of the present invention, which fully controls the functions and tasks of the passenger vehicle, including driving and other control functions;

[0370] The basic components of the system include: computing and processing unit, interactive control unit, display unit, control unit, feedback unit, and electromechanical system unit;

[0371] The technical features of this solution are:

[0372] The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably comprises the modules and corresponding functional configurations described in the binocular stereoscopic vision method, including:

[0373] (1) Interactive control unit,

[0374] Three-dimensional coordinate measurement preferably uses binocular stereo vision devices to be deployed in appropriate locations in passenger vehicles for distribution control, including in front of the front driver and other required and appropriate locations; the virtual mode interactive system of the present invention replaces the existing driving control facilities and equipment;

[0375] The existing method can be simplified and equipped as a spare on the side, which can be pulled out when in use and pushed aside when not in use, including a simplified steering wheel (either a left or right semi-arc or a large semi-circular type or other styles including handles or keys), a speed lever or key: acceleration, deceleration (braking) or even shifting gears, eliminating the accelerator, brake, and clutch pedals; and eliminating the fixed middle partition in the front row;

[0376] (2) Computational processing unit,

[0377] The processor and software support the application implementation of the present invention and the implementation of this system solution, including: three-dimensional coordinate measurement, processing judgment and recognition, tracking and matching, response, operation, 2D / 3D display, intelligent judgment, support for incremental overlay control and anti-shake misoperation control, support for AI integrated application; support for wireless communication, networking and Bluetooth, equipped with Bluetooth headsets and speakers, etc.

[0378] This includes supporting the direct calling of the motion space position coordinates, trajectory, and rotation parameters of the three-dimensional coordinate measurement unit to map the corresponding execution equipment and machine's response actions, thereby replacing and extending the action work;

[0379] (3) Display unit,

[0380] Display the information that needs to be displayed by the three-dimensional coordinate measurement unit, calculation processing unit, etc. accordingly;

[0381] Forward visual display during driving, binocular stereo vision 2D / 3D display at the operating station (optional one-way corrected 2D display or two-way 3D display), etc.

[0382] Including: display of steering response, speed response effect (direction angle, speed value), display of status indication of steering control model, speed control model operation, and information display of existing solutions;

[0383] (4) a control unit,

[0384] Fusion and integration of the present invention and existing methods (basic solutions) to complete the functions of this system solution;

[0385] (5) Feedback unit,

[0386] Fusion and integration of the present invention and existing methods (basic solutions) to complete the functions of this system solution;

[0387] (6) Electromechanical system unit,

[0388] The unit configuration and functions are existing or similar;

[0389] Process steps:

[0390] S1. Start the system. The three-dimensional spatial position coordinate measuring device is ready. The operating system displays a startup menu. After the operator enters, the system automatically enables the interactive mode and interface of the human-computer interaction method of the present invention, presenting the last selected setting parameter value or the system default setting;

[0391] The present system applies the mapping response of direct operation integrated into the operation interaction mode of the present invention, that is, the mapping response of direct operation integrated into step (1) of operation S4 described in 1.(2) of the above invention content 1 is controlled and mapped to the execution device and machine response: the control unit is used to drive the electromechanical system unit;

[0392] S2. If the operator does not need to change the setting parameters, use it directly to enter the driving and control interface, and then proceed to step S4 in sequence;

[0393] S3. Enter the settings interface and adjust the virtual interactive driving adjustable parameters according to the human-computer interaction method of the present invention to select and set the appropriate settings to suit the driving and control needs, ergonomics, and the convenience of adapting to the user's own characteristics;

[0394] The modeling operation area and the inner model (including position, size, and angle) are located between the upper part of the legs and the lower part of the chest in a sitting position, so as to not block the view in front and to be conveniently selected. They are usually located above the legs and near the armrests.

[0395] The system supports standing driving control and mobile driving control (cancel the middle fixed partition)

[0396] It can be placed in any position in front of the operator that is within reach of both hands, usually between the shoulder and chest and the hips, as appropriate and as needed;

[0397] Adjust the system's (default) steering wheel and speed lever models, including fine-tuning sensitivity and left-right distribution, including their relative positional relationships. The left-right distribution can be dynamically adjusted in real time.

[0398] Set the mapping response parameters for the operator's actions, including the proportional parameters of rotation angle change or movement distance. The operator can choose specific settings based on feature preferences and needs; enable incremental overlay control, anti-shake and misoperation control, intelligent judgment, etc.

[0399] Choose to set three specific gestures and one adjustment gesture, representing: ready, go (optional for left and right hands), and release. Adjust the position of the modeling area. The three specific gestures can be rotated around the wrist and elbow and recognized. The go gesture can be combined with the ready gesture.

[0400] Set the disengagement distance interval generated by the disengagement operation;

[0401] For the operation area position determination mode, set the model adaptive parameters, docking and pasting elastic distance: within the modeling operation area, the distance range, docking position (left and right hands can be different) and spacing between the model and the operating body. The elastic distance range for model pasting is the separation distance.

[0402] In the follow mode, the model automatically docks (matches) with the arm and follows it. You can set the distance range for the model to follow, usually within the modeling operation area.

[0403] For the automatic switching mode, set the switching conditions or parameters, usually: automatically switch to follow mode when standing or moving driving, and switch to position-determined mode (the area above the armrest) when sitting and holding the armrest;

[0404] Set display parameters, including conversion ratio, conversion angle, area range, position, brightness, perspective, edge brightness, color, and other parameter values. The operator can adjust the settings as needed;

[0405] The above parameters are valid for the entire modeling operation area;

[0406] The selection setting of whether the operation subject, steering wheel model, speed lever model, or other models are displayed or hidden during driving in a non-operation change state;

[0407] Control and adjustable parameters of other vehicle equipment, sounds, etc.

[0408] The present invention supports providing steering control that can be controlled by other models and operating methods instead of using a steering wheel model. Speed ​​control can also be controlled by other models and operating methods. The relevant model mode is selected and set accordingly.

[0409] After the setting is completed (stored for standby use), the operator can drive and control it;

[0410] Operation:

[0411] The S4 operator uses a binocular stereo vision device to measure the three-dimensional coordinates of his or her workstation for interactive operation, or wears an MR / AR or VR headset (equipped with a binocular stereo vision device) for control;

[0412] After the operator takes the driving position (sitting posture, position determination mode, effective range of three-dimensional coordinate measurement), the system automatically detects (identifies), locks, and tracks the operator (hands), and accordingly determines the position in the operating area or prepares for following or automatically switching modes;

[0413] The operator's arm enters the modeling operation area, is positioned near the model and within the docking distance, and then makes a ready-to-operate gesture. The model automatically docks (matches) its arm in the docking area (left or right hand), and is in an initial posture ready for operation.

[0414] The operator operates by performing gestures (left or right), standard operating movements, operating a steering wheel model, a speed lever model, or other models and methods;

[0415] Use ready-to-operate gestures and disengage-from-operation gestures to distinguish and exclude non-operation periods;

[0416] The system compares and identifies the operating gestures, standard operating actions, displacement positions, etc. with the set gestures, action modes, displacement position ranges, etc., distinguishes them from unconscious actions (daily habits, physiological actions, etc.), and processes abnormal non-standard actions and jittering actions with intelligent recognition and judgment to prevent misoperation and anti-shake control;

[0417] If the position of the operator (hands) changes, the system will automatically dock the model intermittently;

[0418] The operator can monitor driving and manipulating actions, positions or instructions in real time by viewing 2D / 3D displays showing execution, response, operation results, progress, and feedback. The operator can also choose to have the progress content and results reported in voice mode.

[0419] The display may be any suitable 2D / 3D display scene format, including a transparent screen placed appropriately on the front glass, or a head-mounted MR / AR or VR display;

[0420] This solution of the present invention provides virtual control of seated driving;

[0421] It also provides virtual control that supports standing driving;

[0422] and or mobile driving virtual control;

[0423] When equipped with forward-looking vision, the driver can even drive from the back seat, or perform virtual driving in a reclining or lying position.

[0424] response:

[0425] S5. The computing and processing unit processes, determines, and identifies the interactive control unit's operating actions, displacement positions, or operating instructions, generating control and instruction information that is transmitted to the control unit. This information controls the actuators of the electromechanical system unit to execute responses and operations accordingly. The feedback unit collects operational data and feeds it back to the computing and processing unit and the control unit for loop adjustment and control. This information is displayed on the display unit, completing the execution response and operation process corresponding to the operator's arm movements (rotation angle, control position, etc.) and / or body posture instructions, thereby achieving interactive driving and control.

[0426] For steering operations of the steering wheel model, the spatial position coordinates, trajectory, and rotation angle parameters of the operator's action are combined with the orientation parameters; for speed operations of the speed lever model, the spatial displacement position and trajectory parameters of the operator's action are used; and for other operating gestures, corresponding processing, judgment, and validation are performed, and then mapped to the response actions of the actuator device: steering, gear position, speed, etc.

[0427] The system continuously cycles through this process, controlling the electromechanical and mechanical equipment to continuously correspond to the total motion rotation angle and control position mapped by the operator's arm movements, and respond to the operation instructions until the driving journey task is completed;

[0428] S6. Intelligent driving assistance and navigation features are recommended, working either online or offline (preferably with customized AI assistance, recognition, navigation, and multilingual translation). The software supports route planning for travel tasks, displaying necessary route information and signage, and scheduling work and rest.

[0429] The system can use any interactive mode of the human-computer interaction method of the present invention to select, set, modify and maintain default items and parameters for driving and control on a daily basis;

[0430] The direction model and speed rod model can also be replaced by the direction rod or speed rod or key of the real remote control method;

[0431] For other equipment control besides driving, the interactive solution or spirit described in the above solution 1 can be easily integrated into this solution for combined control;

[0432] The virtual interactive system controls driving and operation, but the system still uses manual driving and operation, and manual intervention. It supports combination with intelligent autonomous driving technology for assistance, and AI intelligent driving technology for better assistance.

[0433] The autonomous navigation and autonomous driving variants of the enhanced recognition and navigation functions of the portable device for daily travel for blind persons described in Summary 7 above are combined with this solution to form vehicle autonomous driving and AI autonomous intelligent driving (online support or offline) functions, allowing for virtual driving, control assistance, or automatic and autonomous intelligent driving (autonomous navigation, automatic avoidance, self-driving, berthing, etc.);

[0434] In the driving and controlling virtual mode system described in this solution, the operator controls the system through the virtual mode interactive mode provided by the present invention, which does not require contact and has mobility, and is therefore particularly suitable for use in aerospace under weightless conditions;

[0435] The configuration and structure of the system are system application solutions based on the current typical and common land passenger vehicles, but it can have other equipment node configurations and system network topology structures, and be applicable to various vehicle system applications in space, air, land, sea, underwater, underground, etc.; it can carry out element change application, equivalent substitution and variant implementation, and can adopt other three-dimensional coordinate measurement technologies and devices or equivalent methods. It can be applicable to the application needs of other industries and occasions, and can be applicable to different characteristics and expertise requirements and other functional uses.

[0436] This invention represents a new technological development trend.

[0437] 9. An application technology solution of the present invention's human-computer interaction method and system optimization mode: virtual mode remote control (wired or wireless) control and manipulation work, characterized by the specific configuration, method and steps of the solution are as follows:

[0438] The remote control includes a typical panel (including a touch screen), a remote handle joystick, and other similar functions. The characteristics are that the operator can control and operate the work through the remote control device, and the feedback information can be perceived, the response and execution effect can be felt, and it is usually accompanied by a display;

[0439] The basic components of the system include: computing and processing unit, interactive control unit, machine "vision" unit, display unit, control unit, feedback unit, and electromechanical system unit;

[0440] The technical features of this solution are:

[0441] 1. The system for implementing the human-computer interaction method of the present invention as described in Content 2 of the present invention preferably has the configuration of the modules and corresponding functions described in the binocular stereoscopic vision method, and in this application technical solution, realizes the following functions:

[0442] (1) The virtual keyboard interaction method or virtual touch screen interaction method of the present invention can be specifically applied to realize various required remote control panels, center console control panels, and other similar devices with various panel types (including touch screens) and functions;

[0443] (2) Applying the interactive operation mode of the present invention to realize remote handle or other similar functions: i. Implementing the control function by applying the mapping of (1) integrated direct operation b. Mapping to the corresponding response of the control drive unit of the execution device or machine;

[0444] ii. (2) Integrate character operation to realize the manipulation function: mapping is applied to the character --- robot, manipulator, processing table and other subsystems --- with machine vision unit for execution;

[0445] (3) Using other interaction methods of the present invention, including gestures, body movements or lip movements, handwriting recognition, eye movement interaction, etc., to implement the operation instructions required for the above-mentioned control;

[0446] Each of them can be used individually, in combination or in conjunction with each other, so as to fully realize and complete the purpose of replacing the interactive control and manipulation work of the real remote control device with the virtual mode interaction method of the present invention;

[0447] The unique technical features of the human-machine interaction method and system of the present invention support easy simultaneous interactive control of multiple channels and multiple groups of objects and machines, and can realize remote control and manipulation.

[0448] 2. The system configuration of this solution with the modules and corresponding functions mentioned above specifically includes:

[0449] (1) Interactive control unit,

[0450] The three-dimensional coordinate measurement preferably uses a binocular stereo vision device that is arranged in a suitable range and position in a specific area and designated workstation in the working environment space. It can be multi-point and distributed control placement; including:

[0451] Mobile workstation: VR / MR / AR head-mounted mobile display, 3D coordinate measurement device configuration: horizontally distributed on both sides of the upper frame of the head display, symmetrical on the left and right, facing outward;

[0452] Fixed workstations: including control desks / rooms, on-site, operator workstations, and detection locations;

[0453] (2) Machine "vision" unit,

[0454] The working "visual" configuration of the remotely controlled object equipment and machines is also preferably a binocular stereo vision device;

[0455] Including: subsystems such as robots, robotic arms, processing tables, and forward-looking vision, etc.

[0456] (3) Computational processing unit,

[0457] The processor and software support the application implementation of the present invention and the implementation of this system solution, including: three-dimensional coordinate measurement, processing, judgment and recognition, execution response, operation, machine "vision", interactive 2D / 3D display, distributed detection and fusion processing, intelligent judgment, support for incremental overlay control and anti-shake misoperation control, support for AI integration and application; support for wireless communication, networking and Bluetooth, equipped with Bluetooth headsets and speakers, etc.

[0458] Supports simultaneous interactive control and manipulation of multiple channels, multiple groups (clusters), and distributed devices;

[0459] Including supporting direct call of the motion space position coordinates, trajectory, rotation, orientation, and other parameter values ​​of the three-dimensional coordinate measurement unit to the corresponding manipulation of the directly applied execution operation action;

[0460] (4) Display unit,

[0461] Display the information that needs to be displayed by the three-dimensional coordinate measurement unit, calculation processing unit, etc. accordingly;

[0462] Binocular stereoscopic 2D / 3D display (optional: one-way rectified 2D display or two-way 3D display) of the remotely controlled equipment (robots, robotic arms, processing tools, and other subsystems, as well as forward-looking vision systems) and various panel and joystick models of the remote control device, as well as switching to other binocular stereoscopic displays; control, control response, execution instructions, information display, etc.

[0463] It can be a common display of the working "visual" of multiple channels and multiple groups (groups) of machines;

[0464] Or a simplified version of the simple panel with display-type control, operation setting status, response operation (feedback) parameter value and other information indication display;

[0465] (5) Other units

[0466] The configuration and function of the control unit, electromechanical system unit, and other units are to integrate and combine the present invention with existing methods or similar situations;

[0467] Steps:

[0468] According to the content steps of the human-computer interaction method of the present invention, the functional contents described in the technical feature 1 of the above scheme are respectively implemented and operated in a corresponding manner to complete the various tasks of remote control and manipulation;

[0469] (1) Realize various panel functions and operations by means of a virtual keyboard or a virtual touch screen;

[0470] (2) Implement the control, manipulation, and operation of the remote control object (joystick, etc.) by integrating the operation interaction mode; enable incremental superposition control and anti-shake misoperation control as needed, and make intelligent judgments;

[0471] (3) realizing the effects and operations of operating instructions by other interactive means, including recognition of gestures, body movements or lip movements;

[0472] Feedback and display content, including panels, joystick remote handles or similar functional components, mapped corresponding operating actions, responses, executions, and other information screens are provided to the operator for monitoring, forming an interactive process until the task is completed;

[0473] Common system-integrated control panel types can be considered as wired or wireless remote control short-range (on-site) scenarios. The above applies to: simplified version, simple feedback (near-field) type;

[0474] The virtual mode interactive method of the present invention realizes remote control, and can adjust the style and type according to different characteristics, expertise requirements, and different application scenarios. Various configurations and different types can be switched and called in real time. It is applicable to: intelligent production, manufacturing, processing, smart office, smart home, smart service, entertainment and other fields to realize the virtual mode of corresponding control and operation of remote control devices (including integrated control panels);

[0475] The virtual mode remote control and manipulation interaction provided by the present invention does not require contact, has mobility, and can be distributed and multi-group controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0476] FIG1 is a system structure block diagram of a human-computer interaction method based on three-dimensional coordinate measurement according to the present invention;

[0477] 2 is a schematic diagram showing the effect of a virtual keyboard and virtual mouse interaction mode in a two-dimensional traditional computing desktop scenario according to a preferred embodiment of the present invention in Example 1;

[0478] 3 is a schematic diagram showing the effect of the scene-integrated operation mode in a desktop scenario of a two-dimensional traditional computing type according to a preferred embodiment of the present invention in Example 1;

[0479] 4 is a schematic diagram showing the effects of a virtual keyboard and virtual mouse interaction mode in a wearable scenario of a three-dimensional space computing type according to a preferred embodiment of the present invention in Example 2;

[0480] FIG5 is a schematic diagram showing the effect of the wearable scenario-integrated operation mode of the three-dimensional space computing type according to the preferred embodiment of the present invention in Example 2;

[0481] Figure 6 series a, b, and c are schematic diagrams of a specific embodiment 4 of an application of the present invention: a scroll-shaped flexible screen and housing structure of a scroll-shaped flexible screen (mobile) terminal;

[0482] Figure 7 series a and b are schematic diagrams of a specific embodiment 4 of an application of the present invention: a support arm and a horizontal bracket structure of a scroll-type flexible screen (mobile) terminal;

[0483] Figure 8 series a, b, and c are enlarged schematic diagrams of a specific embodiment 4 of an application of the present invention: a main unit and stand structure and a stand bracket and support arm cross bracket structure of a scroll flexible screen (mobile) terminal; DETAILED DESCRIPTION

[0484] The following describes in detail the method and system for realizing the human-computer interaction of the present invention with preferred specific implementation methods, and illustrates the technical solutions summarized in the claims with examples in combination with the drawings. The specific constituent elements, steps, parameters, structures, etc. shown in the examples and drawings are only examples and illustrations, and are for support and explanation but not intended to limit the present disclosure. The present invention can also be implemented by other element changes, equivalent substitutions, modifications and other topological structures that are different from those described herein. In addition, the constituent elements in the following embodiments that are not recorded in the independent claims representing the highest-level concepts are explained as arbitrary constituent elements.

[0485] FIG1 is a block diagram of a system structure of a human-computer interaction method based on three-dimensional coordinate measurement according to the present invention, comprising:

[0486] The three-dimensional coordinate measurement module is used to measure the spatial position coordinates and posture parameters of the operating subject and / or the operator's operating actions in human-computer interaction: the measured data is processed and calculated accordingly to obtain the spatial position coordinate parameters of the operating subject and / or the operator's operating actions, mainly the three-dimensional coordinates and posture parameters of the main points (keystroke points, contact points, focus points, etc.) and / or joint points, and further obtain related parameter values ​​such as trajectory, posture, and action degree;

[0487] The calculation and processing module is used to process and calculate the parameter data information of other modules, make judgments, identify, execute responses, and operate;

[0488] It includes: extracting the measurement data of the three-dimensional coordinate measurement module and making judgments: comparing with the regional coordinates, or patterns, or paradigm parameters in the pre-built (pre-set library, table), making corresponding judgments and identifications based on the degree of coincidence or fit, and executing (hardware, software) corresponding responses and operations;

[0489] Responsible for the construction and setting of the virtual interaction mode of the system according to the human-computer interaction method based on three-dimensional coordinate measurement as described in the above invention content 1;

[0490] A display module is used to display information required for displaying the measurement, processing, calculation, identification, response, operation, and effects of the three-dimensional coordinate measurement module and the calculation processing module, in accordance with the display content and display requirements of the human-computer interaction method based on three-dimensional coordinate measurement described in the first aspect of the present invention, for monitoring and viewing by the operator;

[0491] The above modules jointly realize the various virtual mode interaction input operation functions of the human-computer interaction method of the present invention, thereby further completing the work tasks of the system.

[0492] The system implementation process includes the following steps, taking the preferred binocular stereo vision method as an example:

[0493] S1. Three-dimensional Coordinate Determination: Configure a binocular stereo vision system with two identical camera sensors at different viewpoints, horizontally distributed and symmetrical (typically, with the X-axis in the same plane and symmetrical about the Y-axis). The parameters and angles are such that the overlapping field of view appropriately covers the operating area. If necessary (e.g., for the first time on a model), calibration is performed to obtain corrected values ​​for the internal and external parameters.

[0494] S2. The system constructs various interaction modes (as default) according to the human-computer interaction method based on three-dimensional coordinate measurement described in Summary 1 of the Invention;

[0495] S3. According to the steps, the operator performs various virtual interaction mode parameters to adapt to ergonomics and needs, convenient settings;

[0496] S4. The operator performs input and operation in the selected and enabled virtual mode interaction mode;

[0497] S5. A computing and processing device (including at least one computing processor) processes the images captured by the left and right cameras simultaneously, matching feature points such as principal points. Based on triangulation methods and epipolar constraint relationship rules, the computing and processing device can calculate (measure) the spatial position coordinates of the operating subject and / or the operator and their actions, primarily the three-dimensional coordinates and posture parameters of the principal points (keystroke points, contact points, focus points, etc.) and / or the joints of the operating subject's skeletal model, as well as values ​​of related parameters such as trajectory, posture, and degree of movement.

[0498] S6. Compare the coordinates, patterns, or parameters of the pre-set database or table with the coordinates, patterns, or paradigms. Based on the degree of coincidence or fit, make a corresponding judgment or identification, and execute (hardware or software) a corresponding response or operation.

[0499] S7. A display device for displaying information required to display the measurement, processing, calculation, identification, response, and operation of the three-dimensional coordinate measurement device and the calculation and processing device, in accordance with the display content and display requirements of the human-computer interaction method based on three-dimensional coordinate measurement described in Summary 1 above;

[0500] The image is provided by two identical camera sensors of a binocular stereo vision device, and for a two-dimensional conventional computing display, either the left or right image is rectified and displayed;

[0501] The method includes: mapping and displaying the operating subject and / or the operator and their actions on the screen, and marking the main points, / or joint points, and / or skeleton models at the corresponding parts;

[0502] Depending on the situation and needs, the screen can be displayed as a video image of the operating subject and / or the operator, a virtual animation or a picture-based appearance, or a 3D reconstruction display; the simplest way is to map the body contour, mark the main points, / or joints, or a skeletal model display;

[0503] For gesture mice, only the display cursor needs to be recognized;

[0504] Operations for integrating characters appear as design parameters of the model;

[0505] Display according to the display requirements of the region or the display parameters of the spatial scene;

[0506] S8. The operator monitors the input operation by viewing the operation scene and the response display screen; realizes the human-computer interaction operation function;

[0507] The operator evaluates the operation situation, operation response, and operation results. The system follows the operator's operations and instructions to complete the control work and tasks required by the operator.

[0508] The two identical cameras and sensors at different viewpoints may be arranged vertically or in other distributions, or in an asymmetrical arrangement, or may be more than two sensors, or even in multi-point, multi-unit (pair), or distributed control arrangements. Other distribution, arrangement, and quantity options may be chosen based on specific needs, with corresponding changes in computational effort and configuration.

[0509] Similarly, according to the needs of the use environment, select the appropriate camera sensor, applicable frequency band, illumination and other technical parameters of the binocular stereo vision device;

[0510] Alternatively, light spots of various frequency bands and cursors may be used to mark or replace the main points and key points of the operating subject.

[0511] The system operation process includes the following steps:

[0512] S1. After the system device (including both 2D conventional computing and 3D spatial computing) is powered on, the connected 3D spatial position coordinate measurement device is ready for operation. The system simultaneously displays a startup menu icon, and the operating system automatically boots into the interaction mode last selected and set by the operator, or into the system default interaction mode. If the operator does not wish to change, the computing device and system are controlled directly using this interaction mode and its settings.

[0513] S2. If the operator needs to change the interaction mode, confirm (click) to enter the startup menu, that is, start the selection and setting process; the startup menu list lists various interaction modes of the present invention, the operator selects and enables the desired interaction mode as needed; if the operator does not need to change the adjustment settings, then you can directly select and enable it;

[0514] The computing processing device of the system is provided with a hot key (with direction), which can be pressed at any time in the boot state to force the system startup menu to pop up on the displayed desktop, providing process operations for selection and setting the interactive mode, etc.

[0515] S3. If you need to change the setting parameters, you can further confirm the startup process to enter the setting submenu, in the setting interface according to the content of the human-computer interaction method of the present invention, set the steps, according to the needs of the system to provide the relevant interaction mode including parameters of various aspects of the adjustment settings, until the setting is completed;

[0516] S4. The operator performs interactive input and operation in the newly enabled interactive mode, controls the computing device and system, and displays the interactive and operational scenes through the system;

[0517] S5. The system is based on the three-dimensional spatial position coordinate measurement, and the computing and processing device processes, judges and identifies, and makes corresponding responses and corresponding operations;

[0518] The operator and the system continuously cycle through the interactive steps S4 and S5, and monitor the scenes and effects displayed by the system until the required work and tasks are achieved and completed;

[0519] S6. Confirm the shutdown menu item and end.

[0520] Example 1: The human-computer interaction method and system of the present invention preferably adopts a binocular stereo vision method as an example of a two-dimensional traditional computing type desktop scenario.

[0521] Figures 2 and 3 are schematic diagrams showing the effects of two main interaction modes for a typical desktop computer scenario of a two-dimensional traditional computing type. Preferably, the binocular stereo vision sensors are distributed horizontally, symmetrically, and facing the operator at both ends of the upper frame of the display screen.

[0522] The display is provided by a binocular stereo vision device, and either the left or right image is selected for correction and then displayed;

[0523] During the above system operation Based on the steps, this embodiment also specifically includes:

[0524] FIG2 shows the interactive mode of using a virtual keyboard and a virtual mouse in the case of a desktop display screen; if no parameter setting needs to be changed, the operator can use them to input and operate;

[0525] If the settings need to be changed, the operator enters the S3 setting process step described in the above system operation process and makes settings on the setting interface;

[0526] The operator adjusts the parameters used in the last setting of the system or the default setting parameters as needed according to the relevant steps in the setting content of the virtual keyboard and virtual mouse interaction mode of the human-computer interaction method of the present invention until the setting is completed;

[0527] After the settings are completed, the operator can use the newly enabled virtual keyboard and virtual mouse interaction mode to input and operate:

[0528] On the virtual keyboard model, the operator performs a "keyboard tapping" action to input and operate on it;

[0529] Mouse operation functions are realized by gesture mouse recognition.

[0530] The default settings for the virtual keyboard and virtual mouse (gestures) provided by the system have the following characteristics for desktop use:

[0531] The modeling operation area is located on the front table at elbow height, usually above the base of the display screen, at an appropriate distance relative to the size of the display screen (optimal measurement range);

[0532] The style and key response of the virtual keyboard model are organized, laid out, and configured by default according to the current standard keyboard style (102 / 104 or 78 / 87, etc.). The size (length and width) is slightly larger than the actual keyboard model. It can be ergonomically configured to facilitate operation. (The angle is generally perpendicular to the arm or parallel to the line of sight.)

[0533] The display angle is the rear viewing angle, the angle is parallel to the screen surface, the display color parameters are configured according to the usual habits, the display ratio (as small as possible is appropriate), and the default is to display according to the display requirements of the region, that is, the system parameters;

[0534] The default action's ready action gesture and the release action gesture are:

[0535] Put your palms together and stretch them flat and raise them up at a certain angle (such as Left and right), definition represents: ready operation,

[0536] The double-handed (or single) fist gesture is defined to represent: disengagement from the operation;

[0537] The default settings for keyboard model adaptation parameters are:

[0538] Longitudinal docking distance range: ≤10cm, docking position: wrist of both hands (first operation axis position), spacing: natural spacing (approx. ),

[0539] Gesture mouse, the default gesture settings given in this embodiment are:

[0540] The gesture of stretching out five fingers of one hand (or gathering the five fingertips) is defined as: cursor, with the middle fingertip as the main point and the corresponding position of the key point.

[0541] The single index finger and thumb touch gesture is defined as: left click,

[0542] The gesture of touching the middle finger (or ring finger) with the thumb is defined as: right click,

[0543] Using cursor gestures, the up and down flipping actions are defined to represent: turning pages up and down;

[0544] For a gesture mouse, the system only needs to identify the display cursor, recognize and determine the key click gesture, and respond accordingly;

[0545] The default value of the response amplitude ratio of cursor operation is 1:1, (including the response amplitude ratio of up and down page actions); (Note: the 1:1 here refers to the ratio of the response amplitude of existing devices)

[0546] By default, the (cursor) anti-shake misoperation control is enabled;

[0547] In fine-tuning situations, you can also enable incremental overlay control of (cursor) movement;

[0548] The operator and the screen of their actions are displayed in correspondence, with the default display being the video image and outline of the finger joint area of ​​the operator's palm, generally appropriate color parameters, and the main points (and / or joint points, and / or skeletal model) marked;

[0549] The scale is displayed according to the display requirements of the area, that is, the system display parameters, and the angle and viewing angle are consistent with the model.

[0550] FIG3 shows a desktop display screen scenario in which the interactive mode of integrating display space operation ((1) integrating direct operation a direct application) is selected; if no parameter setting needs to be changed, the operator can operate in this manner;

[0551] If the settings need to be changed, the operator enters the S3 setting process step described in the above system operation process and makes settings on the setting interface;

[0552] The operator adjusts the system's last set parameters or default settings as needed, following the steps in the above-mentioned human-computer interaction method of the present invention for integrating the display space operation interaction mode into the setting content, until the setting is complete;

[0553] After the settings are completed, the operator can operate in the interactive mode of integrated display space enabled by the new settings.

[0554] The default settings for integrating into the display space provided by the system have the following characteristics for the desktop scenario:

[0555] By default, the entire operation area is a single integrated operation area;

[0556] The default mapping relationship between the merged body and the entity (operation subject) is:

[0557] For (2) the character integration operation, the integrated body is displayed with the design parameters of the character model including color;

[0558] For (1) integration of direct manipulation, the integration body is to use the video image and outline of the operating subject, i.e., the hands and wrists, usually with appropriate color parameters, and to mark the main points, joint points, and / or skeletal model display;

[0559] The mapping conversion ratio is displayed according to the regional space display parameter value, and the conversion angle is that the desktop surface is parallel to the screen surface;

[0560] The default mapping position is that the entity's initial position is on the table surface, at elbow height and shoulder width;

[0561] The starting position of the fusion body is set at the two corners of the lower edge of the front field of the screen, and is displayed in the above-mentioned proportions, angles and colors;

[0562] The activation confirmation and exit confirmation of the object components and objects are as follows: the default in this embodiment is to select the gesture of the gesture mouse to operate;

[0563] Confirm the scope of the activated area. By default, it will be displayed on the original construction main page of the component or object. By default, it will occupy the main display area. The color of the activated display will be configured according to the usual appropriate habits. The default main display area is the entire screen area.

[0564] The default action's ready action gesture and the release action gesture are:

[0565] The five fingers of one hand are stretched together to form a gesture, which means: ready operation.

[0566] The single-handed fist gesture is defined as: disengagement from the operation;

[0567] The default response ratio of the subject action is 1:1, including: movement distance, rotation degree, and angular change parameters;

[0568] Anti-shake and anti-misoperation control are enabled by default;

[0569] Incremental overlay control can also be enabled for precision, fine-tuning, etc.

[0570] Other interaction modes can be easily implemented by following the steps described in the relevant content of the human-computer interaction method of the present invention and referring to the above two main interaction mode embodiments.

[0571] The various interaction modes are used alone or in combination to complete the work;

[0572] The operator monitors the input operation through the operation displayed on the display screen and the corresponding interactive scene; the display can be rectified after the left or right image is displayed;

[0573] The computing and processing devices and systems process, judge and identify, make corresponding responses and operate accordingly until the work and tasks are completed.

[0574] The two-dimensional traditional computing type display screen includes various flat-panel, quasi-flat-panel, foldable and other display screen forms.

[0575] The algorithm for anti-shake misoperation control and incremental superposition control provided by the specific embodiment of the present invention is as follows:

[0576] Anti-jitter control: eliminate jitter by taking the average of adjacent amplitudes as the adopted value (e.g., taking the average with the previous value), or taking the average of multiple values ​​within a certain time interval as the adopted value;

[0577] Anti-misoperation control: If finger manipulation is required (and cannot be performed with a fixed gesture), the system uses ready-to-operate gestures and disengage-from-operation gestures to distinguish and eliminate non-operation periods. At the same time, the system distinguishes between manipulation and unconscious movements (daily habits and physiological processes, etc.), and handles unusual, non-standard movements and jittering movements with intelligent recognition and judgment to prevent misoperation.

[0578] If the operation is based on the trajectory of the action, position mapping, and does not require finger manipulation (can be performed with a fixed gesture), then further operation gestures can be performed to confirm that the operation is valid and perform the operation;

[0579] The default proceed gesture is replaced by the ready gesture, and the operator can also choose to set a new proceed gesture;

[0580] Incremental overlay control:

[0581] Response ratio: If the setting is m:n, it means that the action parameter value of the operator is n, and the action parameter value of the mapped response is m;

[0582] When incremental superposition control is enabled, it means that each mapping response parameter mi becomes an increment and is superimposed, and the final response parameter is the cumulative amount, that is, M = ∑mi;

[0583] The parameters include: movement amount, rotation degree, angular change degree, and other parameters, and the incremental superposition control principle is the same.

[0584] The operator can modify and adjust the above as needed.

[0585] Example 2: An example of a wearable scenario of a three-dimensional space calculation type of a binocular stereo vision embodiment of the human-computer interaction method and system of the present invention.

[0586] Figures 4 and 5 are schematic diagrams showing the effects of two main interaction modes for wearable 3D computing. Preferably, the binocular stereo vision sensors are located at both ends of the upper frame of the glasses, horizontally distributed, bilaterally symmetrical, and facing outward.

[0587] The display is provided by a binocular stereo vision device, and the two-way image realizes a three-dimensional stereoscopic panoramic (immersive) display with full depth, virtuality and reality fusion;

[0588] During the above system operation Based on the steps, this embodiment also specifically includes:

[0589] FIG4 shows an interactive mode of using a virtual keyboard and a virtual mouse in a wearable 3D panoramic immersive display. If no parameter setting needs to be changed, the operator can use them for interactive input and operation.

[0590] If the settings need to be changed, the operator enters the S3 setting process step described in the above system operation process and makes settings on the setting interface;

[0591] The operator adjusts the parameters used in the last setting of the system or the default setting parameters as needed according to the relevant steps in the setting content of the virtual keyboard and virtual mouse interaction mode of the human-computer interaction method of the present invention until the setting is completed;

[0592] After the settings are completed, the operator can use the newly enabled virtual keyboard and virtual mouse interaction mode to input and operate:

[0593] On the virtual keyboard model, the operator performs a "keyboard tapping" action to input and operate on it;

[0594] Mouse operation functions are realized by gesture mouse recognition.

[0595] The system provides the following default settings for the virtual keyboard and virtual mouse (gestures) for wearable 3D devices:

[0596] The modeling operation area is located between the waist and the hips in front of you by default, and is in a single keyboard modeling operation area state;

[0597] The style and key response of the virtual keyboard model are organized, laid out, and configured by default according to the current standard keyboard style (102 / 104 or 78 / 87, etc.). The size (length and width) is slightly larger than the actual keyboard model. It can be ergonomically configured to facilitate operation. (The angle is generally perpendicular to the arm or parallel to the line of sight.)

[0598] The model's mapping display is consistent with the model itself, and the display color parameters are configured according to common practices;

[0599] The display of the operator is superimposed on the entity, and by default, the video image and outline of the operator's palm and wrist area are displayed, with generally appropriate color parameters, and the main points (and / or joint points, and / or skeletal model) marked;

[0600] The default action's ready action gesture and the release action gesture are:

[0601] Put your palms together, stretch them flat and raise them up at a certain angle (such as Left and right), definition represents: ready operation gesture,

[0602] Make a fist with both hands (or one hand), which means: disengage from the operation gesture;

[0603] The default settings for keyboard model adaptation parameters are:

[0604] Longitudinal docking distance range: ≤10cm, docking position: wrist of both hands (first operation base axis position), spacing: natural spacing (approx. ),

[0605] Gesture mouse, the default gesture settings given in this embodiment are:

[0606] The gesture of stretching out five fingers of one hand (or gathering five fingertips) is defined as: cursor, with the middle fingertip as the main point and the corresponding position of the key point.

[0607] The single index finger and thumb touch gesture is defined as: left click,

[0608] The gesture of touching the middle finger (or ring finger) with the thumb is defined as: right click,

[0609] Using cursor gestures, the up and down flipping actions are defined to represent: turning pages up and down;

[0610] For a gesture mouse, the system only needs to identify the display cursor, recognize and determine the key click gesture, and respond accordingly;

[0611] The default value of the response amplitude ratio of cursor operation is 1:1, (including the response amplitude ratio of up and down page actions); (Note: the 1:1 here refers to the ratio of the response amplitude of existing devices)

[0612] By default, the (cursor) anti-shake misoperation control is enabled;

[0613] In fine-tuning situations, you can also enable incremental overlay control of (cursor) movement;

[0614] FIG5 shows a three-dimensional panoramic immersive display wearable situation in which an interactive mode of integrating display space operation is selected ((1) integrating direct operation a direct application); if no parameter setting needs to be changed, the operator can use it to perform interactive operation;

[0615] If the settings need to be changed, the operator enters the S3 setting process step described in the above system operation process and makes settings on the setting interface;

[0616] The operator adjusts the system's last set parameters or default settings as needed according to the relevant steps in the setting content of the display space operation interaction mode integrated into the human-computer interaction method of the present invention, until the settings are completed;

[0617] After the settings are completed, the operator can perform interactive operations using the newly enabled interactive mode integrated into the display space.

[0618] The default settings provided by the system for integrating display space operations have the following characteristics for 3D wearable situations:

[0619] By default, the entire operation area is a single integrated operation area;

[0620] The default mapping relationship between the merged volume and the entity is:

[0621] For (2) the character integration operation, the integrated body is displayed with the design parameters of the character model including color;

[0622] For (1) incorporating direct manipulation, the body is integrated with the video image and outline of the subject's palm and wrist area, usually with appropriate color parameters, and with the main points, joint points, and / or skeletal model displayed;

[0623] The initial state of the fusion body is displayed as overlapping with the entity;

[0624] The activation confirmation and exit confirmation of the object components and objects are as follows: the default in this embodiment is to select the gesture of the gesture mouse to operate;

[0625] Confirm the scope of the activated area. By default, it will be displayed on the original construction main page of the component or object. By default, it will occupy the main display area. The color of the activated display will be configured according to the usual appropriate habits. By default, the main display surface will be in an appropriate position facing the operator and the field of view will be of an appropriate size.

[0626] The default action's ready action gesture and the release action gesture are:

[0627] The five fingers of one hand are stretched together to form a gesture, which means: ready operation.

[0628] The single-handed fist gesture is defined as: disengagement from the operation;

[0629] The default response ratio of the subject action is 1:1, including: movement distance, rotation degree, and angular change parameters;

[0630] Anti-shake and anti-misoperation control are enabled by default;

[0631] Incremental overlay control can also be enabled for precision, fine-tuning, etc.

[0632] Other interaction modes can be easily implemented by following the steps described in the relevant content of the human-computer interaction method of the present invention and referring to the above two main interaction mode embodiments.

[0633] The various interaction modes are used alone or in combination to complete the work;

[0634] The operator monitors the input operation through the interactive scene displayed in three-dimensional space and the response;

[0635] The computing and processing devices and systems process, judge and identify, make corresponding responses and operate accordingly until the work and tasks are completed.

[0636] The three-dimensional space computing wearable scenario includes virtual reality, mixed reality, and augmented reality display technology forms;

[0637] This embodiment is also applicable to applications including: metaverse, virtual simulation, software games, three-dimensional graphics, three-dimensional animation, and other three-dimensional scenes.

[0638] Example 3: An example of a non-wearable scenario of a three-dimensional space calculation type of a specific embodiment of the human-computer interaction method and system of the present invention, preferably a binocular stereo vision method,

[0639] Characteristics of non-wearable 3D spatial computing displays: The 3D coordinate sensors are arranged and oriented in the same direction as the display, facing inward toward the user or operator. The displayed image is three-dimensional and may or may not be panoramic. Depending on the limitations of display technology, it may or may not incorporate a fusion of full depth of field and virtual reality.

[0640] The corresponding virtual models (including models of other interaction modes) and fusion bodies (and regional display requirements) are respectively attributed to the two types of mapping display relationship displays described in the human-computer interaction method of the present invention described in Summary 1 of the Invention, that is, the panoramic depth of field virtual-real fusion is attributed to the mapping display relationship corresponding to the three-dimensional space computing type wearable type, or the non-panoramic depth of field virtual-real fusion is attributed to the mapping display relationship corresponding to the two-dimensional traditional computing type;

[0641] On the basis of the system solution and specific implementation methods, the corresponding methods and steps of the two types are respectively implemented, that is, the specific implementation steps of the non-wearable application are respectively adopted in accordance with the specific implementation steps of the above-mentioned embodiment 2 and the above-mentioned embodiment 1.

[0642] Non-wearable 3D coordinate measurement sensors are typically deployed in multiple locations around the perimeter for inward distributed control. The overlapping areas of the 3D coordinate measurement sensor detection areas require fusion processing to create a continuous, integrated, and panoramic scene effect, achieving coordinated distributed detection and effective system control.

[0643] Non-wearable computing devices are usually powerful configurations, which can be networked, distributed, or even parallel processing, quantum computing devices, and support integrated AI applications;

[0644] An integrated system that supports systematization, networking, and the mixing and joint use of a large number of execution devices and machines, as well as generalized computing devices and machines (smart devices);

[0645] The three-dimensional space computing type display non-wearable includes holographic stereoscopic display technology, other naked-eye three-D display technologies, and a simple method of wearing polarized glasses (not a screen), as well as other display technologies that will appear in the future.

[0646] For example, a specific embodiment of the present invention using a new generation of display technology and style types, structural methods, and new functional uses to solve technical problems is human-computer interaction in a flexible screen display situation;

[0647] Example 4: Scroll flexible screen display (mobile) terminal,

[0648] The development direction of the new generation of display technology is self-luminous, independently driven, and micro-chips, such as OLED, MicroLED, quantum dots, and other emerging technologies. Flexible screens are both a product that embodies their characteristics and a way to meet the needs of portability, thus hopefully realizing the emergence of rollable mobile terminals.

[0649] By using a self-luminous, flexible screen and using similar steps and settings as in Example 1, a rollable mobile computer solution can be formed, which can ultimately lead to the realization of a "pocket-sized" portable computer.

[0650] Furthermore, the integrated communication function will eventually make computers and mobile phones become integrated devices.

[0651] The technical solution of the human-computer interaction method and system of the present invention solves the most important structural technical obstacle currently hindering the widespread commercialization of scroll-type flexible screen devices: How to implement input and operation in a consistent and convenient manner in the case of a flexible screen?

[0652] The interactive method provided by the present invention does not require a physical keyboard or mouse, and does not require touching the flexible screen; thus, it is particularly suitable for the soft characteristics of flexible screen devices and matches the convenience requirements.

[0653] The screen area of ​​the shell of the scroll flexible screen device (at least the front) is made of transparent material, and the binocular stereoscopic vision device is arranged at the end of the scroll, so that the screen and content inside the shell can be clearly seen when it is not unfolded; when the screen is not unfolded (curled up), the current functions of smart phones can be fully realized, and basic content such as phone number, name, address book, message, time, and relatively small-scale narrow pictures and image content can be displayed, and video calls can be realized; small expansion and half expansion and even full expansion screen display are conducive to the high pixel and large screen size requirements of pictures, images and other content, especially the fully expanded and or half expanded screen is wide and the field of view is large, which is especially suitable for the top working state of the host computer, usually working in wired mode, suitable for realizing high-pixel and large-screen clear appreciation of videos, etc. high-end display needs; the support arm plays the role of support, wiring, and or expansion of storage, and or expansion of battery. The host computer provides more powerful computing power and performance, greater memory and storage capacity, a richer software suite, even AI integration, and a wider range of external connectivity ports. When connected to the host computer (wired or wireless), the scroll housing components function collaboratively, with the main functions of control interaction, wireless communication, wireless networking, and / or display driver functionality. When separated from the host computer and connected via a network, the scroll housing supports remote control of the host computer (based on a client-server model, where users connect to the computer remotely over the network, and operational results are reflected via screen sharing).

[0654] Based on the aforementioned three preferred embodiments of the human-computer interaction method and system of the present invention, namely, the application implementation device solution of the scroll flexible screen (mobile) terminal, this embodiment further provides a specific structural example of the specific implementation:

[0655] Figures 6, 7, and 8 are schematic diagrams of the structures and effects of the embodiments, as follows:

[0656] Figure 6 series: reel flexible screen and shell, card sleeve,

[0657] Schematic diagram a: The scroll-shaped flexible screen is in the unfolded state, with binocular stereo vision device cameras horizontally distributed at the upper end, symmetrically on the left and right; a sensor is symmetrically added under the left (or right) shell to support vertical screen;

[0658] Schematic diagram b shows: the scroll-shaped flexible screen is in a fully retracted state, the shell is merged and locked, and the transparent scroll-shaped screen is visible in the shell area (at least the front area);

[0659] As shown in the schematic diagram c: on the back of the shell in the folded state, there is a pair of lock holes for the card sleeve on the left and right, and the data and power cord sockets are shown on the right. The left side shows the appearance of the card sleeve after being locked, and the right side is the front style of the card sleeve; the card sleeve is used to lock the reel shell to the front end of the support arm, and has the function of line switching. The card sleeve shell has a locking tongue mechanism, and the card sleeve is larger in front and smaller in the back horizontally. The larger one is used for the situation where two arms are side by side, and the smaller one is used for the situation where a single arm is unfolded.

[0660] Figure 7 series: support arm and cross bracket,

[0661] As shown in the schematic diagram a: the supporting arms are folded side by side and can be clamped against the front of the ferrule housing;

[0662] As shown in the schematic diagram b: the support arm is in the lowered and extended state, and can be locked with the small rear part of the ferrule housing; the support arm can adopt a multi-section telescopic structure (such as 2 sections, etc.), and can be locked in the extended state;

[0663] Figure 8 series: main unit, stand and bracket,

[0664] As shown in the diagram a: the main unit and stand are integrated, or the stand is separated. The main unit has a hot key (with direction); the stand and the bracket work together to support the arm part and adjust the pitch angle and height or support the horizontal shell (when the screen is folded or extended);

[0665] Schematic diagram b: An enlarged side view of the support structure. The support is inserted and connected to the cross bracket of the support arm at the upper position. The back has a structure and a wiring connection socket for connecting with the cross bracket of the support arm, or the support hook at the bottom of the support is used to fix the side-by-side folded support arms in a horizontal position (usually for the semi-expanded or partially expanded (multi-section) state of the flexible screen);

[0666] The middle part of both sides of the vertical support is a long knuckle support structure that can be flipped 90 degrees. It is usually folded flat and can be lowered to support the horizontal shell when in use, with the screen folded or unfolded.

[0667] Schematic diagram c is shown as an enlarged diagram of the side structure of the support arm cross bracket. The cross bracket fixes one end of the support arm and can rotate. The bottom support hook supports and fixes the support arm that is horizontally extended to both sides. There is a corresponding structure and line connection socket on the back for connecting with the vertical bracket of the vertical frame.

[0668] Steps:

[0669] During the above system operation In the step, in the step S3, after selecting and setting each specific virtual mode interaction method, the operator can select and enable each interaction method to perform input and operation.

[0670] (1) Unexpanded state:

[0671] The left and right shells are combined with a lock (Figure 6.b), which is convenient for movement and can fully provide all the functions of a smartphone. The screen content inside the shell can be clearly seen when the shell is not unfolded, and it can be used completely like a mobile phone (making and answering calls, inputting and displaying text, symbols and other information, making video calls, and other basic content functions of current mobile phones, supporting wireless Bluetooth and wired headsets); it can be wirelessly connected to the host for operation (wired connection is also possible); it can be held in hand, or placed on a stand to support the shell: horizontal screen or vertical screen.

[0672] (2) Expanded state:

[0673] a. When holding the screen without the arm and card holder, you can unfold the screen to any size with both hands to view the image content. However, it is inconvenient to operate manually. You can use lip recognition + eye movement interaction and voice recognition interaction.

[0674] b. When using the support and clamp lock, with the screen half-extended or partially extended (arms side by side), or fully extended, the housing is fixed to both ends of the arms with the clamps and can be held (one-handed) or placed on other flat objects, or the hooks at the bottom of the stand bracket are used to clamp the arms placed horizontally side by side, or the upper part of the stand bracket is inserted and clamped to the horizontal bracket in the fully extended single-arm horizontal state, or the long knuckle supports on both sides of the stand bracket are lowered to support the housing in a horizontal position. At this time, the screen is either folded or supported by the arms;

[0675] Both can be easily input and operated using virtual interaction, and can be input and operated with one hand when held (one-handed);

[0676] All other situations (including stand support) other than mobile support support the working mode of wired connection with the host.

[0677] Obviously, this technical solution can also be implemented with equivalent substitutions, variations, or changes in specific elements, as described in the relevant aspects of the technical solution described in Section 3 above. In addition, this embodiment can also be implemented without certain specific details or with different details. For example, the structural design of the card connection and line switching socket on the back in the enlarged side structure diagrams of Figures 8 series b and c is only an example. Obviously, in actual application, the specific details and types of the details can be implemented in a specific manner and structure according to the actual needs of various product and equipment models, applications, and specific implementations.

[0678] By applying the virtual mode interaction method of the present invention, coupled with its technical solution and the specific examples of the structure and connection method of the above-mentioned embodiment, the problems and obstacles that hinder the appropriate and convenient input and operation of flexible screen devices are completely and thoroughly solved, thereby allowing (scroll) flexible screen devices to be widely used for commercial purposes.

[0679] Description of various scenarios for solving technical problems using the present invention:

[0680] The above describes in detail the preferred specific implementation methods for implementing the two-dimensional and three-dimensional calculation types of the human-computer interaction method and system of the present invention, and provides examples 1, 2, 3, and 4 to provide typical examples of the preferred specific implementation methods, supporting the technical solutions outlined in the claims; combined with the clear and complete description of the multiple technical solutions for solving the technical problems of the present invention listed in the technical solution section of the aforementioned invention content (Invention Contents 3, 4, 5, 6, 7, 8, and 9), the scope and specific applications of the present invention are fully demonstrated. The wide range, versatility, and multi-form situation;

[0681] Nevertheless, the specific application of the present invention is not limited to the scope of the above application solutions, nor is it limited to the specific details of these application solutions and examples. Following and using the method, principle, purpose and spirit of the human-computer interaction method and system technical solution of the present invention, the specific application of the technical solution of the method and system of the present invention and its specific implementation methods can also have a wider range of specific implementations, various element changes, equivalent substitutions, and variants.

[0682] The two-dimensional traditional computing type and three-dimensional spatial computing type described in this invention are explained and understood based on the most typical and common representative computer configurations: traditional computers and emerging three-dimensional computing types VR / MR / AR. They are configuration types of equipment that enhance professional computing capabilities, and their performance and application characteristics are specialized in computing.

[0683] However, the present invention is not limited to machines with high computing power. It is also fully applicable to smart devices with other specialized uses besides professional computing power: devices with computing processing capabilities; they can be regarded as generalized computers and can be used universally.

[0684] The application of the present invention can be the device configuration of various nodes and the implementation of specific technical means (which can be generalized computer intelligent devices), and can be various system network topologies and scales;

[0685] At the same time, the device configuration of the present invention is not limited to a single machine (integrated) or a set (split) device, but is also applicable to a systematic distributed functional layout system: interactive control, computing processing, display, execution, and other functions are distributed on different devices and displayed in a systematic manner;

[0686] Therefore, the present invention may be applied to computer networks, systems, intelligent networks / systems, hybrid networks, systems, various node configurations, and technical means of various topological structures. (The virtual model style (keyboard style) may be any desired panel style, and the control (including mouse) may be a hand lever or other method.)

[0687] The images displayed can be two-dimensional or three-dimensional;

[0688] It is also possible to select or combine various solutions from the above aspects, and use various virtual mode interaction methods alone or in combination;

[0689] In fact, the technical solutions and specific implementations of the method and system of the present invention are applicable to virtualization applications that require interaction between devices and systems with computing processors and display devices, including smart devices.

[0690] The application and implementation of the above-mentioned devices and systems still follow and use the methods, principles, purposes and spirit of the human-computer interaction method and system of the present invention, and are still within the scope of the present invention. They can adapt to different conditions and requirements and be applicable to different functions and uses.

[0691] The human-computer interaction method and system of the present invention are applicable to various fields and industries, including scientific research, production, life, and various social applications. These include intelligent manufacturing, intelligent office, intelligent service, smart home, intelligent entertainment, etc. Human-computer interaction virtualization applications in different occasions and use cases can improve the level of technology application and enhance intelligence (not referring to AI).

[0692] With the development of technology and the updating of devices, new forms of the constituent elements of the present invention may continue to emerge. As long as the spirit and scope of the methods, principles, purposes and solutions of the methods and systems of the present invention are still used, they will still be within the scope of protection of the present invention.

Claims

1. A human-computer interaction method based on three-dimensional coordinate measurement, characterized in that: The operating subject inputs and operates through the virtual keyboard and virtual mouse; it is integrated into the display space and directly operated; and other methods; Various virtual mode interaction methods realize input and operation, do not require contact, and are movable; including: (1) Virtual keyboard, virtual mouse, input, operation, The virtual keyboard and virtual mouse are modeled by the computing device and set by the operator according to the needs; they are operated by the operating subject to realize and provide the input and operation of the keyboard including any combination of letters, numbers, and symbol commands, as well as the mouse operation function; (2) Integrate into display space operation, The operation of integrating into the display space refers to the operation subject and / or the operator corresponding to the space integrated into the display screen of the computing and processing device, entering the virtual and real scene, operating as a part of it, and performing various direct operations and experiences on the object components and objects of the display space scene in a participatory and interactive manner; (3) Other methods, The other interaction methods refer to that the same system can also provide: virtual handwriting input, body movement posture recognition, lip recognition interaction, support for virtual touch screen, gesture recognition interaction, etc., and interaction methods related to the position coordinates, trajectory, posture, etc. of the action for input and operation; The display space scene includes three-dimensional space display occasions generated in various ways, and also includes two-dimensional flat and curved space display occasions; The correlation with the position coordinates, trajectory, posture, etc. of the action refers to the correlation with the position coordinates of the action and / or the elements or trajectory, or posture, or degree of action, etc. generated by the coordinates; The various interaction modes are implemented in a virtual mode to realize input and operation, which does not require contact or touching the screen and is mobile; The above (1) and (2) are the main interaction modes of the present invention, which can be used alone or in combination or simultaneously. The (3) other methods can be used as a supplement to the main interaction method, or can be used alone or in combination; the human-computer interaction method is based on the three-dimensional coordinate measurement technology, device and the hardware, software and display device (including two-dimensional traditional calculation type and three-dimensional space calculation type) of the computing processing device that provides support for it; The virtual keyboard, virtual mouse, input, and operation described in 1.(1) above include the following steps: The virtual keyboard (model) and virtual mouse (gesture, model) are modeled by a computing device, set by an operator, set as needed, and operated by an operating subject, providing and implementing keyboard input and operation in any combination of letters, numbers, and symbol commands, as well as mouse operation functions; Build: S1. The area within the measurement range of the three-dimensional coordinate measuring device, in front of the operator and the surrounding space area is the operation area; In it, usually in front of the operator, a certain area is appropriately designated as the modeling operation area; Generally, the computing device at the bottom is used to model, that is, to virtually construct a model to provide operation input; Displayed with appropriate adjustable display parameters; S2. For two-dimensional traditional computing types, the display is two-dimensional (usually a plane, including a curved surface), a virtual keyboard model is constructed in the modeling operation area, and the style is displayed in the foreground of the scene displayed by the computing processing device, and can be displayed in the lower part, that is, the lower part of the display screen, with appropriate and adjustable conversion parameters, and the corresponding displayed keyboard and each key of the model are mapped to each other; S3. For three-dimensional space computing type, the display is a three-dimensional space stereoscopic situation. For wearable type --- panoramic depth virtual reality fusion panoramic display, a virtual keyboard model is constructed in the modeling operation area, and the model is built in an appropriate position in front of the operator and displayed as the main body; S4. For three-dimensional space computing type, the display is three-dimensional space stereoscopic. For non-wearable, according to the limitation requirements of display technology, if the full depth of field is displayed in a fusion of virtual and real, a virtual keyboard model is constructed in the modeling operation area as described in S3, and the model is built in an appropriate position in front of the operator and displayed as the main body; Alternatively, if the display is not a full-view depth-of-space virtual-real fusion display, a virtual keyboard model style may be constructed as described in S2 and displayed at the same time in the foreground of the scene displayed in the three-dimensional space, in the lower part or other required and appropriate positions with appropriate and adjustable conversion parameters, and the corresponding displayed keyboard and each key of the model are mapped to each other; S5. In the modeling operation area described in S2, S3, and S4, the side outside the virtual keyboard model, such as the right or left, or the top or bottom, is intended to be set as a virtual mouse operation area. It can be set that the operating subject automatically enters the mouse operation area when it leaves the virtual keyboard model within a certain distance range, or the entire modeling operation area is selected to be set to a single virtual mouse function interactive operation state; Select Settings to implement mouse operation functions using gesture mouse recognition; The second option (depending on the need, not recommended) is to build a virtual mouse model to provide operations. For the S2 display situation (applicable to the S2 mode of S4), the displayed mouse and the model are also mapped and converted to each other. Hereinafter, in the description of the display position and mapping conversion relationship of the virtual model in the entire invention, for S4 three-dimensional space computing type non-wearable, according to the limitation requirements of the display technology, since they correspond to the same methods respectively, when it belongs to the previous panoramic depth virtual-real fusion situation, it is the same as the description about S3, i.e. three-dimensional space computing type wearable, and when it belongs to the latter non-panoramic depth virtual-real fusion situation, it is the same as the description about S2, i.e. two-dimensional traditional computing type, and it is applicable; S6. Regarding the operating subject, for the model mode, within the three-dimensional coordinate measurement range and the modeling operation area in the operation area, the operating subject performs corresponding mapping through three-dimensional coordinate measurement and displays in real time and marks the main point (keystroke point, which can be multiple points) and or the bone model at the corresponding part; According to the situation and needs: the operating subject display screen can appear in the form of video images, virtual animation or picture-fitting, or 3D reconstruction display; The simplest way is to display the outline of the main body or the relevant branches of the main point, mark the main point and / or display the skeleton model; For gesture mouse, only the display cursor needs to be recognized; S7. For the model mode, the operator enters the modeling operation area, stands at a suitable distance near the model and then makes a ready gesture, and the keyboard model automatically connects with the operator longitudinally at the first operation base axis position (wrists of both hands) and is in an initial posture ready for operation; The virtual mouse model, after the operator enters the virtual mouse area and takes position, makes a ready gesture, the system automatically locks and tracks, docks (combines and matches) the model and follows, and the pasting distance range is the entire mouse operation area, and the entire mouse operation area is the pad area; set up: The setting generally refers to adjusting the default or previously set parameter values; S8. The size, position, angle, viewing angle, display color and other parameters of the proposed modeling operation area in S1 can be adjusted by the operator according to needs and convenience; Generally speaking, for humans, it is usually placed in front of the operator so that the model and operation do not block the operator's main line of sight to view the display screen scene, within the three-dimensional coordinate measurement detection range, for convenience and need; For the S2 two-dimensional traditional computing display (applicable to S4 and S2 mode), it is generally located below the main line of sight of the screen in front of you. The usual desktop display is on the table top at the height of the elbow joint. For S3 three-dimensional space computing stereoscopic display (applicable to S4 and classified as S3 mode), it can be in any position within the reach of both hands in front of the operator, usually between the shoulder and chest to the hips within the reach of both hands, according to convenience and need; Set area status and range: select the modeling operation area as a single model operation area state, or select the setting of the interval range within which the operating subject leaves the model area and enters other adjacent areas; The organization content, layout style, configuration type, key size ratio, angle, and position of the virtual keyboard and virtual mouse (if selected) models described in S1, S2, S3, S4, and S5 are adjusted and set by the operator according to needs and convenience, adapted to ergonomic settings, and can be regionalized, three-dimensional, non-planar, etc., and suitable for the operator's operating subject characteristics; Includes the option to set the specific response of each key; the usual angle is perpendicular to the line of sight (default); The display color parameters include: edge brightness, letter symbol brightness, color, perspective (keys and disks), etc. The operator can select and set them according to the needs; The mapping display of the model, for the S2 two-dimensional traditional computing display situation (applicable to S4 and attributed to S2 mode) (hereinafter omitted), includes appropriate and adjustable conversion parameters: conversion ratio, conversion angle, and viewing angle, which are provided by the conditional technical parameters and display requirements of the system, the display device, and the three-dimensional coordinate measuring device, and the operator can adjust and select the settings as needed within the range provided by the system; generally, the conversion angle is the conversion of the model surface parallel to the screen surface, and the viewing angle is the rear viewing direction (default); For the S3 three-dimensional space calculation stereoscopic display full depth of field virtual and real fusion situation (applicable to S4 and belong to S3 mode) (omitted below), the display is the model body, the mapping display is consistent with the model body, the ratio is 1:1 and cannot be adjusted, and the angle, viewing angle and display color are the same as the model; Choose to set two specific gestures, defining their meanings: ready operation and disengagement operation; Set the keyboard model adaptive parameters, docking distance: the distance range, docking position and spacing between the model and the operating body in the modeling operation area; For gesture mouse, choose to set three specific gestures and one action gesture, and define them to represent: cursor, left click, right click. You can choose cursor gesture or other gestures to define up and down flip actions to represent: up and down page flip; For the virtual model mouse, you can choose to define the docked single gesture up and down flipping action definition to represent: up and down page turning; Set the cursor response ratio to the action amplitude of the operating subject (the default is 1:1), including the response ratio of the up and down page turning action amplitude; Anti-shake misoperation control can be enabled; Set the display area of ​​the operation subject mapping display; The above parameters are valid for the entire modeling operation area; The operating subject and the marked main point and / or skeleton model are mapped and displayed at the corresponding position according to the display setting requirements of the area, including: scale, angle, viewing angle and brightness, color, perspective, and other parameters; The conversion parameters in the modeling operation area, including: scale, angle, and viewing angle, are consistent with the model conversion parameters described above, and the display color parameters, including: brightness, color, perspective, edge brightness, and other parameter values, can be adjusted by the operator as needed; operate: S9. In the modeling operation area in the operation area, the operator performs input and operation actions on the virtual keyboard model and / or the virtual mouse model; For gesture mouse, three defined gestures and one action gesture are used for operation: cursor, left click, right click, and page turning gestures; response: S10. The computing and processing device measures and processes the spatial position coordinates of the operating subject and the operating action in the modeling operation area based on the three-dimensional coordinate measurement technology and device, mainly the three-dimensional coordinates of the principal point (keystroke point, which can be multiple points), extracts the coordinates, and makes a judgment: compares with the key area of ​​each key of the virtual keyboard model constructed by the computing and processing device in the modeling operation area, that is, the range of its coordinates, and makes a judgment on whether the action "keystroke" is successful (including the keystroke action and the specific key) according to the coincidence between the three-dimensional coordinates measured by the principal point and the pre-built (pre-set library) parameter values ​​of the key area coordinates of the model; executes (hardware, software) corresponding response and operation; For gesture mouse, extract the coordinates of the main point and joint point, compare the joint point and / or skeleton model structure pattern with the pre-set library, and judge according to the fit: identify the cursor, then use the position and movement of the main point of the cursor gesture to map the position and movement of the cursor in the screen, identify the left-click, right-click and page-turning gesture actions, and execute the corresponding response of keystroke and page-turning; For the virtual model mouse, the position and movement of the virtual mouse model and the operating subject in the mouse operation area correspond to the position and movement of the cursor in the mapping screen, and the left-click, right-click and page-turning actions of the mouse model are judged and recognized, and the corresponding responses of keystrokes and page-turning are executed; S11. The virtual model, gestures, operating subjects and real scenes of operating actions, together with responses, are displayed in real time at corresponding positions in the scene on the screen. The operator monitors the keyboard and mouse input operations by watching the screen, completing human-computer interaction; The operator evaluates the operation situation, operation response, and operation results to further complete the operation task; The main point generally refers to the keystroke point, which can be multiple points, including fingertips, contact points, etc. The operator refers to the part and the whole that issues instructions and monitors, and the operating subject refers to the part that completes the operation, including both hands (including arms), robotic arms, etc.; The above-mentioned operation of integrating into the display space as described in 1.(2) includes the following steps: Integration: Integration refers to the operation subject and / or the operator correspondingly integrating into the space of the display screen of the computing and processing device, entering the virtual and real scene, operating as a part of it, and performing various direct operations and experiences on the object components and objects of the display space scene in a participatory and interactive manner; S1. The area within the measurement range of the three-dimensional coordinate measuring device, in front of the operator and the surrounding space area is the operation area; generally, the entire operation area is selected as a single integrated operation area state; Or it can be set outside the aforementioned modeling operation area to integrate into the operation area. It can be set that when the operating subject leaves the modeling operation area, or leaves the virtual model within a certain distance range (including any virtual mouse, which does not conflict with it), it will automatically enter the integrated operation area and function state; S2. In the three-dimensional coordinate measurement range and operation area, the operating subject and / or the operator are mapped and integrated into the display screen space scene through three-dimensional coordinate measurement and displayed in real time, and the main point (contact point, which can be multiple points) and / or the bone model are marked at the corresponding part; The fusion body and the entity (operation subject and / or operator) are in a mapping correspondence relationship. According to the application and use of the fusion operation, it is classified as S4(1) fusion direct operation, and the fusion body is displayed with appropriate and / or adjustable conversion parameters; S4(2) is integrated into the character operation, and the operation actions and postures of the entity are mapped to the corresponding parts of the character; the integrated body is the character body, which is displayed by the design parameters of the character model; For S4(1) fusion direct operation, the choice is made according to the situation and needs: the fusion volume display screen can appear in the form of video images, virtual animation or picture-fitting, or three-dimensional reconstruction display; The simplest way is to integrate body contours, mark principal points and / or display the skeleton model; set up: The setting generally refers to adjusting the default or previously set parameter values; S3. Set the area status and range: select the entire proposed operation area as a single integrated operation area state, or select the modeling operation area within it, or outside the model, as an integrated operation area; For S4(2) character integration operation, the integration mapping is operated based on the ontology display and structure of the character model. The design parameters except color parameters are not adjustable in the integration mapping. For S4(1) fusion direct operation, the appropriate and / or adjustable conversion parameters of the fusion volume mapping display, including the mapping conversion ratio, conversion angle, and mapping position, are provided by the conditional technical parameters and display requirements of the system, the display device, and the three-dimensional coordinate measurement device, and the operator can adjust and select the settings according to the needs and convenience within the range provided by the system; The mapping position refers to the initial and starting position of the entity and the merged body when the operation starts. Generally speaking, it is at the bottom and both sides of the main sight line that does not block the viewing of the display screen scene in front of you. You can choose it according to your convenience and needs: The initial position of the entity is generally the appropriate and natural position within the detection range of the three-dimensional coordinate determination in the above-mentioned integrated operation area: the entire operation area or the modeling operation area within it, or the part outside the model; The entire operating area is usually selected as a single integrated operating area. Generally speaking, for a two-dimensional traditional computing display, it is usually located below the main line of sight of the screen in front of you. The usual desktop display is on the table top at the height of the elbow joint; For three-dimensional computing and stereoscopic display, it can be anywhere in front of the operator that both hands can reach, usually between the shoulder and chest to the hips, which can be reached by both hands, according to convenience and need. The starting position of the fusion body, for the two-dimensional display, is generally at an appropriate position near both ends of the lower edge of the front field of the display screen scene, with a conversion ratio and conversion angle to display the starting state fusion body; For the three-dimensional display, according to the requirements of display technology limitations: full depth virtual and real fusion display, generally the fusion body and the entity are overlapped, and the fusion body is superimposed on the entity body for display (non-adjustable conversion); Non-full depth virtual-real fusion display is to display the initial state fusion body at the appropriate position of the lower edge of the front field with conversion ratio and conversion angle (similar to the case of two-dimensional display); When the operation interface pops up or the activation area appears, the interface or area usually occupies the main display surface. Usually, the starting position of the fusion body is at an appropriate position at the lower edge of the front field of the main display, and is displayed by entity projection conversion (similar to the situation of non-full depth of field virtual-real fusion display); if it does not occupy the main display surface, the fusion body can follow the mapping corresponding display at the corresponding position at the lower edge of the interface or area---projection conversion of the main display surface situation; Set the area where the operating subject is integrated into the display; The color parameters of the fusion body display include: brightness, color, perspective, edge brightness, and other parameter values, which can be adjusted by the operator according to the needs; Set the main display area and position; Set the definition operations of object components, object activation confirmation, and exit confirmation of exit activation. The three defined gestures of gesture mouse can be used to implement the operation (applicable when the gesture mouse is fully integrated into the operation area); Set adjustment to confirm the activated area range, activate the color effect parameters displayed; for minor operations, it can be combined with the focus area range selection method to enlarge the display for easy use; The activated display color effect parameters of the components, objects and regions include: brightness, color, perspective, edge brightness, and other variable parameters, which can be adjusted by the operator as needed; Choose to set two specific gestures, defining their meanings: ready operation and disengagement operation; Set the response ratio of the main action parameters of the operation (the default is 1:1), including: moving distance, rotation degree, angular change, etc.; can perform incremental superposition control, and or anti-shake misoperation control; for precise situations, you can select settings to facilitate fine-tuning; The set parameters are valid for the entire integration operation area; operate: S4. The hardware and software of the computing and processing device need to be designed to support the scene-based operation mode. According to the construction of the software, the operating subject directly operates the operable object components and objects, and performs various participatory and interactive operations and experiences: (1) Integrate direct operation, as a part of the display screen space, directly operate and interact with the operable objects and components in the virtual and real scenes; have remote control, that is, remote: direct operation at far and near; the characteristics are: activate first, then operate; that is, first activate and confirm the target object to be controlled, and then map the operation action to it; a Direct application: directly applying the corresponding mapped operation action to the object components and objects within the display screen area to perform operation and interaction; b. Mapping response application --- through network communication, the data information of the control action is transmitted to the corresponding receiving port through data instructions, mapped to the response action of the corresponding execution device or machine, and replaced and extended by it to complete the action work --- through mapping to the execution device or machine; The operation starts from the initial position of the entity (operating subject and / or operator) and the starting position of the integrated body, and the corresponding mapping is performed; when the operation is completed, the activation is exited, and the entity and the integrated body return to the original initial and starting position states and corresponding displays; the system is ready to carry out a new round of activation and operation; The activation confirmation and exit confirmation operations can be performed dynamically in real time and have the highest priority; (2) Integrate the role operation, map it to the role, and as a member of the virtual and real space scene world, that is, the role player, perform participatory operation and interactive operation; the actions and postures of the operating subject and the operator are mapped to the corresponding parts of the role; the characteristics are: the corresponding mapped operation action is issued from the perspective and position of the role, that is, the execution action; The character directly operates and interacts with the physical components and objects in the virtual and real world scenes around him, completing the corresponding operation actions and control instructions of the operator; the interaction follows the same general rules and experience as the real world; Usually the main display presents the visual field of the character's perspective, and can also display the operator's perspective (global) picture; it can be switched in real time; response: S5. The computing and processing device measures and processes the position coordinates, actions, postures, etc., mainly the three-dimensional coordinates of the main points (contact points, which may be multiple points) and joint points of the operating subject (which may be multiple parts) and / or the operator in the operating area based on the three-dimensional coordinate measurement technology and device, and the results are mapped to the corresponding parts of the fusion body, and corresponding judgments are made respectively: compared with the position coordinates of the corresponding object components and objects, and the pre-built storage (pre-set library) of the operation mode, and compared with the pre-built storage (pre-set library) of the action degree (speed, acceleration) and trajectory, the mode of posture, and the paradigm parameter, and make corresponding judgments and identifications according to the degree of coincidence or fit; execute (hardware, software) corresponding responses and operations; S6. The real scene of the operating subject and / or the operator and the operating action, together with the response, is displayed accordingly in the screen scene, and the operator monitors the integration of the operation by watching the screen; completing the human-computer interaction operation function; The main point generally refers to the contact point, which can be multiple points, including fingertips, joints, etc.; The other methods mentioned in 1.(3) above refer to the fact that the same system can also provide: virtual handwriting input, body movement posture recognition, lip recognition interaction methods; support gesture recognition, virtual touch screen, eye movement interaction; etc., and use interactive methods related to the position coordinates, trajectory, posture, etc. of the action for input and operation; It can be set in the virtual mouse operation area (replacing the mouse) described in S5 of claim 1.(1) above, or in an appropriate area outside the virtual keyboard model and / or virtual mouse model in the modeling operation area, or the entire modeling operation area can be selected to be set as a single operation mode function area; or in an appropriate area outside the modeling operation area described in S1 of claim 1.(2) above, or the entire operation area can be selected to be a single operation area state; the range of settings provided is configured accordingly according to the corresponding requirements and needs of each interaction mode, and the operator can select according to the needs; (1) Virtual handwriting input interaction, S1. The computing and processing device measures and processes the three-dimensional coordinates of the main point of the operation action of the operating subject based on the three-dimensional coordinate measurement technology and device, extracts the coordinates, tracks and obtains the trajectory, and makes a judgment: the structural pattern of the action trajectory of the main point (feature point) is compared with the pre-built (pre-set library), and the recognition judgment is made according to the fit; the hardware and software respond and run accordingly to realize the input operation; S2. The real scene of the operating subject and the operating action, together with the response, is displayed accordingly in the screen scene, and the operator monitors the handwriting input operation by watching the screen; completing human-computer interaction; (2) Body gesture recognition and lip shape recognition interaction, S1. The computing and processing device measures and processes the corresponding three-dimensional coordinates or posture and posture parameters of the operating subject's body movements or mouth shape based on three-dimensional coordinate measurement technology and devices, extracts the coordinates of the principal points and joint points, and makes judgments and identifications: compares the body posture joint points and / or the structure pattern of the skeleton model or the structure pattern of the mouth shape with the pre-built storage (pre-set library), and makes identification judgments based on the degree of fit; executes the corresponding response and operation of the hardware and software to realize the interactive operation; S2. The real scene of the operating subject and the operating action, together with the response, is displayed accordingly in the screen scene, and the operator monitors the body movement posture or lip movement by watching the screen; completing human-computer interaction; (3) Support virtual touch screen, gesture recognition, and eye movement interaction. The support means that the same system can also provide support including: virtual touch screen, gesture recognition, eye movement interaction and other methods for interactive use; corresponding software design needs support and is easy to be integrated and used in the system of the present invention; aVirtual touch screen interaction, The S2, S3, and S4 models in 1.(1) above can be modified as follows to achieve a virtual touch screen interaction mode: For the S2 two-dimensional traditional computing display situation, a plane model of a flat-panel interface type virtual touch screen is constructed in the modeling operation area, and the style is simultaneously displayed on the display screen with appropriate and adjustable conversion parameters; For the wearable S3 three-dimensional display situation, a flat panel interface virtual touch screen plane model is constructed in the modeling operation area, and the plane model is displayed as the main body; For the S4 three-dimensional spatial display situation which is not wearable, the situations are the same as those of S2 and S3 above; The operating subject operates the virtual flat touch screen model, and the computing and processing device processes and extracts the three-dimensional coordinates of the main point (keystroke point, which can be multiple points), compares it with the coordinate area range of each icon and menu item in the library pre-set by the virtual touch screen model, and judges whether the "click" is successful based on the overlap, performs corresponding (hardware and software) response and operation, and marks the main point at the corresponding part, and displays it in real time at the corresponding position in the screen scene together with the response; completing human-computer interaction; b. Gesture recognition, eye movement interaction (existing methods), The gesture recognition computing and processing device extracts the three-dimensional coordinates of the gesture body joint points, performs gesture processing, compares them with the structural patterns in the pre-set library, makes recognition judgments based on the degree of fit, and performs corresponding responses and displays to realize interactive operations; In the eye movement interaction, the computing and processing device extracts the posture parameters of the principal point pupil, performs projection processing, obtains the corresponding aiming position (eye pupil focus point) parameters, compares them with the coordinates of objects in the picture space and virtual and real scenes, makes judgment and extraction of the selected corresponding target body according to the overlap, and performs corresponding response and display; realizes interactive operation; The overlap or fit judgments described in 1.(1), 1.(2) and 1.(3) above, as well as the sensitivity adjustment and the action amplitude adjustment, are set by the system, and the operator can select and set the options provided by the system; The various virtual mode interaction methods provided by the present invention can realize input and operation without contact or touching the screen, and are mobile.

2. A system for implementing the human-computer interaction method based on three-dimensional coordinate measurement as described in claim 1 of the present invention, characterized in that: Includes the following modules: (1) Three-dimensional coordinate measurement module, The three-dimensional coordinate measurement module is used to measure the spatial position coordinate parameters of the operating subject of human-computer interaction and / or the operating action of the operator, and is connected to the calculation processing module, and its functions are realized by its computing power and software support; The operation area and the virtual model and operation actions of the operation subject and / or the operator in the modeling operation area and the operation actions in the area outside the measurement range of the three-dimensional coordinate measurement device are measured: the measured data are processed and calculated accordingly to obtain the spatial position coordinate parameters of the operation actions of the operation subject and / or the operator, mainly the three-dimensional coordinates and posture parameters of the main points (keystroke points, contact points, focus points, etc.) and / or joint points, and / or further obtain the trajectory, posture, action degree and other related parameter values; (2) Computation and processing module, The computing processing module is mainly constructed by at least one processor (including CPU, GPU, NPU, TPU, etc.) and the software running on it, which is used to process, calculate, judge, identify, execute response, run, etc. of parameter data information of other modules; It includes: extracting the measurement data of the three-dimensional coordinate measurement module, making judgments: comparing with the regional coordinates, or patterns, or paradigm parameters in the pre-built (pre-set library), making corresponding judgments and identifications according to the degree of coincidence or fit, and executing (hardware, software) corresponding responses and operations; The system is responsible for performing the human-computer interaction method based on three-dimensional coordinate measurement according to claim 1 above, setting up the proposed modeling operation area and the integrated operation area in the operation area, and constructing a virtual interaction mode (as a system default setting), including: a virtual keyboard, a virtual mouse interaction mode, or an integrated display space operation interaction mode, or other interaction modes; Provide adjustment and selection of relevant parameters for each mode, including: operation area status and range, organization layout style and display characteristic parameters of the constructed virtual keyboard, mouse (if selected) and other models, model display style conversion parameters, specific responses of each key that can be set, model docking distance range, gesture mouse, mapping response ratio and area parameters of the integrated operation action, incremental overlay, anti-shake misoperation control, starting position, ratio and angle settings of the integrated body mapping display, activation related settings, other corresponding settings, or adjustable parameters, etc., which can be adapted to ergonomics and needs, and convenient settings; including the selection and adjustment settings of overlap or fit judgment sensitivity and action amplitude; (3) Display module, Connected with the calculation and processing module, it is used to display the information that needs to be displayed on the measurement, processing, calculation, judgment, recognition and response, operation and effect of the three-dimensional coordinate measurement module and the calculation and processing module for the operator to monitor and watch; Including: display content and display requirements of the human-computer interaction method based on three-dimensional coordinate measurement according to claim 1, including the establishment area of ​​the interactive mode, various virtual models constructed, gesture mouse, integrated direct operation, role operation, or other methods, mapping and response of the operation subject and / or operator and operation action, main point, joint point, and other corresponding information to be displayed; Including: 2D traditional computing type requirement display, 3D space computing type requirement display, full depth virtual-real fusion display, non-full depth virtual-real fusion display; The above modules jointly realize various virtual mode interaction modes and input operation functions of the human-computer interaction method of the present invention, thereby further completing the work tasks of the system; The various virtual mode interaction methods provided by the system of the present invention realize input and operation without contact or touching the screen and are mobile; The realization of the function of the three-dimensional coordinate measurement module is based on three-dimensional coordinate measurement, including: binocular stereo vision method, structured light method, TOF laser time of flight method or method based on AI three-dimensional reconstruction method, etc. The selection of three-dimensional coordinate measurement technology and device is not limited in the present invention. The present invention preferably uses binocular stereo vision technology device method, using two identical camera sensors with different viewpoints, and the overlapping area of ​​the field of view is the measurement range and the appropriate operation area; The three-dimensional coordinate measurement module is preferably a binocular stereo vision technology device, and the specific steps of the system are as follows: S1. A three-dimensional coordinate measurement module, wherein the three-dimensional coordinate measurement preferably uses two identical camera sensors at different viewpoints of a binocular stereoscopic vision device, whose parameters and angles enable the overlapping area of ​​the field of view to appropriately cover the operating area; if necessary, calibration is performed to obtain correction parameters of internal and external parameters; S2. A computing and processing module, for constructing various virtual interaction modes according to the human-computer interaction method based on three-dimensional coordinate measurement according to claim 1; S3. According to the steps, the operator sets the parameters of each virtual interaction mode to suit ergonomics, needs and convenience; S4. The operating subject performs input and operation actions on the virtual keyboard model and / or virtual mouse model; For gesture mouse, three defined gestures and one action gesture are used for operation: cursor, left click, right click, and page turning gestures; Or it is integrated into the display space operation mode, the operation subject and / or the operator is mapped and integrated into the display screen space scene to perform various direct operations and experiences of the operable object components and objects in a participatory and interactive manner; the hardware and software of the system need to be designed to support the integrated operation mode, including the activation operation of the control target: activation first, then operation; Or according to the needs of the occasion, you can choose settings and enter other ways of interaction, including: virtual handwriting input, body movement posture recognition, lip recognition interaction, support gesture recognition, virtual touch screen, eye movement interaction, etc., and use interactive methods related to action position coordinates, trajectory, posture, etc. to input and operate; the corresponding software needs to be designed to support, among which gesture + eye movement interaction in wearable three-dimensional scenes requires at least two binocular vision sensors to be added; S5. A three-dimensional coordinate measurement module processes the images of the sensors of the cameras with different viewpoints at the same time accordingly, matches the characteristic points such as the principal point, and can calculate (measure) the spatial position coordinate parameters of the operating subject and / or the operator and its actions based on the triangulation method and the polar line constraint relationship rules, mainly the three-dimensional coordinates and posture parameters of the principal points (keystroke points, contact points, focus points, etc.) and / or the joint points of the operating subject's skeleton model, and / or further obtains the values ​​of related parameters such as trajectory, posture, and action degree; S6. The calculation processing module extracts the measurement data of the three-dimensional coordinate measurement module, compares it with the regional coordinates, or patterns, or paradigm parameters in the pre-built (pre-set library, table), makes corresponding judgments and identifications according to the degree of coincidence or fit, and executes (hardware, software) corresponding responses and corresponding operations; S7. The display module displays the information required to be displayed on the measurement, processing, calculation, identification, response, operation, etc. performed by the three-dimensional coordinate measurement module and the calculation processing module in accordance with the display content and display requirements of the human-computer interaction method based on three-dimensional coordinate measurement as described in claim 1 above; The picture is provided by two identical camera sensors at different viewpoints of a binocular stereoscopic vision device, and one image is selected for correction and then displayed for a two-dimensional traditional computing display; The operator can monitor the input operation status by watching the operation scene and the corresponding display screen, and realize the human-computer interactive operation function; The system follows the operator's operations and instructions to complete the operations, work, and other tasks required by the operator; The two identical cameras and sensors with different viewpoints can be horizontally distributed, symmetrical on the left and right, which is a preferred solution for easy calculation; However, it can also be distributed up and down or in other distributions, or arranged asymmetrically, and can be more than two sensors, or even multi-point, multi-part (pair), distributed control arrangement, etc., and other distribution, arrangement and quantity selection can be made according to specific needs, and the calculation amount and setting will change accordingly; Similarly, according to the needs of the use environment, select the appropriate camera sensor applicable frequency band and illumination and other technical parameters of the binocular stereo vision device; Alternatively, various frequency band light spots and cursors may be used to mark and replace the main points and key points of the operating subject; The preferred binocular stereo vision method for three-dimensional space calculation type, based on the above, has the following specific features: a wearable solution, (3) Display module, The two-way images of binocular stereo vision realize the full-view, deep, virtual and real fusion three-dimensional stereoscopic panoramic (immersive) display; The three-dimensional space computing type wearable function of the display module of the human-computer interaction system of the present invention is realized; Including: the real scene and response of the operating subject and / or the operator and the operating action is equal to the full-view depth virtual-real fusion three-dimensional space stereoscopic panoramic (immersive) picture display scene real-time display; bNon-wearable solution, (1) Three-dimensional coordinate measurement module, The arrangement and orientation of the non-wearable three-dimensional coordinate determination sensor are the same as the display orientation, facing inwards towards the user or operator; usually multiple (pairs) and distributed arrangements are used, and the positions and quantities are arranged according to the three-dimensional space detection range of the space domain; The three-dimensional coordinate measurement module function of the human-computer interaction system of the present invention is realized; (2) Computation and processing module, The processor and software support the implementation of the three-dimensional space calculation type configuration of the present invention and the preferred binocular stereoscopic vision method, and support better AI integration application; The three-dimensional space calculation type non-wearable function of the calculation processing module of the human-computer interaction system of the present invention is realized; Including: supporting distributed control of multiple points and multiple units (pairs) around, fusion processing of overlapping edges of detection areas of three-dimensional coordinate measurement sensors, forming a continuous, integrated, panoramic scene effect, and realizing coordinated and effective system control of distributed detection; (3) Display module, The two-way images of binocular stereo vision realize three-dimensional space stereo display; The non-wearable function of the three-dimensional space calculation type of the display module of the human-computer interaction system of the present invention is realized; Including: non-wearable three-dimensional space stereoscopic picture display, which may be panoramic or not (need not) panoramic; limited by display technology, it may be panoramic depth virtual and real fusion or not panoramic depth virtual and real fusion, the virtual model and fusion body correspond to the corresponding method and steps of the human-computer interaction method of the present invention as described in claim 1, respectively using the two types of mapping display relationships for corresponding display, as well as operation actions and responses, etc., for real-time display; The non-wearable computing processing module is usually a configuration with strong computing power, which can be a networked, distributed or even parallel processing, quantum computing and other devices, supporting integrated AI combined application; An integrated system that supports systematization, networking, and the mixing and joint use of a large number of execution devices and machines, and generalized computing devices and machines (intelligent devices); The non-wearable three-dimensional display may include: holographic stereoscopic display technology, other naked-eye three-D display technologies, and a simple method of wearing polarized glasses (not a screen), as well as other display technologies that will emerge in the future.

3. An application device scheme for applying the human-computer interaction method and system optimization mode of the present invention: a scroll flexible screen (mobile) terminal, characterized in that: The specific configuration and steps of the equipment are as follows: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Roll-up flexible screen and housing, The housing contains a scroll (with retractable elasticity), two or two parts on the left and right, providing a retractable and unfoldable flexible screen, and also contains hardware and software devices supporting screen display and wireless communication including smart phone (including processor) functions; it can independently complete the mobile phone functions and screen display work, and the housing screen area (at least the front part) is transparent; (2) Three-dimensional coordinate measuring device, Preferably, two identical camera sensors of the binocular stereo vision device are horizontally distributed at the upper front ends of the left and right shells, symmetrically on the left and right; or a sensor is added to the left or right shell symmetrically up and down to support a vertical screen; (3) Support arm, cross bracket and clamping sleeve, The support arm and the horizontal bracket are used to support and hold up the flexible screen when it is unfolded; the card sleeve is used to lock the shell on the back and fix it to the front end of the support arm, and also has the function of data and power cable adapter; The arm has expansion storage, expansion battery and data, power cable and terminal sockets for expansion performance and wired mode connection between the shell and the host firmware; a multi-section structure can be used; (4) Main equipment, stand and bracket, The host includes a computer processor, software, storage, various ports, etc., providing more powerful computing performance and functions, and can be connected to the shell components by wire or wirelessly to work together to support the integrated application of AI; The stand and the bracket work together to support the arm and adjust the pitch angle, height or support the horizontal shell (screen folded or unfolded state); the stand contains data, power cables and terminal connectors, which serve as a wired connection between the host and the arm; Steps: S1. Enable the interactive mode of the human-computer interactive method of the present invention, and set the operating area state and range, model, gesture mouse, conversion parameters, response ratio, area, docking, and other related settings, adjustable parameters, etc. according to the human-computer interactive method of the present invention to adapt to ergonomics, needs, and convenience. The operator operates and works in the selected interactive mode; or directly uses the last selected setting or the system default setting; Input and operation can be realized through various virtual mode interaction methods provided by the present invention, without contact or touching the screen, and with mobility; S2. When not unfolded, it can fully provide all the functions of a smartphone, and can be connected to the host wirelessly or by wire. When held in hand, input and operation can be performed with one hand. S3. In the unfolded state, it is supported horizontally or vertically, with a large screen, and usually works in wired mode, which is particularly suitable for exerting the high-strength performance of the host PC and for high-end demanding scenarios with high pixels, high definition, and large screens at the top of the computer; S4. When the screen and the housing are separated from the host, the host can be remotely controlled through a network connection; All of the above can be conveniently input and operated using various virtual mode interactive methods of the present invention, thereby completing various tasks and tasks of the operator; The flexible screen can be a two-dimensional display or a three-dimensional display; This technical solution can also be applied with specific element changes, including: The housing is fixed to the front end of the support arm by a card set, or the opposite side of the housing can be fixed to the front end of the support arm by a groove to perform line switching, or other equivalent functional forms; The housing contains two reels arranged on the left and right, which can be symmetrical in shape, or the main and secondary parts can be single-axis rolled, and the end edges can be single-stretched with a support rod structure; The reeling mechanism can be reeled in a reeling manner or in a folding manner; The device system configuration may also have other configurations and functional uses, including not requiring independent host configuration, or focusing on the image screen (smart TV), or not requiring mobile phone communication function, etc., and the configuration can be selected according to specific needs; Other three-dimensional coordinate measurement techniques and devices or equivalent methods may be used.

4. An application system solution using the human-machine interaction method and system optimization mode of the present invention: a micro-surgical robot system, characterized in that: The specific configuration and steps of the system are as follows: The basic components of the micro-surgical robot system include: computing and processing unit, interactive control unit, display unit, robot subsystem unit, and accessory units such as robotic surgical instruments and operating tables; Based on this, the technical features of this system solution are: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Interactive control unit, The three-dimensional coordinate measurement preferably uses binocular stereo vision devices to be arranged in multiple points, multiple parts (pairs), and distributed placement in the appropriate range and position of specific areas and designated workstations in the surgical environment space for distributed control; including: Mobile workstation: VR / MR / AR head-mounted movable headset, 3D coordinate measurement device configuration: horizontally distributed on both sides of the upper frame of the front of the headset, symmetrical on the left and right, facing outward; Fixed workstations: including control desks, operator workstations in operating rooms, surgeon workstations in the operating area, assistant or nurse workstations, etc. The arrangement of the three-dimensional coordinate measuring device is selected in a cheap way based on the best and effective range of landing points, areas, fixing and installation conditions required by the surgery and convenient for three-dimensional measurement; the operator station of the main console can be horizontally distributed on both sides of the upper frame of the 3D / 2D display screen (not a head display), symmetrically installed on the left and right, facing the operator, or other split, external, independent seat-type fixing or fixed by various fastening devices; the surgeon station in the operating area can be fixed on the operating table, ceiling or other methods mentioned above; the selection of other points is similar; The 3D / 2D display screen should be configured accordingly to the needs of the designated workstation and should be configured in accordance with the layout of the binocular stereoscopic vision device; (2) Computational processing unit, Computer processors and software support the application of the present invention and the implementation of the system solution, including three-dimensional coordinate measurement, processing judgment recognition and response, 2D / 3D display (VR / MR / AR three-dimensional space calculation), robot subsystem and binocular stereo vision including surgical positioning, registration, tracking and other support, support for incremental superposition control and anti-shake misoperation control, AI medical integration implementation support, etc.; It includes directly calling the action space position coordinates and posture orientation parameters of the three-dimensional coordinate measurement unit to map to the execution action position and orientation of the character; (3) Display unit, Display the information that needs to be displayed by the three-dimensional coordinate measuring unit and the calculation processing unit accordingly; Robot subsystem binocular stereo vision 2D / 3D display (optionally one channel of corrected 2D display or two channels of 3D display), or switch to other positions for binocular stereo vision display; Including: display of surgical positioning, registration, tracking, etc. of the robot subsystem, display of surgical operation movement trajectory, position, orientation and response, and effect (movement and amount, direction angle value, etc.); The configuration and functions of other units adopt existing technology or similar situations; Steps: S1. Enable the interactive mode of the human-computer interactive method of the present invention, and set the operating area state and range, model, gesture mouse, conversion parameters, response ratio, area, docking, and other related settings, adjustable parameters, etc. according to the human-computer interactive method of the present invention to adapt to ergonomics, needs, and convenience. The operator operates and works in the selected interactive mode; or directly uses the last parameter value or default setting; The virtual mode interaction method provided by the present invention realizes input and operation without contact and is movable; the movement position and orientation of the corresponding operating subject arm and / or operator can be completely mapped; S2. During the preoperative preparation planning stage, any virtual interaction method provided by the present invention may be used alone or in combination; The system supports the necessary organization and identification of the route design arrangement, description and other information of the plan, and arranges and sets the display characteristics of the physiological and pathological tissues and structures of the surgical area and parameters such as the viewing angle and orientation; Setting response ratio parameters: The surgeon selects and sets the mapping response ratio (usually reduction) of the robot arm action according to the precision of the operation, including: moving distance, rotation degree, angular change, etc.; Enable incremental overlay movement control and anti-shake misoperation control; Set the area range and magnification ratio parameters: according to the precision of the surgery and the size of the area range, choose to set the focus area and display the magnification ratio; S3. During the operation execution phase, the operation of integrating the role using the operation interaction method of the present invention is selected, that is, the operation S4 step (2) of claim 1 is selected to integrate the role operation, and the mapping is applied to the role: the robot subsystem is executed; S4. The operator operates through the binocular stereoscopic three-dimensional coordinate measuring device at his own workstation; the surgeon operates by wearing a VR / MR / AR headset, or sits in front of the 3D display screen at the main console without a headset; The surgeon can hold the scalpel mold in his hand to operate the corresponding action, or specify the main point at the end of the gesture as the corresponding point of the operation; S5. The operator can monitor the progress of the operation by watching the 3D / 2D images displayed by the binocular stereo vision of the robot subsystem; monitor the consistency of the movement trajectory of the master and slave arms; The robot subsystem arm completely maps the total motion position and orientation of the operator's arm movements to perform the surgical operation; Until the work and tasks of the operation are completed; The arms of the robotic subsystem are usually multiple and multifunctional (including binocular stereo vision), and the selection is dynamic and can be selected and used at any time during the operation; During the operation, if the hands are inconvenient, the selection control can be carried out through eye tracking, lip recognition or voice recognition; The three-dimensional coordinate measuring device is configured with multiple points, multiple parts (pairs), and distributed control, and supports simultaneous detection of multiple workstations and simultaneous operation by multiple people: doctors, assistants, and nurses each choose to operate different robotic arms to participate in, coordinate surgery, and cooperate in operations (the selection and control rights of each position have priority). The system of the present invention supports that the robot subsystem can have more arms to perform operations at the same time, and multiple robot subsystems can participate in the operation together; the doctor does not even need to stay beside the patient, and supports remote surgery through Internet connection; The system configuration and structure described above are based on the current typical system application scheme, but the elements can also be changed: there can be other equipment node configurations and scales, different system network topology structures; the robot subsystem can also be an intelligent machine, equipment, and can be used in situations where it can move and execute autonomously; other three-dimensional coordinate measurement technologies and devices or equivalent methods can be used, and it can be applied to other industry occasions and situations, and can be applied to different characteristics and expertise requirements, and other functional uses.

5. An application device solution for applying the human-computer interaction method and system optimization mode of the present invention: a smart phone, a tablet device terminal, characterized in that: The specific configuration and steps of the equipment are as follows: The basic components of existing smart phones and tablet devices include: computing and processing unit, interactive control unit, display unit, communication unit, and interface accessory unit; The technical features of this solution are: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Interactive control unit, The three-dimensional coordinate measurement preferably uses a binocular stereo vision device with at least one (pair) of sensors arranged horizontally on both ends of the upper frame in front of the screen, symmetrically distributed on the left and right, or symmetrically distributed up and down to support a horizontal screen, or three sensors on the left and right, up and down; (2) Computational processing unit, The processor and software support the implementation of this technical solution, including the measurement, processing, judgment, recognition, response, and display of three-dimensional coordinates, and support for better AI integration and application, etc. The configuration and functions of other units adopt existing technology or similar situations; Steps: S1. Enable the interactive mode of the human-computer interaction method of the present invention, enter the human-computer interaction interface state of the present invention, and present the last selected setting or the system default setting; If the operator does not need to change, just use it directly; S2. Enter the setting interface, and set the operation area status and range, model, gesture mouse, conversion parameters, response ratio, area, docking, and other related settings, adjustable parameters, etc. according to the human-computer interaction method of the present invention to adapt to ergonomics and needs and convenience. The operator operates and works in the selected interactive mode; S3. For devices with relatively small screens such as smart phones, in order to reduce the screen space occupied by the display of model-type interaction methods (keyboard or touch screen, etc.), in the above-mentioned claim 1 of the human-computer interaction method of the present invention, 1. (1) virtual keyboard, virtual mouse, input, operation setting step S8, the following specific operations are set: The appropriate and adjustable conversion parameters of the display style of the model in S8 include: conversion ratio, conversion angle, and viewing angle. The operator can adjust the conversion ratio as much as possible to reduce the display size, so that the space occupied by the display is as narrow as possible and can be recognized; and the construction size of the virtual model of the operation space can be relatively large, which is easy to operate and input without causing misoperation; At the same time, a magnifying glass mechanism is adopted, that is, the corresponding model parts in the small area around the main point are magnified and displayed (the magnification is adjustable) to facilitate identification and observation; The combination of the two reduces the screen space occupied by the model display; The specific response of each key symbol can be jumped, amplified and popped up, and can be accompanied by different frequency sounds, which is convenient for the operator to monitor; Thus, the interactive display occupies little space and the interaction is normal and convenient; The human-computer interaction method of the present invention provides various virtual mode interaction modes to realize input and operation without contact or touching the screen; The scheme described in step S3 of display conversion of the virtual model to reduce the occupied screen space can obviously be applied to any similar human-computer interaction input and operation occasions to save screen space; and can also be applied to specific element changes; other three-dimensional coordinate measurement technologies and devices or equivalent methods can be used; The solution is suitable for any flat, foldable or flat-screen device, including human-computer interaction input and operation virtualization applications of various intelligent machines and devices.

6. An application device scheme using the human-computer interaction method and system optimization mode of the present invention: a portable device for daily social interaction for deaf-mute people, characterized in that: The specific configuration and steps of the equipment are as follows: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Hardware, The portable device using the human-computer interaction method and system of the present invention is provided with at least one (pair) of binocular stereo vision devices for three-dimensional coordinate measurement, a processor, a display, a sound pickup, a speaker, etc., supports wireless communication, networking and Bluetooth, and is equipped with a Bluetooth headset; For two-way transmission and presentation, a device with two screens on both sides and a pair of three-dimensional coordinate measuring devices is selected; (2) Software, Support the implementation of this technical solution, including supporting the formation of interactive content information between the two parties and realizing the organization and presentation or two-way transmission and presentation, and supporting the switching of the presentation content selection screen; Solution 1: preferably using the scroll flexible screen (mobile) terminal device solution of the present invention, The scroll flexible screen (mobile) terminal of the present invention has a double-sided screen display structure on both the front and back sides in the retracted state, and the screen area of ​​the outer shell material is transparent, which is easy to stand up, and at least one (pair) of three-dimensional coordinate measurement is preferably a binocular stereoscopic vision device is configured on the front side; When the two parties are in face-to-face mode, it is a two-way transmission and presentation, and both the front and back sides need to be equipped with a (pair of) three-dimensional coordinate measurement, preferably a binocular stereo vision device; Steps: S1. When this solution is enabled, the system automatically enters the interactive communication mode and interface. By default, deaf-mute gesture recognition and voice recognition are used as the main modes. Other interactive modes include body gestures, handwriting, lip shape or eye movement, which can be selected, set and used in real time. S2. Social side-by-side mode, both parties face the same side of the flexible screen terminal (expandable); the organization and size of the information displayed on the screen can be set, or the last setting parameters or default settings can be used directly; The same pair of three-dimensional coordinate measuring devices and sound pickup microphones respectively perform real-time recognition of the deaf-mute hand gestures, body postures, and / or handwriting, lip shape or eye movement interactions of the deaf-mute person and convert the result content into text and voice information for display and broadcast, and perform voice recognition (pickup, speaker location) of ordinary people's speech, and / or body postures, handwriting, lip shape or eye movement, etc., and convert them into text information for display. Both parties realize continuous interactive communication by watching the grouped text presented on the screen, (ordinary people) and voice broadcast information content; S3. Social two-party face-to-face mode, the flexible screen terminal is supported between the two parties; the two screens each display the information content of the other party transmitted in both directions, or the content of the other party is displayed together for monitoring, and the organization form and size of the information content displayed on the screen can be set, or the last parameter or default setting can be used directly; The three-dimensional coordinate measuring device (configured on both sides) respectively recognizes the hand gestures, body postures, handwriting, lip shape or eye movement of the deaf-mute person in real time and converts the result content into text and voice information and transmits it to the other party's screen for display and broadcast (the side where the microphone and speaker are located); recognizes the speech of ordinary people through voice, and or recognizes the body postures, handwriting, lip shape or eye movement and converts it into text information and transmits it to the screen facing the deaf-mute for display. It is better that the text information presents the group content of both parties; both parties realize the continuous interactive communication by watching the information content presented by the screen text, (ordinary people) and voice broadcast; It is better for ordinary people to pair S2 and S3 with wireless Bluetooth headsets to avoid using external speakers and microphones to pick up sound; The front and back sides are each provided with a pair of binocular stereo vision devices for three-dimensional coordinate measurement, which can be horizontally symmetrically distributed or vertically symmetrically supported horizontally; Solution 2: Preferably use the three-dimensional space computing type glasses device solution of the present invention, a The head-mounted AR / MR / VR of the three-dimensional space computing type of the present invention is a mobile head display similar to glasses, binocular stereo, and the three-dimensional coordinate measurement is preferably configured with a binocular stereo vision device. The two sides of the upper frame of the head display are horizontally distributed, symmetrical on the left and right, and facing outward; Steps: S1. When this solution is enabled, the system automatically enters the interactive communication mode and interface. By default, deaf-mute gesture recognition and voice recognition are used as the main modes. Other interactive modes include body gestures, handwriting, lip shape or eye movement, which can be selected, set and used in real time. S2. The three-dimensional coordinate measuring device facing outward of the head display is used to perform real-time recognition of the deaf-mute hand gestures, body gestures, and / or handwriting, lip shape or eye movement interaction (a pair of binocular stereo vision is required) of the deaf-mute person and convert the result content into voice and / or text information for broadcast and / or display; the speech of ordinary people is recognized by voice, and / or body gestures, handwriting, lip shape or eye movement recognition is converted into text information displayed on the mirror screen, and the deaf-mute person can realize continuous interactive communication by watching the screen text and ordinary people can realize continuous interactive communication by voice broadcast; b. On the basis of the above, a further configuration is added with an outward-facing mirror surface with a transparent display screen, thereby forming a pair of transparent screens and display functions on both sides (the inner side is also a transparent display screen), with an intermediate layer between the two sides that is transparent at ordinary times - the lens is transparent, and can be controlled to be opaque when interaction is required, and the two sides display different contents without crosstalk. The glasses technology solution also has a Bluetooth headset function (or is equipped with an outward-facing pickup microphone and speaker); Steps: S1. When this solution is enabled, the system automatically enters the interactive communication working mode and communication interface. By default, the deaf-mute gesture recognition and voice recognition are the main ones. Other interactive modes include body gestures, handwriting, lip shape or eye movement, which can be selected, set and used in real time. S2. The three-dimensional coordinate measuring device facing outward on the head display can recognize the hand gestures, body postures, handwriting, lip shape or eye movement interaction (need to add a pair of binocular stereo vision) of the deaf-mute person in real time and convert the result content into text and voice information for display on the external mirror screen and Bluetooth voice broadcast; it can also recognize the speech of ordinary people. Or the recognition of body gestures, handwriting, lip shape or eye movement is converted into text information and displayed on the inner mirror screen for the deaf-mute person. Both parties can realize the continuous interactive communication by watching the information presented by the inner and outer mirror screen texts, (ordinary people) and voice broadcasts respectively; It is preferred that the text information presents the group contents of both parties for monitoring; The portable device may be a split structure; The front and back sides are transparent screens and the middle layer is usually transparent and can be controlled to be opaque when in use, which is the preferred solution. However, it is also possible that the two screens do not need to be transparent and in this case or some middle layers do not require normal transparency, so that the wearer is worn when in use (interactive communication) and not worn when not in use; Solution 3: Using the smart phone or tablet device terminal solution of the present invention, The terminal application of the smart phone or tablet device of the present invention is arranged with at least one (pair) of three-dimensional coordinate measurement, preferably binocular stereo vision devices, horizontally distributed at both ends of the upper frame of the front of the screen, symmetrical on the left and right, or symmetrically distributed up and down to support horizontal screen; installed with relevant software App and deaf-mute gesture corpus support; paired with Bluetooth headset (recommended), or use the device pickup microphone and external speaker; Steps: S1. The smartphone or tablet device is placed in front of both parties (shoulder-to-shoulder mode); the application of this solution is activated, and the system automatically enters the interactive communication working state and communication interface. By default, the deaf-mute gesture recognition and voice recognition are mainly used, and other interactive methods include body gestures, handwriting, lip shape or eye movement as auxiliary, which can be selected, set and used in real time; S2. Set the organization form and size of the information content displayed on the screen, or directly use the last parameter or default setting; The three-dimensional coordinate measuring device and the sound pickup microphone respectively recognize the deaf-mute hand gestures, body gestures, and / or handwriting, lip shape or eye movement interactions of the deaf-mute person in real time and convert the result content into text and voice information for display and broadcast, and recognize the speech of ordinary people by voice, and / or body gestures, handwriting, lip shape or eye movement, etc., and convert them into text information for display. Both parties realize the continuous interactive communication by watching the grouped text, (ordinary people) and voice broadcast information content presented on the screen; It is better for ordinary people to wear wireless Bluetooth headsets to avoid mutual interference of sounds; The above three solutions can work offline or online (preferably with customized AI deaf-mute gesture recognition, multi-language translation, etc.); Among the above three solutions, it is recommended that ordinary people use wireless Bluetooth headsets to avoid using external speakers and microphones to pick up sound; and the use distance of the two can be extended to increase the long-distance interaction between the two parties, which can be in different rooms (ordinary people only use voice), etc. The headset in WiFi mode (customized version) uses the same principle, supporting both parties to use it at a greater distance; Support remote interaction through networking (wireless, wired) connection, which requires an additional device. For ordinary people, it can be an ordinary smart device (monocular); The device can be used to add, modify and maintain deaf-mute gesture language materials on a daily basis; The speech recognition and text display function can be used by deaf people with normal throats and voices for self-pronunciation correction training.

7. An application device scheme using the human-computer interaction method and system optimization mode of the present invention: a portable device for daily travel for blind people, characterized in that: The specific configuration and steps of the equipment are as follows: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Hardware, The portable device using the human-computer interaction method and system of the present invention is provided with at least one (pair) of binocular stereo vision devices for three-dimensional coordinate measurement, a processor, a sound pickup, a speaker, etc., supports wireless communication, networking and Bluetooth, and is equipped with stereo dual Bluetooth headsets and left and right vibration prompts; It is recommended to use glasses, with binocular stereo vision devices on the upper frame: the two sides are horizontally distributed, symmetrical on the left and right, facing outwards, and the left and right voice broadcast and vibration devices (which can be adjusted to the front and back position and flipped up and down) are placed on the ear of the glasses frame; (2) Software, Support the implementation of this technical solution, record the offline database package (which can be established with the help of AI) required for the corresponding recognition and judgment of the navigation function, support online support work, and preferably have customized AI recognition, guidance, multi-language translation, etc.; Operation steps: S1. The portable device of this solution is activated, and the system automatically enters the navigation working state, working in offline or online support mode (customized AI recognition, guidance, multi-language translation, etc.); S2. The device detects, identifies, processes and judges the information of people, objects, roads, signs, traffic signals and other things encountered by the wearer in real time through a binocular stereoscopic vision three-dimensional coordinate measuring device, broadcasts and prompts in voice or vibration mode, and issues real-time path guidance navigation, sign translation, obstacle reminders, danger warnings and other content information, so as to help blind people travel safely; The portable device may be a split structure; Obviously, with slight modifications, expansions and extensions to the implementation of the above technical solutions, the functions of identification and guidance can be extracted and strengthened. By strengthening the positioning, identification, target judgment, terrain, map matching, integrated customized AI recognition, judgment, and guidance, online support (recommended) or offline, it can be easily transformed into a device solution for automatic positioning, identification, and navigation of surrounding objects by mobile devices, and used for equipment service applications in situations such as autonomous navigation and automatic driving.

8. An application device and system solution using the human-computer interaction method and system optimization mode of the present invention: virtual interactive driving and control, characterized in that: The specific configuration, method and steps of the equipment and system are as follows: Solution 1: Virtualization of other control and interaction devices. The other control interactions refer to the control of devices other than driving. Currently, most other control interactions in driving use touch screens with processors and voice interaction methods. Based on these methods, this solution uses the virtual mode interaction method of the present invention instead, specifically: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Hardware, The three-dimensional coordinate measurement is preferably a binocular stereo vision device, and at least one (pair) of devices should be placed in front of the front driver or on both sides of the upper frame of the screen. The device system includes a processor, supports wireless communication, networking and Bluetooth, and is equipped with a Bluetooth headset; it is better to support the integrated application of AI; (2) Software, Support the implementation of this technical solution, and record the offline database package of recognition and judgment required by the navigation function, support online support work, and preferably have customized AI recognition, navigation, multi-language translation, etc. Steps: S1. Enable the application of this solution, and the system automatically enters the interactive mode control working state and menu interface of the human-computer interaction method of the present invention. By default, the virtual gesture mouse mode is the main interaction mode, and other interaction modes include body posture movements, lip shape, eye movement, handwriting or voice recognition as auxiliary modes, which can be selected, set and used in real time; S2. The system, in accordance with the human-computer interaction method of the present invention, uses a three-dimensional coordinate measuring device to determine and identify the position of the gesture mouse operation and the key action instruction, performs hardware and software corresponding responses and operations, and broadcasts the process content and results by voice (optional); S3. The operator monitors the operation actions and responses by watching the displayed screen or voice broadcast; Complete control work tasks; The present invention also supports a virtual touch screen interaction mode as an alternative; The configuration and arrangement are convenient implementations based on the combination with the currently more commonly used facilities, but other configuration and arrangement display forms may be used, and a transparent screen may be used to display in a suitable area on the front glass; or a head-mounted AR / MR or VR may be used for interaction and display, and the lenses of AR / MR or VR glasses are usually transparent (transparent screen) as preferred; the operation is realized by the virtual interaction method provided by the present invention, which does not require contact and has mobility; Solution 2: Virtual interactive driving and control system, The virtual interactive driving and control system refers to the systematized control of driving and control in the form of a virtual mode interactive system of the present invention, which fully controls the functions and tasks of the passenger vehicle including driving and other control; The basic components of the system include: computing and processing unit, interactive control unit, display unit, control unit, feedback unit, and electromechanical system unit; The technical features of this solution are: The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functional configurations described in the binocular stereoscopic vision mode, including: (1) Interactive control unit, The three-dimensional coordinate measurement preferably uses binocular stereo vision devices to be arranged in appropriate positions in the passenger vehicle for distribution control, including the front of the front driver and other required and appropriate positions for installation; the virtual mode interactive system of the present invention is used to replace the existing driving control facilities and equipment; The existing method may be simplified and designed to be equipped as a spare on the side, pulled out when in use and pushed aside when not in use, including a simplified steering wheel (or left and right semi-arc or large semi-circle type or other styles including handles or keys), speed lever or key: acceleration, deceleration (braking) or even gear shifting, eliminating the accelerator, brake, and clutch pedals; eliminating the fixed partition in the middle of the front row; (2) Computational processing unit, The processor and software support the application of the present invention and the implementation of the system solution, including: three-dimensional coordinate measurement, processing judgment and identification, tracking matching and response, operation, 2D / 3D display, intelligent judgment, support for incremental superposition control and anti-shake misoperation control, support for AI integrated application; support for wireless communication, networking and Bluetooth, equipped with Bluetooth headsets, speakers, etc. Including supporting direct calling of the motion space position coordinates, trajectory, and rotation parameter values ​​of the three-dimensional coordinate measurement unit to map to the corresponding execution equipment and machine response actions, thereby replacing and extending the action work; (3) Display unit, Display the information that needs to be displayed by the three-dimensional coordinate measuring unit, the calculation processing unit and other units accordingly; Forward vision display during driving, binocular stereoscopic vision 2D / 3D display at the operating station (optionally one-way corrected 2D display or two-way 3D display), etc. Including: display of steering response, speed response effect (direction angle, speed value), display of steering control model, speed control model operation status indication, etc., and information display in existing solutions; (4) a control unit, The present invention is integrated with the existing methods to complete the functions of the system solution; (5) Feedback unit, The present invention is integrated with the existing methods to complete the functions of the system solution; (6) Electromechanical system unit, The unit configuration and functions are existing methods or similar situations; Process steps: S1. Start the system, the three-dimensional space position coordinate measuring device is ready, the operating system displays a startup menu, and after the operator enters, the system automatically enables the interaction mode and interface of the human-computer interaction method of the present invention, and presents the last selected setting parameter or the system default setting; The system of this solution applies the mapping response application of the direct operation integrated into the operation interaction mode of the present invention, that is, the b mapping response application of the direct operation integrated into the operation S4 step (1) of claim 1. (2) described above is controlled and mapped to the execution device and machine response: to (drive) the electromechanical system unit through the control unit; S2. If the operator does not need to change the setting parameters, use it directly to enter the driving and control interface, and then proceed to step S4 in sequence; S3. Enter the setting interface, and adjust the adjustable parameters of virtual interactive driving according to the human-computer interaction method of the present invention, and select and set them to meet the driving and control needs, ergonomics and convenient settings for the user's own characteristics; The modeling operation area and the inner model (including position, size, angle) are selected between the upper part of the legs and the lower part of the chest in a sitting position so as not to block the sight in front and are convenient to choose, and are usually set to a position above the legs and above the handrails; The system supports standing driving control and mobile driving control (cancel the middle fixed partition) It can be placed anywhere within reach of both hands in front of the operator, usually between the shoulder and chest to the hip area, and any position can be selected as needed; Adjust the relevant settings of the steering wheel model and speed lever model proposed by the system (default), fine-tune the sensitivity settings, and adjust the left and right distribution settings; including the relative position relationship between the two, where the left and right distribution can be dynamically converted and adjusted in real time; Set the mapping response ratio parameters of the operation subject's actions, including: the ratio parameter of the rotation angle change or the moving distance. The operator can choose the specific settings according to the feature preferences and needs; enable incremental overlay control and anti-shake misoperation control, intelligent judgment, etc. Choose to set three specific operation gestures and one adjustment gesture, define the representatives: ready operation, proceed operation (left and right hands can be different), disengagement operation, adjust the position of the modeling operation area; the three specific operation gestures can be rotated around the wrist and elbow and can be effectively recognized, and the ready operation gesture can be combined with the proceed operation gesture; set the disengagement distance interval generated by the disengagement operation; For the operation area position determination mode, set the model adaptive parameters, docking and pasting elastic distance: within the modeling operation area, the distance range, docking position (left and right hands can be different) and spacing of the active docking between the model and the operating subject, and the elastic distance range of the model pasting is the separation distance; For the follow mode, the model automatically docks (combines and matches) with the arm and follows the arm, and the distance range for the model to follow is set, usually the modeling operation area is the follow range limit; For the automatic switching mode, set the switching conditions or parameters, usually: automatically switch to the follow-up mode when standing or moving driving, and switch to the position-determined mode (the area above the armrest) when sitting and holding the armrest; Set display parameters, including conversion ratio, conversion angle, area range, position, including brightness, perspective, edge brightness, color, and other parameter values. The operator can adjust the settings as needed; The above parameters are valid for the entire modeling operation area; The operation subject, steering wheel model, speed lever model, or other models, and the selection setting of whether to display or hide the operation subject, steering wheel model, speed lever model, or other models during driving without operation change status; Control and adjustable parameters of other vehicle equipment, sounds, etc. The present invention supports providing steering control without using a steering wheel model, and other models and operation modes can be selected for control. Speed ​​control can also have other models and operation modes, and the mode selection and corresponding settings of the relevant models are performed; After the setting is completed (stored for standby use), the operator can drive and control it; Operation: The S4 operator uses the binocular stereo vision device to measure the three-dimensional coordinates of his own workstation for interactive operation, or wears an MR / AR or VR headset (equipped with a binocular stereo vision device) for control; After the operator takes the driving position (sitting posture, position determination mode, effective range of three-dimensional coordinate measurement), the system automatically detects (identifies), locks, and tracks the operator (both hands), and accordingly determines the position in the operating area or prepares for following or automatic switching mode; The operator's arm enters the modeling operation area, is located near the model and within the docking distance range, and then makes a ready operation gesture. The model automatically docks its arm (combines and matches) in the docking area (left and right hands) and is in an initial posture ready for operation. The operator operates by performing operating gestures (left or right hand), standard operating movements, operating a steering wheel model, a speed lever model, or other models and methods; Use the ready-to-operate gesture and the disengage-from-operation gesture to distinguish and exclude non-operation periods; The system compares and identifies the operating gestures, standard operating actions, displacement positions, etc. with the set gestures, action modes, displacement position ranges, etc., distinguishes them from unconscious actions (daily habits, physiological actions, etc.), and processes abnormal non-standard actions and shaking actions. It has intelligent recognition and judgment to prevent misoperation and anti-shake control. If the position of the operator (hands) changes, the system will automatically dock the model intermittently; The operator can monitor the driving and control operation, position or instruction display mark and the execution, response, operation effect, progress, feedback and other conditions in real time by watching the 2D / 3D display screen; and can choose to broadcast the process content and results by voice; The display may be any suitable 2D / 3D display scene format, including a transparent screen displayed at an appropriate position on the front glass, or a head-mounted MR / AR or VR display; This solution of the present invention provides virtual control of seated driving; It also provides virtual control that supports standing driving; and or mobile driving virtual control; When equipped with forward-looking vision, you can even drive from the back seat, or virtually control driving in a reclining or lying position. response: S5. The computing unit processes, determines and identifies the operation action, displacement position, or operation instruction of the interactive control unit, forms control and instruction information and transmits it to the control unit, controls the execution device of the electromechanical system unit to complete the execution response and operation accordingly, and the feedback unit collects the operation data information and feeds it back to the computing unit and the control unit for loop adjustment and control. Various information is displayed on the display unit together, and the execution response and operation process of the corresponding mapping of the operation subject's arm action (rotation angle, control position, etc.) and or limb posture action instruction are completed, so as to realize driving and control interaction; For the steering operation of the steering wheel model, the spatial position coordinates, trajectory, rotation angle parameters and orientation parameters of the operating subject action are used; for the speed operation of the speed lever model, the spatial displacement position and trajectory parameters of the operating subject action are used; and for other operating action postures, corresponding processing judgments and validation processing are performed and then mapped to the response actions of the execution device: steering, gear position and speed, etc.; The system runs through this process in a continuous cycle, so that the electromechanical and mechanical equipment can continuously correspond to the total motion rotation angle and control position mapped by the operator's arm movements, and respond to the operation instructions until the driving journey task is completed; S6. It is recommended to use intelligent driving assistance and navigation functions, which can work online or offline (customized AI assistance, recognition, navigation, multi-language translation, etc. are preferred); the software supports route planning for travel tasks, necessary information signs along the way, work and rest arrangements, etc.; The system can use any interactive mode of the human-computer interaction method of the present invention to select, set, modify and maintain default items and parameters for driving and control on a daily basis; The direction model and speed rod model can also be replaced by the direction rod or speed rod or key of the real remote control mode; For the control of other equipment besides driving, the interactive scheme or spirit described in the above scheme 1 is adopted, which can be easily integrated into this scheme for combined control; The virtual interaction system controls driving and operation, and the system is still driven and operated manually, with human intervention; it supports the combination with intelligent automatic driving technology for assistance, and the combination with AI intelligent driving technology for better assistance; The autonomous navigation and automatic driving variants of the enhanced identification and navigation functions of the portable device for daily travel of blind persons as described in claim 7 above are combined with this solution to form vehicle automatic driving and AI autonomous intelligent driving (online support or offline) functions, and perform virtual driving, control assistance or automatic and autonomous intelligent driving (autonomous navigation, automatic avoidance, self-driving, berthing, etc.); In the driving and controlling virtual mode system described in this solution, the operator controls through the virtual mode interactive mode provided by the present invention, which does not require contact and has mobility, and is therefore particularly suitable for use in aerospace in a weightless state; The configuration and structure of the system are system application solutions based on the current typical and common land passenger vehicles, but it can have other equipment node configurations and system network topology forms, and be suitable for various vehicle system applications in space, air, land, sea, underwater, underground, etc.; it can carry out element change application, equivalent substitution and variant implementation, it can adopt other three-dimensional coordinate measurement technologies and devices or equivalent methods, it can be suitable for the application needs of other industries and occasions, and it can be suitable for different characteristics and expertise requirements, and other functional uses.

9. An application technology scheme of the human-computer interaction method and system optimization mode of the present invention: virtual mode remote control (wired, wireless) control and manipulation work, characterized in that: The specific configuration and steps of the solution are as follows: The remote control includes a typical panel (including a touch screen), a remote handle joystick, and other similar functions, and is characterized in that the operator can control and operate the work through the remote control device, and the feedback information can be sensed, the response and execution effect can be felt, and it is usually accompanied by a display; The basic components of the system include: computing and processing unit, interactive control unit, machine "vision" unit, display unit, control unit, feedback unit, and electromechanical system unit; The technical features of this solution are:

1. The system for realizing the human-computer interaction method of the present invention as claimed in claim 2 of the present invention preferably has the modules and corresponding functions configured in the binocular stereoscopic vision mode, and the following functions are realized in this application technical solution: (1) The virtual keyboard interaction method or virtual touch screen interaction method of the present invention is specifically applied to realize various required remote control panels, center console control panels, and other similar devices with various panel types (including touch screens) and functions; (2) Applying the interactive operation mode of the present invention to realize remote handle or other similar functions: i. Implementing the control function by applying the mapping operation of (1) integrated direct operation: mapping to the corresponding response of the control drive unit of the execution device or machine; ii. (2) Integrate the role operation to realize the manipulation function: the mapping is applied to the role --- robot, mechanical arm, processing table and other subsystems --- with machine vision unit for execution; (3) Using other interaction methods of the present invention, including gestures, body movements or lip movements, handwriting recognition, eye movement interaction, etc., to implement the operation instructions required for the above-mentioned control; Each of them can be used alone, in combination or in coordination, so as to completely realize and complete the purpose of replacing the interactive control and manipulation work of the real remote control device with the virtual mode interaction mode of the present invention; The unique technical features of the human-machine interaction method and system of the present invention support the easy implementation of simultaneous interactive control of multiple channels and multiple groups of object machines and equipment, and can realize remote control and manipulation work; 2. The system configuration of this solution with the above-mentioned modules and corresponding functions specifically includes: (1) Interactive control unit, The three-dimensional coordinate measurement preferably uses a binocular stereo vision device arranged in a suitable range and position of a specific area and a designated workstation in the working environment space, which can be a multi-point, distributed control placement; including: Mobile workstation: VR / MR / AR head-mounted movable headset, 3D coordinate measurement device configuration: horizontally distributed on both sides of the upper frame of the front of the headset, symmetrical on the left and right, facing outward; Fixed workstations: including control desks, rooms, sites, and other operator workstations and detection locations; (2) Machine "vision" unit, The working "vision" configuration of the remotely controlled object equipment and machines is also preferably a binocular stereoscopic vision device; including: subsystems such as robots, mechanical arms, processing tables, and driving forward vision, etc.; (3) Computational processing unit, The processor and software support the application of the present invention and the implementation of the system solution, including: three-dimensional coordinate measurement, processing, judgment and recognition, execution response, operation, machine "vision", interactive 2D / 3D display, distributed detection fusion processing, intelligent judgment, support for incremental superposition control and anti-shake misoperation control, support for AI integrated application; support for wireless communication, networking and Bluetooth, equipped with Bluetooth headsets, speakers, etc. Supports simultaneous interactive control and manipulation of multiple channels, multiple groups, and distributed devices; Including supporting direct calling of the motion space position coordinates, trajectory, rotation, orientation, etc. of the three-dimensional coordinate measuring unit and mapping the corresponding manipulation of the directly applied execution operation action; (4) a display unit, Display the information that needs to be displayed by the three-dimensional coordinate measuring unit, the calculation processing unit and other units accordingly; The remote control target equipment (robots, mechanical arms, processing tools and other subsystems, and driving forward vision, etc.) works with binocular stereoscopic vision 2D / 3D display (optionally one-way corrected 2D display or two-way image 3D display), and the display of various types of panels, joystick models, etc. of the remote control device of this solution, and or switch to other positions of binocular stereoscopic vision display; control, control response, execution instructions, information display, etc.; It can be a common display of the working "visual" of multiple channels and multiple groups (groups) of machines; Or a simplified version of the simple panel with display of information such as control, setting status of the operation, response operation (feedback) parameters, etc.; (5) Other units The configuration and function of the control unit, electromechanical system unit, and other units are to integrate and combine the present invention with the existing methods or similar situations; Steps: According to the content steps of the human-computer interaction method of the present invention, the functional contents described in the technical feature 1 of the above-mentioned scheme are respectively implemented and operated in a corresponding manner to complete various tasks of remote control and manipulation; (1) Use a virtual keyboard or a virtual touch screen to realize various panel functions and operations; (2) Implement the control, manipulation function and operation of the remote control object (joystick, etc.) by integrating the operation interaction mode; enable incremental superposition control and anti-shake misoperation control as needed, and make intelligent judgments; (3) realizing the operation command and operation by other interactive methods, including recognition of gestures, body movements or lip movements, etc.; Feedback and display content, including panels, joystick remote handles or components with similar functions, mapped corresponding operation actions, responses, executions, and other information screens are provided to operators for monitoring and interaction until the task is completed; Common system-integrated control panel types can be considered as wired or wireless remote control short-range (on-site) situations. The above-mentioned applications include: simplified version, simple feedback (near-field) type; The virtual mode interactive mode of the present invention realizes remote control, and can adjust and set corresponding styles and types according to different characteristics, expertise requirements, and different usage occasions and situations. Various configurations and different types can be switched and called in real time; Applicable: The virtualization mode of corresponding control and operation of remote control devices (including integrated control panels) in the fields of intelligent production, manufacturing, processing, intelligent office, intelligent home, intelligent service, entertainment, etc. The virtual mode remote control and interactive manipulation provided by the present invention does not require contact, has mobility, and can be distributed and multi-group controlled.

Images