Immersive multi-module feedback man-machine interaction method and system based on visual dominance

Abstract

The invention is suitable for the technical field of man-machine interaction, and provides an immersive multi-module feedback man-machine interaction method and system based on visual dominance. According to the invention, binocular monitoring is carried out on the area opposite to display; in a glasses wearing state, unlimited standard man-machine interaction with a user is carried out; in a glasses taking-off state, limiting and screening the plurality of standard interactive applications; matching an application interaction size, and planning an application display position; and carrying out partially limited amplification man-machine interaction with the user. When the glasses are worn, unlimited standard man-machine interaction with a user can be carried out; in the glasses taking-off state, multiple standard interaction applications are subjected to limited screening, application interaction size matching, application display position planning and partially limited amplification man-machine interaction with a user, so that the visual state difference of the user is automatically recognized, self-adaptive adjustment is carried out, it is ensured that the user carries out accurate and continuous interaction operation, and the user experience is improved. And the use experience of effective information browsing is improved.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, and in particular relates to a vision-driven immersive multi-module feedback human-computer interaction method and system. Background Technology

[0002] Human-computer interaction is the technical process of perceiving, modeling, parsing, and mapping user behavior, instructions, or state information, realizing the conversion of human intentions into machine-executable instructions, and feeding back the machine's operating results to humans in a perceptible form.

[0003] Computer devices are currently the most common human-computer interaction medium, and the human-computer interaction process mainly relies on monitors, mice, and keyboards.

[0004] In existing technologies, human-computer interaction in computer devices typically uses uniform and fixed display and interaction parameter settings, which cannot be adaptively adjusted according to the differences in the user's visual state. In particular, for nearsighted users, it is impossible to present and control different interactive feedback based on whether they are wearing glasses or not, which affects the user's experience of accurate and continuous interactive operations and effective information browsing. Summary of the Invention

[0005] The purpose of this invention is to provide a vision-driven immersive multi-module feedback human-computer interaction method and system, which aims to solve the technical problems existing in the prior art mentioned in the background.

[0006] The embodiments of the present invention are implemented as follows: A vision-driven, immersive, multi-module feedback human-computer interaction method, the method specifically includes the following steps: The system performs binocular monitoring on the opposite area of ​​the display, acquires binocular monitoring data, and identifies the binocular monitoring data to determine the user's current facial state and facial spatial position. When the current facial state is that of wearing glasses, multiple standard interactive applications are identified to conduct unrestricted standard human-computer interaction with the user. When the current facial state is the glasses-free state, multiple standard interactive applications are restricted and filtered, multiple important interactive applications are blocked, and multiple ordinary interactive applications are retained. Based on the facial spatial position, determine the vertical distance to the face, match the application interaction size, and plan the application display position according to the facial spatial position; At the application display location, a partially restricted, enlarged human-computer interaction is performed with the user, according to the application interaction size and multiple ordinary interactive applications.

[0007] As a further limitation of the technical solution of this embodiment of the invention, the step of performing binocular monitoring on the opposite area of ​​the display, acquiring binocular monitoring data, and identifying the binocular monitoring data to determine the user's current facial state and facial spatial position specifically includes the following steps: Perform binocular monitoring on the opposite area of ​​the display and acquire binocular monitoring data; Obtain eyeglass feature data; Based on the glasses feature data, feature recognition and matching are performed on the binocular monitoring data to determine the user's current facial state; Facial spatial localization is performed on the binocular monitoring data to determine the user's facial spatial position.

[0008] As a further limitation of the technical solution of this embodiment of the invention, when the current facial state is a state of wearing glasses, determining multiple standard interactive applications and conducting unrestricted standard human-computer interaction with the user specifically includes the following steps: When the current facial state is that the person is wearing glasses, a standard display interface is created; Identify multiple standard interactive applications; The user's first interactive operation is received in the standard display interface; Based on multiple standard interactive applications, the first interactive operation is displayed in a standard, unrestricted interactive response.

[0009] As a further limitation of the technical solution of this embodiment of the invention, when the current facial state is a glasses-free state, the step of restricting and filtering multiple standard interactive applications, blocking multiple important interactive applications, and retaining multiple ordinary interactive applications specifically includes the following steps: When the current facial state is that the glasses are off, a magnified display interface is created; Obtain the application basic information of multiple standard interactive applications; Perform attribute identification on multiple application-related basic information items and record the attribute identification results; Based on the attribute recognition results, the multiple standard interactive applications are classified to determine multiple important interactive applications and multiple ordinary interactive applications. Implement a restriction filter to block several important interactive applications while retaining several ordinary interactive applications.

[0010] As a further limitation of the technical solution of this embodiment of the invention, the step of determining the vertical distance of the face according to the facial spatial position, matching the application interaction size, and planning the application display position according to the facial spatial position specifically includes the following steps: Determine the vertical distance to the face based on the aforementioned facial spatial position; Match the corresponding application interaction size based on the vertical distance to the face; Based on the application interaction size, the display is uniformly divided to obtain multiple uniformly sized interactive display areas, and the center position of each area is determined. Create multiple center lines perpendicular to the display surface, based on the multiple center locations of the aforementioned regions; Calculate the distance between the spatial location of the face and the center lines of the plurality of regions; Based on the aforementioned distances, select the application display location from multiple regional center locations.

[0011] As a further limitation of the technical solution of this embodiment of the invention, the step of partially restricting the enlarged human-computer interaction with the user at the application display position according to the application interaction size and multiple ordinary interactive applications specifically includes the following steps: According to the application display position, select the current display area from the plurality of interactive display areas; In the current display area, a magnified display interface is dynamically displayed; Determine the scaling ratio of the interaction based on the application's interaction dimensions; According to the aforementioned interactive magnification ratio, the application icon, application window, interactive window, and / or interactive icon are magnified in the magnified display interface. The second interactive operation from the user is received in the magnified display interface; Based on multiple common interactive applications, the second interactive operation is magnified and the interactive response is restricted.

[0012] A vision-driven, immersive, multi-module feedback human-computer interaction system, comprising a binocular monitoring and recognition module, a standard interactive feedback module, an application restriction filtering module, a display position planning module, and a magnified interactive feedback module, wherein: The binocular monitoring and recognition module is used to perform binocular monitoring on the opposite area of ​​the display, acquire binocular monitoring data, and recognize the binocular monitoring data to determine the user's current facial state and facial spatial position. The standard interactive feedback module is used to determine multiple standard interactive applications when the current facial state is wearing glasses, and to conduct unrestricted standard human-computer interaction with the user; The application restriction and filtering module is used to restrict and filter multiple standard interactive applications when the current facial state is the glasses-free state, blocking multiple important interactive applications and retaining multiple ordinary interactive applications; The display position planning module is used to determine the vertical distance to the face according to the facial spatial position, match the application interaction size, and plan the application display position according to the facial spatial position. The magnified interactive feedback module is used to perform partially limited magnified human-computer interaction with the user at the application display position, according to the application interaction size and multiple ordinary interactive applications.

[0013] As a further limitation of the technical solution of this embodiment of the invention, the application restriction filtering module specifically includes: The interface creation unit is used to create a magnified display interface when the current facial state is a state of removing glasses; An application information acquisition unit is used to acquire basic application information of multiple standard interactive applications; An attribute recognition unit is used to perform attribute recognition on multiple application basic information items and record the attribute recognition results. The application classification unit is used to classify multiple standard interactive applications according to the attribute recognition results, and to determine multiple important interactive applications and multiple ordinary interactive applications; The restriction filtering unit is used to restrict and filter applications, blocking multiple important interactive applications while retaining multiple ordinary interactive applications.

[0014] As a further limitation of the technical solution of this embodiment of the invention, the display position planning module specifically includes: A facial distance determination unit is used to determine the vertical distance to the face according to the facial spatial position; An interaction size matching unit is used to match the corresponding application interaction size based on the vertical distance of the face; The area location determination unit is used to perform uniform division of the display based on the application interaction size to obtain multiple uniformly sized interactive display areas, and determine the corresponding area center position; The centerline creation unit is used to create multiple centerlines perpendicular to the display surface according to the multiple center positions of the regions; A distance calculation unit is used to calculate the distance between the spatial position of the face and the center lines of the multiple regions; The display location selection unit is used to select an application display location from multiple area center locations according to multiple said distances.

[0015] As a further limitation of the technical solution of this embodiment of the invention, the amplified interactive feedback module specifically includes: The display area selection unit is used to select the current display area from a plurality of interactive display areas according to the application display position; An interface display unit is used to dynamically display a magnified display interface in the current display area; The scaling ratio determination unit is used to determine the scaling ratio of the interaction according to the application interaction size; The magnification display unit is used to magnify and display application icons, application windows, interactive windows and / or interactive icons in the magnification display interface according to the interactive magnification ratio. An operation receiving unit is used to receive a second interactive operation from the user in the magnified display interface; An interactive response unit is used to enlarge and restrict the interactive response of the second interactive operation based on multiple common interactive applications.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention utilizes binocular monitoring of the display area; when wearing glasses, it enables unrestricted standard human-computer interaction with the user; when glasses are removed, it restricts and filters multiple standard interactive applications; matches application interaction sizes and plans application display positions; and provides partially restricted, magnified human-computer interaction with the user. This allows for unrestricted standard human-computer interaction when glasses are on; and when glasses are removed, it restricts and filters multiple standard interactive applications, matches application interaction sizes, plans application display positions, and provides partially restricted, magnified human-computer interaction with the user. This automatically identifies differences in the user's visual state and adaptively adjusts to ensure accurate and continuous interactive operations, improving the user experience for browsing effective information. Attached Figure Description

[0017] Figure 1 A flowchart of a vision-driven immersive multi-module feedback human-computer interaction method provided in an embodiment of the present invention is shown. Figure 2 The diagram illustrates the application architecture of a vision-driven, immersive, multi-module feedback human-computer interaction system provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Understandably, in existing technologies, human-computer interaction in computer devices typically uses uniform and fixed display and interaction parameter settings, which cannot adaptively adjust according to differences in the user's visual state. In particular, for nearsighted users, it is impossible to present and control different interactive feedback based on whether they are wearing glasses or not, affecting the user's experience of accurate and continuous interactive operations and effective information browsing.

[0020] To address the aforementioned issues, this invention discloses a vision-driven immersive multi-module feedback human-computer interaction method and system. This method involves performing binocular monitoring on the display area to acquire binocular monitoring data, identifying this data, and determining the user's current facial state and spatial position. When the user is wearing glasses, multiple standard interactive applications are identified for unrestricted standard human-computer interaction. When the user is not wearing glasses, these applications are restricted and filtered, with several important applications blocked and several ordinary applications retained. The vertical distance to the face is determined based on the facial spatial position, matching the application interaction size, and the application display position is planned accordingly. At the application display position, a partially restricted, magnified human-computer interaction is performed with the user, based on the application interaction size and the multiple ordinary interactive applications. It can perform unrestricted standard human-computer interaction with users when they are wearing glasses; when they are not wearing glasses, it can restrict and filter multiple standard interactive applications, match the application interaction size, plan the application display position, and perform partially restricted magnified human-computer interaction with users. This allows it to automatically identify differences in the user's visual state and make adaptive adjustments to ensure that users can perform accurate and continuous interactive operations, thereby improving the user experience of browsing effective information.

[0021] Specifically, Figure 1 A flowchart of a vision-driven immersive multi-module feedback human-computer interaction method provided in an embodiment of the present invention is shown.

[0022] In a preferred embodiment of the present invention, a vision-driven immersive multi-module feedback human-computer interaction method specifically includes the following steps: Step S101: Perform binocular monitoring on the opposite area of ​​the display, acquire binocular monitoring data, and identify the binocular monitoring data to determine the user's current facial state and facial spatial position.

[0023] In this embodiment of the invention, binocular monitoring is performed on the area opposite the display to obtain binocular monitoring data and glasses feature data. Based on the glasses feature data, feature recognition and matching are performed on the binocular monitoring data to determine the user's current facial state. Specifically, if the glasses feature can be matched, the current facial state is a state of wearing glasses; if the glasses feature cannot be matched, the current facial state is a state of not wearing glasses. At the same time, facial spatial positioning is performed on the binocular monitoring data to determine the user's facial spatial position. Specifically, facial spatial positioning is the process of spatially positioning the user's two eyes and then determining the spatial position of the center between the two eyes.

[0024] It is understood that, in this embodiment of the invention, the default is a user who is nearsighted and wears glasses.

[0025] It is understandable that the area opposite the display is the space opposite the monitor of the computer device.

[0026] Specifically, in another preferred embodiment provided by the present invention, the step of performing binocular monitoring on the opposite area of ​​the display, acquiring binocular monitoring data, and identifying the binocular monitoring data to determine the user's current facial state and facial spatial position specifically includes the following steps: Perform binocular monitoring on the opposite area of ​​the display and acquire binocular monitoring data; Obtain eyeglass feature data; Based on the glasses feature data, feature recognition and matching are performed on the binocular monitoring data to determine the user's current facial state; Facial spatial localization is performed on the binocular monitoring data to determine the user's facial spatial position.

[0027] Furthermore, the vision-driven immersive multi-module feedback human-computer interaction method also includes the following steps: Step S102: When the current facial state is wearing glasses, determine multiple standard interactive applications to conduct unrestricted standard human-computer interaction with the user.

[0028] In this embodiment of the invention, when the current facial state is that of wearing glasses, a standard display interface is created, and multiple standard interactive applications that have been installed on the computer device are identified. In the standard display interface, a first interactive operation performed by the user through the mouse and / or keyboard is received. Based on the multiple standard interactive applications, the first interactive operation is displayed in a standard, unrestricted interactive response in the standard display interface.

[0029] Understandably, unlimited interactive response refers to the unrestricted access to multiple standard interactive applications, where none of the applications have functional limitations.

[0030] It is understandable that during the process of "receiving the first interactive operation performed by the user through the mouse and / or keyboard in the standard display interface", the mouse icon and keyboard window are displayed at a standard size; during the process of "performing an unrestricted interactive response to the first interactive operation in the standard display interface", the application icons and application windows of multiple standard interactive applications are displayed at a standard size.

[0031] Specifically, in another preferred embodiment provided by the present invention, when the current facial state is a glasses-wearing state, determining multiple standard interactive applications to conduct unrestricted standard human-computer interaction with the user specifically includes the following steps: When the current facial state is that the person is wearing glasses, a standard display interface is created; Identify multiple standard interactive applications; The user's first interactive operation is received in the standard display interface; Based on multiple standard interactive applications, the first interactive operation is displayed in a standard, unrestricted interactive response.

[0032] Furthermore, the vision-driven immersive multi-module feedback human-computer interaction method also includes the following steps: Step S103: When the current facial state is the glasses-free state, restrict and filter the multiple standard interactive applications, block multiple important interactive applications, and retain multiple ordinary interactive applications.

[0033] In this embodiment of the invention, when the current facial state is that the glasses are off, a magnified display interface is created, and the application basic information of multiple standard interactive applications is obtained. By performing attribute recognition on the multiple application basic information, the attribute recognition results are recorded. Then, according to the attribute recognition results, the multiple standard interactive applications are classified into multiple important interactive applications and multiple ordinary interactive applications. After that, the multiple important interactive applications are blocked, and the multiple ordinary interactive applications are retained, thus completing the application restriction and filtering.

[0034] It is understood that the attributes of multiple standard interactive applications may include work attributes, learning attributes, entertainment attributes, news attributes, social attributes, financial attributes, and comprehensive attributes. Since users may be at risk of making incorrect interactive operations when their glasses are off, it is necessary to restrict and filter some applications to reduce this risk. In this embodiment of the invention, the attributes of the applications that need to be blocked include work attributes and financial attributes. Therefore, multiple important interactive applications include standard interactive applications corresponding to work and financial attributes; multiple ordinary interactive applications include standard interactive applications corresponding to learning, entertainment, news, social, and comprehensive attributes.

[0035] Specifically, in another preferred embodiment provided by the present invention, when the current facial state is a state of removing glasses, restricting and filtering multiple standard interactive applications, blocking multiple important interactive applications, and retaining multiple ordinary interactive applications specifically includes the following steps: When the current facial state is that the glasses are off, a magnified display interface is created; Obtain the application basic information of multiple standard interactive applications; Perform attribute identification on multiple application-related basic information items and record the attribute identification results; Based on the attribute recognition results, the multiple standard interactive applications are classified to determine multiple important interactive applications and multiple ordinary interactive applications. Implement a restriction filter to block several important interactive applications while retaining several ordinary interactive applications.

[0036] Furthermore, the vision-driven immersive multi-module feedback human-computer interaction method also includes the following steps: Step S104: Determine the vertical distance to the face according to the facial spatial position, match the application interaction size, and plan the application display position according to the facial spatial position.

[0037] In this embodiment of the invention, the vertical distance between the user's face and the display can be determined according to the facial spatial position. Then, based on the vertical distance, the corresponding application interaction size is matched, and based on the application interaction size, the display surface of the display is uniformly divided to obtain multiple uniformly sized interactive display areas. The center position of each interactive display area is determined. Then, based on the multiple center positions, multiple virtual center lines perpendicular to the display surface are created. By calculating the distance between the facial spatial position and the multiple center lines, the multiple distances are compared, and the shortest distance is selected. Then, the target center line corresponding to the shortest distance is selected from the multiple center lines. Finally, the application display position corresponding to the target center line is selected from the multiple center positions.

[0038] It is understood that, in this embodiment of the invention, the greater the vertical distance to the face, the larger the application interaction size, thereby compensating for the blurriness caused by the decline in visual state, and making it more conducive for users to make clear recognition and accurate operation when they are without glasses.

[0039] Specifically, in another preferred embodiment provided by the present invention, the step of determining the vertical distance of the face according to the facial spatial position, matching the application interaction size, and planning the application display position according to the facial spatial position specifically includes the following steps: Determine the vertical distance to the face based on the aforementioned facial spatial position; Match the corresponding application interaction size based on the vertical distance to the face; Based on the application interaction size, the display is uniformly divided to obtain multiple uniformly sized interactive display areas, and the center position of each area is determined. Create multiple center lines perpendicular to the display surface, based on the multiple center locations of the aforementioned regions; Calculate the distance between the spatial location of the face and the center lines of the plurality of regions; Based on the aforementioned distances, select the application display location from multiple regional center locations.

[0040] Furthermore, the vision-driven immersive multi-module feedback human-computer interaction method also includes the following steps: Step S105: At the application display location, according to the application interaction size and multiple ordinary interactive applications, perform partially restricted magnified human-computer interaction with the user.

[0041] In this embodiment of the invention, the current display area corresponding to the application display position is selected from multiple interactive display areas. Then, an enlarged display interface is dynamically displayed in the current display area. The interaction magnification ratio is determined according to the application interaction size. Then, according to the interaction magnification ratio, the application icon, application window, interaction window and / or interaction icon are enlarged in the enlarged display interface. The user's second interaction operation is received in the enlarged display interface. Based on multiple ordinary interactive applications, the second interaction operation is magnified and displayed in the enlarged display interface to restrict the interaction response.

[0042] It is understandable that the current display area may change when the user's facial spatial position changes. Therefore, it is necessary to dynamically display the magnified display interface in different current display areas according to the user's facial spatial position.

[0043] It is understood that, in the embodiments of the present invention, the interaction magnification ratio is positively correlated with the application interaction size; the larger the application interaction size, the larger the interaction magnification ratio.

[0044] It is understood that in this embodiment of the invention, the interactive window is the keyboard window; the interactive icon is the mouse icon.

[0045] It is understood that restricting interactive responses refers to limiting the access permissions of multiple ordinary interactive applications and restricting the interactive state. In this embodiment of the invention, some functions of ordinary interactive applications with comprehensive attributes are subject to usage restrictions, including work functions and financial functions.

[0046] Specifically, in another preferred embodiment provided by the present invention, the step of partially restricting the enlarged human-computer interaction with the user at the application display position, according to the application interaction size and multiple ordinary interactive applications, specifically includes the following steps: According to the application display position, select the current display area from the plurality of interactive display areas; In the current display area, a magnified display interface is dynamically displayed; Determine the scaling ratio of the interaction based on the application's interaction dimensions; According to the aforementioned interactive magnification ratio, the application icon, application window, interactive window, and / or interactive icon are magnified in the magnified display interface. The second interactive operation from the user is received in the magnified display interface; Based on multiple common interactive applications, the second interactive operation is magnified and the interactive response is restricted.

[0047] Furthermore, Figure 2 The diagram illustrates the application architecture of a vision-driven, immersive, multi-module feedback human-computer interaction system provided in an embodiment of the present invention.

[0048] Specifically, in another preferred embodiment provided by the present invention, a vision-driven immersive multi-module feedback human-computer interaction system includes: The binocular monitoring and recognition module 101 is used to perform binocular monitoring on the opposite area of ​​the display, acquire binocular monitoring data, and identify the binocular monitoring data to determine the user's current facial state and facial spatial position.

[0049] In this embodiment of the invention, the binocular monitoring and recognition module 101 performs binocular monitoring on the opposite area of ​​the display to obtain binocular monitoring data and glasses feature data. Based on the glasses feature data, it performs feature recognition and matching on the binocular monitoring data to determine the user's current facial state. Specifically, if the glasses feature can be matched, the current facial state is a state of wearing glasses; if the glasses feature cannot be matched, the current facial state is a state of not wearing glasses. At the same time, it performs facial spatial positioning on the binocular monitoring data to determine the user's facial spatial position. Specifically, facial spatial positioning is the process of spatially positioning the user's two eyes and then determining the spatial position of the center between the two eyes.

[0050] The standard interactive feedback module 102 is used to determine multiple standard interactive applications when the current facial state is a glasses-wearing state, and to conduct unrestricted standard human-computer interaction with the user.

[0051] In this embodiment of the invention, when the current facial state is that of wearing glasses, the standard interactive feedback module 102 creates a standard display interface and determines multiple standard interactive applications that have been installed in the computer device. In the standard display interface, it receives the first interactive operation performed by the user through the mouse and / or keyboard, and then, based on the multiple standard interactive applications, performs an unrestricted interactive response to the first interactive operation in the standard display interface.

[0052] The application restriction filtering module 103 is used to restrict and filter multiple standard interactive applications when the current facial state is the glasses-free state, blocking multiple important interactive applications and retaining multiple ordinary interactive applications.

[0053] In this embodiment of the invention, when the current facial state is that the glasses are off, the application restriction filtering module 103 creates a magnified display interface and obtains the application basic information of multiple standard interactive applications. By performing attribute recognition on the multiple application basic information, the attribute recognition results are recorded. Then, according to the attribute recognition results, the multiple standard interactive applications are classified into multiple important interactive applications and multiple ordinary interactive applications. After that, the multiple important interactive applications are blocked and the multiple ordinary interactive applications are retained, thus completing the application restriction filtering.

[0054] Specifically, in another preferred embodiment provided by the present invention, the application restriction filtering module 103 specifically includes: The interface creation unit is used to create a magnified display interface when the current facial state is a state of removing glasses; An application information acquisition unit is used to acquire basic application information of multiple standard interactive applications; An attribute recognition unit is used to perform attribute recognition on multiple application basic information items and record the attribute recognition results. The application classification unit is used to classify multiple standard interactive applications according to the attribute recognition results, and to determine multiple important interactive applications and multiple ordinary interactive applications; The restriction filtering unit is used to restrict and filter applications, blocking multiple important interactive applications while retaining multiple ordinary interactive applications.

[0055] Furthermore, the vision-driven immersive multi-module feedback human-computer interaction system also includes: The display position planning module 104 is used to determine the vertical distance of the face according to the facial spatial position, match the application interaction size, and plan the application display position according to the facial spatial position.

[0056] In this embodiment of the invention, the display position planning module 104 determines the vertical distance between the user's face and the display based on the facial spatial position. Then, based on the vertical distance, it matches the corresponding application interaction size and, based on the application interaction size, uniformly divides the display surface of the display to obtain multiple uniformly sized interactive display areas. It also determines the center position of each interactive display area. Then, based on the multiple center positions, it creates multiple virtual center lines perpendicular to the display surface. By calculating the distance between the facial spatial position and the multiple center lines, it compares the multiple distances and selects the shortest distance. Then, it selects the target center line corresponding to the shortest distance from the multiple center lines, and finally selects the application display position corresponding to the target center line from the multiple center positions.

[0057] Specifically, in another preferred embodiment provided by the present invention, the display position planning module 104 specifically includes: A facial distance determination unit is used to determine the vertical distance to the face according to the facial spatial position; An interaction size matching unit is used to match the corresponding application interaction size based on the vertical distance of the face; The area location determination unit is used to perform uniform division of the display based on the application interaction size to obtain multiple uniformly sized interactive display areas, and determine the corresponding area center position; The centerline creation unit is used to create multiple centerlines perpendicular to the display surface according to the multiple center positions of the regions; A distance calculation unit is used to calculate the distance between the spatial position of the face and the center lines of the multiple regions; The display location selection unit is used to select an application display location from multiple area center locations according to multiple said distances.

[0058] Furthermore, the vision-driven immersive multi-module feedback human-computer interaction system also includes: The magnified interactive feedback module 105 is used to perform partially limited magnified human-computer interaction with the user at the application display position, according to the application interaction size and multiple ordinary interactive applications.

[0059] In this embodiment of the invention, the magnified interactive feedback module 105 selects the current display area corresponding to the application display position from multiple interactive display areas, and then dynamically displays the magnified display interface in the current display area. According to the application interaction size, the interaction magnification ratio is determined, and then the application icon, application window, interaction window and / or interaction icon are magnified in the magnified display interface according to the interaction magnification ratio. In the magnified display interface, the user's second interaction operation is received, and then based on multiple ordinary interactive applications, the second interaction operation is magnified and restricted interactive response in the magnified display interface.

[0060] Specifically, in another preferred embodiment provided by the present invention, the amplified interactive feedback module 105 specifically includes: The display area selection unit is used to select the current display area from a plurality of interactive display areas according to the application display position; An interface display unit is used to dynamically display a magnified display interface in the current display area; The scaling ratio determination unit is used to determine the scaling ratio of the interaction according to the application interaction size; The magnification display unit is used to magnify and display application icons, application windows, interactive windows and / or interactive icons in the magnification display interface according to the interactive magnification ratio. An operation receiving unit is used to receive a second interactive operation from the user in the magnified display interface; An interactive response unit is used to enlarge and restrict the interactive response of the second interactive operation based on multiple common interactive applications.

[0061] The above-described embodiments are merely examples of several implementations of the present invention, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A vision-driven, immersive, multi-module feedback human-computer interaction method, characterized in that, The method specifically includes the following steps: The system performs binocular monitoring on the opposite area of ​​the display, acquires binocular monitoring data, and identifies the binocular monitoring data to determine the user's current facial state and facial spatial position. When the current facial state is that of wearing glasses, multiple standard interactive applications are identified to conduct unrestricted standard human-computer interaction with the user. When the current facial state is the glasses-free state, multiple standard interactive applications are restricted and filtered, multiple important interactive applications are blocked, and multiple ordinary interactive applications are retained. Based on the facial spatial position, determine the vertical distance to the face, match the application interaction size, and plan the application display position according to the facial spatial position; At the application display location, a partially restricted, enlarged human-computer interaction is performed with the user, according to the application interaction size and multiple ordinary interactive applications.

2. The vision-driven immersive multi-module feedback human-computer interaction method according to claim 1, characterized in that, The process of performing binocular monitoring on the opposite area of ​​the display, acquiring binocular monitoring data, and identifying the binocular monitoring data to determine the user's current facial state and facial spatial position specifically includes the following steps: Perform binocular monitoring on the opposite area of ​​the display and acquire binocular monitoring data; Obtain eyeglass feature data; Based on the glasses feature data, feature recognition and matching are performed on the binocular monitoring data to determine the user's current facial state; Facial spatial localization is performed on the binocular monitoring data to determine the user's facial spatial position.

3. The vision-driven immersive multi-module feedback human-computer interaction method according to claim 1, characterized in that, When the current facial state is that of wearing glasses, determining multiple standard interactive applications to conduct unrestricted standard human-computer interaction with the user specifically includes the following steps: When the current facial state is that the person is wearing glasses, a standard display interface is created; Identify multiple standard interactive applications; The user's first interactive operation is received in the standard display interface; Based on multiple standard interactive applications, the first interactive operation is displayed in a standard, unrestricted interactive response.

4. The vision-driven immersive multi-module feedback human-computer interaction method according to claim 1, characterized in that, When the current facial state is a glasses-free state, the process of restricting and filtering multiple standard interactive applications, blocking multiple important interactive applications, and retaining multiple ordinary interactive applications specifically includes the following steps: When the current facial state is that the glasses are off, a magnified display interface is created; Obtain the application basic information of multiple standard interactive applications; Perform attribute identification on multiple application-related basic information items and record the attribute identification results; Based on the attribute recognition results, the multiple standard interactive applications are classified to determine multiple important interactive applications and multiple ordinary interactive applications. Implement a restriction filter to block several important interactive applications while retaining several ordinary interactive applications.

5. The vision-driven immersive multi-module feedback human-computer interaction method according to claim 4, characterized in that, The steps of determining the vertical distance to the face based on the facial spatial position, matching the application interaction size, and planning the application display position based on the facial spatial position specifically include the following steps: Determine the vertical distance to the face based on the aforementioned facial spatial position; Match the corresponding application interaction size based on the vertical distance to the face; Based on the application interaction size, the display is uniformly divided to obtain multiple uniformly sized interactive display areas, and the center position of each area is determined. Create multiple center lines perpendicular to the display surface, based on the multiple center locations of the aforementioned regions; Calculate the distance between the spatial location of the face and the center lines of the plurality of regions; Based on the aforementioned distances, select the application display location from multiple regional center locations.

6. The vision-driven immersive multi-module feedback human-computer interaction method according to claim 5, characterized in that, The step of enlarging the human-computer interaction with the user in a partially restricted manner at the application display location, according to the application interaction size and multiple ordinary interactive applications, specifically includes the following steps: According to the application display position, select the current display area from the plurality of interactive display areas; In the current display area, a magnified display interface is dynamically displayed; Determine the scaling ratio of the interaction based on the application's interaction dimensions; According to the aforementioned interactive magnification ratio, the application icon, application window, interactive window, and / or interactive icon are magnified in the magnified display interface. The second interactive operation from the user is received in the magnified display interface; Based on multiple common interactive applications, the second interactive operation is magnified and the interactive response is restricted.

7. A vision-driven, immersive, multi-module feedback human-computer interaction system, characterized in that, The system includes a binocular monitoring and recognition module, a standard interactive feedback module, an application restriction filtering module, a display position planning module, and a magnified interactive feedback module, wherein: The binocular monitoring and recognition module is used to perform binocular monitoring on the opposite area of ​​the display, acquire binocular monitoring data, and recognize the binocular monitoring data to determine the user's current facial state and facial spatial position. The standard interactive feedback module is used to determine multiple standard interactive applications when the current facial state is wearing glasses, and to conduct unrestricted standard human-computer interaction with the user; The application restriction and filtering module is used to restrict and filter multiple standard interactive applications when the current facial state is the glasses-free state, blocking multiple important interactive applications and retaining multiple ordinary interactive applications; The display position planning module is used to determine the vertical distance to the face according to the facial spatial position, match the application interaction size, and plan the application display position according to the facial spatial position. The magnified interactive feedback module is used to perform partially limited magnified human-computer interaction with the user at the application display position, according to the application interaction size and multiple ordinary interactive applications.

8. The vision-driven immersive multi-module feedback human-computer interaction system according to claim 7, characterized in that, The application restriction filtering module specifically includes: The interface creation unit is used to create a magnified display interface when the current facial state is a state of removing glasses; An application information acquisition unit is used to acquire basic application information of multiple standard interactive applications; An attribute recognition unit is used to perform attribute recognition on multiple application basic information items and record the attribute recognition results. The application classification unit is used to classify multiple standard interactive applications according to the attribute recognition results, and to determine multiple important interactive applications and multiple ordinary interactive applications; The restriction filtering unit is used to restrict and filter applications, blocking multiple important interactive applications while retaining multiple ordinary interactive applications.

9. The vision-driven immersive multi-module feedback human-computer interaction system according to claim 8, characterized in that, The display location planning module specifically includes: A facial distance determination unit is used to determine the vertical distance to the face according to the facial spatial position; An interaction size matching unit is used to match the corresponding application interaction size based on the vertical distance of the face; The area location determination unit is used to perform uniform division of the display based on the application interaction size to obtain multiple uniformly sized interactive display areas, and determine the corresponding area center position; The centerline creation unit is used to create multiple centerlines perpendicular to the display surface according to the multiple center positions of the regions; A distance calculation unit is used to calculate the distance between the spatial position of the face and the center lines of the multiple regions; The display location selection unit is used to select an application display location from multiple area center locations according to multiple said distances.

10. The vision-driven immersive multi-module feedback human-computer interaction system according to claim 9, characterized in that, The amplified interactive feedback module specifically includes: The display area selection unit is used to select the current display area from a plurality of interactive display areas according to the application display position; An interface display unit is used to dynamically display a magnified display interface in the current display area; The scaling ratio determination unit is used to determine the scaling ratio of the interaction according to the application interaction size; The magnification display unit is used to magnify and display application icons, application windows, interactive windows and / or interactive icons in the magnification display interface according to the interactive magnification ratio. An operation receiving unit is used to receive a second interactive operation from the user in the magnified display interface; An interactive response unit is used to enlarge and restrict the interactive response of the second interactive operation based on multiple common interactive applications.

Images