Human-computer interaction method, system and device

Abstract

The present invention provides a human-computer interaction method, system and device, which establish a communication connection between a drone and a head-mounted display terminal, transmit the external environment image collected by the drone back to the head-mounted display terminal, and adjust the image display mode based on a first action initiated by a wearer of the head-mounted display terminal; further perform line of sight tracking and identification on the wearer to obtain the viewing field information of the wearer in the image display space, thereby adjusting the shooting state of the drone for the external environment; further adjust the shooting state and image data return state of the drone again based on the picture feature information of the external environment image transmitted back in real time, thereby ensuring the image display quality of the head-mounted display terminal and satisfying the wearer's viewing needs for different areas of the external environment in real time; further determine the wearer's image area of ​​interest in the image display space based on the wearer's second action feature information, thereby adjusting the flight state of the drone, realizing synchronous human-computer interaction between the wearer, the head-mounted display terminal and the drone, and improving the scene applicability and reliability of human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a human-computer interaction method, system, or device. Background Art

[0002] Human-computer interaction has become an important development direction of artificial intelligence. The existing human-computer interaction mainly recognizes the user's actions and voices to achieve interaction between users and smart terminals and meet the user's control needs for smart terminals. Most of the existing human-computer interactions are one-to-one interactions between users and a single smart terminal. There are differences in business performance between different smart terminals. If users need to interact with multiple smart terminals at the same time, all smart terminals need to be integrated into a unified platform, which places high demands on both the hardware and software of human-computer interaction. When the smart terminal changes, the unified platform needs to be redesigned, which cannot achieve the scene applicability of different smart terminals and increase the development workload of human-computer interaction. Especially in the human-computer interaction scenario where drones and head-mounted display terminals work together, users can only interact with drones and head-mounted display terminals separately, and cannot achieve unified integrated interactive operations for drones and head-mounted display terminals. Summary of the invention

[0003] The object of the present invention is to provide a human-computer interaction method, system and device, which establish a communication connection between a drone and a head-mounted display terminal, transmit the external environment image collected by the drone back to the head-mounted display terminal, and adjust the image display mode based on the first action initiated by the wearer of the head-mounted display terminal to provide a suitable image display scene for the wearer; also perform line of sight tracking and recognition on the wearer to obtain the viewing field information of the wearer in the image display space, thereby adjusting the drone's shooting state of the external environment and realizing adaptive shooting adjustment of the drone; and further adjust the drone's shooting state based on the picture feature information of the external environment image transmitted back in real time. The system can determine the image area of ​​interest of the wearer in the image display space based on the second motion feature information of the wearer, so as to adjust the flight state of the drone. The drone is connected to the head-mounted display terminal for communication, and the head-mounted display terminal is used as the interaction intermediary between the wearer and the drone, so that the human-computer interaction initiated by the wearer to the head-mounted display terminal can directly control the drone, thereby realizing the synchronous human-computer interaction between the wearer, the head-mounted display terminal and the drone, and improving the scene applicability and reliability of the human-computer interaction.

[0004] The present invention is achieved through the following technical solutions:

[0005] A human-computer interaction method, comprising:

[0006] Based on the working states of the drone and the head-mounted display terminal, a communication connection is established between the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal; based on the first action initiated by the wearer of the head-mounted display terminal, an image display mode of the head-mounted display terminal is adjusted;

[0007] Tracking and identifying the wearer's line of sight to obtain viewing field information of the wearer in the image display space of the head-mounted display terminal; adjusting the drone's shooting state of the external environment based on the viewing field information, and then adjusting the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image transmitted back by the drone in real time;

[0008] The wearer is subjected to second motion tracking and identification to obtain second motion feature information of the wearer, and an image region of interest of the wearer in the image display space is determined based on the second motion feature information; and the flight state of the drone is adjusted based on the position information of the image region of interest.

[0009] Optionally, based on the working states of the drone and the head mounted display terminal, establishing a communication connection between the drone and the head mounted display terminal, so as to transmit the external environment image collected by the drone back to the head mounted display terminal; and based on the first action initiated by the wearer of the head mounted display terminal, adjusting the image display mode of the head mounted display terminal, including:

[0010] Identify the drone and the head-mounted display terminal to determine whether the drone and the head-mounted display terminal are both in a powered-on working state; if so, trigger the drone and the head-mounted display terminal to enter a base station search mode, so as to obtain the wireless communication signal status information of all base stations in the surrounding environment; based on the wireless communication signal status information, select a base station that matches the drone and the head-mounted display terminal, and build a communication channel between the drone and the head-mounted display terminal based on the wireless network address information of the selected base station, so as to transfer the external environment image collected by the drone to the selected base station and then transmit it back to the head-mounted display terminal;

[0011] Dynamically detect the head of the wearer of the head-mounted display terminal to obtain head movement posture information and head movement amplitude information of the wearer; fit the head movement trajectory information of the wearer based on the head movement posture information and the head movement amplitude information; compare the head movement trajectory information with the display control instruction form of the head-mounted display terminal, select a matching display control instruction therefrom, and switch the display interface of the head-mounted display terminal to a single image display window or a multi-image display window based on the matching display control instruction.

[0012] Optionally, performing line of sight tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal; adjusting the shooting state of the drone for the external environment based on the viewing field information, and then adjusting the shooting state of the drone or the image data return state of the drone again based on the picture feature information of the external environment image transmitted back by the drone in real time, including:

[0013] Tracking and identifying the wearer's sight line when the wearer is viewing the display interface, and obtaining the projection area range information of the wearer's sight line on the display interface, which serves as the viewing field information;

[0014] Compare the projection area range information with the area range information of all image display windows currently existing on the display interface, and determine the position information of the wearer's viewing line corresponding to the coverage area of ​​the external environment; then, based on the coverage area position information, instruct the drone to aim and zoom the coverage area of ​​the external environment;

[0015] Performing picture pixel contour recognition on the external environment image transmitted back by the drone in real time to obtain picture jitter offset characteristic information of the external environment image; wherein the picture jitter offset characteristic information includes picture jitter offset direction and offset amount;

[0016] Based on the image jitter offset characteristic information, the drone is subjected to shooting jitter correction; and during the execution of the shooting jitter correction, the drone is suspended from transmitting image data back to the head mounted display terminal through the base station until the shooting jitter correction is completed.

[0017] Optionally, performing second motion tracking and identification on the wearer to obtain second motion feature information of the wearer, determining an image region of interest of the wearer in the image display space based on the second motion feature information; and adjusting the flight state of the drone based on position information of the image region of interest, including:

[0018] Dynamically photograph the wearer's hand to obtain a dynamic image of the wearer's hand; recognize the dynamic image of the hand to obtain a motion trajectory of the wearer's fingertips in the external environment space; based on the transformation relationship between the external environment space and the image display space, map the motion trajectory to the image display space, determine the image screen area corresponding to the wearer's finger in the image display space, and use it as the image area of ​​interest of the wearer in the image display space;

[0019] Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined; and then based on the position of the area of ​​interest and the real-time position of the drone, the flight path of the drone is adjusted so that the drone flies above the area of ​​interest.

[0020] Optionally, the head mounted display terminal is extended with an auxiliary support frame mounted on the shoulder of the wearer, and the auxiliary support frame is used to increase the rotation damping when the wearer turns his head to prevent the head from turning too fast and causing false triggering interference to the shooting work of the drone, which includes:

[0021] Step S1, using the following formula (1), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, control the shooting anti-shake correction of the drone,

[0022]

[0023] In the above formula (1), D represents the anti-shake control value of the drone shooting state; θ0 represents the preset micro-motion angle; Indicates the first head rotation angle detected within the adjacent preset time interval; Indicates the second head rotation angle detected within the adjacent preset time interval; || indicates obtaining the absolute value; & indicates the logical relationship; if[] indicates a judgment function. If the formula in the brackets is true, the function value of the judgment function is 1; if the formula in the brackets is not true, the function value of the judgment function is 0;

[0024] If D=1, the current camera shake correction of the drone will not be exited;

[0025] If D=0, then the current camera anti-shake correction of the drone is exited;

[0026] Step S2, using the following formula (2), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, determining whether to control the auxiliary support frame to trigger the rotation damping,

[0027]

[0028] In the above formula (2), C represents the trigger control value of the rotation damping; T0 represents the time length corresponding to the preset time interval; ω0 represents the minimum angular velocity for triggering the rotation damping;

[0029] If C=1, the auxiliary support frame is controlled to trigger rotation damping;

[0030] If C=0, the auxiliary support frame is not controlled to trigger the rotation damping;

[0031] Step S3, when the auxiliary support frame successfully triggers the rotation damping, the following formula (3) is used to control the magnitude of the rotation damping force applied by the auxiliary support frame according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals:

[0032]

[0033] In the above formula (3), F represents the magnitude of the rotational damping force applied by the auxiliary support frame; α represents the angle between the auxiliary support frame and the shoulder; R represents the preset standard head radius; M represents the preset standard head mass; and m0 represents the mass of the head-mounted display terminal.

[0034] A human-computer interaction system, comprising:

[0035] A communication connection module, used to establish a communication connection between the drone and the head-mounted display terminal based on the working status of the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal;

[0036] an image display mode adjustment module, configured to adjust the image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal;

[0037] A sight tracking and identification module, used to perform sight tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal;

[0038] A drone shooting and data return adjustment module is used to adjust the drone's shooting state of the external environment based on the viewing field information, and then adjust the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image returned by the drone in real time;

[0039] a motion tracking and identifying module, configured to perform a second motion tracking and identifying operation on the wearer, obtain second motion feature information of the wearer, and determine an image region of interest of the wearer in the image display space based on the second motion feature information;

[0040] The UAV flight adjustment module is used to adjust the flight state of the UAV based on the position information of the image area of ​​interest.

[0041] Optionally, the communication connection module is used to establish a communication connection between the drone and the head-mounted display terminal based on the working status of the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal, including:

[0042] Identify the drone and the head-mounted display terminal to determine whether the drone and the head-mounted display terminal are both in a powered-on working state; if so, trigger the drone and the head-mounted display terminal to enter a base station search mode, so as to obtain the wireless communication signal status information of all base stations in the surrounding environment; based on the wireless communication signal status information, select a base station that matches the drone and the head-mounted display terminal, and build a communication channel between the drone and the head-mounted display terminal based on the wireless network address information of the selected base station, so as to transfer the external environment image collected by the drone to the selected base station and then transmit it back to the head-mounted display terminal;

[0043] The image display mode adjustment module is used to adjust the image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal, including:

[0044] Dynamically detect the head of the wearer of the head-mounted display terminal to obtain head movement posture information and head movement amplitude information of the wearer; fit the head movement trajectory information of the wearer based on the head movement posture information and the head movement amplitude information; compare the head movement trajectory information with the display control instruction form of the head-mounted display terminal, select a matching display control instruction therefrom, and switch the display interface of the head-mounted display terminal to a single image display window or a multi-image display window based on the matching display control instruction.

[0045] Optionally, the gaze tracking and identification module is used to perform gaze tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal, including:

[0046] Tracking and identifying the wearer's sight line when the wearer is viewing the display interface, and obtaining the projection area range information of the wearer's sight line on the display interface, which serves as the viewing field information;

[0047] The drone shooting and data return adjustment module is used to adjust the drone's shooting state of the external environment based on the viewing field information, and then adjust the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image returned by the drone in real time, including:

[0048] Compare the projection area range information with the area range information of all image display windows currently existing on the display interface, and determine the position information of the wearer's viewing line corresponding to the coverage area of ​​the external environment; then, based on the coverage area position information, instruct the drone to aim and zoom the coverage area of ​​the external environment;

[0049] Performing picture pixel contour recognition on the external environment image transmitted back by the drone in real time to obtain picture jitter offset characteristic information of the external environment image; wherein the picture jitter offset characteristic information includes picture jitter offset direction and offset amount;

[0050] Based on the image jitter offset characteristic information, the drone is subjected to shooting jitter correction; and during the execution of the shooting jitter correction, the drone is suspended from transmitting image data back to the head mounted display terminal through the base station until the shooting jitter correction is completed.

[0051] Optionally, the motion tracking and identifying module is used to perform a second motion tracking and identifying operation on the wearer to obtain second motion feature information of the wearer, and determine an image region of interest of the wearer in the image display space based on the second motion feature information, including:

[0052] Dynamically photograph the wearer's hand to obtain a dynamic image of the wearer's hand; recognize the dynamic image of the hand to obtain a motion trajectory of the wearer's fingertips in the external environment space; based on the transformation relationship between the external environment space and the image display space, map the motion trajectory to the image display space, determine the image screen area corresponding to the wearer's finger in the image display space, and use it as the image area of ​​interest of the wearer in the image display space;

[0053] The UAV flight adjustment module is used to adjust the flight state of the UAV based on the position information of the image area of ​​interest, including:

[0054] Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined; and then based on the position of the area of ​​interest and the real-time position of the drone, the flight path of the drone is adjusted so that the drone flies above the area of ​​interest.

[0055] A human-computer interaction device comprises: a drone, a head-mounted display terminal and a human-computer interaction system; the human-computer interaction system is used to implement a human-computer interaction method.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present application provides a human-computer interaction method, system and device for establishing a communication connection between a drone and a head-mounted display terminal, transmitting the external environment image collected by the drone back to the head-mounted display terminal, and adjusting the image display mode based on a first action initiated by the wearer of the head-mounted display terminal to provide a suitable image display scene for the wearer; the wearer's line of sight is tracked and identified to obtain the wearer's viewing field information in the image display space, thereby adjusting the drone's shooting state of the external environment and realizing adaptive shooting adjustment of the drone; the drone's shooting state is adjusted again based on the picture feature information of the external environment image transmitted back in real time. and image data return status, ensuring the image display quality of the head-mounted display terminal and meeting the wearer's viewing needs for different areas of the external environment in real time; also based on the wearer's second action feature information, determining the wearer's image area of ​​interest in the image display space, so as to adjust the flight state of the drone, which connects the drone with the head-mounted display terminal for communication, and uses the head-mounted display terminal as an interaction intermediary between the wearer and the drone, so that the human-computer interaction initiated by the wearer to the head-mounted display terminal can directly control the drone, thereby realizing synchronous human-computer interaction between the wearer, the head-mounted display terminal and the drone, and improving the scene applicability and reliability of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0059] Figure 1 A schematic flow chart of a human-computer interaction method provided by the present invention.

[0060] Figure 2 A schematic diagram of the structure of a human-computer interaction system provided by the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0062] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0063] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] See also Figure 1 As shown, an embodiment of the present application provides a human-computer interaction method. The human-computer interaction method includes:

[0065] Based on the working states of the drone and the head-mounted display terminal, a communication connection is established between the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal; based on the first action initiated by the wearer of the head-mounted display terminal, an image display mode of the head-mounted display terminal is adjusted;

[0066] Tracking and identifying the wearer's line of sight to obtain viewing field information of the wearer in the image display space of the head-mounted display terminal; adjusting the drone's shooting state of the external environment based on the viewing field information, and then adjusting the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image transmitted back by the drone in real time;

[0067] The wearer is subjected to second motion tracking and identification to obtain second motion feature information of the wearer, and based on the second motion feature information, an image region of interest of the wearer in the image display space is determined; and based on the position information of the image region of interest, the flight state of the drone is adjusted.

[0068] The beneficial effects of the above embodiments are as follows: the human-computer interaction method establishes a communication connection between the drone and the head-mounted display terminal, transmits the external environment image collected by the drone back to the head-mounted display terminal, and adjusts the image display mode based on the first action initiated by the wearer of the head-mounted display terminal to provide a suitable image display scene for the wearer; the wearer's line of sight is tracked and identified to obtain the wearer's viewing field information in the image display space, thereby adjusting the drone's shooting state of the external environment and realizing adaptive shooting adjustment of the drone; the drone's shooting state and The image data is transmitted back to the status to ensure the image display quality of the head-mounted display terminal and to meet the wearer's viewing needs for different areas of the external environment in real time; based on the wearer's second action feature information, the image area of ​​interest of the wearer in the image display space is determined to adjust the flight status of the drone, which connects the drone to the head-mounted display terminal for communication, and uses the head-mounted display terminal as an interaction intermediary between the wearer and the drone, so that the human-computer interaction initiated by the wearer to the head-mounted display terminal can directly control the drone, realize the synchronous human-computer interaction between the wearer, the head-mounted display terminal and the drone, and improve the scene applicability and reliability of the human-computer interaction.

[0069] In another embodiment, based on the working states of the drone and the head mounted display terminal, a communication connection is established between the drone and the head mounted display terminal, so as to transmit the external environment image collected by the drone back to the head mounted display terminal; and based on the first action initiated by the wearer of the head mounted display terminal, the image display mode of the head mounted display terminal is adjusted, including:

[0070] Identify the drone and the head-mounted display terminal to determine whether the drone and the head-mounted display terminal are both in a powered-on working state; if so, trigger the drone and the head-mounted display terminal to enter a base station search mode, so as to obtain the wireless communication signal status information of all base stations in the surrounding environment; based on the wireless communication signal status information, select a base station that matches the drone and the head-mounted display terminal, and build a communication channel between the drone and the head-mounted display terminal based on the wireless network address information of the selected base station, so as to transfer the external environment image collected by the drone to the selected base station and then transmit it back to the head-mounted display terminal;

[0071] Dynamic detection is performed on the head of a wearer of the head mounted display terminal to obtain head movement posture information and head movement amplitude information of the wearer; based on the head movement posture information and the head movement amplitude information, head movement trajectory information of the wearer is fitted; the head movement trajectory information is compared with a display control instruction form of the head mounted display terminal, a matching display control instruction is selected therefrom, and based on the matching display control instruction, a display interface of the head mounted display terminal is switched to a single image display window or a multiple image display window.

[0072] The beneficial effects of the above embodiments are that the drone and the head-mounted display terminal are two independent electronic devices. If the two electronic devices are directly connected wirelessly, the wireless communication performance of the drone and the head-mounted display terminal will be affected. During the movement of the drone or the head-mounted display terminal, the two cannot always be within the effective wireless communication range of each other, resulting in instability of the direct wireless connection between the drone and the head-mounted display terminal. In order to achieve a stable and continuous wireless communication connection between the drone and the head-mounted display terminal, a wireless communication base station is required as a wireless communication intermediary between the drone and the head-mounted display terminal. To this end, the drone and the head-mounted display terminal are identified to determine whether the drone and the head-mounted display terminal are both in a powered-on working state. When the drone and the head-mounted display terminal are both in a powered-on working state, the drone and the head-mounted display terminal will automatically perform a wireless base station search working mode to obtain the wireless communication signal status information of all base stations in the surrounding environment; wherein the wireless communication signal status information may be, but is not limited to, the wireless signal strength information of the base station, the effective coverage range information of the wireless signal, and the wireless communication bandwidth information, etc.; then based on the wireless communication signal status information of all base stations, the base station with the best wireless communication performance is identified, and based on the wireless network address information of the base station with the best wireless communication performance, a communication channel between the drone and the head-mounted display terminal is established, and the base station is used as a wireless communication intermediary to establish a stable and continuous wireless communication connection between the drone and the head-mounted display terminal, so as to avoid wireless communication interruption between the two, and ensure that the drone transfers the collected external environment image to the selected base station and then transmits it back to the head-mounted display terminal, so as to provide a stable and reliable image display for the wearer. There is also a dynamic detection of the head of the wearer of the head-mounted display terminal to obtain the head movement posture information and head movement amplitude information of the wearer, so that the head movement trajectory of the wearer can be used as the basis for the human-computer interaction between the wearer and the head-mounted display terminal, so that the wearer can achieve the corresponding human-computer interaction without directly performing physical contact operation on the head-mounted display terminal. Specifically, based on the head movement posture information and the head movement amplitude information, the head movement trajectory information of the wearer is fitted, so that the head movement of the wearer can be accurately calibrated. Then the head movement trajectory information is compared with the display control instruction form of the head-mounted display terminal, and the matching display control instruction is selected from it, so that the control action type initiated by the wearer's own head is accurately identified. And based on the matching display control instruction, the display interface of the head-mounted display terminal is switched to a single image display window or a multi-image display window, so that the display interface of the head-mounted display terminal can provide the wearer with a single image display window or a multi-image display window, and each image display window can independently display the external environment image collected by the drone, so that the wearer obtains a different image viewing experience.

[0073] In another embodiment, the wearer is tracked and identified to obtain viewing field information of the wearer in the image display space of the head mounted display terminal; based on the viewing field information, the drone's shooting state of the external environment is adjusted; and based on the picture feature information of the external environment image transmitted back by the drone in real time, the drone's shooting state or the drone's image data transmission state is adjusted again, including:

[0074] Tracking and identifying the wearer's sight line when the wearer is viewing the display interface, and obtaining the projection area range information of the wearer's sight line on the display interface as the viewing field information;

[0075] The projection area range information is compared with the area range information of all image display windows currently existing in the display interface to determine the position information of the wearer's viewing line corresponding to the coverage area of ​​the external environment; based on the coverage area position information, the drone is instructed to aim and zoom the coverage area of ​​the external environment;

[0076] Performing image pixel contour recognition on the external environment image transmitted back by the drone in real time to obtain image jitter offset characteristic information of the external environment image; wherein the image jitter offset characteristic information includes the image jitter offset direction and offset amount;

[0077] Based on the image jitter offset characteristic information, the drone is subjected to shooting jitter correction; and during the execution of the shooting jitter correction, the drone is suspended from transmitting image data back to the head mounted display terminal through the base station until the shooting jitter correction is completed.

[0078] The beneficial effect of the above embodiment is that when the wearer is viewing the display interface of the head-mounted display terminal, the wearer's eye sight is tracked and identified to obtain the projection area range information of the wearer's viewing sight on the display interface, that is, the boundary information of the projection area range formed on the display interface when the wearer's eye sight changes direction, so as to accurately calibrate the viewing range of the wearer on the display interface. One or more image display windows are presented on the display interface. At this time, the projection area range information is compared with the area range information of all the image display windows currently existing in the display interface to determine the position information of the wearer's viewing line of sight corresponding to the coverage area of ​​the external environment. The head-mounted display terminal displays the image of the external environment collected by the drone in real time, that is, the image pictures of all image display windows on the display interface of the head-mounted display terminal are consistent with the external environment, that is, by determining the wearer's gaze coverage area of ​​the image display window, the area of ​​the external environment that the wearer is concerned about can be uniquely determined. At this time, based on the coverage area position information, the drone is instructed to align and zoom the coverage area of ​​the external environment, so that the drone can further obtain clearer and more detailed image pictures of the coverage area of ​​the external environment, thereby meeting the wearer's image viewing needs. In addition, when the drone collects external images during flight, the drone will inevitably experience shooting jitter. Therefore, the pixel contour of the external environment image transmitted back by the drone in real time is recognized to obtain the image jitter offset direction and offset of the external environment image, so as to perform shooting jitter correction on the drone; and during the execution of the shooting jitter correction, the drone is suspended from transmitting image data to the head-mounted display terminal through the base station until the shooting jitter correction is completed. In this way, the shooting quality of the drone's external environment image can be improved, and an effective display image data source can be provided for the head-mounted display terminal.

[0079] In another embodiment, the second motion tracking and identification of the wearer is performed to obtain the second motion feature information of the wearer, and based on the second motion feature information, the image region of interest of the wearer in the image display space is determined; and based on the position information of the image region of interest, the flight state of the drone is adjusted, including:

[0080] The wearer's hand is dynamically photographed to obtain a dynamic image of the wearer's hand; the dynamic image of the hand is recognized to obtain a motion trajectory of the wearer's fingertips in the external environment space; based on the transformation relationship between the external environment space and the image display space, the motion trajectory is mapped to the image display space, and the image screen area corresponding to the wearer's finger in the image display space is determined as the image area of ​​interest of the wearer in the image display space;

[0081] Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined; and then based on the position of the area of ​​interest and the real-time position of the drone, the flight path of the drone is adjusted so that the drone flies above the area of ​​interest.

[0082] The beneficial effect of the above embodiment is that when the wearer is viewing an image through the head-mounted display terminal, the wearer's hand is dynamically photographed to obtain a dynamic image of the wearer's hand, so that the interactive action initiated by the wearer to the head-mounted display terminal through his own hand action can be accurately identified, and the action trajectory of the wearer's fingertips in the external environment space (i.e., the real world space) can be further determined. In addition, based on the transformation relationship between the external environment space and the image display space, the action trajectory is mapped to the image display space, and the image screen area corresponding to the selected by the wearer's finger in the image display space is determined, so that the selected operation made by the wearer on the image in the process of viewing the image from the head-mounted display terminal can be accurately calibrated, thereby determining the image area of ​​interest of the wearer in the image display space. Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined, and combined with the real-time position of the drone, the flight path of the drone is adjusted to make the drone fly above the area of ​​interest. In this way, the drone can shoot the wearer's area of ​​interest in the external environment space at close range, thereby transmitting clearer image data to the head-mounted display terminal in real time, thereby improving the image display experience of the head-mounted display terminal.

[0083] In another embodiment, the head mounted display terminal is extended with an auxiliary support frame mounted on the shoulder of the wearer, and the auxiliary support frame is used to increase the rotation damping when the wearer turns his head to prevent the head from turning too fast and causing false triggering interference to the shooting work of the drone, which includes:

[0084] Step S1, using the following formula (1), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, control the shooting anti-shake correction of the drone,

[0085]

[0086] In the above formula (1), D represents the anti-shake control value of the drone shooting state; θ0 represents the preset micro-motion angle; Indicates the first head rotation angle detected within the adjacent preset time interval; Indicates the second head rotation angle detected within the adjacent preset time interval; || indicates obtaining the absolute value; & indicates the logical relationship; if[] indicates a judgment function. If the formula in the brackets is true, the function value of the judgment function is 1; if the formula in the brackets is not true, the function value of the judgment function is 0;

[0087] If D=1, the current camera shake correction of the drone will not be exited;

[0088] If D=0, then exit the current camera shake correction of the drone;

[0089] Step S2, using the following formula (2), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, determines whether to control the auxiliary support frame to trigger the rotation damping,

[0090]

[0091] In the above formula (2), C represents the trigger control value of the rotation damping; T0 represents the time length corresponding to the preset time interval; ω0 represents the minimum angular velocity for triggering the rotation damping;

[0092] If C=1, the auxiliary support frame is controlled to trigger rotation damping;

[0093] If C=0, the auxiliary support frame is not controlled to trigger the rotation damping;

[0094] Step S3, when the auxiliary support frame successfully triggers the rotation damping, the following formula (3) is used to control the magnitude of the rotation damping force applied by the auxiliary support frame according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals:

[0095]

[0096] In the above formula (3), F represents the magnitude of the rotational damping force applied by the auxiliary support frame; α represents the angle between the auxiliary support frame and the shoulder; R represents the preset standard head radius; M represents the preset standard head mass; and m0 represents the mass of the head-mounted display terminal.

[0097] The beneficial effect of the above embodiment is that, by using the above formula (1), according to the head rotation angle detected by the head mounted display terminal within two adjacent preset time intervals, the shooting anti-shake correction of the drone is controlled, thereby avoiding frequent movement of the drone caused by slight head movement, thereby ensuring the reliability of the system; then by using the above formula (2), according to the head rotation angle detected by the head mounted display terminal within two adjacent preset time intervals, it is determined whether to control the auxiliary support frame to trigger the rotation damping, thereby automatically and intelligently detecting the head rotation state and applying damping, thereby realizing the intelligence of the system; then by using the above formula (3), according to the head rotation angle detected by the head mounted display terminal within two adjacent preset time intervals, the magnitude of the rotation damping force applied by the auxiliary support frame is controlled, thereby ensuring that the damping effect can resist the increase in the rotation speed.

[0098] See also Figure 2 As shown, an embodiment of the present application provides a human-computer interaction system. The human-computer interaction system includes:

[0099] A communication connection module, used to establish a communication connection between the drone and the head-mounted display terminal based on the working status of the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal;

[0100] An image display mode adjustment module, configured to adjust the image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal;

[0101] A sight tracking and identification module, used to perform sight tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal;

[0102] The drone shooting and data return adjustment module is used to adjust the drone's shooting state of the external environment based on the viewing field information, and then adjust the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image returned by the drone in real time;

[0103] A motion tracking and identification module is used to perform a second motion tracking and identification on the wearer to obtain second motion feature information of the wearer, and determine an image region of interest of the wearer in the image display space based on the second motion feature information;

[0104] The UAV flight adjustment module is used to adjust the flight state of the UAV based on the position information of the image area of ​​interest.

[0105] The beneficial effects of the above embodiments are as follows: the human-computer interaction system establishes a communication connection between the drone and the head-mounted display terminal, transmits the external environment image collected by the drone back to the head-mounted display terminal, and adjusts the image display mode based on the first action initiated by the wearer of the head-mounted display terminal to provide a suitable image display scene for the wearer; it also tracks and identifies the wearer's line of sight to obtain the wearer's viewing field information in the image display space, thereby adjusting the drone's shooting state of the external environment and realizing adaptive shooting adjustment of the drone; it also adjusts the drone's shooting state and based on the picture feature information of the external environment image transmitted back in real time. The image data is transmitted back to the status to ensure the image display quality of the head-mounted display terminal and to meet the wearer's viewing needs for different areas of the external environment in real time; based on the wearer's second action feature information, the image area of ​​interest of the wearer in the image display space is determined to adjust the flight status of the drone, which connects the drone to the head-mounted display terminal for communication, and uses the head-mounted display terminal as an interaction intermediary between the wearer and the drone, so that the human-computer interaction initiated by the wearer to the head-mounted display terminal can directly control the drone, realize the synchronous human-computer interaction between the wearer, the head-mounted display terminal and the drone, and improve the scene applicability and reliability of the human-computer interaction.

[0106] In another embodiment, the communication connection module is used to establish a communication connection between the drone and the head-mounted display terminal based on the working status of the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal, including:

[0107] Identify the drone and the head-mounted display terminal to determine whether the drone and the head-mounted display terminal are both in a powered-on working state; if so, trigger the drone and the head-mounted display terminal to enter a base station search mode, so as to obtain the wireless communication signal status information of all base stations in the surrounding environment; based on the wireless communication signal status information, select a base station that matches the drone and the head-mounted display terminal, and build a communication channel between the drone and the head-mounted display terminal based on the wireless network address information of the selected base station, so as to transfer the external environment image collected by the drone to the selected base station and then transmit it back to the head-mounted display terminal;

[0108] The image display mode adjustment module is used to adjust the image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal, including:

[0109] Dynamic detection is performed on the head of a wearer of the head mounted display terminal to obtain head movement posture information and head movement amplitude information of the wearer; based on the head movement posture information and the head movement amplitude information, head movement trajectory information of the wearer is fitted; the head movement trajectory information is compared with a display control instruction form of the head mounted display terminal, a matching display control instruction is selected therefrom, and based on the matching display control instruction, a display interface of the head mounted display terminal is switched to a single image display window or a multiple image display window.

[0110] The beneficial effects of the above embodiments are that the drone and the head-mounted display terminal are two independent electronic devices. If the two electronic devices are directly connected wirelessly, the wireless communication performance of the drone and the head-mounted display terminal will be affected. During the movement of the drone or the head-mounted display terminal, the two cannot always be within the effective wireless communication range of each other, resulting in instability of the direct wireless connection between the drone and the head-mounted display terminal. In order to achieve a stable and continuous wireless communication connection between the drone and the head-mounted display terminal, a wireless communication base station is required as a wireless communication intermediary between the drone and the head-mounted display terminal. To this end, the drone and the head-mounted display terminal are identified to determine whether the drone and the head-mounted display terminal are both in a powered-on working state. When the drone and the head-mounted display terminal are both in a powered-on working state, the drone and the head-mounted display terminal will automatically perform a wireless base station search working mode to obtain the wireless communication signal status information of all base stations in the surrounding environment; wherein the wireless communication signal status information may be, but is not limited to, the wireless signal strength information of the base station, the effective coverage range information of the wireless signal, and the wireless communication bandwidth information, etc.; then based on the wireless communication signal status information of all base stations, the base station with the best wireless communication performance is identified, and based on the wireless network address information of the base station with the best wireless communication performance, a communication channel between the drone and the head-mounted display terminal is established, and the base station is used as a wireless communication intermediary to establish a stable and continuous wireless communication connection between the drone and the head-mounted display terminal, so as to avoid wireless communication interruption between the two, and ensure that the drone transfers the collected external environment image to the selected base station and then transmits it back to the head-mounted display terminal, so as to provide a stable and reliable image display for the wearer. There is also a dynamic detection of the head of the wearer of the head-mounted display terminal to obtain the head movement posture information and head movement amplitude information of the wearer, so that the head movement trajectory of the wearer can be used as the basis for the human-computer interaction between the wearer and the head-mounted display terminal, so that the wearer can achieve the corresponding human-computer interaction without directly performing physical contact operation on the head-mounted display terminal. Specifically, based on the head movement posture information and the head movement amplitude information, the head movement trajectory information of the wearer is fitted, so that the head movement of the wearer can be accurately calibrated. Then the head movement trajectory information is compared with the display control instruction form of the head-mounted display terminal, and the matching display control instruction is selected from it, so that the control action type initiated by the wearer's own head is accurately identified. And based on the matching display control instruction, the display interface of the head-mounted display terminal is switched to a single image display window or a multi-image display window, so that the display interface of the head-mounted display terminal can provide the wearer with a single image display window or a multi-image display window, and each image display window can independently display the external environment image collected by the drone, so that the wearer obtains a different image viewing experience.

[0111] In another embodiment, the eye tracking and recognition module is used to perform eye tracking and recognition on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal, including:

[0112] Tracking and identifying the wearer's sight line when the wearer is viewing the display interface, and obtaining the projection area range information of the wearer's sight line on the display interface as the viewing field information;

[0113] The drone shooting and data return adjustment module is used to adjust the drone's shooting state of the external environment based on the viewing field information, and then adjust the drone's shooting state and the drone's image data return state again based on the image feature information of the external environment image returned by the drone in real time, including:

[0114] The projection area range information is compared with the area range information of all image display windows currently existing in the display interface to determine the position information of the wearer's viewing line corresponding to the coverage area of ​​the external environment; based on the coverage area position information, the drone is instructed to aim and zoom the coverage area of ​​the external environment;

[0115] Performing image pixel contour recognition on the external environment image transmitted back by the drone in real time to obtain image jitter offset characteristic information of the external environment image; wherein the image jitter offset characteristic information includes the image jitter offset direction and offset amount;

[0116] Based on the image jitter offset characteristic information, the drone is subjected to shooting jitter correction; and during the execution of the shooting jitter correction, the drone is suspended from transmitting image data back to the head mounted display terminal through the base station until the shooting jitter correction is completed.

[0117] The beneficial effect of the above embodiment is that when the wearer is viewing the display interface of the head-mounted display terminal, the wearer's eye sight is tracked and identified to obtain the projection area range information of the wearer's viewing sight on the display interface, that is, the boundary information of the projection area range formed on the display interface when the wearer's eye sight changes direction, so as to accurately calibrate the viewing range of the wearer on the display interface. One or more image display windows are presented on the display interface. At this time, the projection area range information is compared with the area range information of all the image display windows currently existing in the display interface to determine the position information of the wearer's viewing line of sight corresponding to the coverage area of ​​the external environment. The head-mounted display terminal displays the image of the external environment collected by the drone in real time, that is, the image pictures of all image display windows on the display interface of the head-mounted display terminal are consistent with the external environment, that is, by determining the wearer's gaze coverage area of ​​the image display window, the area of ​​the external environment that the wearer is concerned about can be uniquely determined. At this time, based on the coverage area position information, the drone is instructed to align and zoom the coverage area of ​​the external environment, so that the drone can further obtain clearer and more detailed image pictures of the coverage area of ​​the external environment, thereby meeting the wearer's image viewing needs. In addition, when the drone collects external images during flight, the drone will inevitably experience shooting jitter. Therefore, the pixel contour of the external environment image transmitted back by the drone in real time is recognized to obtain the image jitter offset direction and offset of the external environment image, so as to perform shooting jitter correction on the drone; and during the execution of the shooting jitter correction, the drone is suspended from transmitting image data to the head-mounted display terminal through the base station until the shooting jitter correction is completed. In this way, the shooting quality of the drone's external environment image can be improved, and an effective display image data source can be provided for the head-mounted display terminal.

[0118] In another embodiment, the motion tracking and identification module is used to perform a second motion tracking and identification on the wearer to obtain second motion feature information of the wearer, and determine the image region of interest of the wearer in the image display space based on the second motion feature information, including:

[0119] The wearer's hand is dynamically photographed to obtain a dynamic image of the wearer's hand; the dynamic image of the hand is recognized to obtain a motion trajectory of the wearer's fingertips in the external environment space; based on the transformation relationship between the external environment space and the image display space, the motion trajectory is mapped to the image display space, and the image screen area corresponding to the wearer's finger in the image display space is determined as the image area of ​​interest of the wearer in the image display space;

[0120] The UAV flight adjustment module is used to adjust the flight state of the UAV based on the position information of the image area of ​​interest, including:

[0121] Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined; and then based on the position of the area of ​​interest and the real-time position of the drone, the flight path of the drone is adjusted so that the drone flies above the area of ​​interest.

[0122] The beneficial effect of the above embodiment is that when the wearer is viewing an image through the head-mounted display terminal, the wearer's hand is dynamically photographed to obtain a dynamic image of the wearer's hand, so that the interactive action initiated by the wearer to the head-mounted display terminal through his own hand action can be accurately identified, and the action trajectory of the wearer's fingertips in the external environment space (i.e., the real world space) can be further determined. In addition, based on the transformation relationship between the external environment space and the image display space, the action trajectory is mapped to the image display space, and the image screen area corresponding to the selected by the wearer's finger in the image display space is determined, so that the selected operation made by the wearer on the image in the process of viewing the image from the head-mounted display terminal can be accurately calibrated, thereby determining the image area of ​​interest of the wearer in the image display space. Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined, and combined with the real-time position of the drone, the flight path of the drone is adjusted to make the drone fly above the area of ​​interest. In this way, the drone can shoot the wearer's area of ​​interest in the external environment space at close range, thereby transmitting clearer image data to the head-mounted display terminal in real time, thereby improving the image display experience of the head-mounted display terminal.

[0123] Correspondingly, the present application also provides a human-computer interaction device, which includes a drone, a head-mounted display terminal and the aforementioned human-computer interaction system; the human-computer interaction system is used to implement the aforementioned human-computer interaction method. The human-computer interaction device is used to realize unified human-computer interaction between the drone, the head-mounted display terminal and the wearer. When the wearer wears the head-mounted display terminal, the human-computer interaction actions initiated by the wearer on the head-mounted display terminal can be automatically converted into human-computer interaction actions on the drone, so that the wearer only needs to interact with the head-mounted display terminal to control the head-mounted display terminal and the drone at the same time, eliminating the need to set up two different interaction relationships between the wearer and the head-mounted display terminal and between the wearer and the drone at the same time, reducing the complexity of the human-computer interaction between the wearer and the head-mounted display terminal and the drone. The effects of the corresponding human-computer interaction method and human-computer interaction system have been described above, and no repetitive description is made here.

[0124] In general, the human-computer interaction method, system and device establish a communication connection between the drone and the head-mounted display terminal, transmit the external environment image collected by the drone back to the head-mounted display terminal, and adjust the image display mode based on the first action initiated by the wearer of the head-mounted display terminal to provide the wearer with a suitable image display scene; the wearer's line of sight is tracked and identified to obtain the wearer's viewing field information in the image display space, so as to adjust the drone's shooting state of the external environment and realize adaptive shooting adjustment of the drone; the drone's shooting state and The image data is transmitted back to the status to ensure the image display quality of the head-mounted display terminal and to meet the wearer's viewing needs for different areas of the external environment in real time; based on the wearer's second action feature information, the image area of ​​interest of the wearer in the image display space is determined to adjust the flight status of the drone, which connects the drone to the head-mounted display terminal for communication, and uses the head-mounted display terminal as an interaction intermediary between the wearer and the drone, so that the human-computer interaction initiated by the wearer to the head-mounted display terminal can directly control the drone, realize the synchronous human-computer interaction between the wearer, the head-mounted display terminal and the drone, and improve the scene applicability and reliability of the human-computer interaction.

[0125] The above is only a specific implementation of the present invention, and any other improvements made based on the concept of the present invention are considered to be within the protection scope of the present invention.

Claims

1. A human-computer interaction method, characterized in that: include: Based on the working states of the drone and the head-mounted display terminal, a communication connection is established between the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal; based on the first action initiated by the wearer of the head-mounted display terminal, an image display mode of the head-mounted display terminal is adjusted; Performing line of sight tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal; Based on the viewing field information, adjusting the drone's shooting state of the external environment, and then adjusting the drone's shooting state and the drone's image data return state again based on the image feature information of the external environment image transmitted back by the drone in real time; Performing second motion tracking and identification on the wearer to obtain second motion feature information of the wearer, and determining an image region of interest of the wearer in the image display space based on the second motion feature information; and adjusting the flight state of the drone based on position information of the image region of interest; The head mounted display terminal is extended with an auxiliary support frame mounted on the shoulder of the wearer, and the auxiliary support frame is used to increase the rotation damping when the wearer turns his head to prevent the head from turning too fast and causing false triggering interference to the shooting work of the drone, which includes: Step S1, using the following formula (1), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, control the shooting anti-shake correction of the drone, In the above formula (1), D represents the anti-shake control value of the drone shooting state; θ0 represents the preset micro-motion angle; Indicates the first head rotation angle detected within the adjacent preset time interval; Indicates the second head rotation angle detected within the adjacent preset time interval; || indicates obtaining the absolute value; & indicates the logical relationship; if[] indicates a judgment function. If the formula in the brackets is true, the function value of the judgment function is 1; if the formula in the brackets is not true, the function value of the judgment function is 0; If D=1, the current anti-shake correction of the drone is not exited; If D=0, then the current anti-shake correction of the drone is exited; Step S2, using the following formula (2), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, determining whether to control the auxiliary support frame to trigger the rotation damping, In the above formula (2), C represents the trigger control value of the rotation damping; T0 represents the time length corresponding to the preset time interval; ω0 represents the minimum angular velocity for triggering the rotation damping; If C=1, the auxiliary support frame is controlled to trigger rotation damping; If C=0, the auxiliary support frame is not controlled to trigger the rotation damping; Step S3, when the auxiliary support frame successfully triggers the rotation damping, the following formula (3) is used to control the magnitude of the rotation damping force applied by the auxiliary support frame according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals: In the above formula (3), F represents the magnitude of the rotational damping force applied by the auxiliary support frame; α represents the angle between the auxiliary support frame and the shoulder; R represents the preset standard head radius; M represents the preset standard head mass; m0 represents the mass of the head mounted display terminal.

2. The human-computer interaction method according to claim 1, characterized in that: Based on the working status of the drone and the head mounted display terminal, a communication connection is established between the drone and the head mounted display terminal, so as to transmit the external environment image collected by the drone back to the head mounted display terminal; Adjusting an image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal includes: Identify the drone and the head-mounted display terminal to determine whether the drone and the head-mounted display terminal are both in a powered-on working state; if so, trigger the drone and the head-mounted display terminal to enter a base station search mode, so as to obtain the wireless communication signal status information of all base stations in the surrounding environment; based on the wireless communication signal status information, select a base station that matches the drone and the head-mounted display terminal, and build a communication channel between the drone and the head-mounted display terminal based on the wireless network address information of the selected base station, so as to transfer the external environment image collected by the drone to the selected base station and then transmit it back to the head-mounted display terminal; Dynamically detect the head of the wearer of the head-mounted display terminal to obtain head movement posture information and head movement amplitude information of the wearer; fit the head movement trajectory information of the wearer based on the head movement posture information and the head movement amplitude information; compare the head movement trajectory information with the display control instruction form of the head-mounted display terminal, select a matching display control instruction therefrom, and switch the display interface of the head-mounted display terminal to a single image display window or a multi-image display window based on the matching display control instruction.

3. The human-computer interaction method according to claim 2, characterized in that: Tracking and identifying the wearer's line of sight to obtain viewing field information of the wearer in the image display space of the head-mounted display terminal; adjusting the drone's shooting state of the external environment based on the viewing field information, and then adjusting the drone's shooting state or the drone's image data return state again based on the image feature information of the external environment image transmitted back by the drone in real time, including: Tracking and identifying the wearer's sight line when the wearer is viewing the display interface, and obtaining the projection area range information of the wearer's sight line on the display interface, which serves as the viewing field information; Compare the projection area range information with the area range information of all image display windows currently existing on the display interface, and determine the position information of the wearer's viewing line corresponding to the coverage area of ​​the external environment; then, based on the coverage area position information, instruct the drone to aim and zoom the coverage area of ​​the external environment; Performing picture pixel contour recognition on the external environment image transmitted back by the drone in real time to obtain picture jitter offset characteristic information of the external environment image; wherein the picture jitter offset characteristic information includes picture jitter offset direction and offset amount; Based on the image jitter offset characteristic information, performing shooting jitter correction on the drone; and During the execution of the shooting shake correction, the drone is suspended from transmitting image data back to the head mounted display terminal through the base station until the shooting shake correction is completed.

4. The human-computer interaction method according to claim 3, characterized in that: Performing second motion tracking and identification on the wearer to obtain second motion feature information of the wearer, and determining an image region of interest of the wearer in the image display space based on the second motion feature information; Then, based on the position information of the image region of interest, adjusting the flight state of the drone includes: Dynamically photograph the wearer's hand to obtain a dynamic image of the wearer's hand; recognize the dynamic image of the hand to obtain a motion trajectory of the wearer's fingertips in the external environment space; based on the transformation relationship between the external environment space and the image display space, map the motion trajectory to the image display space, determine the image screen area corresponding to the wearer's finger in the image display space, and use it as the image area of ​​interest of the wearer in the image display space; Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined; and then based on the position of the area of ​​interest and the real-time position of the drone, the flight path of the drone is adjusted so that the drone flies above the area of ​​interest.

5. A human-computer interaction system, characterized in that: include: A communication connection module, used to establish a communication connection between the drone and the head-mounted display terminal based on the working status of the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal; an image display mode adjustment module, configured to adjust the image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal; A sight tracking and identification module, used to perform sight tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal; A drone shooting and data return adjustment module is used to adjust the drone's shooting state of the external environment based on the viewing field information, and then adjust the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image returned by the drone in real time; a motion tracking and identifying module, configured to perform a second motion tracking and identifying operation on the wearer, obtain second motion feature information of the wearer, and determine an image region of interest of the wearer in the image display space based on the second motion feature information; A UAV flight adjustment module, used for adjusting the flight state of the UAV based on the position information of the image region of interest; The head mounted display terminal is extended with an auxiliary support frame mounted on the shoulder of the wearer, and the auxiliary support frame is used to increase the rotation damping when the wearer turns his head to prevent the head from turning too fast and causing false triggering interference to the shooting work of the drone, which includes: Step S1, using the following formula (1), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, control the shooting anti-shake correction of the drone, In the above formula (1), D represents the anti-shake control value of the drone shooting state; θ0 represents the preset micro-motion angle; Indicates the first head rotation angle detected within the adjacent preset time interval; Indicates the second head rotation angle detected within the adjacent preset time interval; || indicates obtaining the absolute value; & indicates the logical relationship; if[] indicates a judgment function. If the formula in the brackets is true, the function value of the judgment function is 1; if the formula in the brackets is not true, the function value of the judgment function is 0; If D=1, the current camera shake correction of the drone will not be exited; If D=0, then the current camera anti-shake correction of the drone is exited; Step S2, using the following formula (2), according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals, determining whether to control the auxiliary support frame to trigger the rotation damping, In the above formula (2), C represents the trigger control value of the rotation damping; T0 represents the time length corresponding to the preset time interval; ω0 represents the minimum angular velocity for triggering the rotation damping; If C=1, the auxiliary support frame is controlled to trigger rotation damping; If C=0, the auxiliary support frame is not controlled to trigger the rotation damping; Step S3, when the auxiliary support frame successfully triggers the rotation damping, the following formula (3) is used to control the magnitude of the rotation damping force applied by the auxiliary support frame according to the head rotation angle detected by the head mounted display terminal in two adjacent preset time intervals: In the above formula (3), F represents the magnitude of the rotational damping force applied by the auxiliary support frame; α represents the angle between the auxiliary support frame and the shoulder; R represents the preset standard head radius; M represents the preset standard head mass; m0 represents the mass of the head mounted display terminal.

6. The human-computer interaction system according to claim 5, characterized in that: The communication connection module is used to establish a communication connection between the drone and the head-mounted display terminal based on the working status of the drone and the head-mounted display terminal, so as to transmit the external environment image collected by the drone back to the head-mounted display terminal, including: Identify the drone and the head-mounted display terminal to determine whether the drone and the head-mounted display terminal are both in a powered-on working state; if so, trigger the drone and the head-mounted display terminal to enter a base station search mode, so as to obtain the wireless communication signal status information of all base stations in the surrounding environment; based on the wireless communication signal status information, select a base station that matches the drone and the head-mounted display terminal, and build a communication channel between the drone and the head-mounted display terminal based on the wireless network address information of the selected base station, so as to transfer the external environment image collected by the drone to the selected base station and then transmit it back to the head-mounted display terminal; The image display mode adjustment module is used to adjust the image display mode of the head mounted display terminal based on a first action initiated by a wearer of the head mounted display terminal, including: Dynamically detect the head of the wearer of the head-mounted display terminal to obtain head movement posture information and head movement amplitude information of the wearer; fit the head movement trajectory information of the wearer based on the head movement posture information and the head movement amplitude information; compare the head movement trajectory information with the display control instruction form of the head-mounted display terminal, select a matching display control instruction therefrom, and switch the display interface of the head-mounted display terminal to a single image display window or a multi-image display window based on the matching display control instruction.

7. The human-computer interaction system according to claim 6, characterized in that: The eye tracking and identification module is used to perform eye tracking and identification on the wearer to obtain viewing field information of the wearer in the image display space of the head mounted display terminal, including: Tracking and identifying the wearer's sight line when the wearer is viewing the display interface, and obtaining the projection area range information of the wearer's sight line on the display interface, which serves as the viewing field information; The drone shooting and data return adjustment module is used to adjust the drone's shooting state of the external environment based on the viewing field information, and then adjust the drone's shooting state and the drone's image data return state based on the image feature information of the external environment image returned by the drone in real time, including: Compare the projection area range information with the area range information of all image display windows currently existing on the display interface, and determine the position information of the wearer's viewing line corresponding to the coverage area of ​​the external environment; then, based on the coverage area position information, instruct the drone to aim and zoom the coverage area of ​​the external environment; Performing picture pixel contour recognition on the external environment image transmitted back by the drone in real time to obtain picture jitter offset characteristic information of the external environment image; wherein the picture jitter offset characteristic information includes picture jitter offset direction and offset amount; Based on the image jitter offset characteristic information, performing shooting jitter correction on the drone; and During the execution of the shooting shake correction, the drone is suspended from transmitting image data back to the head mounted display terminal through the base station until the shooting shake correction is completed.

8. The human-computer interaction system according to claim 7, characterized in that: The motion tracking and identification module is used to perform second motion tracking and identification on the wearer to obtain second motion feature information of the wearer, and determine the image region of interest of the wearer in the image display space based on the second motion feature information, including: Dynamically photograph the wearer's hand to obtain a dynamic image of the wearer's hand; recognize the dynamic image of the hand to obtain a motion trajectory of the wearer's fingertips in the external environment space; based on the transformation relationship between the external environment space and the image display space, map the motion trajectory to the image display space, determine the image screen area corresponding to the wearer's finger in the image display space, and use it as the image area of ​​interest of the wearer in the image display space; The UAV flight adjustment module is used to adjust the flight state of the UAV based on the position information of the image area of ​​interest, including: Based on the position information of the image area of ​​interest and the transformation relationship, the position of the wearer's area of ​​interest in the external environment space is determined; and then based on the position of the area of ​​interest and the real-time position of the drone, the flight path of the drone is adjusted so that the drone flies above the area of ​​interest.

9. A human-computer interaction device, characterized in that: include: A drone, a head mounted display terminal and a human-computer interaction system as described in any one of claims 5 to 8; the human-computer interaction system is used to implement the human-computer interaction method as described in any one of claims 1 to 4.

Images