Control method, head-mounted display device, control system, and storage medium

Abstract

A control method of a head-mounted display device (400) comprises: obtaining a reference yaw orientation (OA, OA1) of a head (S110); obtaining an actual yaw orientation (OB, OB1, OB2, OB3) of an imaging device (130) (S120); displaying an identifier (10, 11, 12, 20, 30, 40, 41, 42, 50, 60, 70, 80, 90, 102, 103, 104, 105) according to the reference yaw orientation (OA, OA1) and the actual yaw orientation (OB, OB1, OB2, OB3), the identifier (10, 11, 12, 20, 30, 40, 41, 42, 50, 60, 70, 80, 90, 102, 103, 104, 105) being used to indicate a deviation between the reference yaw orientation (OA, OA1) and / or the actual yaw orientation (OB, OB1, OB2, OB3) and the reference yaw orientation (OA, OA1) (S130); wherein the reference yaw orientation (OA, OA1) is a movement direction reference for a control terminal (200, 300) to control horizontal movement of a movable device. The application can prevent the user (101) from losing direction and affecting the control of the movable device. A head-mounted display device (400), a control system and a storage medium are also provided.

Description

Technical Field

[0001] This application relates to the field of control, and more particularly to a control method, a head-mounted display device, a control system, and a storage medium. Background Technology

[0002] Mobile devices (such as unmanned aerial vehicles, underwater / aquatic unmanned robots, and ground intelligent vehicles) can be equipped with imaging devices for capturing images. Users can control the mobile device through a control terminal (such as a remote control) so that the imaging device can capture different images. The images captured by the imaging device can be transmitted to the display device of the control terminal for the user's reference, thereby helping the user to further control the mobile device.

[0003] In traditional control modes for mobile devices, horizontal movement is controlled based on the yaw direction of the device's nose. This yaw direction is the same as or roughly the same as the yaw direction of the imaging device. Users can determine the yaw direction by observing the images captured by the imaging device, thus controlling the mobile device. However, with the emergence of new control modes for mobile devices, this movement direction reference may differ from the yaw direction. In this case, although users can view images captured by the imaging device on the mobile device using a head-mounted display, they cannot know the movement direction reference for horizontal movement control when using the control terminal, making accurate control of the mobile device impossible. Summary of the Invention

[0004] Based on this, this application provides a control method, a head-mounted display device, a control system, and a storage medium, aiming to solve the technical problem that when the movement direction reference of a mobile device during horizontal movement control may be different from the yaw direction of the device head, the user cannot know the horizontal movement reference of the mobile device when using the control terminal and thus cannot accurately control the mobile device.

[0005] In a first aspect, embodiments of this application provide a control method for a head-mounted display device, the head-mounted display device being used to display images captured by an imaging device carried by a mobile device, the mobile device including a body, the body including a head, the method comprising:

[0006] Obtain the reference yaw orientation of the nose;

[0007] Obtain the actual yaw orientation of the imaging device;

[0008] Display an indicator based on the reference yaw heading and the actual yaw heading, the indicator being used to indicate the deviation between the reference yaw heading and / or the actual yaw heading and the reference yaw heading;

[0009] The reference yaw orientation is the movement direction reference for the control terminal to control the horizontal movement of the mobile device.

[0010] Secondly, embodiments of this application provide a head-mounted display device, including:

[0011] A display device, at least for displaying images captured by an imaging device carried by a mobile device, the mobile device including a body, the body including a head;

[0012] One or more processors, working individually or collectively, are used to perform the following steps:

[0013] Obtain the reference yaw orientation of the nose;

[0014] Obtain the actual yaw orientation of the imaging device;

[0015] Display an indicator based on the reference yaw heading and the actual yaw heading, the indicator being used to indicate the deviation between the reference yaw heading and / or the actual yaw heading and the reference yaw heading;

[0016] The reference yaw orientation is the movement direction reference for the control terminal to control the horizontal movement of the mobile device.

[0017] Thirdly, embodiments of this application provide a control system, including:

[0018] A mobile device, the mobile device including a body, the body including a head;

[0019] The aforementioned head-mounted display device;

[0020] Control terminal;

[0021] The head-mounted display device and the control terminal are communicatively connected to the mobile device.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the above-described method.

[0023] This application provides a head-mounted display device and its control method, storage medium, and storage medium. The method includes: acquiring a reference yaw orientation of the head of a mobile device; acquiring the actual yaw orientation of an imaging device carried by the mobile device; displaying an identifier based on the reference yaw orientation and the actual yaw orientation, the identifier indicating the deviation between the reference yaw orientation and / or the actual yaw orientation and the reference yaw orientation; wherein, the reference yaw orientation is a movement direction reference for the horizontal movement control of the mobile device by a control terminal, so that when the movement direction reference of the mobile device during horizontal movement control may differ from the yaw orientation of the head, the user can at least know the reference yaw orientation of the mobile device through the displayed identifier, facilitating accurate operation of the mobile device by the user using the control terminal; when the identifier is used to indicate the deviation between the actual yaw orientation and the reference yaw orientation, it can also facilitate the user to control the actual yaw orientation of the imaging device.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the embodiments of this application. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a control method for a head-mounted display device provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the structure of the control system in one embodiment provided in this application;

[0028] Figure 3A This is a schematic diagram of the structure of an unmanned aerial vehicle in one embodiment provided in this application.

[0029] Figure 3B This is a schematic diagram of the gimbal structure of an unmanned aerial vehicle in one embodiment of this application.

[0030] Figure 4A This is a schematic diagram of the attitude of an unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided in an embodiment of this application;

[0031] Figure 4B This is a method provided in the embodiments of this application. Figure 4AThe diagram shows the reference yaw direction and the actual yaw direction displayed on the head-mounted display device;

[0032] Figure 4C This is a schematic diagram of the attitude, reference yaw direction, and actual yaw direction of another unmanned aerial vehicle provided in the embodiments of this application;

[0033] Figure 4D This is a method provided in the embodiments of this application. Figure 4C The diagram shows the reference yaw direction and the actual yaw direction displayed on the head-mounted display device;

[0034] Figure 4E This is a schematic diagram of the attitude of another unmanned aerial vehicle provided in the embodiments of this application, as well as its reference yaw direction and actual yaw direction;

[0035] Figure 4F This is a method provided in the embodiments of this application. Figure 4E The diagram shows the reference yaw direction and the actual yaw direction displayed on the head-mounted display device;

[0036] Figure 4G This is a schematic diagram of the attitude of another unmanned aerial vehicle provided in the embodiments of this application, as well as its reference yaw direction and actual yaw direction;

[0037] Figure 4H This is a method provided in the embodiments of this application. Figure 4G The diagram shows the reference yaw direction and the actual yaw direction displayed on the head-mounted display device;

[0038] Figure 5A This is a schematic diagram of the attitude of an unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided in an embodiment of this application;

[0039] Figure 5B This is a method provided in the embodiments of this application. Figure 5A The diagram shows a different representation of the reference yaw direction and the actual yaw direction on a head-mounted display.

[0040] Figure 5C This is a schematic diagram of the attitude, reference yaw direction, and actual yaw direction of another unmanned aerial vehicle provided in the embodiments of this application;

[0041] Figure 5D This is a method provided in the embodiments of this application. Figure 5C The diagram shows a different representation of the reference yaw direction and the actual yaw direction on a head-mounted display.

[0042] Figure 5EThis is a schematic diagram of the attitude of another unmanned aerial vehicle provided in the embodiments of this application, as well as its reference yaw direction and actual yaw direction;

[0043] Figure 5F This is a method provided in the embodiments of this application. Figure 5E The diagram shows a different representation of the reference yaw direction and the actual yaw direction on a head-mounted display.

[0044] Figure 6 These are schematic diagrams illustrating the display of various icons on a head-mounted display device in some embodiments provided in this application.

[0045] Figure 7 This is a method provided in the embodiments of this application. Figure 4C The diagram shows the reference yaw direction and the actual yaw direction displayed on a head-mounted display device in another form;

[0046] Figure 8 This is a method provided in the embodiments of this application. Figure 4E The diagram shows the reference yaw direction and the actual yaw direction displayed on a head-mounted display device in another form;

[0047] Figure 9 This is a method provided in the embodiments of this application. Figure 4E The diagram shows the reference yaw direction and the actual yaw direction displayed on a head-mounted display device in another form;

[0048] Figure 10 This is a schematic diagram of a scenario in which a control terminal controls the deflection of an unmanned aerial vehicle according to one embodiment of this application;

[0049] Figure 11 This is a schematic diagram of a scenario where a control terminal controls the deflection of an unmanned aerial vehicle in another embodiment provided in this application.

[0050] Figure 12 This is a schematic diagram of a scenario in which a control terminal controls the deflection of an unmanned aerial vehicle in another embodiment provided in this application.

[0051] Figure 13 This is a schematic diagram of the virtual environment of a head-mounted display device in one embodiment of the present application.

[0052] Figure 14 This is a schematic diagram of the identifier displayed in a top-down view according to one embodiment of the present application;

[0053] Figure 15A and Figure 15B This is a schematic diagram of a scenario for calibrating a reference yaw heading according to an embodiment of this application;

[0054] Figure 16 This is a schematic block diagram of a head-mounted display device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0057] Mobile devices, such as unmanned aerial vehicles, underwater / aquatic unmanned robots, and ground intelligent vehicles, can be equipped with imaging devices for capturing images. Users can control the mobile devices through a control terminal (such as a remote control), enabling the imaging devices to capture different images.

[0058] In traditional control modes for mobile devices, horizontal movement is controlled based on the yaw direction of the device's nose. This yaw direction is the same as or approximately the same as the yaw direction of the imaging device, allowing the user to determine the yaw direction by observing the image captured by the imaging device. However, with the emergence of new control modes for mobile devices, such as those that allow the device to respond to yaw control commands from at least two control terminals (a remote controller and a head-mounted display) to control the device's body yaw, thereby adjusting the nose yaw direction, a new control mode has emerged. By maintaining the movement direction reference unchanged when the head-mounted display controls the yaw rotation, and updating the reference when the remote controller controls the yaw rotation, it is possible to prevent the head-mounted display from altering the horizontal movement direction reference and interfering with the control terminals' operation of the mobile device. However, the reference for the direction of movement when controlling the horizontal movement of a mobile device may differ from the yaw direction of the nose. Although users can view the images captured by the imaging device on the mobile device using a head-mounted display, they cannot know the reference for the direction of movement when using the control terminal, which makes it impossible for users to accurately control the mobile device.

[0059] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0060] For ease of explanation, this application primarily uses unmanned aerial vehicles (UAVs) to illustrate the aforementioned principles of mobile devices. The movement actions within the mobile device correspond to the flight actions of the UAV; that is, in the specific application scenario of the UAV, the direction of movement corresponds to the direction of flight. The implementation principles for other mobile robots are similar and will not be elaborated upon further.

[0061] Please see Figure 1 , Figure 1 This is a flowchart illustrating a control method for a head-mounted display device according to an embodiment of this application. The method can be applied to head-mounted display devices, at least for processes such as displaying markings based on the reference yaw direction of the unmanned aerial vehicle's (UAV) nose and the actual yaw direction of the imaging device mounted on the UAV.

[0062] Optionally, unmanned aerial vehicles, also known as drones, may include rotary-wing drones, such as quadcopter drones, hexacopter drones, and octacopter drones, or fixed-wing drones, or combinations of rotary-wing and fixed-wing drones. This application does not specifically limit these aspects.

[0063] In some implementations, the head-mounted display device may include a virtual reality (VR) display device or a first-person view (FPV) display device; however, it is not limited to these. For example, the head-mounted display device may include flight control glasses, a smart helmet, or a smart headband device. The head-mounted display device may be, for example, a glasses-type display device or a helmet-type display device.

[0064] This application also provides a control system; please refer to [link / reference]. Figure 2 The control system includes an unmanned aerial vehicle (UAV) 100, a first control terminal 200, and a second control terminal 300. The UAV 100 can communicate with the first control terminal 200 and the second control terminal 300. Both the first control terminal 200 and the second control terminal 300 are used to control the UAV 100. The first control terminal 200 may include at least one of a remote controller, a smartphone, and a tablet computer. The second control terminal 300 may include the aforementioned head-mounted display device, and may also include at least one of a remote controller, a smartphone, and a tablet computer. It should be noted that... Figure 2This is merely one possible form of the control system structure provided in one embodiment of this application. Those skilled in the art can use multiple control terminals to control the unmanned aerial vehicle as needed, and at least one of the multiple control terminals may include the head-mounted display device in this embodiment of the application.

[0065] In some embodiments, the unmanned aerial vehicle 100 includes a fuselage 110, a power system 120, an imaging device 130, and a control device. Figure 2 (Not shown in the image). The fuselage 110 may include a nose. In some embodiments, the unmanned aerial vehicle 100 also includes an arm connected to the fuselage 110, the arm being used to mount the power system. In some embodiments, the power system 120 may be directly mounted on the fuselage 110.

[0066] The power system 120 provides flight propulsion for the unmanned aerial vehicle (UAV). The power system 120 may include a motor and a propeller mounted on and driven by the motor. The power system 120 can drive the fuselage 110 of the UAV 100 to rotate around one or more rotation axes. For example, these rotation axes may include a roll axis, a yaw axis, and a pitch axis. When the power system 120 drives the fuselage 110 to rotate around the yaw axis, the yaw direction of the fuselage's nose changes; that is, the yaw rotation of the fuselage 110 can be controlled by controlling the power system 120. It should be understood that the motor can be a DC motor or an AC motor. Furthermore, the motor can be a brushless motor or a brushed motor.

[0067] The imaging device 130 is directly mounted on or mounted on the body 110 via a gimbal for capturing images, which can be pictures and / or videos. Because the connection method between the imaging device 130 and the body 110 differs, the method for controlling the actual yaw direction of the imaging device also differs, leading to different steps in obtaining the actual yaw direction. The following will demonstrate how to obtain the actual yaw direction of the imaging device and display the corresponding indicator on the head-mounted display device, taking into account the different connection methods of the imaging device 130 and the body 110; these details will not be elaborated upon here.

[0068] In some embodiments, the first control terminal 200 may include an input device, wherein the input device can detect the user's control operations (pitch control operations and / or roll control operations) of the first control terminal. The first control terminal can generate control commands (pitch control commands and / or roll control commands) for the unmanned aerial vehicle (UAV) based on the user's control operations detected by the input device. The first control terminal can send the control commands to the UAV so that the UAV can perform flight level according to the control commands. It is understood that the first control terminal can also generate a first yaw control command based on the user's yaw control operations detected by the input device, and the first control terminal can send the first yaw control command to the UAV. Specifically, the input device may include at least one of buttons, joysticks, dials, and touch screens. For example, the input device is a joystick, which is mounted on the body of the remote controller. The remote controller senses the user's yaw control operation on the joystick and generates a corresponding first yaw control command, which is then sent to the UAV through the communication device of the first control terminal to control the UAV 100 to adjust the yaw rotation of the fuselage to adjust the yaw direction of the nose.

[0069] In some embodiments, the second control terminal 300 may include an input device, wherein the input device can detect control operations by the user of the second control terminal, and the second control terminal can generate control commands for the unmanned aerial vehicle (UAV) based on the user's control operations detected by the input device. In some embodiments, the second control terminal 300 is provided with a motion sensor, and a corresponding control command is generated based on the motion of the second control terminal 300 sensed by the motion sensor to control the UAV 100 to perform a corresponding operation. For example, the motion of the second control terminal 300 sensed by the motion sensor (e.g., yaw rotation) generates a second yaw control command to control the fuselage yaw rotation of the UAV 100 to adjust the yaw direction of the nose. The user of the first control terminal can be the same as the user of the second control terminal; in some cases, the user of the first control terminal can be different from the user of the second control terminal. Specifically, the input device of the second control terminal 300 can be the same as or different from the input device of the first control terminal 200.

[0070] The unmanned aerial vehicle (UAV) can respond to yaw control commands sent by the first control terminal 200 and the second control terminal 300 to control the yaw of the UAV 100's fuselage 110, thereby adjusting the yaw direction of the nose. Currently, the horizontal flight control of the UAV 100 is based on the yaw direction of the UAV 100's nose as the flight direction reference. In some embodiments, when both the first control terminal 200 and the second control terminal 300 can change the yaw direction of the UAV 100's nose, the horizontal flight control of the UAV by the first control terminal 200 will be interfered with by the second control terminal 300. This is because when the second control terminal 300 adjusts the yaw direction of the UAV 100's nose, it changes the flight direction reference of the first control terminal 200's horizontal flight control of the UAV 100, which will affect the continuous horizontal flight control of the UAV 100 by the first control terminal 200, resulting in a poor user experience. This application embodiment can solve the problem that the first control terminal 200 is interfered with by the second control terminal 300 when it receives the second yaw control command from the second control terminal 300, and control the fuselage 110 to yaw and keep the flight direction reference (which can be called the reference yaw orientation) of the unmanned aerial vehicle 100 unchanged.

[0071] In some embodiments, the second control terminal 300 can receive images captured by the imaging device transmitted by the unmanned aerial vehicle 100 and display them on a display device. The display device can be integrated into the second control terminal 300, or the display device can be set separately from the second control terminal 300 and communicate with the second control terminal 300. The communication connection can be a wired connection or a wireless connection, and there is no limitation on this.

[0072] For example, the second control terminal 300 is the aforementioned head-mounted display device. The display device of the second control terminal 300 can be integrated into the main body of the head-mounted display device, or it can be connected to the head-mounted display device via a data cable.

[0073] In some implementations, the first yaw control command from the first control terminal is used to control the fuselage yaw rotation of the unmanned aerial vehicle (UAV) and update the reference yaw orientation to obtain an updated reference yaw orientation. For example, the UAV can receive the first yaw control command generated and sent by the first control terminal. The UAV can control the fuselage yaw rotation according to the first yaw control command. When the fuselage yaws, the actual yaw orientation of the nose changes accordingly. Furthermore, the UAV updates the reference yaw orientation according to the first yaw control command to obtain an updated reference yaw orientation. That is, the UAV determines a new reference yaw orientation of the nose based on the first yaw control command and the reference yaw orientation, and updates the reference yaw orientation with the determined new reference yaw orientation of the nose. Here, the determined new reference yaw orientation of the nose is the updated reference yaw orientation of the nose.

[0074] In some embodiments, the reference yaw orientation is updated by the fuselage yaw rotation angle corresponding to the first yaw control command. The updated reference yaw orientation of the nose can be obtained by superimposing the fuselage yaw rotation angle on the reference yaw orientation of the nose. The fuselage yaw rotation angle corresponding to the first yaw control command can be based on the fuselage sensing data output by the UAV's sensors. In some embodiments, the sensors, as mentioned above, may include at least one of the following: an inertial measurement unit (IMU), a compass and gyroscope combination unit. The sensing data can be used to determine the rotation information of the UAV's fuselage in the yaw direction (e.g., at least one of the actual attitude of the fuselage, the angular velocity of the fuselage, and the angular acceleration of the fuselage).

[0075] Because the reference yaw direction can be updated, when users are unaware of the flight direction reference for horizontal flight control of the UAV, i.e., the reference yaw direction, they are prone to losing control of the UAV's flight direction reference, thus affecting their ability to control the UAV. For example, when users view images captured by the imaging device on the UAV through a head-mounted display, they are more likely to lose control of the UAV's flight direction reference. In other words, when users control the UAV to fly forward, the image they see may not be the same image captured by the imaging device during forward flight.

[0076] This application embodiment can display an indicator based on the reference yaw direction of the nose and the actual yaw direction of the imaging device. The indicator is used to indicate the deviation between the reference yaw direction and / or the actual yaw direction and the reference yaw direction. This allows the user to at least know the reference yaw direction of the UAV when the flight direction reference during horizontal flight control is different from the yaw direction of the nose. This facilitates the user's operation of the UAV using the control terminal, such as performing horizontal flight control. When the indicator is used to indicate the deviation between the actual yaw direction and the reference yaw direction, it can also facilitate the user's operation of the actual yaw direction of the imaging device.

[0077] For ease of explanation, the embodiments of this application are mainly described using a remote control as the first control terminal and a head-mounted display device as the second control terminal.

[0078] like Figure 1 As shown, the control method for the head-mounted display device according to an embodiment of this application includes steps S110 to S130. The head-mounted display device is used to display images captured by an imaging device carried by an unmanned aerial vehicle (UAV), the UAV including a fuselage, the fuselage including a nose.

[0079] S110. Obtain the reference yaw orientation of the nose.

[0080] The reference yaw orientation is the flight direction reference for the control terminal to control the horizontal flight of the unmanned aerial vehicle. The control terminal is the aforementioned first control terminal, such as a remote controller, preferably a motion-sensing remote controller.

[0081] The reference yaw heading is the reference for the horizontal flight direction of the unmanned aerial vehicle (UAV) controlled by the first control terminal. For example, the first control terminal can send horizontal control commands (including at least one of pitch control commands and roll control commands) to the UAV to control its horizontal flight. The UAV can receive the horizontal flight control commands sent by the first control terminal. When the UAV responds to the horizontal control to perform horizontal flight, it needs to use the reference yaw heading to determine its horizontal flight direction.

[0082] For example, the head-mounted display device can obtain the reference yaw direction of the UAV's nose from the UAV or a first control terminal. The reference yaw direction of the UAV's nose can be stored in a storage device, which can be the UAV's local storage device or a storage device located outside the UAV. For example, the storage device can be a storage device located in the first control terminal, and the UAV can obtain the reference yaw direction of the nose sent by the first control terminal. In some embodiments, the reference yaw direction of the nose can be the current reference yaw direction of the UAV. Optionally, the current reference yaw direction of the UAV can be the updated reference yaw direction based on a first yaw control command from the first control terminal. Detailed examples of whether the current reference yaw direction of the UAV is the current reference yaw direction or the updated reference yaw direction will be described later and will not be elaborated here. Regardless of whether the current reference yaw direction of the UAV is the current reference yaw direction or the updated reference yaw direction, it has no impact on the principle of displaying the mark on the head-mounted display device in the embodiments of this application. For ease of explanation, the following mainly uses the current reference yaw direction as the reference yaw direction as the example. The principle of the current reference yaw direction of the UAV being the updated reference yaw direction according to the first yaw control command of the first control terminal is similar, and will not be repeated here.

[0083] S120. Obtain the actual yaw orientation of the imaging device.

[0084] For example, the unmanned aerial vehicle adjusts the actual yaw orientation of the imaging device according to a first yaw control command sent by a first control terminal and / or a second yaw control command sent by a second control terminal.

[0085] For example, the yaw control command of the head-mounted display device, i.e., the second yaw control command, can be used to control the fuselage yaw rotation of the unmanned aerial vehicle (UAV) to adjust the actual yaw orientation of the imaging device that is fixedly connected to the fuselage; or the yaw control command of the head-mounted display device can control the gimbal yaw rotation to adjust the actual yaw orientation of the imaging device; or the yaw control command of the head-mounted display device can control both the fuselage yaw rotation and the gimbal yaw rotation of the UAV to adjust the actual yaw orientation of the imaging device. It can be understood that whether the imaging device is fixedly connected to the fuselage or movably connected through the gimbal, the actual yaw orientation of the imaging device is the same as or after adjustment as the actual yaw orientation of the fuselage (nose). Therefore, the actual yaw orientation of the fuselage can be used to represent the actual yaw orientation of the imaging device.

[0086] In some implementations, the yaw control commands of the head-mounted display device are used to control the unmanned aerial vehicle to adjust the actual yaw orientation of the imaging device and to keep the reference yaw orientation of the nose unchanged; in order to prevent the head-mounted display device from interfering with the horizontal flight control of the unmanned aerial vehicle by the control terminal such as the remote controller.

[0087] For example, the head-mounted display device detects the movement of the user's head (e.g., posture, angular velocity, or angular acceleration) to generate the yaw control command. In some embodiments, the head-mounted display device is equipped with a motion sensor, and the head-mounted display device generates the yaw control command based on the movement of the head-mounted display device sensed by the motion sensor.

[0088] In some embodiments, the head-mounted display device acquires images captured by an imaging device carried by an unmanned aerial vehicle (UAV), and displays the images captured by the imaging device via a display device. The user can adjust their head posture based on the images captured by the imaging device, so that the head-mounted display device detects the user's head movement and generates the yaw control command. For example, if the user is interested in a target on the left side of the image, they can turn their head to the left so that the head-mounted display device displays an image of the target of interest.

[0089] The imaging device 130 is directly mounted on or supported by a gimbal on the fuselage 110 for capturing images, which may be pictures and / or videos. In the yaw direction, the yaw rotation of the imaging device 130 is correlated with the yaw rotation of the fuselage 110. Further, the imaging device 130 yaws in response to the yaw rotation of the fuselage 110, or the fuselage 110 yaws in response to the yaw rotation of the imaging device 130.

[0090] In some embodiments, the head-mounted display device sends a second yaw control command to the unmanned aerial vehicle (UAV), and the UAV can control its fuselage yaw rotation according to the second yaw control command. The method of controlling the fuselage yaw rotation according to the second yaw control command can include any of the following:

[0091] One way, such as Figure 3A As shown, in the yaw direction, the imaging device 130 is fixedly connected to the fuselage 110. Specifically, when it is necessary to adjust the actual yaw direction of the imaging device 130, the yaw direction of the fuselage 110 can be adjusted, so that the imaging device 130 can rotate with the fuselage 110 to adjust the yaw direction in order to obtain the required yaw direction.

[0092] Another way, such as Figure 2 and Figure 3BAs shown, in the yaw direction, the imaging device 130 and the fuselage 110 are movably connected. The unmanned aerial vehicle (UAV) may include a gimbal 140, on which the imaging device 130 is mounted, and the gimbal 140 is connected to the fuselage 110. The gimbal 140 can control the yaw rotation of the imaging device 130 to adjust the actual yaw orientation of the imaging device 130. Specifically, the gimbal 140 may include a yaw motor 141, which is used to control the yaw rotation of the imaging device 130. In some embodiments, the UAV controls the fuselage 110 to yaw. The UAV can acquire rotation information of the fuselage 110's yaw rotation (e.g., yaw attitude, yaw angular velocity, or angular acceleration) and control the gimbal 140 to yaw according to the rotation information to control the yaw rotation of the imaging device 130. In this case, the imaging device 130 yaws along with the yaw rotation of the fuselage 110.

[0093] In another manner, as described above, the imaging device 130 and the fuselage 110 are movably connected in the yaw direction. The fuselage 110 yaws along with the yaw rotation of the imaging device 130. The UAV controls the gimbal to rotate according to the second yaw control command. The gimbal 140 controls the yaw rotation of the imaging device 130. The UAV can acquire rotation information of the yaw rotation of the imaging device (e.g., yaw attitude, yaw angular velocity, or angular acceleration) and control the yaw rotation of the fuselage 110 according to the rotation information (e.g., control the power system according to the rotation information to control the yaw rotation of the fuselage 110).

[0094] In some embodiments, the gimbal 140 can control the pitch rotation and / or roll rotation of the imaging device 130 to adjust the pitch orientation and / or roll rotation of the imaging device 130. Specifically, the gimbal 140 may include a pitch motor and / or a roll motor, which are used to control the pitch rotation and / or roll rotation of the imaging device 130.

[0095] In some implementations, the first control terminal sends a first yaw control command to the unmanned aerial vehicle (UAV), which can control its fuselage yaw rotation according to the first yaw control command. The manner in which the UAV controls its fuselage yaw rotation according to the first yaw control command can be the same as the manner in which the UAV controls its fuselage yaw rotation according to the second yaw control command. In some cases, the manner in which the UAV controls its fuselage yaw rotation according to the first yaw control command can be different from the manner in which the UAV controls its fuselage yaw rotation according to the second yaw control command.

[0096] S130. Display an indicator based on the reference yaw heading and the actual yaw heading, the indicator being used to indicate the deviation between the reference yaw heading and / or the actual yaw heading and the reference yaw heading.

[0097] The reference yaw direction is related to the flight direction reference of the UAV, and the actual yaw direction is related to the shooting direction of the imaging device in space, that is, the user's line of sight. By decoupling the flight direction reference of the UAV and the shooting direction of the imaging device, for example, the imaging device can shoot in more directions when the UAV is flying, such as shooting to the left or right, or shooting to the upper or lower side; more types of aerial shooting can be performed when the UAV is flying, such as flying around the target while the shooting direction of the imaging device is facing the center point of the target (circling shooting, or called panning), or flying backward or sideways while flying around the target.

[0098] By displaying indicators to indicate the reference yaw direction and / or the deviation between the actual yaw direction and the reference yaw direction, the user can at least determine the reference yaw direction of the UAV when the flight direction reference during horizontal flight control may differ from the yaw direction of the nose. This facilitates user control of the UAV using the control terminal (remote controller), such as for horizontal flight control. When the indicators are used to indicate the deviation between the actual yaw direction and the reference yaw direction, it also facilitates user control of the actual yaw direction of the imaging device. For example, it can prevent users from being unaware of the UAV's flight direction reference when viewing images captured by the imaging device. For instance, when a user controls the UAV to fly forward, the image seen may not always be in the direction of flight, easily leading to disorientation, such as after the user rotates their body left or right. By displaying these indicators, the user can assist in determining the UAV's flight direction reference and the imaging device's shooting direction, preventing disorientation that could affect the control of the UAV.

[0099] In other words, the embodiments of this application can adjust the attitude of the imaging device mounted on the UAV according to the attitude of the head-mounted display device while the UAV is flying at a predetermined speed and direction (such as adjusting the attitude of the UAV when the imaging device is fixedly connected to the UAV; or adjusting the attitude of the UAV or the gimbal when the imaging device is movably connected to the UAV via a gimbal) to adjust the shooting angle of the imaging device, while maintaining the predetermined speed and direction of flight, and sending the captured image to the head-mounted display device in real time, so that the head-mounted display device can simulate the scene of turning the head, and display an icon on the head-mounted display device (such as on the image sent by the imaging device displayed on the head-mounted display device or on the graphical user interface of the display device of the head-mounted display device) to prevent the user from losing their way after turning their head, making the user experience more realistic.

[0100] In some implementations, an indicator is displayed based on the reference yaw heading and the actual yaw heading. This indicator is used to indicate the reference yaw heading. (See also...) Figure 4A and 4B The identifier includes a first identifier 10. For example, the position of the first identifier in the image can be determined based on the reference yaw direction and the actual yaw direction; the first identifier can then be displayed in the image based on its position. Specifically, Figure 4A In this context, OA represents the reference yaw direction of the UAV's nose, and OB represents the actual yaw direction of the imaging device. Based on the reference yaw direction OA of the UAV's nose and the actual yaw direction OB of the imaging device, it is possible to determine... Figure 4B The position of the first identifier 10 in the image captured by the imaging device displayed on the head-mounted display device is related to the viewing angle of the image and the actual yaw direction OB of the imaging device. For example, the user's line of sight when observing the center of the image is the same as the actual yaw direction of the imaging device. Therefore, by using the position of the first identifier in the image and its position relative to the image center, the reference yaw direction of the UAV's nose can be indicated to the user. That is, the first identifier 10 can indicate the reference yaw direction in the space where the UAV is located.

[0101] For example, such as Figure 4B As shown, the method further includes displaying a third identifier 20 in the image, the third identifier 20 indicating the position of the image center. The user can determine the actual yaw orientation of the imaging device based on the third identifier 20, so that the user can know the current actual yaw orientation of the imaging device in real time, thereby controlling the unmanned aerial vehicle.

[0102] Optionally, please refer to Figure 4A and Figure 4B When the reference yaw direction OA coincides with the actual yaw direction OB, the horizontal position of the first identifier 10 in the image is the same as the horizontal position of the image center. The first identifier can be one or more of the following: a cross, a circle, an arc, a dot, a square, etc., as long as the shape serves an identifying function. It can be understood that the position of the first identifier 10 in the image is the position of its geometric center in the image. For example, when the center of the first identifier 10 in the image can be the same horizontally as the image center and / or vertically, it can indicate to the user that the current shooting direction of the imaging device is the same as the reference flight direction of the unmanned aerial vehicle; such as... Figure 4A and Figure 4BThe first identifier 10 is symmetrical about the left and right sides and about the top and bottom of the image center. This application embodiment mainly uses the example of the first identifier's horizontal position in the image being the same as the horizontal position of the image center as an example. It can be understood that the principle of the first identifier's vertical position in the image being the same as the vertical position of the image center is similar, and will not be elaborated here.

[0103] Optionally, determining the position of the first marker in the image based on the reference yaw direction and the actual yaw direction includes: determining the orientation deviation OB between the reference yaw direction OA and the actual yaw direction; determining the positional deviation between the first marker 10 and the image center based on the orientation deviation; and determining the position of the first marker 10 in the image based on the positional deviation between the first marker 10 and the image center. For example, Figure 4C In the image, the actual yaw direction OB1 is located at a preset angle to the left of the reference yaw direction OA. Therefore, it can be determined that the position of the first marker 10 in the image is located at a preset distance to the right of the center of the image. The first marker 10 can indicate the reference yaw direction OA of the nose, and the image center can indicate the actual yaw direction OB1 of the imaging device. By determining the positional deviation between the first marker and the image center through the orientation deviation, the first marker can indicate the reference yaw direction and / or the deviation between the actual yaw direction and the reference yaw direction.

[0104] In some implementations, the orientation deviation between OA and OB includes the deviation direction and / or the deviation angle, and the positional deviation between the first identifier 10 and the center position of the image includes the deviation direction and / or the deviation distance. Based on the deviation direction in the orientation deviation between OA and OB, the deviation direction of the first identifier relative to the center of the image can be determined, and based on the deviation angle in the orientation deviation between OA and OB, the deviation distance between the first identifier and the center of the image can be determined.

[0105] Optionally, the deviation azimuth between the reference yaw heading OA and the actual yaw heading OB is the same as the deviation azimuth between the first marker and the image center. Please continue to refer to... Figure 4C and Figure 4D When the reference yaw orientation OA is to the right of the actual yaw orientation OB1, the first identifier 10 is deflected to the right relative to the image center; when the reference yaw orientation OA is to the left of the actual yaw orientation OB1, the first identifier is deflected to the left relative to the image center. It can be understood that the orientation deviation between the reference yaw orientation OA and the actual yaw orientation OB can be determined based on the deviation of the first identifier relative to the image center, thereby helping users to control the unmanned aerial vehicle without getting lost.

[0106] Optionally, the deviation angle between the reference yaw orientation OA and the actual yaw orientation OB is positively correlated with the deviation distance of the first marker relative to the image center. For example, please refer to... Figure 4E and Figure 4F When the angle between the actual yaw direction OB and the reference yaw direction OA increases (i.e., the angle between OB2 and OA is greater than the angle between OB1 and OA), the deviation distance of the first marker (in this embodiment, the geometric center of the first marker) relative to the image center is greater (refer to...). Figure 4F and Figure 4D The deviation distance between the first identifier 10 and the center of the image can be used to determine the deviation angle between the reference yaw direction of the aircraft head and the actual yaw direction of the imaging device, and thus adjust the user's control method of the unmanned aerial vehicle.

[0107] For example, please refer to Figure 4F-4H When the deviation angle between the reference yaw direction and the actual yaw direction exceeds a preset deviation angle threshold, the first identifier 10 is located outside the image, or a portion of the first identifier 10 is located outside the image, for example, within the graphical user interface of the display device of a head-mounted display device, but outside the displayed image.

[0108] Please see Figure 4F When the deviation angle between the reference yaw heading and the actual yaw heading exceeds a first deviation angle threshold, a portion of the first identifier 10 is located outside the image; please refer to [link / reference]. Figure 4H When the deviation angle between the reference yaw heading OA and the actual yaw heading OB3 exceeds a second deviation angle threshold, the first identifier 10 is entirely outside the image; wherein the second deviation angle threshold is greater than the first deviation angle threshold. When at least a portion of the first identifier 10 is outside the image, the user can understand that the deviation angle between the reference yaw heading and the actual yaw heading is large.

[0109] Optionally, the preset deviation angle threshold can be determined based on the range of yaw rotation of the imaging device controlled by the gimbal. Optionally, after the gimbal's left and right yaw angle reaches the limit, the actual yaw orientation of the imaging device can be adjusted by controlling the fuselage yaw rotation and keeping the reference yaw orientation unchanged. The preset deviation angle threshold, such as the second deviation angle threshold, can exceed the range of yaw rotation of the imaging device controlled by the gimbal.

[0110] Optionally, when at least a portion of the first identifier is outside the image, the user can control the actual yaw direction of the imaging device to shift toward the reference yaw direction.

[0111] Optional, please refer to Figure 4D , 4F and Figure 4H When at least a portion of the first identifier is located outside the image, a third identifier 20 is displayed in the image, the third identifier 20 indicating the position of the image center. This helps the user understand the center point of the viewing angle of the image at this time.

[0112] Optional, please refer to Figure 4G and 4H When the deviation angle between the reference yaw heading OA and the actual yaw heading OB3 exceeds a preset deviation angle threshold, and the first identifier 10 or a portion thereof is located outside the image, a fourth identifier 30 may be displayed on the image. The fourth identifier 30 indicates the deviation azimuth between the first identifier 10 and the image. By displaying the fourth identifier 30 when the first identifier 10 is located outside the image, the user is prompted about the deviation azimuth between the reference yaw heading and the actual yaw heading.

[0113] In some implementations, the position of the fourth marker in the image can be determined based on the offset direction between the reference yaw direction and the actual yaw direction, and the fourth marker 30 can be displayed according to its position in the image. For example, the fourth marker 30 may be located at one side edge of the image, the side edge corresponding to the relative orientation of the first marker with respect to the center of the image. See also... Figure 4H The first identifier 10 is located outside the image to the right of the image center. The fourth identifier can be displayed on the right side of the image. The fourth identifier is an arrow pointing to the right, which can indicate to the user that the reference yaw direction is located to the right of the actual yaw direction.

[0114] In some implementations, such as Figure 5A and Figure 5B As shown, the first identifier 10 includes a first left-side identifier 11 and a first right-side identifier 12. In some embodiments, such as Figure 5B As shown, the first identifier 10 includes a first left identifier 11 and a first right identifier 12. It can be understood that the first identifier in this embodiment differs from... Figures 4A-4H Although the shapes of the first identifiers are different, their display principles are the same. They all use the first identifier 10 to help users determine the reference for the horizontal flight direction of the unmanned aerial vehicle.

[0115] In some implementations, the appearance of the first left-side identifier differs from that of the first right-side identifier, so that users can distinguish between the left-side and right-side identifiers; for example... Figure 5BAs shown, the first left-side identifier 11 is a left parenthesis, and the second left-side identifier 12 is a right parenthesis, but this is not a limitation. Optionally, the first left-side identifier and the first right-side identifier are symmetrical, so that the user can determine the deviation azimuth and / or deviation angle between the reference yaw orientation and the actual yaw orientation based on the deviation azimuth and deviation distance between the line of symmetry of the first left-side identifier and the center of the image, or based on the deviation distance of the first left-side identifier from the left edge of the image and / or the deviation distance of the first right-side identifier from the right edge of the image.

[0116] For example, such as Figure 5A As shown, when the actual yaw direction coincides with the reference yaw direction, the Figure 5B The first left-side identifier 11 and the first right-side identifier 12 are symmetrically located on both sides of the image center, and the image center is located on the line of symmetry between the first left-side identifier 11 and the first right-side identifier 12.

[0117] It can be understood that the overlap between the actual yaw direction and the reference yaw direction means that the actual yaw direction of the imaging device and the reference yaw direction of the nose are roughly coincident in orientation, with an orientation angle of approximately 0.

[0118] Optionally, when the actual yaw direction coincides with the reference yaw direction, the first left-side marker and the first right-side marker are located at the left and right edges of the image, respectively. This can reduce the impact of the first left-side marker and the first right-side marker on the user's view of the image.

[0119] For example, the location of the first identifier in the image includes the location of the selected target identifier in the image.

[0120] Optionally, determining the position of the first marker in the image based on the reference yaw heading OA and the actual yaw heading OB includes: determining the heading deviation between the reference yaw heading and the actual yaw heading, the heading deviation including the deviation azimuth and / or the deviation angle; selecting a target marker to be displayed from the first left marker and the first right marker based on the deviation azimuth in the heading deviation; and determining the position of the target marker in the image based on the deviation angle in the heading deviation. By selecting the target marker from the first left marker and the first right marker, and displaying the target marker, the user can easily determine at least the deviation azimuth between the reference yaw heading and the actual yaw heading based on the type of the displayed target marker.

[0121] Optional, such as Figure 5A and Figure 5BAs shown, when the actual yaw direction coincides with the reference yaw direction, both the first left-side identifier and the second right-side identifier are selected as the target identifier. When both the first left-side identifier and the second right-side identifier are selected as the target identifier, the user can determine that the orientation deviation between the actual yaw direction and the reference yaw direction is small, such as the actual yaw direction and the reference yaw direction roughly coinciding.

[0122] In some implementations, such as Figure 5C and 5D As shown, when the actual yaw heading OB1 is to the left of the reference yaw heading OA, the first left-side identifier is selected as the target identifier; when the actual yaw heading OB1 is to the right of the reference yaw heading OA, the first right-side identifier is selected as the target identifier. Figure 5C and 5D As shown, the first left-side identifier 11 is selected as the target identifier. When the first left-side identifier 11 is displayed but the first right-side identifier 12 is not displayed, the user can determine that the reference yaw heading is located to the left of the actual yaw heading.

[0123] For example, the position of the target marker in the image can be determined based on the deviation angle in the orientation deviation. In some embodiments, please refer to... Figure 5C-5F When the first left-side identifier is selected as the target identifier, the deviation distance between the position of the first left-side identifier in the image and the left edge of the image is positively correlated with the deviation angle between the actual yaw orientation and the reference yaw orientation. That is, because Figure 5C The deviation angle between the actual yaw heading OB1 and the reference yaw heading OA is less than Figure 5E The deviation angle between the actual yaw heading OB2 and the reference yaw heading OA, therefore Figure 5D The deviation distance between the position of the first left-side identifier 11 in the image and the left edge of the image is less than... Figure 5F The deviation distance between the position of the first left-side identifier 11 in the image and the left edge of the image. In some embodiments, when the first right-side identifier is selected as the target identifier, the deviation distance between the position of the first right-side identifier in the image and the right edge of the image is positively correlated with the deviation angle between the actual yaw direction and the reference yaw direction. This part is similar to the selection of the first left-side identifier as the target identifier, and will not be described again here.

[0124] like Figures 5A-5FAs shown, the greater the deviation distance between the first left-side marker 11 and the left edge of the image, that is, the closer the first left-side marker 11 is to the right edge of the image, the greater the deviation angle between the actual yaw direction and the reference yaw direction. This scheme helps users determine the current reference yaw direction and the actual yaw direction based on the first left-side marker or the first right-side marker selected as the target marker, which is beneficial for user control of the unmanned aerial vehicle.

[0125] It is understood that when the deviation angle between the reference yaw heading and the actual yaw heading exceeds a preset deviation angle threshold, the presentation of the first left-side marker and the first left-side marker will be different. Figure 4E-4H The presentation principle is similar, so I will not go into details here.

[0126] In some implementations, an indicator is displayed based on the reference yaw heading and the actual yaw heading, the indicator being used at least to indicate the deviation between the actual yaw heading and the reference yaw heading, such as... Figures 6 to 9 As shown, the identifier includes a second identifier 40, which in turn includes a second reference identifier 41 and a second active identifier 42. Since the second identifier 40 can be used to indicate the deviation between the actual yaw heading and the reference yaw heading, it is also introduced here for brevity. Figure 4A , 4C and Figure 4E The attitude of the unmanned aerial vehicle, among which, Figure 4A The attitude of the unmanned aerial vehicle shown is Figure 6 The diagram shown corresponds to the displayed logo on the head-mounted display device. Figure 4C The attitude of the unmanned aerial vehicle shown is Figure 7 The diagram shown corresponds to the displayed logo on the head-mounted display device. Figure 4E The attitude of the unmanned aerial vehicle shown is Figure 8 or Figure 9 The diagram shown corresponds to the logo displayed on the head-mounted display device.

[0127] In some implementations, such as Figure 6 and Figure 7 As shown, the second reference mark 41 is a scale with graduations, and the second movable mark 42 can be called a sliding weight; the position of the second movable mark 42 on the second reference mark 41, i.e., the scale, can indicate the deviation between the actual yaw heading OB and the reference yaw heading OA. However, this is not the only possibility. Figure 9 As shown, the deviation angle between the actual yaw heading and the reference yaw heading can be indicated by the deviation angle of the second active identifier 42 relative to the second reference identifier 41.

[0128] Optionally, the second reference identifier 41 indicates the reference yaw direction OA, and the second active identifier indicates the actual yaw direction OB; or the second active identifier indicates the reference yaw direction OA, and the second reference identifier indicates the actual yaw direction OB. This application embodiment mainly uses the example of the second reference identifier 41 indicating the reference yaw direction OA and the second active identifier 42 indicating the actual yaw direction OB for illustration.

[0129] For example, the step of displaying an indicator based on the reference yaw heading and the actual yaw heading includes: displaying the second reference indicator; and displaying the second activity indicator based on the deviation between the reference yaw heading and the actual yaw heading. For example, based on... Figure 4A The reference yaw heading OA mentioned above displays a second reference identifier 41, that is, the second reference identifier 41 is used as the deviation reference between the reference yaw heading and the actual yaw heading, according to... Figure 4A The deviation between the reference yaw heading OA and the actual yaw heading OB is used to determine the positional relationship between the second active identifier 42 and the second reference identifier 41, so that the user can determine the deviation between the reference yaw heading OA and the actual yaw heading OB based on the positional relationship between the second active identifier 42 and the second reference identifier 41.

[0130] For example, displaying the second activity identifier based on the deviation between the reference yaw heading and the actual yaw heading includes: determining the heading deviation between the actual yaw heading and the reference yaw heading; determining the deviation between the second activity identifier and the second reference identifier based on the heading deviation between the actual yaw heading and the reference yaw heading; and displaying the second activity identifier based on the deviation between the second activity identifier and the second reference identifier.

[0131] Optionally, the orientation deviation between the actual yaw orientation and the reference yaw orientation includes the deviation bearing and / or deviation angle, and the deviation between the second active identifier and the second reference identifier includes the deviation bearing and deviation angle or deviation distance.

[0132] In some embodiments, the deviation azimuth in the heading deviation is used to determine the deviation azimuth between the second active identifier and the second reference identifier. For example, when the second reference identifier 41 indicates the reference yaw heading and the second active identifier 42 indicates the actual yaw heading, the deviation azimuth between the actual yaw heading and the reference yaw heading is the same as the deviation azimuth of the second active identifier 42 relative to the second reference identifier 41. Figure 4C , Figure 7As shown, when the actual yaw direction OB1 is to the left of the reference yaw direction OA, the second active identifier 42 is to the left of the second reference identifier 41; when the actual yaw direction OB1 is to the right of the reference yaw direction, the second active identifier 42 is to the right of the second reference identifier 41.

[0133] In some implementations, please refer to 4C, 4E, Figure 7 and Figure 8 The deviation angle in the orientation deviation is used to determine the deviation distance between the second active marker 42 and the second reference marker 41. For example, the second reference marker 41 includes a scale, and the second active marker 42 includes a rider. The deviation angle between the actual yaw orientation OB and the reference yaw orientation OA is positively correlated with the deviation distance from the rider to the center of the scale. The deviation angle between the actual yaw orientation OB and the reference yaw orientation OA is positively correlated with the deviation distance of the second active marker relative to the second reference marker; for example, Figure 4C The deviation angle between the actual yaw heading OB1 and the reference yaw heading OA is less than Figure 4E The deviation angle between the actual yaw heading OB2 and the reference yaw heading OA, as described in the text, is then... Figure 7 The deviation distance between the second activity identifier 42 and the second reference identifier 41 is less than Figure 8 The deviation distance between the second active marker 42 and the second reference marker 41. When the second active marker 42 is located at the leftmost scale of the second reference marker 41, it can be determined that the deviation angle between the reference yaw direction and the actual yaw direction has reached a preset deviation angle threshold.

[0134] In some implementations, please refer to Figures 6 to 8 The second reference mark 41 is provided with a scale, and one side of the second reference mark 41 coincides with that of the second active mark 42, with the second active mark 42 pointing to the scale of the second reference mark 41. Optionally, the center scale on the second reference mark 41 is used to indicate the reference yaw direction, and the scale pointed to by the second active mark 42 is used to indicate the actual yaw direction; or the center scale on the second reference mark is used to indicate the actual yaw direction, and the scale pointed to by the second active mark is used to indicate the reference yaw direction.

[0135] In some implementations, please refer to Figure 9The deviation angle in the orientation deviation is used to determine the deviation angle between the second active marker 42 and the second reference marker 41. For example, the deviation angle in the orientation deviation is indicated by the included angle between the second reference marker 42 and the second active marker 41; for example, the deviation angle between the actual yaw orientation and the reference yaw orientation is positively correlated with the deviation angle of the second active marker 42 relative to the second reference marker 41. The principle of this embodiment is similar to the embodiment where the second reference marker 41 is a scale with graduations and the second active marker 42 can be called a sliding weight, and will not be described again here.

[0136] Optionally, when the actual yaw heading OB coincides with the reference yaw heading OA, the second active identifier and the second reference identifier coincide. For example... Figure 4A When the actual yaw heading OA is the same as the reference yaw heading OB, the positions of the second active identifier and the second reference identifier are the same. In some embodiments, such as Figure 6 and 7 As shown, the second activity identifier 42 and the second reference identifier 41 are at the same horizontal position in the image; or please refer to Figure 9 The second activity identifier 42 and the second reference identifier 41 can also coincide, that is, the included angle is 0.

[0137] It should be noted that you should refer to [link / reference]. Figure 8 and Figure 9 The second reference mark 41 and the second activity mark 42 can be displayed at any position in the image, or at the lower edge of the image, and centered; of course, it is not limited to this, for example, it can also be displayed in an area outside the image in the display interface.

[0138] It is understood that the first identifier 10 and the second identifier 40 mentioned above can appear simultaneously, or one of them can be selected to identify the reference yaw heading and / or the deviation between the reference yaw heading and the actual yaw heading, as long as it can serve as the implementation indication provided by the embodiments of this application.

[0139] In some embodiments, step S130, which displays an identifier based on the reference yaw direction and the actual yaw direction, includes: displaying an identifier based on the updated reference yaw direction and the actual yaw direction of the imaging device. The reference yaw direction can be updated, for example, based on a first yaw control command from a first control terminal; or the identifier can be displayed based on the current reference yaw direction and the actual yaw direction of the imaging device. The reference yaw direction can, for example, be the current reference yaw direction.

[0140] Taking the actual yaw direction of the imaging device being consistent with the actual yaw direction of the aircraft nose as an example, such as Figure 10As shown, the reference yaw direction of the UAV's nose, OA, is consistent with the actual yaw direction OB of the imaging device. When the UAV receives a first yaw control command from the first control terminal, it controls the UAV's fuselage to yaw counterclockwise by an angle C1 in the yaw direction, adjusting the actual yaw direction of the imaging device to OB1. Furthermore, the reference yaw direction OA of the nose is updated according to the first yaw control command to obtain an updated reference yaw direction OA1. Further, the UAV obtains the fuselage yaw rotation angle C1 corresponding to the first yaw control command and updates the reference yaw direction according to the fuselage yaw rotation angle C1. The updated reference yaw direction OA1 can be obtained by superimposing the reference yaw direction of the nose with the fuselage yaw rotation angle. The indicator can be displayed according to the updated reference yaw direction OA1 and the actual yaw direction OB1 of the imaging device. The specific display method is similar to the principle described above and will not be repeated here.

[0141] like Figure 11 As shown, the reference yaw direction of the UAV's nose is OA, which is consistent with the actual yaw direction OB of the imaging device. When the UAV receives a second yaw control command generated and sent by the head-mounted display device, it controls the UAV's fuselage to yaw counterclockwise by an angle C1 in the yaw direction, and the actual yaw direction of the imaging device is adjusted to OB1. The reference yaw direction of the nose is not updated and remains unchanged at OA, that is, the first yaw control command does not change the reference yaw direction of the nose. The aforementioned indicator can be displayed according to the current reference yaw direction OA and the actual yaw direction OB1 of the imaging device. The specific display method is similar to the principle described above and will not be repeated here.

[0142] like Figure 12 As shown, the reference yaw direction of the UAV's nose is OA, and the actual yaw direction of the UAV's imaging device is OB. When the UAV receives a first yaw control command generated and sent by the first control terminal, it controls the UAV's fuselage to yaw counterclockwise by an angle C1 in the yaw direction. The actual yaw direction of the imaging device is adjusted to OB1, and the reference yaw direction OA of the nose is updated according to the first yaw control command to obtain the updated reference yaw direction OA1. The indicator can be displayed based on the updated reference yaw direction OA1 and the actual yaw direction OB1 of the imaging device. When the UAV receives a second yaw control command generated and sent by the head-mounted display device, it controls the UAV's fuselage to yaw counterclockwise by an angle C2 in the yaw direction. The actual yaw direction of the imaging device is adjusted to OB2, and the reference yaw direction of the nose is not updated, maintaining OA1 unchanged. The indicator can be displayed based on the current reference yaw direction OA1 and the actual yaw direction OB2 of the imaging device. The specific display method is similar to the principle described above and will not be repeated here.

[0143] In some implementations, such as Figure 6 As shown, the method further includes: displaying a fifth identifier 50 in the image, the fifth identifier 50 indicating the flight direction of the unmanned aerial vehicle (UAV). The flight direction of the UAV can be its velocity direction. For example, the position of the fifth identifier 50 in the image is determined based on the flight direction of the UAV, and the fifth identifier is displayed based on its position in the image. In some embodiments, a first control terminal can send a horizontal control command to the UAV to control its horizontal flight. When the UAV responds to the horizontal control to perform horizontal flight, it determines its horizontal flight direction using a reference yaw heading.

[0144] Optional, such as Figure 6 As shown, the method further includes: displaying a sixth identifier 60 on the fifth identifier 50, the sixth identifier 60 being used to indicate the attitude of the control terminal; wherein, the attitude of the control terminal is used to determine the control commands sent by the control terminal to the unmanned aerial vehicle. Figure 6 As shown, the fifth identifier 50 is a circle, and the sixth identifier 60 is a diagonal line on both sides of the circle.

[0145] In some implementations, the posture of the sixth identifier on the fifth identifier is determined based on the posture of the control terminal, and the sixth identifier is displayed based on the posture of the sixth identifier on the fifth identifier.

[0146] In some embodiments, the control terminal, such as the first control terminal, is equipped with a motion sensor. This motion sensor senses the motion of the first control terminal, such as its yaw attitude and / or pitch attitude, and generates corresponding control commands to control the unmanned aerial vehicle (UAV) to determine its horizontal flight direction based on a reference yaw orientation. The attitude of the control terminal is related to the control commands sent to the UAV. The greater the left and right tilt angle of the control terminal, the greater the angular velocity of the UAV's left and right flight. For example, a larger yaw angle of the control terminal results in a greater yaw angular velocity of the UAV, and a greater tilt of the sixth identifier on the fifth identifier. By displaying the sixth identifier 60 on the fifth identifier 50, the user can be informed of the attitude of the control terminal and how the UAV's flight direction will be adjusted in the future.

[0147] Optional, please refer to Figure 6The method further includes: predicting the flight trajectory of the unmanned aerial vehicle (UAV) based on its flight direction; and displaying a seventh identifier 70 in the image based on the predicted flight trajectory, the seventh identifier 70 indicating the predicted flight trajectory of the UAV. The flight trajectory may include, for example, the UAV's flight path over a preset time period, which can assist the user in determining the UAV's flight direction, allowing the user to control the UAV based on the predicted trajectory.

[0148] Optional, please refer to Figure 6 The method further includes displaying an eighth identifier 80 in the image, the eighth identifier 80 indicating the horizon or sea level of the environment corresponding to the image. This can help the user determine the horizontal attitude of the unmanned aerial vehicle at this time.

[0149] Optional, please refer to Figure 6 The method further includes displaying a ninth identifier 90 on the image, the ninth identifier 90 indicating the actual yaw orientation of the imaging device. In some embodiments, the ninth identifier 90 may indicate the yaw direction of the actual yaw orientation of the imaging device in space; helping the user understand the current shooting direction of the imaging device in the world coordinate system, such as... Figure 6 As shown, the imaging device is pointing north (N) at this time. Optionally, the ninth marker can be a compass-shaped marker, but it is not limited to this.

[0150] In some implementations, please refer to Figure 13 and Figure 14 , Figure 13 In one embodiment, a virtual environment can be experienced by a user using a head-mounted display device, wherein a first identifier 10 (such as including a first left identifier 11 and a first right identifier 12) is displayed directly in front of the user 101. The first identifier 10 is used to indicate the deviation between the reference yaw direction of the nose and / or the actual yaw direction of the imaging device and the reference yaw direction.

[0151] For example, the head-mounted display device is equipped with a motion sensor, and corresponding control commands are generated based on the motion of the head-mounted display device sensed by the motion sensor, such as controlling and adjusting the actual yaw orientation of the imaging device; this can make the actual yaw orientation of the imaging device follow the yaw orientation of the user's head, making the user experience more realistic and giving them a feeling of being in free flight.

[0152] Please see Figure 13 When user 101 is looking at the camera, a first identifier 10 and / or a second identifier 40 are displayed to indicate the deviation between the reference yaw direction of the nose and / or the actual yaw direction of the imaging device and the reference yaw direction. See Figures 4 to 4 above. Figure 9The explanation is omitted here.

[0153] In some embodiments, the gimbal can control the pitch rotation of the imaging device to adjust the pitch orientation of the imaging device. In some embodiments, the gimbal can control the roll rotation of the imaging device to adjust the roll orientation of the imaging device.

[0154] Please see Figure 13 and Figure 14 When user 101 looks down, the head-mounted display device controls the imaging device to look down via a gimbal; and the head-mounted display device can display at least one of the following on the image captured by the imaging device: a tenth mark 102, an eleventh mark 103, a twelfth mark 104, and a thirteenth mark 105. The tenth mark 102 is used to indicate a circular scale in the user's space, allowing the user to determine their orientation. The eleventh mark 103 is used to indicate the deviation between the actual yaw orientation of the imaging device and the reference yaw orientation, for example, by indicating the deviation numerically. In some embodiments, the value of the eleventh mark 103 is adjusted accordingly when the yaw attitude of the head-mounted display device changes, causing a change in the actual yaw orientation of the imaging device. The twelfth indicator 104 is used to indicate the reference yaw direction of the camera head and can be called a camera head indicator; for example, when the head-mounted display device yaws to the left, the twelfth indicator 104 yaws to the right. Exemplarily, the function and display position determination process of the twelfth indicator 104 are similar to those of the first indicator 10, and can be referred to the aforementioned description of the first indicator. The main difference is that the first indicator 10 is displayed when the user's eye level or pitch posture is within a small range, while the twelfth indicator 104 is displayed when the user's downward tilt posture exceeds a certain degree. By displaying the twelfth indicator 104 and / or the first indicator 10 according to the user's head pitch posture, it is convenient for the user to understand the deviation between the actual yaw direction of the imaging device and the reference yaw direction. The thirteenth indicator 105 is used to indicate the center point of the user's space, such as the user's location.

[0155] In some embodiments, the method further includes: acquiring a first correction command; and in response to the first correction command, adjusting the reference yaw orientation to the actual yaw orientation of the imaging device; adjusting the position of a first identifier in the identifier in the image to the center position of the image or adjusting the position of a second reference identifier in the identifier to the position of a second active identifier in the identifier; and / or, acquiring a second correction command; and in response to the second correction command, adjusting the actual yaw orientation of the imaging device to the reference yaw orientation of the nose; adjusting the center position of the image to the position of the first identifier in the image or adjusting the second active identifier to the position of the second reference identifier.

[0156] During the flight of an unmanned aerial vehicle (UAV), the reference yaw direction of the nose and the actual yaw direction of the imaging device are inconsistent. Users may want the actual yaw direction of the imaging device to align with the reference yaw direction of the nose, such as during a reset operation. The UAV can receive a correction command to adjust the actual yaw direction of the imaging device to match the reference yaw direction of the nose. This correction command may include a first correction command and / or a second correction command. When the UAV receives the first correction command, it can adjust the reference yaw direction of the nose to match the actual yaw direction of the imaging device. When the UAV receives the second correction command, it can adjust the actual yaw direction of the imaging device to match the reference yaw direction.

[0157] like Figure 15A and Figure 15B As shown, there is an angle between the reference yaw direction OA of the nose and the actual yaw direction OB of the imaging device. Figure 15A As shown, in response to the first correction command, the unmanned aerial vehicle adjusts the reference yaw orientation OA to the actual yaw orientation OB of the imaging device; as Figure 15B As shown, in response to the second correction command, the unmanned aerial vehicle adjusts the actual yaw direction OB of the imaging device to the reference yaw direction OA. Specifically, when adjusting the reference yaw direction OA to the actual yaw direction OB of the imaging device, the reference yaw direction OA can be updated, and the identifiers can be displayed based on the updated reference yaw direction OA and the actual yaw direction OB, such as updating the first identifier and the second identifier. Alternatively, when adjusting the actual yaw direction OB of the imaging device to the reference yaw direction OA, the first identifier and the second identifier can be updated based on the reference yaw direction OA and the actual yaw direction OB of the imaging device at each time. The update is, for example, a real-time update.

[0158] By adjusting the position of the first marker in the image to the center of the image or adjusting the second reference marker to the position of the second active marker in the marker when adjusting the reference yaw direction to the actual yaw direction of the imaging device; or by adjusting the center of the image to the position of the first marker in the image or adjusting the second active marker to the position of the second reference marker when adjusting the actual yaw direction of the imaging device to the reference yaw direction of the nose; the markers can be reset so that the markers accurately indicate the reference yaw direction and / or the deviation between the actual yaw direction and the reference yaw direction.

[0159] The correction command may be generated by a first control terminal or a second control terminal. Further, the correction command may be generated by the first control terminal detecting a user's correction operation through an interactive device. In some cases, the correction command may be generated by the second control terminal detecting a user's correction operation through an interactive device. For example, the first correction command is a command sent by the first control terminal to adjust the reference yaw direction of the unmanned aerial vehicle (UAV), and the second correction command is a command sent by the second control terminal to adjust the reference yaw direction of the UAV. The user can manipulate the first control terminal to output the first correction command, and manipulate the second control terminal to output the second correction command. For example, the first and / or second control terminals may be equipped with a reset button or other triggering devices, such as a microphone. When the user turns too far to the left or right, or wants to turn and move, the reference yaw direction and the actual yaw direction of the imaging device may not coincide, resulting in abnormal control. By triggering the reset button or other triggering devices, the reference yaw direction and the actual yaw direction of the imaging device can be reset to zero, making the actual yaw direction of the user's imaging device coincide with the reference yaw direction, thus enabling normal flight control.

[0160] The control method for a head-mounted display device provided in this application includes: acquiring a reference yaw orientation of the nose of an unmanned aerial vehicle (UAV); acquiring the actual yaw orientation of an imaging device carried by the UAV; and displaying an identifier based on the reference yaw orientation and the actual yaw orientation, wherein the identifier is used to indicate the deviation between the reference yaw orientation and / or the actual yaw orientation and the reference yaw orientation; wherein the reference yaw orientation is a flight direction reference for the control terminal to control the horizontal flight of the UAV, so that when the flight direction reference of the UAV during horizontal flight control is different from the yaw orientation of the imaging device, the user can at least know the reference yaw orientation of the UAV through the displayed identifier, which facilitates the user to accurately control the UAV using the control terminal, such as to perform horizontal flight control of the UAV; when the identifier is used to indicate the deviation between the actual yaw orientation and the reference yaw orientation, it can also facilitate the user to control the actual yaw orientation of the imaging device.

[0161] Please refer to the above embodiments. Figure 16 , Figure 16 This is a schematic block diagram of a head-mounted display device 400 provided in an embodiment of this application.

[0162] The head-mounted display device 400 includes a display unit 410, which is used at least to display images captured by an imaging device carried by an unmanned aerial vehicle.

[0163] The head-mounted display device 400 also includes one or more processors 401, which operate individually or collectively to perform the steps of the aforementioned control method for the head-mounted display device.

[0164] For example, the head-mounted display device 400 also includes a memory 402. The processor 401 and the memory 402 are connected via a bus 403, such as an I2C (Inter-integrated Circuit) bus.

[0165] Specifically, the processor 401 can be a microcontroller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP), etc.

[0166] Specifically, the memory 402 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0167] The processor 401 is used to run a computer program stored in the memory 402, and implements the steps of the aforementioned control method for the head-mounted display device when executing the computer program.

[0168] For example, the processor 401 is configured to run a computer program stored in the memory 402, and when executing the computer program, perform the following steps:

[0169] Obtain the reference yaw orientation of the nose;

[0170] Obtain the actual yaw orientation of the imaging device;

[0171] Display an indicator based on the reference yaw heading and the actual yaw heading, the indicator being used to indicate the deviation between the reference yaw heading and / or the actual yaw heading and the reference yaw heading;

[0172] The reference yaw orientation is the flight direction reference for the control terminal to control the horizontal flight of the unmanned aerial vehicle.

[0173] The specific principles and implementation methods of the head-mounted display device provided in this application embodiment are similar to the control methods of the head-mounted display device in the foregoing embodiments, and will not be repeated here.

[0174] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps of the control method for the head-mounted display device provided in the above embodiments.

[0175] The computer-readable storage medium can be an internal storage unit of the head-mounted display device described in any of the foregoing embodiments, such as the hard disk or memory of the head-mounted display device. The computer-readable storage medium can also be an external storage device of the head-mounted display device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the head-mounted display device.

[0176] It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0177] It should also be understood that the term “and / or” as used in this application and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0178] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method for a head-mounted display device, characterized in that, The head-mounted display device is used to display images captured by an imaging device carried by a mobile device, the mobile device including a body, the body including a head, and the method comprising: Obtain the reference yaw orientation of the nose; Obtain the actual yaw orientation of the imaging device; A label is displayed based on the reference yaw heading and the actual yaw heading, the label being used to indicate the deviation between the reference yaw heading and / or the actual yaw heading and the reference yaw heading; The reference yaw orientation is the movement direction reference for the control terminal to control the horizontal movement of the mobile device.

2. The control method according to claim 1, characterized in that, The identifier includes a first identifier, and the display of the identifier based on the reference yaw heading and the actual yaw heading includes: The position of the first marker in the image is determined based on the reference yaw heading and the actual yaw heading; The first identifier is displayed in the image according to its position in the image.

3. The control method according to claim 2, characterized in that, When the reference yaw direction coincides with the actual yaw direction, the horizontal position of the first marker in the image is the same as the horizontal position of the image center.

4. The control method according to claim 2, characterized in that, Determining the position of the first marker in the image based on the reference yaw heading and the actual yaw heading includes: Determine the orientation deviation between the reference yaw orientation and the actual yaw orientation; The positional deviation between the first identifier and the image center of the image is determined based on the orientation deviation; The position of the first identifier in the image is determined based on the positional deviation between the first identifier and the center of the image.

5. The control method according to claim 4, characterized in that, The orientation deviation includes the deviation direction and / or the deviation angle, and the position deviation includes the deviation direction and / or the deviation distance. The deviation direction in the orientation deviation is used to determine the deviation direction of the first identifier relative to the center of the image, and the deviation angle in the orientation deviation is used to determine the deviation distance between the first identifier and the center of the image.

6. The control method according to claim 5, characterized in that, The deviation between the reference yaw heading and the actual yaw heading is the same as the deviation between the first marker and the image center.

7. The control method according to claim 5, characterized in that, The deviation angle between the reference yaw orientation and the actual yaw orientation is positively correlated with the deviation distance of the first marker relative to the center of the image.

8. The control method according to claim 5, characterized in that, When the deviation angle between the reference yaw heading and the actual yaw heading exceeds a preset deviation angle threshold, the first identifier is located outside the image; as well as A fourth identifier is displayed on the image, the fourth identifier being used to indicate the deviation of the first identifier from the orientation of the image.

9. The control method according to any one of claims 1-8, characterized in that, The method further includes: A third identifier is displayed in the image, which is used to indicate the location of the image center.

10. The control method according to any one of claims 2-8, characterized in that, The first identifier includes a first left-side identifier and a first right-side identifier.

11. The control method according to claim 10, characterized in that, When the actual yaw direction coincides with the reference yaw direction, the first left-side marker and the first right-side marker are symmetrically located on both sides of the image center of the image.

12. The control method according to claim 11, characterized in that, When the actual yaw orientation coincides with the reference yaw orientation, the first left-side marker and the first right-side marker are located at the left and right edges of the image, respectively.

13. The control method according to claim 10, characterized in that, The appearance of the first left-side sign is different from that of the first right-side sign.

14. The control method according to claim 10, characterized in that, The position of the first identifier in the image includes the position of the selected target identifier in the image; determining the position of the first identifier in the image based on the reference yaw direction and the actual yaw direction includes: Determine the heading deviation between the reference yaw heading and the actual yaw heading, wherein the heading deviation includes the deviation bearing and / or the deviation angle; Based on the deviation direction in the orientation deviation, select the target identifier to be displayed from the first left-side identifier and the first right-side identifier; The position of the target identifier in the image is determined based on the deviation angle in the orientation deviation.

15. The control method according to claim 14, characterized in that, The step of selecting the target identifier to be displayed from the first left-side identifier and the first right-side identifier based on the deviation direction in the orientation deviation includes: When the reference yaw heading is to the left of the actual yaw heading, the first left-side identifier is selected as the target identifier; When the reference yaw heading is to the right of the actual yaw heading, the first right-side identifier is selected as the target identifier.

16. The control method according to claim 14, characterized in that, Determining the position of the target marker in the image based on the deviation angle in the orientation deviation includes: When the first left-side identifier is selected as the target identifier, the deviation distance between the position of the first left-side identifier in the image and the left edge of the image is positively correlated with the deviation angle between the actual yaw orientation and the reference yaw orientation; When the first right-side marker is selected as the target marker, the deviation distance between the position of the first right-side marker in the image and the right edge of the image is positively correlated with the deviation angle between the actual yaw direction and the reference yaw direction.

17. The control method according to claim 14, characterized in that, When the actual yaw heading coincides with the reference yaw heading, both the first left-side identifier and the second right-side identifier are selected as the target identifier.

18. The control method according to any one of claims 1-8, characterized in that, The identifier includes a second identifier, which includes a second reference identifier and a second active identifier; the display of the identifier based on the reference yaw heading and the actual yaw heading includes: Display the second reference identifier; The second activity indicator is displayed based on the deviation between the reference yaw heading and the actual yaw heading.

19. The control method according to claim 18, characterized in that, The step of displaying the second activity indicator based on the deviation between the reference yaw heading and the actual yaw heading includes: Determine the orientation deviation between the actual yaw orientation and the reference yaw orientation; The deviation between the second active marker and the second reference marker is determined based on the orientation deviation between the actual yaw orientation and the reference yaw orientation. The second activity identifier is displayed based on the deviation between the second activity identifier and the second reference identifier; Wherein, the second reference identifier indicates the reference yaw direction, and the second activity identifier indicates the actual yaw direction; or the second activity identifier indicates the reference yaw direction, and the second reference identifier indicates the actual yaw direction.

20. The control method according to claim 19, characterized in that, The heading deviation between the actual yaw heading and the reference yaw heading includes the deviation bearing and / or the deviation angle; the deviation between the second active marker and the second reference marker includes the deviation bearing and the deviation angle or deviation distance. Wherein, the deviation orientation in the orientation deviation is used to determine the deviation orientation between the second active identifier and the second reference identifier, and the deviation angle in the orientation deviation is used to determine the deviation angle or deviation distance between the second active identifier and the second reference identifier.

21. The control method according to claim 20, characterized in that, When the second reference marker indicates the reference yaw heading and the second active marker indicates the actual yaw heading, the deviation azimuth between the actual yaw heading and the reference yaw heading is the same as the deviation azimuth of the second active marker relative to the second reference marker.

22. The control method according to claim 20, characterized in that, The deviation angle between the actual yaw heading and the reference yaw heading is positively correlated with the deviation angle or deviation distance of the second active marker relative to the second reference marker.

23. The control method according to claim 19, characterized in that, When the actual yaw heading coincides with the reference yaw heading, the second active identifier and the second reference identifier coincide.

24. The control method according to any one of claims 1-8, characterized in that, The method further includes: A fifth identifier is displayed in the image, which is used to indicate the direction of movement of the mobile device.

25. The control method according to claim 24, characterized in that, The method further includes: A sixth identifier is displayed on the fifth identifier, the sixth identifier being used to indicate the attitude of the control terminal; The attitude of the control terminal is used to determine the control commands sent by the control terminal to the mobile device.

26. The control method according to any one of claims 1-8, characterized in that, The method further includes: Predict the movement trajectory of the mobile device based on its direction of movement; A seventh identifier is displayed in the image based on the predicted movement trajectory of the mobile device, the seventh identifier being used to indicate the predicted movement trajectory of the mobile device.

27. The control method according to any one of claims 1-8, characterized in that, The method further includes: An eighth identifier is displayed in the image, which is used to indicate the horizon or sea level of the environment corresponding to the image.

28. The control method according to any one of claims 1-8, characterized in that, The method further includes: A ninth identifier is displayed on the image, which indicates the actual yaw heading.

29. The control method according to claim 19, characterized in that, The method further includes: Obtain a first correction command; and in response to the first correction command, adjust the reference yaw orientation to the actual yaw orientation of the imaging device; adjust the position of the first marker in the marker in the image to the center position of the image or adjust the position of the second reference marker in the marker to the position of the second active marker in the marker; and / or, Obtain a second correction command; and in response to the second correction command, adjust the actual yaw direction of the imaging device to the reference yaw direction of the nose; adjust the center position of the image to the position of the first marker in the image or adjust the second active marker to the position of the second reference marker.

30. A head-mounted display device, characterized in that, include: A display device, at least for displaying images captured by an imaging device carried by a mobile device, the mobile device including a body, the body including a head; One or more processors, working individually or collectively, are used to perform the following steps: Obtain the reference yaw orientation of the nose; Obtain the actual yaw orientation of the imaging device; A label is displayed based on the reference yaw heading and the actual yaw heading, the label being used to indicate the deviation between the reference yaw heading and / or the actual yaw heading and the reference yaw heading; The reference yaw orientation is the movement direction reference for the control terminal to control the horizontal movement of the mobile device.

31. The head-mounted display device according to claim 30, characterized in that, The identifier includes a first identifier, and when the processor executes the step of displaying the identifier based on the reference yaw heading and the actual yaw heading, it is used for: The position of the first marker in the image is determined based on the reference yaw heading and the actual yaw heading; The first identifier is displayed in the image according to its position in the image.

32. The head-mounted display device according to claim 31, characterized in that, When the reference yaw direction coincides with the actual yaw direction, the horizontal position of the first marker in the image is the same as the horizontal position of the image center.

33. The head-mounted display device according to claim 31, characterized in that, When the processor executes the step of determining the position of the first marker in the image based on the reference yaw direction and the actual yaw direction, it is used to: Determine the orientation deviation between the reference yaw orientation and the actual yaw orientation; The positional deviation between the first identifier and the image center of the image is determined based on the orientation deviation; The position of the first identifier in the image is determined based on the positional deviation between the first identifier and the center of the image.

34. The head-mounted display device according to claim 33, characterized in that, The orientation deviation includes the deviation direction and / or the deviation angle, and the position deviation includes the deviation direction and / or the deviation distance. The deviation direction in the orientation deviation is used to determine the deviation direction of the first identifier relative to the center of the image, and the deviation angle in the orientation deviation is used to determine the deviation distance between the first identifier and the center of the image.

35. The head-mounted display device according to claim 34, characterized in that, The deviation between the reference yaw heading and the actual yaw heading is the same as the deviation between the first marker and the image center.

36. The head-mounted display device according to claim 34, characterized in that, The deviation angle between the reference yaw orientation and the actual yaw orientation is positively correlated with the deviation distance of the first marker relative to the center of the image.

37. The head-mounted display device according to claim 34, characterized in that, When the deviation angle between the reference yaw heading and the actual yaw heading exceeds a preset deviation angle threshold, the first identifier is located outside the image; as well as A fourth identifier is displayed on the image, the fourth identifier being used to indicate the deviation of the first identifier from the orientation of the image.

38. The head-mounted display device according to any one of claims 30-37, characterized in that, The processor is also used to perform: A third identifier is displayed in the image, which is used to indicate the location of the image center.

39. The head-mounted display device according to any one of claims 31-37, characterized in that, The first identifier includes a first left-side identifier and a first right-side identifier.

40. The head-mounted display device according to claim 39, characterized in that, When the actual yaw direction coincides with the reference yaw direction, the first left-side marker and the first right-side marker are symmetrically located on both sides of the image center of the image.

41. The head-mounted display device according to claim 40, characterized in that, When the actual yaw orientation coincides with the reference yaw orientation, the first left-side marker and the first right-side marker are located at the left and right edges of the image, respectively.

42. The head-mounted display device according to claim 39, characterized in that, The appearance of the first left-side sign is different from that of the first right-side sign.

43. The head-mounted display device according to claim 39, characterized in that, The position of the first identifier in the image includes the position of the selected target identifier in the image; when the processor executes the step of determining the position of the first identifier in the image based on the reference yaw direction and the actual yaw direction, it is used to: Determine the heading deviation between the reference yaw heading and the actual yaw heading, wherein the heading deviation includes the deviation bearing and / or the deviation angle; Based on the deviation direction in the orientation deviation, select the target identifier to be displayed from the first left-side identifier and the first right-side identifier; The position of the target identifier in the image is determined based on the deviation angle in the orientation deviation.

44. The head-mounted display device according to claim 43, characterized in that, When the processor executes the step of selecting the target identifier to be displayed from the first left-side identifier and the first right-side identifier based on the deviation azimuth in the orientation deviation, it is used to: When the reference yaw heading is to the left of the actual yaw heading, the first left-side identifier is selected as the target identifier; When the reference yaw heading is to the right of the actual yaw heading, the first right-side identifier is selected as the target identifier.

45. The head-mounted display device according to claim 43, characterized in that, When the processor executes the step of determining the position of the target identifier in the image based on the deviation angle in the orientation deviation, it is used to: When the first left-side identifier is selected as the target identifier, the deviation distance between the position of the first left-side identifier in the image and the left edge of the image is positively correlated with the deviation angle between the actual yaw orientation and the reference yaw orientation; When the first right-side marker is selected as the target marker, the deviation distance between the position of the first right-side marker in the image and the right edge of the image is positively correlated with the deviation angle between the actual yaw direction and the reference yaw direction.

46. ​​The head-mounted display device according to claim 43, characterized in that, When the actual yaw heading coincides with the reference yaw heading, both the first left-side identifier and the second right-side identifier are selected as the target identifier.

47. The head-mounted display device according to any one of claims 30-37, characterized in that, The processor executes the identifier including a second identifier, the second identifier including a second reference identifier and a second active identifier; when displaying the identifier according to the reference yaw heading and the actual yaw heading, it is used for: Display the second reference identifier; The second activity indicator is displayed based on the deviation between the reference yaw heading and the actual yaw heading.

48. The head-mounted display device according to claim 47, characterized in that, When the processor executes the step of displaying the second activity indicator based on the deviation between the reference yaw heading and the actual yaw heading, it is used for: Determine the orientation deviation between the actual yaw orientation and the reference yaw orientation; The deviation between the second active marker and the second reference marker is determined based on the orientation deviation between the actual yaw orientation and the reference yaw orientation. The second activity identifier is displayed based on the deviation between the second activity identifier and the second reference identifier; Wherein, the second reference identifier indicates the reference yaw heading, and the second activity identifier indicates the actual yaw heading; Alternatively, the second activity identifier indicates the reference yaw heading, and the second reference identifier indicates the actual yaw heading.

49. The head-mounted display device according to claim 48, characterized in that, The heading deviation between the actual yaw heading and the reference yaw heading includes the deviation bearing and / or the deviation angle; the deviation between the second active marker and the second reference marker includes the deviation bearing and the deviation angle or deviation distance. Wherein, the deviation orientation in the orientation deviation is used to determine the deviation orientation between the second active identifier and the second reference identifier, and the deviation angle in the orientation deviation is used to determine the deviation angle or deviation distance between the second active identifier and the second reference identifier.

50. The head-mounted display device according to claim 49, characterized in that, When the second reference marker indicates the reference yaw heading and the second active marker indicates the actual yaw heading, the deviation azimuth between the actual yaw heading and the reference yaw heading is the same as the deviation azimuth of the second active marker relative to the second reference marker.

51. The head-mounted display device according to claim 49, characterized in that, The deviation angle between the actual yaw heading and the reference yaw heading is positively correlated with the deviation angle or deviation distance of the second active marker relative to the second reference marker.

52. The head-mounted display device according to claim 48, characterized in that, When the actual yaw heading coincides with the reference yaw heading, the second active identifier and the second reference identifier coincide.

53. The head-mounted display device according to any one of claims 30-37, characterized in that, The processor is also used to perform: A fifth identifier is displayed in the image, which is used to indicate the direction of movement of the mobile device.

54. The head-mounted display device according to claim 53, characterized in that, The processor is also used to perform: A sixth identifier is displayed on the fifth identifier, the sixth identifier being used to indicate the attitude of the control terminal; The attitude of the control terminal is used to determine the control commands sent by the control terminal to the mobile device.

55. The head-mounted display device according to any one of claims 30-37, characterized in that, The processor is also used to perform: Predict the movement trajectory of the mobile device based on its direction of movement; A seventh identifier is displayed in the image based on the predicted movement trajectory of the mobile device, the seventh identifier being used to indicate the predicted movement trajectory of the mobile device.

56. The head-mounted display device according to any one of claims 30-37, characterized in that, The processor is also used to perform: An eighth identifier is displayed in the image, which is used to indicate the horizon or sea level of the environment corresponding to the image.

57. The head-mounted display device according to any one of claims 30-37, characterized in that, The processor is also used to perform: A ninth identifier is displayed on the image, which indicates the actual yaw heading.

58. The head-mounted display device according to claim 48, characterized in that, The processor is also used to perform: Obtain a first correction command; and in response to the first correction command, adjust the reference yaw orientation to the actual yaw orientation of the imaging device; adjust the position of the first marker in the marker in the image to the center position of the image or adjust the position of the second reference marker in the marker to the position of the second active marker in the marker; and / or, Obtain a second correction command; and in response to the second correction command, adjust the actual yaw direction of the imaging device to the reference yaw direction of the nose; adjust the center position of the image to the position of the first marker in the image or adjust the second active marker to the position of the second reference marker.

59. A control system, characterized in that, include: A mobile device, the mobile device including a body, the body including a head; The head-mounted display device as described in any one of claims 30-58; Control terminal; The head-mounted display device and the control terminal are communicatively connected to the mobile device.

60. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement: The steps of the control method for the head-mounted display device as described in any one of claims 1-29.

Images