Aerial imaging control method, device and system and storage medium

By monitoring the output image from the image source and adjusting the angle of the optical reflector, real-time compensation for aerial imaging is achieved based on the observer's eye position, solving the problem of image asynchrony in aerial imaging technology and improving the user interaction experience.

CN119045210APending Publication Date: 2024-11-29SAIC GENERAL MOTORS +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411350424.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-26
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In existing aerial imaging technologies, differences in image generation controllers and imaging principles lead to asynchronous images and poor user interaction.

Method used

When monitoring the output image from the image source, the position of the aerial image and the observer's eye position are determined according to the preset sampling period. The angle of the optical reflective plate is adjusted using the observation elevation angle and observation deviation angle to achieve image compensation and ensure that the position of the aerial image is suitable for the observer to view.

Benefits of technology

The system collects the observer's eye position in real time and adjusts the position of the aerial image to meet the user's interactive needs, avoids image asynchrony, and improves the user's interactive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045210A_ABST
    Figure CN119045210A_ABST
Patent Text Reader

Abstract

The invention discloses an aerial imaging control method, device and system and a storage medium, and the method comprises the steps: determining the position of aerial imaging according to a preset sampling period when monitoring that an image source outputs an image, and determining the position of an eye point of an observer according to the preset sampling period; determining an observation elevation angle and an observation deviation angle according to the position of the aerial imaging in the current period and the position of the eye point of the observer; adjusting the angle of an optical reflection flat plate through the observation elevation angle and the observation deviation angle so as to compensate the aerial imaging position; and when the position of the aerial imaging in the current sampling period is adjusted to the position suitable for the observer to watch, determining that the adjustment of the position of the aerial imaging is finished. By adopting the scheme provided by the invention, the picture synchronization effect can be improved, and the user interaction experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to an aerial imaging control method, device, system and storage medium. Background Art

[0002] With the continuous development of science and technology, the number of display carriers and display media of entertainment systems continues to increase. In particular, the development of aerial imaging technology can realize the integration of multiple display media. Through the synchronization and superposition of multiple screens, it provides strong technical support for human-computer interactive experience, immersive viewing and other functions.

[0003] At present, due to differences in image generation controllers and imaging principles, controlled imaging technology is prone to cause image asynchrony from the user's perspective, resulting in a poor interactive experience.

[0004] Therefore, how to provide an aerial imaging control method to improve the effect of picture synchronization and enhance the user interaction experience has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides an aerial imaging control method, device, system and storage medium to improve the effect of picture synchronization and enhance the user interaction experience.

[0006] The present application provides an aerial imaging control method, comprising:

[0007] When the image source outputs an image, the position of the aerial image is determined according to a preset sampling period, and the position of the observer's eye point is determined according to the preset sampling period;

[0008] Determine the observation altitude angle and the observation deviation angle according to the position of the aerial image in the current cycle and the position of the observer's eye point;

[0009] Adjusting the angle of the optical reflective plate by the observation altitude angle and the observation deviation angle to compensate for the position of the aerial imaging;

[0010] When the position of the aerial image in the current sampling period is adjusted to a position suitable for viewing by the observer, it is determined that the adjustment of the position of the aerial image is completed.

[0011] The beneficial effects of the present application are: when the image source outputs an image, the position of the aerial image is determined according to a preset sampling period, and the position of the observer's eye point is determined according to the preset sampling period; the observation altitude angle and the observation deviation angle are determined according to the position of the aerial image in the current period and the position of the observer's eye point; the angle of the optical reflection plate is adjusted by the observation altitude angle and the observation deviation angle to compensate for the position of the aerial image; when the position of the aerial image in the current sampling period is adjusted to a position suitable for the observer to watch, it is determined that the position of the aerial image has been adjusted. Since the observer's eye point position can be collected in real time, and the position of the aerial image can be adjusted accordingly according to the observer's eye point position, so that the picture can be in the best position in real time from the user's perspective, adapting to the user's interaction or viewing needs, thereby avoiding the situation of picture asynchrony and improving the user's interactive experience.

[0012] In one embodiment, when the image source outputs an image, light from the image source is reflected by a reflective plate and enters a flat lens with a negative refractive index, and aerial imaging is achieved through refraction of the flat lens. The determining of the position of the aerial imaging according to a preset sampling period includes:

[0013] The position of the aerial image is determined according to the angle of the reflection plate and the refractive index of the plate lens.

[0014] In one embodiment, determining the position of the observer's eye point according to a preset sampling period includes:

[0015] capturing a facial image of the observer according to a facial image capturing device;

[0016] Recognize the facial image to determine the location of the eye points in the facial image;

[0017] The position of the observer's eye points is determined according to the position of the eye points in the facial image.

[0018] In one embodiment, adjusting the angle of the optical reflective plate by the observation altitude angle and the observation deviation angle comprises:

[0019] Determine the product of the observation altitude angle and the first preset proportional coefficient as the altitude compensation value of the observation altitude angle, and determine the product of the observation deviation angle and the second preset proportional coefficient as the deviation compensation value of the observation deviation angle;

[0020] The height compensation value and the deviation compensation value are used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt platform.

[0021] In one embodiment, adjusting the angle of the optical reflective plate by the observation altitude angle and the observation deviation angle comprises:

[0022] The height compensation value of the observation altitude angle is determined according to the following first preset formula:

[0023] Δθ=K p ·[e1(t)-e1(t-1)]+K i ·e1(t)+K d [e1(t)-2e1(t-1)+e1(t-2)];

[0024] Where Δθ is the height compensation value; e1 is the difference between the target value and the current value of the observed height angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the two previous sampling times of the current sampling time, respectively; K p , K i , K d They are respectively the preset adjustment coefficients corresponding to the height compensation values;

[0025] The deviation compensation value of the observation deviation angle is determined according to the following second preset formula:

[0026] Δα=K′ p ·[e2(t)-e2(t-1)]+K′ i ·e2(t)+K' d [e2(t)-2e2(t-1)+e2(t-2)];

[0027] Where Δα is the height compensation value, e2 is the difference between the target value and the current value of the observed deviation angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the previous two sampling times of the current sampling time, respectively; K′ p , K′ i , K′ d They are respectively the preset adjustment coefficients corresponding to the deviation compensation values;

[0028] The height compensation value and the deviation compensation value are used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt platform.

[0029] In one embodiment, the method further comprises:

[0030] The planar image interacting with the aerial imaging is mapped to a preset area according to the following formula:

[0031] X target =x rot +cos(α rot )×(r width ·x source -x rot )-sin(α rot)×(r height ·y source -y rot );

[0032] y target =sec(α rot )×[y rot +sin(α rot )×(r width ·x source -x rot )+cos(α rot )×(r height ·y source -y rot )];

[0033] Among them, X rot and rot are the horizontal and vertical coordinates of the equivalent rotation center; α rot is the parallax equivalent rotation angle; x source and source The pixel coordinates of the original image in the first picture; r width and r height are respectively the ratios of the width and height of the first picture original image to the first picture display array.

[0034] In one embodiment, the method further comprises:

[0035] When an interaction event between the aerial image and the plane image is detected, querying the interaction event library to determine whether the interaction event belongs to a known interaction event;

[0036] When the interaction event does not belong to a known interaction event, determining whether the interaction event between the aerial image and the plane image deviates from a known interaction event;

[0037] When the interaction event between the aerial imaging and the plane image deviates from a known interaction event, the interaction event is expanded to an interaction event library.

[0038] The present application also provides an aerial imaging control device, comprising:

[0039] A first determination module is used to determine the position of the aerial image according to a preset sampling period when the image source outputs an image, and to determine the position of the observer's eye point according to the preset sampling period;

[0040] The second determination module is used to determine the observation altitude angle and the observation deviation angle according to the position of the aerial imaging in the current period and the position of the observer's eye point;

[0041] An adjustment module, used for adjusting the angle of the optical reflective plate through the observation altitude angle and the observation deviation angle, so as to compensate for the position of the aerial imaging;

[0042] The third determination module is used to determine that the adjustment of the position of the aerial imaging is completed when the position of the aerial imaging in the current sampling period is adjusted to a position suitable for viewing by the observer.

[0043] In one embodiment, when the image source outputs an image, light from the image source is reflected by a reflective plate and enters a flat lens with a negative refractive index, and aerial imaging is achieved through refraction of the flat lens. The first determining module includes:

[0044] The first determining submodule is used to determine the position of the aerial imaging according to the angle of the reflecting plate and the refractive index of the plate lens.

[0045] In one embodiment, the first determining module includes:

[0046] A collection submodule, used for collecting a facial image of the observer according to a facial image collection device;

[0047] A recognition submodule, used for recognizing the facial image to determine the position of the eye points in the facial image;

[0048] The second determination submodule is used to determine the position of the observer's eye points according to the position of the eye points in the facial image.

[0049] In one embodiment, the adjustment module includes:

[0050] A third determination submodule is used to determine the product of the observation altitude angle and the first preset proportional coefficient as the altitude compensation value of the observation altitude angle, and to determine the product of the observation deviation angle and the second preset proportional coefficient as the deviation compensation value of the observation deviation angle;

[0051] The first adjustment submodule is used to use the height compensation value and the deviation compensation value as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust the angle, and then adjust the angle of the optical reflective plate through the angle adjustment of the pan-tilt platform.

[0052] In one embodiment, the adjustment module includes:

[0053] The fourth determination submodule is used to determine the height compensation value of the observation altitude angle according to the following first preset formula:

[0054] Δθ=K p ·[e1(t)-e1(t-1)]+K i ·e1(t)+K d[e1(t)-2e1(t-1)+e1(t-2)];

[0055] Where Δθ is the height compensation value; e1 is the difference between the target value and the current value of the observed height angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the two previous sampling times of the current sampling time, respectively; K p , K i , K d They are respectively the preset adjustment coefficients corresponding to the height compensation values;

[0056] The fifth determination submodule is used to determine the deviation compensation value of the observation deviation angle according to the following second preset formula:

[0057] Δα=K′ p ·[e2(t)-e2(t-1)]+K′ i ·e2(t)+K' d [e2(t)-2e2(t-1)+e2(t-2)];

[0058] Where Δα is the height compensation value, e2 is the difference between the target value and the current value of the observed deviation angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the previous two sampling times of the current sampling time, respectively; K′ p , K′ i , K′ d They are respectively the preset adjustment coefficients corresponding to the deviation compensation values;

[0059] The second adjustment submodule is used to use the height compensation value and the deviation compensation value as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust the angle, and then adjust the angle of the optical reflective plate through the angle adjustment of the pan-tilt platform.

[0060] In one embodiment, the apparatus further comprises:

[0061] A mapping module is used to map a plane image interacting with the aerial imaging to a preset area according to the following formula:

[0062] x target =x rot +cos(α rot )×(r width ·x source -x rot )-sin(α rot )×(r height ·y source -y rot );

[0063] y target =sec(α rot )×[yrot +sin(α rot )×(r width ·x source -x rot )+cos(α rot )×(r height ·y source -y rot )];

[0064] Among them, x rot and rot are the horizontal and vertical coordinates of the equivalent rotation center; α rot is the parallax equivalent rotation angle; x source and source The pixel coordinates of the original image in the first picture; r width and r height are respectively the ratios of the width and height of the first picture original image to the first picture display array.

[0065] In one embodiment, the apparatus further comprises:

[0066] A query module, configured to query an interaction event library to determine whether the interaction event belongs to a known interaction event when an interaction event between the aerial imaging and the plane image is detected;

[0067] A judgment module, configured to judge whether the interaction event between the aerial image and the plane image deviates from a known interaction event when the interaction event does not belong to a known interaction event;

[0068] The expansion module is used to expand the interaction event to the interaction event library when the interaction event between the aerial imaging and the plane image deviates from the known interaction event.

[0069] The present application also provides an aerial imaging control system, comprising:

[0070] at least one processor; and,

[0071] a memory communicatively connected to the at least one processor; wherein,

[0072] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the aerial imaging control method recorded in any of the above embodiments.

[0073] The present application also provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor corresponding to the aerial imaging control system, the aerial imaging control system can implement the aerial imaging control method recorded in any of the above embodiments.

[0074] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0075] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0077] Figure 1 This is a flow chart of an aerial imaging control method in one embodiment of the present application;

[0078] Figure 2 This is a schematic diagram of the cross-media aerial imaging principle in one embodiment of the present application;

[0079] Figure 3 This is a flow chart of the pan / tilt control in one embodiment of the present application;

[0080] Figure 4 This is a schematic diagram of a cross-media display synchronization control method in an embodiment of the present application;

[0081] Figure 5 This is a schematic diagram of a cross-media linkage display effect in one embodiment of the present application;

[0082] Figure 6 This is a schematic diagram of the structure of an aerial imaging control device in one embodiment of the present application;

[0083] Figure 7 The figure is a schematic diagram of the hardware structure of an aerial imaging control system in one embodiment of the present application. DETAILED DESCRIPTION

[0084] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application.

[0085] When considering designing a cross-media screen control, due to factors such as differences in image generation controllers, imaging principles, display resolutions, and parallax caused by changes in relative viewing angles, it is easy to produce problems such as screen asynchrony, stiff linkage, and poor interactive experience. Therefore, it is necessary to solve the problems of screen equivalent mapping and screen synchronization so that cross-media screens can achieve linkage functions such as line of sight compensation, gesture interaction, and voice interaction.

[0086] Figure 1 Flow chart of an aerial imaging control method in one embodiment of the present application. Figure 1 As shown, the method can be implemented as the following steps S101-S104:

[0087] In step S101, when the image source outputs an image, the position of the aerial image is determined according to a preset sampling period, and the position of the observer's eye point is determined according to the preset sampling period;

[0088] In step S102, the observation altitude angle and the observation deviation angle are determined according to the position of the aerial imaging in the current period and the position of the observer's eye point;

[0089] In step S103, the angle of the optical reflective plate is adjusted by the observation altitude angle and the observation deviation angle to compensate for the position of the aerial imaging;

[0090] In step S104, when the position of the aerial image in the current sampling period is adjusted to a position suitable for viewing by the observer, it is determined that the adjustment of the position of the aerial image is completed.

[0091] Figure 2 This is a schematic diagram of cross-media aerial imaging in one embodiment of the present application. Figure 2 As shown, 201 is a camera module, 202 is an observer's eye point, wherein the camera module is used to collect an observer's image, in particular, collect an observer's eye image to determine the observer's eye point. 203 is a first picture, 204 is a second picture, wherein the display contents of the first picture and the second picture may be consistent or inconsistent. When the display contents of the first picture and the second picture are consistent, multi-screen synchronous display can be achieved; when the display contents of the first picture and the second picture are inconsistent, a visual superposition effect is generated. 205 is an image source, which is used to generate an image for the second picture; 206 is an optical path magnifying lens group; 207 is an optical reflective plate; 208 is a plate lens with a negative refractive index; and 209 is a multi-axis gimbal. After the image source 205 generates a picture, the light path passes through the refraction of the optical path magnifying lens group 206 and the reflection of the optical reflective plate 207, and the second picture 204 is presented in the axially symmetrical air through the negative refractive index plate lens 208. The optical reflective plate is movably connected to the multi-axis gimbal, and the multi-axis gimbal can adjust the position of the second image by changing the angle of the optical reflective plate. Cofficient (θ)·γ+θ=π / 2, adjustment coefficient a Cofficient is a function of θ.

[0092] In this application, in order to achieve the real-time linkage effect of multiple media aerial imaging, it is necessary to adjust the position of the aerial imaging according to the observer's observation angle. Therefore, when the image source outputs an image, the position of the aerial imaging is determined according to the preset sampling period, and the position of the observer's eye point is determined according to the preset sampling period. Among them, the preset sampling period is a pre-set value, and the appropriate adoption period can be determined according to the equipment performance and actual needs. For example, 0.02 seconds to 0.04 seconds can be selected. In order to determine the position of the aerial imaging, such as Figure 2 As shown, when the image source outputs an image, the light of the image source enters the flat lens with a negative refractive index after being reflected by the reflective plate, and the aerial image is formed by the refraction of the flat lens. Therefore, the position of the aerial image can be determined according to the angle of the reflective plate and the refractive index of the flat lens. In order to determine the position of the observer's eye point, the observer's facial image can be collected by a facial image collection device; the facial image is recognized to determine the position of the eye point in the facial image; and the observer's eye point is determined according to the position of the eye point in the facial image.

[0093] The observation altitude angle and the observation deviation angle are determined according to the position of the aerial image in the current cycle and the position of the observer's eye point. Figure 2 As shown, the angle α in the figure is the observation deviation angle, which indicates the horizontal deviation angle of the angle between the center line of the eye point and the XZ plane when the observer observes the aerial image, which can be estimated by the deviation of the pixel point between the center of the observer's eye point and the horizontal symmetric center of the image screen in the image captured by the camera. The angle θ in the figure is the observation altitude angle, which indicates the vertical deviation angle of the angle between the line connecting the center of the eye point and the center of the second screen when the observer observes the aerial image, and the XY plane, which can be calculated based on the distance between the screen and the observer's face, H1 is the height of the plane where the center of the second screen is located and the flat lens, and H2 is the height of the eye point and the flat lens, which can be calculated based on the captured image of the observer.

[0094] The angle of the optical reflective plate is adjusted by the observation altitude angle and the observation deviation angle to compensate for the position of the aerial imaging. In order to adjust the position of the second screen image, the present application drives the change of the angle of the optical reflective plate by changing the angle of the multi-axis gimbal, thereby changing the position of the image. In one embodiment of the present application, the product of the observation altitude angle and the first preset proportional coefficient is determined as the height compensation value of the observation altitude angle, and the product of the observation deviation angle and the second preset proportional coefficient is determined as the deviation compensation value of the observation deviation angle; for example, a certain compensation sampling period, such as 2 seconds, can be set, and a suitable first preset proportional coefficient and second preset proportional coefficient can be set, which can be obtained by conversion according to the optical principle diagram, or can be selected according to experience. Then, the altitude compensation value and the deviation compensation value are used as the adjustment value of the gimbal to control the gimbal to adjust the angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the gimbal.

[0095] In another embodiment of the present application, the height compensation value of the observation altitude angle and the deviation compensation value of the observation deviation angle are determined based on the incremental PID control algorithm. Specifically, the height compensation value of the observation altitude angle is determined according to the following first preset formula:

[0096] Δθ=K p ·[e1(t)-e1(t-1)]+K i ·e1(t)+K d [e1(t)-2e1(t-1)+e1(t-2)];

[0097] Where Δθ is the height compensation value; e1 is the difference between the target value and the current value of the observed height angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the two previous sampling times of the current sampling time, respectively; K p , K i , K d They are respectively the preset adjustment coefficients corresponding to the height compensation values;

[0098] At the same time, the deviation compensation value of the observation deviation angle is determined according to the following second preset formula:

[0099] Δa=K′ p ·[e2(t)-e2(t-1)]+K′ i ·e2(t)+K' d [e2(t)-2e2(t-1)+e2(t-2)];

[0100] Where Δα is the height compensation value, e2 is the difference between the target value and the current value of the observed deviation angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the previous two sampling times of the current sampling time, respectively; K′ p , K′ i, K′ d They are respectively the preset adjustment coefficients corresponding to the deviation compensation values;

[0101] The height compensation value and the deviation compensation value are then used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and the angle of the optical reflective plate is then adjusted through the angle adjustment of the pan-tilt platform.

[0102] Figure 3 This is a flow chart of the pan / tilt control in one embodiment of the present application. Figure 3 As shown, when adjusting the gimbal, first determine whether the deviation compensation value of the observation deviation angle is less than the angle deviation threshold. When the deviation compensation value of the observation deviation angle is greater than the angle deviation threshold, the Z-axis angle of the gimbal is controlled to adjust according to the deviation compensation value of the observation deviation angle; otherwise, the Z-axis angle of the gimbal is not adjusted. At the same time, determine whether the height compensation value of the observation altitude angle is less than the altitude deviation threshold. When the height compensation value is greater than the altitude deviation threshold, the Y-axis angle of the gimbal is controlled to adjust according to the height compensation value of the observation altitude angle; otherwise, the Y-axis of the gimbal is not adjusted. When the deviation compensation value of the observation deviation angle is less than the angle deviation threshold, and / or the height compensation value of the observation altitude angle is less than the altitude deviation threshold, due to the existence of slight parallax, in order to improve the picture display effect, the observation time difference can be compensated. Specifically, first determine the parallax equivalent rotation center (x rot ,y rot ) and parallax equivalent rotation angle α rot , where x rot and rot is the coordinate of the intersection of the line connecting the observation point and the center point of the second picture on the plane of the first picture, and the parallax equivalent rotation angle α rot It is proportional to the observation deviation angle α, that is, α rot = kα, where k is a preset adjustment coefficient. Then, the plane image interacting with the aerial imaging is mapped to the preset area by the following formula:

[0103] x target =x rot +cos(α rot )×(r width ·x source -x rot )-sin(α rot )×(r height ·y source -y rot );

[0104] y target =sec(α rot )×[y rot +sin(α rot )×(r width ·xsource -x rot )+cos(α rot )×(r height ·y source -y rot )];

[0105] Among them, x rot and rot are the horizontal and vertical coordinates of the equivalent rotation center; α rot is the parallax equivalent rotation angle; x source and source The pixel coordinates of the original image in the first picture; r width and r height are respectively the ratios of the width and height of the first picture original image to the first picture display array.

[0106] In addition, in order to make the image adjustment process smooth, the angular velocity of the gimbal can also be controlled through the cascade PID to achieve a more stable angle compensation. Specifically, the height compensation value of the observation altitude angle and the deviation compensation value of the observation deviation angle are obtained respectively; then the first angular velocity and the second angular velocity of the gimbal are determined by querying the first correspondence table and the second correspondence table, wherein the first correspondence table is a pre-stored correspondence table of the height compensation value of the observation altitude angle and the first angular velocity of the gimbal Y-axis adjustment, and the first correspondence table is a pre-stored correspondence table of the deviation compensation value of the observation deviation angle and the first angular velocity of the gimbal Z-axis adjustment; finally, the angular velocity compensation value of the gimbal adjustment is determined according to the following formula:

[0107]

[0108]

[0109] Among them, Δω1 and Δω2 are the first angular velocity compensation value and the second angular velocity compensation value of the gimbal respectively; and are the target angular velocity values ​​of the Y axis and the Z axis of the gimbal, respectively; ω1 and ω2 are the difference between the first angular velocity target value of the gimbal and the current first angular velocity value, and the difference between the second angular velocity target value of the gimbal and the current second angular velocity value, respectively; t, t-1, and t-2 represent the current sampling moment, the previous sampling moment, and the two previous sampling moments of the current sampling moment, respectively; K" p , K” d are the preset adjustment coefficients corresponding to the first angular velocity compensation value of the gimbal; K”' p , K”' d They are respectively the preset adjustment coefficients corresponding to the second angular velocity compensation value of the gimbal.

[0110] It can be understood that, in the above PID control algorithm, each preset adjustment coefficient can be a pre-set value, or can be adjusted by self-learning of the neural network to achieve the optimal combination of control parameters.

[0111] Finally, when the position of the aerial imaging in the current sampling period is adjusted to a position suitable for viewing by the observer, it is determined that the adjustment of the position of the aerial imaging is completed.

[0112] Of course, in order to prevent the observer from moving, after the adjustment is completed, the above strategy still needs to be executed according to the preset sampling period.

[0113] The beneficial effects of the present application are: when the image source outputs an image, the position of the aerial image is determined according to a preset sampling period, and the position of the observer's eye point is determined according to the preset sampling period; the observation altitude angle and the observation deviation angle are determined according to the position of the aerial image in the current period and the position of the observer's eye point; the angle of the optical reflection plate is adjusted by the observation altitude angle and the observation deviation angle to compensate for the position of the aerial image; when the position of the aerial image in the current sampling period is adjusted to a position suitable for the observer to watch, it is determined that the position of the aerial image has been adjusted. Since the observer's eye point position can be collected in real time, and the position of the aerial image can be adjusted accordingly according to the observer's eye point position, so that the picture can be in the best position in real time from the user's perspective, adapting to the user's interaction or viewing needs, thereby avoiding the situation of picture asynchrony and improving the user's interactive experience.

[0114] In one embodiment, when the image source outputs an image, the light of the image source enters a flat lens with a negative refractive index after being reflected by a reflective plate, and aerial imaging is achieved through refraction by the flat lens. The above step S101 of determining the position of the aerial imaging according to a preset sampling period may also be implemented as the following steps:

[0115] The position of the aerial image is determined according to the angle of the reflection plate and the refractive index of the plate lens.

[0116] In one embodiment, the above step S101 of determining the position of the observer's eye point according to the preset sampling period can also be implemented as the following steps A1-A3:

[0117] In step A1, a facial image of an observer is captured by a facial image capturing device;

[0118] In step A2, the facial image is identified to determine the position of the eye points in the facial image;

[0119] In step A3, the positions of the observer's eyes are determined according to the positions of the eyes in the facial image.

[0120] In one embodiment, the above step S103 can also be implemented as the following steps B1-B2:

[0121] In step B1, the product of the observation altitude angle and the first preset proportional coefficient is determined as the altitude compensation value of the observation altitude angle, and the product of the observation deviation angle and the second preset proportional coefficient is determined as the deviation compensation value of the observation deviation angle;

[0122] In step B2, the height compensation value and the deviation compensation value are used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt platform.

[0123] In this embodiment, a certain compensation sampling period, such as 2 seconds, can be set, and a proportional coefficient, such as 0.9 to 1.2, can be set. The product of the observation altitude angle and the preset proportional coefficient is the height compensation value of the observation altitude angle, and the product of the observation deviation angle and the preset proportional coefficient is the deviation compensation value of the observation deviation angle. Then, the altitude compensation value and the deviation compensation value are used as the adjustment values ​​of the pan-tilt to control the pan-tilt to adjust the angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt.

[0124] By using this method to compensate for the observation angle deviation and the observation altitude angle deviation, not only can the stability of the equipment be increased, but also direct calculation can be performed without iteration, thus achieving high calculation efficiency.

[0125] When adjusting the gimbal, the deviation compensation value of the observation angle is adjusted by controlling the Z-axis angle of the gimbal; the height compensation value of the observation altitude angle is adjusted by controlling the Y-axis angle of the gimbal.

[0126] In one embodiment, the above step S103 can also be implemented as the following steps C1-C2:

[0127] In step C1, the height compensation value of the observation altitude angle is determined according to the following first preset formula:

[0128] Δθ=K p ·[e1(t)-e1(t-1)]+K i ·e1(t)+K d [e1(t)-2e1(t-1)+e1(t-2)];

[0129] Where Δθ is the height compensation value; e1 is the difference between the target value and the current value of the observed height angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the two previous sampling times of the current sampling time, respectively; K p , K i , K dThey are respectively the preset adjustment coefficients corresponding to the height compensation values;

[0130] In step C2, the deviation compensation value of the observation deviation angle is determined according to the following second preset formula:

[0131] Δα=K′ p ·[e2(t)-e2(t-1)]+K′ i ·e2(t)+K' d [e2(t)-2e2(t-1)+e2(t-2)];

[0132] Where Δα is the height compensation value, e2 is the difference between the target value and the current value of the observed deviation angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the previous two sampling times of the current sampling time, respectively; K′ p , K′ i , K′ d They are respectively the preset adjustment coefficients corresponding to the deviation compensation values;

[0133] In step C3, the height compensation value and the deviation compensation value are used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt platform.

[0134] In this embodiment, the height compensation value of the observation altitude angle and the deviation compensation value of the observation deviation angle are determined based on the incremental PID control algorithm. Specifically, the first and second preset formulas are substituted by the latest three sampling values ​​to determine the corresponding compensation values. The advantage is that the control method has high control accuracy after steady state, and because of the unpredictability of the observation angle of view, only the latest three sampling values ​​are associated, and the calculation efficiency is high. In addition, in order to eliminate excessively frequent movements of the equipment in actual control, a dead zone, that is, an angle deviation threshold, can also be added.

[0135] In one embodiment, the method can also be implemented as follows:

[0136] The planar image interacting with the aerial imaging is mapped to a preset area according to the following formula:

[0137] x target =x rot +cos(α rot )×(r width ·x source -x rot )-sin(α rot )×(r height ·y source -y rot );

[0138] y target =sec(αrot )×[y rot +sin(α rot )×(r width ·x source -x rot )+cos(α rot )×(r height ·y source -y rot )];

[0139] Among them, X rot and rot are the horizontal and vertical coordinates of the equivalent rotation center; α rot is the parallax equivalent rotation angle; x source and source The pixel coordinates of the original image in the first picture; r width and r height are respectively the ratios of the width and height of the first picture original image to the first picture display array.

[0140] In one embodiment, the method can also be implemented as the following steps D1-D3:

[0141] In step D1, when an interaction event between the aerial imaging and the plane image is detected, the interaction event library is queried to determine whether the interaction event belongs to a known interaction event;

[0142] In step D2, when the interaction event does not belong to a known interaction event, it is determined whether the interaction event between the aerial image and the plane image deviates from a known interaction event;

[0143] In step D3, when the interaction event between the aerial imaging and the plane image deviates from the known interaction event, the interaction event is expanded to the interaction event library.

[0144] Figure 4 This is a schematic diagram of a cross-media display synchronization control method in an embodiment of the present application, as shown in Figure 4. When an interaction event between an aerial image and the plane image is detected, the interaction event library is first queried to determine whether the interaction event belongs to a known interaction event. The interaction event can be an interaction intention collected and determined by the interaction perception module, such as voice, gesture, facial expression, or even multi-modal fusion input.

[0145] Then, match this interaction event from the stored interaction event library. When the interaction event does not belong to a known interaction event, determine whether the interaction event between the aerial imaging and the plane image deviates from the known interaction event. Specifically, match the closest target preset interaction event according to the maximum similarity, obtain the preset screen synchronization parameters of the matched target preset interaction event, generate the first screen and the second screen according to the preset screen synchronization parameters, perform cross-media screen output, and update the screen according to the interaction synchronization. When the cross-media screen output is completed and the current interaction event ends, return to the screen of this interaction to generate the equivalent time T Return_1& T Return_2, and compare with the preset interaction event, compare the difference in the equivalent generation time of the screen to determine whether it deviates from the known interaction event.

[0146] When the interaction event between the aerial imaging and the plane image deviates from a known interaction event, the interaction event is expanded to an interaction event library.

[0147] It should be noted that, when the interaction event between the aerial imaging and the plane image does not deviate from the known interaction event, it is determined whether the interaction screens corresponding to the interaction event between the aerial imaging and the plane image are synchronized. When the interaction screens corresponding to the interaction event between the aerial imaging and the plane image are not synchronized, the screen synchronization parameters are updated.

[0148] For example, when the difference in equivalent generation time exceeds the first preset value range, it indicates a deviation from a known interaction event, and the interaction event is used as a new interaction event to expand the interaction event library; otherwise, when the difference in equivalent generation time is within the first preset value range, it is determined whether it deviates from screen synchronization. When the difference in equivalent generation time is within the second preset value range, it indicates that there is no deviation from screen synchronization. Otherwise, when the interaction screens corresponding to the interaction events of the aerial imaging and the plane image are not synchronized, the screen synchronization parameters are updated.

[0149] Figure 5It is a schematic diagram of the cross-media linkage display effect in one embodiment of the present application. The light of the reflector screen 501 passes through the flat lens 503, and after multiple reflections, produces a light path as shown in 502, and produces a suspended real image 506 at a certain height from the surface of the structure. The control method proposed in this application generates a screen equivalent mapping and a screen synchronization signal to the light-emitting surface 504, and generates a linkage screen as shown in 505. Screens 505 and 506 are superimposed in the observation angle, thereby presenting a sense of combining virtual and real (in this case, it is manifested as a cross-media display effect of white hair floating on green water to produce ripples). When an interactive request is received, such as controlling the screen 506 to move to the left, the processor uses the control method proposed in this article to update the screen 505 to respond (in this case, it is manifested as the duck swimming to the left, and the ripples also follow the movement and spread).

[0150] Next, the interaction between aerial imaging and plane images is further introduced. Specifically, for some simple control instructions, some aerial imaging movement trajectories, aerial imaging playback animations and corresponding plane image linkage animation effects can be pre-stored. For example, aerial imaging is Figure 5 The duck in the video can be a water ripple. Some simple aerial imaging movement trajectories can be stored in advance. At this time, the linkage animation of the plane image corresponding to the aerial imaging movement trajectory can be the ripples left after the aerial imaging moves. The plane image is a picture formed by a large number of LED lights. By adjusting the switching sequence of the LED lights in the corresponding positions and the changes in brightness, the corresponding ripples can be displayed. The user can control the aerial imaging duck to move in the corresponding direction through gestures. Figure 2 The camera module 201 in the image processing apparatus is used to obtain the user gesture, thereby determining the moving trajectory of the aerial imaging according to the user gesture. When the similarity between the moving trajectory of the aerial imaging and the pre-stored first preset trajectory is greater than the preset similarity (such as greater than 95%), the animation corresponding to the first preset trajectory is played.

[0151] When the similarity between the moving trajectory of the aerial imaging and all pre-stored trajectories is less than the preset similarity, it means that the interaction event between the aerial imaging and the plane image has deviated from the known interaction event, and the control instruction is determined to be a complex control instruction. At this time, the interaction event can be expanded to the interaction event library.

[0152] The method of expanding the interaction event to the interaction event library is as follows:

[0153] A large number of video frames can be stored in advance, and the position information and direction information of the aerial image in each video frame can be annotated, and each video frame can be associated with the LED array display mode of the plane image. When the similarity between the monitored moving trajectory of the aerial image and all the pre-stored trajectories is less than the preset similarity, the position information and direction information of each point of the moving trajectory are analyzed, and the video frames corresponding to the position information and direction information of each point are loaded one by one according to the annotated information of the position information and direction information of the aerial image, so as to form a coherent animation. On this basis, the LED array display mode of the plane image associated with the corresponding video frame is loaded, so that the ripples in the plane image match the trajectory of the aerial image, and the response to the complex control instructions is completed. The loading order of the video frames and the corresponding LED array display mode of the plane image during this interactive event are expanded to the interactive event library as new interactive events.

[0154] Figure 6 FIG. 1 is a schematic diagram of the structure of an aerial imaging control device in one embodiment of the present application. Figure 6 As shown, including:

[0155] The first determination module 601 is used to determine the position of the aerial image according to a preset sampling period when the image source outputs an image, and to determine the position of the observer's eye point according to the preset sampling period;

[0156] The second determination module 602 is used to determine the observation altitude angle and the observation deviation angle according to the position of the aerial imaging in the current period and the position of the observer's eye point;

[0157] An adjustment module 603, used to adjust the angle of the optical reflective plate according to the observation altitude angle and the observation deviation angle, so as to compensate for the position of the aerial imaging;

[0158] The third determination module 604 is configured to determine that the adjustment of the position of the aerial imaging is completed when the position of the aerial imaging in the current sampling period is adjusted to a position suitable for viewing by the observer.

[0159] In one embodiment, when the image source outputs an image, light from the image source is reflected by a reflective plate and enters a flat lens with a negative refractive index, and aerial imaging is achieved through refraction of the flat lens. The first determining module includes:

[0160] The first determining submodule is used to determine the position of the aerial imaging according to the angle of the reflecting plate and the refractive index of the plate lens.

[0161] In one embodiment, the first determining module includes:

[0162] A collection submodule, used for collecting a facial image of the observer according to a facial image collection device;

[0163] A recognition submodule, used for recognizing the facial image to determine the position of the eye points in the facial image;

[0164] The second determination submodule is used to determine the position of the observer's eye points according to the position of the eye points in the facial image.

[0165] In one embodiment, the adjustment module includes:

[0166] A third determination submodule is used to determine the product of the observation altitude angle and the first preset proportional coefficient as the altitude compensation value of the observation altitude angle, and to determine the product of the observation deviation angle and the second preset proportional coefficient as the deviation compensation value of the observation deviation angle;

[0167] The first adjustment submodule is used to use the height compensation value and the deviation compensation value as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust the angle, and then adjust the angle of the optical reflective plate through the angle adjustment of the pan-tilt platform.

[0168] In one embodiment, the adjustment module includes:

[0169] The fourth determination submodule is used to determine the height compensation value of the observation altitude angle according to the following first preset formula:

[0170] Δθ=K p ·[e1(t)-e1(t-1)]+K i ·e1(t)+K d [e1(t)-2e1(t-1)+e1(t-2)];

[0171] Where Δθ is the height compensation value; e1 is the difference between the target value and the current value of the observed height angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the two previous sampling times of the current sampling time, respectively; K p , K i , K d They are respectively the preset adjustment coefficients corresponding to the height compensation values;

[0172] The fifth determination submodule is used to determine the deviation compensation value of the observation deviation angle according to the following second preset formula:

[0173] Δα=K′ p ·[e2(t)-e2(t-1)]+K′ i ·e2(t)+K' d [e2(t)-2e2(t-1)+e2(t-2)];

[0174] Where Δα is the height compensation value, e2 is the difference between the target value and the current value of the observed deviation angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the previous two sampling times of the current sampling time, respectively; K′ p , K′ i , K′ d They are respectively the preset adjustment coefficients corresponding to the deviation compensation values;

[0175] The second adjustment submodule is used to use the height compensation value and the deviation compensation value as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust the angle, and then adjust the angle of the optical reflective plate through the angle adjustment of the pan-tilt platform.

[0176] In one embodiment, the apparatus further comprises:

[0177] A mapping module is used to map a plane image interacting with the aerial imaging to a preset area according to the following formula:

[0178] x target =x rot +cos(α rot )×(r width ·x source -x rot )-sin(α rot )×(r height ·y source -y rot );

[0179] y target =sec(α rot )×[y rot +sin(α rot )×(r width ·x source -x rot )+cos(α rot )×(r height ·y source -y rot )];

[0180] Among them, x rot and rot are the horizontal and vertical coordinates of the equivalent rotation center; α rot is the parallax equivalent rotation angle; x source and source The pixel coordinates of the original image in the first picture; r width and r height are respectively the ratios of the width and height of the first picture original image to the first picture display array.

[0181] In one embodiment, the apparatus further comprises:

[0182] A query module, configured to query an interaction event library to determine whether the interaction event belongs to a known interaction event when an interaction event between the aerial imaging and the plane image is detected;

[0183] A judgment module, configured to judge whether the interaction event between the aerial image and the plane image deviates from a known interaction event when the interaction event does not belong to a known interaction event;

[0184] The expansion module is used to expand the interaction event to the interaction event library when the interaction event between the aerial imaging and the plane image deviates from the known interaction event.

[0185] Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an aerial imaging control system in an embodiment of the present application. Figure 7 As shown, the aerial imaging control system comprises:

[0186] at least one processor 720; and,

[0187] A memory 704 in communication with the at least one processor 720; wherein,

[0188] The memory 704 stores instructions that can be executed by the at least one processor 720, and the instructions are executed by the at least one processor 720 to implement the aerial imaging control method described in any of the above embodiments.

[0189] Reference Figure 7 The aerial imaging control system 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0190] The processing component 702 generally controls the overall operation of the aerial imaging control system 700. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0191] The memory 704 is configured to store various types of data to support the operation of the airborne imaging control system 700. Examples of such data include instructions for any application or method operating on the airborne imaging control system 700, such as text, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0192] The power supply assembly 706 provides power to the various components of the aerial imaging control system 700. The power supply assembly 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the aerial imaging control system 700.

[0193] The multimedia component 708 includes a screen that provides an output interface between the aerial imaging control system 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 may also include a front camera and / or a rear camera. When the aerial imaging control system 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0194] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the aerial imaging control system 700 is in an operation mode, such as an alarm mode, a recording mode, a voice recognition mode, and a voice output mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0195] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0196] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the aerial imaging control system 700. For example, the sensor assembly 714 may include a sound sensor. In addition, the sensor assembly 714 may detect the open / closed state of the aerial imaging control system 700, the relative positioning of components, such as the display and keypad of the aerial imaging control system 700, and the sensor assembly 714 may also detect the operating state of the aerial imaging control system 700 or a component of the aerial imaging control system 700, the orientation or acceleration / deceleration of the aerial imaging control system 700, and the temperature change of the aerial imaging control system 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0197] The communication component 716 is configured to enable the aerial imaging control system 700 to provide the ability to communicate with other devices and cloud platforms in a wired or wireless manner. The aerial imaging control system 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0198] In an exemplary embodiment, the aerial imaging control system 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the aerial imaging control method described in any of the above embodiments.

[0199] The present application also provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor corresponding to the aerial imaging control system, the aerial imaging control system can implement the aerial imaging control method recorded in any of the above embodiments.

[0200] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0201] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An aerial imaging control method, characterized in that: include: When the image source outputs an image, the position of the aerial image is determined according to a preset sampling period, and the position of the observer's eye point is determined according to the preset sampling period; Determine the observation altitude angle and the observation deviation angle according to the position of the aerial image in the current cycle and the position of the observer's eye point; Adjusting the angle of the optical reflective plate by the observation altitude angle and the observation deviation angle to compensate for the position of the aerial imaging; When the position of the aerial image in the current sampling period is adjusted to a position suitable for viewing by the observer, it is determined that the adjustment of the position of the aerial image is completed.

2. The method according to claim 1, characterized in that When the image source outputs an image, the light of the image source enters a flat lens with a negative refractive index after being reflected by a reflective plate, and aerial imaging is achieved through refraction of the flat lens. The position of the aerial imaging is determined according to a preset sampling period, including: The position of the aerial image is determined according to the angle of the reflection plate and the refractive index of the plate lens.

3. The method according to claim 1, characterized in that The step of determining the position of the observer's eye point according to a preset sampling period includes: capturing a facial image of the observer according to a facial image capturing device; Recognize the facial image to determine the location of the eye points in the facial image; The position of the observer's eye points is determined according to the position of the eye points in the facial image.

4. The method according to claim 1, characterized in that The step of adjusting the angle of the optical reflective plate by the observation altitude angle and the observation deviation angle comprises: Determine the product of the observation altitude angle and the first preset proportional coefficient as the altitude compensation value of the observation altitude angle, and determine the product of the observation deviation angle and the second preset proportional coefficient as the deviation compensation value of the observation deviation angle; The height compensation value and the deviation compensation value are used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt platform.

5. The method according to claim 1, characterized in that The step of adjusting the angle of the optical reflective plate by the observation altitude angle and the observation deviation angle comprises: The height compensation value of the observation altitude angle is determined according to the following first preset formula: Δθ=K p ·[e1(t)-e1(t-1)]+K i ·e1(t)+K d ·[e1(t)-2e1(t-1)+e1(t-2)]; Where Δθ is the height compensation value; e1 is the difference between the target value and the current value of the observed height angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the two previous sampling times of the current sampling time, respectively; K p , K i , K d They are respectively the preset adjustment coefficients corresponding to the height compensation values; The deviation compensation value of the observation deviation angle is determined according to the following second preset formula: Δα=K′ p ·[e2(t)-e2(t-1)]+K′ i ·e2(t)+K′ d ·[e2(t)-2e2(t-1)+e2(t-2)]; Where Δα is the height compensation value, e2 is the difference between the target value and the current value of the observed deviation angle; t, t-1, and t-2 represent the current sampling time, the previous sampling time, and the previous two sampling times of the current sampling time, respectively; K′ p , K′ i , K′ d They are respectively the preset adjustment coefficients corresponding to the deviation compensation values; The height compensation value and the deviation compensation value are used as adjustment values ​​of the pan-tilt platform to control the pan-tilt platform to adjust its angle, and then the angle of the optical reflective plate is adjusted by adjusting the angle of the pan-tilt platform.

6. The method according to claim 1, characterized in that The method further comprises: The planar image interacting with the aerial imaging is mapped to a preset area according to the following formula: X target =x rot ++cos(α rot )×(r width ·x source -x rot ))-sin(a rot )×(r height ·y source -y rot ); and target =sec(α rot )×[and rot +sin(α rot )×(r width ·x source -x rot )+cos(α rot )×(r height ·and source -and rot )]; Among them, x rot and rot are the horizontal and vertical coordinates of the equivalent rotation center; α rot is the parallax equivalent rotation angle; x source and source The pixel coordinates of the original image in the first picture; r width and r height are respectively the ratios of the width and height of the first picture original image to the first picture display array.

7. The method according to claim 1, characterized in that The method further comprises: When an interaction event between the aerial image and the plane image is detected, querying the interaction event library to determine whether the interaction event belongs to a known interaction event; When the interaction event does not belong to a known interaction event, determining whether the interaction event between the aerial image and the plane image deviates from a known interaction event; When the interaction event between the aerial imaging and the plane image deviates from a known interaction event, the interaction event is expanded to an interaction event library.

8. An aerial imaging control device, characterized in that: include: A first determination module is used to determine the position of the aerial image according to a preset sampling period when the image source outputs an image, and to determine the position of the observer's eye point according to the preset sampling period; The second determination module is used to determine the observation altitude angle and the observation deviation angle according to the position of the aerial imaging in the current period and the position of the observer's eye point; An adjustment module, used for adjusting the angle of the optical reflective plate through the observation altitude angle and the observation deviation angle, so as to compensate for the position of the aerial imaging; The third determination module is used to determine that the adjustment of the position of the aerial imaging is completed when the position of the aerial imaging in the current sampling period is adjusted to a position suitable for viewing by the observer.

9. An aerial imaging control system, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the aerial imaging control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor corresponding to the aerial imaging control system, the aerial imaging control system can implement the aerial imaging control method as described in any one of claims 1 to 7.