Image control method, apparatus, and system

Abstract

The application discloses an image control method, device and system. The method comprises the following steps: determining target indication position information and target pose information; determining the indication direction of a remote controller as a target indication direction according to the target pose information; adjusting a display image according to the relative position information between each element in the display image, the target indication position information and the target indication direction, to obtain an output image; and outputting the output image. In the application, the output image can change along with the change of the target indication direction, the target indication position information and the relative position information between each element in the display image, so that the diversity of the output image is high, and the diversity of the interaction between the user and the output image and the user experience are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image control method, apparatus, and system. Background Technology

[0002] Currently, in scenarios such as teaching or meetings, users can mark the displayed images on the image display device using markers emitted by marking devices such as laser remote controls. They can also send interactive events to the displayed images using marking devices such as remote controls. Then, the image display device controls the displayed images to change based on the location information of the markers and the sent interactive events, thereby realizing the interaction between the user and the displayed images.

[0003] However, traditional display control methods result in limited content and a lack of interaction between the user and the displayed images, leading to a poor user experience. Summary of the Invention

[0004] In view of the above problems, this application proposes an image control method, apparatus and system that can improve the diversity of displayed content.

[0005] In a first aspect, embodiments of this application provide an image control method, the method comprising: determining target indication position information and target pose information, wherein the target indication position information refers to the indication position information of the remote controller for the displayed image, and the target pose information refers to the pose information of the remote controller, the remote controller being used to indicate the displayed image; determining the indication direction of the remote controller based on the target pose information, as the target indication direction; adjusting the displayed image based on the relative position information between elements in the displayed image, the target indication position information, and the target indication direction, to obtain an output image; and outputting the output image.

[0006] Secondly, embodiments of this application provide an image control device, comprising: an information determination module, configured to determine target indication position information and target pose information, wherein the target indication position information refers to the indication position information of the remote controller for the displayed image, and the target pose information refers to the pose information of the remote controller, the remote controller being used to indicate the displayed image; a direction determination module, configured to determine the indication direction of the remote controller based on the target pose information, as the target indication direction; an adjustment module, configured to adjust the displayed image based on the relative position information between elements in the displayed image, the target indication position information, and the target indication direction, to obtain an output image; and an output module, configured to output the output image.

[0007] Thirdly, embodiments of this application provide an image control device, including:

[0008] One or more processors;

[0009] Memory;

[0010] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, and the applications are configured to perform the methods described above.

[0011] Fourthly, embodiments of this application provide an image control system, which includes at least an image display device, a remote controller, and an image control device, wherein...

[0012] Image display devices are used to display images;

[0013] The remote control is used to instruct the displayed images;

[0014] The image control device is used to perform the method described in the first aspect above.

[0015] This application provides an image control method, apparatus, and system. In this application, the target indication direction is determined based on the target pose information, and the displayed image is adjusted based on the relative position information between elements in the displayed image, the target indication position information, and the target indication direction to obtain an output image. The output image can change with the change of the target indication direction, the target indication position information, and the relative position information between elements in the displayed image, resulting in a high degree of diversity in the output image. This improves the diversity of interaction between the user and the output image and enhances the user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of an image control system applicable to an application environment of this application embodiment is shown.

[0018] Figure 2 A flowchart of an image control method according to an embodiment of this application is shown.

[0019] Figure 3a A schematic diagram of a calibration image is shown in an embodiment of this application.

[0020] Figure 3b A schematic diagram of yet another calibration image in an embodiment of this application is shown.

[0021] Figure 4 A schematic diagram of a screen coordinate system according to an embodiment of this application is shown.

[0022] Figure 5 A schematic diagram of a remote control coordinate system according to an embodiment of this application is shown.

[0023] Figure 6 A flowchart of an image control method according to another embodiment of this application is shown.

[0024] Figure 7 A schematic diagram illustrating one method for obtaining the actual size in an embodiment of this application is shown.

[0025] Figure 8 A flowchart of an image control method according to another embodiment of this application is shown.

[0026] Figure 9 A schematic diagram of an output image according to an embodiment of this application is shown;

[0027] Figure 10a A schematic diagram of an image display embodiment of this application is shown;

[0028] Figure 10b It shows Figure 10a A schematic diagram of the output image corresponding to the displayed image;

[0029] Figure 11a A schematic diagram of yet another display image in an embodiment of this application is shown;

[0030] Figure 11b It shows Figure 11a A schematic diagram of the intermediate image corresponding to the displayed image;

[0031] Figure 11c It shows Figure 10a A schematic diagram of the output image corresponding to the displayed image;

[0032] Figure 12 A block diagram of an image control device according to one embodiment of this application is shown.

[0033] Figure 13 A block diagram of an image control device according to yet another embodiment of this application is shown.

[0034] Figure 14 A structural block diagram of a computer-readable storage medium according to an embodiment of this application is shown. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0036] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0038] Reference Figure 1 , Figure 1 A schematic diagram of an image control system applicable to an embodiment of this application is shown. The image control system includes an image display device 101, an image control device 103, and a remote controller 104.

[0039] The image display device 101 (thickest solid line frame) can be any type or structure of image display device, such as a projector, monitor, and television. The resolution of the image display device can be 1080p, 2K, or 4K, etc. The image display device can be placed in a fixed position, and the displayed image is... Figure 1 106 in the middle.

[0040] When the image display device 101 is a projector, the projector can project the acquired image to be projected onto the projection area (equivalent to...). Figure 1 (At position 106 in the text), the projection area can be equipped with a white projection screen. The white projection screen displays the image to be projected, which is the display image in this application. The display image refers to all the content output by the image display device. For example, if the image display device outputs the acquired target image in a part of the screen, while other parts of the screen are black, then the display image refers to the entire image composed of the target image and the black area.

[0041] Remote controller 103 is used to mark the displayed image, and it also emits indicator 107 for marking the displayed image. The remote controller may be a remote controller that emits laser or colored rays.

[0042] The image control device 104 is used to execute the steps of the image control method in this application. The image control device can be any type of electronic device including a processor and memory, such as a computer or tablet. The image control device can receive a displayed image sent by an image display device and a captured image taken by a camera, and perform a marker positioning operation based on the displayed image and the captured image. The image control device 104 can also control changes in the displayed image based on interactive events. These interactive events can refer to events such as withdrawing, restoring, editing, drawing a trajectory, and viewing attributes of the marker's indicated position.

[0043] In some embodiments, the image display device 101 implements part or all of the image control device 104 through software or hardware functions. That is, the image display device 101 can also serve as the image control device 104 to execute the steps of the image control method of this application.

[0044] Optionally, the image control system may further include a camera 102, which may be a high-definition camera or a digital camera, and may have a zoom function. The camera 102 is used to capture images of the display image. The captured image may include at least the image to be displayed, and may also include other content besides the image to be displayed. The camera's shooting area 105 (the area corresponding to the solid-line frame) includes the display image 106, and the shooting area 105 is larger than the display image 106.

[0045] Reference Figure 2 , Figure 2 This application illustrates a flowchart of an image control method according to an embodiment of the present application. The method is used in an image control device, which may be... Figure 1 The method for method 104 includes:

[0046] S110. Determine the target indication location information and the target pose information.

[0047] Among them, target indication position information refers to the indication position information of the remote control for the displayed image, and target pose information refers to the pose information of the remote control. The remote control is used to indicate the displayed image.

[0048] In this application, the image display device acquires and displays a display image. For example, the image display device is a projector, which projects the acquired image to be projected onto a projection area, and the image displayed in the projection area serves as the display image. As another example, a monitor acquires and displays a webpage, and the interface of the monitor displaying the webpage serves as the display image.

[0049] Users can transmit indicator markers via remote control, which are used to indicate (or mark) content in the displayed image. The indicator markers transmitted by the remote control can be of any shape, such as dots, squares, cursors, or palm-shaped dots, for example... Figure 1 As shown, the indicator is a dot.

[0050] In this embodiment, the indication position information can refer to the indication position of the remote controller towards the displayed image on the image display device; that is, the position of the indication marker emitted by the remote controller in the displayed image is the indication position information. The target indication position information can refer to the indication position information of the remote controller towards the displayed image on the image display device at the current moment. The target indication position information can be the position information in the image coordinate system where the displayed image is located.

[0051] The remote controller's pose information can include the remote controller's coordinates and the sensor information from its inertial measurement unit (IMU). Target pose information can refer to the remote controller's pose at the current moment. For example, the remote controller can be equipped with a Time-of-Flight (TOF) ranging unit and an IMU. The TOF ranging unit determines the remote controller's coordinates, and the IMU determines its sensor information.

[0052] An inertial measurement unit (IMU) is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object. Gyroscopes and accelerometers are the main components of an IMU, and their accuracy directly affects the accuracy of the inertial system.

[0053] A TOF ranging unit refers to a ranging element based on TOF technology. TOF is short for Time of Flight. Broadly speaking, time-of-flight technology can be understood as a technique that measures the time it takes for an object, particle, or wave to travel a certain distance in a fixed medium (where the medium, distance, and time are all known or measurable), thereby further understanding certain properties of ions or media. TOF ranging is a two-way ranging technique that primarily uses the time it takes for a signal to travel back and forth between two asynchronous transceivers (or reflected surfaces) to measure the distance between nodes. Traditional ranging techniques are divided into two-way ranging and one-way ranging. In environments where the signal level is easily modulated or in non-line-of-sight (Line-of-Sight) environments, the results estimated by the RSSI (Received Signal Strength Indication) ranging method are relatively ideal; in line-of-sight (Line-of-Sight) environments, the TOF distance estimation method can compensate for the shortcomings of the RSSI distance estimation method.

[0054] In one implementation, the remote controller can transmit an indicator and determine the target indication location information based on the transmitted indicator. The process of determining the target indication location information may include: if the remote controller transmits an indicator, acquiring a captured image of the corresponding display image, the captured image including the captured indicator of the corresponding indicator; determining the position information of the captured indicator in the captured image as the captured position information; and determining the target indication location information based on a second mapping parameter and the captured position information as the indication location information. The second mapping parameter is used to characterize the mapping relationship between the captured image and the display image.

[0055] The shooting marker can be determined from the captured image using traditional OpenCV algorithms. Alternatively, the gray values ​​of pixels in the captured image with gray values ​​greater than a preset maximum gray value can be adjusted to a first value, and the gray values ​​of pixels with gray values ​​not greater than the preset maximum gray value can be adjusted to a second value, resulting in an adjusted captured image. This adjusted captured image is then convolved with a preset convolution kernel to obtain the convolution result. The position information of values ​​in the convolution result greater than a preset threshold is used as the position information of the shooting marker. Here, the preset maximum gray value can refer to the maximum gray value that the image display device can display, the preset threshold can be a value set based on requirements, the first value can be 1, the second value can be 0, and the size of the preset convolution kernel can be determined based on the size of the marker.

[0056] The process of obtaining the second mapping parameter may include: displaying a calibration image including display feature points through an image display device, and capturing the displayed calibration image using a camera to obtain a captured calibration image; calculating the second mapping parameter using perspective transformation or a homography matrix algorithm based on the pixel coordinates of the captured feature points and their corresponding display feature points in the captured calibration image. The second mapping parameter may be a homography matrix, and the mapping parameter is obtained based on at least four display feature points and four corresponding captured feature points. In some embodiments, more display feature points and corresponding captured feature points may be set to solve for a homography matrix with higher accuracy using a larger number of display feature points and corresponding captured feature points.

[0057] Feature points can be the corner points of a checkerboard grid or the centers of circular patterns within a dot matrix. (See reference...) Figure 3a and Figure 3b , Figure 3a A schematic diagram of a calibration image according to an embodiment of this application is shown. Figure 3b This illustration shows another type of calibration image according to an embodiment of this application. The calibration image can have various patterns. Figure 3a The pattern in the image is a "chessboard". Figure 3b The pattern in it is a "dot matrix".

[0058] like Figure 3a As shown, the feature points are the corner points of the bounding box on the edge. Figure 3a (The point marked by the small circle in the middle), for example Figure 3b As shown, the feature point is the center of the circle whose edge is the circle. Figure 3b (The point marked by the small circle in the middle).

[0059] In the specific implementation of this application, the selection rules for display feature points can be set according to the accuracy requirements, for example... Figure 3a and Figure 3b The calibration image shown can select 4 points at regular intervals of rows or columns as display feature points.

[0060] Display feature points can be determined in the calibration image using traditional OpenCV algorithm libraries, and shooting feature points can be determined in the corresponding captured calibration image.

[0061] S120. Based on the target pose information, determine the remote control's pointing direction as the target pointing direction.

[0062] The target indication direction can be determined based on the sensor information of the inertial measurement unit and the coordinate information of the remote controller in the target pose information. The target indication direction can be a vector in the screen coordinate system of the image display device.

[0063] like Figure 4 As shown, the origin of the screen coordinate system is the remote control 401. The y-axis of the screen coordinate system is the direction of gravity, the z-axis is the direction perpendicular to the screen 401, and the x-axis is the direction determined by the y-axis and z-axis.

[0064] For example, the remote controller includes three Time-of-Flight (TOF) ranging units with different transmission directions. The spatial coordinates Pi(xi,yi,zi), i=1,2,3 of three points on the screen are obtained through the TOF ranging units. The spatial coordinates of these three points are in the remote controller coordinate system.

[0065] like Figure 5 As shown, the origin of the remote control coordinate system is the location of remote control 501, the z-axis is the direction of the indicator emitted by the remote control, and the x-axis is the direction of gravity.

[0066] When the position of the remote control changes, the coordinate system of the remote control also changes. The origin, z-axis direction, and y-axis direction of the remote control coordinate system may change, while the x-axis direction remains unchanged and always remains in the direction of gravity.

[0067] Using the spatial coordinates Pi(xi,yi,zi) of the three points mentioned above, i=1,2,3, the plane where the screen of the image display device is located can be determined. The plane can be represented as A1x+B1y+C1z+D1=0, and we can solve for A1=(y2-y1)*(z3-z1)-(y3-y1)*(z2-z1), B1=(z2-z1)*(x3-x1)-(z3-z1)*(x2-x1), C1=(x2-x1)*(y3-y1)-(x3-x1)*(y2-y1), D1=-(A1x1)*(x2-x1)*(y3-y1)*(y3-x1)*(y2-y1), D1=-(A1x1)*(x2-x1)*(y3-y1)*(y3-x1)*(y2-y1), D1=-(A1x1)*(x2-x1)*(y3-y1)*(y2-y1), From the inertial navigation unit, we can obtain the gravity direction G1(x4,y4,z4). Since the screen of the image display device is parallel to the gravity direction, we can regard G1 as the y-axis (vector) of the screen coordinate system. Then the z-axis (vector) of the screen coordinate system is represented as (A1,B1,C1) (since the z-axis points vertically to the screen, C1 needs to be greater than 0. If C1<0, then the z-axis is (-A1,-B1,-C1)). We can then obtain the x-axis (vector) of the screen coordinate system: y×z (the product of the y-axis vector and the z-axis vector).

[0068] Meanwhile, the direction p1(0,0,1) pointed to by the remote control is represented as q1(p1·x, p1·y, p1·z) in the screen coordinate system, and this vector q2 is the target indication direction.

[0069] S130. Adjust the displayed image according to the relative position information between each element in the displayed image, the target indication position information, and the target indication direction to obtain the output image.

[0070] The displayed image may include multiple elements. The relative position information of each element in the three-dimensional space represented by the image is used as the relative position information between the elements in the displayed image. For example, if the displayed image includes a car and a person, and the car is behind the person, then the relative position information between the elements is that the car is 2m behind the person, and the angle between the car and the person is 100 degrees.

[0071] In one implementation, the target element indicated by the remote control in the displayed image can be determined by the relative position information between each element in the displayed image, the target indication position information, and the target indication direction, and the display image can be controlled to display an output mark at the location of the target element to obtain the output image.

[0072] In another embodiment, the target can be indicated to determine the actual observation angle of each element in the displayed image. Based on the relative position information between the elements in the displayed image, the observation angle of the displayed image is adjusted to the actual observation angle, and an output identifier is displayed at the position corresponding to the target indicated position information in the image after the observation angle is adjusted, thus obtaining the output image.

[0073] S140, Output image.

[0074] After obtaining the output image, the image display device can be controlled to output the image for easy and intuitive observation.

[0075] As one implementation, after S140, the method may further include: controlling the output image to change in response to an instruction event for the output identifier.

[0076] Command events can refer to commands sent to the location indicated by the remote control (the location of the output identifier mentioned above). Command events can include selection, deselection, drawing trajectory, editing, deletion, retraction, clearing, restoration, and viewing attributes.

[0077] The remote control can be equipped with physical or virtual buttons. Buttons can be explicit, always displayed on the image; or implicit, activated through a specific method, such as a laser pausing at a specific position for a certain time, or a special gesture design. Multiple buttons can be included, each corresponding to a single command event.

[0078] Upon receiving a command event, the system performs a corresponding operation at the location indicated by the output identifier to change the output image. For example, if the command event is to view attributes, the system views the attributes at the location indicated by the output identifier based on the command event.

[0079] In this embodiment, the target indication direction is determined based on the target pose information, and the display image is adjusted based on the relative position information between each element in the display image, the target indication position information, and the target indication direction to obtain an output image. The output image can change as the target indication direction, the target indication position information, and the relative position information between each element in the display image change, resulting in a high degree of diversity in the output image, thereby improving the diversity of interaction between the user and the output image and the user experience.

[0080] Reference Figure 6 , Figure 6 This application illustrates a flowchart of an image control method according to another embodiment. The method is used in an image control device, which may be... Figure 1 In section 104, the method may include:

[0081] S210. If the remote controller does not transmit an indication marker, acquire the first indication position information, the first position information, and the position information of at least three first marker points in the displayed image.

[0082] The first indication position information refers to the position information of the indication transmitted by the remote control in the display image when the remote control last transmitted the indication. The first indication position information is the position information in the image coordinate system where the display image is located. The first pose information refers to the pose information of the remote control when the remote control last transmitted the indication.

[0083] The first marker point can refer to the point determined by the remote control from the displayed image at the current moment, and the position information of the first marker point can be obtained. The position information of the first marker point can be the position information in the remote control coordinate system.

[0084] In this embodiment, the remote controller includes at least three Time-of-Flight (TOF) ranging units with different transmission angles. At the current moment, the ranging signals emitted by the at least three TOF ranging units are used as the target point on the screen of the image display device as the first marker point. At this time, the method for obtaining the position information of the at least three first marker points includes: controlling each TOF ranging unit to send a ranging signal to the first marker point corresponding to the TOF ranging unit, and controlling each TOF ranging unit to receive a return signal returned by the ranging signal sent by the TOF ranging unit; determining the position information of the first marker point corresponding to each TOF ranging unit based on the ranging signal and the return signal corresponding to each TOF ranging unit.

[0085] The relative positions and relative angles between the multiple TOF ranging units included in the remote controller can remain constant. However, the actual pose of the remote controller may differ at different times, resulting in different first marker points and their position information for each TOF ranging unit.

[0086] Optionally, the method for obtaining the first indication position information includes: determining a target distance based on the position information of at least three second marker points, wherein the target distance is the actual distance between the remote controller and the displayed image when the remote controller last transmitted the indication marker; determining a target indication direction of the remote controller based on the transmission direction of the remote controller last transmitted the indication marker, wherein the target indication direction is the indication direction in the remote controller coordinate system; determining the indication direction of the remote controller in the target coordinate system based on the target indication direction and the target coordinate system, as the final indication direction; and determining the first indication position information based on the target distance and the indication direction.

[0087] The method for obtaining the target coordinate system includes: obtaining the position information of at least three second marker points in the displayed image, wherein the at least three second marker points are determined when the remote controller transmits an indication mark, and the position information of the at least three second marker points is the coordinates in the remote controller coordinate system in which the remote controller is located; and determining the target coordinate system based on the position information of the at least three second marker points and the first pose information.

[0088] In the remote control coordinate system, the spatial coordinates Pi(xi,yi,zi), i=4,5,6 of three second marker points on the screen of the image display device are obtained through the TOF ranging unit. These three second marker points determine the plane in which the screen of the image display device is located. The plane can be represented as A2x+B2y+C2z+D2=0, which can be solved as A2=(y6-y5)*(z7-z5)-(y7-y5)*(z6-z5), B2=(z6-z5)*(x7-x5)-(z7-z5)*(x6-x5), C2=(x6-x5)*(y7-y5)-(x7-x5)*(y7-y5)*(x6-x5), C2=(x6-x5)*(y7-y5)-(x7-x5)*(y7-y5)*(x6-x5). 5)*(y6-y5), D2=-(A2x5+B2y5+C2z5), the gravity direction G2(x8,y8,z8) can be obtained from the inertial navigation unit. The screen of the image display device is parallel to the gravity direction, so G2 can be regarded as the y-axis (vector) of the screen coordinate system. Then the z-axis (vector) of the screen coordinate system is represented as (A2,B2,C2) (since the z-axis points vertically to the screen, C2 needs to be greater than 0. If C2<0, then the z-axis is (-A2,-B2,-C2)). Then the x-axis (vector) of the screen coordinate system can be obtained: y×z (the product of the vector of the y-axis and the vector of the z-axis).

[0089] Based on the above A2x+B2y+C2z+D2=0, the target distance is determined to be... The remote control is pointing towards a target in the direction p2(0,0,1). The final pointing direction in the target coordinate system is q2 = (p2·x, p2·y, p2·z), where x, y, and z are the directions of the three axes of the target coordinate system, respectively. Since the origin of the target coordinate system is the remote control, the pointing position information of the remote control in the screen coordinate system (i.e., the target coordinate system) where the image display device's screen is located is represented as follows: This indicates location information. This is the fourth indication location information.

[0090] Based on the first mapping parameter and the fourth indication position information, the first indication position information is determined. The first mapping parameter represents the correspondence between the actual size of the displayed image and the displayed size of the displayed image in the image coordinate system. Using the fourth indication position information and the first mapping parameter, the position point of the fourth indication position information in the image coordinate system is determined, and the position information of this position point is used as the first indication position information.

[0091] S220. Determine the first position change information based on the first pose information and the position information of at least three first marker points.

[0092] After obtaining the first pose information, the fifth indication position information is determined based on the position information of at least three first marker points. The process of determining the fifth indication position information is the same as that of determining the fourth indication position information, and will not be repeated here.

[0093] Based on the fourth and fifth indicated position information, the relative position information of the fourth and fifth indicated position information is determined as the first position change information, which is the position change information in the target coordinate system.

[0094] S230. Determine the second position change information based on the first mapping parameters and the first position change information.

[0095] After obtaining the first mapping parameters, the first position change information in the target coordinate system is converted into the position change information in the image coordinate system, which is then used as the second position change information.

[0096] The actual size of a displayed image can refer to the size of the image displayed on the screen of an image display device. This can be the screen size itself, or the actual size in the target coordinate system, expressed in meters (m). For example, an actual size of 1.5m × 1.2m can be used. The display size can also refer to the size of the displayed image in image coordinates. For example, a display size of 256 × 256 indicates 256 × 256 pixels.

[0097] The first mapping parameter can be determined based on the actual size of the displayed image and the display size, and the first mapping parameter can be used to characterize the correspondence between the actual size of the displayed image and the display size.

[0098] Optionally, the method for obtaining the true size of the displayed image includes: obtaining second and third indication position information, wherein the second indication position information is the coordinates of the indicator in the target coordinate system when the remote controller sends an indication signal to a third marker point of the displayed image at a first angle from the target position, and the third indication position information is the coordinates of the indicator in the target coordinate system when the remote controller sends an indication signal to a fourth marker point of the displayed image at a second angle from the target position; determining the relative true distance between the third and fourth marker points based on the second and third indication position information; determining the target scale relationship based on the display size of the displayed image and the relative pixel distance, wherein the relative pixel distance is the relative distance between the third and fourth marker points in the image coordinate system; and determining the true size of the displayed image based on the relative true distance and the target scale relationship. The target position can be any position, and this application does not limit it. The first angle and the second angle are different, the second and third indication position information are different, and the third and fourth marker points can be corner points of the displayed image (e.g., opposite corner points or adjacent corner points of a rectangular displayed image).

[0099] like Figure 7 As shown, the remote control emits indicator signals at different angles from the same location (target location) to the third marker point 703 and the fourth marker point 704 of the display image 702 (solid lines indicate that the remote control 701 emits the indicator signal at the first angle, and dashed lines indicate that the remote control 701 emits the indicator signal at the second angle). The third marker point 703 corresponds to the second indicator position information, and the fourth marker point corresponds to the third indicator position information. The relative actual distance between the third marker point and the fourth marker point is the actual width of the display image.

[0100] The determination methods for the second and third indication location information are the same as those for the determination method for the fourth indication location information, and will not be repeated here.

[0101] The relative pixel distance between the third and fourth marker points can be expressed as a combination of horizontal and vertical pixel distances. For example, a relative pixel distance of (120, 320) indicates that the relative pixel distance includes a horizontal pixel distance of 120 and a vertical pixel distance of 320. The target scaling relationship can include both horizontal and vertical scaling relationships. For example, a target scaling relationship of (20, 30) indicates that the width of the display size is 20 times the horizontal pixel distance in the relative pixel distance, and the height of the display size is 30 times the vertical pixel distance in the relative pixel distance.

[0102] The actual horizontal and vertical distances can be determined based on the relative actual distances. The actual width and height can be determined based on the target proportions. For example, if the actual horizontal distance is 20cm, the actual vertical distance is 10cm, and the target proportions are (20, 30), the actual width is determined to be 20×20=400cm and the height is determined to be 30×10=300cm.

[0103] In one implementation, two corner points representing the width (or height) of the displayed image can be taken as the third and fourth marker points, respectively. The relative distance between them is the true width (or true height). Based on the aspect ratio of the displayed image, the true size can be determined.

[0104] S240. Determine the target indication position information based on the first indication position information and the second position change information.

[0105] Starting from the first indicated position information and considering the second position change information as the change, the changed position information is determined as the target indicated position information. The second position change information may include the lateral movement distance (which can be represented by the number of pixels, with positive values ​​indicating leftward movement and negative values ​​indicating rightward movement) and the vertical movement distance (which can be represented by the number of pixels, with positive values ​​indicating upward movement and negative values ​​indicating downward movement).

[0106] S250. Determine the target pose information; based on the target pose information, determine the remote control's pointing direction as the target pointing direction.

[0107] S260. Based on the relative position information between elements in the displayed image, the target indication position information, and the target indication direction, adjust the displayed image to obtain an output image with an output identifier; output the output image.

[0108] The descriptions of S250-S260 are the same as those of S110-S140 above, and will not be repeated here.

[0109] In this embodiment, even when the remote control does not transmit an indicator, it can still obtain target indication position information with high accuracy. This makes the output image obtained based on the displayed image more closely match the pose information of the remote control, improving the accuracy of the output image and thus enhancing the interaction accuracy and user experience. Simultaneously, the remote control can achieve accurate interaction without transmitting an indicator, saving energy consumption.

[0110] Reference Figure 8 , Figure 8 This illustration shows a flowchart of an image control method according to another embodiment of the present application. The method is used in an image control device, which may be... Figure 1 In section 104, the method may include:

[0111] S310. Determine the target indication position information and target pose information; based on the target pose information, determine the remote control indication direction as the target indication direction.

[0112] The description of S310 is the same as that of S110-S120 above, and will not be repeated here.

[0113] S320. Based on the target indication position information, the target indication direction, and the relative position information, determine the first adjustment indication position information corresponding to the target indication position information in the display image, and add a first output identifier in the display image at the position corresponding to the first adjustment indication position information to obtain the output image.

[0114] Based on the target indication direction and target indication position information, a first indicator line of the remote control in the three-dimensional space represented by the displayed image can be determined. Based on the first indicator line and the relative position information between various elements, a first target element indicated by the remote control in the displayed image can be determined. The position where the first target element intersects with the first indicator line is taken as the first target position. The position information of the first target position (in the image coordinate system) is taken as the first adjustment indication position information. A first output identifier is added to the first target position represented by the first adjustment indication position information. The style of the first output identifier can be referred to the description of S110 above, and will not be repeated here.

[0115] like Figure 9 As shown, the remote controller 901 transmits an indicator to the display image. The first indicator line corresponding to the indicator is 902 (that is, the target indicator direction). The target indicator position information corresponding to the indicator represents the position of the indicator 903. The first adjustment indicator position information corresponding to the indicator represents the position of the first output indicator 904.

[0116] S330. Based on the target indication direction, determine the first observation angle for the displayed image. Based on the relative position information, adjust the observation angle of each element to the first observation angle to obtain the output image.

[0117] Based on the indicated direction, the current observation angle of the displayed image can be determined as the first observation angle. Then, based on the relative position information of each element, the observation angle of each element can be adjusted to the first observation angle, and the adjusted image can be used as the output image.

[0118] As shown in Figure 10, a in Figure 10 shows the original observation angle of each element in the displayed image. The remote controller 1001 transmits an indicator 1003 to the displayed image in the target direction 1002. According to the target direction, the observation angle is determined as the first observation angle. b in Figure 10 is a schematic diagram after the observation angle of each element in the displayed image is adjusted to the first observation angle.

[0119] S340. Based on the target indication direction, determine the second observation angle for the displayed image. Based on the relative position information, adjust the observation angle of each element to the second observation angle to obtain an intermediate image. Based on the target indication position information, the target indication direction, and the relative position information, determine the second adjustment indication position information corresponding to the target indication position information in the intermediate image. Add a second output identifier to the position corresponding to the second adjustment indication position information in the intermediate image to obtain an output image.

[0120] First, adjust the observation angle to a second observation angle based on the target indication direction to obtain an intermediate image. Then, based on the target indication direction and position information, determine the second indicator line of the remote control in the three-dimensional space represented by the intermediate image. Based on this second indicator line and the relative position information between various elements, determine the second target element indicated by the remote control in the intermediate image. The position where the second target element intersects with the second indicator line is taken as the second target position. The position information of the second target position (in the image coordinate system) is taken as the second adjustment indicator position information. A second output identifier is added to the second target position represented by the second adjustment indicator position information. The style of the second output identifier can be referred to in the description of S110 above, and will not be repeated here.

[0121] like Figure 11a The original observation angles of each element in the displayed image are shown. The remote control 1101 transmits an indicator 1103 to the displayed image in the target direction 1102 (i.e., the direction of the second indicator line). Based on the target direction, the observation angle is determined as the second observation angle. The original observation angles of each element in the displayed image are then adjusted to the second observation angle to obtain an intermediate image, as shown below. Figure 11b As shown, the second indication location information is further determined, and a second output identifier is added at the second indication location information to obtain the output image, as shown. Figure 11c As shown, the second output identifier is 1004.

[0122] S350, output image.

[0123] The description of S350 is the same as that of S140 above, and will not be repeated here.

[0124] In this embodiment, the displayed image is adjusted using different methods to make the adjusted output image more closely match the pose information of the remote control, resulting in a more accurate output image and thus improving the interaction accuracy and user experience.

[0125] Reference Figure 12 , Figure 12 A block diagram of an image control device according to an embodiment of this application is shown. The device 1100 includes:

[0126] The information determination module 1110 is used to determine the target indication position information and the target pose information. The target indication position information refers to the indication position information of the remote control for the displayed image, and the target pose information refers to the pose information of the remote control. The remote control is used to indicate the displayed image.

[0127] The direction determination module 1120 is used to determine the direction indicated by the remote control based on the target pose information, and use it as the target indication direction;

[0128] The adjustment module 1130 is used to adjust the displayed image according to the relative position information between each element in the displayed image, the target indication position information, and the target indication direction to obtain the output image;

[0129] Output module 1140 is used to output images.

[0130] Optionally, the information acquisition module 1110 is further configured to, if the remote controller does not transmit an indicator, acquire first indicator position information, first pose information, and position information of at least three first marker points in the displayed image. The first indicator position information refers to the position information of the indicator transmitted by the remote controller in the displayed image when the remote controller last transmitted the indicator. The first indicator position information is the position information in the image coordinate system where the displayed image is located. The first pose information refers to the pose information of the remote controller when the remote controller last transmitted the indicator. Based on the first pose information and the position information of the at least three first marker points, determine first position change information. The first position change information is the position change information in the target coordinate system. Based on the first mapping parameter and the first position change information, determine second position change information. The first mapping parameter represents the correspondence between the actual size of the displayed image and the display size of the displayed image in the image coordinate system. Based on the first indicator position information and the second position change information, determine target indicator position information.

[0131] Optionally, the information acquisition module 1110 is further configured to acquire the position information of at least three second marker points in the displayed image, wherein the at least three second marker points are determined when the remote controller transmits an indication mark, and the position information of the at least three second marker points is the coordinates in the remote controller coordinate system in which the remote controller is located; and determine the target coordinate system based on the position information of the at least three second marker points and the first pose information.

[0132] Optionally, the information acquisition module 1110 is further configured to: determine the target distance based on the position information of at least three second marker points, wherein the target distance is the actual distance between the remote controller and the displayed image when the remote controller last transmitted the indicator; determine the target indication direction of the remote controller based on the transmission direction of the indicator last transmitted by the remote controller, wherein the target indication direction is the indication direction in the remote controller coordinate system; determine the indication direction of the remote controller in the target coordinate system based on the target indication direction and the target coordinate system, as the final indication direction; and determine the first indication position information based on the target distance and the indication direction.

[0133] Optionally, the information acquisition module 1110 is further configured to acquire second indication position information and third indication position information. The second indication position information is the coordinates of the indicator in the target coordinate system when the remote controller transmits the indication indicator to the third marker point of the displayed image at a first angle from the target position. The third indication position information is the coordinates of the indicator in the target coordinate system when the remote controller transmits the indication indicator to the fourth marker point of the displayed image at a second angle from the target position. Based on the second and third indication position information, the relative true distance between the third and fourth marker points is determined. Based on the display size of the displayed image and the relative pixel distance, the target scale relationship is determined. The relative pixel distance is the relative distance between the third and fourth marker points in the image coordinate system. Based on the relative true distance and the target scale relationship, the true size of the displayed image is determined.

[0134] Optionally, the remote controller includes at least three TOF ranging units with different transmission angles; the information acquisition module 1110 is further configured to control each TOF ranging unit to send a ranging signal to a first marker corresponding to the TOF ranging unit, and to control each TOF ranging unit to receive a return signal returned by the ranging signal sent by the TOF ranging unit; and to determine the position information of the first marker corresponding to each TOF ranging unit based on the ranging signal and the return signal corresponding to each TOF ranging unit.

[0135] Optionally, the information acquisition module 1110 is further configured to acquire a captured image of the corresponding display image if the remote controller transmits an indicator, the captured image including the captured indicator of the corresponding indicator; determine the position information of the captured indicator in the captured image as captured position information; and determine the target indicator position information as indicator position information based on the second mapping parameter and the captured position information, the second mapping parameter being used to characterize the mapping relationship between the captured image and the display image.

[0136] Optionally, the adjustment module 1130 is further configured to: determine first adjustment indication position information corresponding to the target indication position information in the displayed image based on the target indication position information, the target indication direction, and the relative position information; and add a first output identifier in the displayed image at the position corresponding to the first adjustment indication position information to obtain an output image; or, determine a first observation angle for the displayed image based on the target indication direction; and adjust the observation angle of each element to the first observation angle based on the relative position information to obtain an output image; or, determine a second observation angle for the displayed image based on the target indication direction; and adjust the observation angle of each element to the second observation angle based on the relative position information to obtain an intermediate image; and determine a second adjustment indication position information corresponding to the target indication position information in the intermediate image based on the target indication position information, the target indication direction, and the relative position information; and add a second output identifier in the intermediate image at the position corresponding to the second adjustment indication position information to obtain an output image.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] In the several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or other forms.

[0139] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0140] Reference Figure 13 , Figure 13 A block diagram of an image control device according to another embodiment of this application is shown. The image control device 1200 may include one or more components such as a processor 1210, a memory 1220, and one or more application programs, wherein the one or more application programs may be stored in the memory 1220 and configured to be executed by one or more processors 1210, and the one or more programs are configured to perform the methods as described in the foregoing method embodiments.

[0141] Processor 1210 may include one or more processing cores. Processor 1210 connects to various parts within the image control device 1200 using various interfaces and lines, and performs various functions and processes data of the image control device 1200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 1220, and by calling data stored in memory 1220. Optionally, processor 1210 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1210 and may be implemented separately using a communication chip.

[0142] The memory 1220 may include random access memory (RAM) or read-only memory (ROM). The memory 1220 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1220 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the terminal 1200 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0143] Please refer to Figure 14 , Figure 14 A structural block diagram of a computer-readable storage medium according to an embodiment of this application is shown. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0144] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image control method, characterized in that, The method includes: Determine the target indication position information and the target pose information. The target indication position information refers to the indication position information of the remote controller in relation to the displayed image, and the target pose information refers to the pose information of the remote controller. The remote controller is used to indicate the displayed image. The target indication direction is determined based on the target pose information and the coordinate information of the remote controller; Based on the relative position information between elements in the displayed image, the target indication position information, and the target indication direction, the target element indicated by the remote control in the displayed image is determined, and the displayed image is controlled to display an output identifier at the location of the target element to obtain an output image; or, Using the target indication direction, determine the actual observation angle of each element in the displayed image. Based on the relative position information, adjust the observation angle of each element in the displayed image to the corresponding actual observation angle. Then, display an output marker at the position corresponding to the target indication position information in the adjusted display image to obtain the output image; or, Based on the target indication location information, the target indication direction and the relative position information, a first adjustment indication location information corresponding to the target indication location information is determined in the display image, and a first output identifier is added to the position in the display image corresponding to the first adjustment indication location information to obtain an output image; Output the output image.

2. The method according to claim 1, characterized in that, The method for obtaining the target indication location information includes: If the remote controller does not transmit an indicator, the first indicator position information, the first position information, and the position information of at least three first marker points in the display image are obtained. The first indicator position information refers to the position information of the indicator transmitted by the remote controller in the display image when the remote controller last transmitted the indicator. The first indicator position information is the position information in the image coordinate system where the display image is located. The first position information refers to the position information of the remote controller when the remote controller last transmitted the indicator. Based on the first pose information and the position information of each of the at least three first marker points, the first position change information is determined, wherein the first position change information is the position change information in the target coordinate system; Based on the first mapping parameter and the first position change information, the second position change information is determined. The first mapping parameter represents the correspondence between the actual size of the displayed image and the display size of the displayed image in the image coordinate system. The target indication location information is determined based on the first indication location information and the second location change information.

3. The method according to claim 2, characterized in that, The method for obtaining the target coordinate system includes: The position information of at least three second marker points in the displayed image is obtained. The at least three second marker points are determined when the remote controller is used to transmit an indication mark. The position information of the at least three second marker points is the coordinates in the remote controller coordinate system in which the remote controller is located. The target coordinate system is determined based on the position information of each of the at least three second marker points and the first pose information.

4. The method according to claim 3, characterized in that, The method for obtaining the first indicated location information includes: The target distance is determined based on the position information of each of the at least three second marker points. The target distance is the actual distance between the remote controller and the displayed image when the remote controller last transmitted the indicator. The target indication direction of the remote controller is determined based on the transmission direction of the previous transmission indication indicator of the remote controller, and the target indication direction is the indication direction in the coordinate system of the remote controller; Based on the target indication direction and the target coordinate system, the indicating direction of the remote control in the target coordinate system is determined as the final indication direction; The first indicated location information is determined based on the target distance and the indicated direction.

5. The method according to claim 2, characterized in that, The method for obtaining the true size of the displayed image includes: Acquire second and third indication position information. The second indication position information is the coordinates of the indication mark in the target coordinate system when the remote controller sends an indication mark to the third marker point of the displayed image at the target position at a first angle. The third indication position information is the coordinates of the indication mark in the target coordinate system when the remote controller sends an indication mark to the fourth marker point of the displayed image at the target position at a second angle. Based on the second indicated location information and the third indicated location information, determine the relative true distance between the third marker point and the fourth marker point; The target proportional relationship is determined based on the display size of the displayed image and the relative pixel distance, wherein the relative pixel distance is the relative distance between the third marker point and the fourth marker point in the image coordinate system; The true size of the displayed image is determined based on the relative true distance and the target scale relationship.

6. The method according to claim 2, characterized in that, The remote controller includes at least three Time-of-Flight (TOF) ranging units with different emission angles; the method for acquiring the position information of each of the at least three first marker points includes: Each TOF ranging unit is controlled to send a ranging signal to the first marker point corresponding to the TOF ranging unit, and each TOF ranging unit is controlled to receive a return signal returned by the ranging signal sent to it. Based on the ranging signal and return signal corresponding to each TOF ranging unit, the position information of the first marker point corresponding to each TOF ranging unit is determined.

7. The method according to claim 1, characterized in that, The acquisition of the target location information includes: If the remote control emits an indicator, a captured image corresponding to the displayed image is obtained, and the captured image includes the captured indicator corresponding to the indicator; The position information of the shooting identifier in the captured image is determined and used as the shooting position information; Based on the second mapping parameter and the shooting location information, the target indication location information is determined as the indication location information. The second mapping parameter is used to characterize the mapping relationship between the captured image and the displayed image.

8. The method according to claim 1, characterized in that, The method further includes: Based on the target indication direction, a first observation angle for the displayed image is determined. Based on the relative position information, the observation angles of each element are adjusted to the first observation angle to obtain the output image; or, Based on the target indication direction, a second observation angle for the displayed image is determined. Based on the relative position information, the observation angle of each element is adjusted to the second observation angle to obtain an intermediate image. Based on the target indication position information, the target indication direction, and the relative position information, a second adjustment indication position information corresponding to the target indication position information is determined in the intermediate image. A second output identifier is added to the position corresponding to the second adjustment indication position information in the intermediate image to obtain an output image.

9. An image control device, characterized in that, The device includes: The information determination module is used to determine target indication position information and target pose information. The target indication position information refers to the indication position information of the remote controller for the displayed image, and the target pose information refers to the pose information of the remote controller. The remote controller is used to indicate the displayed image. The direction determination module is used to determine the direction indicated by the remote control based on the target pose information, and use it as the target indication direction; An adjustment module is configured to: determine the target element indicated by the remote control in the displayed image based on the relative position information between elements in the displayed image, the target indication position information, and the target indication direction; and control the displayed image to display an output identifier at the location of the target element to obtain an output image; or, determine the actual observation angle of each element in the displayed image based on the target indication direction, adjust the observation angle of each element in the displayed image to the corresponding actual observation angle based on the relative position information, and display an output identifier at the position corresponding to the target indication position information in the displayed image after the observation angle adjustment to obtain an output image; or, determine a first adjustment indication position information corresponding to the target indication position information in the displayed image based on the target indication position information, the target indication direction, and the relative position information, and add a first output identifier at the position corresponding to the first adjustment indication position information in the displayed image to obtain an output image. The output module is used to output the output image.

10. An image control device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-8.

11. An image control system, characterized in that, The image control system includes at least an image display device, a remote controller, and an image control device, wherein... The image display device is used to display images; The remote control is used to instruct the displayed image; The image control device is used to perform the method as described in any one of claims 1-8.

Images