Mobile terminal-based control data acquisition method, device, medium and equipment

Abstract

The application provides a mobile terminal-based control data acquisition method, device, medium and equipment. The acquisition method is applied to a human-computer interaction device and includes the following steps: acquiring real-time images of a location where a mobile terminal is located and inertial measurement information of the mobile terminal; the mobile terminal displays a preset feature map; screen position information of the mobile terminal and a feature map image are identified from the real-time images; posture information of the mobile terminal is obtained according to the feature map image; and control data of the human-computer interaction device is obtained according to the inertial measurement information, the posture information and the screen position information. The application acquires control data through a mobile terminal to replace a controller for controlling a human-computer interaction device, and can reduce the production cost of a human-computer interaction device based on a controller.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction device control technology, specifically to a method, apparatus, medium, and device for acquiring control data based on a mobile terminal. Background Technology

[0002] Human-computer interaction (HCI) devices are devices that allow users to communicate with and control a system through a controller. Common controllers include mice, keyboards, and gamepads. Among these, gamepads include 6DOF gamepads, which are controllers that obtain six degrees of freedom (6DoF) attitude data based on the gamepad's state. The 6DOF gamepad transmits this six DoF attitude data to the system to communicate and control it. However, 6DOF gamepads are often used with HCI devices that require six DoF attitude data for control. However, existing 6DOF gamepads have extremely high production costs, which significantly increases the production costs of HCI devices that require six DoF attitude data for control. Summary of the Invention

[0003] The purpose of this application is to overcome the shortcomings and deficiencies in the prior art and provide a method, apparatus, medium and device for acquiring control data based on a mobile terminal. By acquiring control data through a mobile terminal to replace the controller for controlling human-computer interaction devices, the production cost of human-computer interaction devices based on controllers can be reduced.

[0004] The first aspect of this application provides a method for acquiring control data based on a mobile terminal, applied to a human-computer interaction device, the method comprising:

[0005] The system acquires real-time images of the mobile terminal's location and the mobile terminal's inertial measurement information; the mobile terminal displays a preset feature map.

[0006] The screen position information and feature map image of the mobile terminal are identified from the real-time image;

[0007] The attitude information of the mobile terminal is obtained based on the feature map image;

[0008] Based on the inertial measurement information, the attitude information, and the screen position information, control data for the human-computer interaction device is obtained.

[0009] A second aspect of this application provides a control data acquisition device based on a mobile terminal, applied to a human-computer interaction device, the device comprising:

[0010] The real-time image acquisition module acquires real-time images of the location of the mobile terminal and the inertial measurement information of the mobile terminal; the mobile terminal displays a preset feature map.

[0011] A real-time image recognition module is used to identify the screen position information and feature map image of the mobile terminal from the real-time image;

[0012] The posture information acquisition module is used to obtain the posture information of the mobile terminal based on the feature map image;

[0013] The control data acquisition module is used to obtain control data for the human-computer interaction device based on the inertial measurement information, the attitude information, and the screen position information.

[0014] A third aspect of this application provides a computer-readable storage medium storing a computer program, characterized in that: when the computer program is executed by a processor, it implements the steps of the mobile terminal-based control data acquisition method as described above.

[0015] A fourth aspect of this application provides an electronic device including a storage device, a processor, and a computer program stored in the storage device and executable by the processor, wherein the processor executes the computer program to implement the steps of the mobile terminal-based control data acquisition method as described above.

[0016] Compared to related technologies, this application obtains the control data of the human-computer interaction device by using the feature map displayed on the mobile terminal and the inertial measurement information of the mobile terminal. This allows users to control the human-computer interaction device through the control data, eliminating the need to produce a remote control device to provide control data during the manufacturing process of the human-computer interaction device, thereby reducing the production cost of human-computer interaction devices based on controllers.

[0017] To provide a clearer understanding of this application, the specific embodiments of this application will be described below in conjunction with the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for acquiring control data based on a mobile terminal, according to an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the module connections of a mobile terminal-based control data acquisition device according to an embodiment of this application.

[0020] Figure 3 An electronic device according to one embodiment of this application.

[0021] 100. A control data acquisition device based on a mobile terminal; 101. A real-time image acquisition module; 102. A real-time image recognition module; 103. A posture information acquisition module; 104. A control data acquisition module. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0023] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0024] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The singular forms "a," "the," and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The word "if" as used herein can be interpreted as "when," "when," or "in response to determination."

[0025] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0026] Please see Figure 1 This is a flowchart of a mobile terminal-based control data acquisition method according to an embodiment of this application. The mobile terminal-based control data acquisition method is applied to a human-computer interaction (HCI) device. Here, an HCI device refers to a device that communicates with a system and performs control through a controller. In this embodiment, an HCI device specifically refers to a device that achieves control based on six degrees of freedom (DOF) posture data, such as AR (Augmented Reality) devices, VR (Virtual Reality) devices, MR (Mixed Reality) devices, and XR (Extended Reality) devices. The HCI device particularly includes at least a display device and a host computer. The display device can display the virtual image processed by the host computer. Preferably, the HCI device is an all-in-one device integrating the display device and the host computer, such as an AR all-in-one device or a VR all-in-one device.

[0027] The method for acquiring control data based on a mobile terminal includes:

[0028] S1: Acquire real-time images of the location of the mobile terminal and the inertial measurement information of the mobile terminal; the mobile terminal displays a preset feature map.

[0029] A mobile terminal refers to a handheld electronic device with a screen display and an inertial measurement unit, such as a mobile phone or a smart tablet. The mobile terminal connects to a human-computer interaction device via a signal connection to enable data or signal transmission between them. This signal connection can be wired or wireless, such as Bluetooth, Wi-Fi, or radio frequency (RF) connections.

[0030] Real-time images of the mobile terminal's location refer to images captured by the mobile terminal and surrounding objects, which can be obtained through a camera. This camera is connected to a human-computer interaction (HCI) device to transmit the acquired real-time images. Preferably, when the HCI device is an all-in-one device, the camera is mounted on its outer surface, and the camera's shooting direction is the same as the user's line of sight, allowing the camera to directly acquire real-time images based on the user's gaze, saving the user the effort of adjusting the camera.

[0031] At least one state of the feature map is an image displaying multiple feature graphics. Preferably, the multiple feature graphics displayed in the feature map are asymmetrically distributed. The asymmetrical distribution includes vertical asymmetry, horizontal asymmetry, and central asymmetry. The asymmetrical distribution of feature graphics can better reflect the changes in the screen orientation of the mobile terminal.

[0032] The inertial measurement information of a mobile terminal is acquired through an inertial measurement unit (IMU) installed within the terminal. This IMU includes inertial rotation and displacement information. Typically, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the mobile terminal along three independent axes of the carrier coordinate system, while the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system. By measuring the angular velocity and acceleration of the mobile terminal in three-dimensional space, the inertial rotation and displacement information are calculated. Specifically, integrating the angular velocity in three-dimensional space yields the inertial rotation information, and integrating the acceleration in three-dimensional space yields the inertial displacement information.

[0033] S2: Identify the screen position information and feature map image of the mobile terminal from the real-time image.

[0034] Since the screen of a mobile terminal displays a feature map, the screen location information and the feature map image of the mobile terminal can be obtained by identifying the position of the feature map in the real-time image.

[0035] Screen position information refers to the position and range of the mobile terminal's screen in a real-time image; while feature map image refers to the image when the feature map is displayed on the mobile terminal's screen. Specifically, it is obtained by acquiring the image of the mobile terminal's screen from the real-time image based on the position and range of the mobile terminal's screen in the real-time image, thereby obtaining the image of the feature map displayed on the screen.

[0036] S3: Obtain the attitude information of the mobile terminal based on the feature map image.

[0037] The attitude information of a mobile terminal refers to the orientation or rotation direction of the mobile terminal's screen in space.

[0038] Since at least one state of the feature map is an image displaying multiple feature graphics, the orientation of the feature map can be calculated based on changes in multiple information parameters such as the distribution position, distance, and relative size of each feature graphic, thereby obtaining the attitude information of the mobile terminal. For example, when the screen of the mobile terminal is tilted upwards, the distance between the upper and lower feature graphics in the feature map image shrinks, and the relative size of the upper feature graphic also becomes smaller than the relative size of the lower feature graphic. Based on the above-mentioned changing parameters, the result of the mobile terminal screen tilting upwards can be calculated, and accurate tilt information can also be calculated, thereby obtaining the attitude information of the mobile terminal.

[0039] S4: Based on the inertial measurement information, the attitude information, and the screen position information, obtain the control data for the human-computer interaction device.

[0040] The control data includes six degrees of freedom control data. Specifically, the control data includes the rotation and displacement information of the object on the three-dimensional coordinate axes in space. For example, the three-dimensional coordinate axes in space are represented by the X-axis, Y-axis and Z-axis. The six degrees of freedom control data includes six pieces of information, such as the rotation and displacement information of the object on the X-axis, the rotation and displacement information of the object on the Y-axis and the rotation and displacement information of the object on the Z-axis.

[0041] By weighting the inertial measurement information of the mobile terminal with its attitude information and screen position information, the accuracy of the final attitude information of the mobile terminal can be improved. Specifically, as mentioned above, since the attitude information of the mobile terminal reflects its orientation and the screen position information reflects its position, the attitude information and screen position information can be combined to form the first six degrees of freedom initial information. The inertial measurement rotation information and inertial measurement displacement information of the mobile terminal can be used as the second six degrees of freedom initial information. Then, by weighting the first six degrees of freedom initial information and the second six degrees of freedom initial information, the control data of the six degrees of freedom of the mobile terminal can be obtained.

[0042] Compared to related technologies, this application obtains the control data of the human-computer interaction device by using the feature map displayed on the mobile terminal and the inertial measurement information of the mobile terminal. This allows users to control the human-computer interaction device through the control data, eliminating the need to produce a remote control device to provide control data during the manufacturing process of the human-computer interaction device, thereby reducing the production cost of human-computer interaction devices based on controllers.

[0043] In one feasible embodiment, the human-computer interaction device is equipped with two positioning cameras;

[0044] Step S2: The step of identifying the screen position information and feature map image of the mobile terminal from the real-time image includes:

[0045] S201: Obtain the screen position information of the mobile terminal through the two positioning cameras.

[0046] Specifically, the mobile terminal is matched with two positioning cameras. After a successful match, the two positioning cameras and the mobile terminal form a triangle. Based on the principle of triangles, the horizontal and vertical distances between the mobile terminal and the two positioning cameras can be calculated using formulas, thereby obtaining the screen position information of the mobile terminal.

[0047] S202: Based on the screen position information, construct a tracking frame for the screen of the mobile terminal; the tracking frame changes as the screen of the mobile terminal changes.

[0048] The tracking frame of a mobile terminal's screen is generated based on the screen's position information. It can maintain tracking of the screen's position to avoid losing track of it.

[0049] S203: Based on the image within the tracking frame, obtain the feature map image displayed on the screen of the mobile terminal.

[0050] In this embodiment, the position of the mobile terminal's screen is acquired using two positioning cameras. Based on the position of the mobile terminal's screen, a feature map image of the screen display is obtained, thereby accurately acquiring the feature map image. In this embodiment, the feature map image is an image with multiple feature patterns and a solid-color background. Information such as the direction, distance, and relative size between the various feature patterns in the feature map image is used to calculate the mobile terminal's attitude information.

[0051] In one feasible embodiment, the feature map includes a first feature map and a second feature map, wherein the mobile terminal displays the first feature map first, and then displays the second feature map. Specifically, the first feature map and the second feature map can be understood as two display states of the feature map. Displaying the first feature map first and then the second feature map can be achieved by image replacement; alternatively, displaying the first feature map first and then the second feature map can be achieved by changing the fill color. For example, the first feature map may have multiple feature graphics and a solid color background, similar to the second feature map, but the feature graphics of the first feature map have the same color as the solid color background. By changing the color of the solid color background of the first feature map, the feature graphics become different from the solid color background, thus displaying the second feature map. The implementation method of the mobile terminal displaying the first feature map first and then the second feature map is not limited. It can be that a display time limit is set for the first feature map, the mobile terminal displays the first feature map first, and after the display time limit is exceeded, the mobile terminal displays the second feature map. Alternatively, the mobile terminal displays the first feature map first, and when the human-computer interaction device recognizes the screen position information of the mobile terminal, it sends a map change signal to the mobile terminal, and the mobile terminal responds to the map change signal to display the second feature map.

[0052] The first feature map is a solid color image, including but not limited to a pure white image, to facilitate the identification of the location and range of the first feature map; the second feature map is an image with multiple feature graphics and a solid color background, and the color of the feature graphics in the second feature map is different from the color of the solid color background, including but not limited to images with white feature graphics and black backgrounds, to facilitate the identification of each feature graphic in the second feature map.

[0053] Step S2: The step of identifying the screen position information and feature map image of the mobile terminal from the real-time image includes:

[0054] S211: When the mobile terminal displays the first feature map, the real-time image is identified by a preset first feature map recognition algorithm to identify the screen position information of the mobile terminal.

[0055] The preset first feature map recognition algorithm can be a deep learning algorithm or a color space recognition algorithm. By combining a solid color image with the preset first feature map recognition algorithm, the position and range of the first feature map can be accurately and efficiently identified from a real-time image, thereby obtaining the screen position information of the mobile terminal.

[0056] S212: Based on the screen position information, construct a tracking frame for the screen of the mobile terminal; the tracking frame changes as the screen of the mobile terminal changes.

[0057] The tracking frame of the mobile terminal screen is generated based on the screen position information of the mobile terminal. It can maintain the tracking of the mobile terminal screen position during the brief process of the screen changing from displaying the first feature map to displaying the second feature map, so as to avoid losing the position of the mobile terminal screen.

[0058] S213: When the mobile terminal displays the image of the second feature map, the feature map image is obtained based on the image within the tracking box.

[0059] In this embodiment, the position of the mobile terminal's screen is obtained through the first feature map, and then, based on the position of the mobile terminal's screen, the image of the second feature map is obtained when the mobile terminal displays the image of the second feature map, thereby accurately obtaining the feature map image.

[0060] In a feasible embodiment, step S3: obtaining the pose information of the mobile terminal based on the feature map image, includes:

[0061] S31: Perform feature pattern recognition on the image of the feature map to identify the graphic information of the second feature map.

[0062] The graphic information of the second feature map includes information such as the direction, distance and relative size between the various feature graphics of the second feature map.

[0063] S32: Calculate the posture information of the mobile terminal based on the graphic information.

[0064] In this embodiment, the attitude information of the mobile terminal can be obtained from the graphic information of the second feature map.

[0065] The attitude information of the mobile terminal is calculated based on the graphic information using the ICP algorithm (Iterative Closest Point). The ICP calculation method finds the corresponding point pairs between the source point cloud and the target point cloud, constructs a rotation and translation matrix based on the corresponding point pairs, and uses the obtained matrix to transform the information data of the mobile terminal in the real space to the coordinate system of the virtual space. If the value of the transformed error function is greater than the threshold, the above calculation is iterated until the preset error requirement is met.

[0066] In one feasible embodiment, the feature graphic is a circular graphic;

[0067] Step S31: The step of performing feature pattern recognition on the image of the feature map to identify the graphic information of the second feature map includes:

[0068] S3101: Perform circle detection on the image of the feature map to obtain the position, distance and relative size of each feature graphic in the second feature map.

[0069] Circle detection refers to the process of identifying circles in a target image using a circle detection algorithm to obtain information about each circle in the target image. In this example, the circle detection can also identify ellipses formed when a circular shape is tilted.

[0070] S3102: The position, distance, and relative size of each feature graphic are determined as the graphic information.

[0071] In this embodiment, through steps S3101-S3102, the circle detection algorithm can obtain graphic information such as the position, distance, and relative size of each circular graphic.

[0072] In one feasible embodiment, the feature graphic is a white graphic.

[0073] Step S31: The step of performing feature pattern recognition on the image of the feature map to identify the graphic information of the second feature map includes:

[0074] S3111: Perform white spot detection on the image of the feature map to obtain the position, distance and relative size of each feature graphic in the second feature map.

[0075] White spot detection refers to the process of identifying white shapes (white spots) in a target image using a white spot detection algorithm, in order to obtain information about each white shape in the target image.

[0076] S3112: The position, distance, and relative size of each feature graphic are determined as the graphic information.

[0077] In this embodiment, through steps S3111-S3112, the white spot detection algorithm can be used to obtain graphic information such as the position, distance, and relative size of each circular graphic.

[0078] In a feasible embodiment, step S4: obtaining control data for the human-computer interaction device based on the inertial measurement information, the attitude information, and the screen position information, includes:

[0079] S41: Perform extrinsic parameter calibration based on the screen position information and the inertial measurement information to obtain the extrinsic parameter information of the mobile terminal; the extrinsic parameter information is used to indicate the rotation direction and position information of the mobile terminal in the coordinate system of the virtual space constructed by the human-computer interaction device.

[0080] The principle of extrinsic parameter calibration based on screen position information and inertial measurement information is based on rotation matrices and translation vectors. Specifically, these rotation matrices and translation vectors represent the transformation relationship between the three-dimensional coordinate system established based on the inertial sensor and the three-dimensional coordinate system of the virtual space constructed based on the human-computer interaction device. By calculating the transformation relationship, the rotation direction and position information of the mobile terminal in the coordinate system of the virtual space can be calculated, thereby obtaining the extrinsic parameter information of the mobile terminal.

[0081] S42: Obtain control data for the human-computer interaction device based on the attitude information, the external parameter information, and the inertial measurement information.

[0082] In this embodiment, the control data is obtained based on the attitude information, extrinsic parameter information and inertial measurement information of the mobile terminal, which can improve the accuracy of the obtained six-degree-of-freedom control data.

[0083] In a feasible embodiment, step S42: obtaining control data for the human-computer interaction device based on the attitude information, the extrinsic parameter information, and the inertial measurement information, includes:

[0084] The attitude information, extrinsic parameter information, and inertial measurement information are fused and calculated using Kalman filtering or nonlinear optimization methods to obtain the six-degree-of-freedom attitude data of the mobile terminal; this six-degree-of-freedom attitude data of the mobile terminal is then used as the control data for the human-computer interaction device. Specifically, the fusion calculation method is not limited; for example, weighted calculation or averaging calculation can be used.

[0085] In this embodiment, the accuracy of the obtained six-degree-of-freedom attitude data can be improved by using Kalman filtering or nonlinear optimization methods.

[0086] Please see Figure 2This application also provides a mobile terminal-based control data acquisition device 100, applied to a human-computer interaction device, the device comprising:

[0087] The real-time image acquisition module 101 acquires real-time images of the location of the mobile terminal and the inertial measurement information of the mobile terminal; the mobile terminal displays a preset feature map.

[0088] The real-time image recognition module 102 is used to identify the screen position information and feature map image of the mobile terminal from the real-time image;

[0089] The posture information acquisition module 103 is used to obtain the posture information of the mobile terminal based on the feature map image;

[0090] The control data acquisition module 104 is used to obtain control data for the human-computer interaction device based on the inertial measurement information, the attitude information and the screen position information.

[0091] A mobile terminal refers to a handheld electronic device with a screen display and an inertial measurement unit, such as a mobile phone or a smart tablet. The mobile terminal connects to a human-computer interaction device via a signal connection to enable data or signal transmission between them. This signal connection can be wired or wireless, such as Bluetooth, Wi-Fi, or radio frequency (RF) connections.

[0092] Real-time images of the mobile terminal's location refer to images captured by the mobile terminal and surrounding objects, which can be obtained through a camera. This camera is connected to a human-computer interaction (HCI) device to transmit the acquired real-time images. Preferably, when the HCI device is an all-in-one device, the camera is mounted on its outer surface, and the camera's shooting direction is the same as the user's line of sight, allowing the camera to directly acquire real-time images based on the user's gaze, saving the user the effort of adjusting the camera.

[0093] At least one state of the feature map is an image displaying multiple feature graphics. Preferably, the multiple feature graphics displayed in the feature map are asymmetrically distributed. The asymmetrical distribution includes vertical asymmetry, horizontal asymmetry, and central asymmetry. The asymmetrical distribution of feature graphics can better reflect the changes in the screen orientation of the mobile terminal.

[0094] Since the screen of a mobile terminal displays a feature map, the screen location information and the feature map image of the mobile terminal can be obtained by identifying the position of the feature map in the real-time image.

[0095] Screen position information refers to the position and range of the mobile terminal's screen in a real-time image; while feature map image refers to the image when the feature map is displayed on the mobile terminal's screen. Specifically, it is obtained by acquiring the image of the mobile terminal's screen from the real-time image based on the position and range of the mobile terminal's screen in the real-time image, thereby obtaining the image of the feature map displayed on the screen.

[0096] The attitude information of a mobile terminal refers to the orientation or rotation direction of the mobile terminal's screen in three-dimensional space.

[0097] The inertial measurement information of a mobile terminal is acquired through an inertial measurement unit (IMU) installed within the mobile terminal. This inertial measurement information includes the mobile terminal's angular rate and acceleration. Typically, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the mobile terminal along three independent axes of the carrier coordinate system, while the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system. By measuring the angular velocity and acceleration of the mobile terminal in three-dimensional space, the inertial measurement rotation and displacement information of the mobile terminal are calculated, thus obtaining the initial information of the mobile terminal's first six degrees of freedom.

[0098] By weighting the initial six degrees of freedom (DOF) information of the mobile terminal calculated from its inertial measurement information with its attitude information and screen position information, the accuracy of the final attitude information of the mobile terminal can be improved. Specifically, as mentioned above, since the attitude information of the mobile terminal reflects its orientation and the screen position information reflects its position, the attitude information and screen position information can be combined to form the initial six degrees of freedom (DOF) information. By weighting the initial six degrees of freedom (DOF) information of the first and second DOF information, the control data of the mobile terminal with six degrees of freedom can be obtained. The control data of six degrees of freedom includes the rotation and displacement information of the object on the three-dimensional coordinate axes in space. For example, if the three-dimensional coordinate axes in space are represented by the X-axis, Y-axis, and Z-axis, the control data of six degrees of freedom includes the rotation and displacement information of the object on the X-axis, the rotation and displacement information of the object on the Y-axis, and the rotation and displacement information of the object on the Z-axis, etc.

[0099] Compared to related technologies, this application obtains the control data of the human-computer interaction device by using the feature map displayed on the mobile terminal and the inertial measurement information of the mobile terminal. This allows users to control the human-computer interaction device through the control data, eliminating the need to produce a remote control device to provide control data during the manufacturing process of the human-computer interaction device, thereby reducing the production cost of human-computer interaction devices based on controllers.

[0100] In one feasible embodiment, the human-computer interaction device is equipped with two positioning cameras;

[0101] The real-time image recognition module 102 further includes:

[0102] The first screen location information acquisition module acquires the screen location information of the mobile terminal through the two positioning cameras.

[0103] The first tracking frame generation module constructs a tracking frame for the screen of the mobile terminal based on the screen position information; the tracking frame changes as the screen of the mobile terminal changes.

[0104] The first feature map image acquisition module obtains the feature map image displayed on the screen of the mobile terminal based on the image within the tracking box.

[0105] Specifically, the mobile terminal is matched with two positioning cameras. After a successful match, the two positioning cameras and the mobile terminal form a triangle. Based on the principle of triangles, the horizontal and vertical distances between the mobile terminal and the two positioning cameras can be calculated using formulas, thereby obtaining the screen position information of the mobile terminal.

[0106] The tracking frame of a mobile terminal's screen is generated based on the screen's position information. It can maintain tracking of the screen's position to avoid losing track of it.

[0107] In this embodiment, the position of the mobile terminal's screen is acquired using two positioning cameras. Based on the position of the mobile terminal's screen, a feature map image of the screen display is obtained, thereby accurately acquiring the feature map image. In this embodiment, the feature map image is an image with multiple feature patterns and a solid-color background. Information such as the direction, distance, and relative size between the various feature patterns in the feature map image is used to calculate the mobile terminal's attitude information.

[0108] In one feasible embodiment, the feature map includes a first feature map and a second feature map, wherein the mobile terminal first displays the first feature map and then displays the second feature map.

[0109] The first feature map is a solid color image, including but not limited to a pure white image, to facilitate the identification of the location and range of the first feature map; the second feature map is an image with multiple feature graphics and a solid color background, and the color of the feature graphics in the second feature map is different from the color of the solid color background, including but not limited to images with white feature graphics and black backgrounds, to facilitate the identification of each feature graphic in the second feature map.

[0110] The real-time image recognition module 102 further includes:

[0111] The second screen location information acquisition module, when the mobile terminal displays the first feature map, identifies the screen location information of the mobile terminal by recognizing the real-time image through a preset first feature map recognition algorithm.

[0112] The second tracking frame generation module constructs a tracking frame for the screen of the mobile terminal based on the screen position information; the tracking frame changes as the screen of the mobile terminal changes.

[0113] The second feature map image acquisition module obtains the feature map image based on the image within the tracking box when the mobile terminal displays the image of the second feature map.

[0114] In this embodiment, the position of the mobile terminal's screen is obtained through the first feature map, and then, based on the position of the mobile terminal's screen, the image of the second feature map is obtained when the mobile terminal displays the image of the second feature map, thereby accurately obtaining the feature map image.

[0115] Here, the first and second feature maps can be understood as two display states of the feature maps. The display can be achieved by replacing the first feature map with the second feature map, or by changing the fill color. For example, the first feature map might have multiple feature graphics and a solid-color background, similar to the second feature map, but the feature graphics and the solid-color background are the same color. By changing the color of the solid-color background, the feature graphics and the solid-color background are different colors, thus displaying the second feature map. The implementation of the mobile terminal displaying the first feature map first and then the second feature map is not limited. It could be that a display time limit is set for the first feature map, the mobile terminal displays the first feature map first, and after the time limit is exceeded, the mobile terminal displays the second feature map. Alternatively, the mobile terminal can display the first feature map first, and when the human-computer interaction device recognizes the screen position information of the mobile terminal, it sends a map change signal to the mobile terminal, and the mobile terminal responds to the map change signal to display the second feature map.

[0116] In one feasible embodiment, the attitude information acquisition module 103 includes:

[0117] The graphic information acquisition module performs feature graphic recognition on the image of the feature map to identify the graphic information of the second feature map.

[0118] The graphic information of the second feature map includes information such as the direction, distance and relative size between the various feature graphics of the second feature map.

[0119] The posture information calculation module calculates the posture information of the mobile terminal based on the graphic information.

[0120] In this embodiment, the attitude information of the mobile terminal can be obtained from the graphic information of the second feature map.

[0121] The attitude information of the mobile terminal is calculated based on the graphic information using the ICP algorithm (Iterative Closest Point). The ICP calculation method finds the corresponding point pairs between the source point cloud and the target point cloud, constructs a rotation and translation matrix based on the corresponding point pairs, and uses the obtained matrix to transform the information data of the mobile terminal in the real space to the coordinate system of the virtual space. If the value of the transformed error function is greater than the threshold, the above calculation is iterated until the preset error requirement is met.

[0122] In one feasible embodiment, the feature graphic is a circular graphic;

[0123] The graphic information acquisition module includes:

[0124] The first detection module performs circle detection on the image of the feature map to obtain the position, distance, and relative size of each feature graphic in the second feature map.

[0125] Circle detection refers to the process of identifying circles in a target image using a circle detection algorithm to obtain information about each circle in the target image. In this example, the circle detection can also identify ellipses formed when a circular shape is tilted.

[0126] The first graphic information acquisition module determines the position, distance, and relative size of each feature graphic as the graphic information.

[0127] In this embodiment, the circle detection algorithm can be used to obtain graphic information such as the position, distance, and relative size of each circular graphic.

[0128] In one feasible embodiment, the feature graphic is a white graphic.

[0129] The graphic information acquisition module includes:

[0130] The second detection module performs white spot detection on the image of the feature map to obtain the position, distance, and relative size of each feature graphic in the second feature map.

[0131] White spot detection refers to the process of identifying white shapes (white spots) in a target image using a white spot detection algorithm, in order to obtain information about each white shape in the target image.

[0132] The second graphic information acquisition module determines the position, distance, and relative size of each feature graphic as the graphic information.

[0133] In this embodiment, the white spot detection algorithm can be used to obtain graphic information such as the position, distance, and relative size of each circular graphic.

[0134] In one feasible embodiment, the manipulation data acquisition module 104 includes:

[0135] The extrinsic parameter information acquisition module performs extrinsic parameter calibration based on the screen position information and the inertial measurement information to obtain the extrinsic parameter information of the mobile terminal; the extrinsic parameter information is used to indicate the rotation direction and position information of the mobile terminal in the coordinate system of the virtual space constructed by the human-computer interaction device.

[0136] The principle of extrinsic parameter calibration based on screen position information and inertial measurement information is based on rotation matrices and translation vectors. Specifically, these rotation matrices and translation vectors represent the transformation relationship between the three-dimensional coordinate system established based on the inertial sensor and the three-dimensional coordinate system of the virtual space constructed based on the human-computer interaction device. By calculating the transformation relationship, the rotation direction and position information of the mobile terminal in the coordinate system of the virtual space can be calculated, thereby obtaining the extrinsic parameter information of the mobile terminal.

[0137] The data acquisition module obtains control data for the human-computer interaction device based on the attitude information, the extrinsic parameter information, and the inertial measurement information.

[0138] In this embodiment, the control data is obtained based on the attitude information, extrinsic parameter information and inertial measurement information of the mobile terminal, which can improve the accuracy of the obtained six-degree-of-freedom control data.

[0139] In one feasible embodiment, the six-degree-of-freedom data acquisition module uses Kalman filtering or nonlinear optimization to fuse the attitude information, extrinsic parameter information, and inertial measurement information to obtain the six-degree-of-freedom attitude data of the mobile terminal; the six-degree-of-freedom attitude data of the mobile terminal is then used as control data for the human-computer interaction device. Specifically, the fusion calculation method is not limited; for example, weighted calculation or averaging calculation can be used.

[0140] In this embodiment, the accuracy of the obtained six-degree-of-freedom attitude data can be improved by using Kalman filtering or nonlinear optimization methods.

[0141] This application also provides a computer-readable storage medium storing a computer program, characterized in that: when the computer program is executed by a processor, it implements the steps of the mobile terminal-based control data acquisition method as described above.

[0142] This application also provides an electronic device, including a storage device, a processor, and a computer program stored in the storage device and executable by the processor. When the processor executes the computer program, it implements the steps of the mobile terminal-based control data acquisition method as described above.

[0143] The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0144] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of manipulation steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function selected in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0148] Please see Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. The electronic device 300 can specifically be a computer, mobile phone, tablet computer, interactive flat panel, etc. In an exemplary embodiment of this application, the electronic device 300 is an interactive flat panel. The electronic device 300 may include: at least one processor 310, at least one memory 320, at least one display, at least one network interface 340, user interface 350, and at least one communication bus 360.

[0149] The communication bus 360 is used to enable communication between these components.

[0150] The user interface 350 may include a display screen and a camera; the user interface 350 may also include standard wired and wireless interfaces.

[0151] The network interface 340 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface).

[0152] The processor 310 may include one or more processing cores. The processor 310 connects to various parts within the electronic device 300 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 320, and by calling data stored in the memory 320. Optionally, the processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 310 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 310.

[0153] The memory 320 may include random access memory (RAM) or read-only memory. Optionally, the memory 320 may include a non-transitory computer-readable storage medium. The memory 320 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 320 may also be at least one storage device located remotely from the aforementioned processor 310. Figure 3 As shown, the memory 320, which serves as a computer storage medium, may include an operating system, a network communication module, and a user.

[0154] exist Figure 3In the electronic device 300 shown, the user interface 350 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 310 can be used to call the operation application of the smart interactive flat panel stored in the memory 320, such as the live room entry program based on masked interaction; and execute the relevant operations of the registration method of any touch frame device or the data processing method of the touch frame device in the above embodiments, with corresponding functions and beneficial effects.

[0155] The fourth embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. The instructions are adapted to be loaded by a processor and executed by the steps of the above-described registration method or data processing method for the touch frame device. For details of the execution process, please refer to the specific descriptions shown in the embodiments, which will not be repeated here. The device containing the storage medium can be an electronic device such as a personal computer, laptop computer, smartphone, or tablet computer.

[0156] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0157] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0159] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for acquiring control data based on a mobile terminal, characterized in that, The method, applied to human-computer interaction devices, includes: The system acquires real-time images of the mobile terminal's location and the mobile terminal's inertial measurement information; the mobile terminal's screen displays a preset feature map. The screen position information and feature map image of the mobile terminal are identified from the real-time image; The attitude information of the mobile terminal is obtained based on the feature map image; Based on the inertial measurement information, the attitude information, and the screen position information, control data for the human-computer interaction device is obtained, including: The extrinsic parameters of the mobile terminal are obtained by calibrating the extrinsic parameters based on the screen position information and the inertial measurement information; the extrinsic parameter information is used to indicate the rotation direction and position information of the mobile terminal in the coordinate system of the virtual space constructed by the human-computer interaction device. Based on the attitude information, the extrinsic parameter information, and the inertial measurement information, control data for the human-computer interaction device is obtained.

2. The method for acquiring control data based on a mobile terminal according to claim 1, characterized in that, The step of obtaining control data for the human-computer interaction device based on the attitude information, the extrinsic parameter information, and the inertial measurement information includes: The attitude information, extrinsic parameter information, and inertial measurement information are fused and calculated using Kalman filtering or nonlinear optimization methods to obtain the six-degree-of-freedom attitude data of the mobile terminal; the six-degree-of-freedom attitude data of the mobile terminal is used as the control data for the human-computer interaction device.

3. The method for acquiring control data based on a mobile terminal according to claim 1, characterized in that, The human-computer interaction device is equipped with two positioning cameras; The step of identifying the screen position information and feature map image of the mobile terminal from the real-time image includes: The screen position information of the mobile terminal is obtained through the two positioning cameras; Based on the screen position information, a tracking frame for the mobile terminal's screen is constructed; the tracking frame changes as the mobile terminal's screen changes. Based on the image within the tracking frame, a feature map image displayed on the screen of the mobile terminal is obtained.

4. The method for acquiring control data based on a mobile terminal according to claim 1, characterized in that, The feature map includes a first feature map and a second feature map, wherein the mobile terminal first displays the first feature map and then displays the second feature map; The step of identifying the screen position information and feature map image of the mobile terminal from the real-time image includes: When the mobile terminal displays the first feature map, the real-time image is identified by a preset first feature map recognition algorithm to identify the screen position information of the mobile terminal. Based on the screen position information, a tracking frame for the mobile terminal's screen is constructed; the tracking frame changes as the mobile terminal's screen changes. When the mobile terminal displays the image of the second feature map, the feature map image is obtained based on the image within the tracking box.

5. The method for acquiring control data based on a mobile terminal according to claim 4, characterized in that, The step of obtaining the pose information of the mobile terminal based on the feature map image includes: The image of the feature map is subjected to feature pattern recognition to identify the graphic information of the second feature map; The posture information of the mobile terminal is calculated based on the graphic information.

6. The method for acquiring control data based on a mobile terminal according to claim 5, characterized in that, The feature graphic is a circular graphic; The step of performing feature pattern recognition on the image of the feature map to identify the graphic information of the second feature map includes: Perform circle detection on the image of the feature map to obtain the position, distance, and relative size of each feature graphic in the second feature map; The position, distance, and relative size of each feature graphic are determined as the graphic information.

7. The method for acquiring control data based on a mobile terminal according to claim 5, characterized in that, The feature graphic is a white graphic; The step of performing feature pattern recognition on the image of the feature map to identify the graphic information of the second feature map includes: White spot detection is performed on the image of the feature map to obtain the position, distance and relative size of each feature graphic in the second feature map; The position, distance, and relative size of each feature graphic are determined as the graphic information.

8. The method for acquiring control data based on a mobile terminal according to claim 4, characterized in that: The first feature map is a solid color image, and the first feature map recognition algorithm is a deep learning algorithm or a color space recognition algorithm.

9. A control data acquisition device based on a mobile terminal, characterized in that, The device is used in human-computer interaction devices and includes: A real-time image acquisition module is used to acquire real-time images of the location of the mobile terminal and the inertial measurement information of the mobile terminal; the screen of the mobile terminal displays a preset feature map. A real-time image recognition module is used to identify the screen position information and feature map image of the mobile terminal from the real-time image; The posture information acquisition module is used to obtain the posture information of the mobile terminal based on the feature map image; The control data acquisition module is used to obtain control data for the human-computer interaction device based on the inertial measurement information, the attitude information, and the screen position information, including: The extrinsic parameters of the mobile terminal are obtained by calibrating the extrinsic parameters based on the screen position information and the inertial measurement information; the extrinsic parameter information is used to indicate the rotation direction and position information of the mobile terminal in the coordinate system of the virtual space constructed by the human-computer interaction device. Based on the attitude information, the extrinsic parameter information, and the inertial measurement information, control data for the human-computer interaction device is obtained.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the mobile terminal-based control data acquisition method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that: It includes a storage device, a processor, and a computer program stored in the storage device and executable by the processor, wherein the processor executes the computer program to implement the steps of the mobile terminal-based control data acquisition method as described in any one of claims 1 to 8.

Images