A large display screen pointing interaction method and system based on a smart phone

Abstract

The application belongs to the technical field of human-computer interaction, and provides an interactive method and system for large display screen pointing based on a smart phone. A plurality of marks are embedded in an exhibition picture of a large display screen according to the background characteristics of current display content. Image data captured by the smart phone is obtained, and preprocessing is performed on the image data. Detection and decoding operation of the marks are performed according to the preprocessed image data, unique code information corresponding to the marks and pose data in a camera coordinate system are extracted. Spatial mapping operation is performed according to the pose data of the marks in the camera coordinate system and pixel coordinates of the marks on the display screen, a plurality of candidate projection points of the optical center of the smart phone in a screen coordinate system are solved. Interpolation calculation is performed on the candidate projection points, and final mapping coordinates of the optical center of the smart phone on the display screen are calculated. Interaction is performed according to the final mapping coordinates. The application can easily complete positioning and selection of a large screen display object.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to an interaction method and system based on the pointing of a large display screen on a smartphone. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Large-screen displays, also known as giant display screens, are typically used in museums, art galleries, and science museums. They are devices designed to provide large-screen displays, allowing multiple viewers to watch and enjoy simultaneously. Because of their considerable size, visitors often need to maintain a certain distance from the screen to view it, making close-range interactive technologies such as touchscreens unsuitable for these scenarios.

[0004] Therefore, these displays are typically used only for one-way information presentation, or the content on the screen is switched and controlled uniformly by the tour guide via a console, tablet, or laser pointer. They do not support users selecting different types of content on the screen according to their individual needs. Furthermore, in applications such as games, interactive tools such as game controllers, infrared cameras, motion capture devices, and eye trackers are frequently used to achieve on-screen positioning and interactive operations. However, these require the deployment of additional interactive equipment, resulting in significant limitations and high costs in terms of equipment arrangement.

[0005] Currently, there are still significant drawbacks in technical solutions that allow users to interact with large display screens without needing to carry or deploy dedicated interactive devices:

[0006] The solution of using a smartphone to capture and recognize a single QR code to achieve discrete object interaction is limited by the single-point recognition characteristic of QR codes. It cannot achieve continuous tracking of interactive icons with user operations, and it is difficult to meet the real-time interaction requirements in large display screen scenarios.

[0007] Meanwhile, the solution of using smartphone cameras to capture background information of large displays for positioning will cause the background features required for positioning on the mobile phone to change after the interactive content is displayed on the screen, thus causing the positioning function to fail and failing to meet the interactive needs of dynamic switching of exhibition content in large display environments.

[0008] In addition, the solution of controlling the cursor position in a large display device by swiping the smartphone screen relies solely on the user's finger swipe operation and is not related to the user's actual spatial position. When the user moves within a large space, the cursor cannot smoothly migrate to the new field of view with the user's position, which can easily lead to the cursor being lost. Furthermore, it is not possible to quickly recall the cursor to the user's field of view through swiping operations. This solution is only suitable for small space scenarios and is difficult to adapt to large display environments. Summary of the Invention

[0009] To address the aforementioned problems, this invention proposes an interactive method and system for pointing on a large display screen based on a smartphone. This invention enables multiple users in large display scenarios to obtain an interactive cursor that can move continuously and smoothly and follow the user's spatial position in real time. Moreover, the positioning and following of the interactive cursor does not depend on the background content of the large display screen. Even if the screen background content changes dynamically, the interactive function can still be stably implemented, allowing users to easily locate and select objects displayed on a large screen using their personal smartphones.

[0010] According to some embodiments, the present invention adopts the following technical solution:

[0011] An interaction method based on pointing a large smartphone display includes the following steps:

[0012] Several pre-set icons will be embedded into the exhibition screen of the large display screen based on the background characteristics of the current content being displayed;

[0013] The system acquires image data captured by a smartphone and performs preprocessing steps, including denoising, grayscale adjustment, and correction.

[0014] Based on the preprocessed image data, the identification and decoding operations are performed to extract the unique encoding information corresponding to the identification and the pose data in the camera coordinate system.

[0015] Based on the extracted pose data in the camera coordinate system of each marker and the corresponding pixel coordinates of the marker on the display screen, a spatial mapping operation is performed to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system; interpolation calculation is performed on the candidate projection points to calculate the final mapping coordinates of the optical center of the smartphone on the display screen.

[0016] In response to a smartphone pointing at a large display screen, the interactive icons on the large display screen are controlled to move in real time according to the final mapped coordinates. In response to a user's instruction to select or control information or programs on the large display screen based on the final mapped coordinates, the selection or control is executed.

[0017] As an alternative implementation, the process of pre-setting the labels includes: setting the color, transparency, size, and distribution area of ​​the AprilTag labels according to the background content currently displayed on the large display screen, so as to ensure that the labels do not cover the predetermined exhibition content, do not affect the visual experience, and that the camera on the smartphone can effectively extract each label in the predetermined physical space.

[0018] As an alternative implementation, the process of detecting and decoding the identifier based on the preprocessed image data includes: using an edge detection algorithm to extract edge pixels, and then using line segment fitting and contour tracking algorithms to find all potential candidate regions for the corresponding identifier;

[0019] Encoding parsing and recognition are performed. For the candidate quadrilateral regions that pass the screening, the quadrilateral regions are transformed by perspective and corrected to standard squares. Then, adaptive binarization is performed to obtain a black and white coded pattern. The corresponding coded information is extracted from the processed image for decoding. After successful decoding, the family and unique ID of the corresponding identifier are recorded.

[0020] As an alternative implementation, the pose data includes a three-dimensional position and pose quaternion identified in the camera coordinate system.

[0021] As an alternative implementation, the process of performing spatial mapping operations based on the extracted pose data of each marker in the camera coordinate system and the corresponding pixel coordinates of the marker on the display screen to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system includes: for a single marker, converting the pose quaternion of the marker in the camera coordinate system into a rotation matrix representation. The rotation matrix of the smartphone relative to the intermediate coordinate system for The transpose of the matrix is ​​used to calculate the rotation angles of the intermediate coordinate system relative to the X, Y, and Z axes of the camera coordinate system using trigonometric functions. , , We obtain the rotation transformation matrix M from the camera coordinate system to the intermediate coordinate system, where the origin of the intermediate coordinate system coincides with the origin of the camera coordinate system, and the coordinate axes are parallel to the world coordinate system.

[0022] The position of the marker in the camera coordinate system is rotated and transformed using the rotation transformation matrix M to obtain the position of the marker in the intermediate coordinate system.

[0023] Based on the pixel size of the marker in the screen coordinate system and the actual size in the world coordinate system, calculate the mapping ratio coefficient. Based on the preset pixel position of the marker in the screen coordinate system, its position in the intermediate coordinate system, and the mapping ratio coefficient, calculate the mapping point of the camera optical center in the screen coordinate system.

[0024] Based on the distance between the camera and the screen in the intermediate coordinate system, the displacement of the intersection point of the camera coordinate system and the intermediate coordinate system z-axis with the world coordinate system XY plane is calculated, and then the actual mapped position of the camera optical center in the screen coordinate system is obtained.

[0025] As an alternative implementation method, the process of solving multiple candidate projection points of the optical center of the smartphone in the screen coordinate system by performing spatial mapping calculations based on the extracted pose data of each marker in the camera coordinate system and the pixel coordinates of the corresponding marker on the display screen includes: for multiple markers, taking the unit quaternion corresponding to the ideal stable state with zero rotation as a reference, the stability index of the corresponding marker is obtained by solving the dot product of the pose quaternion and the unit quaternion.

[0026] The stability index is normalized to obtain the normalized interpolation coefficients corresponding to each identifier;

[0027] The coordinates of the cursor in the screen coordinate system of the display screen are calculated by weighting the coordinates of the optical center mapping corresponding to all valid identifiers and their corresponding interpolation coefficients.

[0028] The cursor is displayed at the coordinates of the cursor on the screen in response to user interaction.

[0029] As a further defined implementation, before the process of normalizing the stability index, the method further includes: removing pose data with a stability index less than a set value to obtain a set of effective pose data, wherein each element in the set contains a quaternion of the position and attitude corresponding to the position in the camera coordinate system, and normalizing the stability index of the effective pose data.

[0030] As a further defined implementation, before the process of eliminating pose data with a stability index less than a set value, the method further includes: performing distance correction on the calculated stability index; if the distance from the position of a certain marker in the camera coordinate system to the camera optical center exceeds a preset effective recognition range, multiplying its stability index by a predetermined penalty coefficient to reduce the weight of markers that exceed the set distance.

[0031] As an alternative implementation, the identifier is the AprilTag identifier.

[0032] An interactive system based on a large smartphone display, comprising:

[0033] The image embedding module is used to embed several pre-set icons, based on the background features of the currently displayed content, into the exhibition screen of a large display screen;

[0034] The image acquisition and processing module is used to acquire image data captured by the smartphone and perform preprocessing such as noise reduction, grayscale adjustment and correction on it in sequence.

[0035] The image recognition module is used to perform identification and decoding operations on the preprocessed image output by the image acquisition and processing module, and to extract the unique encoding information corresponding to the identification and the pose data in the camera coordinate system.

[0036] The interpolation mapping module is used to perform spatial mapping operations based on the pose data of each marker in the camera coordinate system extracted by the image recognition module and the pixel coordinates of the corresponding marker on the display screen to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system; and to perform interpolation calculations on the candidate projection points to calculate the final mapped coordinates of the optical center of the smartphone on the display screen.

[0037] The human-computer interaction module is used to respond to the operation of pointing at the large display screen on the smartphone, control the interactive icons on the large display screen to move in real time according to the final mapping coordinates, and execute the selection or control in response to the user's instruction to select or control information or programs on the large display screen based on the final mapping coordinates.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] This invention proposes a method for interacting with large public displays based on smartphones, aiming to enable users to scan and recognize embedded logos on the screen to achieve real-time interaction. Compared to the shortcomings of traditional QR codes, such as susceptibility to background interference and low recognition rate, the logos in this invention are pre-optimized for scene recognition, possessing distinct edge features, strong resistance to occlusion and changes in lighting, and support for pose calculation to achieve high-precision spatial positioning.

[0040] This invention utilizes pre-optimized markers, allowing users to easily locate and select displayed objects on the screen using their smartphones. It also supports a drawing-style operation where users can leave traces by long-pressing the phone screen. Compared to existing technologies, this invention overcomes the size and recognition limitations of traditional visual markers, eliminating the need for additional interactive devices and making it more suitable for scenarios where multiple users interact simultaneously on large display screens in spacious environments.

[0041] This invention features a multimedia interactive guide based on a large display screen, showcasing diverse content formats, including single-point triggering of animated videos and audio narration on the large display screen, and a drawing-style operation where users can save their actions by long-pressing the screen. This diverse content presentation attracts user interest and enhances user engagement and immersion. Furthermore, the exhibition content is highly flexible and adaptable to various application scenarios, making it suitable not only for museum and art gallery exhibitions but also for a wide range of venues such as school education, community cultural activities, and commercial displays, thus significantly expanding the system's application scope.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0044] Figure 1 A schematic diagram of the hardware facilities for a smartphone and a large display screen positioning interaction mode according to one embodiment;

[0045] Figure 2 This is a schematic diagram of the system operation process in one embodiment;

[0046] Figure 3 This is a schematic diagram of the execution flow of an image recognition module according to one embodiment;

[0047] Figure 4 This is a schematic diagram of the execution flow of an interpolation mapping module according to one embodiment;

[0048] Figure 5 This is a schematic diagram of a system application according to one embodiment. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0053] Example 1

[0054] An interaction method based on pointing a large smartphone display includes the following steps:

[0055] Based on the background characteristics of the current content being displayed, several pre-set logos will be embedded into the exhibition screen of the large display screen (which may be referred to as the large screen or big screen);

[0056] The system acquires image data captured by a smartphone and performs preprocessing steps, including denoising, grayscale adjustment, and correction.

[0057] Based on the preprocessed image data, the identification and decoding operations are performed to extract the unique encoding information corresponding to the identification and the pose data in the camera coordinate system.

[0058] Based on the extracted pose data in the camera coordinate system of each marker and the corresponding pixel coordinates of the marker on the display screen, a spatial mapping operation is performed to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system; interpolation calculation is performed on the candidate projection points to calculate the final mapping coordinates of the optical center of the smartphone on the display screen.

[0059] In response to a smartphone pointing at a large display screen, the interactive icons on the large display screen are controlled to move in real time according to the final mapped coordinates. In response to a user's instruction to select or control information or programs on the large display screen based on the final mapped coordinates, the selection or control is executed.

[0060] Example 2

[0061] An interactive system based on a large smartphone display, comprising:

[0062] The image embedding module is used to embed several pre-set icons, based on the background features of the currently displayed content, into the exhibition screen of a large display screen;

[0063] The image acquisition and processing module is used to acquire image data captured by a smartphone (hereinafter referred to as the mobile phone) and perform preprocessing such as denoising, grayscale adjustment and correction on it in sequence;

[0064] The image recognition module is used to perform identification and decoding operations on the preprocessed image output by the image acquisition and processing module, and to extract the unique encoding information corresponding to the identification and the pose data in the camera coordinate system.

[0065] The interpolation mapping module is used to perform spatial mapping operations based on the pose data of each marker in the camera coordinate system extracted by the image recognition module and the pixel coordinates of the corresponding marker on the display screen to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system; and to perform interpolation calculations on the candidate projection points to calculate the final mapped coordinates of the optical center of the smartphone on the display screen.

[0066] The human-computer interaction module is used to respond to the operation of pointing at the large display screen on the smartphone, control the interactive icons on the large display screen to move in real time according to the final mapping coordinates, and execute the selection or control in response to the user's instruction to select or control information or programs on the large display screen based on the final mapping coordinates.

[0067] This embodiment uses Figure 1The scenario shown, taking the AprilTag identifier as an example, describes the execution details of each module in the system. Before the detailed description, the following explanation is provided:

[0068] To clearly define the positioning logic of this embodiment and ensure the reproducibility of coordinate calculation and spatial mapping, the definitions and parameter specifications of the four coordinate systems used in this embodiment are first clarified. All subsequent calculations involving coordinates and poses are based on the following standards:

[0069] (1) Screen Coordinate System (SCS):

[0070] Origin: The center of each display screen, that is, the intersection of the diagonals of each screen;

[0071] Axis direction: X-axis is horizontal to the right along the screen, Y-axis is vertical downward along the screen;

[0072] Unit: pixels (px);

[0073] Applicable scenarios: Used to accurately describe the pixel layout position of the AprilTag on the display screen and the mapping position of the user interaction cursor on the screen.

[0074] (2) Camera Coordinate System (CCS):

[0075] Origin point: The optical center of a smartphone camera (lens optical center);

[0076] Axis direction: The X-axis is aligned with the horizontal rightward direction of the phone screen, the Y-axis is aligned with the vertical downward direction of the phone screen, and the Z-axis extends forward and points towards the display screen.

[0077] Unit: meter (m);

[0078] Applicable scenarios: Used to describe the spatial pose of the AprilTag relative to the smartphone camera, including the three-dimensional position coordinates and pose information of the tag relative to the camera.

[0079] (3) World Coordinate System (WCS):

[0080] Origin point: The bottom left corner of each display screen;

[0081] Axis directions: The X-axis is parallel to the X-axis of the screen coordinate system (horizontally to the right), the Y-axis is parallel to the Y-axis of the screen coordinate system (vertically downward), and the Z-axis is perpendicular to the screen plane and points outward (in the same direction as the Z-axis of the screen coordinate system).

[0082] Unit: meter (m);

[0083] Applicable scenarios: As an intermediate coordinate system, it is used to realize the mutual conversion between the screen coordinate system and the camera coordinate system, unify the spatial reference benchmark of multiple smartphones and multiple AprilTags, and ensure the positioning consistency when multiple devices interact collaboratively.

[0084] (4) Image Coordinate System (ICS):

[0085] Origin: The central vertex of the image captured by a smartphone camera;

[0086] Axis direction: U-axis is horizontal to the right along the image, V-axis is vertical to the bottom along the image;

[0087] Unit: pixels (px);

[0088] Applicable scenarios: Used to describe the pixel position of the AprilTag in images captured by mobile phones. It serves as a bridge between the camera coordinate system and the screen coordinate system, providing raw pixel data for subsequent perspective transformations and pose calculations.

[0089] In specific implementation, the image embedding module is used for:

[0090] Based on the background content displayed on the large screen, the physical attributes of the AprilTag are designed manually. This means that the color, transparency, size, and distribution area of ​​the AprilTag are designed according to the content of the background image, ensuring that the phone's RGB camera can effectively extract the tag within the specified physical space. Compared to QR codes, the AprilTag offers more flexible size design, and its edge features are not affected by the background pattern, significantly improving the extraction success rate.

[0091] In the specific implementation process, the image acquisition and processing module uses a mobile phone lens that supports multiple frame rates to acquire multiple frames of video stream captured by the mobile phone lens, and then transmits the acquired images to the image recognition module after preprocessing.

[0092] In the specific implementation process, the process of identifying the AprilTag identifier attribute based on the image information transmitted by the image acquisition and processing module in the image recognition module is as follows:

[0093] An edge detection algorithm is used to extract significant edge pixels. Then, line segment fitting and contour tracking algorithms are used to find all potential candidate regions for AprilTag identifiers.

[0094] Encoding parsing and recognition are performed. For candidate quadrilateral regions that pass the screening, perspective transformation is applied to correct them into standard squares, followed by adaptive binarization to obtain a clear black-and-white encoded pattern. The encoding information corresponding to the selected AprilTag identifier and its model is extracted from the corrected image and decoded. Upon successful decoding, the family and unique ID of the AprilTag identifier are recorded.

[0095] In this embodiment, the interpolation mapping module is used to extract the ID numbers of all AprilTag identifiers in the image recognized by the image recognition module. , … n represents the total number of identifiers and their positions in the camera coordinate system. , … Posture Quaternion ,in These are the imaginary components of the attitude quaternion, used to characterize the rotation axis information of the target in three-dimensional space. The real part of the attitude quaternion represents the rotation angle information of the target about the rotation axis. And the pixel position is indicated on the screen. , … The position of the optical center of the phone mapped to the screen coordinate system is calculated. , … Then, the obtained multiple optical center positions are interpolated to obtain the final position of the optical center mapped on the screen. ;

[0096] The human-computer interaction module is used to detect the user's finger touch operation on the smartphone screen, obtain the corresponding touch coordinates, and then realize the selection and playback control of the content displayed on the large display screen. When the interactive icon moves on the screen and the system detects a long press operation on the phone screen, the movement trajectory can be drawn on the screen to achieve a trajectory retention effect similar to electronic graffiti. In addition, the human-computer interaction module supports multi-point collaborative interaction, which can simultaneously recognize the touch commands of multiple smartphones, enabling multiple users to select, play, and perform other interactive operations on different image or video content in parallel.

[0097] This embodiment proposes an interactive method based on pointing on a large smartphone display. Combining image embedding, image acquisition, image recognition, and interpolation mapping technologies, it allows users to remotely view related information on the large display screen from their personal perspective and engage in collaborative interaction. The system features real-time multi-user interaction, display of the large screen background and related content, over-the-air interaction, and data management and updates, providing an intuitive and highly focused interactive experience. Furthermore, the solution provided in this embodiment requires no additional sensing or interaction devices; users only need to hold their own smartphone, reducing the burden of use and demonstrating broad application prospects.

[0098] On large display screens, varying numbers and densities of AprilTag icons are placed against the background of different content to ensure that the content of interest is not obscured while improving recognition accuracy. Furthermore, the QR codes can be designed with appropriate decorative patterns to enhance their aesthetic appeal.

[0099] The mobile system captures each frame of the real-time video stream using its built-in camera. It then performs Gaussian filtering for noise reduction, grayscale adjustment, and gamma correction preprocessing on the raw image data to improve image quality and feature extractability. Subsequently, it performs AprilTag identification and pose calculation on the preprocessed images to obtain the three-dimensional position coordinates and pose parameters of each tag in the camera coordinate system. Based on these pose parameters, it constructs a spatial mapping relationship between the optical center of the mobile phone camera and the large display screen. Simultaneously, it performs interpolation calculations based on the pose data of multiple identified AprilTag tags to ultimately determine the mapped coordinate position of each user's mobile phone in the screen coordinate system.

[0100] A real-time feedback mechanism is provided for user operations, ensuring that every touch action is instantly reflected on the large display screen. Through rapid response and rendering, every user positioning and click interaction is seamlessly responded to, guaranteeing a smooth user experience.

[0101] This embodiment can also be flexibly configured according to different application scenarios and needs. In addition to museums and art galleries, it can also be applied to science and technology museums, libraries, exhibitions, and other venues.

[0102] Taking a large-screen interactive system based on a smartphone as an example, by supporting various display environments such as computers, projectors, and large display screens, large-screen content designers can naturally integrate the AprilTag identifier with the background content without affecting the viewer's perception of the background image. Then, using the phone's built-in camera function, single-frame images are continuously acquired from the captured video stream. Using image processing, object detection, and recognition technologies, the information of the AprilTag identifier is matched with the information in its corresponding model family information database. Based on the matched icon information and attributes, spatial interpolation mapping is performed to map the phone's spatial position onto the display screen in real time.

[0103] The system provided in this embodiment, such as Figure 1 As shown, the system supports multiple users holding smartphones to interact collaboratively with a large display screen. While viewing the content displayed on the large screen, users can interact with the screen in real-time using their smartphones; that is, as the phone's posture changes, the corresponding interactive cursor on the large screen moves synchronously. A single tap on the smartphone screen triggers a pop-up window with detailed information about the displayed content and plays a video introduction. A long press on the smartphone screen allows for real-time recording of the interaction trajectory on the large screen, achieving a trajectory recording effect similar to electronic graffiti, enriching the multi-user collaborative interaction scenarios. Therefore, it greatly enhances the user's interactive experience and sense of participation, making the exhibition process richer, more intuitive, and more educational.

[0104] Example 3

[0105] Based on the system provided in Embodiment 2, its working method is described in detail, such as... Figure 1 As shown, in specific applications, this embodiment takes an outdoor display screen as an example, which only requires a large display screen, a wireless router and a smartphone. This embodiment does not require additional interactive tools such as controllers, infrared cameras, motion capture devices and eye trackers.

[0106] like Figure 2 As shown, when a user holds their smartphone facing a large display screen, a network connection is established between the two. If both are on the same network, the system generates a continuously movable interactive icon on the screen. This icon moves in real-time following the phone's posture, achieving a screen positioning and movement effect similar to a traditional mouse, facilitating convenient large-screen interaction. The user does not need to look at the phone throughout the process and can directly interact with the content displayed on the screen. If the two are not on the same network, a new network connection or establishment is initiated.

[0107] After the interactive icon (also known as a dynamic icon) moves with the user's phone and lands in the area of ​​interest, the system determines whether to trigger an interaction based on the user's selection. If so, the user can interact by long-pressing or tapping / singling the phone screen to execute control commands. In this embodiment, long-pressing the phone screen displays a graffiti trajectory in the interactive area, while tapping the phone screen plays an image or video in the interactive area.

[0108] Of course, in other embodiments, different interactive actions can be selected to correspond to different interactive instructions / effects depending on the scenario, which will not be exhaustively listed here.

[0109] An interaction based on pointing to a large smartphone display includes the following steps:

[0110] Step S101: The large screen system places different numbers and densities of AprilTag icons (in this embodiment, the tagStandard41h12 model is selected) on the background of the content to be displayed according to different needs. The icons are arranged reasonably in the blank area of ​​the display screen. At the same time, the transparency and color value of the AprilTag icons can be adjusted according to the background pattern to ensure that the core exhibition content is not covered and the visual experience is not affected. The icons can also be used to improve the recognition accuracy by utilizing their distinct edge features. Furthermore, the icons can be integrated into decorative elements according to the content design to increase the aesthetic appeal.

[0111] Step S102: Image acquisition and recognition.

[0112] The system performs operations such as acquiring and preprocessing stacked images obtained from the mobile phone, and identifying AprilTag information to obtain attribute information such as the unique ID, spatial coordinates, and pose data of the tags.

[0113] Step S103: Real-time interpolation mapping of non-uniformly superimposed AprilTag identifiers.

[0114] When an image captured by a mobile phone camera contains multiple AprilTag identifiers, interpolation coefficients are calculated based on the poses of multiple identifiers, and finally, interpolation is performed according to weights. This can effectively reduce the impact of pose errors of a single AprilTag identifier and improve the stability and accuracy of screen plane estimation.

[0115] Step S104: User interaction detection and response.

[0116] The system uses the interaction module to parse the user's operation commands based on the location of the AprilTag icon on the screen when the user touches the phone.

[0117] Step S105: End or cycle monitoring.

[0118] After completing one round of interactive operations, the system returns to the user interaction detection state and continues to listen to the user's touch actions, supporting continuous interactive operations.

[0119] In different application scenarios, the system can flexibly adjust according to the scene background and interactive content to adapt to various application needs, achieve accurate display and interaction of different exhibition content, and meet the flexible use of various scenarios.

[0120] like Figure 3 As shown, this includes edge detection and contour analysis of each frame of the acquired real-time video stream, encoding and parsing of the AprilTag identifier, and pose calculation. The specific implementation method is as follows:

[0121] Step S101: Mobile phone camera calibration.

[0122] Based on smartphone recognition of QR codes and calculation of cursor positions, camera calibration is required during the preprocessing stage. In this embodiment, the Zhang Zhengyou calibration method is used, establishing a mapping relationship between the image coordinate system and the world coordinate system by capturing multiple images of the planar checkerboard calibration board in different poses. In this embodiment, this process only needs to be performed during system deployment; it is not required during end-user operation.

[0123] Specifically, in this embodiment, by adjusting the relative posture (translation and rotation angles) of the handheld mobile phone and the calibration board, 15 clear images containing complete checkerboard features were acquired, ensuring that the calibration board covered different areas in the images without severe occlusion or distortion. Based on the equivalent focal length and camera image resolution obtained from the calibration, the field of view (FOV) in the horizontal and vertical directions of the camera was derived.

[0124] Step S102: Image acquisition and preprocessing.

[0125] Users move the cursor on a large screen using their mobile phone's camera. This operation requires the camera to capture images in real time while facing the screen. If no image data is captured, it is captured again. If image data is captured, preprocessing such as Gaussian filtering for noise reduction, grayscale calculation, and gamma correction is performed to prepare for subsequent recognition.

[0126] Step S1021: Gaussian filtering for noise reduction.

[0127] The Gaussian function is: ,in Indicates coordinates as Gaussian filter weights at the specified location; These represent the horizontal and vertical coordinate offsets with the center of the filter kernel as the origin, respectively. The standard deviation of the Gaussian filter is taken in this embodiment. The filter kernel size is 3×3. For edge pixels, their original values ​​are directly preserved to avoid out-of-bounds distortion.

[0128] Step S1022: Grayscale image calculation.

[0129] Using formula Converting a color image to a grayscale image involves using the R, G, B color mode, a digital color representation based on the three primary colors of red, green, and blue. Each primary color channel typically uses a preset value range to represent its brightness level. By independently controlling and superimposing the brightness values ​​of the three primary color channels, the color output and image display of the corresponding pixels can be achieved.

[0130] Step S1023: Gamma correction.

[0131] To address the nonlinear brightness response characteristics of the camera sensor, a gamma coefficient γ = 1.0 (without a calibration reference value) was set, and a nonlinear transformation formula was used. Adjust the image brightness distribution. Among these, For the corrected brightness, The values ​​represent the original brightness, both ranging from 0 to 255. A 256-level gamma lookup table (LUT) is pre-calculated within the program to optimize computational efficiency. Adjustments can be made if there is loss of detail in dark areas in the actual scene. Adjust the brightness to 0.8 to brighten dark areas, and adjust it to 1.2 to darken overexposed bright areas, thus achieving adaptive optimization of brightness distribution.

[0132] Step S103: Identification and pose estimation of Apriltag identifiers.

[0133] This step can utilize the ProcessImage algorithm from the jp.keijiro.apriltag library to recognize and estimate the pose of Apriltags. The inputs are preprocessed image information (in this embodiment, image width, height, and downsampling factor), the camera field of view, and the physical dimensions of the Apriltag (in meters). The outputs are the Apriltag ID, the translation vector of the tag relative to the camera in the camera coordinate system, and the pose quaternion, i.e., the tag's position. and attitude quaternions .

[0134] If the calculation process fails, the process is redirected back to the step of using the phone's camera to capture image data from the large screen.

[0135] Figure 4This document illustrates the flowchart of the interpolation mapping module in the interactive system based on a large smartphone display screen in this embodiment. First, it receives information related to the AprilTag identifiers identified by the image recognition module. This identifier information includes at least the AprilTag ID and its position and pose quaternions in the camera coordinate system. Since the distribution of AprilTag identifiers on the screen may be uneven, directly using a single icon would lead to unstable mapping. Therefore, the system interpolates the pose data of multiple icons to obtain a smoother, more continuous mapping basis. Subsequently, combining the known pixel positions of each AprilTag identifier on the display screen, the system calculates the projection coordinates of the phone camera's optical center on the screen through spatial geometric relationships and coordinate transformations. These coordinates represent the user's current viewing position on the screen, providing accurate position mapping for subsequent interactive operations such as selection, triggering, and drawing. The specific steps are as follows:

[0136] Step S101: Calculate the position of the mobile phone optical center in the screen coordinate system based on a single Apriltag identifier.

[0137] For an Apriltag captured by the mobile phone camera, the mapping position of the phone's optical center in the screen coordinate system can be directly calculated based on the position parameters of the tag in the screen coordinate system and the camera coordinate system through spatial geometric projection and coordinate transformation. .

[0138] make This is the preset pixel position of the Apriltag identifier in the screen coordinate system. Let this be its position in the camera coordinate system. To facilitate explaining how to calculate it... This embodiment introduces an intermediate coordinate system ( The origin of this coordinate system coincides with the origin of the camera coordinate system, and the coordinate axes are parallel to the world coordinate system.

[0139] Step S1011: Calculate the transformation matrix M between the camera coordinate system and the intermediate coordinate system based on the attitude quaternion R.

[0140] First, assign Apriltag to the pose quaternion in the camera coordinate system. Convert to a rotated matrix representation :

[0141] ;

[0142] Rotation matrix of the mobile phone relative to the intermediate coordinate system for The transpose of is represented as follows:

[0143] ;

[0144] Then, by using trigonometric functions, the rotation angles of the intermediate coordinate system relative to the X, Y, and Z axes of the camera coordinate system are calculated. , , :

[0145] ;

[0146] The rotation angle obtained through the above solution , , We obtain the rotation transformation matrix M that transforms the camera coordinate system to the intermediate coordinate system (where the origin of the intermediate coordinate system coincides with the origin of the camera coordinate system, and the coordinate axes are parallel to the world coordinate system):

[0147] ;

[0148] Step S1012: Calculate the mapping position of the camera optical center in the screen coordinate system based on the intermediate coordinate system.

[0149] Position the Apriltag marker in the camera coordinate system By performing a rotation transformation using the aforementioned transformation matrix M, the position of the marker in the intermediate coordinate system can be obtained. :

[0150] ;

[0151] Since the origin of the intermediate coordinate system is also the optical center of the camera, therefore, This refers to the displacement in the X, Y, and Z directions from the center point to the optical center of the phone, displayed in the intermediate coordinate system. Since the intermediate coordinate system is parallel to the world coordinate system, These represent the displacements of the center point relative to the camera's optical center in the X, Y, and Z directions of the world coordinate system, respectively. Conversely, These represent the displacements of the camera's optical center mapping points relative to the marker center point in the X, Y, and Z directions of the world coordinate system.

[0152] make and These represent the pixel dimensions (in pixels) of the Apriltag identifier in screen coordinates and its actual size (in meters) in world coordinates, respectively. Their mapping scaling factor... for:

[0153] ;

[0154] but These represent the displacements of the camera optical center's mapped point relative to the marker center point in the X, Y, and Z directions of the screen coordinate system, respectively. From this, the mapped point of the camera optical center in the screen coordinate system can be calculated. :

[0155] ;

[0156] It is important to note the mapping position of the camera's optical center in the screen coordinate system. It is calculated in the intermediate coordinate system. In fact, it is the intersection of the Z-axis of the intermediate coordinate system and the screen space, but it is not the actual (i.e., the camera coordinate system) mapping position of the camera's optical center in the screen coordinate system. Therefore, the following steps are needed to obtain the actual mapped position of the camera optical center in the screen coordinate system.

[0157] Step S1013: The actual mapped position of the camera optical center in the screen coordinate system.

[0158] As mentioned earlier, the intermediate coordinate system is formed by rotating the camera coordinate system sequentially around the X-axis. Rotation around the Y-axis Rotation around the Z-axis This is obtained. Conversely, the camera coordinate system can be viewed as a result of rotating the intermediate coordinate system sequentially around the x-axis. Rotate in the opposite direction around the Y-axis Rotate in the opposite direction around the z-axis This is obtained. Thus, the distance between the camera and the screen in the intermediate coordinate system is... This allows us to calculate the displacement of the intersection point between the camera coordinate system, the intermediate coordinate system z-axis, and the world coordinate system XY plane (i.e., the physical screen). and As shown:

[0159] ;

[0160] Therefore, the actual mapped position of the camera's optical center in the screen coordinate system for:

[0161] ;

[0162] Step S102: Calculate the position of the camera optical center in the screen coordinate system based on multiple Apriltag identifiers.

[0163] The calculation method given above is based on an Apriltag identifier to calculate the mapping position of the camera's optical center in the screen coordinate system. However, in practical applications, mobile phone cameras often capture multiple Apriltag identifiers simultaneously. Therefore, it is possible to calculate a mapping position of the camera's optical center in the screen coordinate system based on each Apriltag identifier, and then interpolate the results to obtain the cursor position, thereby improving the accuracy of the calculation.

[0164] Considering the differences in imaging accuracy and shooting angle among various Apriltag markers in the image, and the influence of environmental noise, the mapped coordinates obtained based on different markers will have certain deviations. Therefore, it is necessary to assign different coefficients to different markers to establish a reasonable interpolation calculation method in order to obtain a more accurate and stable cursor position.

[0165] In this embodiment, the setting of each Apriltag identifier coefficient is related to the stability of the Apriltag identifier. Existing research has shown that attitude rotation is the main source of Apriltag positioning error; the smaller the attitude deviation, the smaller its deviation from the unit quaternion, the more stable the identifier attitude, and the corresponding position. The smaller the error, the better. Therefore, the Apriltag stability index is introduced. This is used to evaluate the contribution and impact of each identifier on the accuracy of cursor position calculation. Then, different identifiers are assigned corresponding interpolation weights, and the final cursor position is obtained through weighted calculation. For example... Figure 4 As shown, the specific implementation process consists of the following three sub-steps:

[0166] Step S1021: Calculate the stability index of the Apriltag identifier.

[0167] Stability Indicators The calculation is based on the unit quaternion corresponding to the ideal stable state of "zero rotation". For reference, the solution is obtained by using the dot product of the attitude quaternion and the unit quaternion:

[0168] ;

[0169] in: The closer the value is to 1, the more stable the pose and the higher the reliability of the pose data; the closer the value is to 0, the greater the pose disturbance and the lower the reliability of the pose data.

[0170] To further improve the accuracy of stability assessment, distance correction is performed by combining the location data of the markers: if the position of a certain Apriltag marker in the camera coordinate system... Distance to the optical center of the camera If the detection range exceeds the preset effective range (e.g., 0.5 m ~ 5 m), then its stability index will be adjusted. By multiplying by a penalty factor of 0.25, the weight of long-distance markers is reduced, enabling the stability index to comprehensively reflect both attitude stability and position effectiveness.

[0171] Step S1022: Calculate the interpolation coefficients.

[0172] interpolation coefficients The physical meaning is the weighted weight assigned to the pose data of each Apriltag identifier, and its value is related to the stability index of the Apriltag identifier. Positive correlation: The higher the stability index, the larger the interpolation coefficient, and the higher its contribution to the final interpolation calculation, thus suppressing the impact of large error data on the final result.

[0173] Before calculating the interpolation coefficients, effective pose data must be screened: first, stability indices must be eliminated. Pose data with a value less than 0.2 is considered unreliable and prone to introducing significant errors; therefore, a set of valid pose data is obtained and retained. ,in Includes the position of the corresponding Apriltag identifier in the camera coordinate system. With attitude quaternions .

[0174] The effective stability indices after screening are normalized to obtain the normalized interpolation coefficients corresponding to each identifier. :

[0175] ;

[0176] in and .

[0177] Step S1023: Calculate weighted interpolation mapping based on pose data.

[0178] Based on the above steps, the optical center mapping coordinates corresponding to the n valid identifiers are obtained. , … And combine it with the corresponding interpolation coefficients obtained in the previous step. , … Get the cursor coordinates in the screen coordinate system. :

[0179] ;

[0180] ;

[0181] The system will be on the screen The cursor position is displayed in real time, and the cursor responds accordingly to user selections, stroke inputs, and other interactive operations, ultimately achieving precise interaction between the mobile phone and the large screen.

[0182] Figure 5This is a specific application example of this embodiment in a circular curved screen interactive scenario. This application scenario further verifies the adaptability and practicality of this embodiment on large-scale splicing display devices. It can be directly applied to actual scenarios such as large-scale exhibitions and venue displays. The circular curved screen is composed of four display screens spliced ​​together to form a complete curved display surface to meet the needs of large-space, immersive display and interaction. Each display screen has a pixel resolution of 3840×2160, ensuring the high definition and integrity of the overall display image of the curved screen. The physical size of each screen is 9.60 meters × 3.37 meters. The circular curved screen formed by splicing the four screens has a curvature that adapts to the human visual observation angle, enabling a display and interactive experience with no blind spots and high focus.

[0183] It should be noted that this invention employs a local flattening processing strategy. Although the curved screen presents a curved surface overall, considering the relatively small field of view of the mobile phone camera, the system approximates the local area of ​​the curved screen captured by the mobile phone within its current field of view as a plane. Based on this assumption, the core calculation process can be performed using the method of this embodiment. Simultaneously, to eliminate the cumulative error caused by the global curvature, the system performs targeted adaptation processing for the overall curved surface characteristics of the circular curved screen: by pre-calibrating the arc geometric parameters of each spliced ​​screen of the curved screen, the coordinates of the mobile phone's optical center mapped onto the screen coordinate system are corrected using spatial surface compensation and displayed on the screen.

[0184] In summary, to provide users with a more intuitive and richer experience in large-screen exhibitions, this embodiment offers a collaborative interaction solution based on a smartphone and a large display screen. Through real-time linkage between the smartphone and the large display screen, interactive functions such as real-time positioning and target selection are achieved. Simultaneously, it supports mapping the phone's location trajectory to the large display screen, enabling personalized interactive operations such as drawing and doodling. Ultimately, this provides users with a more intuitive, richer, and immersive exhibition interaction experience, effectively overcoming the shortcomings of existing technologies. Unlike previous large-screen interaction methods, this embodiment proposes a method that does not require additional interactive devices. Users can perform positioning and selection interactions on the same screen simply by holding their personal smartphone, ensuring consistency between input and output, and users do not need to wear any additional devices.

[0185] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0186] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An interaction method based on pointing a large display screen on a smartphone, characterized in that, Includes the following steps: Several pre-set icons will be embedded into the exhibition screen of the large display screen based on the background characteristics of the current content being displayed; The system acquires image data captured by a smartphone and performs preprocessing steps, including denoising, grayscale adjustment, and correction. Based on the preprocessed image data, the identification and decoding operations are performed to extract the unique encoding information corresponding to the identification and the pose data in the camera coordinate system. Based on the extracted pose data in the camera coordinate system of each marker and the corresponding pixel coordinates of the marker on the display screen, a spatial mapping operation is performed to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system; interpolation calculation is performed on the candidate projection points to calculate the final mapping coordinates of the optical center of the smartphone on the display screen. In response to a smartphone pointing at a large display screen, the interactive icons on the large display screen are controlled to move in real time according to the final mapped coordinates. In response to a user's instruction to select or control information or programs on the large display screen based on the final mapped coordinates, the selection or control is executed. The process of calculating multiple candidate projection points of the optical center of the smartphone in the screen coordinate system by performing spatial mapping operations based on the extracted pose data of each marker in the camera coordinate system and the corresponding pixel coordinates of the marker on the display screen includes: for a single marker, converting the pose quaternion of the marker in the camera coordinate system into a rotation matrix representation. The rotation matrix of the smartphone relative to the intermediate coordinate system for The transpose of the matrix is ​​used to calculate the rotation angles of the intermediate coordinate system relative to the X, Y, and Z axes of the camera coordinate system using trigonometric functions. , , We obtain the rotation transformation matrix M from the camera coordinate system to the intermediate coordinate system, where the origin of the intermediate coordinate system coincides with the origin of the camera coordinate system, and the coordinate axes are parallel to the world coordinate system. The position of the marker in the camera coordinate system is rotated and transformed using the rotation transformation matrix M to obtain the position of the marker in the intermediate coordinate system. Based on the pixel size of the marker in the screen coordinate system and the actual size in the world coordinate system, calculate the mapping ratio coefficient. Based on the preset pixel position of the marker in the screen coordinate system, its position in the intermediate coordinate system, and the mapping ratio coefficient, calculate the mapping point of the camera optical center in the screen coordinate system. Based on the distance between the camera and the screen in the intermediate coordinate system, the displacement of the intersection point of the camera coordinate system, the intermediate coordinate system z-axis, and the world coordinate system XY plane is calculated, and then the actual mapped position of the camera optical center in the screen coordinate system is obtained. The process of performing spatial mapping calculations based on the extracted pose data of each marker in the camera coordinate system and the corresponding pixel coordinates of the marker on the display screen to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system includes: for multiple markers, taking the unit quaternion corresponding to the ideal stable state with zero rotation as a reference, the stability index of the corresponding marker is obtained by solving the dot product of the pose quaternion and the unit quaternion. The stability index is normalized to obtain the normalized interpolation coefficients corresponding to each identifier; The coordinates of the cursor in the screen coordinate system of the display screen are calculated by weighting the coordinates of the optical center mapping corresponding to all valid identifiers and their corresponding interpolation coefficients. The cursor is displayed at the coordinates of the cursor on the screen in response to user interaction.

2. The interaction method based on pointing a large display screen of a smartphone as described in claim 1, characterized in that, The process of pre-setting the signs includes: setting the color, transparency, size, and distribution area of ​​the AprilTag signs based on the background content currently displayed on the large display screen, to ensure that the signs do not cover the planned exhibition content, do not affect the visual experience, and that the smartphone camera can effectively extract each sign in the planned physical space.

3. The interaction method based on pointing a large display screen of a smartphone as described in claim 1, characterized in that, The process of detecting and decoding the identifiers based on the preprocessed image data includes: using an edge detection algorithm to extract edge pixels, and then using line segment fitting and contour tracking algorithms to find all potential candidate regions for the corresponding identifiers; Encoding parsing and recognition are performed. For the candidate quadrilateral regions that pass the screening, the quadrilateral regions are transformed by perspective and corrected to standard squares. Then, adaptive binarization is performed to obtain a black and white coded pattern. The corresponding coded information is extracted from the processed image for decoding. After successful decoding, the family and unique ID of the corresponding identifier are recorded.

4. The interaction method based on pointing a large display screen of a smartphone as described in claim 1, characterized in that, The pose data includes three-dimensional position and pose quaternions identified in the camera coordinate system.

5. The interaction method based on pointing a large display screen of a smartphone as described in claim 1, characterized in that, Before normalizing the stability index, the process also includes: removing pose data with stability indices less than a set value to obtain a set of effective pose data. Each element in the set contains the position and attitude quaternions of the corresponding identifier in the camera coordinate system. The stability index of the effective pose data is then normalized.

6. The interaction method based on pointing a large display screen of a smartphone as described in claim 5, characterized in that, Before the process of eliminating pose data with stability indices less than a set value, the process also includes: performing distance correction on the calculated stability indices. If the distance from the position of a certain marker in the camera coordinate system to the camera optical center exceeds the preset effective recognition range, its stability index is multiplied by a predetermined penalty coefficient to reduce the weight of markers that exceed the set distance.

7. An interaction method based on pointing a large display screen of a smartphone as described in any one of claims 1-6, characterized in that, The identifier is the AprilTag identifier.

8. An interactive system based on a large display screen of a smartphone, employing the method of claim 1, characterized in that, include: The image embedding module is used to embed several pre-set icons, based on the background features of the currently displayed content, into the exhibition screen of a large display screen; The image acquisition and processing module is used to acquire image data captured by the smartphone and perform preprocessing such as noise reduction, grayscale adjustment and correction on it in sequence. The image recognition module is used to perform identification and decoding operations on the preprocessed image output by the image acquisition and processing module, and to extract the unique encoding information corresponding to the identification and the pose data in the camera coordinate system. The interpolation mapping module is used to perform spatial mapping operations based on the pose data of each marker in the camera coordinate system extracted by the image recognition module and the pixel coordinates of the corresponding marker on the display screen to solve for multiple candidate projection points of the optical center of the smartphone in the screen coordinate system; and to perform interpolation calculations on the candidate projection points to calculate the final mapped coordinates of the optical center of the smartphone on the display screen. The human-computer interaction module is used to respond to the operation of pointing at the large display screen on the smartphone, control the interactive icons on the large display screen to move in real time according to the final mapping coordinates, and execute the selection or control in response to the user's instruction to select or control information or programs on the large display screen based on the final mapping coordinates.

Images