Smart pen cursor display method and device, storage medium and electronic equipment

Abstract

The invention discloses a smart pen cursor display method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining an infrared image of display equipment corresponding to a smart pen, and enabling the display equipment to be provided with at least three infrared positioning points; determining a spatial three-dimensional coordinate of the intelligent pen based on the image two-dimensional coordinate of each infrared positioning point in the infrared image; and calculating a screen two-dimensional coordinate corresponding to the smart pen based on the spatial three-dimensional coordinate, so that the display device displays a cursor corresponding to the smart pen. As the image processing difficulty of the infrared image is low, the spatial three-dimensional coordinate of the intelligent pen for shooting the image can be obtained with less computing power according to the image two-dimensional coordinate of each infrared positioning point in the infrared image, and the spatial three-dimensional coordinate can be converted to determine the screen two-dimensional coordinate of the cursor of the intelligent pen in the display equipment.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a smart pen cursor display method, device, storage medium and electronic device. Background Art

[0002] With the rapid development of electronic devices with display screens, smart pen devices that are paired with display devices have been widely used in daily life. In the process of using a display device with a smart pen, it involves the recognition and display of the indicator cursor of the position pointed by the smart pen. The accuracy of the positioning of the smart pen cursor will directly affect the indication effect and writing effect of the smart pen in the display device. Therefore, a method that can accurately display the position of the smart pen cursor is needed. Summary of the invention

[0003] The present application provides a smart pen cursor display method, device, storage medium and electronic device, which can solve the technical problem of inaccurate display position of the cursor on the screen in the related art.

[0004] In a first aspect, an embodiment of the present application provides a smart pen cursor display method, which is applied to a smart pen. The method includes:

[0005] Acquire an infrared image of a display device corresponding to the smart pen, where the display device is equipped with at least three infrared positioning points;

[0006] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0007] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the two-dimensional screen coordinates are sent to a display device so that the display device displays a cursor corresponding to the smart pen based on the two-dimensional screen coordinates; or, the three-dimensional spatial coordinates are sent to a display device so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0008] In a second aspect, an embodiment of the present application provides a smart pen cursor display method, which is applied to a display device, where the display device is equipped with at least three infrared positioning points, and the method includes:

[0009] Receive an infrared image sent by a smart pen corresponding to a display device, where the infrared image is an image obtained by photographing the display device by the smart pen, and determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image; or receive the spatial three-dimensional coordinates sent by the smart pen corresponding to the display device, where the spatial three-dimensional coordinates are coordinates determined by the smart pen based on the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device, where the infrared positioning points are installed on the display device;

[0010] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the cursor corresponding to the smart pen is displayed based on the two-dimensional screen coordinates.

[0011] In a third aspect, an embodiment of the present application provides a smart pen, which includes: an infrared camera, a system-level chip, and a wireless communication module, wherein:

[0012] An infrared camera, used to obtain an infrared image of a display device corresponding to the smart pen, the display device being installed with at least three infrared positioning points;

[0013] The system-level chip is connected to the infrared camera and the wireless communication module respectively, and is used to determine the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, and calculate the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates;

[0014] A wireless communication module, used to send the two-dimensional coordinates of the screen to the display device, so that the display device displays a cursor corresponding to the smart pen based on the two-dimensional coordinates of the screen;

[0015] Alternatively, the wireless communication module is used to send the three-dimensional spatial coordinates to the display device, so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0016] In a fourth aspect, an embodiment of the present application provides a display device, the display device comprising:

[0017] Each infrared positioning point is used to emit infrared light;

[0018] A wireless communication module, used for receiving the two-dimensional screen coordinates sent by the smart pen corresponding to the display device, where the two-dimensional screen coordinates are coordinates determined by the smart pen based on the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device;

[0019] A control system, connected to the wireless communication module, for displaying a cursor corresponding to the smart pen based on the two-dimensional coordinates of the screen;

[0020] or,

[0021] A wireless communication module, used for receiving the three-dimensional spatial coordinates sent by the smart pen corresponding to the display device, the three-dimensional spatial coordinates being the coordinates determined by the smart pen according to the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device;

[0022] The control system is connected to the wireless communication module and is used to calculate the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and to display the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0023] In a fifth aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the above-mentioned method.

[0024] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is suitable for being loaded by the processor and executing the steps of the above-mentioned method.

[0025] In a seventh aspect, an embodiment of the present application provides a smart pen cursor display system, comprising: a smart pen and a display device; wherein:

[0026] Smart pens are used for:

[0027] Acquire an infrared image of a display device corresponding to the smart pen, where the display device is equipped with at least three infrared positioning points;

[0028] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0029] Calculating the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and sending the two-dimensional screen coordinates to the display device;

[0030] Display devices are used for:

[0031] The cursor corresponding to the smart pen is displayed based on the two-dimensional coordinates of the screen.

[0032] In an eighth aspect, an embodiment of the present application provides a smart pen cursor display system, comprising: a smart pen and a display device; wherein:

[0033] Smart pens are used for:

[0034] Acquire an infrared image of a display device corresponding to the smart pen and send it to the display device;

[0035] Display devices are used for:

[0036] Receiving an infrared image sent by a smart pen corresponding to a display device, where the infrared image is obtained by photographing the display device by the smart pen, and the display device is equipped with at least three infrared positioning points;

[0037] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0038] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the cursor corresponding to the smart pen is displayed based on the two-dimensional screen coordinates.

[0039] In a ninth aspect, an embodiment of the present application provides a smart pen cursor display system, comprising: a smart pen and a display device; wherein:

[0040] Smart pens are used for:

[0041] Acquire an infrared image of a display device corresponding to the smart pen, where the display device is equipped with at least three infrared positioning points;

[0042] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0043] Sending the spatial three-dimensional coordinates to a display device;

[0044] Display devices are used for:

[0045] Receiving a spatial three-dimensional coordinate sent by a smart pen corresponding to a display device, the spatial three-dimensional coordinate is a coordinate determined by the smart pen according to the image two-dimensional coordinates of at least three infrared positioning points in an infrared image of the display device, the infrared positioning points being installed on the display device;

[0046] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the cursor corresponding to the smart pen is displayed based on the two-dimensional screen coordinates.

[0047] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0048] The present application provides a smart pen cursor display method, which is applied to a smart pen, obtains an infrared image of a display device corresponding to the smart pen, and the display device is equipped with at least three infrared positioning points; determines the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image; calculates the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates, and sends the screen two-dimensional coordinates to the display device, so that the display device displays the cursor corresponding to the smart pen based on the screen two-dimensional coordinates. Since the infrared positioning point and other objects can be directly distinguished and displayed in the infrared image, the spatial three-dimensional coordinates of the smart pen that shoots the image can be obtained according to the image two-dimensional coordinates of each infrared positioning point in the infrared image, and the spatial three-dimensional coordinates are further converted to determine the screen two-dimensional coordinates of the smart pen cursor in the display device. The image processing difficulty of the infrared image is low, so the calculation efficiency of the cursor two-dimensional coordinates can be improved and accurate smart pen cursor positioning can be completed with less computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 An exemplary system architecture diagram of a smart pen cursor display method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of an interactive process of a smart pen cursor display method provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the positions of infrared positioning points of a display device provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of an infrared image of a display device provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of an interactive process of a smart pen cursor display method provided in an embodiment of the present application;

[0055] Figure 6 A flowchart of a smart pen cursor display method provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a module architecture of a smart pen and a display device provided in an embodiment of the present application;

[0057] Figure 8 A correction effect diagram of a two-dimensional screen coordinate on a motion trajectory provided in an embodiment of the present application;

[0058] Fig. 9 A schematic diagram of an interactive process of a smart pen cursor display method provided in an embodiment of the present application;

[0059] Fig.10 A schematic diagram of an interactive process of a smart pen cursor display method provided in an embodiment of the present application;

[0060] Fig.11 A schematic diagram of the structure of a smart pen provided in an embodiment of the present application;

[0061] Fig.12 A schematic diagram of the structure of a display device provided in an embodiment of the present application;

[0062] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0064] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0065] Traditional human-computer interaction technology relies heavily on external input devices, such as mouse, keyboard, display screen, etc. The limitations of these hardware interaction devices can no longer meet users' more intelligent interaction needs. The stylus pen can replace the traditional external mouse and keyboard to complete the input function due to its advantages such as portability, lightness, and compliance with user writing habits. However, the conventional stylus pen needs to contact the screen of the display device to determine the input content, which limits the user's range of action to the screen. In some writing scenarios, users may need to indicate or input the cursor even when they are not touching the display device. For example, in lectures, meetings, classrooms, etc., users may walk to other locations in the venue or classroom during the explanation and presentation. At this time, in order to meet the user's needs for indicating and writing on non-contact touch devices, a new human-computer interaction method, air writing, has emerged.

[0066] As a new means of human-computer interaction, writing in the air allows users to control smart devices (such as smart writing pen devices) to write in the air without touching other external devices. The display device can identify the writing trajectory in the air to obtain the writing data that the user expects to input. In the process of using the display device with the smart pen, the positioning accuracy of the indicator cursor at the position pointed by the smart pen will directly affect the indication effect and writing effect of the smart pen in the display device. In the common positioning and display scheme of the smart pen cursor, a gyroscope component is generally set inside the smart pen, integrating an acceleration sensor, a gravity sensor, etc., to identify the relative motion trajectory of the smart pen and display the motion trajectory of the cursor. However, such relative position recognition can only determine the relative motion state of the smart pen from the end point to the starting point of the motion in one motion. Among them, the absolute position of the smart pen pointer cursor cannot be reflected. For example, when the smart pen is pointed to the left side of the display device, the position of the cursor may appear on the right side of the display device, which causes a bad user experience.

[0067] Therefore, an embodiment of the present application provides a smart pen cursor display method to solve the above-mentioned technical problem of inaccurate display position of the cursor on the screen.

[0068] See also Figure 1 , Figure 1 An exemplary system architecture diagram of a smart pen cursor display method provided in an embodiment of the present application.

[0069] like Figure 1 As shown, the system architecture may include a smart pen 101, a network 102, and a display device 103. The network 102 is used to provide a medium for a communication link between the smart pen 101 and the display device 103. The network 102 may include various types of wired communication links or wireless communication links, for example: the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, a microwave communication link, an ultra-wideband wireless communication link, and the like.

[0070] The smart pen 101 can interact with the display device 103 through the network 102 to receive messages from the display device 103 or send messages to the display device 103, or the smart pen 101 can interact with the display device 103 through the network 102 to receive messages or data sent by other users to the display device 103. Specifically, the smart pen 101 is equipped with an infrared camera for capturing infrared images.

[0071] The display device 103 may be a business display device that provides various services. It should be noted that the display device 103 may be hardware or software. When the display device 103 is hardware, it may be implemented as a distributed display device cluster consisting of multiple display devices, or it may be implemented as a single display device. When the display device 103 is software, it may be implemented as multiple software or software modules (for example, used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here. Specifically, at least three infrared positioning points are installed in the display device 103, and the installation positions of all infrared positioning points in the display device 103 are not in the same straight line, wherein each infrared positioning point is respectively installed at a different vertex of the display device 103, or each infrared positioning point is installed behind the display screen of the display device 103. In the display screen, the diaphragm and the liquid crystal layer are a stacked structure. When each infrared positioning point is installed behind the display screen of the display device 103, the infrared light of each infrared positioning point penetrates the diaphragm and the liquid crystal layer and then transmits out to support being photographed by an infrared camera.

[0072] In an embodiment of the present application, the smart pen 101 can obtain an infrared image of the display device 103 corresponding to the smart pen 101, and the display device 103 is installed with at least three infrared positioning points; then, the smart pen 101 determines the spatial three-dimensional coordinates of the smart pen 101 based on the image two-dimensional coordinates of each infrared positioning point in the infrared image; finally, the smart pen 101 calculates the screen two-dimensional coordinates corresponding to the smart pen 101 based on the spatial three-dimensional coordinates, and sends the screen two-dimensional coordinates to the display device 103, so that the display device 103 displays the cursor corresponding to the smart pen 101 based on the screen two-dimensional coordinates; or the smart pen 101 sends the spatial three-dimensional coordinates to the display device 103, so that the display device 103 calculates the screen two-dimensional coordinates corresponding to the smart pen 101 based on the spatial three-dimensional coordinates, and displays the cursor corresponding to the smart pen 101 based on the screen two-dimensional coordinates.

[0073] In another embodiment of the present application, the smart pen 101 can obtain an infrared image of the display device 103 corresponding to the smart pen 101, and send the infrared image to the display device 103, where the display device 103 is equipped with at least three infrared positioning points; the display device 103 receives the infrared image sent by the smart pen 101; the display device 103 determines the spatial three-dimensional coordinates of the smart pen 101 based on the image two-dimensional coordinates of each infrared positioning point in the infrared image; or the display device 103 directly receives the spatial three-dimensional coordinates sent by the smart pen 101; further, the display device 103 calculates the screen two-dimensional coordinates corresponding to the smart pen 101 based on the spatial three-dimensional coordinates; finally, the display device 103 displays the cursor corresponding to the smart pen 101 based on the screen two-dimensional coordinates.

[0074] In another embodiment of the present application, the smart pen 101 can obtain an infrared image of the display device 103 corresponding to the smart pen 101, and the display device 103 is installed with at least three infrared positioning points; then, the smart pen 101 determines the spatial three-dimensional coordinates of the smart pen 101 based on the two-dimensional image coordinates of each infrared positioning point in the infrared image; the smart pen 101 sends the spatial three-dimensional coordinates to the display device 103; the display device 103 receives the spatial three-dimensional coordinates sent by the smart pen 101 corresponding to the display device 103; then, the display device 103 calculates the two-dimensional screen coordinates corresponding to the smart pen 101 based on the spatial three-dimensional coordinates, and displays the cursor corresponding to the smart pen 101 based on the two-dimensional screen coordinates.

[0075] It should be understood that Figure 1 The number of smart pens, networks, and display devices in the figure is only illustrative, and any number of smart pens, networks, and display devices may be used according to implementation requirements.

[0076] In the embodiment of the present application, it is necessary to convert the two-dimensional image coordinates of the positioning point in the image captured by the smart pen into three-dimensional spatial coordinates, and then calculate the absolute position of the smart pen in space that captures the image, so as to accurately locate the smart pen pointer on the display device. Among them, the calculation process of converting the two-dimensional image coordinates into three-dimensional spatial coordinates can be completed by the smart pen or by the display device.

[0077] Next, combine Figure 2 Introduction, when the smart pen converts the two-dimensional coordinates of the image into the three-dimensional coordinates of the space, the smart pen specifically performs the following steps in the smart pen cursor display method.

[0078] See also Figure 2 , Figure 2 A schematic diagram of an interactive process of a smart pen cursor display method provided in an embodiment of the present application. The execution subject of the embodiment of the present application can be an electronic device that executes the smart pen cursor display, or a processor in an electronic device that executes the smart pen cursor display method, or a smart pen cursor display service in an electronic device that executes the smart pen cursor display method. For the convenience of description, the specific execution process of the smart pen cursor display method is introduced below by taking the execution subject being a processor in an electronic device as an example.

[0079] like Figure 2 As shown, the smart pen cursor display method, applied to the smart pen, may at least include:

[0080] S202: Acquire an infrared image of a display device corresponding to the smart pen, where the display device is installed with at least three infrared positioning points.

[0081] Optionally, relative position recognition can only determine the relative motion state of the smart pen from the end point to the starting point of a motion, while absolute position recognition can determine the three-dimensional absolute position of the smart pen in space. Therefore, when locating the position of the smart pen's pointer on the display screen, it is necessary to accurately identify the absolute position of the smart pen and the pointing direction of the smart pen. Then, starting from the absolute position of the smart pen, the intersection of the pointing direction and the display screen can be determined according to its pointing direction, which is the accurate cursor indication position of the smart pen.

[0082] Optionally, the method of absolute position recognition is generally to capture the movement between the smart pen and the display device through a camera device, and the device equipped with the camera device captures the image of the other device, and by identifying the two-dimensional coordinates of the device in the image, the absolute position of the smart pen in space is deduced according to the absolute position calibrated by the display device itself, and the movement position of the smart pen in the air is determined. However, this solution has high requirements on the pixels of the camera device and the computing power of the processor for image and video processing, and the cost is relatively high. In addition, the structural space of the smart pen is limited, and it is difficult to popularize the application of high-computing-power processors.

[0083] Optionally, when the smart pen and the display device are in use, the position of the display device is usually fixed. Then, when the three-dimensional coordinates of the smart pen in space can be determined by the relative positions of the smart pen and the display device, the relative positions of the smart pen and the display device can be reflected in the image of the display device taken by the smart pen, that is, according to the position of the display device on the image, the position of the camera that took the image, that is, the position of the smart pen with the camera installed can be inferred. In this process, in order to reduce the computing power required for image processing, the image only needs to directly reflect the position of the display device, that is, to distinguish the display device from other objects in the space, and the specific objects, colors, pixel clarity, etc. in the environment do not need to be too high. In this way, the accurate position of the smart pen can be obtained through simple image processing methods.

[0084] Furthermore, infrared cameras are usually equipped with infrared filters, such as IR filters, which can filter out most of the visible light when collecting images. In the collected images, objects emitting infrared light in the environment will be clearly distinguished from other objects. Based on this, the display device can be installed with infrared positioning points. The position of the display device can be determined according to the installation position of the infrared positioning points on the display device. Since the spatial three-dimensional coordinates of the smart pen need to be calculated, there should be no less than three infrared positioning points, and all infrared positioning points cannot form a straight line. At the same time, an infrared camera including at least an infrared filter is installed in the smart pen to capture the infrared image of the display device.

[0085] It should be noted that the infrared positioning point can be an infrared positioning light plate. For the convenience of calculation, please refer to Figure 3 , Figure 3 A schematic diagram of the positions of infrared positioning points of a display device provided in an embodiment of the present application. In a feasible implementation, taking the existence of four infrared positioning points, namely infrared positioning point 1, infrared positioning point 2, infrared positioning point 3, and infrared positioning point 4 as an example, the positions on the display device can be as follows: Figure 3 (A), at the four corners of the display screen of the display device, or as Figure 3As shown in (B), it is located behind the display screen of the display device. In order not to affect the display of the display screen, it can be located at the same position as the LED light bar, so that the infrared light can penetrate the film and liquid crystal of the display screen and be transmitted out.

[0086] Optionally, see Figure 4 , Figure 4 This is a schematic diagram of an infrared image of a display device provided in an embodiment of the present application. Figure 4 As shown in the figure, the infrared image corresponding to the display device is obtained using a smart pen. In the infrared image, because other visible light except infrared light is filtered, it appears dim, while the infrared positioning point appears bright in the image because it emits red light. Objects that emit infrared light and those that do not emit infrared light are clearly distinguished in the image. If a clear and complete image is required, the light and background of the acquisition environment will have higher requirements. When acquiring infrared images, the infrared positioning point can also be accurately acquired in an environment with dim light and complex background environment.

[0087] S204: Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image.

[0088] Optionally, the area of ​​each infrared positioning point in the infrared image is clearly distinguishable from other areas, so it is easy to determine the two-dimensional image coordinates of each infrared positioning point. At this time, if the real three-dimensional coordinates of each infrared positioning point in space are known, the spatial three-dimensional coordinates of the camera that took the image can be directly inferred through the mapping conversion of the two-dimensional image coordinates of each infrared positioning point to the real three-dimensional coordinates, and then the spatial three-dimensional coordinates of the smart pen equipped with a camera can be obtained.

[0089] S206, calculating the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and sending the two-dimensional screen coordinates to the display device, so that the display device displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0090] Optionally, since the coordinates of the cursor are two-dimensional coordinates for the display device when it is displayed on the screen of the display device, after determining the three-dimensional spatial coordinates of the smart pen, it is necessary to further determine the two-dimensional screen coordinates required for displaying the cursor in the display device based on the three-dimensional spatial coordinates. In addition, when displaying the cursor of the smart pen, the two-dimensional screen coordinates need to be sent to the display device so that the display device can obtain the position information of the smart pen and display the position indicated by the cursor of the smart pen based on this. The communication method used when sending the two-dimensional screen coordinates to the display device can be a common data transmission method such as Bluetooth, Wi-Fi, etc. Considering cost control, in a feasible implementation, there is Bluetooth communication between the smart pen and the display device, and the smart pen sends the two-dimensional screen coordinates to the display device via the Bluetooth communication link. The display device receives the two-dimensional screen coordinates via Bluetooth communication and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0091] Furthermore, considering the flexible settings of various display devices such as the size of the display device, the size of the display screen, and the display settings, in order to enable flexible adaptation between multiple display devices and multiple smart pens, when the display device receives the two-dimensional screen coordinates, it can perform adaptive display conversion processing on the two-dimensional screen coordinates according to its own device settings and display parameters, obtain the screen display coordinates corresponding to the smart pen, and display the cursor corresponding to the smart pen based on the screen display coordinates. This can enhance the adaptability between multiple display devices and multiple smart pens, make the use scenarios of display devices and smart pens more extensive, and enhance the user experience of using smart pens.

[0092] In an embodiment of the present application, a smart pen cursor display method is provided, which is applied to a smart pen, and an infrared image of a display device corresponding to the smart pen is obtained, and the display device is equipped with at least three infrared positioning points; based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, the spatial three-dimensional coordinates of the smart pen are determined; based on the spatial three-dimensional coordinates, the screen two-dimensional coordinates corresponding to the smart pen are calculated, and the screen two-dimensional coordinates are sent to the display device, so that the display device displays the cursor corresponding to the smart pen based on the screen two-dimensional coordinates. Since the infrared positioning point and other objects can be directly distinguished and displayed in the infrared image, the spatial three-dimensional coordinates of the smart pen that captured the image can be obtained according to the image two-dimensional coordinates of each infrared positioning point in the infrared image, and the spatial three-dimensional coordinates are further converted to determine the screen two-dimensional coordinates of the smart pen cursor in the display device. The image processing difficulty of the infrared image is low, so the calculation efficiency of the cursor two-dimensional coordinates can be improved, and accurate smart pen cursor positioning can be completed with less computing power.

[0093] In another possible embodiment, combining Figure 5When the display device converts the two-dimensional coordinates of the image into the three-dimensional coordinates of the space, the smart pen specifically performs the following steps in the smart pen cursor display method.

[0094] See also Figure 5 , Figure 5 A schematic diagram of the interactive process of a smart pen cursor display method provided in an embodiment of the present application.

[0095] like Figure 5 As shown, the smart pen cursor display method, applied to the smart pen, may at least include:

[0096] S502: Acquire an infrared image of a display device corresponding to the smart pen, where the display device is installed with at least three infrared positioning points.

[0097] Optionally, the smart pen acquires an infrared image of a display device corresponding to the smart pen, the display device is installed with at least three infrared positioning points, and the smart pen performs image processing and calculation of spatial three-dimensional coordinates.

[0098] S504: Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image.

[0099] Optionally, the smart pen determines the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image. It should be noted that the process and calculation method of determining the spatial three-dimensional coordinates of the smart pen can be the same as the processing principle and process in the smart pen mentioned above, and will not be repeated here.

[0100] S506. Send the three-dimensional spatial coordinates to the display device, so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0101] Optionally, after the spatial writing pen determines the spatial three-dimensional coordinates, the spatial three-dimensional coordinates are directly sent to the display device, so that the display device calculates the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates, and displays the cursor corresponding to the smart pen based on the screen two-dimensional coordinates. The display device calculates the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates. It should be noted that the process and calculation method of calculating the screen two-dimensional coordinates are the same as the processing principle and process in the smart pen mentioned above, and will not be repeated here. The process and calculation method of displaying the cursor corresponding to the smart pen are also the same as the processing principle and process in the smart pen mentioned above, and will not be repeated here.

[0102] In an embodiment of the present application, a smart pen cursor display method is provided, which is applied to a smart pen, and an infrared image of a display device corresponding to the smart pen is obtained, and the display device is equipped with at least three infrared positioning points; based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, the spatial three-dimensional coordinates of the smart pen are determined; the spatial three-dimensional coordinates are sent to the display device, so that the display device calculates the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates, and displays the cursor corresponding to the smart pen based on the screen two-dimensional coordinates. The smart pen obtains the infrared image and processes the infrared image to determine the spatial three-dimensional coordinates of the smart pen, and then sends the spatial three-dimensional coordinates to the display device, which performs subsequent data processing and displays the cursor position of the smart pen after data processing on its own display screen. The smart pen and the display device are each responsible for a part of the calculation, which increases the applicable scenarios of the smart pen and the display device.

[0103] See also Figure 6 , Figure 6 A flowchart of a smart pen cursor display method provided in an embodiment of the present application.

[0104] like Figure 6 As shown, the smart pen cursor display method, applied to the smart pen, may at least include:

[0105] S602: Acquire an infrared image of a display device corresponding to the smart pen, where the display device is equipped with at least three infrared positioning points.

[0106] Regarding step S602, please refer to the detailed description in step S202, which will not be repeated here.

[0107] S604: Based on the wireless positioning communication signal between the smart pen and the display device, receive display device information sent by the display device, where the display device information at least includes the real space coordinates corresponding to each infrared positioning point.

[0108] Optionally, as a hardware device, the smart pen relies on a processor installed therein to receive, send and process data. Considering the small size of the smart pen, a microcontroller unit (MCU), also known as a single chip microcomputer or a single chip microcomputer, can be installed in the smart pen as a main control chip to realize processor work such as data processing. Figure 7 , Figure 7 A schematic diagram of the module architecture of a smart pen and a display device provided in an embodiment of the present application. Figure 7As shown in the figure, the main control chip (MCU) of the smart pen is to appropriately reduce the frequency and specifications of the central processing unit (CPU), and integrate the memory, timer, USB, A / D conversion, UART, PLC, DMA and other peripheral interfaces, and even LCD drive circuits on a single chip to form a chip-level computer, which can perform different combination controls for different application scenarios and is suitable for smart pens with limited size specifications. Please continue to read Figure 7 There are wireless communication modules in both the smart pen and the display device for wireless communication, so that the smart pen can transmit coordinate data to the display device through the wireless communication link and display the cursor corresponding to the smart pen based on the coordinate data of the display device.

[0109] Optionally, see Figure 7 , the smart pen is equipped with an infrared camera. After the infrared camera collects the infrared image of the display screen, it is necessary to calculate the spatial three-dimensional coordinates of the camera that took the image based on the real three-dimensional coordinates of each infrared positioning point in space, through the mapping conversion of the image two-dimensional coordinates of each infrared positioning point to the real three-dimensional coordinates, and then obtain the spatial three-dimensional coordinates of the smart pen installed with the camera. When the display device is installed with the infrared positioning point, the position of the infrared positioning point is recorded in the device information of the display device. Therefore, the display device can be set to wirelessly connect with the matching smart pen after startup and send the display device information to the smart pen after successful connection, so that the smart pen can receive the display device information sent by the display device based on the wireless positioning communication signal between the display device and the display device.

[0110] Specifically, after the smart pen is paired with a display device for the first time, the display device information received for the first time can be stored. After each pairing is started, the real space coordinates of the infrared positioning point installed in the display device can be directly determined based on the device identification of the display device, such as device ID, device model, etc.

[0111] S606: Determine the three-dimensional spatial coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image and the real space coordinates corresponding to each infrared positioning point.

[0112] Optionally, based on the two-dimensional image coordinates of each infrared positioning point in the infrared image and the real space coordinates corresponding to each infrared positioning point, the three-dimensional spatial coordinates of the smart pen can be calculated. Figure 7 When performing image algorithm processing and coordinate calculation processing, the algorithm processing involved in the calculation can be integrated into the system-on-chip (SOC).

[0113] Alternatively, if Figure 7As shown, the smart pen may also include a switch button. When the user presses the switch button, the system-on-chip (SOC) of the smart pen senses the state change of the switch button, and learns the indication signal triggered by the switch button being pressed when the smart pen is in the sleep state. In response to the indication signal, the camera start request is transmitted to the main control chip (MCU), so that the main control chip controls the camera to collect images, and then the SOC obtains the infrared image and performs subsequent coordinate calculations based on the infrared image. Among them, the switch button is only used to wake up the working mode of the smart pen, so it is sufficient to press it once to trigger the indication signal, and it does not need to be kept pressed. That is, the switch button automatically rebounds after being pressed once. When the smart pen is in the working mode, if the switch button is pressed once, it can trigger the end of work signal. The smart pen enters the sleep mode based on the end of work signal to achieve the effect of shutting down the function. The simple button operation makes it easy for users to control the smart pen to achieve the functional requirements of indication, shutdown, etc.

[0114] Specifically, in a feasible embodiment, when calculating the spatial three-dimensional coordinates of the smart pen, the PnP (Perspective-n-Point) algorithm can be used. The PnP (Perspective-n-Point) algorithm is a method for solving the correspondence between 3D and 2D points. It describes how to estimate the position and posture of the camera when n 3D spatial points and their positions are known. If the 3D position of a feature point in two images is known, at least 3 points are needed to calculate the movement of the camera. That is, according to the PnP algorithm, the spatial three-dimensional coordinates of the smart pen can be calculated through the conversion mapping between the image two-dimensional coordinates of each infrared positioning point and the real space coordinates corresponding to each infrared positioning point.

[0115] S608. Acquire the posture data of the smart pen through the posture calculation module, and determine the pointing direction of the smart pen according to the posture data.

[0116] Optionally, after obtaining the spatial three-dimensional coordinates of the smart pen, it is also necessary to convert them into two-dimensional screen coordinates on the two-dimensional display screen. When determining the two-dimensional screen coordinates of the smart pen, it is necessary to determine the pointing direction of the smart pen in order to determine the indicated position of the smart pen on the screen from its three-dimensional coordinate position. At this time, a posture calculation module can be set in the smart pen, and the posture calculation module at least includes an acceleration sensor to obtain the posture information of the smart pen, so that the posture data of the smart pen can be obtained according to the posture calculation module in the smart pen, and the pointing direction of the smart pen can be determined according to the posture data, wherein the posture data can include information such as the pointing direction and angle of the smart pen.

[0117] S610, calculating the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates and the indicated direction.

[0118] Optionally, after determining the pointing direction of the smart pen, spatial trigonometric functions are performed on the three-dimensional spatial coordinates to obtain the position where the pointing direction of the smart pen intersects with the plane of the smart tablet, which is the actual cursor pointing position of the smart pen, that is, the two-dimensional screen coordinates corresponding to the smart pen are obtained.

[0119] S612: Collect the motion trajectory of the smart pen based on the motion trajectory collection module, where the motion trajectory is the position change trajectory of the smart pen relative to the position of the motion end point collected last time.

[0120] Optionally, the above absolute spatial position calculation process is relatively long. When the communication distance between the smart pen and the display device is far, the overall process from data collection to processing the calculation results will increase, which means that the update frame rate of the absolute spatial position of the smart pen is low. Based on this, in order to smooth the display effect of the handwriting movement of the smart pen, a motion trajectory acquisition module can be added to the smart pen to collect the motion trajectory of the smart pen. The motion trajectory is the position change trajectory of the smart pen relative to the end point of the previous motion acquisition, that is, the position trajectory change based on the absolute spatial position. Please refer to Figure 7 The motion trajectory acquisition module reports the collected motion trajectory of the smart pen to the SOC chip of the smart pen. The SOC chip is responsible for image processing and also performs motion trajectory processing and calculation.

[0121] In a feasible embodiment, the motion trajectory acquisition module can be an inertial measurement unit (IMU), which can include at least multiple sensors such as accelerometers, gyroscopes, and geomagnetic sensors. The accelerometer detects the acceleration signals of the object in the three independent axes of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system, measures the angular velocity and acceleration of the object in three-dimensional space, and calculates the posture of the object. The geomagnetic sensor can more effectively determine the direction of the pen tip. The motion trajectory acquisition module has a small amount of data calculation and the update speed can reach milliseconds. In this way, when the absolute spatial position is not updated, the motion trajectory of the smart pen during this period can be captured, and the cursor position display effect of the complete smart pen can be completed.

[0122] Specifically, the gravity acceleration obtained by IMU. w and angular velocity v w , we can find the translation p of the IMU sensor in the global coordinate system at time t w , rotate R wb and speed.V w The numerical value is calculated as follows:

[0123] a w =R wb (a m -ba )+g;

[0124] v w (t+1)=v w (t)+a w dt;

[0125] R wb (t+1)=R wb (t)exp((w m -b g )dt);

[0126]

[0127] S614: Send the motion trajectory to the display device, so that the display device moves the cursor corresponding to the smart pen according to the motion trajectory.

[0128] Optionally, see Figure 7 The main control chip of the smart pen sends the motion trajectory to the display device through wireless communication, so that the control system of the display device receives the motion trajectory sent by the smart pen based on the wireless communication module, and displaces the display cursor corresponding to the smart pen according to the motion trajectory, thereby supplementing the motion trajectory of the smart pen, making its displacement effect smoother and improving the user experience.

[0129] S616: Refresh the two-dimensional coordinates of the screen at a first frequency and refresh the motion trajectory at a second frequency, where the first frequency is lower than the second frequency.

[0130] Optionally, due to the characteristics of the gyroscope in the motion trajectory acquisition module, only the parameters of gravity acceleration and angular velocity can be detected, and the cumulative error of these two parameters will cause the cursor position of the smart pen to gradually deviate from the actual pointing position, and this error cannot be automatically corrected during use. Then in the embodiment of the present application, the absolute position of the smart pen can be accurately determined, and the landing point of the cursor's motion trajectory can be corrected based on the accurate two-dimensional screen coordinates. The two-dimensional screen coordinates are obtained based on image processing, and their refresh frequency is lower than the refresh frequency of the motion trajectory. Therefore, the two-dimensional screen coordinates are refreshed at a first frequency and the motion trajectory is refreshed at a second frequency, and the first frequency is less than the second frequency.

[0131] S618. When the two-dimensional screen coordinates are refreshed, the position of the cursor displayed according to the motion trajectory is corrected based on the refreshed two-dimensional screen coordinates.

[0132] Optionally, when the screen 2D coordinates are refreshed, the cursor display position is corrected to the position corresponding to the screen 2D coordinates. If no new screen 2D coordinates are calculated, the cursor position is displayed according to the motion trajectory. Figure 8 , Figure 8This is a correction effect diagram of a two-dimensional screen coordinate on a motion trajectory provided by an embodiment of the present application. Figure 8 As shown, the black solid dots are the screen 2D coordinates corresponding to the smart pen cursor, the dotted hollow dots and the dotted lines are the motion tracks corresponding to the smart pen cursor, when the screen 2D coordinates are calculated, the motion track moves based on the screen 2D coordinates, and when the screen 2D coordinates are refreshed, the position of the cursor displayed according to the motion track is corrected according to the refreshed screen 2D coordinates. In this way, the accumulated errors of other sensors can be calibrated in time based on the screen 2D coordinates corresponding to the absolute position of the smart pen in space, making the indicated position of the smart pen more accurate.

[0133] In an embodiment of the present application, a smart pen cursor display method is provided, which is applied to the smart pen. The spatial three-dimensional coordinates of the smart pen are calculated by conversion mapping between the image two-dimensional coordinates of each infrared positioning point and the real space coordinates corresponding to each infrared positioning point; the posture data of the smart pen is obtained according to the posture calculation module in the smart pen, and the indication direction of the smart pen is determined according to the posture data, and the screen two-dimensional coordinates corresponding to the smart pen are calculated based on the spatial three-dimensional coordinates and the indication direction. The image processing difficulty of the infrared image is low, so the calculation efficiency of the cursor two-dimensional coordinates can be improved and accurate smart pen cursor positioning can be completed with less computing power; the display device displaces the display cursor corresponding to the smart pen according to the motion trajectory, thereby supplementing the motion trajectory of the smart pen, making its displacement effect smooth, and improving the user experience; based on the two-dimensional screen coordinates corresponding to the absolute position of the smart pen in space, the accumulated errors of other sensors are calibrated in time, so that the indication position of the smart pen is more accurate.

[0134] In the embodiment of the present application, it is necessary to convert the two-dimensional image coordinates of the positioning point in the image captured by the smart pen into three-dimensional spatial coordinates, and then calculate the absolute position of the smart pen in space that captures the image, so as to accurately locate the smart pen pointer on the display device. Among them, the series of calculation processes of processing the image to obtain the two-dimensional image coordinates, the three-dimensional spatial coordinates, etc. can be completed by the smart pen or by the display device.

[0135] Next, combine Fig. 9 Introduction, when the display device processes the image and calculates the coordinates, the display device specifically performs the following steps in the smart pen cursor display method.

[0136] See also Fig. 9 , Fig. 9 A schematic diagram of the interactive process of a smart pen cursor display method provided in an embodiment of the present application.

[0137] like Fig. 9 As shown, the smart pen cursor display method, applied to a display device, may at least include:

[0138] S902: Receive an infrared image sent by a smart pen corresponding to the display device, where the infrared image is obtained by photographing the display device with the smart pen, and the display device is installed with at least three infrared positioning points.

[0139] Optionally, in the specific implementation process, the display device can also realize the calculation and processing of images and coordinates, and receive the infrared image sent by the smart pen corresponding to the display device. The infrared image is obtained by the smart pen shooting the display device. The display device is installed with at least three infrared positioning points.

[0140] S904: Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image.

[0141] Optionally, the display device can determine the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image. It should be noted that the process and calculation method of determining the spatial three-dimensional coordinates of the smart pen can be the same as the processing principle and process in the smart pen, which will not be repeated here.

[0142] S906. Calculate the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and display the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0143] Optionally, the display device can directly calculate the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates of the smart pen, so as to facilitate subsequent display. It should be noted that the process and method of calculating the two-dimensional screen coordinates can be the same as the processing principle and process in the smart pen mentioned above, and will not be repeated here. After that, the display device can directly display the cursor corresponding to the smart pen based on the two-dimensional screen coordinates, without the need to transmit coordinate data with the smart pen, thereby reducing data transmission delay and increasing the applicable scenarios of the smart pen and the display device.

[0144] In an embodiment of the present application, a smart pen cursor display method is provided, which is applied to a display device, receives an infrared image sent by a smart pen corresponding to the display device, the infrared image is obtained by the smart pen shooting the display device, and the display device is installed with at least three infrared positioning points; based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, the spatial three-dimensional coordinates of the smart pen are determined; based on the spatial three-dimensional coordinates, the screen two-dimensional coordinates corresponding to the smart pen are calculated, and the cursor corresponding to the smart pen is displayed based on the screen two-dimensional coordinates. The display device calculates and processes the image and coordinates, determines the spatial three-dimensional coordinates of the smart pen, directly processes the data, and displays the cursor position of the smart pen after the data processing on its own display screen, thereby reducing the data transmission delay and increasing the applicable scenarios of the smart pen and the display device.

[0145] Next, combine Fig.10When the display device converts the two-dimensional coordinates of the image into the three-dimensional coordinates of the space, the display device specifically performs the following steps in the smart pen cursor display method.

[0146] See also Fig.10 , Fig.10 A schematic diagram of the interactive process of a smart pen cursor display method provided in an embodiment of the present application.

[0147] like Fig.10 As shown, the smart pen cursor display method, applied to a display device, may at least include:

[0148] S1002, receiving the three-dimensional spatial coordinates sent by the smart pen corresponding to the display device, the three-dimensional spatial coordinates are coordinates determined by the smart pen based on the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device, and the infrared positioning points are installed on the display device.

[0149] Optionally, the display device may directly receive the three-dimensional spatial coordinates sent by the corresponding smart pen, wherein the three-dimensional spatial coordinates are coordinates determined by the smart pen based on the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device.

[0150] S1004. Calculate the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and display the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0151] Optionally, after receiving the three-dimensional spatial coordinates, the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates. The calculation process and method are not described in detail.

[0152] In an embodiment of the present application, a smart pen cursor display method is provided, which is applied to a display device, receives a spatial three-dimensional coordinate sent by a smart pen corresponding to the display device, the spatial three-dimensional coordinate is a coordinate determined by the smart pen according to the image two-dimensional coordinates of at least three infrared positioning points in the infrared image of the display device, and the infrared positioning points are installed on the display device; the screen two-dimensional coordinate corresponding to the smart pen is calculated based on the spatial three-dimensional coordinate, and the cursor corresponding to the smart pen is displayed based on the screen two-dimensional coordinate. The smart pen obtains and processes the infrared image to determine the spatial three-dimensional coordinate of the smart pen, and then sends the spatial three-dimensional coordinate to the display device, which performs subsequent data processing and displays the cursor position of the smart pen after data processing on its own display screen. The smart pen and the display device are each responsible for a part of the calculation, which increases the applicable scenarios of the smart pen and the display device.

[0153] See also Fig.11 , Fig.11 This is a schematic diagram of the structure of a smart pen provided in an embodiment of the present application. Fig.11As shown, the smart pen 1100 includes: an infrared camera 1110, a system-level chip 1120 and a wireless communication module 1130, including:

[0154] Infrared camera 1110, used to obtain an infrared image of a display device corresponding to the smart pen, the display device being installed with at least three infrared positioning points;

[0155] The system-level chip 1120 is connected to the infrared camera and the wireless communication module respectively, and is used to determine the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, and calculate the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates;

[0156] The wireless communication module 1130 is used to send the two-dimensional coordinates of the screen to the display device, so that the display device displays a cursor corresponding to the smart pen based on the two-dimensional coordinates of the screen;

[0157] Alternatively, the wireless communication module 1130 is used to send the three-dimensional spatial coordinates to the display device, so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0158] In an embodiment of the present application, a smart pen is provided, wherein an infrared camera is used to obtain an infrared image of a display device corresponding to the smart pen, and the display device is equipped with at least three infrared positioning points; a system-level chip is connected to the infrared camera and the wireless communication module respectively, and is used to determine the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, and calculate the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates; the wireless communication module is used to send the screen two-dimensional coordinates to the display device, so that the display device displays the cursor corresponding to the smart pen based on the screen two-dimensional coordinates. Since the infrared positioning points and other objects can be directly distinguished and displayed in the infrared image, the spatial three-dimensional coordinates of the smart pen that takes the image can be obtained according to the image two-dimensional coordinates of each infrared positioning point in the infrared image, and the spatial three-dimensional coordinates are further converted to determine the screen two-dimensional coordinates of the cursor of the smart pen in the display device. The image processing difficulty of the infrared image is low, so the calculation efficiency of the cursor two-dimensional coordinates can be improved and accurate smart pen cursor positioning can be completed with less computing power.

[0159] See also Fig.12 , Fig.12 This is a schematic diagram of the structure of a display device provided in an embodiment of the present application. Fig.12 As shown, the display device 1200 includes: at least three infrared positioning points 1210, a wireless communication module 1220 and a control system 1230, including:

[0160] Each infrared positioning point 1210 is used to emit infrared light;

[0161] The wireless communication module 1220 is used to receive the two-dimensional screen coordinates sent by the smart pen corresponding to the display device, where the two-dimensional screen coordinates are coordinates determined by the smart pen according to the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device;

[0162] A control system 1230, connected to the wireless communication module, is used to display a cursor corresponding to the smart pen based on the two-dimensional coordinates of the screen;

[0163] or,

[0164] The wireless communication module 1220 is used to receive the spatial three-dimensional coordinates sent by the smart pen corresponding to the display device, where the spatial three-dimensional coordinates are the coordinates determined by the smart pen according to the image two-dimensional coordinates of at least three infrared positioning points in the infrared image of the display device;

[0165] The control system 1230 is connected to the wireless communication module, and is used to calculate the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and display the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0166] Optionally, the installation positions of all infrared positioning points 1210 in the display device 1200 are not on the same straight line. When all infrared positioning points 1210 are not on the same straight line, it can be ensured that each adjacent positioning point can form a closed polygon after being connected, which is used as the basis of the spatial coordinate algorithm to realize the conversion calculation between two-dimensional coordinates and three-dimensional coordinates.

[0167] Optionally, each infrared positioning point 1210 is installed at a different vertex of the display device 1200. Furthermore, each infrared positioning point 1210 can also be installed at a different edge line of the display device 1200 to achieve flexible installation of the infrared positioning point 1210.

[0168] Optionally, each infrared positioning point 1210 is installed behind the display screen of the display device 1200, and the display screen includes a membrane and a liquid crystal layer; wherein the infrared light of each infrared positioning point is the light transmitted after penetrating the membrane and the liquid crystal layer. In addition, each infrared positioning point 1210 can also be installed on the back of the display screen, so that the infrared light penetrates the membrane and the liquid crystal layer of the display screen and is transmitted.

[0169] In an embodiment of the present application, a display device is provided, each infrared positioning point is used to emit infrared light; a wireless communication module is used to receive the screen two-dimensional coordinates sent by the smart pen corresponding to the display device, and the screen two-dimensional coordinates are the coordinates determined by the smart pen according to the image two-dimensional coordinates of at least three infrared positioning points in the infrared image of the display device; a control system is connected to the wireless communication module and is used to display the cursor corresponding to the smart pen based on the screen two-dimensional coordinates. The display device calculates and processes the image and coordinates, determines the spatial three-dimensional coordinates of the smart pen, and directly processes the data, and displays the cursor position of the smart pen after the data processing on its own display screen, thereby reducing the data transmission delay and increasing the applicable scenarios of the smart pen and the display device.

[0170] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0171] See also Fig.13 , Fig.13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.13 As shown, the electronic device 1300 may include: at least one electronic device processor 1301 , at least one network interface 1304 , a user interface 1303 , a memory 1305 , and at least one communication bus 1302 .

[0172] The communication bus 1302 is used to realize the connection and communication between these components.

[0173] Among them, the user interface 1303 may include a display screen (Display) and a camera (Camera). The optional user interface 1303 may also include a standard wired interface and a wireless interface. In particular, when the electronic device 1300 is a smart pen, the camera may be an infrared camera, and the infrared camera at least includes an infrared filter.

[0174] The network interface 1304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0175] Among them, the electronic device processor 1301 may include one or more processing cores. The electronic device processor 1301 uses various interfaces and lines to connect various parts within the entire electronic device 1300, and executes various functions and processes data of the electronic device 1300 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1305, and calling data stored in the memory 1305. Optionally, the electronic device processor 1301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The electronic device processor 1301 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the electronic device processor 1301, and it can be implemented separately through a chip.

[0176] Among them, the memory 1305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 1305 includes a non-transitory computer-readable storage medium. The memory 1305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1305 may also be optionally at least one storage device located away from the aforementioned electronic device processor 1301. As Fig.13 As shown, the memory 1305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a smart pen cursor display program.

[0177] When the electronic device 1300 is a smart pen, Fig.13 In the electronic device 1300 shown, the user interface 1303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the electronic device processor 1301 can be used to call the smart pen cursor display program stored in the memory 1305, and specifically perform the following operations:

[0178] Acquire an infrared image of a display device corresponding to the smart pen, where the display device is equipped with at least three infrared positioning points;

[0179] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0180] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the two-dimensional screen coordinates are sent to the display device, so that the display device displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0181] In some embodiments, before the electronic device processor 1301 determines the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, it also specifically performs the following steps: based on the wireless positioning communication signal between the smart pen and the display device, receiving the display device information sent by the display device, the display device information includes at least the real space coordinates corresponding to each infrared positioning point; when the electronic device processor 1301 determines the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, it specifically performs the following steps: based on the image two-dimensional coordinates of each infrared positioning point in the infrared image and the real space coordinates corresponding to each infrared positioning point, determine the spatial three-dimensional coordinates of the smart pen.

[0182] In some embodiments, the smart pen includes a posture calculation module. When the electronic device processor 1301 calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, it specifically performs the following steps: obtaining the posture data of the smart pen through the posture calculation module, and determining the indication direction of the smart pen according to the posture data; calculating the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates and the indication direction.

[0183] In some embodiments, the smart pen includes a motion trajectory collection module, and the electronic device processor 1301 also executes the collection of the motion trajectory of the smart pen based on the motion trajectory collection module, where the motion trajectory is the position change trajectory of the smart pen relative to the position of the motion end point collected last time; the motion trajectory is sent to the display device, so that the display device moves the cursor corresponding to the smart pen according to the motion trajectory.

[0184] In some embodiments, the electronic device processor 1301 also executes refreshing the screen two-dimensional coordinates at a first frequency and refreshing the motion trajectory at a second frequency, where the first frequency is less than the second frequency; when the screen two-dimensional coordinates are refreshed, the position of the cursor displayed according to the motion trajectory is corrected based on the refreshed screen two-dimensional coordinates.

[0185] In some embodiments, the infrared image is collected based on an infrared camera in the smart pen, and the infrared camera includes at least an infrared filter.

[0186] When the electronic device 1300 is a display device, Fig.13 In the electronic device 1300 shown, the user interface 1303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the electronic device processor 1301 can be used to call the smart pen cursor display program stored in the memory 1305, and specifically perform the following operations:

[0187] Receiving an infrared image sent by a smart pen corresponding to a display device, where the infrared image is obtained by photographing the display device by the smart pen, and the display device is equipped with at least three infrared positioning points;

[0188] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0189] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the cursor corresponding to the smart pen is displayed based on the two-dimensional screen coordinates.

[0190] When the electronic device 1300 is a smart pen, Fig.13 In the electronic device 1300 shown, the user interface 1303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the electronic device processor 1301 can be used to call the smart pen cursor display program stored in the memory 1305, and specifically perform the following operations:

[0191] Acquire an infrared image of a display device corresponding to the smart pen, where the display device is equipped with at least three infrared positioning points;

[0192] Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image;

[0193] The three-dimensional spatial coordinates are sent to a display device, so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays a cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

[0194] When the electronic device 1300 is a display device, Fig.13In the electronic device 1300 shown, the user interface 1303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the electronic device processor 1301 can be used to call the smart pen cursor display program stored in the memory 1305, and specifically perform the following operations:

[0195] Receiving a spatial three-dimensional coordinate sent by a smart pen corresponding to a display device, the spatial three-dimensional coordinate is a coordinate determined by the smart pen according to the image two-dimensional coordinates of at least three infrared positioning points in an infrared image of the display device, the infrared positioning points being installed on the display device;

[0196] The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the cursor corresponding to the smart pen is displayed based on the two-dimensional screen coordinates.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0198] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the above process or function according to the embodiment of this specification is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium or transmitted by the above computer-readable storage medium. The above computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The above computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVD)), or semiconductor media (for example, solid state drives (SSD)), etc.

[0200] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0201] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0202] The above is a description of a smart pen cursor display method, device, storage medium and electronic device provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A smart pen cursor display method, characterized in that: Applied to a smart pen, the method comprises: Acquire an infrared image of a display device corresponding to the smart pen, wherein the display device is equipped with at least three infrared positioning points; Determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image; The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the two-dimensional screen coordinates are sent to the display device so that the display device displays a cursor corresponding to the smart pen based on the two-dimensional screen coordinates; or, the three-dimensional spatial coordinates are sent to the display device so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

2. The method according to claim 1, characterized in that Before determining the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image, the method further includes: Based on the wireless positioning communication signal between the smart pen and the display device, receiving display device information sent by the display device, wherein the display device information at least includes the real space coordinates corresponding to each infrared positioning point; The determining of the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image comprises: The three-dimensional spatial coordinates of the smart pen are determined based on the two-dimensional image coordinates of each infrared positioning point in the infrared image and the real space coordinates corresponding to each infrared positioning point.

3. The method according to claim 1, characterized in that The smart pen includes a posture calculation module, and the calculation of the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates includes: Acquiring the posture data of the smart pen through the posture calculation module, and determining the indication direction of the smart pen according to the posture data; The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates and the indicated direction.

4. The method according to claim 1, characterized in that: The smart pen includes a motion trajectory acquisition module, and the method further includes: The motion trajectory of the smart pen is collected based on the motion trajectory collection module, where the motion trajectory is a position change trajectory of the smart pen relative to the position of the motion end point collected last time; The motion trajectory is sent to the display device, so that the display device moves the cursor corresponding to the smart pen according to the motion trajectory.

5. The method according to claim 4, characterized in that The method further comprises: Refresh the two-dimensional screen coordinates at a first frequency and refresh the motion trajectory at a second frequency, the first frequency being less than the second frequency; When the two-dimensional screen coordinates are refreshed, the position of the cursor displayed according to the motion track is corrected based on the refreshed two-dimensional screen coordinates.

6. The method according to claim 1, characterized in that The infrared image is collected based on an infrared camera in the smart pen, and the infrared camera at least includes an infrared filter.

7. A smart pen cursor display method, characterized in that: Applied to a display device, the display device is equipped with at least three infrared positioning points, and the method comprises: Receive an infrared image sent by a smart pen corresponding to the display device, where the infrared image is an image obtained by photographing the display device by the smart pen, and determine the spatial three-dimensional coordinates of the smart pen based on the two-dimensional image coordinates of each infrared positioning point in the infrared image; or receive the spatial three-dimensional coordinates sent by the smart pen corresponding to the display device, where the spatial three-dimensional coordinates are coordinates determined by the smart pen based on the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device, where the infrared positioning points are installed on the display device; The two-dimensional screen coordinates corresponding to the smart pen are calculated based on the three-dimensional spatial coordinates, and the cursor corresponding to the smart pen is displayed based on the two-dimensional screen coordinates.

8. A smart pen, characterized in that: The smart pen includes: an infrared camera, a system-level chip and a wireless communication module, wherein: The infrared camera is used to obtain an infrared image of a display device corresponding to the smart pen, and the display device is equipped with at least three infrared positioning points; The system-level chip is connected to the infrared camera and the wireless communication module respectively, and is used to determine the spatial three-dimensional coordinates of the smart pen based on the image two-dimensional coordinates of each infrared positioning point in the infrared image, and calculate the screen two-dimensional coordinates corresponding to the smart pen based on the spatial three-dimensional coordinates; The wireless communication module is used to send the two-dimensional screen coordinates to the display device, so that the display device displays a cursor corresponding to the smart pen based on the two-dimensional screen coordinates; Alternatively, the wireless communication module is used to send the three-dimensional spatial coordinates to the display device, so that the display device calculates the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and displays the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

9. A display device, characterized in that: The display device includes: at least three infrared positioning points, a control system and a wireless communication module, wherein: Each infrared positioning point is used to emit infrared light; The wireless communication module is used to receive the two-dimensional screen coordinates sent by the smart pen corresponding to the display device, wherein the two-dimensional screen coordinates are coordinates determined by the smart pen according to the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device; The control system is connected to the wireless communication module and is used to display a cursor corresponding to the smart pen based on the two-dimensional coordinates of the screen; or, The wireless communication module is used to receive the three-dimensional spatial coordinates sent by the smart pen corresponding to the display device, wherein the three-dimensional spatial coordinates are coordinates determined by the smart pen according to the two-dimensional image coordinates of at least three infrared positioning points in the infrared image of the display device; The control system is connected to the wireless communication module, and is used to calculate the two-dimensional screen coordinates corresponding to the smart pen based on the three-dimensional spatial coordinates, and display the cursor corresponding to the smart pen based on the two-dimensional screen coordinates.

10. The display device according to claim 9, characterized in that The installation positions of all infrared positioning points in the display device are not on the same straight line.

11. The display device according to claim 9 or 10, characterized in that: Each infrared positioning point is respectively installed at a different vertex of the display device.

12. The display device according to claim 9 or 10, characterized in that: Each infrared positioning point is installed behind a display screen of the display device, and the display screen includes a membrane and a liquid crystal layer; The infrared light at each infrared positioning point is the light transmitted after penetrating the membrane and the liquid crystal layer.

13. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 6 or 7.

14. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 or 7 when executing the program.

Images