Interaction method for display device, mobile terminal and storage medium

Through the camera of the mobile terminal filtering and changing the light spots in the display screen, the complex operation of the TV screen is solved, and efficient and low-cost interactive operation is achieved, which is suitable for a variety of scenarios.

CN118945404APending Publication Date: 2024-11-12深圳市当智科技有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410862527.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing TV screen operation mode is complex, and users need to press buttons multiple times, resulting in insufficient convenience, especially when there are many control patterns in large-screen TVs.

Method used

The camera of the mobile terminal is connected to the display device, and the exposure value is used to filter the light spot, combined with the perspective transformation relationship, the trigger coordinates in the display screen are obtained, and sent to the display device to realize interactive operation, reduce the operating burden of the display device, and improve the interaction accuracy and speed.

Benefits of technology

It realizes exquisite interaction with display devices, reduces costs and enhances applicability, and is suitable for the application of laser pens in conference, teaching and entertainment scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945404A_ABST
    Figure CN118945404A_ABST
Patent Text Reader

Abstract

The invention discloses an interaction method for a display device, a mobile terminal and a storage medium. The interaction method for the display device comprises the following steps: establishing connection with the display device; in response to a calibration picture displayed by the display device, acquiring a first shot image containing the calibration picture, and determining a coordinate conversion relationship from a shooting space coordinate system of the first shot image to a display picture coordinate system of the display device according to the first shot image and the calibration picture; acquiring a second shot image containing a display picture of the display device with an exposure value lower than a first threshold value; determining a target pixel coordinate of which the brightness value is greater than a second threshold value in the second shot image, and determining a trigger coordinate corresponding to the target pixel coordinate in the display picture coordinate system according to the coordinate conversion relation; and sending the trigger coordinate to the display device. The device is more exquisite in implementation, lower in cost and higher in applicability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of display devices, and in particular to an interaction method for a display device, a mobile terminal and a storage medium. Background Art

[0002] With the popularity of large-screen TVs, the operation methods on TV screens are becoming more and more complicated, and the interactive experience of TV screen content is gradually gaining attention. For example, for the operation of selecting and clicking on a large number of control patterns on the TV screen, the user is generally required to use the direction buttons of the remote control to operate one by one to move the check box to the corresponding control pattern. This method requires the user to perform multiple button operations and is not convenient enough. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes an interactive method for a display device, a mobile terminal and a storage medium, which are more sophisticated to implement, lower cost and more applicable.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] In a first aspect, the present invention discloses an interaction method for a display device, comprising the following steps: establishing a connection with the display device; in response to the display device displaying a calibration screen, acquiring a first captured image including the calibration screen, and determining a coordinate transformation relationship from a capturing space coordinate system of the first captured image to a display screen coordinate system of the display device based on the first captured image and the calibration screen; acquiring a second captured image including a display screen of the display device with an exposure value lower than a first threshold; determining target pixel coordinates having a brightness value greater than a second threshold in the second captured image, and determining trigger coordinates corresponding to the target pixel coordinates in the display screen coordinate system based on the coordinate transformation relationship; and sending the trigger coordinates to the display device.

[0006] Preferably, the first captured image and the second captured image are acquired with a same resolution that is lower than a third threshold.

[0007] Preferably, determining the coordinate transformation relationship from the shooting space coordinate system of the first captured image to the display screen coordinate system of the display device includes: obtaining the coordinates of multiple feature points in the calibration screen in the first captured image and in the display screen coordinate system; determining the coordinate transformation relationship from the shooting space coordinate system to the display screen coordinate system according to the coordinates of the multiple feature points in the first captured image and in the display screen coordinate system.

[0008] Preferably, determining the target pixel coordinates having a brightness value greater than a second threshold in the second captured image includes:

[0009] The brightness value of the pixel in the second captured image is determined by the RGB value of the pixel, so as to determine the target pixel coordinates having a brightness value greater than a second threshold in the second captured image according to the RGB value of the pixel in the second captured image; or,

[0010] The second captured image is converted into a grayscale image, and target pixel coordinates having grayscale values ​​greater than a second threshold are determined from the grayscale image.

[0011] Preferably, determining the target pixel coordinates having a brightness value greater than a second threshold in the second captured image includes:

[0012] Scanning the second captured image row by row or column by column to obtain target pixel coordinates whose brightness value is greater than a second threshold; or,

[0013] Determine the mapping coordinates of each vertex of the display screen in the second captured image according to the coordinates of each vertex of the display screen in the display screen coordinate system and the coordinate conversion relationship; scan the quadrilateral area formed by the mapping coordinates of each vertex of the display screen in the second captured image row by row or column by column to obtain the target pixel coordinates whose brightness value is greater than the second threshold.

[0014] Preferably, the first threshold is 1 ms.

[0015] Preferably, the acquiring of a second captured image including a display screen of the display device with an exposure value lower than a first threshold value includes: in response to a light spot generated by a light spot generator being projected onto the display screen, acquiring a second captured image including the display screen with an exposure value lower than the first threshold value; and sending the trigger coordinates to the display device includes: acquiring a control instruction of the light spot generator, and sending the control instruction and the trigger coordinates to the display device at the same time.

[0016] Preferably, before acquiring the second captured image including the display screen of the display device with an exposure value lower than the first threshold, the method further includes:

[0017] In the process of gradually increasing the exposure value of the camera from the lowest, a plurality of calibration images including the display screen are captured, the display screen includes a designated area, and each of the calibration images has a one-to-one correspondence with the exposure value; identifying in time sequence whether there is a target pixel coordinate with a brightness value greater than a second threshold value in the designated area in each of the calibration images; when it is identified that the target pixel coordinate exists in the designated area in the calibration image, using the exposure value corresponding to the calibration image as the exposure value for obtaining a second captured image including the display screen; or,

[0018] The exposure value of the camera is adjusted until only the target pixel coordinates are displayed in the image to be captured, and the current exposure value is used as the exposure value when obtaining the second captured image including the display screen.

[0019] In a second aspect, the present invention discloses a mobile terminal, comprising a camera, a processor and a memory, wherein the camera is connected to the processor, and a computer program is stored in the memory, and the computer program can be read and executed by the processor to implement the interactive method for a display device as described in the first aspect.

[0020] In a third aspect, the present invention discloses a storage medium storing a computer program, wherein the computer program is configured to be executable by a processor to execute the interactive method for a display device described in the first aspect.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: the interactive method, mobile terminal and storage medium for a display device proposed in the present invention realize interaction by means of a camera on a mobile terminal, and obtain a captured image with an exposure value lower than a first threshold value, wherein the exposure function of the camera is utilized, and the higher the light intensity of the light spot within the exposure range, the brighter and clearer the light spot imaged on the captured image is, thereby realizing screening of different light intensities, and screening the light spot in the display screen of the display device according to this principle, and combining the coordinate conversion relationship of the perspective transformation, the shooting space coordinate system is converted into the display screen coordinate system, and the trigger coordinates corresponding to the light spot in the display screen coordinate system are obtained and sent to the display terminal, and all these processes on the captured image can be performed on the mobile terminal, and by means of the powerful image processing performance of the mobile terminal, the memory of the display device is not occupied, and the display device only needs to be responsible for receiving the trigger coordinates and control instructions to perform the control operation, which can minimize the operating burden of the display device and improve the parsing speed and accuracy of the trigger coordinates, thereby making the interactive method more sophisticated to implement, lower cost, and more applicable, and can be applied to scenes such as conferences / teaching / entertainment involving laser pens.

[0022] In a further solution, a control instruction of the light spot display is also obtained to send the control instruction and the trigger coordinates to the display device at the same time. This can avoid the situation where when the trigger coordinates are sent by the mobile terminal to the display device and the control instruction is sent by the light spot display (such as a laser pen) to the display device, the two signals (trigger coordinate signals and control instruction signals) may be difficult to match one by one in a timely manner during high-frequency interactive triggering, resulting in a relatively large difference in the arrival time of the two signals (trigger coordinate signals and control instruction signals); thereby improving the accuracy of real-time matching of control instructions and trigger coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of an interactive method for a display device disclosed in a preferred embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of a device framework for an interactive method for a display device in a specific embodiment;

[0025] Figure 3 is a schematic diagram of a calibration original image including a chessboard preset by a display device in a specific embodiment;

[0026] Figure 4 is a schematic diagram of a first captured image obtained;

[0027] Figure 5 is a schematic diagram of a second captured image obtained;

[0028] Figure 6 is a schematic diagram of a calibration original image obtained by converting the coordinates of the second captured image into a display screen coordinate system;

[0029] Figure 7 It is a captured image obtained after reducing the camera exposure value in a specific example. DETAILED DESCRIPTION

[0030] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0031] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit / signal communication.

[0032] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0034] like Figure 1 As shown, a preferred embodiment of the present invention discloses an interactive method for a display device, comprising the following steps:

[0035] S1: Establish connection with the display device;

[0036] like Figure 2 As shown, the interactive method for the display device in this embodiment is performed by a mobile terminal 10, which may be, for example, a mobile phone, a tablet, etc.; the display device may be, for example, a television, a projector, etc., and the mobile terminal 10 is placed in a position where it can capture the display screen 20 displayed by the display device through its camera. In the interactive process, a spot display 30 is also involved, which may be, for example, a laser pen, etc., and the spot display 30 irradiates the display screen 20 to form a spot 31 (for example, a laser point).

[0037] In a specific example, before starting the interactive function operation, the user first starts the corresponding shooting interactive function in the APP on the mobile terminal 10, and then fixes the mobile terminal 10 to the display screen 20 of the display device. The camera of the mobile terminal 10 can be directly facing or obliquely facing the display screen 20 of the display device. Figure 2 .

[0038] In this step, in a specific embodiment, the mobile terminal and the display device may establish a wireless connection such as Bluetooth or Wifi.

[0039] S2: in response to the display device displaying the calibration picture, acquiring a first captured image including the calibration picture, and determining a coordinate transformation relationship from a capture space coordinate system of the first captured image to a display picture coordinate system of the display device according to the first captured image and the calibration picture;

[0040] The first captured image is acquired at a resolution lower than a third threshold.

[0041] Determining the coordinate transformation relationship from the shooting space coordinate system of the first captured image to the display screen coordinate system of the display device includes: obtaining the coordinates of multiple feature points in the calibration screen in the first captured image and in the display screen coordinate system; determining the coordinate transformation relationship from the shooting space coordinate system to the display screen coordinate system according to the coordinates of the multiple feature points in the first captured image and in the display screen coordinate system.

[0042] In one embodiment, the display device is based on Figure 3 The preset calibration original image 100 shown in the figure first displays a chessboard 200 (or a feature image such as a QR code) on the display screen 20, and then the user fixes the position of the mobile terminal, and the camera on the mobile terminal or the independent camera on other devices captures the display screen 20 with the chessboard 200 according to the preset lower resolution and automatic exposure mode, and obtains the following Figure 4 The first captured image 300 is shown. The image processor of the mobile terminal can identify the vertex coordinates (such as A1, B1, C1, D1, etc.) of the checkerboard 200 in the captured image coordinate system O1 of the first captured image 300, and obtain Figure 3 The original vertex coordinates (such as A, B, C, D, etc.) of the chessboard 200 of the display screen coordinate system O are shown; the mobile terminal can obtain the relevant coordinate information of the calibration original image 100 preset in the display device through wireless data transmission; or the calibration original image 100 displayed on the display device is sent to the display device by the mobile terminal to realize display, and at this time the mobile terminal can obtain the relevant coordinate information of the calibration original image 100 from itself. Figure 3 The original vertex coordinates (such as A, B, C, D, etc.) of the chessboard 200 of the display screen coordinate system O are shown. Based on the two sets of vertex coordinates, the transformation matrix H of the transmission transformation can be calculated. The transformation matrix H is used to represent the coordinate transformation relationship between the two coordinate systems (O and O1).

[0043] When the camera acquires the first captured image, a low resolution is used but a low exposure rate is not used so that the checkerboard pattern can be captured more clearly to obtain a conversion matrix H. In subsequent steps, the touch coordinates are further acquired based on the conversion matrix H, so that the interaction of the display device can be realized through the mobile terminal.

[0044] S3: acquiring a second captured image including a display screen of the display device at an exposure value lower than the first threshold;

[0045] The first threshold is 1 ms, that is, the second captured image including the display screen of the display device is acquired by performing exposure within 1 ms.

[0046] The second captured image is acquired with a resolution lower than the third threshold. In a specific embodiment, the first captured image and the second captured image are acquired with the same resolution. Before capturing the first captured image and the second captured image, the resolution of the camera is adjusted so that the resolution is small enough to make the acquired image data small and speed up the subsequent processing. However, the resolution should not be too small, and the display screen of the entire display device needs to be seen, such as 640×480, 800×600, 1024×768, etc.

[0047] In a specific embodiment, step S3 specifically includes: in response to the light spot generated by the light spot generator being projected onto the display screen, a second captured image including the display screen is acquired with an exposure value lower than the first threshold. Specifically, when the user holds a light spot generator (such as a laser pen) to illuminate the display screen, the camera of the mobile terminal is set to a low exposure mode, or an external independent camera is set to a low exposure mode, and the camera exposure value is lowered. At this time, the second captured image captured by the camera is relatively dark, and most of the display screen and environmental information in the second captured image can be filtered out, so that only the light spot information in the second captured image can be retained as much as possible, and the visible light spot generated by the light spot generator is brighter than the screen and the environment. At this time, the visible small light spot generator projects a small light spot to an area of ​​the display screen 20, such as Figure 5 ZB shown 0 Among them, by automatically adjusting the camera exposure value to be lower than the first threshold, it can be applicable to the small light spots generated by most laser pens on the market.

[0048] In one specific example, Figure 7 As shown, it is a second captured image obtained after lowering the camera exposure value, in which the white dot is the light spot emitted by the captured laser pen, and other display screen information is almost not displayed due to the low exposure value, so the position of the light spot in the captured image can be easily identified.

[0049] S4: determining the target pixel coordinates whose brightness value is greater than the second threshold in the second captured image, and determining the trigger coordinates corresponding to the target pixel coordinates in the display screen coordinate system according to the coordinate conversion relationship;

[0050] Step S4 specifically includes:

[0051] S41: determining the coordinates of target pixels whose brightness values ​​are greater than a second threshold in the second captured image;

[0052] On the one hand, step S41 can adopt any one of the following two specific schemes for determining the brightness value to obtain the target pixel coordinates:

[0053] (1) In a first specific solution, the luminance value of the pixel point in the second captured image can be directly determined by the RGB value in the pixel point, that is, the luminance value of the pixel point in the second captured image is determined by the RGB value in the pixel point, so as to determine the target pixel coordinates whose luminance value is greater than the second threshold value in the second captured image according to the RGB value of the pixel point in the second captured image;

[0054] (2) In the second specific solution, the second captured image may be converted into a grayscale image, and then confirmed based on the grayscale image; that is, the second captured image is converted into a grayscale image, and the target pixel coordinates whose grayscale values ​​are greater than the second threshold are determined from the grayscale image.

[0055] The meanings of the second threshold in the above two specific solutions are different. The first specific solution determines the second threshold by the range of RGB values, and the second specific solution determines the second threshold by the grayscale value. Specifically, the second captured image is originally an RGB image, wherein in the first specific solution, the corresponding brightness can be calculated by the weighted RGB values ​​of each pixel, or the brightness of the pixel can be defined by the maximum value of the RGB values ​​of each pixel; in the second specific solution, each pixel of the grayscale image can only be characterized by the degree of black and white, and the value is between 0-255, 0 is the darkest, and 255 is the brightest.

[0056] On the other hand, step S41 may adopt any one of the following two specific schemes for finding the target pixel coordinates to obtain the target pixel coordinates:

[0057] (1) scanning the second captured image row by row or column by column to obtain target pixel coordinates whose brightness values ​​are greater than a second threshold;

[0058] Specifically, if a pixel grayscale value is greater than a second threshold value such as 128 (the grayscale range is 0-255) obtained by scanning row by row or column by column from the head or tail, or a brightness value determined according to the pixel RGB value is greater than the second threshold value, it is considered to be the target pixel corresponding to the touch point; this second threshold value, combined with the camera exposure value, can be used to adapt to laser pens of different brightness, etc.

[0059] (2) determining the mapping coordinates of each vertex of the display screen in the second captured image according to the coordinates of each vertex of the display screen in the display screen coordinate system and the coordinate conversion relationship; scanning the quadrilateral area formed by the mapping coordinates of each vertex of the display screen in the second captured image row by row or column by column to obtain the target pixel coordinates whose brightness value is greater than the second threshold;

[0060] Specifically, refer to Figure 5 and Figure 6, according to the vertex coordinates of the calibrated original image 100 (for example, (0, 1080) (1920, 1080) (1920, 0) (0, 0)) and the inverse matrix of the above-mentioned conversion matrix H, the theoretical vertex coordinates (K1, K2, K3, K4) of the display screen 20 corresponding to the second captured image 400 are calculated; then scan downward row by row from the point with the largest y value; or scan upward row by row from the point with the smallest y value; or scan rightward column by column from the point with the smallest x value; or scan leftward column by column from the point with the largest x value, and obtain a pixel gray value greater than the second threshold value such as 128 (grayscale range is 0-255) or a brightness value determined according to the pixel RGB value is greater than the second threshold value, then it is considered to be the target pixel corresponding to the touch point. This method does not need to scan the entire image, can speed up the recognition efficiency, and can eliminate the interference of strong light points outside the domain. Among them, when using this method, the default display screen has at most two bright spots, and there is no interference from other fixed bright spot areas. The static bright spot interference can be eliminated by subtracting the image.

[0061] Based on the image transformation involved in the entire processing process, the amount of processed data will increase. For example, scanning from beginning to end will slow down the speed of acquiring touch coordinates. Therefore, this further solution is proposed. In this solution, the mobile terminal processor only needs to scan and identify in the quadrilateral area surrounded by K1, K2, K3, and K4 to determine ZB 0 position, so that there is no need to scan and identify the pixels in the entire captured image every time, which can reduce the computing burden of the CPU processor of the mobile terminal; therefore, this solution can also achieve a faster response experience effect for CPUs with poor computing power.

[0062] S42: determining the trigger coordinates corresponding to the target pixel coordinates in the display screen coordinate system according to the coordinate conversion relationship;

[0063] This step specifically includes: determining the mapping coordinates of the target pixel coordinates in the display screen coordinate system according to the target pixel coordinates and the coordinate conversion relationship; and determining the mapping coordinates of the target pixel coordinates in the display screen coordinate system as the trigger coordinates corresponding to the light spot in the display screen coordinate system.

[0064] refer to Figure 5 and Figure 6 In step S4, first determine the coordinates ZB of the small light spot in the image coordinate system O1 of the second image 400. 0 , and then combined with the transformation matrix H to transfer to the display screen coordinate system O of the calibration original image 100, to obtain the mapping coordinate ZB 1 =H×ZB 0 , the mapping coordinate ZB 1 The trigger coordinates corresponding to the light spot in the display screen coordinate system can be determined.

[0065] S5: Send the trigger coordinates to the display device.

[0066] In some embodiments, the mobile terminal sends the trigger coordinates to the display device, and the display device can directly display the mouse mark at the trigger coordinates. Further, the display device can also receive control instructions from a spot display or remote control, and can perform corresponding touch operations in combination with the trigger coordinates and the control instructions.

[0067] In a further embodiment, the mobile terminal also obtains the control instruction of the spot display, and the mobile terminal sends the trigger coordinates and the control instruction to the display device. The display device can perform touch operations on the corresponding position in the display screen according to the trigger coordinates and the control instruction. Specifically, the mobile terminal establishes a wireless connection with the spot display or remote control, etc. After the control button of the spot display or remote control is pressed, the control instruction is sent to the mobile terminal so that the mobile terminal obtains the control instruction of the spot display or remote control, etc.; the mobile terminal can bind the control instruction with the trigger coordinates corresponding to the currently identified spot in the display screen coordinate system and send them together to the display device. In a specific embodiment, the remote control can be combined with a laser with a laser pen (for example, the spot display is a combination of a remote control and a laser with a laser pen). For example, when the user presses the A button on the remote control, the coordinates ZB are triggered. 1 Click command for the location.

[0068] By sending the control instructions and the trigger coordinates to the display device at the same time, it is possible to avoid the situation where, when the trigger coordinates are sent by the mobile terminal to the display device and the control instructions are sent by a light spot display (such as a laser pen) to the display device, the two signals (trigger coordinate signals and control instruction signals) may be difficult to match one-to-one in a timely manner when they are triggered interactively at a high frequency, resulting in a relatively large difference in the arrival time of the two signals (trigger coordinate signals and control instruction signals); thereby improving the accuracy of real-time matching of control instructions and trigger coordinates.

[0069] The touch operation in this step may be, for example, single-point touch or multi-point touch. For single-point touch or two-point touch, the second captured image may be simply converted into a grayscale image in the specific solution of searching for the target pixel coordinates.

[0070] In a further embodiment, a calibration step is further included before executing step S2.

[0071] In one embodiment, the calibration step specifically includes:

[0072] A1: When the exposure value of the camera is gradually increased from the lowest, a plurality of calibration images including a display screen are captured, the display screen includes a designated area, and each calibration image has a one-to-one correspondence with the exposure value;

[0073] At this time, the display device enters the calibration mode, and the exposure value of the camera gradually increases from the lowest, while continuously shooting to obtain the calibration image. Each calibration image taken corresponds to an exposure value. At this time, the user can hold a visible light laser pen and shine it at a designated area of ​​the display screen. The designated area can be a rectangular area in the middle of the display screen, and the display screen shows a pure color in the designated area.

[0074] A2: Identify in time sequence whether there is a target pixel coordinate with a brightness value greater than a second threshold value in a specified area in each calibration image;

[0075] Specifically, the mobile terminal processor continuously identifies the designated area in the captured image, wherein the designated area of ​​the calibrated original image can be obtained by performing an inverse matrix operation of the transformation matrix H to obtain the designated area corresponding to the captured calibration image.

[0076] A3: When it is recognized that the target pixel coordinates exist in the designated area in the calibration image, the exposure value corresponding to the calibration image is used as the exposure value when the second captured image including the display screen is obtained in step S2.

[0077] When the processor identifies the existence of pixels with grayscale values ​​higher than the second threshold in this area in a certain captured image, the corresponding exposure value when capturing the calibration image is recorded, and the laser pen interaction is subsequently recognized using the exposure value, i.e., the second captured image including the display screen is acquired using the exposure value in step S2. This can prevent some laser pens from being too dim to be captured by a camera with a too low exposure value.

[0078] In actual applications, the exposure value of the image captured by the mobile terminal can be easily adjusted by the user, so the calibration step in another embodiment may include: adjusting the exposure value of the camera until only the light spot is displayed in the image to be captured, and using the current exposure value as the exposure value when obtaining the second captured image containing the display screen in step S2. In a specific embodiment, the calibration step includes: displaying the actual captured image in the corresponding function interface of the APP of the mobile terminal, at which time the camera function in the mobile terminal can be called up, and the user can swipe the screen of the mobile terminal to adjust the exposure of the captured image by swiping. When it is seen that the captured image in the screen of the mobile terminal is almost only the visible light laser point, click the lock button to lock the current exposure, and the subsequent mobile terminal will use this exposure for shooting and recognition during the laser interaction process; in this way, the user can easily adjust it by himself.

[0079] The interactive method for a display device disclosed in the first embodiment of the present invention utilizes the exposure function of a camera. When the exposure is faster, the higher the light intensity of the light spot within the shooting range, the brighter and clearer the light spot imaged on the grayscale photo is, thereby realizing the screening of different light intensities; therefore, when the light intensity diffusely reflected by the laser pen or infrared laser pen on the display screen is greater than the light intensity of the display screen, it can be clearly captured by the camera; then combined with the perspective transformation relationship, the shooting space coordinates are converted into the internal plane coordinates of the display device to achieve the acquisition of the trigger coordinates. It can also be further combined with other remote control / handle buttons to integrate more functional interactive controls. This solution can realize the screen interaction of the display device with the help of a light spot display (visible light laser pen) in conjunction with a general camera of a mobile terminal (mobile phone, tablet computer, etc.). It is more sophisticated to implement, with lower cost and stronger applicability. It can be applied to scenes such as conferences / teaching / entertainment involving laser pens.

[0080] Embodiment 2 of the present invention discloses a mobile terminal, including a camera, a processor and a memory, wherein the camera is connected to the processor, and a computer program is stored in the memory, and the computer program can be read by the processor to be executed to implement the interactive method for a display device in embodiment 1, wherein all images acquired by the interactive method for a display device are acquired through the camera.

[0081] The memory is a storage device that stores programs and data executed by the processor in a non-volatile manner. The memory is composed of semiconductor storage elements such as magnetic storage devices, flash read-only memories (ROMs), or other types of non-volatile storage devices. The memory may also include random access memories (RAMs) that constitute the working area of ​​the processor. The memory stores control programs executed by the processor. The processor may be composed of a single processor or a combination of multiple processing groups. The processor executes the control program to control various parts of the mobile terminal.

[0082] A third embodiment of the present invention discloses a storage medium, in which a computer program is stored. The computer program is configured to be executable by a processor to execute the interactive method for a display device in the first embodiment.

[0083] Optionally, the storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0084] The background section of the present invention may contain background information about the problem or environment of the present invention, rather than describing the prior art by others. Therefore, the content contained in the background section is not an admission by the applicant that the prior art is available.

[0085] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of mutual contradiction, the technical personnel in this field can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope defined by the attached claims.

Claims

1. An interactive method for a display device, characterized in that: The following steps are involved: Establishing a connection with the display device; In response to the display device displaying a calibration screen, acquiring a first captured image including the calibration screen, and determining a coordinate transformation relationship from a capture space coordinate system of the first captured image to a display screen coordinate system of the display device according to the first captured image and the calibration screen; Acquire a second captured image including a display screen of the display device at an exposure value lower than a first threshold; Determine, in the second captured image, target pixel coordinates whose brightness value is greater than a second threshold, and determine, according to the coordinate conversion relationship, trigger coordinates corresponding to the target pixel coordinates in the display screen coordinate system; The trigger coordinates are sent to the display device.

2. The interactive method for a display device according to claim 1, characterized in that: The first captured image and the second captured image are acquired at a same resolution that is lower than a third threshold.

3. The interactive method for a display device according to claim 1, characterized in that: The step of determining a coordinate transformation relationship between a shooting space coordinate system of the first shot image and a display screen coordinate system of the display device comprises: Obtaining coordinates of a plurality of feature points in the calibration screen in the first captured image and in the display screen coordinate system respectively; A coordinate conversion relationship from the shooting space coordinate system to the display screen coordinate system is determined according to the coordinates of the plurality of feature points in the first shot image and the coordinates of the feature points in the display screen coordinate system.

4. The interactive method for a display device according to claim 1, characterized in that: The step of determining the target pixel coordinates whose brightness value is greater than a second threshold in the second captured image comprises: The brightness value of the pixel in the second captured image is determined by the RGB value of the pixel, so as to determine the target pixel coordinates having a brightness value greater than a second threshold in the second captured image according to the RGB value of the pixel in the second captured image; or, The second captured image is converted into a grayscale image, and target pixel coordinates having grayscale values ​​greater than a second threshold are determined from the grayscale image.

5. The interactive method for a display device according to claim 1, characterized in that: The step of determining the target pixel coordinates whose brightness value is greater than a second threshold in the second captured image comprises: Scanning the second captured image row by row or column by column to obtain target pixel coordinates whose brightness value is greater than a second threshold; or, Determine the mapping coordinates of each vertex of the display screen in the second captured image according to the coordinates of each vertex of the display screen in the display screen coordinate system and the coordinate conversion relationship; scan the quadrilateral area formed by the mapping coordinates of each vertex of the display screen in the second captured image row by row or column by column to obtain the target pixel coordinates whose brightness value is greater than the second threshold.

6. The interactive method for a display device according to claim 1, characterized in that: The first threshold is 1 ms.

7. The interactive method for a display device according to claim 1, characterized in that: The acquiring of a second captured image including the display screen of the display device at an exposure value lower than the first threshold value comprises: in response to a light spot generated by a light spot generator being projected onto the display screen, acquiring a second captured image including the display screen at an exposure value lower than the first threshold value; The sending the trigger coordinates to the display device includes: acquiring a control instruction of the light spot generator, and sending the control instruction and the trigger coordinates to the display device simultaneously.

8. The interactive method for a display device according to claim 1, characterized in that: Before acquiring a second captured image including a display screen of the display device at an exposure value lower than the first threshold, the method further includes: In the process of gradually increasing the exposure value of the camera from the lowest, a plurality of calibration images including the display screen are captured, the display screen includes a designated area, and each of the calibration images has a one-to-one correspondence with the exposure value; identifying in time sequence whether there is a target pixel coordinate with a brightness value greater than a second threshold value in the designated area in each of the calibration images; when it is identified that the target pixel coordinate exists in the designated area in the calibration image, using the exposure value corresponding to the calibration image as the exposure value for obtaining a second captured image including the display screen; or, The exposure value of the camera is adjusted until only the target pixel coordinates are displayed in the image to be captured, and the current exposure value is used as the exposure value when obtaining the second captured image including the display screen.

9. A mobile terminal, characterized in that: The invention comprises a camera, a processor and a memory, wherein the camera is connected to the processor, and a computer program is stored in the memory, and the computer program can be read and executed by the processor to implement the interactive method for a display device as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to be executable by a processor to execute the interactive method for a display device as claimed in any one of claims 1 to 8.

Citation Information

Cited By

  • Camera calibration method and device

    CN119904532A