Method for accurately clicking screen by blind operation based on same-screen device

Abstract

The invention discloses a method for accurately clicking a screen by blind operation based on a same-screen device. The method comprises the following steps: S1, creating an icon Fn for setting a screen clicking position on the same-screen device; s2, creating a button Kn used for starting click operation; s3, the same-screen device is connected with the mobile phone, the same-screen device and the mobile phone are in the same-screen state, the user touches and drags the icons Fn with the hand, the Fn are dragged to the positions, needing to be clicked, of the mobile phone screen, and coordinate values of the Fn are recorded; s4, moving the button Kn to a position suitable for finger operation; and S5, the button Kn is touched, and the screen sharing device sends a command of touching the screen of the mobile phone to the mobile phone, wherein the coordinate of the command is the coordinate of the icon Fn. Accurate click operation on any point on the screen of the same-screen device is achieved in a blind operation mode by converting accurate click operation on a certain position in the middle of the screen of the same-screen device into click operation on a button which is relatively fixed in position and large in area on the side position of the screen of the same-screen device. Therefore, focus switching and transferring can be realized quickly.

Description

Technical Field

[0001] This invention belongs to the field of mobile phone photography, and specifically relates to a method for precise blind screen clicking based on a screen mirroring device. Background Technology

[0002] In mobile phone photography, when there are two or more subjects in the shot, in order to highlight the main character of the video, it is often necessary to focus the lens on one of the subjects and then, as needed, shift the focus to another subject. For example, below. Figure 1 When shooting product introductions, we need to shift the focus back and forth between the face and the medicine. Of course, there are many similar scenarios in reality.

[0003] For mobile phone video shooting, you can easily shift the focus by tapping the subject in the camera interface. The operation is relatively simple. However, in the actual video shooting process, especially when shooting with both hands using a stabilizer or gimbal, the phone shakes due to hand movements, making it not easy to accurately tap a specific spot on the screen.

[0004] To reduce screen shake, we usually use a Bluetooth remote control to indirectly tap the phone screen. Based on this, more advanced screen mirroring devices have emerged on the market. These devices have the ability to control the phone screen in reverse, and can also display a mirrored image of the phone screen on their own screen. This effectively achieves indirect screen control, solving the problem of shifting focus. Compared to directly tapping the phone screen, tapping the screen mirroring device significantly reduces body movement, thus reducing screen shake.

[0005] While using the screen of a screen mirroring device can effectively solve the problems of precise focusing and shifting the focus point, it's still not easy to accurately tap a specific location on the device while holding a stabilizer or gimbal with both hands, all while keeping your hands steady. This is because during shooting, our eyes are often not fixed on the screen mirroring device, but rather on the phone, while simultaneously using peripheral vision to observe the subject and move the camera. In this situation, accurately tapping a specific location on the screen mirroring device with your finger is quite inconvenient. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a method for precise blind operation of a screen mirroring device. By transforming a precise click operation on a specific location in the center of the screen mirroring device into a click operation on a relatively fixed and larger button on the edge of the screen mirroring device, a precise click operation on any point on the screen mirroring device can be achieved blindly without the need for careful visual observation, thereby enabling rapid focus switching.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] This invention provides a method for precise blind screen clicking based on a screen mirroring device, comprising the following steps:

[0009] S1: On a screen mirroring device with reverse touch screen functionality, an embedded program is used to create one or more icons Fn (Xfn, Yfn) on the screen of the screen mirroring device to set the click position of the screen.

[0010] S2: Create the same number of buttons Kn(Xkn, Ykn) as Fn to initiate the click action;

[0011] S3: When the screen mirroring device is connected to the phone, the two are in screen mirroring mode. The system enters the settings interface through the settings button of the screen mirroring device. In the settings interface, the user touches and drags the Fn icons, places multiple Fn icons on the phone screen where they need to be clicked, and records their coordinate values ​​(Xfn, Yfn).

[0012] S4: Move button Kn (Xkn, Ykn) to a position suitable for finger operation;

[0013] S5: After the setup is complete, the system returns to the operation interface. At this time, touch the button Kn on the screen mirroring device, and the system will start the screen mirroring device to reverse the touch screen operation of the mobile phone screen, and send a command to the mobile phone with the coordinates (Xfn, Yfn) of the icon Fn.

[0014] It should be noted that the screen mirroring device system needs to have the functions of screen mirroring with mobile phones and reverse touch control of mobile phone screens.

[0015] There are many methods and technologies for achieving screen mirroring on mobile phones, including Aircast, Miracast, TypeCDP, and mobile app technologies that use precise values ​​similar to remote desktop technology. All of these can achieve the screen mirroring function. For example, TypeCDP to HDMI conversion can be achieved using CS5261AN or AD9310. These are all existing technologies and are the basic conditions for realizing this invention.

[0016] Similarly, there are many methods and technologies for achieving reverse touch control of mobile phone screens, such as UIBC reverse touch control technology based on WIFI, HID touch screen based on Bluetooth / USB, or HID mouse based on Bluetooth / USB that supports absolute coordinates. All of these can achieve reverse control of mobile phone screens. These are all existing technologies and are the basic conditions for realizing this invention.

[0017] Based on the aforementioned screen mirroring and reverse touchscreen functionality, the software implementation method and basic control logic of this invention are as follows:

[0018] The specific implementation method for dragging the icon Fn to move its position in step S3 is as follows:

[0019] Step S31: Press the settings button, and the system enters the settings interface. The user touches and slides the screen of the screen mirroring device to determine whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S32; otherwise, end.

[0020] Step S32: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Fn (Xfn, Yfn). If yes, proceed to step S33; otherwise, end.

[0021] Step S33: Determine whether the coordinates (Xtp, Ytp) are equal to the icon Fn (Xfn, Yfn). If they are equal, the process ends; otherwise, proceed to step S34.

[0022] Step S34: Assign values ​​to icon Fn: Xfn=Xtp, Yfn=Ytp, and move icon Fn to the new coordinate position of the subject to be focused.

[0023] In step S4, the position of button Kn can also be adjusted by the user. Generally, this is done by entering the button setting mode through a special key combination. In this setting mode, button Kn can be touched and moved to a position on the screen of the screen mirroring device that is convenient for operation with the thumb. The setting flowchart is as follows:

[0024] Step S41: Press the settings button, and the system enters the settings interface. The user touches and slides the screen of the screen mirroring device to determine whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S42; otherwise, end.

[0025] Step S42: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Kn (Xkn, Ykn). If yes, proceed to step S33; otherwise, end.

[0026] Step S43: Determine whether the coordinates (Xtp, Ytp) are equal to the icon Kn (Xkn, Ykn). If they are equal, the process ends; otherwise, proceed to step S34.

[0027] Step S44: Assign values ​​to icon Kn: Xkn=Xtp, Ykn=Ytp, and move icon Kn to a new coordinate position that is easy to operate.

[0028] In step S5, when the user touches the screen within the effective range of button Kn in the operation interface, the system sends the coordinates of the touch operation as the coordinates of icon Fn (Xfn, Yfn) to achieve precise focus in photography. Otherwise, it sends the normal reverse touch coordinates (Xtp, Ytp). The operation flowchart is as follows:

[0029] Step S51: The user touches the screen of the sliding screen mirroring device. It is determined whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S52. If not, end.

[0030] Step S52: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Kn (Xkn, Ykn). If yes, proceed to step S53; otherwise, proceed to step S54.

[0031] Step S53: Send a reverse control signal to the mobile phone screen for the coordinates (Xfn, Yfn) of icon Fn, and then end;

[0032] Step S54: Send the reverse control signal of the mobile phone screen at the current touch coordinates (Xtp, Ytp), and then end.

[0033] The beneficial effects of this invention are: compared with the prior art, by transforming the precise click operation on a certain position in the middle of the screen of the screen sharing device into a click operation on a relatively fixed and larger button on the edge of the screen of the screen sharing device, the precise click operation on any point on the screen of the screen sharing device can be achieved by blind operation without the need for careful viewing, so as to quickly achieve the switching and transfer of focus.

[0034] In certain special situations, app focusing applications can be useful: Some camera apps have built-in focus shift functions. For example, the Blackmagic Cam photography app has a built-in focus shift function that allows you to set three focus points, and each focus point can have its focus shift duration set. If you position the Fn icon on one of the Blackmagic Cam's built-in focus points, then blindly clicking the Kn button will not only shift the focus point but also complete the focus shift according to the time set by Blackmagic Cam. Attached Figure Description

[0035] Figure 1 This is a flowchart of a method for precise screen clicking in the blind.

[0036] Figure 2 This is a flowchart showing how to move the icon Fn to its current position.

[0037] Figure 3 This is a flowchart of setting the position of button Kn.

[0038] Figure 4 This is a flowchart illustrating how the touch button Kn controls the icon Fn to control the phone screen in reverse.

[0039] Figure 5 This is the architecture diagram of the screen mirroring system.

[0040] Figure 6 This is a diagram illustrating the reverse control of a mobile phone screen using a screen mirroring device.

[0041] Figure 7 This is a schematic diagram of the user interface of the screen mirroring device in focus shift mode.

[0042] Figure 8 This is a schematic diagram of the settings interface of the screen mirroring device in focus shift mode. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0044] To achieve the above objectives, the technical solution of the present invention is as follows:

[0045] See Figure 1 This embodiment provides a method for precise blind screen clicking based on a screen mirroring device, including the following steps:

[0046] S1: On a screen mirroring device with reverse touch screen functionality, an embedded program is used to create one or more icons Fn (Xfn, Yfn) on the screen of the screen mirroring device to set the click position of the screen.

[0047] S2: Create the same number of buttons Kn(Xkn, Ykn) as Fn to initiate the click action;

[0048] S3: When the screen mirroring device is connected to the phone, the two are in screen mirroring mode. The system enters the settings interface through the settings button of the screen mirroring device. In the settings interface, the user touches and drags the Fn icons, places multiple Fn icons on the phone screen where they need to be clicked, and records their coordinate values ​​(Xfn, Yfn).

[0049] S4: Move button Kn (Xkn, Ykn) to a position suitable for finger operation;

[0050] S5: After the setup is complete, the system returns to the operation interface. At this time, touch the button Kn on the screen mirroring device, and the system will start the screen mirroring device to reverse the touch screen operation of the mobile phone screen, and send a command to the mobile phone with the coordinates (Xfn, Yfn) of the icon Fn.

[0051] It should be noted that the screen mirroring device system needs to have the functions of screen mirroring with mobile phones and reverse touch control of mobile phone screens.

[0052] There are many methods and technologies for achieving screen mirroring on mobile phones, including Aircast, Miracast, TypeCDP, and mobile app technologies that use precise values ​​similar to remote desktop technology. All of these can achieve the screen mirroring function. For example, TypeCDP to HDMI conversion can be achieved using CS5261AN or AD9310. These are all existing technologies and are the basic conditions for realizing this invention.

[0053] Similarly, there are many methods and technologies for achieving reverse touch control of mobile phone screens, such as UIBC reverse touch control technology based on WIFI, HID touch screen based on Bluetooth / USB, or HID mouse based on Bluetooth / USB that supports absolute coordinates. All of these can achieve reverse control of mobile phone screens. These are all existing technologies and are the basic conditions for realizing this invention.

[0054] Based on the aforementioned screen mirroring and reverse touchscreen functionality, the software implementation method and basic control logic of this invention are as follows:

[0055] See Figure 2 The specific implementation method for dragging the icon Fn to move its position in step S3 is as follows:

[0056] Step S31: Press the settings button, and the system enters the settings interface. The user touches and slides the screen of the screen mirroring device to determine whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S32; otherwise, end.

[0057] Step S32: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Fn (Xfn, Yfn). If yes, proceed to step S33; otherwise, end.

[0058] Step S33: Determine whether the coordinates (Xtp, Ytp) are equal to the icon Fn (Xfn, Yfn). If they are equal, the process ends; otherwise, proceed to step S34.

[0059] Step S34: Assign values ​​to icon Fn: Xfn=Xtp, Yfn=Ytp, and move icon Fn to the new coordinate position of the subject to be focused.

[0060] See Figure 3 In step S4, the position of button Kn can also be adjusted by the user. Generally, a special key combination is used to enter the button setting mode. In this setting mode, button Kn can be touched and moved to a position on the screen of the screen mirroring device that is convenient for operation with the thumb. The setting flowchart is as follows:

[0061] Step S41: Press the settings button, and the system enters the settings interface. The user touches and slides the screen of the screen mirroring device to determine whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S42; otherwise, end.

[0062] Step S42: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Kn (Xkn, Ykn). If yes, proceed to step S33; otherwise, end.

[0063] Step S43: Determine whether the coordinates (Xtp, Ytp) are equal to the icon Kn (Xkn, Ykn). If they are equal, the process ends; otherwise, proceed to step S34.

[0064] Step S44: Assign values ​​to icon Kn: Xkn=Xtp, Ykn=Ytp, and move icon Kn to a new coordinate position that is easy to operate.

[0065] See Figure 4 In step S5, when the user touches the screen within the effective range of button Kn in the operation interface, the system sends the coordinates of the touch operation as the coordinates of icon Fn (Xfn, Yfn), realizing reverse control of the phone screen, such as precise focus in photography. Otherwise, normal reverse control touch coordinates (Xtp, Ytp) are sent. The operation flowchart is as follows:

[0066] Step S51: The user touches the screen of the sliding screen mirroring device. It is determined whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S52. If not, end.

[0067] Step S52: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Kn (Xkn, Ykn). If yes, proceed to step S53; otherwise, proceed to step S54.

[0068] Step S53: Send a reverse control signal to the mobile phone screen for the coordinates (Xfn, Yfn) of icon Fn, and then end;

[0069] Step S54: Send the reverse control signal of the mobile phone screen at the current touch coordinates (Xtp, Ytp), and then end.

[0070] See Figure 5 For a screen mirroring system that implements the above functions, its system architecture includes:

[0071] Main control IC for screen mirroring device;

[0072] WiFi module;

[0073] Bluetooth module;

[0074] A display screen with touch control functionality;

[0075] CS5261 chip;

[0076] The main control IC is model S800, which supports FreeRTOS and Linux systems. The RTOS system we are currently using has video signal processing capabilities and supports signal input functions such as serial port, GPIO, USB, WIFI, and HDMI_RX. It also supports LCD screen signal output such as RGB and MIPI. The software protocols support Aircast, Miracast projection technology protocols, as well as HID and UIBC technology protocols.

[0077] The WIFI module provides WIFI screen mirroring functionality and is connected via a USB interface.

[0078] The Bluetooth module, which provides functions such as Bluetooth HID touchscreen protocol, is connected via a serial port;

[0079] The display screen is used to provide the UI interface display and to display the mobile phone screen on the same screen. It is output through the MIPI port and is used to generate touch signals. It is connected through both the I2C port and IO.

[0080] The CS5261 chip is used to convert the mobile phone's TYPECDP signal into an HDMI signal, which is then connected via the HDMI_RX port.

[0081] The remaining power supply circuit, battery charging circuit, and TYPEC / CC line detection circuit are existing technologies and will not be described in detail in this application.

[0082] See Figure 6-8 Based on the aforementioned hardware system, we can implement the mobile phone screen mirroring function by writing software. In the settings mode, the user can interact with the screen via touch screen, and in the screen mirroring mode, the user can control the mobile phone screen via touch screen. This application combines the user interaction function via touch screen and the user control function via touch screen according to certain logic.

[0083] The screen mirroring device has many working modes. Since the main application of this invention is to switch and shift the focus point when shooting video, the working mode using this method is temporarily named the focus shift mode.

[0084] Based on this model, the screen mirroring device is set to focus-shift mode. See the attached diagram for its user interface. Figure 5 :

[0085] Mode key: Used to switch working modes. In addition to working in focus shift mode, this product can also work in other working modes.

[0086] Settings Button: Pressing this button will enter the focus shift mode settings interface. Pressing this button again will exit the settings interface and save the settings. The main contents of the settings are the coordinates (Xkn, Ykn) of button Kn and the coordinates (Xfn, Yfn) of icon Fn. Using different colors for button Kn and icon Fn makes it easier to visually distinguish different click points.

[0087] In the user interface, touching the button icon Kn will send a reverse control signal to the phone screen, with coordinates (Xfn, Yfn) corresponding to the coordinates of the icon Fn. As shown in the diagram, Fn has icons F0, F1, and F2, while Kn has icons K0, K1, and K2. K0 corresponds to F0, K1 to F1, and K2 to F2. If icons F0 or F1 are positioned on the subject being photographed, touching buttons K0 or K1 will cause the camera to focus on and capture the corresponding subject. If icon F2 is positioned on the start recording button, touching button K2 will start or stop recording.

[0088] When you touch an empty area in the user interface, the system will send a command to the phone in reverse, specifying the current touch coordinates (Xtp, Ytp) to control the phone screen. Similarly, the location of the Fn icon in the user interface will be treated the same way as an empty area; the system will send a command to the phone in reverse, specifying the current touch coordinates (Xtp, Ytp) to control the phone screen.

[0089] The image below shows the settings interface of the screen mirroring device in focus shift mode.

[0090] In the settings interface, you can drag the focus icon Fn to any position on the screen.

[0091] To set the position of button Kn, first press and hold button Kn to activate it and set it, and then drag and drop it to adjust its position.

[0092] After setting, press the setting button again to save the coordinates (Xfn, Yfn) and (Xkn, Ykn), and return to the operation interface.

[0093] Then, press the settings button again to enter the focus shifting operation interface. At this time, touching different buttons Kn on the screen mirroring device will send the coordinates of the corresponding focus Fn, thereby achieving precise focus shifting operation.

[0094] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for precise blind screen clicking based on a screen mirroring device, characterized in that, Including the following steps: S1: On a screen mirroring device with reverse touch screen functionality, an embedded program is used to create one or more icons Fn (Xfn, Yfn) on the screen of the screen mirroring device to set the click position of the screen. S2: Create the same number of buttons Kn(Xkn, Ykn) as Fn to initiate the click action; S3: When the screen mirroring device is connected to the phone, the two are in screen mirroring mode. The system enters the settings interface through the settings button of the screen mirroring device. In the settings interface, the user touches and drags the Fn icons, places multiple Fn icons on the phone screen where they need to be clicked, and records their coordinate values ​​(Xfn, Yfn). S4: Move button Kn (Xkn, Ykn) to a position suitable for finger operation; S5: After the setup is complete, the system returns to the operation interface. At this time, touch the button Kn on the screen mirroring device, and the system will start the screen mirroring device to reverse the touch screen operation of the mobile phone screen, and send a command to the mobile phone with the coordinates (Xfn, Yfn) of the icon Fn.

2. The method for precise blind screen clicking based on a screen mirroring device as described in claim 1, characterized in that, The specific implementation method for dragging the icon Fn to move its position in step S3 is as follows: Step S31: Press the settings button, and the system enters the settings interface. The user touches and slides the screen of the screen mirroring device to determine whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S32; otherwise, end. Step S32: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Fn (Xfn, Yfn). If yes, proceed to step S33; otherwise, end. Step S33: Determine whether the coordinates (Xtp, Ytp) are equal to the icon Fn (Xfn, Yfn). If they are equal, the process ends; otherwise, proceed to step S34. Step S34: Assign values ​​to icon Fn: Xfn=Xtp, Yfn=Ytp, and move icon Fn to the new coordinate position of the subject to be focused.

3. The method for precise blind screen clicking based on a screen mirroring device as described in claim 1, characterized in that, In step S4, the position of button Kn can also be adjusted by the user. Generally, this is done by entering the button setting mode through a special key combination. In this setting mode, button Kn can be touched and moved to a position on the screen of the screen mirroring device that is convenient for operation with the thumb. The setting flowchart is as follows: Step S41: Press the settings button, and the system enters the settings interface. The user touches and slides the screen of the screen mirroring device to determine whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S42; otherwise, end. Step S42: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Kn (Xkn, Ykn). If yes, proceed to step S33; otherwise, end. Step S43: Determine whether the coordinates (Xtp, Ytp) are equal to the icon Kn (Xkn, Ykn). If they are equal, the process ends; otherwise, proceed to step S34. Step S44: Assign values ​​to icon Kn: Xkn=Xtp, Ykn=Ytp, and move icon Kn to a new coordinate position that is easy to operate.

4. The method for precise blind screen clicking based on a screen mirroring device as described in claim 1, characterized in that, In step S5, when the user touches the screen within the effective range of button Kn in the operation interface, the system sends the coordinates of the touch operation as the coordinates of icon Fn (Xfn, Yfn) to achieve precise focus in photography. Otherwise, it sends the normal reverse touch coordinates (Xtp, Ytp). The operation flowchart is as follows: Step S51: The user touches the screen of the sliding screen mirroring device. It is determined whether it is a touch screen event coordinate (Xtp, Ytp). If it is, proceed to step S52. If not, end. Step S52: Determine whether the touch screen event coordinates (Xtp, Ytp) are within the range of the icon Kn (Xkn, Ykn). If yes, proceed to step S53; otherwise, proceed to step S54. Step S53: Send a reverse control signal to the mobile phone screen for the coordinates (Xfn, Yfn) of icon Fn, and then end; Step S54: Send the reverse control signal of the mobile phone screen at the current touch coordinates (Xtp, Ytp), and then end.

Images