Intelligent calibration method for mouse cursor position in end-to-end mouse control

Abstract

The invention relates to the technical field of end-to-end communication, intelligent calibration of mouse cursor positions and the like, and provides an intelligent calibration method for mouse cursor positions in end-to-end mouse control, which comprises the following steps of: acquiring a screen resolution of main control equipment and a screen picture of controlled equipment, and adjusting the screen picture of the controlled equipment according to the resolution of the main control equipment; dividing a screen of the controlled equipment into equal-proportion grid areas; searching and determining the current position of a mouse cursor of the controlled equipment, and mapping the current position to the grid area; and acquiring coordinates of a mouse cursor of the master control equipment, comparing the coordinates with grid coordinates of the controlled equipment, if the coordinates are inconsistent, calculating an offset distance needing to be adjusted, and adjusting the cursor position of the controlled end until the coordinates are consistent with the coordinates of the master control end. According to the method, accurate synchronization of mouse positions among heterogeneous systems is realized through establishment of the equal-proportion grid model, image recognition and positioning and coordinate compensation based on the grid difference value, the problem of non-synchronization of cursor positions caused by resolution difference, event hijacking or abnormal response is solved, and the control accuracy and the system robustness are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of end-to-end communication, intelligent calibration of mouse cursor position and the like, and particularly relates to a method for intelligent calibration of mouse cursor position in end-to-end mouse control. BACKGROUND

[0002] In the prior art, end-to-end communication has realized control of mouse actions such as movement, clicking and scrolling, but cannot make the mouse position of the master and slave devices the same in proportion in two systems with different screen resolutions, different mouse accuracies and different moving speeds, that is, the mouse cannot realize position synchronization in both ends in the case of the same resolution but different moving speeds and accuracies. In the case that the slave device runs a program to hijack and embezzle mouse events or part of mouse events cannot be transmitted to the mouse cursor control program, the mouse position cannot be kept in both sides. Therefore, when the cursor of the slave device cannot be consistent with the cursor position of the master device, the cursor of the slave device needs to be actively controlled to move to the corresponding problem, so as to realize a scheme of synchronizing the mouse cursor positions of the master and slave devices in both sides.

[0003] In summary, in the prior art, when the cursor of the slave device cannot be consistent with the cursor position of the master device, there is no technical scheme to realize synchronization of the mouse cursor positions of the master and slave devices in both sides. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application provides a method for intelligent calibration of mouse cursor position in end-to-end mouse control, so as to realize accurate synchronization of the mouse cursor positions of the master and slave devices in both sides.

[0005] The method for intelligent calibration of mouse cursor position in end-to-end mouse control provided by the present application comprises the following steps.

[0006] A current screen resolution of a master device is acquired, and a current screen picture of a slave device is acquired, the screen picture of the slave device is divided into a plurality of grid areas in the same proportion as the screen resolution of the master device according to the screen resolution of the master device;

[0007] A current position of a mouse cursor is found and determined in the screen picture of the slave device, the position of the mouse cursor found and determined is mapped into a corresponding grid area, and horizontal and vertical coordinates of the grid area are obtained;

[0008] The current coordinate position of the mouse cursor on the master device is acquired, the coordinates of the mouse cursor on the master device are compared with the grid area coordinates of the mouse cursor of the controlled device, if the two end coordinates are inconsistent, the offset distance that the mouse cursor of the controlled device needs to adjust is automatically calculated, and the position of the mouse cursor of the controlled device is adjusted according to the offset distance until the position of the mouse cursor of the controlled device is consistent with the position of the mouse cursor of the master device.

[0009] Compared with the prior art, the present application has the following beneficial effects:

[0010] The present application provides a kind of intelligent calibration method of mouse cursor position in end-to-end mouse control, by acquiring the current screen resolution of master device, and the current screen picture of controlled device is acquired, according to the screen resolution of the master device, the screen picture of the controlled device is divided into multiple grid areas with the same proportion as the screen resolution of the master device, the current position of mouse cursor is found and determined in the screen picture of the controlled device, the position of the mouse cursor found and determined is mapped into corresponding grid area, and the horizontal and vertical coordinates of grid area are obtained, the current coordinate position of the mouse cursor on the master device is acquired, the coordinates of the mouse cursor on the master device are compared with the grid area coordinates of the mouse cursor of the controlled device, if the two end coordinates are inconsistent, the offset distance that the mouse cursor of the controlled device needs to adjust is automatically calculated, and the position of the mouse cursor of the controlled device is adjusted according to the offset distance until the position of the mouse cursor of the controlled device is consistent with the position of the mouse cursor of the master device, so that when the controlled device cursor cannot be consistent with the position of the master device cursor, the accurate synchronization of the mouse cursor position of the master device and the controlled device is realized. BRIEF DESCRIPTION OF DRAWINGS

[0011] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. Some specific embodiments of the present application will be described in detail hereinafter with reference to the drawings in an exemplary and non-limiting manner. The same reference signs in the drawings indicate the same or similar components or parts, and those skilled in the art should understand that the drawings are not necessarily drawn to scale. In the drawings:

[0012] Fig. 1 is a flowchart of the intelligent calibration method of mouse cursor position in end-to-end mouse control of the embodiment of the present application;

[0013] Fig. 2 is a state diagram when there is a position deviation between the master mouse and the controlled device mouse in the picture of the master device before calibration in the embodiment of the present application;

[0014] Fig. 3 Fig. 4 is a schematic diagram of a state where the master mouse and the slave mouse are overlapped in the screen of the master device after calibration in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the persons skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by the persons skilled in the art without creative labor should belong to the protection scope of the present application.

[0016] Referring to Figs. 1-3 The present embodiment provides a kind of intelligent calibration method of mouse cursor position in end-to-end mouse control, comprising the following steps:

[0017] S101, the current screen resolution of master device is acquired, and the current screen picture of slave device is acquired, according to the screen resolution of the master device, the screen picture of the slave device is divided into multiple grid areas with the same proportion as the screen resolution of the master device;

[0018] S102, the current position of mouse cursor is found and determined in the screen picture of the slave device, the position of the mouse cursor found and determined is mapped to the corresponding grid area, and the horizontal and vertical coordinates of the grid area are obtained;

[0019] S103, the current coordinate position of mouse cursor on the master device is acquired, the coordinates of mouse cursor on the master device are compared with the grid area coordinates of the slave mouse cursor, if the coordinates of the master device and the slave device are inconsistent, the offset distance that the slave mouse cursor needs to adjust is automatically calculated, the position of the slave mouse cursor is adjusted according to the offset distance until the position of the slave mouse cursor is consistent with the position of the master mouse cursor.

[0020] It should be noted that in the prior art, due to the differences in screen resolution, mouse accuracy, mouse speed, event loss, etc. between the master device and the controlled device, the cursor position cannot be one-to-one corresponding, and the synchronization accuracy is insufficient. In addition, in the prior art, the controlled end system may have program hijacking or mouse event transmission failure, resulting in mouse position drift or loss, and the master end and the controlled end cursor are not synchronized. In the embodiment, the current screen resolution of the master device is obtained, and the current screen picture of the controlled device is obtained, to provide basic geometric mapping data for subsequent synchronization. By actively obtaining the resolution and picture data of the master and slave devices, it is different from the traditional method of only sending mouse events by the master, and it has the prerequisite for picture alignment calibration, thereby establishing a unified coordinate mapping model, solving the cursor mapping deviation caused by different resolutions. It can be understood that the traditional technology does not actively compare the resolution of the master and slave ends, and only relies on event simulation transmission; the embodiment is based on the picture mapping strategy, and prepares for subsequent grid mapping, with structural improvement.

[0021] It should also be noted that in the embodiment, according to the screen resolution of the master device, the screen picture of the controlled device is divided into a plurality of grid areas with the same proportion as the screen resolution of the master device, thereby realizing the establishment of a coordinate alignment discrete model by using a grid mapping method, forming a logical alignment framework in space, so that two screens under different resolutions have a logical proportionally-mappable basis, thereby accurately estimating the offset and quantitatively adjusting the target. It can be understood that in the traditional system, only pixel position is used for cursor control, and in the embodiment, grid segmentation is used to improve fault tolerance with disturbance resistance.

[0022] It should also be noted that in the embodiment, the current position of the mouse cursor is found and determined in the picture of the controlled device, and is mapped into the grid area to obtain the horizontal and vertical coordinates, thereby realizing the identification of the actual position of the current controlled end cursor and the conversion into grid coordinates instead of original pixel coordinates, ensuring that even in the case of partial mouse event hijacking of the controlled device, actual mouse position and instruction offset, the current accurate position can still be determined through visual or system-level positioning. It can be understood that the prior art mostly controls the cursor by event triggering, and cannot handle the real position perception problem after event exception or hijacking, and the embodiment effectively solves this problem through image and coordinate mapping.

[0023] It also needs to be explained that in the embodiment, the master device mouse coordinates are acquired and compared with the controlled device grid coordinates, thereby establishing a set of benchmarks for difference analysis, matching and comparing two different coordinate systems (master pixel coordinates and controlled grid coordinates), effectively finding the synchronous offset position, quantifying the adjustable parameters, and solving the unknown difference problem caused by the lack of event precision. It should be noted that in the embodiment, the coordinate comparison method is combined with the grid area, which is to decouple the influence of different system parameters, and is not a direct pixel comparison, so as to improve the cross-platform adaptability.

[0024] It also needs to be explained that in the embodiment, if the coordinates are inconsistent, the offset distance that the controlled device mouse cursor needs to adjust is calculated, and its position is adjusted until it is consistent, thereby realizing the automatic compensation of the controlled end mouse position and restoring the master-slave device cursor synchronization. The adjustment is a fine control based on the calculation result of the grid offset amount, which can automatically return to the synchronous state after the problems such as cursor drift, event hijacking, and packet loss, and guarantee the control accuracy and human-computer interaction consistency.

[0025] In general, the intelligent calibration method of end-to-end mouse cursor position proposed in the embodiment of the application is not a simple extension of mouse control logic, but a comprehensive solution combining cross-resolution mapping, grid modeling, anomaly detection, and closed-loop offset correction. The technical features have synergistic compatibility, and give a complete solution path for specific technical problems. It realizes: from inconsistent resolution to area isometric mapping; from event simulation to actual screen perception; from one-way control to two-way coordinate synchronization correction.

[0026] Preferably, after completing the offset adjustment of the mouse cursor position of the controlled device and the mouse cursor position of the master device once, the movement ratio value corresponding to the relative coordinate difference of the mouse cursors of the master device and the controlled device in the offset adjustment is stored; when calibrating the mouse cursor position next time, the master device mouse cursor coordinates are converted according to the movement ratio value stored last time to estimate the position where the controlled device mouse cursor should be, and the controlled device mouse cursor is adjusted to the estimated position.

[0027] It should be noted that in the embodiment, after the offset adjustment of the mouse cursor position of the controlled device and the mouse cursor position of the host device is completed, the movement ratio value corresponding to the relative coordinate difference of the mouse cursors of the host device and the controlled device in the offset adjustment is stored, so that the core mathematical quantity coordinate difference ratio in the one-time calibration operation is extracted and stored through the state memory mechanism, the system obtains the self-learning ability, can remember the last calibration result, and thus the mapping function of mutual conversion between the host and the controlled is established. When the mouse cursor position calibration is performed next time, the mouse cursor coordinate of the host device is converted according to the movement ratio value stored last time, so that the host end coordinate is corrected based on the historical ratio value, and the control continuity and fault tolerance are greatly improved. In addition, in the embodiment, the position where the mouse cursor of the controlled device should be is estimated, and the mouse cursor of the controlled device is adjusted to the estimated position, so that a predictive adjustment strategy is provided, and the real-time measurement of the current offset is no longer relied on, but the historical ratio is combined to adjust in advance, the real-time and precision of the mouse position calibration are effectively improved, and the embodiment is particularly suitable for heterogeneous devices with large network delay and slow event response.

[0028] Preferably, when the screen picture of the controlled device is divided into a plurality of grid regions, the size standard of each grid unit on the screen of the host device is determined based on the proportional relationship between the number of horizontal pixels and the number of vertical pixels in the current screen resolution of the host device; and the grid division lines are drawn in the screen picture of the controlled device along the horizontal direction and the vertical direction respectively according to the size standard of each grid unit on the screen of the host device, so that the size of each grid unit in the screen of the controlled device is consistent with the size of the host device grid unit in proportion. The screen picture of the controlled device is divided by the grid division lines drawn along the horizontal direction and the vertical direction, and the plurality of grid regions are obtained.

[0029] It should be noted that in the embodiment, the size standard of each grid unit on the screen of the master device is determined based on the proportional relationship between the number of horizontal pixels and the number of vertical pixels in the current screen resolution of the master device, so as to provide the horizontal and vertical pixel proportion as the division basis, instead of using fixed pixel size or fixed grid number, to construct a logical region division mode matching the visual layout of the master device. On the master device, no matter the display resolution, the reference grid with unified structure proportion and consistent geometric characteristics can be divided, which is helpful for the establishment of mapping logic. In addition, according to the size standard of each grid unit on the screen of the master device, grid division lines are drawn in the screen image of the controlled device along the horizontal direction and the vertical direction respectively, so that the size of each grid unit in the screen of the controlled device is consistent with the size of the master device grid unit in proportion, thereby explicitly indicating that the grid division operation is performed on the controlled device side, but the grid standard of the controlled side depends on the master side proportion, emphasizing the consistency and directionality of the grid division, realizing the logical mapping unit with consistent horizontal and vertical size proportion under the heterogeneous display system, and effectively avoiding the cursor position misplacement caused by the difference in screen aspect ratio. Even if the resolutions are inconsistent, one-to-one relative position mapping can be realized, which fundamentally solves the mouse position error problem between different devices. At the same time, by drawing the grid division lines along the horizontal direction and the vertical direction, the screen image of the controlled device is divided to obtain a plurality of grid regions, thereby establishing a spatial reference model for algorithm positioning, searching and mapping in the actual drawing line division manner, and forming a grid region.

[0030] Preferably, when searching and determining the current position of the mouse cursor in the screen image of the controlled device, the image recognition technology is used to automatically recognize and locate the position of the mouse cursor icon in the screen image of the controlled device. It should be noted that in the embodiment, the image recognition technology (such as template matching, image feature extraction, etc.) is used to automatically recognize and locate the position of the mouse cursor icon in the screen image of the controlled device, and the mouse icon is searched directly in the screen image, so as to avoid the coordinate reading based on the event interface. Even if the mouse event channel is invalid, the actual visual position of the controlled end cursor can be accurately obtained through image recognition, which guarantees the availability and robustness of the position synchronization basis.

[0031] Preferably, the automatic calculation of the offset distance that the controlled device mouse cursor needs to adjust comprises: obtaining the actual coordinate position of the master device mouse cursor and the grid coordinate position of the controlled device mouse cursor; calculating the horizontal and vertical offset amounts according to the horizontal and vertical differences between the actual coordinates of the master device mouse cursor and the grid coordinates of the controlled device mouse cursor; and converting the calculated horizontal and vertical offset amounts into the actual offset distance of the controlled device mouse cursor in proportion to obtain the offset distance that the controlled device mouse cursor needs to adjust. It should be noted that in this embodiment, the horizontal and vertical differences are calculated based on the master actual coordinates and the controlled grid coordinates, and the real offset distance that the controlled end needs to adjust is converted by proportion, so that the cross-device mouse coordinate synchronization is more accurate, and the mathematical consistency and adaptability of coordinate conversion in a heterogeneous display environment are supported.

[0032] Preferably, the adjustment of the controlled device cursor position is realized by generating a mouse control instruction, and the mouse control instruction is a mouse movement instruction containing specific offset amount information. It should be noted that in this embodiment, a mouse movement control instruction with explicit offset amount parameters is generated, so that the system accurately moves the cursor according to the physical level. Compared with the jump coordinate type control, this embodiment supports continuous action simulation and automatic synchronization, and has stronger adaptability to dynamic windows, game environments, etc.

[0033] Preferably, the movement proportion value corresponding to the relative coordinate difference between the master device and the controlled device mouse cursor in the offset adjustment is automatically updated after each calibration and stored in the database. It should be noted that after each calibration, the proportion value is updated in real time and written into the database for continuous maintenance and reuse. The proportion value can be used as a dynamic learning model parameter to improve the long-term stability and cross-device generalization ability of the system.

[0034] Preferably, the intelligent calibration method is suitable for realizing accurate synchronization of mouse cursors at both ends when the resolutions of the master device and the controlled device are inconsistent or the mouse movement speeds and accuracies are inconsistent. It should be noted that this embodiment proposes a universal applicable model for adaptability boundary, which has universal adaptability to different resolutions, mouse speeds and accuracies.

[0035] Preferably, the intelligent calibration method is implemented through an AI program running in the host device, direct control of the host device mouse cursor is realized through user operation of the host device, real-time position of the host device mouse cursor is synchronized to the AI program, and synchronization state of the host device mouse cursor and the controlled device mouse cursor is judged through the AI program. It should be noted that the intelligent calibration method is implemented through an AI program running in the host device, direct control of the host device mouse cursor is realized through user operation of the host device, real-time position of the host device mouse cursor is synchronized to the AI program, and synchronization state of the host device mouse cursor and the controlled device mouse cursor is judged through the AI program, thereby realizing the change from passive synchronization to intelligent driving synchronization.

[0036] Preferably, when it is detected that the positions of the host device mouse cursor and the controlled device mouse cursor are not synchronized, the AI program calculates the offset amount that the controlled device mouse cursor needs to adjust, and adjusts the position of the controlled device mouse cursor through an external control instruction until synchronization is completed; in the whole process, the host device performs operation without modifying the existing source system of the controlled device, and the capability of the existing source system of the controlled device is ensured to remain unchanged. It should be noted that in the embodiment, the AI program calculates the offset amount that the controlled device mouse cursor needs to adjust, and adjusts the position of the controlled device mouse cursor through an external control instruction until synchronization is completed, in the whole process, the host device performs operation without modifying the existing source system of the controlled device, and the capability of the existing source system of the controlled device is ensured to remain unchanged, thereby realizing external expansion of the system without upgrading the existing source system of the controlled device, i.e., the original capability of the source system is not modified, and the AI can replace the manual monitoring capability without embedding the modified source system, and the system can be expanded externally without damage. Exemplarily, the monitoring program runs in the host device, and the execution program also runs in the host device, the effect is synchronized to the controlled device, and monitoring and entity are separated.

[0037] It should be noted that the above embodiments are only preferred specific embodiments of the present application, the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application, and the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for intelligent calibration of mouse cursor position in end-to-end mouse control, characterized in that, The method comprises the following steps: acquiring the current screen resolution of the host device and the current screen image of the controlled device, dividing the screen image of the controlled device into a plurality of grid areas in the same proportion as the screen resolution of the host device; finding and determining the current position of the mouse cursor in the screen image of the controlled device, mapping the found and determined position of the mouse cursor into the corresponding grid area, and obtaining the horizontal and vertical coordinates of the grid area; acquiring the current coordinate position of the mouse cursor on the host device, comparing the coordinate of the mouse cursor on the host device with the grid coordinate of the mouse cursor of the controlled device, if the coordinates of the two ends are inconsistent, automatically calculating the offset distance that the mouse cursor of the controlled device needs to adjust, adjusting the position of the mouse cursor of the controlled device according to the offset distance until the position of the mouse cursor of the controlled device is consistent with the position of the mouse cursor of the host device.

2. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 1, wherein, After completing the offset adjustment of the position of the mouse cursor of the controlled device and the position of the mouse cursor of the host device once, the moving proportion value corresponding to the relative coordinate difference of the mouse cursor of the host device and the mouse cursor of the controlled device in the offset adjustment is stored; when calibrating the position of the mouse cursor next time, the mouse cursor coordinate of the host device is converted according to the moving proportion value stored last time, so as to estimate the position where the mouse cursor of the controlled device should be, and adjust the mouse cursor of the controlled device to the estimated position.

3. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 1, wherein, When dividing the screen image of the controlled device into a plurality of grid areas, based on the proportional relationship between the horizontal pixel number and the vertical pixel number in the current screen resolution of the host device, the size standard of each grid unit on the screen of the host device is determined; according to the size standard of each grid unit on the screen of the host device, grid division lines are drawn in the horizontal direction and the vertical direction in the screen image of the controlled device respectively, so that the size of each grid unit in the screen of the controlled device is consistent with the size of the grid unit of the host device in proportion, and the screen image of the controlled device is divided by the grid division lines drawn in the horizontal direction and the vertical direction, to obtain the plurality of grid areas.

4. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 1, wherein, When finding and determining the current position of the mouse cursor in the screen image of the controlled device, the position where the mouse cursor icon is located is automatically recognized and positioned in the screen image of the controlled device by image recognition technology.

5. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 1, wherein, When automatically calculating the offset distance that the mouse cursor of the controlled device needs to adjust, the following steps are included: acquiring the actual coordinate position of the mouse cursor of the host device and the grid coordinate position of the mouse cursor of the controlled device; calculating the offset amount in the horizontal direction and the vertical direction respectively according to the horizontal and vertical difference between the actual coordinate of the mouse cursor of the host device and the grid coordinate of the mouse cursor of the controlled device; converting the calculated offset amount in the horizontal direction and the vertical direction into the actual offset distance of the mouse cursor of the controlled device in proportion, to obtain the offset distance that the mouse cursor of the controlled device needs to adjust.

6. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 1, wherein, The adjustment of the position of the mouse cursor of the controlled device is realized by generating a mouse control instruction, and the mouse control instruction is a mouse moving instruction containing specific offset amount information.

7. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 2, wherein, The moving ratio value corresponding to the relative coordinate difference of the master device and the controlled device mouse cursor in the offset adjustment is automatically updated and stored in the database after each calibration is completed.

8. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to claim 1, wherein, The intelligent calibration method is suitable for realizing accurate synchronization of mouse cursors at both ends when the resolution, mouse moving speed and accuracy of the master device and the controlled device are inconsistent.

9. The intelligent calibration of mouse cursor position in end-to-end mouse control method according to any one of claims 1-8, wherein, The intelligent calibration method is realized through an AI program, the AI program runs in the master device, and the direct control of the master device mouse cursor is realized through the user operation of the master device; the real-time position of the master device mouse cursor is synchronized to the AI program, and the synchronization state of the master device mouse cursor and the controlled device mouse cursor is judged through the AI program.

10. The method of intelligent calibration of mouse cursor position in end-to-end mouse control according to claim 9, wherein, When it is detected that the positions of the master device mouse cursor and the controlled device mouse cursor are out of synchronization, the AI program calculates the offset amount that the controlled device mouse cursor needs to adjust, and adjusts the position of the controlled device mouse cursor through an external control instruction until the synchronization is completed; in the whole process, the master device executes the operation without modifying the existing source system of the controlled device, and ensures that the capability of the existing source system of the controlled device remains unchanged.

Images