Key display operation device and mobile phone control system

Abstract

The invention discloses a key display operation device and a mobile phone control system.The key display operation device is characterized in that when a trigger event is detected, a mouse moving instruction is generated, a mouse is displayed, then whether a system API is called successfully or not is judged, if yes, a full-screen suspension frame is displayed, and the full-screen suspension frame is used for monitoring the moving position of the mouse in real time and controlling mouse moving and click operation; if not, a screenshot instruction is sent, image elements corresponding to screenshot content are obtained, and the mouse moving position and click operation are determined according to the image elements. A driver can realize WeChat information reading, WeChat information replying, navigation opening and music opening of the mobile phone through simple key clicking operation, the accuracy rate is high, the operation is simple, the potential safety hazard caused by distraction of the driver due to mobile phone operation is reduced, and great economic and social values are achieved.

Description

[0001] The present invention is a divisional application, the original application number is 2019113300789, the application date is 2019.12.21, and the name is key display operation method. Technical Field

[0002] The invention relates to the field of vehicle-mounted displays, and in particular to a key display operating device and a mobile phone control system. Background Art

[0003] In the era of mobile Internet, smart phones are rapidly popularized, and their performance is becoming more and more powerful, with better running speed and efficiency than general car navigation. Although, some existing car navigation systems also have voice assistants, which can process navigation, music and other functions by voice. However, in the known field, voice assistants need to use specific languages ​​(such as standard Mandarin, English, etc.), and have high requirements for the speaker's speaking speed, pronunciation accuracy, etc. The voice recognition function often has a low recognition rate or is prone to misrecognition. In this case, it is easy to distract the driver and cause safety hazards. Moreover, general car navigation cannot respond to WeChat replies, and the usage rate of WeChat on mobile phones is very high in life today. During driving, if the driver bows his head to operate the mobile phone by hand to send WeChat messages, his sight is completely away from the road ahead, and it is easy to fail to make timely judgments on some emergencies on the road, which has a very large safety hazard. Summary of the invention

[0004] Based on this, it is necessary to provide a key display operation device and a mobile phone control system to improve driving safety in response to the existing technical problems.

[0005] In order to solve the problems in the prior art, the first aspect of the present invention provides a key display operation device, the method is applied to a mobile phone display screen, and the key display operation device comprises:

[0006] A generation module, used for generating a mouse movement instruction when a trigger event is detected;

[0007] The judgment module is used to judge whether the system API is successfully called;

[0008] A control module is used to monitor the position of the mouse movement in real time according to the mouse movement instruction and in combination with the full-screen floating frame and control the mouse movement and click operation;

[0009] The second control module is used to send a screenshot instruction, obtain the image element corresponding to the screenshot content, and determine the mouse movement position and click operation according to the image element.

[0010] In one embodiment, the generation module is used to store the correspondence between the mouse movement operation and the control instruction on the mobile phone display screen, each of the control instructions corresponds to a movement position and a click operation in one direction of the mobile phone display screen, or a movement instruction and a click operation at a specific position, and the final movement position and click operation of the mouse are determined based on the correspondence.

[0011] The second aspect of the present invention further provides a mobile phone control system, which is applied to a mobile phone and includes the key display operation device described above.

[0012] In summary, the above technical solution is adopted to generate a mouse movement instruction when a trigger event is detected, display the mouse, and then determine whether the system API is successfully called. If so, display a full-screen floating frame, which is used to monitor the position of the mouse movement in real time. When the full-screen floating frame detects a mouse movement instruction, the mouse movement direction and movement distance are determined based on the mouse movement instruction, and the mouse is moved and clicked based on the mouse movement direction and movement distance; if not, send a screenshot instruction, obtain the image element corresponding to the screenshot content, and determine the position of the mouse movement and the click operation based on the image element. Drivers can read information (text / voice) on WeChat, reply to information (text / voice) on WeChat, open navigation (one-click to go home or one-click to return to the company), and open music (one-click to open the favorites music list) through simple button clicks. It has high accuracy and simple operation, reduces the safety hazards caused by distraction of drivers due to mobile phone operation, and has great economic and social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of a flow chart of a key display operation method provided by an embodiment of the present invention;

[0014] Figure 2 for Figure 1 A detailed flow chart of a key display operation method shown;

[0015] Figure 3 for Figure 1 A schematic diagram of an image processing method is shown;

[0016] Figure 4 A structural block diagram of a key display operation device provided by an embodiment of the present invention;

[0017] Figure 5 A structural block diagram of a mobile phone control system provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0018] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0019] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0021] There are some voice assistants on the market that use voice to control mobile phones to perform some tasks (such as Siri on Apple phones, Jovi on VIVO phones, etc.). These software all use the permissions provided by the system and call the interfaces provided by third-party software (such as playing the songs in the current list in QQ Music, etc.). However, when it is necessary to use interface functions that are not provided by third-party apps (such as the third album in the personal collection album in QQ Music), these voice control assistants are powerless due to functional limitations.

[0022] See also Figures 1 to 3 The first aspect of the present invention provides a key display operation method, which is applied to a mobile phone display screen, and the method comprises:

[0023] Step 1: Generate a mouse movement command and display the mouse when a trigger event is detected. The trigger event can be a joystick movement, a button press, or a voice command.

[0024] Step 2: Determine whether the system API (English full name: Application Programming Interface) is successfully called.

[0025] If so, the system API call is successful, and a full-screen floating frame is displayed. The full-screen floating frame can be a transparent or colored floating frame, and the full-screen floating frame is consistent with the screen display range of the mobile phone. The full-screen floating frame is used to monitor the position of the mouse movement in real time. When the full-screen floating frame detects the mouse movement instruction, the mouse movement direction and movement distance are determined based on the mouse movement instruction, and the mouse is moved and clicked based on the mouse movement direction and movement distance.

[0026] Wherein, determining the moving direction and moving distance of the mouse based on the mouse moving instruction comprises the steps of:

[0027] 1. determining a moving direction of the mouse based on the mouse moving instruction;

[0028] 2. determining the number of times the mouse movement instruction acts on the mobile terminal, and determining the movement distance of the mouse;

[0029] 3. Determine the duration of the mouse movement instruction acting on the mobile terminal and determine the continuous movement distance of the mouse.

[0030] If not, the system API call fails, then a screenshot instruction is sent to obtain the image element corresponding to the screenshot content, and the mouse movement position and click operation are determined according to the image element. The image element includes shape element, color element, character element and preset template element.

[0031] The reason for this setting is that some automatic clicking software on the market only completes the information reading and simulation operation of interface elements by calling the auxiliary functions of the mobile phone system. When the system does not provide the corresponding functions, it is impossible to obtain information such as the status or position of the required elements, nor can it perform simulation operations restricted by the system (such as long press, etc.). The application function is restricted by the calling interface API of the permissions provided by the system, and cannot meet all the needs of automated execution tasks well. At this time, the execution uses image processing to judge the elements through shapes, colors, characters or preset templates, and control the position of the mouse movement and click operations. Even if the system API is not supported or the call fails, the mouse can be used normally to accurately control the movement and click operations of the mouse.

[0032] In the first embodiment, if the system API call is successful, the method of real-time monitoring of the mouse movement position through a full-screen floating frame is described in detail:

[0033] When the mobile phone APP receives the mouse movement command (moves to the target coordinate point (x0, y0)),

[0034] (a) First, a transparent full-screen system floating window is opened. The floating window defines a built-in response function for mouse movement and clicks. When the mouse moves or clicks on it, the current coordinates of the mouse and other position information (x1, y1) can be obtained;

[0035] (b) Calculate the difference between the current mouse coordinates and the target coordinates (Dx, Dy), where Dx = x0-x1, Dy = y0-y1. The difference is the distance the mouse should move in the X and Y directions next time.

[0036] (c) The mobile phone APP sends the moving distance (Dx, Dy) to the Bluetooth composite device through the Bluetooth communication protocol;

[0037] (d) After receiving the moving distance (Dx, Dy), the composite device determines whether Dx and Dy are within the limit range of a single mouse movement. If so, Dx is moved in the X direction and Dy is moved in the Y direction at one time. If Dx or Dy exceeds the limit range of a single mouse movement, Dx or Dy is split into multiple movements and moved in the X direction or Y direction multiple times until the total distance moved in the X direction reaches Dx and the total distance moved in the Y direction reaches Dy.

[0038] (e) Since the mouse may freeze or lose Bluetooth packets during movement, the mouse may not be able to move accurately in the X direction Dx or in the Y direction Dy, so it is necessary to confirm again whether the mouse reaches the target coordinate point (x0, y0) after movement;

[0039] (f) Assume that the coordinates captured by the floating window after the mouse moves are (x2, y2), and make the difference (Dx, Dy) again, where Dx = x0-x2, Dy = y0-y2. When Dx is not equal to 0 or Dy is not equal to 0, it means that the mouse has not moved to the target coordinates, and repeat back to the above step (c) to move the mouse to the target coordinate point again; when Dx and Dy are both 0, it means that the mouse has accurately moved to the target coordinate point, the mobile phone APP closes the full-screen system floating window, and the movement process is successfully completed.

[0040] In the second embodiment, if the system API call fails, the image elements are identified in the screenshot through image processing technology, and the position of the mouse movement and the click operation are determined according to the image elements. The identification can be carried out in the following four ways:

[0041] 1. Use OCR (character recognition) technology to perform character recognition on the entire image or a specified area. The recognized text will be converted into text and saved in the memory queue of the APP.

[0042] 2. Use shape recognition function to identify specified shapes such as square, circle, ellipse, etc. in the image to obtain their coordinates, quantity information, etc.

[0043] 3. Use the color recognition function to identify the number of areas of specified color on the image and record their area, coordinate information, etc.

[0044] 4. Match the current image with the pre-made image template through the matching function to obtain the number of matching target points, the zoom factor and their coordinate positions whose similarity with the template reaches a certain threshold.

[0045] The step of obtaining the image element corresponding to the screenshot content includes:

[0046] 1. A full-screen content original image acquisition unit is used to acquire the original image of the full-screen content of the mobile phone screen when the screenshot instruction is detected.

[0047] 2. An image element recognition unit is used to recognize shape elements, color elements, character elements and preset template elements of the image content.

[0048] 3. An execution unit, used to determine the position of the mouse movement and the click operation according to the image element.

[0049] Some automatic clicking software on the market only completes information reading and simulation operations of interface elements by calling the auxiliary functions of the mobile phone system. When the system does not provide corresponding functions, it is impossible to obtain information such as the status or position of the required elements, nor can it perform simulation operations restricted by the system (such as long pressing, etc.). The application functions are restricted by the calling interface of the permissions provided by the system and cannot meet all the needs of automated execution tasks well.

[0050] Please also read Figure 4 The second aspect of the present invention further provides a key display operation device, the key display operation device is applied to a mobile phone display screen, and the key display operation device comprises:

[0051] The generation module is used to generate a mouse movement instruction when a trigger event is detected. The trigger event can be a joystick movement, a key press, or a voice command.

[0052] The judgment module is used to judge whether the system API is successfully called.

[0053] A control module is used to monitor the position of the mouse movement in real time according to the mouse movement instruction and in combination with the full-screen floating frame and control the mouse movement and click operation.

[0054] The second control module is used to send a screenshot instruction, obtain the image element corresponding to the screenshot content, and determine the mouse movement position and click operation according to the image element.

[0055] In one embodiment, the generation module is used to store the correspondence between the mouse movement operation and the control instruction on the mobile phone display screen, each of the control instructions corresponds to a movement position and a click operation in one direction of the mobile phone display screen, or a movement instruction and a click operation at a specific position, and the final movement position and click operation of the mouse are determined based on the correspondence.

[0056] Please also read Figure 5 The third aspect of the present invention further provides a mobile phone control system, which is applied to a mobile phone and includes the key display operation device described above. The key display operation device can be built into the mobile phone and become an independently running APP.

[0057] In summary, the above technical solution is adopted to generate a mouse movement instruction when a trigger event is detected, display the mouse, and then determine whether the system API is successfully called. If so, display a full-screen floating frame, which is used to monitor the position of the mouse movement in real time. When the full-screen floating frame detects a mouse movement instruction, the mouse movement direction and movement distance are determined based on the mouse movement instruction, and the mouse is moved and clicked based on the mouse movement direction and movement distance; if not, send a screenshot instruction, obtain the image element corresponding to the screenshot content, and determine the position of the mouse movement and the click operation based on the image element. Drivers can read information (text / voice) on WeChat, reply to information (text / voice) on WeChat, open navigation (one-click to go home or one-click to return to the company), and open music (one-click to open the favorites music list) through simple button clicks. It has high accuracy and simple operation, reduces the safety hazards caused by distraction of drivers due to mobile phone operation, and has great economic and social value.

[0058] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A key display operation device, characterized in that: The key display operation device is applied to a mobile phone display screen, and the key display operation device comprises: A generation module, used to generate a mouse movement instruction when a trigger event is detected. The trigger event may be a joystick movement, a button press, or a voice command; The judgment module is used to judge whether the system API is successfully called; A control module is used to monitor the position of the mouse movement in real time according to the mouse movement instruction and in combination with the full-screen floating frame and control the mouse movement and click operation; The second control module is used to send a screenshot instruction, obtain the image element corresponding to the screenshot content, and determine the mouse movement position and click operation according to the image element.

2. The key display operation device according to claim 1, characterized in that: The generation module is used to store the correspondence between the mouse movement operation and the control instruction on the mobile phone display screen, each of the control instructions corresponds to a movement position and a click operation in one direction of the mobile phone display screen, or a movement instruction and a click operation at a specific position, and the final movement position and click operation of the mouse are determined based on the correspondence.

3. The key display operation method according to claim 1, characterized in that: The image elements include shape elements, color elements, character elements and preset template elements.

4. The key display operation device according to claim 1, characterized in that: The second control module includes: A full-screen content original image acquisition unit, used to acquire an original image of the full-screen content of the mobile phone screen when the screenshot instruction is detected; An image element recognition unit, used to recognize shape elements, color elements, character elements and preset template elements of image content; An execution unit is used to determine the position of the mouse movement and the click operation according to the image element.

5. A mobile phone control system, characterized in that: The mobile phone control system is applied to a mobile phone, and the mobile phone control system comprises the key display operation device according to any one of claims 1 to 4.

Images