A touchscreen mobile phone auxiliary device, method and equipment

Abstract

This invention provides a touchscreen mobile phone assistive device, method, and apparatus. The device includes an input module, a transmission module, and an accessibility service software module. The input module receives user input operations and transmits the user input information to the accessibility service software module via the transmission module. The accessibility service software module listens to user input information, displays indicator lines on the touchscreen mobile phone application interface showing the correspondence between the input module buttons and icons / text controls in the application, and issues corresponding application operation commands to the application based on user input information. This invention provides a solution to the common gesture operation difficulties faced by elderly people with severely impaired motor perception or pathological hand tremors when using touchscreen mobile phones, thereby improving the user experience, by replacing the touchscreen with an assistive device.

Description

Technical Field

[0001] This invention relates to the field of touchscreen mobile phone auxiliary technology, specifically to a touchscreen mobile phone auxiliary device, method, and equipment. Background Technology

[0002] Smartphones with touchscreens are now an integral part of modern digital and information-based society. Touchscreens allow users to perform various operations directly on the screen using gestures. Touchscreens and a few physical buttons are the mainstream form of smartphones. Except for a few functions such as power on / off, volume adjustment, and wake-up, which are operated via physical buttons, most applications are interacted with through touchscreen gestures. However, many elderly people with severely impaired motor perception or pathological tremors in their hands experience difficulties using touchscreen gestures and cannot use smartphones normally. Currently, most touchscreen phone assistive devices are used to enhance the user's gaming experience; for example, some games come with game controllers. However, game controllers are mostly designed for game software and are highly specialized, making them difficult for non-professional users to understand. Therefore, it is necessary to develop a touchscreen phone assistive device to help users with touchscreen gesture interaction difficulties use some commonly used apps on smartphones smoothly. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a touchscreen mobile phone auxiliary device, method and equipment to assist people with touchscreen gesture interaction difficulties in using smart touchscreen mobile phones normally and improve user experience.

[0004] Technical solution: To achieve the above-mentioned objectives, in a first aspect, the present invention provides a touchscreen mobile phone auxiliary device, including an input module, a transmission module, and an accessibility service software module:

[0005] The input module is used to receive user input operations and transmit the user input information to the accessibility service software module through the transmission module.

[0006] The accessibility service software module is used to listen to user input information, display indicator lines on the touch screen mobile application software interface to show the correspondence between the input module keys and the icons / text controls in the touch screen mobile application software, and issue corresponding application operation instructions to the touch screen mobile application software based on the user input information.

[0007] Preferably, the input module includes buttons, directional keys, and a scroll wheel; the buttons are distributed above and / or below the touchscreen phone screen to facilitate correspondence with the operation controls at the top and / or bottom of the phone screen.

[0008] Preferably, the buttons above and / or below the touchscreen phone screen are arranged in a single row, with a maximum of 5 buttons per row. When the number of text / graphic controls above / below the touchscreen phone application interface is the same as the number of buttons above / below the input module, the correspondence between text / graphic controls and buttons is assigned according to the principle of proximity. When there are more than 5 text / graphic controls with corresponding functions above / below the touchscreen phone application, one button may correspond to multiple text / graphic controls. The user can switch the selected object using the directional keys and press the confirmation button in the center of the directional keys to confirm the selection.

[0009] Preferably, the accessibility service software module also displays guidance information to instruct users on how to operate the software, as well as user feedback information, on the touchscreen mobile application interface.

[0010] Preferably, the accessibility service software module maps user operations on the input module to touch gestures on the touchscreen mobile application interface according to the accessibility service configuration file, and dispatches the gesture events to the touchscreen mobile application; the accessibility service configuration file contains the correspondence between the key values ​​of the input module buttons and the center coordinates of the clickable controls on the touchscreen mobile application interface.

[0011] Preferably, the accessibility service configuration file also includes indicator lines between the input module buttons and clickable controls in the touchscreen mobile application interface, as well as the location and interactive animation information of the operation feedback; the accessibility service software module displays the indicator lines and interactive animations of the operation feedback in the touchscreen mobile application interface when issuing corresponding application operation instructions to the touchscreen mobile application based on the user input information, according to the accessibility service configuration file.

[0012] Preferably, the indicator line includes a starting point, an ending point, and a line connecting them. Based on the correspondence between the input module buttons and controls, the starting point is the perpendicular point from the center of the input module button to the top / bottom edge of the screen, and the ending point is located at the center coordinate position of the control corresponding to the button. The indicator line is formed by connecting the starting point, intermediate point one, intermediate point two, and the ending point in that order. The x-coordinate of intermediate point one is the same as the x-coordinate of the starting point, the x-coordinate of intermediate point two is the same as the x-coordinate of the ending point, and the y-coordinate of the two intermediate points is the median of the y-coordinates of the starting point and the ending point.

[0013] As a preferred option, when the position of controls in the touchscreen mobile application interface is not fixed, a cursor is displayed on the touchscreen mobile application interface, and selection is made by using the directional keys in the input module. Based on the coordinates of the cursor, the function of the control at the corresponding position is activated.

[0014] Secondly, the present invention also provides a touchscreen mobile phone auxiliary method, comprising the following steps:

[0015] The input module is connected to the accessibility service software module via the transmission module, and the accessibility service software module starts listening.

[0016] The accessibility service software module listens to user input information, displays indicator lines on the touch screen mobile application interface showing the correspondence between the input module buttons and the icons / text controls in the touch screen mobile application, as well as guidance information to guide user operation, issues corresponding application operation instructions to the touch screen mobile application based on user input information, and displays operation feedback related to user input information on the touch screen mobile application interface.

[0017] Thirdly, the present invention also provides a mobile touchscreen device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements the functions of the accessibility service software module in the touchscreen mobile phone auxiliary device: including listening to user input information, displaying indicator lines on the touchscreen mobile phone application software interface indicating the correspondence between the input module keys and the icons / text controls in the touchscreen mobile phone application software, and issuing corresponding application operation instructions to the touchscreen mobile phone application software according to the user input information.

[0018] Beneficial Effects: The touchscreen mobile phone assistive device provided by this invention allows various types of input operations to be completed by replacing the touchscreen. A transmission module transmits user input information received by the input module to an accessibility service software module, which then sends corresponding application operation instructions, displays guidance information, indicator lines, and provides operation feedback to the touchscreen mobile phone application software. This provides a solution for the gesture operation difficulties commonly encountered by elderly people with severe motor perception decline or pathological hand tremors when using touchscreen mobile phones. Furthermore, the layout and functions of the input module in the touchscreen mobile phone assistive device provided by this invention are adaptable to software frequently used by the elderly, further enhancing the user experience. Furthermore, the touchscreen mobile phone assistive device provided by this invention allows for the selection and activation of non-fixed controls in the app using directional keys in the input module, which facilitates software development, and the algorithm-generated indicator lines improve versatility. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a schematic diagram of the structure of a touchscreen mobile phone auxiliary device provided in an embodiment of the present invention;

[0021] Figure 2 This is a diagram showing the correspondence between the input module buttons and the text / graphics in the touchscreen mobile phone application software in a touchscreen mobile phone auxiliary device provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the effect of a specific key in the input module of a touchscreen mobile phone auxiliary device provided in an embodiment of the present invention when it corresponds to text / graphics in multiple touchscreen mobile phone applications.

[0023] Figure 4 This is a schematic diagram showing the correspondence between the keypad of the accessibility service software module in a touchscreen mobile phone auxiliary device and the icons / text controls in the touchscreen mobile phone application software, as provided in an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of user operation feedback information displayed on the touchscreen mobile phone application software interface in an accessibility service software module of a touchscreen mobile phone auxiliary device provided in an embodiment of the present invention.

[0025] Figure 6 This is a state transition diagram of the "call" software page, as exemplified by an embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram of the basic page of the "call" software as an example of an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of the page corresponding to the "Add New Contact" control in the "Call" software, as exemplified by an embodiment of the present invention.

[0028] Figure 9 This is a schematic diagram of the "Dial Keyboard" page of the "Call" software as an example of an embodiment of the present invention;

[0029] Figure 10 This is a flowchart of a touchscreen mobile phone auxiliary method provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0031] To facilitate understanding of the embodiments of the present invention, a detailed description of a touchscreen mobile phone auxiliary device disclosed in the embodiments of the present invention will be provided first.

[0032] This invention provides an auxiliary device for touchscreen mobile phones, such as... Figure 1 As shown, it includes an input module, a transmission module, and an accessibility service software module, wherein:

[0033] The input module is used to receive various types of input operations from users.

[0034] The input module includes buttons, arrow keys, and a scroll wheel.

[0035] Here, the directional keys include left, right, up, and down buttons, and a confirmation button located in the center.

[0036] The above-mentioned input module has a specific layout according to the structure of the touch screen phone. It can be placed at the top, bottom and top sides of the touch screen phone screen. The top and bottom are buttons, which are arranged in a single row with a maximum of 5 buttons. The top and bottom sides, apart from the buttons, are arranged from left to right and top to bottom as follows: back button, start button, directional keys and scroll wheel.

[0037] Furthermore, the aforementioned input modules correspond to different interactive functions of touchscreen mobile applications based on specific layouts and different types of input operations. Specifically, these different interactive functions include confirming, pausing / playing, returning, scrolling, zooming, toggling, selecting, and dragging.

[0038] For example, when the application software's interaction function is to confirm or pause / play, it corresponds to the user pressing the buttons in the input modules located at the top and bottom of the touchscreen phone. For example, when the application software's interaction function is to return, it corresponds to the user pressing the return button in the input module located at the top left of the touchscreen phone.

[0039] For example, when the application software's interactive functions are scrolling and zooming, it corresponds to the user scrolling the scroll wheel in the input module at the bottom right of the touchscreen phone. For example, when the application software's interactive functions are switching, selecting, and dragging, it corresponds to the user pressing the directional keys in the input module at the bottom left of the touchscreen phone.

[0040] Furthermore, the number and arrangement of buttons in the input modules at the top and bottom of the touchscreen phone correspond to the text / graphics at the top and bottom of the touchscreen phone application software. Specifically, for example... Figure 2 As shown in (a), when the number of text / graphics at the bottom of the touch screen mobile application software interface is the same as the number of buttons at the bottom of the input module, the 5 buttons correspond one-to-one with the 5 closest text / graphics.

[0041] For example, such as Figure 2 As shown in (b), when the number of text / graphics below the touch screen mobile application software interface is less than the number of buttons below the input module, the three text / graphics correspond to the nearest buttons, namely the 1st, 3rd and 5th buttons, respectively.

[0042] Furthermore, the number of buttons in the input modules above and below the touchscreen phone is a maximum of 5. When there are more than 5 text / graphics with corresponding functions above or below in the touchscreen phone application software, a specific button in the input module will correspond to multiple text / graphics. When the specific button is pressed, the first text / graphic with corresponding function will be selected. The selected text / graphic will be displayed to the user through bolding and highlighting effects. The user can switch the selected object using the directional keys and press the confirmation button in the center of the directional keys to confirm the selection.

[0043] Specifically, for example, such as Figure 3 As shown, when there are 6 text / graphics with corresponding functions at the bottom of the touchscreen mobile application, the number of buttons at the bottom of the input module is 5. In this case, the first two buttons and the last two buttons at the bottom of the input module correspond to the two closest text / graphics and the two closest text / graphics, respectively. The third button corresponds to the third and fourth text / graphics of the application. After the user presses the third button, the text / graphic with the third function will be selected, and the third text / graphic will be bolded and highlighted. The user can then press the right arrow key to switch the selected object to the fourth text / graphic. At this time, the fourth text / graphic will be bolded and highlighted. Finally, the user can press the confirmation button in the center of the arrow key to confirm the selection.

[0044] The transmission module is used to identify the touchscreen phone and send the device model to the accessibility service software module, obtain the input information from the input module, and transmit the input information to the accessibility service software module.

[0045] Here, the transmission module transmits data with the accessibility service software module via serial communication.

[0046] The accessibility service software module is used to enable listening, render overlay updates, receive input information, and issue application operation commands.

[0047] Here, the rendering overlay is intended for rendering in the touchscreen mobile application interface, including displaying guidance information, indicator lines, and operation feedback.

[0048] Furthermore, the content of the overlay will be rendered according to the device model and the current state of the application software.

[0049] Furthermore, the displayed guidance information aims to convey to the user: "Which functions in touchscreen mobile applications can be achieved by operating the directional keys and scroll wheel of the input module?" Specifically, for example, regarding functions related to performing actions in touchscreen mobile applications, the guidance information informs the user that pressing the directional keys can select and execute the corresponding function; regarding video playback in touchscreen mobile applications, the guidance information informs the user that pressing the directional keys can control the video playback progress, playback order, and playback start / pause.

[0050] For example, the onboarding information tells users that scrolling the mouse wheel can adjust the volume of a touchscreen mobile application, and that scrolling the mouse wheel can adjust the scaling of objects in the touchscreen mobile application interface.

[0051] Furthermore, the indicator line includes the connection between the input module and the icons / text in the touchscreen mobile application software. The connection serves as a prompt to prevent accidental pressing of nearby buttons during selection, thus preventing misoperation. Specifically, for example, such as... Figure 4 As shown, when there are three corresponding text / graphics at the top of the touchscreen mobile application, the indicator lines of the 1st, 3rd, and 5th buttons above the input module point to the three corresponding text / graphics at the top of the application, respectively. When there are five corresponding text / graphics at the bottom of the touchscreen mobile application, the indicator lines of the 1st, 2nd, 3rd, 4th, and 5th buttons below the input module point to the five corresponding text / graphics at the bottom of the application, respectively.

[0052] Furthermore, the indicator line can display interactive animations, serving as feedback after a user presses a corresponding button during input, informing the user that a button connected to the indicator line has been pressed. Specifically, for example, after a user presses a button, the indicator line connected to that button may highlight, revert to its original color after magnification, or a light spot may slide along the indicator line.

[0053] Furthermore, the operation feedback refers to the prompts displayed on the touchscreen mobile application interface during the user's input operation, including the operation buttons, gestures, location of the operation, and the function performed by the operation. The operation buttons may also display color changes and scaling animations to provide feedback to the user. Specifically, for example... Figure 5 As shown, when the user inputs the video playback fast-forward event by pressing and holding the right arrow key, the gesture is pressing down with the index finger, the operation position is located at the right arrow key, the function of the operation is reflected in the text description "Fast forwarding", and the color of the right arrow key changes to dark, providing the user with the feedback that "the right arrow key has been pressed".

[0054] Furthermore, the gestures and positions of the operations in the operation feedback can also be presented with visual animation effects to make them easy for users to understand, thereby improving the user experience.

[0055] Here, the input information is sent to the current application on the mobile touchscreen device via the accessibility service software module to trigger corresponding events, achieving the same functionality as operating the application software using touchscreen gestures. Specifically, for example, when implementing the function of a touchscreen mobile application software to take a photo, pressing a button can trigger events such as taking a photo, switching modes, previewing photos, and switching photos; scrolling the scroll wheel can trigger events such as adjusting the focus and zooming the photo.

[0056] For example, when implementing the function of a touch screen mobile application, long-pressing the right directional key can trigger a fast-forward playback progress event, long-pressing the left directional key can trigger a rewind playback progress event, pressing the middle directional key can trigger a playback start event, and pressing the middle directional key again can trigger a playback pause event.

[0057] Furthermore, the accessibility service software module also includes settings and operation functions. Specifically, for example, it allows for adjusting the sensitivity of the scroll wheel and arrow keys in the input module to improve user efficiency and accuracy, thereby enhancing the user experience.

[0058] Furthermore, the accessibility service software module also maps the user's operation on the input module to touch gestures on the touch screen mobile application interface according to the accessibility service configuration file, and dispatches the gesture events to the touch screen mobile application to realize the activation of the corresponding function on the touch screen mobile application APP by operating the input module.

[0059] The accessibility service software module in this embodiment is designed for touchscreen phones running Android 7.0 and later. Users must first authorize the accessibility service function on their touchscreen phone before activating the software module. Specifically, for example, the user accesses the touchscreen phone's settings page, authorizes the accessibility service, and the accessibility service software module's internal system broadcast detects the change in user status, activates the accessibility service, and thus activates the software module.

[0060] Furthermore, the accessibility service software module can receive input information transmitted by the transmission module through the corresponding listening interface, and call and transmit the specified events according to the accessibility service configuration file.

[0061] Furthermore, the accessibility service software module uses a gesture dispatch method to simulate the button presses on the user operation input module as a gesture touch on a touchscreen phone interface, and dispatches the gesture dispatch event to the target APP through the accessibility service. This causes the target APP to cancel any currently ongoing gestures and only respond to the simulated gesture touches corresponding to the button presses on the user operation input module. As a result, the user can achieve the same function as performing a gesture touch on a touchscreen phone interface by operating the input module.

[0062] Here, the gesture dispatch method dispatches an event to the touchscreen mobile application software control that responds to a gesture touch event, causing the touchscreen mobile application software control to respond via a button operation, thereby simulating a gesture touch operation through button operation. Specifically, for example, the accessibility service software module dispatches a control response event of pressing an input module button to the target app to replace the control response event of touching the corresponding control, allowing the user to trigger the function of the corresponding control on the target app by pressing the button on the input module. In this example, for the Android system, the Java `GestureDescription` class utility can be used to implement simulated gesture operations. For example, simulating a gesture touch click at the coordinates (x1, y1) relative to the top left corner of the touchscreen mobile phone screen, a partial code example is shown below.

[0063] public void autoClick(int startTimeMs,int durationMs,int x,int y){

[0064] boolean isCalled=dispatchGesture(gestureDescription(startTimeMs,durationMs,x,y),null,null);

[0065] System.out.println(isCalled);

[0066] }

[0067] public GestureDescription gestureDescription(int startTimeMs,intdurationMs,int x,int y){

[0068] Path path=new Path();

[0069] path.moveTo(x,y);

[0070] return createGestureDescription(newGestureDescription.StrokeDescription(path,startTimeMs,durationMs));

[0071] }

[0072] public GestureDescription createGestureDescription(GestureDescription.StrokeDescription...strokes){

[0073] GestureDescription.Builder builder=newGestureDescription.Builder();

[0074] for(GestureDescription.StrokeDescription stroke:strokes){

[0075] builder.addStroke(stroke);

[0076] }

[0077] return builder.build();

[0078] }

[0079] autoClick(t1,d1,x1,y1);

[0080] Furthermore, to simulate a gesture touch click on a control in the target APP using the gesture dispatch method, it is necessary to know the center coordinates of the clickable control in the target APP. The center coordinates are the coordinates of the geometric center of the control relative to the upper left corner of the touch screen phone screen. The center coordinates are stored in the accessibility service configuration file. By calling the accessibility service configuration file, the center coordinates of the control can be read, so that the target APP can trigger the corresponding gesture touch click function at the corresponding control center coordinate position.

[0081] Furthermore, the accessibility service configuration file also includes the key values ​​of the input module buttons, as well as indicator lines and operation feedback in the rendering overlay. Here, there is a one-to-one correspondence between the key values ​​of the buttons in the accessibility service configuration file, the center coordinates of the clickable controls on the target APP page, the indicator lines, and the operation feedback. This ensures that the buttons on the input module correspond one-to-one with the clickable controls on the target APP page, so that when the input module buttons are operated to activate the control function, the indicator lines between the target control and the buttons, and the operation feedback corresponding to the input operation, can present corresponding visual animation effects.

[0082] Furthermore, the accessibility service configuration file needs to be created and allocated according to the mobile phone model and system software version during the accessibility service software module initialization phase. The contents of the accessibility service configuration file are shown in the table below. It contains data for interacting with the user interface, namely the key values ​​of the input module buttons, the touch screen mobile phone page ID, the center coordinates of the clickable controls on the page, and the position and interactive animation information of the indicator lines and operation feedback.

[0083] Table 1 Contents of the Accessibility Service Configuration File

[0084]

[0085] Furthermore, the position of the indicator line in the accessibility service configuration file is generated according to an algorithm. The indicator line consists of a starting point, an ending point, and a line connecting two intermediate points. Based on the correspondence between the input module buttons and controls, the starting point is the vertical point from the center of the input module button to the top / bottom edge of the screen. The ending point is located at the center coordinate position of the control corresponding to the button. The x-coordinate of intermediate point one is the same as the x-coordinate of the starting point, and the x-coordinate of intermediate point two is the same as the x-coordinate of the ending point. The y-coordinate of the two intermediate points is the median of the y-coordinates of the starting point and the ending point. These four points are connected in the order of starting point, intermediate point one, intermediate point two, and ending point to form the indicator line.

[0086] Furthermore, when the position of controls in the target app is not fixed, since the accessibility service configuration file cannot store the coordinates of the controls in advance, it is impossible to match the pressed button with the desired control one-to-one using coordinates and key values. In this case, the user can select the control by using the arrow keys in the input module and activate its function through the accessibility service software module. Specifically, for example, when the user uses the arrow keys in the input module, the accessibility service software module will display a cursor at the top of the target app interface by rendering an overlay. The user can change the cursor's position using the arrow keys, move the cursor over the desired control, and then press the "OK" button. Based on the cursor's coordinates, the accessibility service software module will activate the corresponding control's function in the target app.

[0087] This section uses the "call" software on a touchscreen phone as an example to introduce the specific implementation steps, and then explains how the accessibility service software module allows users to activate the touchscreen phone's call function through the operation input module.

[0088] like Figure 6 As shown, the "Call" software page has multiple states. In order to realize the function of making a call, the accessibility service software module needs to control the touch screen phone to jump between various page states and activate the corresponding functions of the controls on the page.

[0089] For example, such as Figure 7 As shown, the "Call" software consists of four basic pages, each containing clickable controls. The center coordinates of each control are stored in the accessibility service configuration file during the initialization phase of the accessibility service software module, and correspond one-to-one with the key values ​​of the input module buttons in the accessibility service configuration file. Indicator lines exist between controls and buttons to show their correspondence. When a user wants to perform the function corresponding to a control, such as page navigation, they can press the button corresponding to the control at the bottom of the "Call" software page. When the user presses the corresponding button below the input module, the transmission module transmits the input information to the accessibility service software module. After listening to the input information, the accessibility service software module finds the center coordinates of the control corresponding to the key value of the pressed button according to the accessibility service configuration file, calls and transmits the specified event, and simulates a gesture touch operation at the center coordinate position of the control through the gesture dispatch method, activating the page navigation function corresponding to the control.

[0090] When a user wants to add a new contact, they can follow the indicator line and press the third button above the input module corresponding to the "Add New Contact" control on the "Contacts" page to enter the "Add New Contact" page state. Figure 8As shown, users can then proceed with subsequent operations. If a user wants to return to a previous page, they can press the back button on the upper left of the input module to activate its corresponding "back" control, thus returning to the previous page. If a user wants to save information about a new contact, they can press the fifth button at the top of the touchscreen to save that information.

[0091] When a user wants to make a call, they can follow the instructions on the "Dial Keyboard" page, such as... Figure 9 As shown, you can enter a number by pressing the arrow keys on the input module corresponding to the "Number" control, and dial a number by pressing the third button below the input module after you finish entering the number.

[0092] Based on the same technical concept, this invention also provides a touchscreen mobile phone auxiliary method, which can be found in the following embodiments.

[0093] like Figure 10 The diagram shown is a flowchart of a touchscreen mobile phone assistance method provided by an embodiment of the present invention, which specifically includes the following steps.

[0094] The transmission module connects to the touchscreen phone via a data cable interface. It transmits data to the accessibility service software module through serial communication.

[0095] The accessibility service software module is now monitoring.

[0096] After the transmission module identifies the touchscreen phone, it sends the device model to the accessibility service software module.

[0097] The accessibility service software module displays guidance information and indicator lines in the application software interface of touch screen mobile phones by rendering an overlay, based on the device model and the current status of the application.

[0098] Users use the input module to perform various types of input operations. The input module obtains various types of input information from users and transmits the input information to the accessibility service software module through the transmission module.

[0099] Here, users can refer to the guidance information and indicator lines displayed in the touch screen mobile application interface to avoid accidental operation. The indicator lines can present interactive animations based on the user's input operation, serving as feedback after the user presses the corresponding key when performing an input operation.

[0100] After receiving input information, the accessibility service software module sends a command to the current application on the touchscreen phone to trigger the corresponding event.

[0101] The current application on the touchscreen phone executes the corresponding operation based on the command.

[0102] Here, the accessibility service software module can display operation feedback related to input operations and current application functions on the touch screen mobile application interface through a rendering overlay, and confirm the current operation to the user through visual feedback.

[0103] Based on the same technical concept, this invention also provides a mobile touchscreen device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements the functions of the accessibility service software module in the touchscreen mobile phone auxiliary device, including listening to user input information, displaying indicator lines on the touchscreen mobile phone application software interface to indicate the correspondence between the input module keys and the icons / text controls in the touchscreen mobile phone application software, and issuing corresponding application operation instructions to the touchscreen mobile phone application software based on user input information.

Claims

1. A touchscreen mobile phone auxiliary device, characterized in that, It includes an input module, a transmission module, and an accessibility service software module; the input module is used to receive user input operations and transmit the user input information to the accessibility service software module through the transmission module; The accessibility service software module is used to listen to user input information, display indicator lines on the touchscreen mobile application interface to show the correspondence between the input module buttons and the icons / text controls in the touchscreen mobile application, and issue corresponding application operation commands to the touchscreen mobile application based on the user input information; the input module includes buttons; the buttons are distributed at the top and / or bottom of the touchscreen mobile screen to facilitate correspondence with the operation controls at the top and / or bottom of the mobile screen; the indicator lines include a start point, an end point, and a line connecting them, and according to the correspondence between the input module buttons and controls, the start point is the perpendicular point from the center of the input module button to the top / bottom boundary of the screen, and the end point is located at the center coordinate position of the control corresponding to the button.

2. The touchscreen mobile phone auxiliary device according to claim 1, characterized in that, The input module includes arrow keys and a scroll wheel.

3. The touchscreen mobile phone auxiliary device according to claim 2, characterized in that, The buttons above and / or below the touchscreen phone screen are arranged in a single row, with a maximum of 5 buttons per row. When the number of text / graphic controls above / below the touchscreen phone application interface is the same as the number of buttons above / below the input module, the correspondence between text / graphic controls and buttons is assigned according to the principle of closest proximity. When there are more than 5 text / graphics with corresponding functions at the top / bottom of a touchscreen mobile application, there is a button that corresponds to multiple text / graphic controls. The user can switch the selected object using the arrow keys and press the confirmation button in the center of the arrow keys to confirm the selection.

4. The touchscreen mobile phone auxiliary device according to claim 1, characterized in that, The accessibility service software module also displays guidance information to instruct users on how to operate the software, as well as user feedback information, on the touchscreen mobile application interface.

5. The touchscreen mobile phone auxiliary device according to claim 1, characterized in that, The accessibility service software module maps user operations on the input module to touch gestures on the touchscreen mobile application interface according to the accessibility service configuration file, and dispatches the gesture events to the touchscreen mobile application. The accessibility service configuration file contains the correspondence between the key values ​​of the input module buttons and the center coordinates of the clickable controls on the touchscreen mobile application interface.

6. The touchscreen mobile phone auxiliary device according to claim 5, characterized in that, The accessibility service configuration file also includes indicator lines between the input module buttons and clickable controls in the touchscreen mobile application interface, as well as the location and interactive animation information of the operation feedback. According to the accessibility service configuration file, the accessibility service software module displays indicator lines and interactive animations of the operation feedback in the touchscreen mobile application interface when issuing corresponding application operation instructions to the touchscreen mobile application based on user input information.

7. The touchscreen mobile phone auxiliary device according to claim 1, characterized in that, The indicator line is formed by connecting the starting point, intermediate point one, intermediate point two, and the ending point in that order. The x-coordinate of intermediate point one is the same as that of the starting point, the x-coordinate of intermediate point two is the same as that of the ending point, and the y-coordinate of the two intermediate points is the median of the y-coordinates of the starting point and the ending point.

8. The touchscreen mobile phone auxiliary device according to claim 1, characterized in that, When the position of controls in a touchscreen mobile application interface is not fixed, a cursor is displayed on the touchscreen mobile application interface. Users can select controls by using the arrow keys in the input module, and activate the function of the corresponding control based on the coordinates of the cursor.

9. A touchscreen mobile phone assistance method implemented using the touchscreen mobile phone assistance device according to any one of claims 1-8, characterized in that, Includes the following steps: The input module is connected to the accessibility service software module via the transmission module, and the accessibility service software module starts listening. The accessibility service software module listens to user input information, displays indicator lines on the touch screen mobile application interface showing the correspondence between the input module buttons and the icons / text controls in the touch screen mobile application, as well as guidance information to guide user operation, issues corresponding application operation instructions to the touch screen mobile application based on user input information, and displays operation feedback related to user input information on the touch screen mobile application interface.

10. A mobile touchscreen device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the functions of the accessibility service software module in the touchscreen mobile phone auxiliary device according to any one of claims 1-8: including listening to user input information, displaying indicator lines on the touchscreen mobile phone application software interface to indicate the correspondence between the input module keys and the icons / text controls in the touchscreen mobile phone application software, and issuing corresponding application operation instructions to the touchscreen mobile phone application software according to the user input information.

Images