A method and device for operating a mobile terminal user interface and electronic device thereof

Abstract

The present application discloses a method, device, and electronic device for operating a mobile terminal user interface. The method comprises: obtaining an operation instruction; according to the operation instruction, controlling the operation terminal to operate the mobile terminal user interface so that the operation terminal performs an operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to operate the mobile terminal user interface is learned from a preset video through a preset artificial intelligence model. The present invention allows the operation terminal to automatically operate the mobile terminal according to the operation instruction by issuing an operation instruction, so as to facilitate handicapped users to operate the mobile terminal.

Description

Technical Field

[0001] The present invention relates to the field of automatic control, and in particular to a method, a device, and an electronic device for operating a mobile terminal user interface. Background Art

[0002] With the rapid development of mobile Internet, the UI interaction interface of Internet products is becoming more and more complex and gorgeous, and more and more entrances and functions are piled up on the user interface of mobile terminals.

[0003] Users with cognitive impairments, motor disabilities, and other disabilities may have difficulty understanding the meaning of the complex and numerous UI elements on the mobile terminal user interface due to their own reasons, which makes them unable to operate the mobile terminal. Or due to various obstacles, they can only perform simple operations such as clicking and dragging and cannot operate the mobile terminal smoothly. Summary of the invention

[0004] The present application provides a method, device and electronic device for operating a mobile terminal user interface, which allows an operating terminal to automatically operate a mobile terminal according to the operating instructions by issuing operating instructions, thereby facilitating handicapped users to operate the mobile terminal.

[0005] The embodiment of the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method for operating a mobile terminal user interface, the method comprising:

[0007] Get operation instructions;

[0008] According to the operation instruction, the operation terminal is controlled to operate the mobile terminal user interface so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through a preset artificial intelligence model.

[0009] In some embodiments, learning from a preset video using a preset artificial intelligence model includes:

[0010] Obtaining the preset video;

[0011] Analyze the preset video and identify the operation object and operation subject in the preset video;

[0012] The key operation path for the operation subject to operate the operation object is extracted, so that the operation terminal can correctly reproduce the corresponding operation through the key operation path.

[0013] In some embodiments, the method further comprises:

[0014] Critical information corresponding to the critical operation path is set, wherein the critical information corresponds to at least one of the critical operation paths.

[0015] In some embodiments, the method further comprises:

[0016] Identify the operation instruction by using the preset artificial intelligence model, and find the key information corresponding to the operation instruction;

[0017] According to the key operation path corresponding to the key information, the operation terminal is controlled to operate the mobile terminal user interface.

[0018] In some embodiments, the method further comprises:

[0019] Acquire an image of the mobile terminal user interface;

[0020] According to the image, the operation process of the operation terminal on the mobile terminal user interface is corrected in real time.

[0021] In some embodiments, the step of correcting the operation process in real time according to the image includes:

[0022] According to the image, a preset artificial intelligence model determines whether the user interface is a user interface corresponding to the operation process; if so,

[0023] Then, continue to execute the operation process; otherwise,

[0024] Control the mobile terminal to switch to a user interface corresponding to the operation process.

[0025] In some embodiments, the determining by a preset artificial intelligence model whether the user interface is a user interface corresponding to the operation process includes:

[0026] Training the preset artificial intelligence model;

[0027] Obtaining samples of several mobile terminal user interfaces;

[0028] Label each sample accordingly;

[0029] According to the machine learning algorithm, the preset artificial intelligence model is trained using samples of the mobile terminal user interface and labels corresponding to each of the samples to obtain the trained preset artificial intelligence model.

[0030] In some embodiments, the method further comprises:

[0031] The operation instruction is a voice instruction;

[0032] The operation terminal is a mechanical arm with a stylus.

[0033] In a second aspect, an embodiment of the present invention further provides a device for operating a mobile terminal user interface, the device comprising:

[0034] An acquisition module, used for acquiring operation instructions;

[0035] An operation module is used to control the operation terminal to operate the mobile terminal user interface according to the operation instruction, so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through an artificial intelligence model.

[0036] In a third aspect, an embodiment of the present invention further provides a computer device, including:

[0037] at least one processor; and,

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0040] In a fourth aspect, an embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a touch-controlled device to execute the method for operating a mobile terminal user interface as described above.

[0041] The beneficial effect of the embodiments of the present invention is that, different from the prior art, the embodiments of the present invention provide a method, device, and electronic device for operating a mobile terminal user interface. The method includes: obtaining an operation instruction; according to the operation instruction, controlling the operation terminal to operate the mobile terminal user interface so that the operation terminal performs an operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through a preset artificial intelligence model. The embodiments of the present invention issue simple operation instructions to allow the operation terminal to automatically operate the mobile terminal according to the operation instructions, making it convenient for users with disabilities to operate the mobile terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0043] Figure 1 It is a flowchart of a method for operating a mobile terminal user interface provided by an embodiment of the present invention;

[0044] Figure 2 yes Figure 1 A detailed flowchart of step S20 in FIG.

[0045] Figure 3 is a flowchart of a correction operation process provided by an embodiment of the present invention;

[0046] Figure 4 It is a structural schematic diagram of a device for operating a mobile terminal user interface provided by an embodiment of the present invention;

[0047] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for operating a mobile terminal user interface provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0050] Step S10: obtaining operation instructions;

[0051] Specifically, the operation instruction includes a voice instruction, a gesture instruction, a key instruction, and other recognizable instructions. For example, the operation instruction is a voice instruction.

[0052] Step S20: According to the operation instruction, control the operation terminal to operate the mobile terminal user interface so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through a preset artificial intelligence model.

[0053] Specifically, the operating terminal includes a mechanical arm with a stylus, a mouse, a keyboard and other devices or equipment that can operate the mobile terminal user interface. The mobile terminal includes computer devices that can be used on the move, such as mobile phones, tablets, PDAs and laptops.

[0054] In an embodiment of the present invention, the operation instruction is a voice instruction, and the voice instruction is to open Alipay. The operation terminal is a robotic arm with a stylus. According to the operation instruction, the operation terminal is controlled to operate the mobile terminal user interface so that the operation terminal performs the operation corresponding to the operation instruction, that is, the robotic arm with the stylus is controlled to operate the mobile terminal to open Alipay on the mobile terminal.

[0055] For details, please refer to Figure 2 , Figure 2 yes Figure 1 A detailed flowchart of step S20 in FIG.

[0056] like Figure 2 Said step S20, according to the operation instruction, controls the operation terminal to operate the mobile terminal user interface so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path of the operation terminal performing the operation on the mobile terminal user interface is learned from the preset video through the preset artificial intelligence model, including:

[0057] Step S21: Obtain the preset video;

[0058] Specifically, the format of the video can be arbitrary, including wmv, ast, avi, 3gp, etc., and the preset video includes a clear video of operating the user interface of the mobile terminal that can be used for learning by the artificial intelligence model.

[0059] Step S22: analyzing the preset video to identify the operation object and the operation subject in the preset video;

[0060] Specifically, the operation object is any mobile terminal, and the operation subject can be a stylus, a human finger, a mouse, a keyboard, or other device or body part that can operate the user interface of the mobile terminal. For example, the preset video content is a person operating a mobile phone to open Alipay, then the operation object is a mobile phone, and the operation subject is a person or a person's finger.

[0061] In an embodiment of the present invention, the video data is analyzed using a preset artificial intelligence model, and before the operation objects and operation subjects in the preset video are identified, the artificial intelligence model must be trained first. Specifically, the first sample data and the second sample data of a number of operation objects and operation subjects are collected; corresponding labels are added to each of the first sample data and the second sample data; according to the deep learning algorithm, the first sample data and the second sample data of the several operation objects and operation subjects and the corresponding labels of each of the first sample data and the second sample data are used to train the model to obtain the preset artificial intelligence model.

[0062] Specifically, a deep learning algorithm is a method of using data to train a model and then using the model for prediction. The first sample data of several operation objects are collected in advance, such as mobile phones of different models and brands, laptops of different models and brands, tablets of different models and brands, etc., and the second sample data of several operation subjects are collected, such as hands of people of different skin colors and genders, styluses of different sizes and appearances, etc. Or the first sample data and the second sample data of several operation objects and operation subjects are obtained from a database. Then, a corresponding label is added to each of the first sample data and the second sample data. When the first sample data and the second sample data of all the second operation objects and operation subjects are added with corresponding labels, the model is trained using the first sample data and the second sample data of several operation objects and operation subjects and the corresponding labels of each of the first sample data and the second sample data, thereby obtaining the artificial intelligence model. The more the first sample data and the second sample data of the operation objects and operation subjects, the more situations are covered, and thus the recognition ability of the artificial intelligence model is higher.

[0063] In the embodiment of the present invention, in order to ensure that the recognition rate of the artificial intelligence model is high enough, a preset threshold can be set. If the test recognition rate reaches the preset threshold, it means that the recognition ability of the trained artificial intelligence model meets expectations, and the trained artificial intelligence model can be directly used as a trained artificial intelligence model to recognize the video. Otherwise, the various parameters of the artificial intelligence model are continuously adjusted, and the artificial intelligence model is trained again until the recognition rate of the artificial intelligence model reaches the preset threshold.

[0064] In the embodiment of the present invention, it is also possible to analyze the image of the user interface of the operation object and identify the program corresponding to the operation process of the operation subject on the operation object.

[0065] Step S23: extracting the key operation path for the operation subject to operate the operation object, so that the operation terminal correctly reproduces the corresponding operation through the key operation path;

[0066] Specifically, the target video to be imitated is extracted frame by frame to obtain the image of each frame, the artificial intelligence model is used to identify the type of object in the image of each frame, the operation object and the operation subject in the video are identified, and the key task path for the operation subject to operate the operation object is extracted.

[0067] Furthermore, the key information corresponding to the key task path is set, and the key information may correspond to multiple key task paths. For example, the operation corresponding to the first key task path is to open WeChat, and the operation corresponding to the second key task path is to open the health code in the WeChat interface. The key information corresponding to the first key task path and the second key task path are both set to open the WeChat health code. Further, the priorities of multiple key task paths are set, for example, the first key task path is set to the highest priority, and the second key task path is set to the second priority. When the operation instruction corresponding to the key information is identified, the operation terminal executes the operations corresponding to the first key task path and the second key task path in sequence, that is, executes the operations of opening WeChat and opening the health code in the WeChat interface in sequence.

[0068] See also Figure 3 , Figure 3 : is a flowchart of a correction operation process provided by an embodiment of the present invention; wherein the method comprises:

[0069] Step S30: Acquire an image of the mobile terminal user interface;

[0070] In an embodiment of the present invention, an image of the mobile terminal user interface can be acquired through a camera, and the image can be uploaded to the artificial intelligence model.

[0071] Step S40: Correcting the operation process of the operating terminal on the mobile terminal user interface in real time according to the image.

[0072] Specifically, the process of correcting the operation of the operating terminal on the mobile terminal user interface in real time according to the image includes:

[0073] According to the image, a preset artificial intelligence model determines whether the user interface is a user interface corresponding to the operation process; if so,

[0074] Then, continue to execute the operation process; otherwise,

[0075] Control the mobile terminal to switch to a user interface corresponding to the operation process.

[0076] Specifically, the determining, by a preset artificial intelligence model, whether the user interface is a user interface corresponding to the operation process includes:

[0077] Training the preset artificial intelligence model;

[0078] Obtaining samples of several mobile terminal user interfaces;

[0079] Label each sample accordingly;

[0080] In an embodiment of the present invention, it is determined based on the image that Alipay is running on the user interface of the mobile terminal. Further, it is determined whether the user interface is the user interface corresponding to the operation process. For example, if the operation process is to open the WeChat health code and the corresponding user interface is WeChat, the operation terminal is controlled to close Alipay and open WeChat. If the user interface corresponding to the operation process is WeChat, the operation is performed according to the key operation path.

[0081] Specifically, before correcting the operation process of the operating terminal on the mobile terminal user interface in real time based on the image, it is necessary to train the preset artificial intelligence model according to a machine learning algorithm using samples of the mobile terminal user interface and labels corresponding to each of the samples to obtain the trained preset artificial intelligence model.

[0082] In the embodiment of the present invention, before using the preset artificial intelligence model to analyze the video data, the artificial intelligence model must be trained first. Specifically, samples of the mobile terminal user interface are collected; corresponding labels are added to each sample of the mobile terminal user interface; and according to the deep learning algorithm, the model is trained using the samples of the mobile terminal user interface and the corresponding labels of each sample of the mobile terminal user interface to obtain the preset artificial intelligence model.

[0083] Specifically, a deep learning algorithm is a method that uses data to train a model and then uses the model for prediction. Collect several samples of the mobile terminal user interface in advance, such as the WeChat friend interface, WeChat chat interface, Alipay payment interface, etc. Or obtain several samples of the mobile terminal user interface from the database, and then label each sample of the mobile terminal user interface with a corresponding label. When all samples of the mobile terminal user interface are labeled with a corresponding label, use several samples of the mobile terminal user interface and the corresponding labels of each sample of the mobile terminal user interface to train the model, thereby obtaining the artificial intelligence model. The more sample data of the mobile terminal user interface, the more situations are covered, and thus the higher the recognition ability of the artificial intelligence model.

[0084] In the embodiment of the present invention, in order to ensure that the recognition rate of the artificial intelligence model is high enough, a preset threshold can be set. If the test recognition rate reaches the preset threshold, it means that the recognition ability of the trained artificial intelligence model meets expectations, and the trained artificial intelligence model can be directly used as a trained artificial intelligence model to recognize the mobile terminal user interface. Otherwise, the various parameters of the artificial intelligence model are continuously adjusted, and the artificial intelligence model is trained again until the recognition rate of the artificial intelligence model reaches the preset threshold.

[0085] Furthermore, the method also includes automatically identifying unfinished critical task paths and performing operations according to the unfinished critical task paths. Specifically, automatically identifying unfinished critical task paths and performing operations according to the unfinished critical task paths includes: starting from the lowest priority critical task path, identifying whether the current user interface is the user interface corresponding to the critical task path.

[0086] In one embodiment of the present invention, there are two critical task paths with different priorities. The first critical task path with the highest priority corresponds to an operation of opening WeChat, and the second critical task path with the second priority corresponds to an operation of opening a health code in the WeChat interface. The key information corresponding to the voice instruction is opening the WeChat health code, and the current user interface is Alipay. The artificial intelligence model determines that the user interface is different from the user interfaces corresponding to the second critical task path and the first critical task path. The operation performed by the operating terminal is, according to the voice instruction, to execute the operations of closing Alipay, opening WeChat and opening the health code in the WeChat interface in sequence.

[0087] In another embodiment of the embodiment of the present invention, there are two critical task paths with different priorities, the first critical task path with the highest priority corresponds to an operation of opening WeChat, and the second critical task path with the second priority corresponds to an operation of opening a health code in the WeChat interface. The key information corresponding to the voice instruction is opening the WeChat health code, and the current user interface is WeChat. The artificial intelligence model determines that the current user interface is the same as the user interface corresponding to the second critical task path, and the operation performed by the operating terminal is to skip the first critical path and directly execute the second critical path, that is, directly execute the operation of opening the health code in the WeChat interface.

[0088] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a device for operating a mobile terminal user interface:

[0089] like Figure 4 As shown, the device comprises:

[0090] An acquisition module, used for acquiring operation instructions;

[0091] An operation module is used to control the operation terminal to operate the mobile terminal user interface according to the operation instruction, so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through an artificial intelligence model.

[0092] In the embodiment of the present invention, the acquisition module includes:

[0093] A voice command acquisition module is used to acquire voice operation commands;

[0094] An image acquisition module, used to acquire a user interface image of the mobile terminal;

[0095] The video acquisition module is used to acquire the preset video.

[0096] In the embodiment of the present invention, the operation module includes:

[0097] An operation terminal module, used to perform an operation corresponding to the operation instruction according to the key operation path;

[0098] Artificial intelligence module, used to simulate human behavior;

[0099] In an embodiment of the present invention, the artificial intelligence module includes:

[0100] Speech recognition module, converting voice commands into text;

[0101] A video analysis module, for analyzing the preset video and identifying an operation object and an operation subject in the preset video;

[0102] A video learning module, which analyzes the preset video and identifies the operation object and operation subject in the preset video;

[0103] A user interface recognition module, which analyzes the image of the user interface of the mobile terminal and determines whether the user interface is the user interface corresponding to the operation process;

[0104] In the embodiment of the present invention, the video analysis module is specifically used for:

[0105] The video data is analyzed using a preset artificial intelligence model, and before the operation objects and operation subjects in the preset video are identified, the artificial intelligence model must be trained first. Specifically, the first sample data and the second sample data of a number of operation objects and operation subjects are collected; corresponding labels are added to each of the first sample data and the second sample data; according to the deep learning algorithm, the first sample data and the second sample data of the several operation objects and operation subjects and the corresponding labels of each of the first sample data and the second sample data are used to train the model to obtain the preset artificial intelligence model.

[0106] Specifically, a deep learning algorithm is a method of using data to train a model and then using the model for prediction. The first sample data of several operation objects are collected in advance, such as mobile phones of different models and brands, laptops of different models and brands, tablets of different models and brands, etc., and the second sample data of several operation subjects are collected, such as hands of people of different skin colors and genders, styluses of different sizes and appearances, etc. Or the first sample data and the second sample data of several operation objects and operation subjects are obtained from a database. Then, a corresponding label is added to each of the first sample data and the second sample data. When the first sample data and the second sample data of all the second operation objects and operation subjects are added with corresponding labels, the model is trained using the first sample data and the second sample data of several operation objects and operation subjects and the corresponding labels of each of the first sample data and the second sample data, thereby obtaining the artificial intelligence model. The more the first sample data and the second sample data of the operation objects and operation subjects, the more situations are covered, and thus the recognition ability of the artificial intelligence model is higher.

[0107] In the embodiment of the present invention, in order to ensure that the recognition rate of the artificial intelligence model is high enough, a preset threshold can be set. If the test recognition rate reaches the preset threshold, it means that the recognition ability of the trained artificial intelligence model meets expectations, and the trained artificial intelligence model can be directly used as a trained artificial intelligence model to recognize the video. Otherwise, the various parameters of the artificial intelligence model are continuously adjusted, and the artificial intelligence model is trained again until the recognition rate of the artificial intelligence model reaches the preset threshold.

[0108] In the embodiment of the present invention, the video learning module is specifically used for:

[0109] The target video to be imitated is extracted frame by frame to obtain the image of each frame, the artificial intelligence model is used to identify the type of object in the image of each frame, the operation object and the operation subject in the image are identified, and the key task path for the operation subject to operate the operation object is extracted.

[0110] In the embodiment of the present invention, the user interface identification module is specifically used to:

[0111] According to the machine learning algorithm, the preset artificial intelligence model is trained using samples of the mobile terminal user interface and labels corresponding to each of the samples to obtain the trained preset artificial intelligence model.

[0112] Specifically, in an embodiment of the present invention, the preset artificial intelligence model determines the program or interface running on the mobile terminal based on the image. For example, it determines based on the image that WeChat is running on the mobile terminal. Further, it determines whether the user interface is the user interface corresponding to the operation process. For example, if the user interface corresponding to the operation process is Alipay, the operation terminal is controlled to close Alipay and open WeChat. If the user interface corresponding to the operation process is WeChat, the operation is performed according to the key operation path.

[0113] In the embodiment of the present invention, before using the preset artificial intelligence model to analyze the video data, the artificial intelligence model must be trained first. Specifically, samples of the mobile terminal user interface are collected; corresponding labels are added to each sample of the mobile terminal user interface; and according to the deep learning algorithm, the model is trained using the samples of the mobile terminal user interface and the corresponding labels of each sample of the mobile terminal user interface to obtain the preset artificial intelligence model.

[0114] Specifically, a deep learning algorithm is a method that uses data to train a model and then uses the model for prediction. Collect several samples of the mobile terminal user interface in advance, such as the WeChat friend interface, WeChat chat interface, Alipay payment interface, etc. Or obtain several samples of the mobile terminal user interface from the database, and then label each sample of the mobile terminal user interface with a corresponding label. When all samples of the mobile terminal user interface are labeled with a corresponding label, use several samples of the mobile terminal user interface and the corresponding labels of each sample of the mobile terminal user interface to train the model, thereby obtaining the artificial intelligence model. The more sample data of the mobile terminal user interface, the more situations are covered, and thus the higher the recognition ability of the artificial intelligence model.

[0115] In the embodiment of the present invention, in order to ensure that the recognition rate of the artificial intelligence model is high enough, a preset threshold can be set. If the test recognition rate reaches the preset threshold, it means that the recognition ability of the trained artificial intelligence model meets expectations, and the trained artificial intelligence model can be directly used as a trained artificial intelligence model to recognize the mobile terminal user interface. Otherwise, the various parameters of the artificial intelligence model are continuously adjusted, and the artificial intelligence model is trained again until the recognition rate of the artificial intelligence model reaches the preset threshold.

[0116] Furthermore, the method also includes automatically identifying unfinished critical task paths and performing operations according to the unfinished critical task paths. Specifically, automatically identifying unfinished critical task paths and performing operations according to the unfinished critical task paths includes: starting from the lowest priority critical task path, identifying whether the current user interface is the user interface corresponding to the critical task path.

[0117] In one embodiment of the present invention, there are two critical task paths with different priorities. The first critical task path with the highest priority corresponds to an operation of opening WeChat, and the second critical task path with the second priority corresponds to an operation of opening a health code in the WeChat interface. The key information corresponding to the voice instruction is opening the WeChat health code, and the current user interface is Alipay. The artificial intelligence model determines that the user interface is different from the user interfaces corresponding to the second critical task path and the first critical task path. The operation performed by the operating terminal is, according to the voice instruction, to execute the operations of closing Alipay, opening WeChat and opening the health code in the WeChat interface in sequence.

[0118] In another embodiment of the embodiment of the present invention, there are two critical task paths with different priorities, the first critical task path with the highest priority corresponds to an operation of opening WeChat, and the second critical task path with the second priority corresponds to an operation of opening a health code in the WeChat interface. The key information corresponding to the voice instruction is opening the WeChat health code, and the current user interface is WeChat. The artificial intelligence model determines that the current user interface is the same as the user interface corresponding to the second critical task path, and the operation performed by the operating terminal is to skip the first critical path and directly execute the second critical path, that is, directly execute the operation of opening the health code in the WeChat interface.

[0119] Please refer to Figure 5 , Figure 5 is a schematic diagram of an electronic device of the present invention;

[0120] like Figure 5 As shown, the electronic device 50 includes: one or more processors 51 and a memory 52, Figure 5 A processor 51 is taken as an example.

[0121] The processor 51 and the memory 52 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0122] Processor 51 is used to obtain operation instructions; according to the operation instructions, control the operation terminal to operate the mobile terminal user interface so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through a preset artificial intelligence model.

[0123] The embodiment of the present invention can help handicapped patients operate the mobile terminal by controlling the operation terminal to automatically operate the mobile terminal user interface according to the user's operation instructions, so that the handicapped patients can let the operation terminal complete complex operations by issuing simple instructions.

[0124] The memory 52 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules of the method for operating the mobile terminal user interface in the embodiment of the present application. The processor 51 executes various functional applications and data processing of the controller by running the non-volatile software programs, instructions and modules stored in the memory 52, that is, the method for operating the mobile terminal user interface in the above method embodiment is implemented.

[0125] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The one or more modules are stored in the memory 52, and when executed by the one or more processors 51, perform the method of operating the mobile terminal user interface in any of the above method embodiments, for example, perform the above described Figure 2 The method comprises steps S10 to S20.

[0127] The electronic device includes a mobile terminal and a robotic arm having one or more processors and a memory.

[0128] It should be noted that the above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in the present embodiment, please refer to the method provided in the embodiment of the present application.

[0129] An embodiment of the present invention also provides a non-volatile computer storage medium, which stores computer executable instructions. The computer executable instructions are executed by one or more processors, so that the one or more processors can execute the method of operating the mobile terminal user interface in any of the above method embodiments.

[0130] The above described device or equipment embodiments are merely illustrative, wherein the unit modules described as separate components may or may not be physically separated, and the components displayed as module units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network module units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence or in other words, the part that contributes to the relevant technology, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for operating a mobile terminal user interface, characterized in that: The method comprises: Get operation instructions; According to the operation instruction, the operation terminal is controlled to operate the mobile terminal user interface so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through a preset artificial intelligence model; The learning from the preset video by using the preset artificial intelligence model includes: Obtaining the preset video; Analyze the preset video and identify the operation object and operation subject in the preset video; The key operation path for the operation subject to operate the operation object is extracted, so that the operation terminal can correctly reproduce the corresponding operation through the key operation path.

2. The method according to claim 1, characterized in that: The method further comprises: Critical information corresponding to the critical operation path is set, wherein the critical information corresponds to at least one of the critical operation paths.

3. The method according to claim 2, characterized in that The method further comprises: Identify the operation instruction by using the preset artificial intelligence model, and find the key information corresponding to the operation instruction; According to the key operation path corresponding to the key information, the operation terminal is controlled to operate the mobile terminal user interface.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire an image of the mobile terminal user interface; According to the image, the operation process of the operation terminal on the mobile terminal user interface is corrected in real time.

5. The method according to claim 4, characterized in that The step of correcting the operation process in real time according to the image comprises: According to the image, a preset artificial intelligence model determines whether the user interface is a user interface corresponding to the operation process; if so, Then, continue to execute the operation process; otherwise, Control the mobile terminal to switch to a user interface corresponding to the operation process.

6. The method according to claim 5, characterized in that The determining, by a preset artificial intelligence model, whether the user interface is a user interface corresponding to the operation process includes: Training the preset artificial intelligence model; Obtaining samples of several mobile terminal user interfaces; Label each sample accordingly; According to the machine learning algorithm, the preset artificial intelligence model is trained using samples of the mobile terminal user interface and labels corresponding to each of the samples to obtain the trained preset artificial intelligence model.

7. The method according to claim 1, characterized in that The operation instruction is a voice instruction, and the operation terminal is a robotic arm with a stylus.

8. A device for operating a mobile terminal user interface, characterized in that: The device comprises: An acquisition module, used for acquiring operation instructions; an operation module, configured to control the operation terminal to operate the mobile terminal user interface according to the operation instruction, so that the operation terminal performs the operation corresponding to the operation instruction, wherein the key operation path for the operation terminal to perform the operation on the mobile terminal user interface is learned from a preset video through an artificial intelligence model; Wherein, the operation module is specifically used for: Obtaining the preset video; Analyze the preset video and identify the operation object and operation subject in the preset video; The key operation path for the operation subject to operate the operation object is extracted, so that the operation terminal can correctly reproduce the corresponding operation through the key operation path.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

Images