Equipment control method based on large model

Abstract

The invention relates to an equipment control method based on a large model, and aims to utilize professional experience of developers to supplement information gain and meet individual requirements of users. When the capacity of an existing large model is insufficient, the system allows a developer to upload expected operation steps or directly interact with screen information by opening an interface to the developer, and a control command is provided. Operation steps and task description are not limited by special formats, and unexpected information processing can be included. Meanwhile, in order to overcome the defects of a large model in the aspect of processing spatial position information, an independent image recognition model is introduced to recognize screen elements, and percentage coordinates are used for representing element positions. The identified element names and coordinates are numbered, the numbers are sorted according to the position information, and the sorting rule can be from top to bottom, from left to right and the like. The sorted element names and number lists (not including coordinate information) and sorting rules are provided for a developer or a large model, so that the confusion of the large model is reduced, and the processing speed and safety are improved. And the large model returns a control instruction using the ID of the element, and the system maps the ID back to the coordinate so as to realize accurate control of the equipment. The success rate can be calculated through any combination of information such as satisfaction, speed and whether the information is harmful or not. According to the method, the efficiency and safety of equipment control are improved by introducing a developer to participate in and solve position information processing limitation.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence and human-computer interaction technology, and specifically relates to a device control method based on a large model, combined with image recognition technology and multimodal data processing. Background Art

[0002] With the rapid development of artificial intelligence and big model technology, people hope to use big models to control mobile phones, computers and other devices to achieve more intelligent and personalized interactions. However, when converting user needs into specific device operations, it is usually a process of information gain, and the information content of user instructions is far less than the amount of information required for actual operations. The existing big models cannot completely fill this information gap within a period of time, and the same instruction may have different expected results in the context of different users. Therefore, it is difficult to accurately meet the personalized needs of each user by relying solely on big models.

[0003] Traditional solutions, such as marking elements on screenshots or sending coordinate information along with screenshots to large models, have problems such as low efficiency, mutual overlap, and easy confusion. These methods will destroy the original screenshot information, and when there are too many annotations, the processing efficiency of the large model will decrease and the difficulty of understanding will increase. In addition, the large model has deficiencies in processing spatial position information (such as precise coordinates), especially when the coordinate values ​​are large, the large model is not sensitive to coordinate information.

[0004] Therefore, there is an urgent need for a new method to effectively open interfaces to developers, meet users' personalized needs, solve the shortcomings of large models in processing location information, and improve the efficiency and safety of device control. Summary of the invention

[0005] The present invention aims at the problems existing in the prior art and provides a device control method based on a large model, which opens an interface to developers, supplements information gain, and meets the personalized needs of users. At the same time, the percentage coordinate and element numbering method are adopted to solve the shortcomings of the large model in processing position information.

[0006] The main technical solutions are as follows.

[0007] 1. Open interfaces to developers: The system is open to developers, allowing them to participate in the device control process. The open methods include:

[0008] A) Receive the expected operation steps and corresponding task descriptions uploaded by the developer, including the name and developer ID. There is no special format restriction for the operation steps and task descriptions, and the operation steps may include unexpected information handling (for example, handling pop-up windows, etc.).

[0009] Or B) Open screen information to developers, receive developer commands and developer IDs, and the process is completed by developers calling the large model. Developers can choose the appropriate model and supplement appropriate information based on their own experience and preferences.

[0010] 2. Use percentage coordinates and element numbers: In order to solve the problem of insufficient position information processing in large models, the system introduces an independent image recognition model to identify the device screen, obtain the percentage coordinates and name of each element, and number them.

[0011] 3. Element recognition and classification: Screenshot recognition is divided into two parts: target detection and classification recognition.

[0012] Object detection: Detect elements on the screen. Categories and coordinates include but are not limited to Chinese icons, images, buttons, checkboxes, input boxes, sliders, and text.

[0013] Classification and recognition: Classify and recognize the detected elements, including any combination of icon classification, text recognition, and image recognition.

[0014] 4. Element numbering and sorting: The identified element names and coordinates are numbered, and the numbers are sorted according to the position information. The sorting rules can be from top to bottom, from left to right, etc. The sorted element name and number list (excluding coordinate information) and sorting rules are provided to developers or large models to reduce confusion in large models and improve processing speed and security. Developers use element numbers in coordinate-related commands, and the system eventually restores the numbers to coordinate information.

[0015] 5. Command generation and execution: The developer or the big model generates a control command containing the element number. After receiving the command, the system restores the element number to a specific coordinate to achieve precise control of the device. If necessary, offset information can be added to the coordinates to specify the non-center position of the click element.

[0016] 6. System evaluation and optimization: The system automatically evaluates the effectiveness of the developer's work based on user needs and operation processes. The success rate can be calculated by any combination of satisfaction, speed, whether it is harmful, etc. The verification results are provided to the developer through the verification interface to help them improve the operation steps. Based on the success rate of historical calls, the system decides whether to use the developer's work in the future when there are similar needs. The first few calls require the user to specify the developer information, forcing the system to select the specified developer work and complete the cold start process.

[0017] Through the above technical solutions, the purpose of accurately controlling the equipment is achieved without increasing the processing burden of large models, the personalized needs of users are met, and the efficiency and safety of equipment control are improved. Beneficial Effects

[0018] The present invention solves the limitation of large models in meeting the personalized needs of users by opening interfaces to developers. The participation of developers can effectively supplement information gain, meet the personalized needs of different users, and improve the accuracy and efficiency of device control.

[0019] The use of percentage coordinates and element numbers solves the difficulty of large models in processing position information. The identified element names and coordinates are numbered, the numbers are sorted according to the position information, and the sorted element names and number lists (excluding coordinate information) and sorting rules are provided to developers or large models to reduce confusion for large models and improve processing speed and security. The use of percentage coordinates avoids the problem of inconsistent coordinates caused by different device screen resolutions and simplifies the operation logic.

[0020] The system optimizes the developer's usage strategy based on the success rate of historical calls, improves the reliability of device control, and has high practical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 : Flow chart of the system equipment control method of the present invention.

[0022] As shown in Figure 1, the system includes the following functional modules and interactive relationships.

[0023] User device: The user sends a request to the system of the present invention through the device, such as an operation instruction or a task request.

[0024] The system of the present invention: the system core module is responsible for receiving user requirements, calling image recognition models, multimodal large models and verification services, and interacting with developers.

[0025] Screenshot recognition model: performs target detection and classification recognition on the screenshots of the user's device, generates the percentage coordinates, names and numbers of the elements, and returns the results to the system of the present invention.

[0026] Multimodal big model: Provides developers or systems with multimodal data processing capabilities, including understanding contextual requirements and generating operational instructions.

[0027] Verification service: Verify the developer's operation steps and task completion status, and return the verification results to the system of the present invention to support the developer in optimizing his work.

[0028] Developer: The developer obtains user requirements and image recognition results through the system of the present invention, generates device operation instructions, and uploads task descriptions and operation steps for system calls. DETAILED DESCRIPTION

[0029] Example 1: Operational process of developer opening interface 1. Open interface The system is open to developers, allowing them to participate in the device control process. The open methods include:

[0030] A) Receive the expected operation steps and corresponding task descriptions uploaded by the developer, including the task name and developer ID. There is no special format for the operation steps and task descriptions, and the operation steps may include unexpected information processing (for example: click to skip, if there is an ad pop-up window).

[0031] Or B) Open screen information to developers, including user messages, screenshots, and image recognition results. The element IDs and element names in the image recognition results are sorted according to the location information and provided to developers, without including the original coordinate information. Developers receive this information and generate control instructions. The process is completed by the developer calling the large model.

[0032] 2. Receive instructions from developers Developers generate control instructions based on the information obtained. The element ID is used in the instruction, and the system eventually restores the element ID to specific coordinate information.

[0033] The interface for receiving developer instruction information may be in JSON format. For example.

[0034] Click on screen elements: { "type": "cmd_click", "payload": { "elementId": [element ID], "offset": [offset] } }.

[0035] Long press on a screen element: { "type": "cmd_longclick", "payload": { "elementId": [element ID], "time": "Duration", "offset": [offset] } }.

[0036] To drag screen elements: { "type": "cmd_slide", "payload": { "elementId": [source element ID, target element ID], "time": "Time-consuming", "offset": [source offset, target offset] } }.

[0037] Move the pointer over a screen element: { "type": "cmd_moveto", "payload": { "elementId": [element ID], "offset": [offset] } }.

[0038] 3. Instruction Execution The system receives instructions from developers and restores the element ID to specific coordinate information.

[0039] Execute the corresponding device control operation according to the instruction type and parameters.

[0040] The completion instruction has a timeout mechanism and will automatically complete after the timeout. After sending the completion instruction, the system will shut down the developer's ability to obtain screen information and control the device.

[0041] 4. Recording and evaluation of results The system records the requirements and operation results when users call on developers to work.

[0042] Based on the success rate of historical calls, decide whether to use the developer's work when there are similar needs in the future.

[0043] The system automatically evaluates the effectiveness of the developer's work based on user needs and the operation process. The success rate can be calculated through any combination of information such as satisfaction (how satisfied the user is with the result), speed (the time it takes to complete the operation), and whether it is harmful (whether the operation introduces unsafe factors).

[0044] The verification results are provided to developers through the verification interface to help them improve their operation steps.

[0045] Example 2: Predefinition and execution of operation steps 1. Operation steps upload The developer uploads the desired operation steps and the corresponding task description, including the task name and developer ID, through open method A.

[0046] An operation step may contain multiple instructions, arranged in order of execution, and may include unexpected information handling.

[0047] 2. Task matching and execution When a user proposes a relevant task, the system matches the corresponding operation steps according to the user's needs.

[0048] The system executes the instructions in the operation steps. The large model can handle unexpected situations during the operation (such as pop-up windows) and continue to execute the operation steps.

[0049] Example 3: Image recognition and element numbering 1. Screen information acquisition and recognition Take a screenshot of the device.

[0050] An independent image recognition model is used to analyze the screenshots. The image recognition process is divided into two parts.

[0051] Object detection: Detect elements on the screen. The categories and coordinates include but are not limited to Chinese icons, images, buttons, checkboxes, input boxes, sliders, and text. Get the coordinate information of the elements. The coordinates are expressed in percentage coordinates.

[0052] Classification and recognition: Classify and recognize detected elements, including any combination of icon classification, text recognition, and image recognition. Icon recognition and image recognition use the original content as the element name, without using extended meanings, to reduce categories and ambiguity.

[0053] 2. Element numbering and sorting The identified element names and coordinates are numbered, and the numbers are sorted according to the position information. The sorting rules can be from top to bottom, from left to right, etc.

[0054] Provide the sorted list of element names and numbers (without coordinate information) and sorting rules to developers or large models to reduce confusion in large models and improve processing speed and security.

[0055] 3. Information sending Provide developers or large models with a sorted list of element names and numbers and sorting rules.

[0056] The original coordinate information is not included to reduce confusion in large models and improve processing speed and safety.

[0057] Example 4: Instruction Generation and Execution 1. Large model instruction generation The large model generates a control instruction including element numbers according to the received element name and number list and sorting rules.

[0058] The command is in JSON format and contains "type" and "payload" fields.

[0059] 2. Instruction Execution The system receives the instruction and restores the element number to specific coordinate information.

[0060] Execute the corresponding device control operation according to the instruction type and parameters.

[0061] Instruction examples.

[0062] Open an application: { "type": "cmd_app", "payload": { "app_name": "Application Name", "url_scheme": "Apply URL Scheme" } } Among them, url_scheme can be replaced according to the requirements of different systems.

[0063] Enter text: { "type": "cmd_input", "payload": { "input": "Input text" } }.

[0064] Feedback to users: { "type": "reply_message", "payload": { "message": "Message content" } }.

[0065] Complete the operation: { "type": "finished", "payload": {} }.

[0066] Example 5: System evaluation and optimization 1. Automatic evaluation The system automatically evaluates the effectiveness of developers' work based on user needs and operation processes.

[0067] The success rate can be calculated using any combination of information such as satisfaction (how satisfied the user is with the result), speed (how long it takes to complete the operation), and whether it is harmful (whether the operation introduces unsafe factors).

[0068] 2. Verification interface The verification results are provided to developers through the verification interface to help them improve their operation steps and increase the success rate.

[0069] 3. Strategy Optimization The system optimizes the usage strategy for developer work based on the success rate of historical calls.

[0070] When there are similar requirements in the future, automatically decide whether to use the developer's work.

[0071] Through the above implementation modes, the present invention achieves the purpose of accurately controlling the device without increasing the processing burden of the large model. The identified element names and coordinates are numbered by percentage coordinates and element numbering, and sorted according to the position information, providing a sorted element name and number list and sorting rules, avoiding the large model from processing a large amount of coordinate information, and improving the processing speed and security. The interface is opened to developers, allowing professionals to participate in the information gain process, meeting the personalized needs of users, and having high practical value and application prospects. It should be understood that various modifications, equivalent substitutions and improvements to the present invention are within the scope of protection of the present invention.

Claims

1. A device control method based on a large model, characterized in that: The following steps are involved: Use device screen information to send user needs and necessary task information to the big model; The large model generates control instructions based on the information received; The system executes the control instructions to realize the control of the device; The system is open to developers to supplement information gain and meet the personalized needs of different users. The open methods include: A) Receive the expected operation steps and corresponding task descriptions uploaded by the developer, including the name and developer ID. The operation steps and task descriptions have no special format restrictions, and the operation steps may include unexpected information processing; Or B) open screen information to developers, receive developer commands and developer IDs, and the process is completed by the developer calling the large model.

2. The method according to claim 1, characterized in that The control instructions include but are not limited to clicking, long pressing, dragging, inputting, moving a pointer, sending a message to a user, opening an application, and completing an operation.

3. The method according to claim 2, characterized in that The completion instruction has a timeout mechanism and will automatically complete after the timeout. After sending the completion instruction, the system will shut down the developer's ability to obtain screen information and control the device based on the screen information.

4. The method according to claim 1, characterized in that: The system records the requirements entered by the user when calling the developer's work, marks the content of the developer's work, and decides whether to use the developer's work in the future when there are similar requirements based on the success rate of historical calls.

5. The method according to claim 4, characterized in that The first few calls require the user to specify the developer information, forcing the system to select the specified developer to work and complete the cold start process.

6. The method according to claim 1, characterized in that The interface of open method B includes opening screenshot recognition to developers and receiving element IDs in the image recognition results selected by developers.

7. The method according to claim 6, characterized in that Screenshot recognition is divided into two parts: object detection and classification recognition; The categories and coordinates of target detection include but are not limited to Chinese icons, images, buttons, checkboxes, input boxes, sliders, and texts. The content of classification recognition includes any combination of icon classification, text recognition, and image recognition.

8. The method according to claim 7, characterized in that The identified element names and coordinates are numbered, and the numbers are sorted according to the position information. The sorting rules can be from top to bottom, from left to right, etc.; the sorted element name and number list (excluding coordinate information) and sorting rules are provided to developers or large models to reduce confusion in large models and improve processing speed and security; developers use element numbers in coordinate-related commands, and the system eventually restores the numbers to coordinate information.

9. The method according to claim 7, characterized in that: Icon recognition and image recognition use the original content as the element name, without using extended meanings.

10. The method according to claim 7, characterized in that Coordinates are expressed in percentage form.

11. The method according to claim 8, characterized in that The coordinates contain offset information.

12. The method according to claim 8, characterized in that The interface for receiving developer instruction information may be in JSON format.

13. The method according to claim 12, characterized in that When using click, long press, drag, pointer movement and other operation commands, the relevant JSON definitions are as follows: Click on screen elements: { "type": "cmd_click", "payload": { "elementId": [element ID], "offset": [offset] } } Long press on a screen element: { "type": "cmd_longclick", "payload": { "elementId": [element ID], "time": "Duration", "offset": [offset] } } To drag screen elements: { "type": "cmd_slide", "payload": { "elementId": [source element ID, target element ID], "time": "Time-consuming", "offset": [source offset, target offset] } } Move the pointer over a screen element: { "type": "cmd_moveto", "payload": { "elementId": [element ID], "offset": [offset] } }。 14. The method according to claim 2 or 12, characterized in that: When using commands such as opening an application, text input, replying to user information, and completing an operation, the relevant JSON definitions are as follows: Open an application: { "type": "cmd_app", "payload": { "app_name": "Application Name", "url_scheme": "Apply URL Scheme" } } Among them, url_scheme can be replaced according to the requirements of different systems; Enter text: { "type": "cmd_input", "payload": { "input": "Input text" } } Provide feedback to users and record it: { "type": "reply_message", "payload": { "message": "Message content" } } When the evaluation task is complete, end all operations: { "type": "finished", "payload": {} }。 15. The method according to claim 1, characterized in that The system automatically evaluates the effectiveness of the developer's work based on user needs and operation processes. The evaluation content includes any combination of information such as satisfaction, speed, and whether it is harmful.

16. The method according to claim 15, characterized in that The verification results are provided to developers through the verification interface.

Images