The invention relates to an equipment control method based on a
large model, and aims to utilize professional experience of developers to supplement
information gain and meet individual requirements of users. When the capacity of an existing
large model is insufficient, the
system allows a developer to upload expected operation steps or directly interact with screen information by opening an interface to the developer, and a control command is provided. Operation steps and task description are not limited by special formats, and unexpected
information processing can be included. Meanwhile, in order to overcome the defects of a
large model in the aspect of
processing spatial position information, an independent image recognition model is introduced to recognize screen elements, and percentage coordinates are used for representing element positions. The identified element names and coordinates are numbered, the numbers are sorted according to the position information, and the sorting rule can be from top to bottom, from left to right and the like. The sorted element names and number lists (not including coordinate information) and sorting rules are provided for a developer or a large model, so that the
confusion of the large model is reduced, and the
processing speed and safety are improved. And the large model returns a control instruction using the ID of the element, and the
system maps the ID back to the coordinate so as to realize accurate control of the equipment. The success rate can be calculated through any combination of information such as satisfaction, speed and whether the information is harmful or not. According to the method, the efficiency and safety of equipment control are improved by introducing a developer to participate in and solve position
information processing limitation.