RPA robot intelligent element positioning and picking method and system

Abstract

The invention discloses an RPA robot intelligent element positioning and picking method and system. According to the method, firstly a problem that a remote desktop, a virtual system and other elements cannot be positioned and operated is solved through employing a computer vision technology combining template matching, OCR and image retrieval, and a proposed new scheme can more accurately and stably locate and pick up the positions and contents of the elements on the picture; secondly, an element positioning scheme based on template matching is adopted, a large number of labeling samples arenot needed, and the scheme is more stable and controllable; thirdly, pixel point features, corner point features and convolution features of a deep neural network are fused, so that the template matching effect is more accurate and stable, and meanwhile, the OCR technology can further assist in positioning elements with text information; and finally, a template library retrieval engine is constructed by adopting an image retrieval technology, and which application the page picture to be processed belongs to can be accurately judged, so that the problem of low retrieval and matching speed caused by template library growth in multiple scenes is solved.

Description

technical field [0001] The invention relates to the technical field of page element positioning, in particular to a method and system for positioning and picking intelligent elements of an RPA robot. Background technique [0002] With the gradual disappearance of my country's demographic dividend, the national economy has entered a "new steady state". Many enterprises will reduce labor costs and improve personnel efficiency as their core development demands, and use intelligent and automated technologies to empower and transform existing business operations, management and offices. The mode of entrusting simple tasks to machines has become a new trend and will bring new momentum to economic and social development. RPA (Robot Process Automation) technology is a process automation technology. Through the process editor provided by RPA software, business operation processes that can be automatically executed can be designed and configured, and packaged into the form of "software...

AI Technical Summary

Problems solved by technology

[0018] First of all, the computer vision technology combining template matching, OCR and image retrieval is used to solve the problem that elements such as remote desktops and virtual systems cannot be located and operated in RPA application scenarios. The proposed new solution can locate and pick up more accurately and stably The position and content of elements on the image;

Images