RPA automation method, system and equipment based on improved fast regional convolutional neural network model, and medium

Abstract

The invention discloses an RPA automation method, system and device based on an improved fast regional convolutional neural network model, and a medium, and relates to the technical field of robot process automation, and the method comprises the steps: employing an image annotation tool to make a target data set; building an improved fast regional convolutional neural network model; a feature pyramid network layer is added in front of a region candidate network, and original region interest pooling is changed into region interest alignment; training the model in a preheating learning rate mode; and inputting a to-be-identified picture into the trained neural network model to obtain a detection result. The improved 50-layer residual neural network is used for feature extraction, normal residual convolution is replaced with grouping convolution, and the training and reasoning efficiency is improved. And a feature pyramid network layer is added, so that the detection capability of shallow high-resolution small targets of a bottom-layer feature map is enhanced. The method has the advantages of high target detection capability, strong adaptability and anti-interference performance, and high robustness.

Description

Technical Field

[0001] The present invention relates to the field of robotic process automation technology, and in particular to an RPA automation method, system, device and medium based on an improved fast regional convolutional neural network model. Background Art

[0002] Robotic Process Automation (RPA) automates user operation processes by mimicking the manual operation methods of end users on computers. It uses horizontal and business-flow-friendly technologies to automate repetitive, rule-based tasks that require people to switch between different systems, simulating human interaction to complete repetitive tasks. RPA generally uses optical character recognition (OCR) for text recognition, determines its shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text. For printed characters, it uses an optical method to convert the text in paper documents into black and white dot matrix image files, and uses recognition software to convert the text in the image into text format for further editing and processing by word processing software.

[0003] By combining RPA and OCR, we can realize the workflow of automatically processing large amounts of text data, improving work efficiency and accuracy. However, traditional OCR text recognition is to cut the characters in the text and then recognize the individual characters, and the text coordinates are relatively strict. Problems such as text misalignment, blur, low resolution, and wrong lines will cause the recognition effect to generate garbled characters, and the recognition effect is extremely unstable. In addition, OCR technology is limited to text recognition and cannot recognize other types of data, such as images or graphics. For RPA that must process non-text and other types of data, traditional OCR is difficult to meet the requirements. Therefore, in the scenario of RPA automating the processing of complex UI (user interface) interfaces, especially non-text elements such as scroll bar radio boxes, time elements, pictures and animations, sliders, calendars, views, etc., OCR is difficult to meet the requirements. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is: how to solve the problem of RPA scenarios in which traditional OCR text recognition is to cut characters in the text and then recognize individual characters, and has strict requirements on text coordinates. Problems such as text misalignment, blur, low resolution, and wrong lines will cause the recognition effect to generate garbled characters, the recognition effect is extremely unstable, and it is difficult to recognize non-text data.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: an RPA automation method based on an improved fast regional convolutional neural network model, comprising: in a robotic process automation scenario, collecting UI interfaces that need to be automated, and using image annotation tools to produce a target data set; building an improved fast regional convolutional neural network model, and the backbone feature extraction network adopts an improved 50-layer residual neural network; adding a feature pyramid network layer before the regional candidate network to achieve multi-scale feature fusion, and changing the original regional interest pooling to regional interest alignment; training the model by preheating the learning rate, using binary cross entropy and smoothed L1 loss as classification loss function and regression loss function, and using stochastic gradient descent to optimize the loss function; inputting the picture to be identified into the trained neural network model to obtain the detection result.

[0007] As a preferred solution of the RPA automation method based on the improved fast regional convolutional neural network model described in the present invention, wherein: the target data set is prepared including data annotation using image annotation tools, the generated format is .json file, and the label box format is <left> 、 <top> 、 <right> 、 <bottom>.

[0008] As a preferred solution of the RPA automation method based on the improved fast regional convolutional neural network model described in the present invention, the backbone feature extraction network adopts an improved 50-layer residual neural network, including replacing the first layer 7×7 convolution kernel with three 3×3 convolution kernels; in the residual block, group convolution is used to replace the normal residual convolution; in the identical connection part, 1×1 convolution is only used to change the dimension, and a pooling layer is added separately for downsampling, which is expressed as: , , in, and To input the image width and height, and is the output width and height, and is the width and height of the convolution kernel, is the number of filled circles, is the step size of the convolution kernel; it is calculated that the image feature size obtained by three consecutive 3×3 convolutions and a 7×7 convolution is the same, but the parameter amount is: , in, is the number of convolution kernels.

[0009] As a preferred solution of the RPA automation method based on the improved fast regional convolutional neural network model described in the present invention, the multi-scale feature fusion includes adding a feature pyramid network layer after improvement, inputting the feature map generated by the feature pyramid network layer and the prediction box output by the regional candidate network, and determining the feature map corresponding to the prediction box, which is expressed as: , in, and is the width and height of the prediction box, is the matching feature map, is a constant coefficient and 224 is the size of pre-training on the image network.

[0010] As a preferred solution of the RPA automation method based on the improved fast regional convolutional neural network model described in the present invention, wherein: the original regional interest pooling is changed to regional interest alignment, including using regional interest alignment to cancel the quantization operation, using a bilinear interpolation method to obtain the image value at the pixel point whose coordinates are floating point numbers, and converting the entire feature aggregation process into a continuous operation. The back propagation is expressed as: , in, is the candidate region, is a point in the candidate area, Represents the distance between two points. is the pixel value of a point on the feature map. is the floating point coordinate after pooling, for and The difference in ordinate, for and The difference in the horizontal axis, is the interpolated coordinate pixel value, After pooling The candidate region Points, are integer coordinate points on the feature map.

[0011] As a preferred solution of the RPA automation method based on the improved fast regional convolutional neural network model described in the present invention, the training model by preheating the learning rate includes the loss functions of the improved fast regional convolutional neural network model being the loss functions of the regional candidate network and the regional convolutional neural network, including classification loss and regression loss, expressed as: , in, To improve the loss function of the fast regional convolutional neural network model, is the cross entropy loss, is the smooth L1 loss, For the The classification prediction probability of samples, For the The position regression prediction coordinate value of samples, is the total number of classification items, For the The true category labels of samples, is the balance coefficient, is the total number of regression terms, For the The actual position coordinates of the samples.

[0012] As a preferred solution of the RPA automation method based on the improved fast regional convolutional neural network model described in the present invention, the cross entropy loss is expressed as: , The smooth L1 loss is expressed as: , in, is the smooth L1 loss function; , in, That is, the smooth L1 loss function with x as the dependent variable, x is the difference between the predicted value and the true value, is a hyperparameter.

[0013] Another object of the present invention is to provide a system for an RPA automation method based on an improved fast regional convolutional neural network model, which can solve the RPA automation problem based on an improved fast regional convolutional neural network model by constructing an RPA automation system based on an improved fast regional convolutional neural network model.

[0014] To solve the above technical problems, the present invention provides the following technical solutions: an RPA automation system based on an improved fast regional convolutional neural network model, comprising a data acquisition module, a model building module, a feature fusion module, a training module and a detection module; the data acquisition module is used to collect UI interfaces that need to be automated in a robotic process automation scenario, and use an image annotation tool to produce a target data set; the model building module is used to build an improved fast regional convolutional neural network model, and the backbone feature extraction network uses an improved 50-layer residual neural network; the feature fusion module is used to add a feature pyramid network layer before the regional candidate network to achieve multi-scale feature fusion, and change the original regional interest pooling to regional interest alignment; the training module is used to train the model by preheating the learning rate, using binary cross entropy and smooth L1 loss as the classification loss function and regression loss function, and using the stochastic gradient descent method to optimize the loss function; the detection module is used to input the picture to be identified into the trained neural network model to obtain the detection result.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the RPA automation method based on an improved fast regional convolutional neural network model as described above are implemented.

[0016] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of an RPA automation method based on an improved fast regional convolutional neural network model as described above are implemented.

[0017] The beneficial effects of the present invention are as follows: The RPA automation method based on the improved fast regional convolutional neural network model provided by the present invention adopts an improved fast regional convolutional neural network model to solve the RPA scenario where OCR is difficult to recognize non-text data in the face of complex UI interfaces. An improved 50-layer residual neural network is used for feature extraction, and grouped convolution is used to replace normal residual convolution to improve training and reasoning efficiency. A feature pyramid network layer is added before the regional candidate network to achieve multi-scale feature fusion and enhance the detection capability of shallow high-resolution small targets in the underlying feature map. Regional interest pooling is replaced with regional interest alignment to improve detection accuracy; during training, a warm-up learning rate is used to make the model convergence speed faster and the model effect better. Compared with the prior art, the present invention proposes an RPA automation method based on an improved fast regional convolutional neural network model, which has high target detection capability, strong adaptability and anti-interference, and high robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 A flowchart of an RPA automation method based on an improved fast regional convolutional neural network model is provided for the first embodiment of the present invention.

[0020] Figure 2 A schematic diagram of an application system of an RPA automation method based on an improved fast regional convolutional neural network model provided for the second embodiment of the present invention.

[0021] Figure 3 A schematic diagram of an original fast regional convolutional neural network model of an RPA automation method based on an improved fast regional convolutional neural network model provided for the second embodiment of the present invention.

[0022] Figure 4 A schematic diagram of an improved 50-layer residual neural network for an RPA automation method based on an improved fast regional convolutional neural network model provided for the second embodiment of the present invention.

[0023] Figure 5 A schematic diagram of an improved fast regional convolutional neural network model for an RPA automation method based on an improved fast regional convolutional neural network model is provided for the second embodiment of the present invention.

[0024] Figure 6 A structural diagram of an RPA automation system based on an improved fast regional convolutional neural network model provided for the third embodiment of the present invention.

[0025] In the figure: 100, data acquisition module; 200, model building module; 300, feature fusion module; 400, training module; 500, detection module. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0027] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0028] Example 1, reference Figure 1 , which is the first embodiment of the present invention, and this embodiment provides an RPA automation method based on an improved fast regional convolutional neural network model, including: in a robotic process automation (RPA) scenario, collecting UI interfaces that need to be automated, and using image annotation tools to create a target data set; building an improved fast regional convolutional neural network model, and the backbone feature extraction network uses an improved 50-layer residual neural network; adding a feature pyramid network layer before the regional candidate network to achieve multi-scale feature fusion, and changing the original regional interest pooling to regional interest alignment; training the model by preheating the learning rate, using binary cross entropy and smoothed L1 loss as classification loss function and regression loss function, and using stochastic gradient descent to optimize the loss function; inputting the image to be identified into the trained neural network model to obtain the detection result.

[0029] First, based on the basic text recognition of optical character recognition, an improved fast regional convolutional neural network model is used to solve the problem that optical character recognition is difficult to recognize non-text data in complex UI interface scenarios.

[0030] Secondly, in order to improve the reasoning efficiency, an improved 50-layer residual neural network is used to extract features from the backbone network of the original fast regional convolutional neural network model.

[0031] Thirdly, in view of the low accuracy and efficiency of the original fast regional convolutional neural network model in detecting small targets, a feature pyramid network layer is introduced to enhance the detection capability of small targets with shallow high resolution in the underlying feature map.

[0032] Finally, replacing regional interest pooling with regional interest alignment can avoid double quantization errors and the impact of misalignment on small targets, which can improve the accuracy of the detection model.

[0033] Compared with the prior art, the RPA automation method based on the improved fast regional convolutional neural network model proposed in the present invention has higher target detection capability, strong adaptability and anti-interference, and higher robustness.

[0034] S1. In the robotic process automation scenario, collect the UI interfaces that need to be automated and use image annotation tools to create the target dataset.

[0035] S2. Build an improved fast regional convolutional neural network model, and the backbone feature extraction network adopts an improved 50-layer residual neural network.

[0036] S3. Add a feature pyramid network layer before the region candidate network to achieve multi-scale feature fusion and change the original region interest pooling to region interest alignment.

[0037] S4. Train the model by warming up the learning rate, use binary cross entropy and smooth L1 loss as the classification loss function and regression loss function, and use stochastic gradient descent to optimize the loss function.

[0038] S5. Input the image to be identified into the trained neural network model to obtain the detection result.

[0039] Furthermore, the data annotation in step S1 is all generated using image annotation tools in the .json format. The format of the label box is: <left> <top> <right> <bottom>.

[0040] Furthermore, the backbone network of step S2 uses an improved 50-layer residual neural network: the first layer 7×7 convolution kernel is replaced with three 3×3 convolution kernels, which significantly reduces the number of parameters and calculations while keeping the receptive field unchanged; in the residual block, grouped convolution is used to replace the normal residual convolution, ensuring that multi-scale features can be extracted at a finer granularity while maintaining the same number of parameters and calculations; in the identical connection part, 1×1 convolution is only used to change the dimension, and a separate pooling layer is added for downsampling to avoid the simultaneous occurrence of 1×1 convolution and downsampling, which causes information loss. The formula is as follows: , , in, and To input the image width and height, and is the output width and height, and is the width and height of the convolution kernel, is the number of filled circles, is the stride of the convolution kernel.

[0041] , , in, is the input shape latitude, is the output shape latitude, and are the number of input and output image channels respectively, divided into g groups, then the size of each convolution kernel is k*k*C2 / g. According to the formula, the number of parameters of the conventional convolution kernel is: k*k*C1*C2, and the number of grouped convolution parameters is: k*k*C1*C2 / g.

[0042] Furthermore, in step S3, before the improvement, the feature map generated by the region-based convolutional neural network as the backbone network and the prediction box output by the region candidate network are input; after the improvement, the feature pyramid network layer is added, and the feature map generated by the feature pyramid network layer and the prediction box output by the region candidate network are input. The feature map corresponding to the prediction box is determined according to the following formula to perform the subsequent region interest alignment operation, which is expressed as: , in, and is the width and height of the prediction box, is the matching feature map, is a constant coefficient, and the present invention takes 4,224 as the pre-training size on the image network.

[0043] In step S3, the regional interest pooling used before the improvement caused quantization errors due to double rounding. After the improvement, the regional interest alignment was used to cancel the quantization operation, and the bilinear interpolation method was used to obtain the image values ​​at the pixel points with floating-point coordinates, thereby converting the entire feature aggregation process into a continuous operation. The back propagation formula is: , in, is the candidate region, is a point in the candidate area, Represents the distance between two points. is the pixel value of a point on the feature map. is the floating point coordinate after pooling, for and The difference in ordinate, for and The difference in the horizontal axis, is the interpolated coordinate pixel value, After pooling The candidate region Points, are integer coordinate points on the feature map.

[0044] Furthermore, a warm-up learning rate is used in step S4; the loss functions of the improved fast regional convolutional neural network model are the loss functions of the regional candidate network and the region-based convolutional neural network, both of which include classification loss and regression loss, expressed as: , in, To improve the loss function of the fast regional convolutional neural network model, is the cross entropy loss, is the smooth L1 loss, For the The classification prediction probability of samples, For the The position regression prediction coordinate value of samples, is the total number of classification items, For the The true category labels of samples, is the balance coefficient, is the total number of regression terms, For the The actual position coordinates of the samples.

[0045] The cross entropy loss is expressed as: , The smooth L1 loss is expressed as: , in, is the smooth L1 loss function; , in, That is, the smooth L1 loss function with x as the dependent variable, x is the difference between the predicted value and the true value, is a hyperparameter.

[0046] Example 2, reference Figure 2~Figure 5 , which is the second embodiment of the present invention, provides an RPA automation method based on an improved fast regional convolutional neural network model based on the above embodiment.

[0047] The present invention proposes an RPA automation method based on an improved fast regional convolutional neural network model, in which optical character recognition is used to recognize traditional text interfaces and programs, and the improved fast regional convolutional neural network model is used to recognize complex UI interfaces and programs. Figure 2 shown.

[0048] The original fast regional convolutional neural network model is a two-stage deep learning detection algorithm. Its biggest innovation is that it generates candidate boxes based on the anchor mechanism (instead of selective search) by adding a regional candidate network, and finally integrates feature extraction, candidate box selection, border regression and classification into one network, thereby effectively improving detection accuracy and efficiency. The specific process of the original fast regional convolutional neural network model is to scale the input image and enter the convolution layer to extract features to obtain a feature map, and then send the feature map to the regional candidate network to generate a series of possible candidate boxes. Next, the original feature map and all candidate boxes output by the regional candidate network are input into the regional interest pooling layer to extract and collect candidates, and calculate a fixed size 7×7 candidate feature map, which is sent to the fully connected layer for target classification and coordinate regression. The schematic diagram of the original fast regional convolutional neural network model is shown in the attached figure. Figure 3 shown.

[0049] The present invention designs a novel improved fast regional convolutional neural network model, which provides the ability to detect small targets in complex UI interfaces. The backbone network uses an improved 50-layer residual neural network, which reduces the amount of calculation and improves the detection accuracy. The schematic diagram of the improved 50-layer residual neural network is shown in the attached figure. Figure 4 shown.

[0050] The original fast regional convolutional neural network model extracts features through convolution, and the generated feature map is only single-layer, and the resolution is usually small, which is not conducive to the detection of small objects and multi-scale objects. Therefore, this improvement introduces a feature pyramid network layer to generate multi-layer fused feature maps, increase the resolution of feature maps, and enhance the detection efficiency of multiple small targets. The original fast regional convolutional neural network model uses regional interest pooling for two quantizations, which leads to a loss of accuracy. Therefore, improvements are made to this pooling, and the use of regional interest alignment improves the positioning accuracy. The original fast regional convolutional neural network model finally uses a fully connected network. This part of the fully connected network occupies most of the parameters of the network, and after regional interest pooling, each region of interest must pass through the fully connected network once, and there is no shared calculation. This improvement uses a fully convolutional network to replace this part of the fully connected network to achieve a lightweight network. The schematic diagram of the improved fast regional convolutional neural network model is shown in the attached figure. Figure 5 shown.

[0051] The 50-layer residual neural network of the backbone network extracts features from the input image (height*width), and the outputs are conv2_x residual block 2, conv3_x residual block 3, conv4_x residual block 4 and conv5_x residual block 5. The corresponding shapes are (256, 256, 256), (128, 128, 512), (64, 64, 1024), and (32, 32, 2048) according to the height, width and number of channels. From the conv5_x high-level network to the conv2_x layer by layer, the network is shallow. Different network feature layers represent different meanings. The shallow network pays more attention to detail information, which is conducive to the detection of high-resolution small targets. The high-level network pays more attention to semantic information, which is more conducive to accurate detection of targets. Each layer is used as input to the feature pyramid network layer to achieve feature fusion and predict objects of different sizes on feature maps of multiple scale levels, thereby improving the multi-scale detection capability of the improved fast regional convolutional neural network model. conv5_x is input to the feature pyramid network layer, performs 1*1*256 convolution to reduce dimension, and connects to two branches. One branch performs 3*3*256 convolution to generate C5 with a dimension of (32, 32, 256). The improvement here does not downsample C5 to generate C6, because there is no need to detect larger images in practice; the other branch performs top-down and generates C5' with a dimension of (64, 64, 256) after 2 times upsampling. conv4_x is input to the feature pyramid network layer, performs 1*1*256 convolution to reduce dimension, and then adds it to the high-level C5' to achieve feature fusion. Then it connects to two branches. One branch performs 3*3*256 convolution to generate C4 with a dimension of (64, 64, 256), and the other branch performs top-down and generates C4' with a dimension of (128, 128, 256) after 2 times upsampling. conv3_x is input to the feature pyramid network layer, and after performing 1*1*256 convolution to reduce the dimension, it is added with the high-level C4' to achieve feature fusion. Then two branches are connected. One branch performs 3*3*256 convolution to generate C3 with a dimension of (128, 128, 256), and the other branch performs top-down and 2x upsampling to generate C3' with a dimension of (256, 256, 256).

[0052] conv2_x is input to the feature pyramid network layer, and after performing 1*1*256 convolution to reduce the dimension, it is added with the high-level C3' to achieve feature fusion, and then 3*3*256 convolution is performed to generate C2 with a dimension of (256, 256, 256). C5, C4, C3, and C2 are multi-scale fusion feature maps, which are respectively input to the region candidate network. The region candidate network includes two branches, which respectively classify the foreground and background of the input feature map and regress the label box offset of the anchor point. The anchor point generated by each feature map is k. According to non-maximum suppression (IOU=threshold), m prediction boxes are selected and passed to the final region-based convolutional neural network.

[0053] The parameters are set as follows: H, W∈(600-1000), k=1000, threshold=0.7, m=1000.

[0054] The input of the region-based convolutional neural network is the first four feature maps generated by the feature pyramid network layer and the prediction boxes output by the region candidate network. The four feature maps are C2, C3, C4, and C5. The region interest alignment operation corresponding to each prediction box is determined on the feature map according to the formula. The network is trained according to the loss function of the formula.

[0055] Use a warm-up learning rate, starting from the initial small learning rate, and increase each step a little bit until the relatively large learning rate set initially is reached, and then use the initially set learning rate for training. This makes the model converge faster and the model effect better.

[0056] Training steps: The first step is to train the region candidate network, use the pre-trained model to initialize the shared convolution and region candidate network weights, and train the region candidate network end-to-end to generate region candidate boxes; the second step is to train the region-based convolutional neural network, use the same pre-trained model to initialize the shared convolution, initialize a new shared convolutional network with the same structure as the first step, instead of the convolutional network trained in the first step, lock the weights of the region candidate network trained in the first step, and train the region-based convolutional neural network in combination with the candidates obtained from the region candidate network; the third step is to tune the region candidate network, use the shared convolution and region-based convolutional neural network trained in the first step, fix the shared convolution layer, and continue to train the region candidate network; the fourth step is to tune the region-based convolutional neural network, use the shared convolution and region candidate network trained in the third step, continue to train and fine-tune the region-based convolutional neural network, and obtain the final complete improved fast region convolutional neural network model after training.

[0057] The training parameters are set as follows: Weight decay = 0.0002, mometum = 0.8, lre = 0.002, and the learning rate is adjusted using step decay.

[0058] Example 3, reference Figure 6 , which is the third embodiment of the present invention. It is different from the previous two embodiments in that it provides an RPA automation system based on an improved fast regional convolutional neural network model, including: a data acquisition module 100, a model building module 200, a feature fusion module 300, a training module 400 and a detection module 500.

[0059] The data collection module 100 is used to collect UI interfaces that need to be automated in the Robotic Process Automation (RPA) scenario, and to create a target data set using an image annotation tool.

[0060] The model building module 200 is used to build an improved fast regional convolutional neural network model, and the backbone feature extraction network adopts an improved 50-layer residual neural network.

[0061] The feature fusion module 300 is used to add a feature pyramid network layer before the region candidate network to achieve multi-scale feature fusion and change the original region interest pooling into region interest alignment.

[0062] The training module 400 is used to train the model by preheating the learning rate, using binary cross entropy and smooth L1 loss as the classification loss function and regression loss function, and using the stochastic gradient descent method to optimize the loss function.

[0063] The detection module 500 is used to input the image to be identified into the trained neural network model to obtain the detection result.

[0064] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0065] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or in conjunction with such instruction execution system, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, apparatus or device, or in conjunction with such instruction execution system, apparatus or device.

[0066] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0067] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0068] Embodiment 4 is the fourth embodiment of the present invention, which is different from the first three embodiments in that it provides a detailed description of the implementation steps used in the present invention.

[0069] In order to verify the effectiveness of the method proposed in this paper, the present invention conducts a comparative study on the recognition effects of the traditional optical character recognition, the original fast regional convolutional neural network model, and the improved fast regional convolutional neural network model, i.e., the method of the present invention, in a complex UI interface environment. Table 1 is a comparison of the number of targets detected by the three methods. It can be seen that the method proposed in this invention has higher detection accuracy than the traditional control method.

[0070] The RPA automation method designed by the present invention based on the improved fast regional convolutional neural network model has better target recognition performance and detection capability than the traditional optical character recognition method.

[0071] The comparison table of the number of targets detected by the three methods is shown in Table 1.

[0072] Table 1 Comparison of the number of targets detected by the three methods

[0073] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.< / bottom> < / right> < / top> < / left> < / bottom> < / right> < / top> < / left>

Claims

1. An RPA automation method based on an improved fast regional convolutional neural network model, characterized in that: include, In the robotic process automation scenario, collect the UI interfaces that need to be automated and use image annotation tools to create the target dataset; An improved fast regional convolutional neural network model was built, and the backbone feature extraction network adopted an improved 50-layer residual neural network; A feature pyramid network layer is added before the region candidate network to achieve multi-scale feature fusion and change the original region interest pooling to region interest alignment; The model is trained by warming up the learning rate, using binary cross entropy and smooth L1 loss as the classification loss function and regression loss function, and the stochastic gradient descent method is used to optimize the loss function; Input the image to be identified into the trained neural network model to obtain the detection result.

2. The RPA automation method based on the improved fast regional convolutional neural network model according to claim 1, characterized in that: The target dataset is made by using image annotation tools, including data annotation, and the generated format is .json file, and the label box format is <left> 、 <top> 、 <right> 、 <bottom> 。< / bottom> < / right> < / top> < / left> 3. The RPA automation method based on the improved fast regional convolutional neural network model as claimed in claim 2, characterized in that: The backbone feature extraction network adopts an improved 50-layer residual neural network, including replacing the first layer 7×7 convolution kernel with three 3×3 convolution kernels; In the residual block, grouped convolution is used to replace the normal residual convolution; In the identical connection part, the 1×1 convolution is only used to change the dimension, and a single pooling layer is added for downsampling, which is expressed as, ， ， in, and To input the image width and height, and is the output width and height, and is the width and height of the convolution kernel, is the number of filled circles, is the step size of the convolution kernel; It is calculated that the image feature size obtained by three consecutive 3×3 convolutions and one 7×7 convolution is the same, but the number of parameters is ， in, is the number of convolution kernels.

4. The RPA automation method based on the improved fast regional convolutional neural network model as claimed in claim 3, characterized in that: The multi-scale feature fusion includes adding a feature pyramid network layer after improvement, inputting the feature map generated by the feature pyramid network layer and the prediction box output by the region candidate network, and determining the feature map corresponding to the prediction box, which is expressed as: ， in, and is the width and height of the prediction box, is the matching feature map, is a constant coefficient and 224 is the size of pre-training on the image network.

5. The RPA automation method based on the improved fast regional convolutional neural network model according to claim 4, characterized in that: The method of changing the original regional interest pooling to regional interest alignment includes canceling the quantization operation by regional interest alignment, obtaining the image value of the pixel point whose coordinates are floating point numbers by using the bilinear interpolation method, and converting the entire feature aggregation process into a continuous operation. The back propagation is expressed as: ， in, is the candidate region, is a point in the candidate area, Represents the distance between two points. is the pixel value of a point on the feature map. is the floating point coordinate after pooling, for and The difference in ordinate, for and The difference in the horizontal axis, is the interpolated coordinate pixel value, After pooling The candidate region Points, are integer coordinate points on the feature map.

6. The RPA automation method based on the improved fast regional convolutional neural network model according to claim 5, characterized in that: The training model by warming up the learning rate includes improving the loss functions of the fast regional convolutional neural network model to be the loss functions of the regional candidate network and the regional convolutional neural network, including classification loss and regression loss, expressed as: ， in, To improve the loss function of the fast regional convolutional neural network model, is the cross entropy loss, is the smooth L1 loss, For the The classification prediction probability of samples, For the The position regression prediction coordinate value of samples, is the total number of classification items, For the The true category labels of samples, is the balance coefficient, is the total number of regression terms, For the The actual position coordinates of the samples.

7. The RPA automation method based on the improved fast regional convolutional neural network model according to claim 6, characterized in that: The cross entropy loss is expressed as, ， The smooth L1 loss is expressed as, ， in, is the smooth L1 loss function; ， in, That is, the smooth L1 loss function with x as the dependent variable, x is the difference between the predicted value and the true value, is a hyperparameter.

8. A system using an RPA automation method based on an improved fast regional convolutional neural network model as described in any one of claims 1 to 7, characterized in that: It includes a data collection module (100), a model building module (200), a feature fusion module (300), a training module (400) and a detection module (500); The data collection module (100) is used to collect UI interfaces that need to be automated in a robotic process automation scenario, and to create a target data set using an image annotation tool; The model building module (200) is used to build an improved fast regional convolutional neural network model, and the backbone feature extraction network adopts an improved 50-layer residual neural network; The feature fusion module (300) is used to add a feature pyramid network layer before the region candidate network to achieve multi-scale feature fusion and change the original region interest pooling into region interest alignment; The training module (400) is used to train the model by preheating the learning rate, using binary cross entropy and smooth L1 loss as the classification loss function and regression loss function, and using the stochastic gradient descent method to optimize the loss function; The detection module (500) is used to input the image to be identified into a trained neural network model to obtain a detection result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of an RPA automation method based on an improved fast regional convolutional neural network model described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of an RPA automation method based on an improved fast regional convolutional neural network model described in any one of claims 1 to 7 are implemented.

Images