RPA robot production method and medium based on software screenshots

Abstract

The present invention relates to a method and medium for producing an RPA robot based on software screenshots, and relates to the field of computer information processing technology, including: entering the operating interface of a software system, automatically operating the operating interface according to a preset operating mode, taking a screenshot of the interface after each operation and saving it, and recording the operation coordinate position corresponding to the screenshot after each operation; for the saved screenshots, identifying the screenshots with the same interface and retaining one of the screenshots, and obtaining screenshots with different interfaces; based on the retained screenshots, obtaining preset fields, and obtaining the corresponding relationship between the screenshot with any preset field and the preset field; based on the corresponding relationship and the page where the preset field is located, obtaining the operation coordinates of the operation to be performed, and producing an RPA robot that takes a screenshot of the page where the preset field is located. Compared with the existing technology, the workload of RPA robot production can be reduced, and labor costs can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing technology, and in particular to a method and medium for producing an RPA robot based on software screenshots. Background Art

[0002] There are a large number of stroke patients, and they account for a huge proportion of the causes of death and disability, which has brought a heavy burden to families and society. Therefore, after stroke patients are discharged from the hospital, the hospital must promptly upload 357 treatment data indicators of the stroke patients during their hospitalization to the prescribed stroke database to further conduct stroke case quality control and data analysis, optimize treatment measures, improve treatment rates, and reduce the disease burden.

[0003] With the rapid development of information technology and the widespread application of artificial intelligence, it is necessary to develop a software robot based on RPA (Robotic Process Automation) technology to automatically extract, review, and report stroke center data. This robot will be used by thousands of hospital stroke centers nationwide to assist hospitals in reporting stroke data. However, RPA development requires customization for different vendors and / or data source software versions. Each company has multiple versions of the same system, and each version requires a new robot, which significantly increases the manual RPA workload. Reducing the workload of RPA robot development has become a pressing issue. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide an RPA robot production method and medium based on software screenshots, which has the characteristics of reducing the workload of RPA robot production and reducing labor costs.

[0005] In a first aspect, an embodiment provides a method for creating an RPA robot based on software screenshots, including:

[0006] Enter the operation interface of the software system, automatically operate the operation interface according to the preset operation mode, take and save a screenshot of the interface after each operation, and record the operation coordinate position corresponding to the screenshot after each operation;

[0007] For the saved screenshots, identify the screenshots with the same interface and keep one of them, and obtain the screenshots with different interfaces;

[0008] Based on the retained screenshots, obtain the preset fields, and obtain the corresponding relationship between the screenshots containing any preset field and the preset field;

[0009] Based on the corresponding relationship and the page where the preset field is located, the operation coordinates of the operation to be performed are obtained, and an RPA robot is created to take a screenshot of the page where the preset field is located.

[0010] In one embodiment, the automatic operation of the operation interface according to a preset operation mode includes:

[0011] Perform coordinate position operations one by one according to the preset coordinate position sequence.

[0012] In one embodiment, the coordinate position operation is performed according to a preset coordinate position sequence, including:

[0013] According to the preset coordinate position sequence, the coordinate position is moved 3px each time, and the coordinate position operation is performed one by one.

[0014] In one embodiment, the automatic operation of the operation interface according to a preset operation mode includes:

[0015] Perform automatic operations according to the preset operation type and record the operation type corresponding to the screenshot after each operation, which includes single-click operation and double-click operation;

[0016] The step of obtaining the operation coordinates of the operation to be performed based on the corresponding relationship and the page where the preset field is located, and creating an RPA robot that takes a screenshot of the page where the preset field is located, includes:

[0017] Based on the corresponding relationship and the page where the preset field is located, the operation coordinates and operation type of the operation to be performed are obtained, and an RPA robot is created to take a screenshot of the page where the preset field is located.

[0018] In one embodiment, the coordinate position operation is performed according to a preset coordinate position sequence, including:

[0019] According to the preset coordinate position sequence, the operations of each coordinate position are performed on all operation types.

[0020] In one embodiment, the step of identifying the saved screenshots with the same interface and retaining one of the screenshots to obtain the retained screenshots with different interfaces includes:

[0021] Enter the operation interface of the software system, and save the screenshot before executing the first automatic operation. For the screenshots after any operation, compare them with the saved screenshots, and determine whether there is a screenshot in the saved screenshot that is the same as the screenshot after any operation. If so, delete the screenshot after any operation. If not, save the screenshot after any operation, so as to obtain a different screenshot of the retained interface.

[0022] In one embodiment, the step of identifying the saved screenshots with the same interface and retaining one of the screenshots to obtain the retained screenshots with different interfaces includes:

[0023] Enter the operation interface of the software system, take a screenshot and save it before executing the first automatic operation, and save the screenshots after each operation in chronological order;

[0024] Based on all the saved screenshots, for any current screenshot, determine whether there is a screenshot in the previous screenshots that is the same as the current screenshot. If so, delete the current screenshot to obtain a screenshot with a different interface.

[0025] In one embodiment, the formats in which the screenshots are saved may include:

[0026] NO_x_y.jpg;

[0027] Among them, NO represents the serial number of the screenshot, starting from 1 and increasing by 1; x and y represent the x-axis coordinate and y-axis coordinate of the operation coordinate position respectively.

[0028] In one embodiment, the automatic operation of the operation interface according to a preset operation mode includes:

[0029] Perform automatic operations according to the preset operation type and record the operation type corresponding to the screenshot after each operation, which includes single-click operation and double-click operation;

[0030] The formats in which screenshots can be saved include:

[0031] NO_x_y_click.jpg;

[0032] Among them, NO represents the sequence number of the screenshot, starting from 1 and increasing by 1; x and y represent the x-coordinate and y-coordinate of the operation coordinate position respectively; click represents the operation type.

[0033] In a second aspect, an embodiment provides a computer-readable storage medium, wherein the medium stores a program, and the program can be loaded by a processor to execute the RPA robot production method described in any one of the above embodiments.

[0034] The beneficial effects of the present invention are:

[0035] Since the operation interface is automatically operated according to the preset operation method, a screenshot of the interface after each operation is taken and saved, and the operation coordinate position corresponding to the screenshot after each operation is recorded, so that the screenshot that can be obtained for each operation coordinate position can be obtained. Since for the saved screenshots, the screenshots with the same interface are identified and one of the screenshots is retained, different screenshots of the retained interface are obtained; based on the retained screenshots, the preset fields are obtained, and the correspondence between the screenshots with any preset field and the preset field is obtained; so that the screenshot interface where the preset field to be extracted is located can be found. Based on the above correspondence and the page where the preset field is located, the operation coordinates of the operation to be performed are obtained, and an RPA robot is produced to take a screenshot of the page where the preset field is located. In this way, the coordinate position required for each operation to obtain the screenshot interface can be obtained based on the screenshot interface where the preset field is located. In this way, an RPA robot can be produced to extract the coordinate position required for the operation of the preset field, and a screenshot can be taken based on the coordinate position to obtain a screenshot of the extracted preset field, thereby realizing the automatic production of the RPA robot that extracts the screenshot required for the preset field, reducing the workload of RPA robot production and greatly reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of an RPA robot production method based on software screenshots in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0038] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0039] The serial numbers assigned to the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0040] To facilitate the description of the inventive concept of this application, the RPA robot production technology is briefly described below.

[0041] Currently, data extraction at stroke centers requires manual data extraction for each version of the data source software. The stroke center requires a total of 357 fields, and the data for these fields comes from various systems. RPA is needed to create a robot for collecting and processing each field. The information systems used by different hospitals are likely provided by different information companies, which may also have multiple versions of the same system. With over a hundred HIS software companies nationwide, locating the specific software and interface for a field is extremely labor-intensive. If RPA could also automatically locate the software system and interface where a field resides, labor costs would be significantly reduced.

[0042] During their research, the applicant discovered that if RPA is to automatically locate the software system and interface where a field is located, first finding a screenshot of the software interface where the field is located and then automatically locating it from the screenshot is a feasible option. However, in order to further improve work efficiency and reduce the workload of RPA robot production, realizing automatic screenshots and producing robots that can take screenshots of the corresponding pages has become an urgent problem to be solved.

[0043] In view of this, the embodiment of the present application provides an RPA robot production method and medium based on software screenshots. In the operation interface of the entered software system, the operation interface is automatically operated according to the preset operation method, the interface after each operation is screenshoted and saved, and the operation coordinate position corresponding to the screenshot after each operation is recorded. In this way, the screenshot that can be obtained at each operation coordinate position can be obtained. For the saved screenshots, the screenshots with the same interface are identified and one of the screenshots is retained to obtain screenshots with different interfaces; based on the retained screenshots, the preset fields are obtained, and the corresponding relationship between the screenshots with any preset field and the preset fields is obtained; in this way, the screenshot interface where the preset field to be extracted is located can be found. Based on the above correspondence and the page where the preset field is located, the operation coordinates of the operation to be performed are obtained, and an RPA robot is made to take a screenshot of the page where the preset field is located. In this way, the coordinate position of the operation required each time the screenshot interface is obtained can be obtained based on the screenshot interface where the preset field is located. In this way, an RPA robot can be made to extract the coordinate position required for the operation of the preset field, and take a screenshot based on the coordinate position to obtain a screenshot of the extracted preset field, thereby realizing the automatic production of the RPA robot that extracts the screenshot required for the preset field, reducing the workload of RPA robot production and greatly reducing labor costs.

[0044] The RPA robot production method based on software screenshots of an embodiment of the present application can be applied to a computer system, which includes a processor and a computer storage medium, and at least one processor is used to load and execute a program of the RPA robot production method based on software screenshots and perform processing.

[0045] For an RPA robot production method based on software screenshots in this application embodiment, please refer to Figure 1 ,include:

[0046] Step S10, entering the operation interface of the software system, automatically operating the operation interface according to the preset operation mode, taking a screenshot of the interface after each operation and saving it, and recording the operation coordinate position corresponding to the screenshot after each operation.

[0047] During their research, the applicant discovered that, in order to improve work efficiency and reduce the workload for RPA robot development, the traditional manual data extraction process was insufficient to meet the automated development requirements of RPA robots. To address this technical difficulty, a feasible solution could be to take a screenshot of the software interface where the field is located and automatically locate the field to be extracted from the screenshot. Therefore, an RPA robot capable of automatically creating software screenshots is needed.

[0048] After entering the software system's interface, different pages may appear at different operation locations. For example, some operations may result in a pop-up window, others in more detailed information, and others in a page update. These new pages may contain fields that need to be extracted. Therefore, we need to take screenshots of the new page after each operation to create an RPA robot that automatically takes screenshots. This requires both automated operations within the page and automatic screenshots of the page after the automated operation, enabling the RPA robot to automatically create the page.

[0049] In one embodiment, the operation interface is automatically operated according to a preset operation mode, including: performing coordinate position operations one by one according to a preset coordinate position sequence. That is, in response to the preset operation sequence and operation mode instructions, coordinate position operations are performed one by one according to the preset coordinate position sequence. For example, after entering a certain operation interface, automatic operation can be performed according to the preset coordinate position sequence from left to right and from top to bottom. In this way, after each operation of a coordinate position is performed, the operation of the next coordinate position is performed according to the preset coordinate position sequence, until all preset coordinate position operations are completed or all coordinate position operations of the entire page are completed. It can be understood by those skilled in the art that the coordinate position sequence here can also be other coordinate position sequences, and those skilled in the art can set it according to actual needs, all within the scope of protection of this application.

[0050] In one embodiment, performing the coordinate position operation according to a preset coordinate position sequence may further include: performing the coordinate position operation according to the preset coordinate position sequence, shifting the coordinate position by 3px each time. Those skilled in the art will appreciate that the size of the shift can be configured according to actual needs.

[0051] Operating the software interface to obtain a new interface may be a single-click operation, a double-click operation, or other operation types. To ensure the comprehensiveness of displaying the new interface, in one embodiment, automatic operations are performed according to preset operation types, and the operation type corresponding to the screenshot after each operation is recorded, where the operation types include single-click operations and double-click operations.

[0052] In one embodiment, if there are more than two preset operation types, when performing coordinate-by-coordinate position operations, after performing all preset operation types at each operation position, proceed to the next operation position to perform operations. For example, after performing a single-click operation and a double-click operation at the current operation position, proceed to the next coordinate position to perform a single-click operation and a double-click operation. Alternatively, after performing a coordinate-by-coordinate position operation of one operation type, perform a coordinate-by-coordinate operation of the next operation type. It will be understood by those skilled in the art that the operation of each coordinate position requires the execution of all operation types. The specific operation method can be set as needed and is within the scope of protection of this application.

[0053] In one embodiment, after entering the operation interface of the software system, before performing automatic operation, a page screenshot is first taken.

[0054] By taking and saving a screenshot of the interface after each operation and recording the corresponding operation coordinates for each screenshot, you can determine the coordinates of the operation required to obtain the screenshot. If you subsequently build an RPA robot, you can determine the coordinates of the operation that needs to be automatically executed to obtain the target screenshot. If there are multiple operation types, since the operation type corresponding to the screenshot after each operation is recorded, you can determine the type of operation that needs to be automatically executed to obtain the target screenshot. This allows you to determine the coordinates and type of operation required to automatically execute the operation to obtain the target screenshot.

[0055] For example, after clicking the operation coordinate position (x, y) of the current software page, a screenshot M is obtained, and the disease type field to be extracted can be extracted from the screenshot M. If the screenshot M is used as the target screenshot to create an RPA robot, since the specific operation coordinate position and operation type corresponding to the screenshot are known, the RPA robot that obtains the target screenshot can be automatically created based on the operation coordinate position and operation type corresponding to the screenshot M, thereby realizing the automatic creation of the RPA robot.

[0056] Step S20: For the saved screenshots, identify the screenshots with the same interface and retain one of them, thereby obtaining the retained screenshots with different interfaces.

[0057] For a software interface, since there is a screenshot at each preset coordinate position, if there are 1000 preset coordinate positions, there will be 1000 screenshots. If there are two types of operations at each coordinate position, there will be 2000 screenshots. The applicant found in the research that there are a large number of duplicate screenshots among the 2000 screenshots, that is, screenshots with the same interface. For example, it was found that the interfaces of 300 of the 2000 screenshots are the same, but we only need to keep one of the screenshots to obtain information. Therefore, in order to improve recognition efficiency and reduce computer resource usage, in one embodiment, step S20 may include: entering the operation interface of the software system, and saving the screenshot before executing the first automatic operation. For the screenshots after any operation, they are compared with the saved screenshots, and it is determined whether there is a screenshot identical to the screenshot after any operation in the saved screenshots. If so, the screenshot after any operation is deleted. If not, the screenshot after any operation is saved, thereby obtaining screenshots of different interfaces.

[0058] Based on the above embodiment, enter the operating interface of the software system, and before executing the first automatic operation, take a screenshot and save it to obtain the first screenshot. For any screenshot after an operation, it is compared with the saved screenshot. For example, if a screenshot is obtained after the first operation, it can be compared with the first screenshot. Determine whether there is a screenshot in the saved screenshot that is the same as the screenshot after the any operation. If so, delete the screenshot after the any operation. If not, save the screenshot after the any operation. If the screenshot obtained after the first operation is the same as the first screenshot, delete the screenshot obtained after the first operation to avoid duplication of screenshots. If the screenshot obtained after the first operation is different from the first screenshot, save the screenshot obtained after the first operation. And so on, all screenshots with different interfaces are retained. In this embodiment, each time a screenshot is obtained, it is compared with the previously saved screenshot. If they are the same, delete it, which can reduce the storage space occupied.

[0059] In one embodiment, as an alternative to the above embodiment, step S20 may include: entering the operating interface of the software system, saving a screenshot before executing the first automatic operation, and saving the screenshots after each operation in chronological order; based on all the saved screenshots, for any current screenshot, determining whether there is a screenshot in the previous screenshots that is the same as any current screenshot, and if so, deleting any current screenshot, thereby obtaining a screenshot of a different interface that is retained.

[0060] Through any one of the embodiments of the above step S20, duplicate images are cleared to ensure that each interface has only one image.

[0061] In one embodiment, when there is only one type of operation at each coordinate position, the screenshot saving format can be expressed as:

[0062] NO_x_y.jpg;

[0063] Where NO represents the screenshot number, starting at 1 and increasing by 1. x and y represent the x-axis and y-axis coordinates of the operation position, respectively. For example, if a screenshot is saved in the format of 102_15_603.jpg, it means that the screenshot number is 102 and the operation position coordinates are (15, 603).

[0064] In the case where there are multiple types of operations at each coordinate position, in one embodiment, the format in which the screenshot is saved can be expressed as:

[0065] NO_x_y_click.jpg;

[0066] Here, "NO" represents the screenshot's sequence number, starting at 1 and incrementing by 1; "x" and "y" represent the x and y coordinates of the operation coordinate position, respectively; and "click" represents the operation type. For example, if a screenshot can be saved in the format "102_15_603_1.jpg," this indicates that the screenshot's sequence number is 102, the operation coordinate position is (15, 603), and the operation type is 1. If "operation type 1" represents a single-click operation, then the operation type is a single-click operation. Therefore, if a screenshot in the format "102_15_603_1.jpg" is the target image from which the preset fields need to be extracted, the RPA robot that generated the screenshot can be determined based on the screenshot's format information. Specifically, the RPA robot's information can include capturing a screenshot after a single-click operation at the coordinate position (15, 603) and saving it to the preset location.

[0067] Step S30: Based on the retained screenshots, the preset fields are obtained, and the corresponding relationship between the screenshots containing any preset field and the preset field is obtained.

[0068] Taking one of the preset fields as "Department Name" as an example, OCR can be used to identify screenshots with "Department Name". In this way, the correspondence between the preset field and the screenshot can be obtained, and the screenshots without the preset field can be deleted, leaving only the screenshots with the preset field.

[0069] Step S40: Based on the corresponding relationship and the page where the preset field is located, the operation coordinates of the operation to be performed are obtained, and an RPA robot is created to take a screenshot of the page where the preset field is located.

[0070] By knowing the correspondence between screenshots and pre-set fields, the screenshot required to access the pre-set "Department Name" field can be determined. In the above embodiment, since the operation coordinates corresponding to the screenshots are recorded, an RPA robot can be created based on the relationship between the screenshots and the operation coordinates to obtain the screenshots of the "Department Name" field.

[0071] In one embodiment, when there are multiple operation types, the operation coordinates and operation type of the operation to be performed are obtained based on the corresponding relationship and the page where the preset field is located, and an RPA robot is created to take a screenshot of the page where the preset field is located.

[0072] In one embodiment of the present application, a computer-readable storage medium is provided. A program is stored on the storage medium. The stored program includes a method that can be loaded by a processor and process any of the above embodiments.

[0073] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0074] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A method for creating an RPA robot based on software screenshots, characterized in that: include: Enter the operation interface of the software system and automatically operate the operation interface according to the preset operation mode, including: performing coordinate position operations one by one according to the preset coordinate position sequence, and recording the operation type corresponding to the screenshot after each operation, the operation type including single-click operation and double-click operation; taking and saving a screenshot of the interface after each operation, and recording the operation coordinate position corresponding to the screenshot after each operation; For the saved screenshots, identify the screenshots with the same interface and keep one of them, and obtain the screenshots with different interfaces; Based on the retained screenshots, obtain the preset fields, and obtain the corresponding relationship between the screenshots containing any preset field and the preset field; Based on the corresponding relationship and the page where the preset field is located, the operation coordinates and operation type of the operation to be performed are obtained, and an RPA robot is created to take a screenshot of the page where the preset field is located.

2. The RPA robot production method according to claim 1, characterized in that: The coordinate position operation is performed according to the preset coordinate position sequence, including: According to the preset coordinate position sequence, the coordinate position is moved 3px each time, and the coordinate position operation is performed one by one.

3. The RPA robot production method according to claim 1, characterized in that: The coordinate position operation is performed according to the preset coordinate position sequence, including: According to the preset coordinate position sequence, the operations of each coordinate position are performed on all operation types.

4. The RPA robot production method according to claim 1, characterized in that: The method of identifying the screenshots with the same interface and retaining one of the screenshots to obtain the retained screenshots with different interfaces includes: Enter the operation interface of the software system, and save the screenshot before executing the first automatic operation. For the screenshots after any operation, compare them with the saved screenshots, and determine whether there is a screenshot in the saved screenshot that is the same as the screenshot after any operation. If so, delete the screenshot after any operation. If not, save the screenshot after any operation, so as to obtain a different screenshot of the retained interface.

5. The RPA robot production method according to claim 1, characterized in that: The method of identifying the screenshots with the same interface and retaining one of the screenshots to obtain the retained screenshots with different interfaces includes: Enter the operation interface of the software system, take a screenshot and save it before executing the first automatic operation, and save the screenshots after each operation in chronological order; Based on all the saved screenshots, for any current screenshot, determine whether there is a screenshot in the previous screenshots that is the same as the current screenshot. If so, delete the current screenshot to obtain a screenshot with a different interface.

6. The RPA robot production method according to claim 1, characterized in that: The formats in which screenshots can be saved include: NO_x_y.jpg; Among them, NO represents the serial number of the screenshot, starting from 1 and increasing by 1; x and y represent the x-axis coordinate and y-axis coordinate of the operation coordinate position respectively.

7. The RPA robot production method according to claim 1, characterized in that: The automatic operation of the operation interface according to the preset operation mode includes: Perform automatic operations according to the preset operation type and record the operation type corresponding to the screenshot after each operation, which includes single-click operation and double-click operation; The formats in which screenshots can be saved include: NO_x_y_click.jpg; Among them, NO represents the sequence number of the screenshot, starting from 1 and increasing by 1; x and y represent the x-coordinate and y-coordinate of the operation coordinate position respectively; click represents the operation type.

8. A computer-readable storage medium, characterized in that The medium stores a program, which can be loaded by a processor and execute the RPA robot production method according to any one of claims 1 to 7.

Images