Software robot adjusting system suitable for self-adaptive upgrading

Abstract

The invention relates to the technical field of software robots, in particular to a software robot adjusting system suitable for self-adaptive upgrading, which is provided with a storage module, a robot module, a verification module, a change processing module and an adaptation module, the verification module is used for calling a current operation interface and a corresponding component triggering interface to carry out similarity matching, the variation processing module is used for determining coordinates of a difference region, and the adaptation module is used for judging and calling the variation processing module, determining a relative position relation between the difference region and an operation component required to be triggered in the next step and the area of the difference region; whether the robot module is controlled to execute a process or not and whether triggering verification is executed or not are judged; or controlling the robot module to execute the process. According to the method, the current operation interface is comprehensively matched with the corresponding component triggering interface, so that the current operation interface can be self-adjusted, and the reliability of executing the preset process by the robot module is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software robots, and in particular to a software robot adjustment system suitable for adaptive upgrading. Background Art

[0002] With the rapid development of new technology waves, the industrial and corporate office models are shifting to new models of informatization and digitization. Software robots have emerged. Their typical characteristics are based on fixed software operation paths, simulating human behavior to perform complex and repetitive manual operations, thereby assisting or replacing manual behavior. Software robots can run uninterruptedly around the clock, effectively reducing human errors in previous office operations, greatly improving current office efficiency, effectively reducing corporate costs, and freeing employees from large, repetitive, mechanical and rule-based labor, realizing process automation and process control, improving work efficiency and reducing costs.

[0003] China Patent Publication No.: CN111709701A, discloses a robot office system, including an activity control module, a main window module, a function control module and a property editing module. The system is based on the .net technology development platform and python automated office technology; the visual design module in the system can completely record the process of simulating manual operation in the main window module by dragging and dropping; at the same time, different variables and parameters can be defined in the main window module to declare different logic and algorithm processes; after the complete process is developed, the process can be executed 24 hours a day to the maximum extent according to the needs of actual business volume; the invention can realize automated computer operation business processes, and complete the visual design and development of different business scenarios, business needs, operation processes, algorithms and logics through the development program set by the customer, so that the computer operations that originally needed to be completed manually can be realized in office automation.

[0004] Chinese patent publication number: CN113641569A, discloses a robot process automation method, including using RPA technology to record the user's basic operations and generate test cases, running the test cases to play back in the information system, and converting the test cases into automated scripts; simulating the information system through automated scripts to obtain web page data; inspecting the above simulated operations through manual judgment, judging whether the information system is running normally through the inspection results, and then comparing and judging the image features in the browser page with the image features in the database through image similarity comparison and difference marking methods; simulating the manual judgment method through machine learning algorithms to obtain configuration recommendations for the judgment results; after repeated training and testing, finally establishing an accurate operation rule library and judgment rule library. This method sets RPA operations for daily affairs, optimizes the basic processes of the enterprise, collects and extracts information to generate user-required forms, and automatically enters the business system.

[0005] However, there are still the following problems in the prior art:

[0006] In actual situations, most office software robots run according to processes based on existing operating paths and do not have the ability to independently judge and handle emergencies. If an abnormality occurs in the system and causes the current operating interface to change, the office software robot will not be able to continue to execute according to the preset process, resulting in reduced work efficiency or even forced suspension of work. Summary of the invention

[0007] To this end, the present invention provides a software robot adjustment system suitable for adaptive upgrading, which is used to solve the problem that most office software robots in the prior art run according to the process based on the existing operation path and have no ability to judge and handle emergencies by themselves. If an abnormality occurs in the system and causes the current operation interface to change, the office software robot will not be able to continue to execute according to the preset process, resulting in reduced work efficiency or even forced suspension of work.

[0008] To achieve the above object, the present invention provides a software robot adjustment system suitable for adaptive upgrading, which includes:

[0009] A storage module, which is used to store preset component triggering processes and several component triggering interfaces;

[0010] A robot module connected to the storage module, used to generate operation instructions based on the component triggering process and perform corresponding operations, including triggering operation components and inputting text;

[0011] A verification module, which is connected to the storage module and the robot module respectively, and is used to respond to the robot module performing an operation, call the current operation interface and the corresponding component trigger interface for similarity matching, and obtain a matching result;

[0012] A mutation processing module, which is connected to the verification module, is used to determine the coordinates of the difference area, call the current operation interface and the corresponding component trigger interface to perform feature elimination processing, and then send it to the verification module to obtain the matching result again;

[0013] An adaptation module, which is connected to the verification module and the abnormality processing module respectively, and is used to control the robot module based on the matching result of the verification module, including:

[0014] Calling the abnormality processing module to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification;

[0015] Or, control the robot module execution process;

[0016] The trigger verification includes generating a verification control instruction based on the difference area coordinates, and controlling the robot module to execute the verification control instruction to verify whether the robot module execution process is controlled, and the verification control instruction includes a number of instruction trigger coordinates.

[0017] Furthermore, the verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching, including:

[0018] To determine the image similarity between the current operation interface and the corresponding component trigger interface;

[0019] for extracting a first keyword of the current operation interface and a second keyword of the corresponding component triggering interface;

[0020] To determine the keyword similarity between the first keyword and the second keyword;

[0021] It is used to determine the ratio of image similarity to the reference image similarity as the similarity influencing factor;

[0022] To determine the ratio of keyword similarity to the benchmark keyword similarity as the keyword impact factor;

[0023] The weighted sum of the similarity impact factor and the keyword impact factor is used to determine the similarity matching coefficient.

[0024] Furthermore, the verification module is used to obtain a matching result, wherein:

[0025] If the similarity matching coefficient is greater than or equal to the coefficient threshold, determining the matching result is a match;

[0026] If the similarity matching coefficient is less than the coefficient threshold, the matching result is determined to be a mismatch.

[0027] Furthermore, the variation processing module is used to determine the coordinates of the difference area, including:

[0028] Used to split the current operation interface and the component trigger interface in the same coordinate system to compare the image similarity between the current local operation interface and the corresponding local component trigger interface;

[0029] to identify the difference area based on the comparison result, so as to determine the coordinate value of each pixel point in the difference area, and obtain the coordinates of the difference area;

[0030] If the image similarity is less than a predetermined difference region image similarity threshold, the corresponding local operation interface is determined to be a difference region.

[0031] Furthermore, the mutation processing module performs feature elimination, including:

[0032] Determining the difference area contours corresponding to the current operation interface and the component triggering interface based on the difference area coordinates;

[0033] It is used to remove the image features within the contour of the difference area.

[0034] Further, the adaptation module controls the robot module based on the matching result of the verification module, including:

[0035] If the matching result of the verification module is a match, the robot module is controlled to execute the process;

[0036] If the matching result of the verification module is mismatch, the mutation processing module is called to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area to determine whether to control the robot module to execute the process and whether to perform trigger verification.

[0037] Furthermore, the adaptation module determines the relative position relationship between the difference area and the operation component to be triggered in the next step, including:

[0038] Used to determine the coordinates of the operating component to be triggered based on the component trigger interface;

[0039] Comparing the coordinates with the coordinates of the difference area to obtain a relative position relationship;

[0040] The relative position relationship includes that the operating component is located outside the difference area and the operating component is located inside the difference area.

[0041] Furthermore, the adaptation module determines whether to control the robot module to execute the process and whether to perform trigger verification, including:

[0042] To determine the area of ​​the difference area;

[0043] If the current operation interface meets the process execution conditions, the adaptation module determines and controls the robot module to execute the process;

[0044] If the current operation interface does not meet the process execution conditions, the adaptation module determines to control the robot to perform trigger verification;

[0045] The process execution condition includes that the operating component is located outside the difference area and the area of ​​the difference area is smaller than a predetermined area.

[0046] Furthermore, the adaptation module generates a verification control instruction based on the difference area coordinates, including:

[0047] for identifying target features in the difference area, determining coordinates of the target features, and determining the coordinates as instruction triggering coordinates;

[0048] for identifying edge coordinates of the difference area, randomly generating a predetermined number of coordinates outside the edge coordinates, and determining the coordinates as instruction triggering coordinates;

[0049] The target features include keywords and icons.

[0050] Further, the adaptation module verifies whether the robot module is controlled to execute the process, including:

[0051] To obtain the current operation interface after executing the trigger verification, and call the verification module to obtain the matching result for the current operation interface;

[0052] If the matching result is a match, the robot module is controlled to execute the process;

[0053] If the matching result is no match, the robot module execution process will not be controlled.

[0054] Compared with the prior art, the present invention is provided with a storage module, a robot module, a verification module, a mutation processing module and an adaptation module, wherein the robot module is used to generate an operation instruction and perform a corresponding operation, the verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching and obtain a matching result, the mutation processing module is used to determine the coordinates of the difference area to obtain the matching result again, and the adaptation module is used to determine whether to call the mutation processing module, determine the relative position relationship between the difference area and the operation component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform the trigger verification; or, control the robot module to execute the process. The present invention can perform self-adjustment by fully matching the current operation interface with the corresponding component trigger interface, thereby improving work efficiency and work accuracy.

[0055] In particular, the current operation interface is called to perform similarity matching with the corresponding component trigger interface, and the matching degree is calculated based on keywords and images to provide a data basis for determining whether to call the anomaly processing module to eliminate features. In actual working conditions, the robot module triggers the operation component according to the operation instructions, but if the operation is wrong or unknown content appears in the current operation interface, such as the operation is other steps, or an unknown pop-up window or error interface appears, the current operation interface is partially covered or completely covered, making the operation steps discontinuous and making the robot module unable to continue to work according to the preset steps. Based on this, consider performing similarity matching on the current operation interface and the corresponding component trigger interface to facilitate subsequent timely intervention in the execution of the robot module operation and improve the stability and reliability of the robot module operation.

[0056] In particular, for the unmatched interface, the mutation processing module is called to determine the coordinates of the difference area for feature elimination. When working according to the operation process and encountering an overlay window, if the current operation interface is continued to be operated based on the corresponding component trigger interface, it will lead to operation errors and even cause the interface to change, so that the process is terminated. Based on this, consider determining the feature area of ​​the current operation interface and eliminating the feature area that interferes with the current window, so as to provide a data basis for determining whether to control the execution process of the robot module and whether to perform trigger verification, so that the robot module can continue to work according to the preset steps in the event of an emergency, and can perform self-adjustment, thereby improving work efficiency and accuracy.

[0057] In particular, the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area are determined to determine whether to control the robot module to execute the process and whether to perform trigger verification. In the actual operation process, the difference area may cover the component. At this time, it is necessary to verify the difference area. The coverage of the current operation interface is determined by determining the relative position relationship and the area of ​​the difference area. The current operation interface with the covered component is verified so that the robot module can continue to work according to the preset steps and can perform self-adjustment, thereby improving work efficiency and work accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of the structure of a software robot adjustment system suitable for adaptive upgrading according to an embodiment of the invention;

[0059] Figure 2 A logic block diagram of a verification module of an embodiment of the invention for obtaining a matching result;

[0060] Figure 3 A logic block diagram of an adaptation module of an embodiment of the invention controlling a robot module based on a matching result of the verification module;

[0061] Figure 4 This is a logic block diagram of an adaptation module according to an embodiment of the invention for determining whether to control the robot module to execute a process and whether to perform trigger verification. DETAILED DESCRIPTION

[0062] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0064] It should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0065] See also Figure 1-Figure 4 As shown, Figure 1 The structure diagram of the software robot adjustment system suitable for adaptive upgrading according to the embodiment of the invention is shown in FIG. Figure 2 A logic block diagram of a verification module used to obtain a matching result according to an embodiment of the invention. Figure 3 The logic block diagram of the adaptation module of the embodiment of the invention controlling the robot module based on the matching result of the verification module is as follows: Figure 4 The adaptive module of the embodiment of the invention determines whether to control the robot module to execute the process and whether to execute the trigger verification. The software robot adjustment system suitable for adaptive upgrading of the present invention comprises:

[0066] The storage module is used to store preset component triggering processes and several component triggering interfaces. It can be understood that the component triggering interface is the interface to which any component in the operating system jumps after being triggered, which will not be described in detail.

[0067] A robot module connected to the storage module, used to generate operation instructions based on the component triggering process and perform corresponding operations, including triggering operation components and inputting text;

[0068] A verification module is respectively connected to the storage module and the robot module, and is used for responding to the robot module performing operations, calling the current operation interface and the corresponding component trigger interface for similarity matching, and obtaining a matching result; it can be understood that after the robot module performs the corresponding operation based on the operation instruction, it will generate the corresponding current operation interface, and the verification module responds to the robot module performing operations, that is, after each operation is performed, the verification module will call the corresponding current operation interface for verification to ensure the accuracy of the operation and improve the adaptive upgrade capability.

[0069] A mutation processing module, which is connected to the verification module, is used to determine the coordinates of the difference area, call the current operation interface and the corresponding component trigger interface to perform feature elimination processing, and then send it to the verification module to obtain the matching result again;

[0070] An adaptation module, which is connected to the verification module and the abnormality processing module respectively, and is used to control the robot module based on the matching result of the verification module, including:

[0071] Calling the abnormality processing module to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification;

[0072] Or, control the robot module execution process;

[0073] The trigger verification includes generating a verification control instruction based on the difference area coordinates, and controlling the robot module to execute the verification control instruction to verify whether the robot module execution process is controlled, and the verification control instruction includes a number of instruction trigger coordinates.

[0074] Specifically, there is no limitation on the source of the preset component trigger process. For example, the manual operation steps can be recorded to construct and record the corresponding component trigger process. Technical personnel in this field can construct it according to actual needs. It is only necessary to ensure that the process is continuous and uninterrupted and the operation can be completed. This will not be repeated.

[0075] It can be understood that the component triggering process is the timing relationship of the triggering of the operation components. For example, taking the example that the entire workflow needs to trigger five operation components, the component triggering process includes the triggering sequence of the five operation components, so that the robot module can trigger the operation components in a predetermined order.

[0076] Specifically, the operation component is an interactive component in the interface, and the corresponding function is realized by triggering the corresponding operation component, such as icons, buttons, etc., which will not be repeated here.

[0077] It is understandable that, in implementation, the component triggering interface and the current operation interface are image data, and both can be obtained in the form of screenshots.

[0078] Specifically, there is no limitation on the specific structures of the storage module, robot module, verification module, mutation processing module and adaptation module, which can be composed of logical components, including field programmable processors, computers or microprocessors in computers.

[0079] Specifically, there is no limitation on the specific way in which the robot module generates operation instructions.

[0080] In some possible implementations, the component triggering process may include the coordinates of the corresponding component, that is, the robot module may generate a click instruction based on the coordinates of the corresponding component, control the cursor to click the corresponding coordinates, and realize the function of triggering the operation component.

[0081] In some possible implementations, the robot module can identify the coordinates of the currently required triggering operation component based on the image processing algorithm, and then control the cursor to click the corresponding coordinates to realize the function of the triggering operation component.

[0082] Specifically, the verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching, including:

[0083] To determine the image similarity between the current operation interface and the corresponding component trigger interface;

[0084] for extracting a first keyword of the current operation interface and a second keyword of the corresponding component triggering interface;

[0085] To determine the keyword similarity between the first keyword and the second keyword;

[0086] It is used to determine the ratio of image similarity to the reference image similarity as the similarity influencing factor;

[0087] To determine the ratio of keyword similarity to the benchmark keyword similarity as the keyword impact factor;

[0088] The weighted sum of the similarity impact factor and the keyword impact factor is used to determine the similarity matching coefficient.

[0089] Specifically, there is no limitation on the method for calculating image similarity. For example, the method of calculating image cosine similarity (Cosine Simi l ar ity) can be used to represent the current operation interface as a vector, and the similarity of the two images is characterized by calculating the cosine distance between the vectors. Of course, other methods can also be used, which will not be repeated here.

[0090] Specifically, there is no limitation on the method of obtaining the keywords. For example, OCR technology can be used to recognize and convert the text content in the current operation interface into an editable text format, and then extract keywords from the text. Of course, other methods can also be used, which will not be repeated here.

[0091] Specifically, there is no limitation on the method for calculating the keyword similarity. For example, the Word2Vec word embedding model can be used to capture the semantic relationship between keywords through the vector representation of keywords, and the similarity between keywords can be measured by calculating the cosine similarity between two keyword vectors;

[0092] Of course, other methods may also be used. Those skilled in the art only need to ensure that the keyword similarity can be calculated, which will not be described in detail.

[0093] Specifically, considering the personalized customization of the system, the operation interfaces triggered by the same operation component may be different, and a certain difference is allowed, so the benchmark image similarity is selected in the interval [0.85, 1].

[0094] Specifically, considering system updates or UI changes, the keywords corresponding to the operation components may be replaced by synonyms, so the benchmark keyword similarity is selected in the interval [0.95, 1].

[0095] Specifically, the weight coefficient of the similarity impact factor is 0.35, and the weight coefficient of the keyword impact factor is 0.65.

[0096] Specifically, the verification module is used to obtain matching results, where:

[0097] If the similarity matching coefficient is greater than or equal to the coefficient threshold, determining the matching result is a match;

[0098] If the similarity matching coefficient is less than the coefficient threshold, the matching result is determined to be a mismatch.

[0099] Specifically, the coefficient threshold is pre-calculated and selected within the interval (0.915, 1).

[0100] Specifically, the current operation interface is called to perform similarity matching with the corresponding component trigger interface, and the matching degree is calculated based on keywords and images to provide a data basis for determining whether to call the anomaly processing module to eliminate features. In actual working conditions, the robot module triggers the operation component according to the operation instructions, but if the operation is wrong or unknown content appears in the current operation interface, such as the operation is other steps, or an unknown pop-up window or error interface appears, the current operation interface is partially covered or completely covered, making the operation steps discontinuous and making the robot module unable to continue to work according to the preset steps. Based on this, consider performing similarity matching on the current operation interface and the corresponding component trigger interface to facilitate timely intervention in the execution of the robot module operation and improve the stability and reliability of the robot module.

[0101] Specifically, the mutation processing module is used to determine the coordinates of the difference area, including:

[0102] It is used to split the current operation interface and the component trigger interface in the same coordinate system to compare the image similarity between the current local operation interface and the corresponding local component trigger interface. It can be understood that the local operation interface is obtained after the current operation interface is split, and the corresponding local component trigger interface is obtained after the component trigger interface is split;

[0103] to identify the difference area based on the comparison result, so as to determine the coordinate value of each pixel point in the difference area, and obtain the coordinates of the difference area;

[0104] If the image similarity is less than a predetermined difference region image similarity threshold, the corresponding local operation interface is determined to be a difference region.

[0105] It can be understood that a pixel is the basic unit of a digital image, which represents a minimum point in the image. Each pixel has a specific coordinate position in the image, usually expressed in the form of rows and columns. In this implementation, the coordinates of the pixel are determined as the coordinates of the difference area.

[0106] Specifically, there is no limit on the specific number of parts to be split, as long as it can ensure that the difference areas can be identified. Technical personnel in this field can determine it based on the computing power of the logical components, which will not be elaborated here.

[0107] It is understandable that the method for calculating the image similarity of the difference region is consistent with the method for calculating the image similarity, which will not be described in detail.

[0108] Specifically, the difference region image similarity threshold is the same as the image similarity threshold.

[0109] Specifically, for the unmatched interface, the mutation processing module is called to determine the coordinates of the difference area for feature elimination. When working according to the operation process and encountering an overlay window, if the current operation interface is continued to be operated based on the corresponding component trigger interface, it will lead to operation errors and even cause the interface to change, so that the process is terminated. Based on this, consider determining the feature area of ​​the current operation interface and eliminating the feature area that interferes with the current window, so as to provide a data basis for determining whether to control the execution process of the robot module and whether to perform trigger verification, so that the robot module can continue to work according to the preset steps in the event of an emergency, and can perform self-adjustment, thereby improving work efficiency and accuracy.

[0110] Specifically, the mutation processing module performs feature elimination, including:

[0111] Used to determine the difference area contours corresponding to the current operation interface and the component trigger interface based on the difference area coordinates. It can be understood that the difference area coordinates include the coordinates of all pixel points in the difference area, and the edge coordinates can be identified to determine the difference area contour, which will not be repeated here.

[0112] It is used to remove the image features within the contour of the difference area.

[0113] It is understandable that, in order to facilitate subsequent comparison, the same image may be used for filling after the elimination is completed, which will not be described in detail.

[0114] Specifically, the adaptation module controls the robot module based on the matching result of the verification module, including:

[0115] If the matching result of the verification module is a match, the robot module is controlled to execute the process;

[0116] If the matching result of the verification module is mismatch, the mutation processing module is called to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area to determine whether to control the robot module to execute the process and whether to perform trigger verification.

[0117] Specifically, the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area are determined to determine whether to control the robot module to execute the process and whether to perform trigger verification. In the actual operation process, the difference area may cover the component. At this time, it is necessary to verify the difference area. The coverage of the current operation interface is determined by determining the relative position relationship and the area of ​​the difference area. The current operation interface with the covered component is verified so that the robot module can continue to work according to the preset steps and can perform self-adjustment, thereby improving work efficiency and work accuracy.

[0118] Specifically, the adaptation module determines the relative position relationship between the difference area and the operation component to be triggered in the next step, including:

[0119] It is used to determine the coordinates of the operating component to be triggered based on the component trigger interface. It can be understood that the component trigger interface includes the operating component, so the coordinates of the operating component can be determined in advance and recorded, which will not be repeated here.

[0120] Comparing the coordinates with the coordinates of the difference area to obtain a relative position relationship;

[0121] The relative position relationship includes that the operating component is located outside the difference area and the operating component is located inside the difference area.

[0122] Specifically, if the operation component is outside the difference area, it means that the difference area does not contain the operation component required to be triggered in the next step; if the operation component is inside the difference area, it means that the difference area contains the operation component required to be triggered in the next step, and the difference area needs to be analyzed and verified to make the operation process return to the preset component triggering process.

[0123] Specifically, the adaptation module determines whether to control the robot module to execute the process and whether to perform trigger verification, including:

[0124] To determine the area of ​​the difference area;

[0125] If the current operation interface meets the process execution conditions, the adaptation module determines and controls the robot module to execute the process;

[0126] If the current operation interface does not meet the process execution conditions, the adaptation module determines to control the robot to perform trigger verification;

[0127] The process execution condition includes that the operating component is located outside the difference area and the area of ​​the difference area is smaller than a predetermined area.

[0128] Specifically, the predetermined area is set to 0.5 times the current operation interface.

[0129] Specifically, the adaptation module generates a verification control instruction based on the difference area coordinates, including:

[0130] for identifying target features in the difference area, determining coordinates of the target features, and determining the coordinates as instruction triggering coordinates;

[0131] for identifying edge coordinates of the difference area, randomly generating a predetermined number of coordinates outside the edge coordinates, and determining the coordinates as instruction triggering coordinates;

[0132] The target features include keywords and icons.

[0133] Specifically, for some difference areas, the difference area can be cancelled by simply clicking anywhere outside the difference area; but for some difference areas, a specified button must be clicked to cancel the difference area.

[0134] It can be understood that the target features are keywords and icons that may potentially appear in the difference area, for example, the keyword "OK", the keyword "Cancel", the close icon, the minimize icon, etc., which will not be repeated here.

[0135] It is understandable that there is no limitation on the method of identifying icons, and it can be implemented using a pre-trained neural network image processing model. Of course, other methods can also be used, which will not be elaborated here.

[0136] Specifically, the adaptation module verifies whether the control robot module executes the process, including,

[0137] To obtain the current operation interface after executing the trigger verification, and call the verification module to obtain the matching result for the current operation interface;

[0138] If the matching result is a match, the robot module is controlled to execute the process;

[0139] If the matching result is no match, the robot module execution process will not be controlled.

[0140] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A software robot adjustment system suitable for adaptive upgrading, characterized in that: include: A storage module, which is used to store preset component triggering processes and several component triggering interfaces; A robot module connected to the storage module, used to generate operation instructions based on the component triggering process and perform corresponding operations, including triggering operation components and inputting text; A verification module, which is connected to the storage module and the robot module respectively, and is used to respond to the robot module performing an operation, call the current operation interface and the corresponding component trigger interface for similarity matching, and obtain a matching result; A mutation processing module, which is connected to the verification module, is used to determine the coordinates of the difference area, call the current operation interface and the corresponding component trigger interface to perform feature elimination processing, and then send it to the verification module to obtain the matching result again; An adaptation module, which is connected to the verification module and the abnormality processing module respectively, and is used to control the robot module based on the matching result of the verification module, including: Calling the abnormality processing module to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification; Or, control the robot module execution process; The trigger verification includes generating a verification control instruction based on the difference area coordinates, and controlling the robot module to execute the verification control instruction to verify whether the robot module execution process is controlled, and the verification control instruction includes a number of instruction trigger coordinates.

2. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching, including: To determine the image similarity between the current operation interface and the corresponding component trigger interface; for extracting a first keyword of the current operation interface and a second keyword of the corresponding component triggering interface; To determine the keyword similarity between the first keyword and the second keyword; It is used to determine the ratio of image similarity to the reference image similarity as the similarity influencing factor; To determine the ratio of keyword similarity to the benchmark keyword similarity as the keyword impact factor; The weighted sum of the similarity impact factor and the keyword impact factor is used to determine the similarity matching coefficient.

3. The software robot adjustment system suitable for adaptive upgrading according to claim 2, characterized in that: The verification module is used to obtain a matching result, wherein: If the similarity matching coefficient is greater than or equal to the coefficient threshold, determining the matching result is a match; If the similarity matching coefficient is less than the coefficient threshold, the matching result is determined to be a mismatch.

4. The software robot adjustment system suitable for adaptive upgrading according to claim 3, characterized in that: The mutation processing module is used to determine the coordinates of the difference area, including: Used to split the current operation interface and the component trigger interface in the same coordinate system to compare the image similarity between the current local operation interface and the corresponding local component trigger interface; to identify the difference area based on the comparison result, so as to determine the coordinate value of each pixel point in the difference area, and obtain the coordinates of the difference area; If the image similarity is less than a predetermined difference region image similarity threshold, the corresponding local operation interface is determined to be a difference region.

5. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The mutation processing module performs feature elimination. include, Determining the difference area contours corresponding to the current operation interface and the component triggering interface based on the difference area coordinates; It is used to remove the image features within the contour of the difference area.

6. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module controls the robot module based on the matching result of the verification module, including: If the matching result of the verification module is a match, the robot module is controlled to execute the process; If the matching result of the verification module is mismatch, the mutation processing module is called to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area to determine whether to control the robot module to execute the process and whether to perform trigger verification.

7. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module determines the relative position relationship between the difference area and the operation component to be triggered in the next step, including: Used to determine the coordinates of the operating component to be triggered based on the component trigger interface; Comparing the coordinates with the coordinates of the difference area to obtain a relative position relationship; The relative position relationship includes that the operating component is located outside the difference area and the operating component is located inside the difference area.

8. The software robot adjustment system suitable for adaptive upgrading according to claim 7, characterized in that: The adaptation module determines whether to control the robot module to execute the process and whether to perform trigger verification, including: To determine the area of ​​the difference area; If the current operation interface meets the process execution conditions, the adaptation module determines and controls the robot module to execute the process; If the current operation interface does not meet the process execution conditions, the adaptation module determines to control the robot to perform trigger verification; The process execution condition includes that the operating component is located outside the difference area and the area of ​​the difference area is smaller than a predetermined area.

9. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module generates a verification control instruction based on the difference area coordinates, including: for identifying target features in the difference area, determining coordinates of the target features, and determining the coordinates as instruction triggering coordinates; for identifying edge coordinates of the difference area, randomly generating a predetermined number of coordinates outside the edge coordinates, and determining the coordinates as instruction triggering coordinates; The target features include keywords and icons.

10. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module verifies whether the control robot module executes the process, include, To obtain the current operation interface after executing the trigger verification, and call the verification module to obtain the matching result for the current operation interface; If the matching result is a match, the robot module is controlled to execute the process; If the matching result is no match, the robot module execution process will not be controlled.

Images