A software robot adjustment system suitable for adaptive upgrading

Abstract

The present invention relates to the field of software robot technology, and in particular to a software robot adjustment system suitable for adaptive upgrading. The present invention provides a storage module, a robot module, a verification module, a mutation processing module, and an adaptation module. The robot module is used to generate operation instructions and perform corresponding operations. The verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching. The mutation processing module is used to determine the coordinates of the difference area. The adaptation module is used to determine whether to call the mutation processing module, determine the relative position relationship between the difference area and the operation component to be triggered in the next step, and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification; or, control the robot module to execute the process. The present invention comprehensively matches the current operation interface with the corresponding component trigger interface, enabling it to perform self-adjustment, thereby improving the reliability of the robot module in executing the preset process.

Description

Technical Field

[0001] The present invention relates to the technical field of software robots, and in particular to a software robot adjustment system suitable for adaptive upgrading. Background Art

[0002] With the rapid development of new technologies, and the shift in industrial and corporate office models towards information-based and digital models, software robots have emerged. Their defining characteristic is that they simulate human behavior based on fixed software operation paths to perform complex and repetitive manual operations, thereby assisting or replacing manual tasks. Software robots can operate 24 / 7, effectively reducing human errors in previous office operations, significantly improving current office efficiency and effectively reducing enterprise costs. They free employees from extensive, repetitive, mechanical, and rule-based labor, enabling process automation and control, improving work efficiency and reducing costs.

[0003] Chinese patent publication number CN111709701A discloses a robotic office system, comprising an activity control module, a main window module, a function control module, and a property editing module. The system is based on the .NET technology development platform and Python automated office technology. The visual design module in the system can completely record simulated manual operation processes in the main window module by dragging and dropping. Furthermore, different variables and parameters can be defined in the main window module to declare different logic and algorithmic processes. Once the complete process is developed, it can be executed 24 hours a day, depending on actual business volume. This invention can automate computer operation business processes. Through customer-defined development programs, it can complete the visual design and development of different business scenarios, business requirements, operational processes, algorithms, and logic, thus automating computer operations that previously required manual work.

[0004] Chinese patent publication number CN113641569A discloses a robotic process automation method, which includes using RPA technology to record basic user operations and generate test cases. The test cases are then replayed in an information system, converting the test cases into automated scripts. The automated scripts simulate the information system to obtain webpage data. The simulated operations are inspected manually, and the inspection results determine whether the information system is operating normally. Image features in the browser page are then compared with those in a database using image similarity comparison and difference marking. A machine learning algorithm simulates manual judgment to generate configuration recommendations based on the judgment results. Through repeated training and testing, an accurate operation rule library and judgment rule library are ultimately established. This method sets RPA operations for daily transactions, optimizes basic enterprise processes, collects and extracts information, generates user-required forms, and automatically enters them into the business system.

[0005] However, the prior art still has the following problems:

[0006] In actual situations, most office software robots run according to procedures based on existing operating paths and do not have the ability to independently judge and handle emergencies. If an abnormality occurs in the system and causes the current operating interface to change, the office software robot will not be able to continue to execute according to the preset process, resulting in reduced work efficiency or even forced suspension of work. Summary of the Invention

[0007] To this end, the present invention provides a software robot adjustment system suitable for adaptive upgrading, which is used to solve the problem in the prior art that most office software robots operate according to the existing operation path and process, and have no ability to judge and handle emergencies on their own. If an abnormality occurs in the system and the current operation interface changes, the office software robot will not be able to continue to execute according to the preset process, resulting in reduced work efficiency or even forced suspension of work.

[0008] To achieve the above objectives, the present invention provides a software robot adjustment system suitable for adaptive upgrading, which includes:

[0009] A storage module, which is used to store preset component triggering processes and several component triggering interfaces;

[0010] A robot module, connected to the storage module, configured to generate operation instructions based on the component triggering process and perform corresponding operations, including triggering operation components and inputting text;

[0011] a verification module, connected to the storage module and the robot module respectively, for responding to the robot module executing an operation, calling a current operation interface and a corresponding component triggering interface for similarity matching to obtain a matching result;

[0012] A mutation processing module, connected to the verification module, is used to determine the coordinates of the difference area, call the current operation interface and the corresponding component trigger interface to perform feature elimination processing, and then send it to the verification module to obtain a matching result again;

[0013] An adaptation module, which is connected to the verification module and the mutation processing module respectively, and is used to control the robot module based on the matching result of the verification module, including:

[0014] Calling the abnormality processing module to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification;

[0015] Or, control the execution process of the robot module;

[0016] The trigger verification includes generating a verification control instruction based on the difference area coordinates, controlling the robot module to execute the verification control instruction, and then verifying whether the robot module is controlled to execute the process, wherein the verification control instruction includes a number of instruction trigger coordinates.

[0017] Furthermore, the verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching, including:

[0018] To determine the image similarity between the current operation interface and the corresponding component trigger interface;

[0019] for extracting a first keyword of the current operation interface and a second keyword of the corresponding component triggering interface;

[0020] To determine the keyword similarity between the first keyword and the second keyword;

[0021] Used to determine the ratio of image similarity to the reference image similarity as the similarity influencing factor;

[0022] To determine the keyword similarity and the benchmark keyword similarity ratio as the keyword impact factor;

[0023] The weighted sum of the similarity impact factor and the keyword impact factor is used to determine the similarity matching coefficient.

[0024] Furthermore, the verification module is used to obtain a matching result, wherein:

[0025] If the similarity matching coefficient is greater than or equal to the coefficient threshold, determining the matching result is a match;

[0026] If the similarity matching coefficient is less than the coefficient threshold, the matching result is determined to be a mismatch.

[0027] Furthermore, the variation processing module is used to determine the coordinates of the difference area, including:

[0028] Used to split the current operation interface and the component trigger interface in the same coordinate system to compare the image similarity between the current local operation interface and the corresponding local component trigger interface;

[0029] Identify the difference area based on the comparison result, determine the coordinate value of each pixel point in the difference area, and obtain the coordinates of the difference area;

[0030] If the image similarity is less than a predetermined difference region image similarity threshold, the corresponding local operation interface is determined to be a difference region.

[0031] Furthermore, the mutation processing module performs feature elimination, including:

[0032] Determining the difference area outlines corresponding to the current operation interface and the component triggering interface based on the difference area coordinates;

[0033] It is used to remove the image features within the contour of the difference area.

[0034] Furthermore, the adaptation module controls the robot module based on the matching result of the verification module, including:

[0035] If the matching result of the verification module is a match, the robot module is controlled to execute the process;

[0036] If the matching result of the verification module is mismatch, the anomaly processing module is called to determine the relative position relationship between the difference area and the operation component to be triggered in the next step and the area of ​​the difference area to determine whether to control the robot module execution process and whether to perform trigger verification.

[0037] Furthermore, the adaptation module determines the relative position relationship between the difference area and the operation component to be triggered in the next step, including:

[0038] Used to determine the coordinates of the operating component to be triggered based on the component trigger interface;

[0039] Comparing the coordinates with the coordinates of the difference area to obtain a relative position relationship;

[0040] The relative position relationship includes: the operating component is located outside the difference area; and the operating component is located inside the difference area.

[0041] Furthermore, the adaptation module determines whether to control the robot module to execute the process and whether to perform trigger verification, including:

[0042] To determine the area of ​​the difference area;

[0043] If the current operation interface meets the process execution conditions, the adaptation module determines and controls the robot module to execute the process;

[0044] If the current operation interface does not meet the process conditions, the adaptation module determines to control the robot to perform trigger verification;

[0045] The process execution condition includes that the operating component is located outside the difference area and the area of ​​the difference area is smaller than a predetermined area.

[0046] Furthermore, the adaptation module generates a verification control instruction based on the coordinates of the difference area, including:

[0047] for identifying target features within the difference area, determining coordinates of the target features, and determining the coordinates as instruction trigger coordinates;

[0048] identifying edge coordinates of the difference area, randomly generating a predetermined number of coordinates outside the edge coordinates, and determining the coordinates as instruction trigger coordinates;

[0049] The target features include keywords and icons.

[0050] Furthermore, the adaptation module verifies whether the control robot module executes the process, including:

[0051] To obtain the current operation interface after executing the trigger verification, and call the verification module to obtain the matching result for the current operation interface;

[0052] If the matching result is a match, the robot module is controlled to execute the process;

[0053] If the matching result is no match, the robot module execution process is not controlled.

[0054] Compared with the prior art, the present invention provides a storage module, a robot module, a verification module, a mutation processing module, and an adaptation module. The robot module is used to generate operation instructions and perform corresponding operations. The verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching to obtain matching results. The mutation processing module is used to determine the coordinates of the difference area to obtain matching results again. The adaptation module is used to determine whether to call the mutation processing module, determine the relative position relationship between the difference area and the operation component to be triggered in the next step, and the area of ​​the difference area, to determine whether to control the robot module to execute the process and whether to perform trigger verification; or, to control the robot module to execute the process. The present invention comprehensively matches the current operation interface with the corresponding component trigger interface, enabling it to self-adjust, thereby improving work efficiency and accuracy.

[0055] In particular, the current operation interface is called to perform similarity matching with the corresponding component trigger interface, and the matching degree is calculated based on keywords and images to provide a data basis for determining whether to call the anomaly processing module to eliminate features. In actual working conditions, the robot module triggers the operation component according to the operation instructions, but if the operation is wrong or unknown content appears on the current operation interface, such as the operation is other steps, or an unknown pop-up window or error interface appears, the current operation interface may be partially covered or completely covered, making the operation steps discontinuous and making the robot module unable to continue to work according to the preset steps. Based on this, consider performing similarity matching on the current operation interface and the corresponding component trigger interface to facilitate subsequent timely intervention in the execution of operations by the robot module and improve the stability and reliability of the robot module.

[0056] In particular, for the unmatched interface, the mutation processing module is called to determine the coordinates of the difference area for feature elimination. When working according to the operation process and encountering an overlay window, if the current operation interface is continued to be operated based on the corresponding component trigger interface, it will lead to operational errors and even cause the interface to change, so that the process is terminated. Based on this, consider determining the feature area of ​​the current operation interface and eliminating the feature area that interferes with the current window, providing a data basis for determining whether to control the execution process of the robot module and whether to perform trigger verification, so that the robot module can continue to work according to the preset steps in the event of an emergency, and can perform self-adjustment, thereby improving work efficiency and accuracy.

[0057] In particular, the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area are determined to determine whether to control the robot module to execute the process and whether to perform trigger verification. In the actual operation process, the difference area may cover the component. At this time, the difference area needs to be verified. The coverage status of the current operation interface is determined by determining the relative position relationship and the area of ​​the difference area. The current operation interface with the covered component is verified so that the robot module can continue to work according to the preset steps and can perform self-adjustment, thereby improving work efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic structural diagram of a software robot adjustment system suitable for adaptive upgrading according to an embodiment of the invention;

[0059] Figure 2 A logic block diagram of a verification module for obtaining matching results according to an embodiment of the invention;

[0060] Figure 3 A logic block diagram of an adaptation module according to an embodiment of the invention controlling a robot module based on a matching result of the verification module;

[0061] Figure 4 This is a logic block diagram of the adaptation module of an embodiment of the invention determining whether to control the robot module to execute the process and whether to perform trigger verification. DETAILED DESCRIPTION

[0062] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] It should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the term "connection" should be understood in a broad sense. For example, it can mean a fixed connection, a detachable connection, or an integral connection; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0065] See also Figures 1-4 As shown, Figure 1 This is a schematic diagram of the structure of a software robot adjustment system suitable for adaptive upgrading according to an embodiment of the invention. Figure 2 This is a logic block diagram of the verification module of an embodiment of the invention for obtaining matching results. Figure 3 This is a logic block diagram of the adaptation module of an embodiment of the invention controlling the robot module based on the matching result of the verification module. Figure 4 This is a logic diagram of the adaptation module of the embodiment of the invention determining whether to control the execution process of the robot module and whether to perform trigger verification. The present invention is a software robot adjustment system suitable for adaptive upgrading, which includes:

[0066] The storage module is used to store preset component triggering processes and several component triggering interfaces. It can be understood that the component triggering interface is the interface that jumps to after any component in the operating system is triggered, which will not be repeated here.

[0067] A robot module, connected to the storage module, configured to generate operation instructions based on the component triggering process and perform corresponding operations, including triggering operation components and inputting text;

[0068] The verification module is connected to the storage module and the robot module respectively, and is used to respond to the robot module to perform operations, call the current operation interface and the corresponding component trigger interface for similarity matching, and obtain the matching result; it can be understood that after the robot module performs the corresponding operation based on the operation instruction, it will generate the corresponding current operation interface, and the verification module responds to the robot module to perform operations, that is, after each operation is performed, the verification module will call the corresponding current operation interface for verification to ensure the accuracy of the operation and improve the adaptive upgrade capability.

[0069] A mutation processing module, connected to the verification module, is used to determine the coordinates of the difference area, call the current operation interface and the corresponding component trigger interface to perform feature elimination processing, and then send it to the verification module to obtain a matching result again;

[0070] An adaptation module, which is connected to the verification module and the mutation processing module respectively, and is used to control the robot module based on the matching result of the verification module, including:

[0071] Calling the abnormality processing module to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification;

[0072] Or, control the execution process of the robot module;

[0073] The trigger verification includes generating a verification control instruction based on the difference area coordinates, controlling the robot module to execute the verification control instruction, and then verifying whether the robot module is controlled to execute the process, wherein the verification control instruction includes a number of instruction trigger coordinates.

[0074] Specifically, there is no limitation on the source of the preset component triggering process. For example, the corresponding component triggering process can be constructed and recorded by recording manual operation steps. Those skilled in the art can construct it according to actual needs. It is only necessary to ensure that the process is continuous and uninterrupted and the operation can be completed. This will not be repeated.

[0075] It can be understood that the component triggering process is the temporal relationship of the triggering of the operation components. For example, taking the example of the entire workflow needing to trigger five operation components, the component triggering process includes the triggering sequence of the five operation components, so that the robot module can trigger the operation components in a predetermined order.

[0076] Specifically, the operation component is an interactive component in the interface, and the corresponding function is realized by triggering the corresponding operation component, such as icons, buttons, etc., which will not be repeated here.

[0077] It is understandable that, in implementation, the component triggering interface and the current operation interface are image data, and both can be obtained in the form of screenshots.

[0078] Specifically, there is no limitation on the specific structures of the storage module, robot module, verification module, mutation processing module and adaptation module, which can be composed of logical components, including field programmable processors, computers or microprocessors in computers.

[0079] Specifically, there is no limitation on the specific way in which the robot module generates operation instructions.

[0080] In some possible implementations, the component triggering process may include the coordinates of the corresponding component, that is, the robot module may generate a click instruction based on the coordinates of the corresponding component, control the cursor to click the corresponding coordinates, and realize the function of triggering the operation component.

[0081] In some possible implementations, the robot module can identify the coordinates of the currently required triggering operation component based on the image processing algorithm, and then control the cursor to click the corresponding coordinates to realize the function of the triggering operation component.

[0082] Specifically, the verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching, including:

[0083] To determine the image similarity between the current operation interface and the corresponding component trigger interface;

[0084] for extracting a first keyword of the current operation interface and a second keyword of the corresponding component triggering interface;

[0085] To determine the keyword similarity between the first keyword and the second keyword;

[0086] Used to determine the ratio of image similarity to the reference image similarity as the similarity influencing factor;

[0087] To determine the keyword similarity and the benchmark keyword similarity ratio as the keyword impact factor;

[0088] The weighted sum of the similarity impact factor and the keyword impact factor is used to determine the similarity matching coefficient.

[0089] Specifically, there is no limitation on the method of calculating image similarity. For example, the method of calculating the cosine similarity of the images can be used to represent the current operation interface as a vector, and the similarity of the two images is characterized by calculating the cosine distance between the vectors. Of course, other methods can also be used, which will not be repeated here.

[0090] Specifically, there is no limitation on the method of obtaining the keywords. For example, OCR technology can be used to recognize and convert the text content in the current operation interface into an editable text format, and then extract keywords from these texts. Of course, other methods can also be used, which will not be repeated here.

[0091] Specifically, there is no limitation on the method for calculating keyword similarity. For example, the Word2Vec word embedding model can be used to capture the semantic relationship between keywords through the vector representation of keywords, and the similarity between keywords can be measured by calculating the cosine similarity between two keyword vectors.

[0092] Of course, other methods may also be used. Those skilled in the art only need to ensure that the keyword similarity can be calculated, and details will not be given here.

[0093] Specifically, considering the personalized customization of the system, the operation interfaces triggered by the same operation component may be different. A certain degree of difference is allowed, so the benchmark image similarity is selected in the interval [0.85, 1].

[0094] Specifically, considering that system updates or UI changes may cause the keywords corresponding to the operation components to be replaced by synonyms, the benchmark keyword similarity is selected in the interval [0.95, 1].

[0095] Specifically, the weight coefficient of the similarity impact factor is 0.35, and the weight coefficient of the keyword impact factor is 0.65.

[0096] Specifically, the verification module is used to obtain matching results, where

[0097] If the similarity matching coefficient is greater than or equal to the coefficient threshold, determining the matching result is a match;

[0098] If the similarity matching coefficient is less than the coefficient threshold, the matching result is determined to be a mismatch.

[0099] Specifically, the coefficient threshold is pre-calculated and selected within the interval (0.915, 1).

[0100] Specifically, the current operation interface is called to perform similarity matching with the corresponding component trigger interface, and the matching degree is calculated based on keywords and images to provide a data basis for determining whether to call the anomaly processing module to eliminate features. In actual working conditions, the robot module triggers the operation component according to the operation instructions, but if the operation is wrong or unknown content appears on the current operation interface, such as the operation is other steps, or unknown pop-ups or error interfaces appear, the current operation interface may be partially or completely covered, making the operation steps discontinuous and the robot module unable to continue to work according to the preset steps. Based on this, consider performing similarity matching on the current operation interface and the corresponding component trigger interface to facilitate timely intervention in the execution of operations by the robot module and improve the stability and reliability of the robot module.

[0101] Specifically, the mutation processing module is used to determine the coordinates of the difference area, including:

[0102] It is used to split the current operation interface and the component trigger interface in the same coordinate system to compare the image similarity between the current local operation interface and the corresponding local component trigger interface. It can be understood that the local operation interface is obtained by splitting the current operation interface, and the corresponding local component trigger interface is obtained by splitting the component trigger interface.

[0103] Identify the difference area based on the comparison result, determine the coordinate value of each pixel point in the difference area, and obtain the coordinates of the difference area;

[0104] If the image similarity is less than a predetermined difference region image similarity threshold, the corresponding local operation interface is determined to be a difference region.

[0105] It can be understood that a pixel is the basic unit of a digital image, which represents a minimum point in the image. Each pixel has a specific coordinate position in the image, usually expressed in the form of rows and columns. In this embodiment, the coordinates of the pixel are determined as the coordinates of the difference area.

[0106] Specifically, there is no limit on the specific number of parts to be split, as long as it can ensure that the difference areas can be identified. Those skilled in the art can determine it based on the computing power of the logical components, which will not be elaborated here.

[0107] It is understandable that the method for calculating the image similarity of the difference region is consistent with the method for calculating the image similarity, and will not be described in detail here.

[0108] Specifically, the difference region image similarity threshold is the same as the image similarity threshold.

[0109] Specifically, for the unmatched interface, the mutation processing module is called to determine the coordinates of the difference area for feature elimination. When working according to the operation process and encountering an overlay window, if the current operation interface is continued to be operated based on the corresponding component trigger interface, it will lead to operational errors and even cause the interface to change, so that the process is terminated. Based on this, consider determining the feature area of ​​the current operation interface and eliminating the feature area that interferes with the current window, providing a data basis for determining whether to control the execution process of the robot module and whether to perform trigger verification, so that the robot module can continue to work according to the preset steps in the event of an emergency, and can perform self-adjustment, thereby improving work efficiency and accuracy.

[0110] Specifically, the mutation processing module performs feature elimination, including:

[0111] Used to determine the difference area outline corresponding to the current operation interface and the component trigger interface based on the difference area coordinates. It can be understood that the difference area coordinates include the coordinates of all pixel points in the difference area, and the edge coordinates can be identified to determine the difference area outline. This will not be repeated.

[0112] It is used to remove the image features within the contour of the difference area.

[0113] It is understandable that, in order to facilitate subsequent comparison, the same image can be used for filling after the elimination is completed, and this will not be repeated here.

[0114] Specifically, the adaptation module controls the robot module based on the matching result of the verification module, including:

[0115] If the matching result of the verification module is a match, the robot module is controlled to execute the process;

[0116] If the matching result of the verification module is mismatch, the anomaly processing module is called to determine the relative position relationship between the difference area and the operation component to be triggered in the next step and the area of ​​the difference area to determine whether to control the robot module execution process and whether to perform trigger verification.

[0117] Specifically, the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area are determined to determine whether to control the robot module to execute the process and whether to perform trigger verification. In the actual operation process, the difference area may cover the component. At this time, the difference area needs to be verified. The coverage status of the current operation interface is determined by determining the relative position relationship and the area of ​​the difference area. The current operation interface with the covered component is verified so that the robot module can continue to work according to the preset steps and can perform self-adjustment, thereby improving work efficiency and accuracy.

[0118] Specifically, the adaptation module determines the relative position relationship between the difference area and the operation component to be triggered in the next step, including:

[0119] It is used to determine the coordinates of the operating component to be triggered based on the component trigger interface. It can be understood that the component trigger interface includes the operating component. Therefore, the coordinates of the operating component can be determined in advance and recorded, which will not be repeated here.

[0120] Comparing the coordinates with the coordinates of the difference area to obtain a relative position relationship;

[0121] The relative position relationship includes: the operating component is located outside the difference area; and the operating component is located inside the difference area.

[0122] Specifically, if the operation component is located outside the difference area, it means that the difference area does not contain the operation component required to be triggered in the next step; if the operation component is located inside the difference area, it means that the difference area contains the operation component required to be triggered in the next step, and the difference area needs to be analyzed and verified to make the operation process return to the preset component triggering process.

[0123] Specifically, the adaptation module determines whether to control the robot module to execute the process and whether to perform trigger verification, including:

[0124] To determine the area of ​​the difference area;

[0125] If the current operation interface meets the process execution conditions, the adaptation module determines and controls the robot module to execute the process;

[0126] If the current operation interface does not meet the process conditions, the adaptation module determines to control the robot to perform trigger verification;

[0127] The process execution condition includes that the operating component is located outside the difference area and the area of ​​the difference area is smaller than a predetermined area.

[0128] Specifically, the predetermined area is set to 0.5 times the current operation interface.

[0129] Specifically, the adaptation module generates a verification control instruction based on the difference area coordinates, including:

[0130] for identifying target features within the difference area, determining coordinates of the target features, and determining the coordinates as instruction trigger coordinates;

[0131] identifying edge coordinates of the difference area, randomly generating a predetermined number of coordinates outside the edge coordinates, and determining the coordinates as instruction trigger coordinates;

[0132] The target features include keywords and icons.

[0133] Specifically, for some difference areas, the difference area can be cancelled by simply clicking anywhere outside the difference area; but for some difference areas, a specified button must be clicked to cancel the difference area.

[0134] It is understandable that the target features are keywords and icons that may potentially appear in the difference area, for example, the keyword "OK", the keyword "Cancel", the close icon, the minimize icon, etc., which will not be repeated here.

[0135] It is understandable that there is no limitation on the method of identifying icons, and it can be implemented using a pre-trained neural network image processing model. Of course, other methods can also be used, which will not be elaborated here.

[0136] Specifically, the adaptation module verifies whether the control robot module executes the process, including,

[0137] To obtain the current operation interface after executing the trigger verification, and call the verification module to obtain the matching result for the current operation interface;

[0138] If the matching result is a match, the robot module is controlled to execute the process;

[0139] If the matching result is no match, the robot module execution process is not controlled.

[0140] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0141] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A software robot adjustment system suitable for adaptive upgrading, characterized in that: include: A storage module, which is used to store preset component triggering processes and several component triggering interfaces; A robot module, connected to the storage module, configured to generate operation instructions based on the component triggering process and perform corresponding operations, including triggering operation components and inputting text; a verification module, connected to the storage module and the robot module respectively, for responding to the robot module executing an operation, calling a current operation interface and a corresponding component triggering interface for similarity matching to obtain a matching result; A mutation processing module, connected to the verification module, is used to determine the coordinates of the difference area, call the current operation interface and the corresponding component trigger interface to perform feature elimination processing, and then send it to the verification module to obtain a matching result again; An adaptation module, which is connected to the verification module and the mutation processing module respectively, and is used to control the robot module based on the matching result of the verification module, including: Calling the abnormality processing module to determine the relative position relationship between the difference area and the operating component to be triggered in the next step and the area of ​​the difference area, so as to determine whether to control the robot module to execute the process and whether to perform trigger verification; Or, control the execution process of the robot module; Wherein, the trigger verification includes generating a verification control instruction based on the coordinates of the difference area, controlling the robot module to execute the verification control instruction and then verifying whether the robot module is controlled to execute the process, wherein the verification control instruction includes a number of instruction trigger coordinates; The verification module is used to call the current operation interface and the corresponding component trigger interface for similarity matching, including: To determine the image similarity between the current operation interface and the corresponding component trigger interface; for extracting a first keyword of the current operation interface and a second keyword of the corresponding component triggering interface; To determine the keyword similarity between the first keyword and the second keyword; Used to determine the ratio of image similarity to the reference image similarity as the similarity influencing factor; To determine the keyword similarity and the benchmark keyword similarity ratio as the keyword impact factor; The weighted sum of the similarity impact factor and the keyword impact factor is used to determine the similarity matching coefficient.

2. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The verification module is used to obtain a matching result, wherein: If the similarity matching coefficient is greater than or equal to the coefficient threshold, determining the matching result is a match; If the similarity matching coefficient is less than the coefficient threshold, the matching result is determined to be a mismatch.

3. The software robot adjustment system suitable for adaptive upgrading according to claim 2, characterized in that: The variation processing module is used to determine the coordinates of the difference area, including: Used to split the current operation interface and the component trigger interface in the same coordinate system to compare the image similarity between the current local operation interface and the corresponding local component trigger interface; Identify the difference area based on the comparison result, determine the coordinate value of each pixel point in the difference area, and obtain the coordinates of the difference area; If the image similarity is less than a predetermined difference region image similarity threshold, the corresponding local operation interface is determined to be a difference region.

4. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The mutation processing module performs feature elimination. include, Determining the difference area outlines corresponding to the current operation interface and the component triggering interface based on the difference area coordinates; It is used to remove the image features within the contour of the difference area.

5. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module controls the robot module based on the matching result of the verification module, including: If the matching result of the verification module is a match, the robot module is controlled to execute the process; If the matching result of the verification module is mismatch, the anomaly processing module is called to determine the relative position relationship between the difference area and the operation component to be triggered in the next step and the area of ​​the difference area to determine whether to control the robot module execution process and whether to perform trigger verification.

6. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module determines the relative position relationship between the difference area and the operation component to be triggered in the next step, including: Used to determine the coordinates of the operating component to be triggered based on the component trigger interface; Comparing the coordinates with the coordinates of the difference area to obtain a relative position relationship; The relative position relationship includes: the operating component is located outside the difference area; and the operating component is located inside the difference area.

7. The software robot adjustment system suitable for adaptive upgrading according to claim 6, characterized in that: The adaptation module determines whether to control the robot module to execute the process and whether to perform trigger verification, including: To determine the area of ​​the difference area; If the current operation interface meets the process execution conditions, the adaptation module determines and controls the robot module to execute the process; If the current operation interface does not meet the process conditions, the adaptation module determines to control the robot to perform trigger verification; The process execution condition includes that the operating component is located outside the difference area and the area of ​​the difference area is smaller than a predetermined area.

8. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module generates a verification control instruction based on the difference area coordinates, including: for identifying target features within the difference area, determining coordinates of the target features, and determining the coordinates as instruction trigger coordinates; identifying edge coordinates of the difference area, randomly generating a predetermined number of coordinates outside the edge coordinates, and determining the coordinates as instruction trigger coordinates; The target features include keywords and icons.

9. The software robot adjustment system suitable for adaptive upgrading according to claim 1, characterized in that: The adaptation module verifies whether the control robot module executes the process, include, To obtain the current operation interface after executing the trigger verification, and call the verification module to obtain the matching result for the current operation interface; If the matching result is a match, the robot module is controlled to execute the process; If the matching result is no match, the robot module execution process is not controlled.

Images