String recognition method, device and medium for robotic process automation

Abstract

The present invention relates to the field of communication network management, and specifically discloses a string recognition method, device, and medium for use in robotic process automation. The method comprises: taking a screenshot of a graphical user interface to be recognized, and identifying the starting position and first character text through machine vision according to string rules; analyzing whether the first character text is complete; if so, outputting and terminating string recognition; otherwise, calculating the end position of the string to be recognized and adjusting the width of the interface frame to obtain a complete second character text. The present invention first estimates the end position of the string, then adjusts the window width to obtain the complete string and thus performs string recognition. Compared with the traditional method of uniformly expanding the window width to the maximum width before recognition, this method occupies fewer system resources and improves string recognition efficiency.

Description

Technical Field

[0001] The present invention relates to the field of communication network management, and in particular to a character string recognition method, device and medium used in robotic process automation. Background Art

[0002] With the development of modern network communication technology, the management of enterprises and factories has become increasingly reliant on network management tools. While this has saved significant manpower and resources, it has also introduced some operational inconveniences. For example, in power management systems, because the communication network management system manufacturer is reluctant to open interfaces to the underlying systems, upper-level systems (other application systems) cannot directly access communication network management functions. Operations must be performed manually by logging into the underlying communication network management system of each device one by one. Furthermore, the power management system has a large number of communication network management systems, and the same operation cannot be issued all at once through the interface. Instead, manual operations must be repeated across multiple systems, resulting in high duplication and significant waste of manpower.

[0003] To address these issues, developers often use Robotic Process Automation (RPA) technology. RPA software robots simulate human mouse and keyboard operations on a computer, automating processes or a series of tasks according to pre-defined rules, freeing management personnel from repetitive and tedious tasks. In power system applications, machine vision is used to identify the network management interface of the power communication system. Clicking on the identified interface elements issues commands, allowing the same command to be automatically executed across multiple underlying communication network management systems, reducing manual repetitive operations.

[0004] Among all the configuration operations, obtaining the device's IP address string from the system interface and configuring each device based on the IP address string is an operation with a high workload. However, in actual application, see Figure 2 When RPA software robots recognize the underlying communications network management system interface, the starting position, length, and ending position of the IP address strings displayed in the interface vary significantly, often exceeding the system's default text box width. This prevents traditional machine vision technology from recognizing the complete IP address string. Furthermore, it's impossible to configure tools to drag the text box to a specific, quantifiable position to accommodate the reading of each IP address string. Simulating a uniform, fixed width for the text box to capture the complete IP address string results in the RPA software robot seeing more background information, which becomes noise and affects the speed and accuracy of IP address character recognition. Summary of the Invention

[0005] In order to overcome the above problems, the present invention provides a string recognition method, device and medium for use in robotic process automation.

[0006] The present invention provides a string recognition method for use in robotic process automation, comprising the following steps:

[0007] Taking a screenshot of the graphical user interface to be recognized to obtain a first image to be recognized;

[0008] Recognize characters from the first image to be recognized using machine vision according to character string rules, and obtain the starting position and first character text of the character string to be recognized; wherein the character string rules include the relative position range of the character string to be recognized in the graphical user interface, character string length rules, and character string text rules;

[0009] Analyzing whether the first character text is complete according to the character string rule;

[0010] If yes, the first character text is output and the character string recognition method ends;

[0011] Otherwise, calculating the end position of the character string to be recognized according to the character string rule, the starting position and the first character text;

[0012] Automatically adjusting the width of the frame of the graphical user interface where the character string to be recognized is located according to the terminal position;

[0013] A second image to be recognized is obtained by re-screening, and characters are recognized from the second image to be recognized by machine vision according to character string rules to obtain and output a second character text.

[0014] Preferably, the string rule is the string rule of the IP address, specifically including: the string format is YYYY, where Y includes Z, ZZ or ZZZ, Z is a character from 0 to 9 occupying m*n pixel positions, and Y is separated by the character "."; where m and n are both positive integers.

[0015] Preferably, the specific steps of the machine vision recognition are as follows:

[0016] Using a machine vision algorithm to obtain the first character position of the IP address string from an image of the area corresponding to the relative position range;

[0017] Using the expansion algorithm, determine whether there is an IP address character within the m*n pixel range of the next character position;

[0018] If yes, record this character position as the last character position and repeat the previous step;

[0019] Otherwise, the character string is identified based on the first character position and the last character position through the corrosion algorithm, and the recognition is terminated.

[0020] Preferably, the dilation algorithm is used to determine whether there is a character pixel within the m*n pixel range, specifically:

[0021] Determine the pixel step number X for the horizontal expansion operation that can achieve the highest recognition accuracy 准 value;

[0022] Apply X to the m*n pixel image 准 The step size is used to perform horizontal expansion operation to determine whether there are character pixels.

[0023] Preferably, when m is 9, the X 准 =7.

[0024] Preferably, the character string is identified by using a corrosion algorithm, specifically:

[0025] Determine the optimal erosion step size X 腐 value;

[0026] For the entire string of character images from the first character position to the last character position, perform a step of X 腐 The horizontal corrosion operation of the "010" structure is performed to identify the characters in each character position.

[0027] Preferably, when n is 9, the X 腐 =1.

[0028] The present invention also provides a character string recognition device for use in robotic process automation, comprising: a screenshot module, a first recognition module, an analysis module, a terminal position calculation module, an execution module, and a second recognition module;

[0029] The screenshot module is used to take a screenshot of the graphical user interface to be identified to obtain a first image to be identified;

[0030] The first recognition module is configured to recognize characters from the first image to be recognized by machine vision according to a character string rule, and obtain a starting position and a first character text of the character string to be recognized; wherein the character string rule includes a relative position range of the character string to be recognized in the graphical user interface, a character string length rule, and a character string text rule;

[0031] The analysis module is used to analyze whether the first character text is complete according to the character string rule; if so, output the first character text and end the character string recognition;

[0032] The end position calculation module is used to calculate the end position of the character string to be recognized according to the character string rule, the starting position and the first character text when the result of the analysis module is no;

[0033] The execution module is used to automatically adjust the width of the frame of the graphical user interface where the character string to be recognized is located according to the terminal position;

[0034] The second recognition module is used to re-screenshot to obtain a second image to be recognized, and recognize characters from the second image to be recognized through machine vision according to character string rules to obtain and output second character text.

[0035] The present invention provides a computer-readable storage medium, which includes a stored computer program. When the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned string recognition method for robotic process automation.

[0036] The beneficial effects of the present invention include at least:

[0037] By first estimating the end position of the string and then adjusting the window width to obtain the complete string, the string can be read. Compared with the traditional method of uniformly expanding the window width to the maximum width before recognition, this method takes up less system resources and improves the efficiency of string recognition.

[0038] Preferably, this solution proposes an optimization of the segmentation graph operation algorithm for the IP address string, so as to accurately and efficiently judge the integrity of the IP address string and improve the efficiency of IP address string recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention will be further described below with reference to the accompanying drawings, in which:

[0040] Figure 1 A flow chart of a method according to one embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a graphical user interface according to another embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of character pixels of the entire IP address according to another embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] See also Figure 1As one embodiment of the present invention, a string recognition method for robotic process automation is disclosed, comprising the following steps:

[0045] S1. Taking a screenshot of the graphical user interface to be recognized to obtain a first image to be recognized;

[0046] S2. Recognize characters from the first image to be recognized using machine vision according to character string rules, and obtain a starting position and first character text of the character string to be recognized; wherein the character string rules include a relative position range where the character string to be recognized may appear in the graphical user interface, a character string length rule, and a character string text rule;

[0047] S3. Analyze whether the first character text is complete according to the string rule;

[0048] S42: If yes, output the first character text and end the string recognition method;

[0049] S41. Otherwise, calculate the end position of the character string to be recognized based on the character string rule, the starting position, and the first character text;

[0050] S5. Automatically adjust the width of the frame of the graphical user interface where the character string to be recognized is located based on the terminal position;

[0051] S6. Re-take a screenshot to obtain a second image to be recognized, and recognize characters from the second image to be recognized through machine vision according to the character string rule to obtain and output a second character text.

[0052] This embodiment estimates the end position of the string and then adjusts the window width to obtain the complete string for string recognition. Compared with the traditional method of uniformly expanding the window width to the maximum width before recognition, it occupies fewer system resources and improves the efficiency of string recognition.

[0053] See also Figure 2 and Figure 3 As another embodiment of the present invention, this embodiment identifies IP address strings of different lengths and their meanings based on the IP address string encoding specification of the power communication system. Specifically, an expansion operation algorithm for IP address character segmentation and recognition adapted for the power communication network management system is disclosed.

[0054] In the electric power communication network management system, there are 11 types of IP address characters, and each character occupies an m*n pixel grid; where m and n are both positive integers. Based on this characteristic, this embodiment studies and proposes a graphics operation algorithm adapted for IP address character segmentation in the electric power communication network management system, achieving accurate and efficient determination of the integrity of the IP address string.

[0055] The string rule of this embodiment is the string rule of the IP address, specifically: the string format is YYYY, where Y includes Z, ZZ, or ZZZ, Z is a character from 0 to 9 occupying m*n pixel points, and Y is separated by the character ".", that is, 11 IP address characters.

[0056] In some embodiments, brackets or spaces are set at both ends of the IP address string by default, that is, "(", ")" or Figure 3 The last vacancy has no impact on the implementation of this solution.

[0057] The specific steps of the IP string recognition method used in robotic process automation of this embodiment are as follows:

[0058] S1. Taking a screenshot of the graphical user interface to be recognized to obtain a first image to be recognized;

[0059] S2. According to the IP character string rule, the IP character is recognized from the first image to be recognized by machine vision, and the starting position and the first character text of the IP character string to be recognized are obtained;

[0060] S3. Analyze whether the first character text is complete according to IP character string rules;

[0061] S42: If yes, output the first character text and end the IP character string recognition method;

[0062] S41. Otherwise, calculate the end position of the IP string to be identified based on the IP string rule, the starting position, and the first character text;

[0063] S5. Automatically adjust the width of the frame of the graphical user interface where the IP string to be identified is located based on the terminal position;

[0064] S6. Re-take a screenshot to obtain a second image to be identified. According to the IP character string rule, the IP characters are identified from the second image to be identified by machine vision to obtain and output a second character text.

[0065] Among them, through machine vision recognition, the steps include:

[0066] A1. Use a machine vision algorithm to obtain the position of the first IP character in the IP address string from the image of the area corresponding to the relative position range.

[0067] A2. Using an expansion algorithm, determine whether the next IP character position has an IP character in the range of m*n pixels.

[0068] A31. If yes, record the IP character position as the last character position of the IP and repeat the previous step.

[0069] A32. Otherwise, based on the first IP character position and the last IP character position, the IP character string is identified through the corrosion algorithm, and the identification is terminated.

[0070] The traditional opening and closing operation algorithms are to perform dilation and then erosion or erosion and then dilation operations on the image. Due to the application of IP address characters in the power communication network management system, the traditional dilation and erosion have the disadvantages of low efficiency and insufficient accuracy respectively, so the opening and closing operations are not suitable for direct application.

[0071] Preferably, step A2, using a dilation algorithm to determine whether there is an IP character pixel within the m*n pixel range, specifically includes the following steps:

[0072] A21. Confirm the number of pixel steps X for the horizontal expansion operation that can achieve the highest recognition accuracy. 准 value;

[0073] A22. Apply X to the m*n pixel image 准 The step size is used to perform horizontal expansion operation to determine whether there are IP character pixels.

[0074] Preferably, step A32 further includes the following sub-steps:

[0075] A321. Determine the optimal step size X for the corrosion operation 腐 value;

[0076] A322, for the entire string of IP character images from the first IP character position to the last IP character position, perform a step of X 腐 The horizontal corrosion operation of the "010" structure is performed to identify the IP characters in each IP character position.

[0077] The principle of the above dilation algorithm is to expand the boundary points of the binary object, merge all background points that are in contact with the object into the object, and expand the boundary of the object outward.

[0078] Traditional expansion algorithms often use small-step horizontal expansion or horizontal and vertical expansion for traversal. In the IP address character recognition of power communication network management systems, due to the long length of the IP address string and the difficulty in determining its precise end position, the system has difficulty determining when to terminate the algorithm, resulting in excessive computation and low efficiency.

[0079] According to the morphological characteristics of IP address characters in the power communication network management system, this embodiment proposes using the expansion algorithm for the IP address character string recognition scenario of the power communication network management system. In the m columns of pixels of a single character position, a right shift is performed with a step size of X pixels. Whether the character position is empty is determined based on whether the character pixel appears within the step size range. Let X 准is the X step value when both recognition accuracy and recognition efficiency reach the optimal balance, and X 准 ≤m. In the unit character space, based on the determined single-step optimal X 准 , drag the text recognition box to the right by the corresponding length from the starting column, and you can accurately identify whether the bit is the end bit of the IP address string by dragging the recognition box once. At the same time, the time required for recognition is t 总 By identifying the process time t 识别 and the program call time t 调 Composition, where t 调 Can be regarded as a fixed value, t 识别 Approximately log of X n Trends in X. Use X 准 As a step size, it can reduce the amount of calculation while maintaining character recognition accuracy, ensuring the efficient operation of the RPA system.

[0080] This technology combines the research on expansion and corrosion algorithms and proposes a morphological opening and closing operation algorithm adapted to the IP address character segmentation of the power communication network management system: First, use X 准 Perform expansion operation on the step size to determine whether there is an IP address character in the character position;

[0081] If it exists, temporarily record this character position as the end of the string, and continue to perform the expansion operation on the next character position;

[0082] If it does not exist, the string recognition algorithm ends and the text recognition box is dragged right to the end of the recorded string. After the IP address string is determined in the recognition box, the text in the stack box is applied with the erosion algorithm to determine the meaning of the characters and output the complete IP address string recognition result.

[0083] like Figure 3 As shown, in the power communication network management system, each IP address character occupies a different number of pixels. Taking the most commonly used font size as an example, the pixel space is 9x9 (i.e., m=9). Since a character position can be one of 12 characters, as shown in the figure below: The pixel format of the 12 characters in the power communication network management system is described as follows:

[0084] When X takes a certain value, it means recognizing the pixels in the X columns before the character. If there is a pixel point that is not empty, it is determined that the subsequent position is not empty; if the characters in the X column range are all empty, it is inferred whether the subsequent position is empty.

[0085] The underlying communication network management system interface has low resolution and other issues due to its early development. As a result, there are display differences on current operating terminals. Therefore, a universal X value is required at the pixel level to ensure that the algorithm can achieve optimal accuracy and efficiency across all systems.

[0086] For example, when using X=1 step length, the first column of pixels of each character in the corresponding image is identified. There are 12 characters (0-9, symbol ".", empty character) that meet the requirement that the pixel column is all empty. Based on the fact that the pixel column is all empty, it is inferred that the subsequent space is also empty. Among the 12 characters, only the empty character meets the requirement. Therefore, the accuracy of judging that the subsequent space is an empty character is 1 / 12=8.3% at this time. When using X=2 step length, the first 1-2 columns of pixels of each character in the corresponding image are identified. There are 2 characters (symbol "." and empty character) that meet the requirement that the pixel range is empty. Based on the fact that the pixel range is empty, it is inferred that the subsequent space is an empty character. The accuracy is 1 / 2=50% at this time. When X is in the range of 3-5, the situation is similar to X=2, with an accuracy of 50%. When using X=7 step length, there is only 1 character (empty character) that meets the requirement that the first 7 columns of pixels are empty. Based on the fact that the pixels in this range are empty, the accuracy of inferring that the subsequent space is empty is 1 / 1=100%. When X is between 8 and 9, the situation is similar to that of X=7, with an accuracy rate of 100%. According to the above process, the distribution of the accuracy rate of IP address character recognition in the power communication network management system as X changes is summarized in Table 1 below.

[0087] Step size X 1 2 3 4 5 6 7 8 9 Accuracy 8.3％ 50％ 50％ 50％ 50％ 50％ 100％ 100％ 100％

[0088] Table 1

[0089] It can be seen that when X=7, 8, and 9, the accuracy reaches the highest value of 100%.

[0090] System testing revealed that the time required to adjust the window width each time remained essentially constant. However, as the pixel column X for a single recognition increased, the recognition range expanded, and the recognition time t increased. Therefore, the total time t required to adjust the window width once, complete recognition, and determine whether the window is empty gradually increased. The time distribution for a single recognition as X changed is shown in Table 2.

[0091]

[0092] Table 2

[0093] Therefore, combining Table 1 and Table 2, when X is in the range of 7-9, the accuracy is the highest, as shown by the blue line in the figure above. However, when X=7, the time consumption is the least, and when X is in the range of 8-9, the time consumption is slightly longer. Therefore, in this example, X 准 Taking 7, the recognition accuracy is the highest and the system operation efficiency is the highest. In this case, only one lateral shift is required to ensure high accuracy in identifying whether the position is empty.

[0094] Since the step size of a single right shift is 7 pixels, the range is relatively large. To avoid excessive noise in the recognized range and reduce the ability to restore and recognize characters, the erosion algorithm is tried after the dilation algorithm to remove the boundary noise of the recognized image.

[0095] This embodiment further proposes a corrosion operation algorithm adapted to the IP address character segmentation of the electric power communication network management system to confirm the meaning of the IP address characters.

[0096] The principle of the erosion algorithm is to shrink the boundaries of the image, which can be used to eliminate small and meaningless objects.

[0097] According to the morphological characteristics of IP address characters in the power communication network management system, this technology proposes to use multi-step (X 腐 ) to achieve precise recognition of IP address character pixels and accurately describe their meaning.

[0098] When using the erosion algorithm, it is also necessary to determine the erosion step size. In the above example, among the 12 character set column pixels, the only column that is completely empty is the first column. Therefore, for the IP address string of the power system in the above example, only a step size of X is required. 腐 = 1, eliminating noise and improving the accurate recognition of IP address character pixels.

[0099] The above is based on X 准 Multiple expansion operations and X-based 腐 A corrosion operation is performed to adapt the morphological opening and closing operations of the power system IP address string recognition and segmentation in this embodiment; the clarity of the IP address character recognition is guaranteed, the integrity of the IP address string is accurately and efficiently judged, and the efficiency of IP address string recognition is improved.

[0100] The present invention also discloses a character string recognition device for use in robotic process automation, comprising: a screenshot module, a first recognition module, an analysis module, a terminal position calculation module, an execution module, and a second recognition module;

[0101] The screenshot module is used to take a screenshot of the graphical user interface to be identified to obtain a first image to be identified;

[0102] The first recognition module is configured to recognize characters from a first image to be recognized using machine vision according to character string rules, and obtain a starting position and first character text of the character string to be recognized; wherein the character string rules include a relative position range in which the character string to be recognized may appear in the graphical user interface, a character string length rule, and a character string text rule;

[0103] The analysis module is used to analyze whether the first character text is complete according to the string rules; if so, the first character text is output and the string recognition is terminated;

[0104] The end position calculation module is used to calculate the end position of the character string to be identified based on the character string rule, the starting position and the first character text when the analysis result of the analysis module is negative;

[0105] The execution module is used to automatically adjust the width of the frame of the graphical user interface where the character string to be recognized is located according to the terminal position;

[0106] The second recognition module is used to re-screenshot to obtain a second image to be recognized, and recognize characters from the second image to be recognized through machine vision according to character string rules to obtain and output second character text.

[0107] The present invention also discloses a terminal device, including a processor and a storage device, wherein the storage device is used to store one or more programs; when the one or more programs are executed by the processor, the processor implements the above-mentioned string recognition method for robotic process automation. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the test equipment, and uses various interfaces and lines to connect various parts of the entire test equipment.

[0108] The storage device can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the storage device and calling data stored in the storage device. The storage device may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, etc.; the data storage area may store data created based on the use of the terminal device, etc. In addition, the storage device may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0109] Among them, if the module / unit integrated into the string recognition device used in robotic process automation is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in at least one computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. Computer-readable media can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium.

[0110] It should be noted that the embodiments of the devices and apparatuses described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the objectives of the present embodiment as needed.

Claims

1. A string recognition method for robotic process automation, characterized in that: The steps are as follows: Taking a screenshot of the graphical user interface to be recognized to obtain a first image to be recognized; Recognize characters from the first image to be recognized using machine vision according to character string rules, and obtain the starting position and first character text of the character string to be recognized; wherein the character string rules include the relative position range of the character string to be recognized in the graphical user interface, character string length rules, and character string text rules; Analyzing whether the first character text is complete according to the character string rule; If yes, the first character text is output and the character string recognition method ends; Otherwise, calculating the end position of the character string to be recognized according to the character string rule, the starting position and the first character text; Automatically adjusting the width of the frame of the graphical user interface where the character string to be recognized is located according to the terminal position; Re-screening to obtain a second image to be recognized, and according to the character string rule, using machine vision to recognize characters from the second image to be recognized, to obtain and output a second character text; The string rule is a string rule for an IP address, specifically including: a string format of YYYY, where Y includes Z, ZZ, or ZZZ, Z is a character from 0 to 9 occupying m*n pixel locations, and Y is separated by the character ".", where m and n are both positive integers; The specific steps of machine vision recognition are as follows: Using a machine vision algorithm to obtain the first character position of the IP address string from an image of the area corresponding to the relative position range; Using the expansion algorithm, determine whether there is an IP address character within the m*n pixel range of the next character position; If yes, record this character position as the last character position and repeat the previous step; Otherwise, the character string is identified based on the first character position and the last character position through the erosion algorithm, and the recognition ends; The expansion algorithm is used to determine whether there is a character pixel within the m*n pixel range, specifically: Determine the pixel step number X for the horizontal expansion operation that can achieve the highest recognition accuracy 准 value; Apply X to the m*n pixel image 准 The step size is used to perform horizontal expansion operation to determine whether there are character pixels.

2. The character string recognition method for robotic process automation according to claim 1, characterized in that: When m is 9, the X 准 =7.

3. The character string recognition method for robotic process automation according to claim 1, wherein: The character string is identified by the corrosion algorithm, specifically: Determine the optimal erosion step size X 腐 value; For the entire string of character images from the first character position to the last character position, perform a step of X 腐 The horizontal corrosion operation of the "010" structure is performed to identify the characters in each character position.

4. The character string recognition method for robotic process automation according to claim 3, wherein: When n is 9, the X 腐 =1.

5. A string recognition device for use in robotic process automation, characterized in that: include: Screenshot module, first recognition module, analysis module, termination bit calculation module, execution module and second recognition module; The screenshot module is used to take a screenshot of the graphical user interface to be identified to obtain a first image to be identified; The first recognition module is configured to recognize characters from the first image to be recognized by machine vision according to a character string rule, and obtain a starting position and a first character text of the character string to be recognized; wherein the character string rule includes a relative position range of the character string to be recognized in the graphical user interface, a character string length rule, and a character string text rule; The analysis module is used to analyze whether the first character text is complete according to the character string rule; if so, output the first character text and end the character string recognition; The end position calculation module is used to calculate the end position of the character string to be recognized based on the character string rule, the starting position and the first character text when the analysis result of the analysis module is no; The execution module is used to automatically adjust the width of the frame of the graphical user interface where the character string to be recognized is located according to the terminal position; The second recognition module is configured to re-screenshot and obtain a second image to be recognized, and recognize characters from the second image to be recognized by machine vision according to a character string rule to obtain and output a second character text; The string rule is a string rule for an IP address, specifically including: a string format of YYYY, where Y includes Z, ZZ, or ZZZ, Z is a character from 0 to 9 occupying m*n pixel locations, and Y is separated by the character ".", where m and n are both positive integers; The specific steps of machine vision recognition are as follows: Using a machine vision algorithm to obtain the first character position of the IP address string from an image of the area corresponding to the relative position range; Using the expansion algorithm, determine whether there is an IP address character within the m*n pixel range of the next character position; If yes, record this character position as the last character position and repeat the previous step; Otherwise, the character string is identified based on the first character position and the last character position through the erosion algorithm, and the recognition ends; The expansion algorithm is used to determine whether there is a character pixel within the m*n pixel range, specifically: Determine the pixel step number X for the horizontal expansion operation that can achieve the highest recognition accuracy 准 value; Apply X to the m*n pixel image 准 The step size is used to perform horizontal expansion operation to determine whether there are character pixels.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the string recognition method for robotic process automation according to any one of claims 1 to 4.

Images