Automated testing method, apparatus, computer device, and storage medium

Abstract

The application relates to an automatic test method and device, computer equipment and a storage medium, and relates to the technical field of computers. The method can be used in the field of financial technology or other related fields. The method comprises the following steps: displaying an automatic test interface, obtaining test text information input in the automatic test interface for a business system to be tested, obtaining at least one test operation text and test object text corresponding to the test operation text according to the test text information; obtaining a plurality of operation instruction information pre-bound with the test operation text, and obtaining an operation object position corresponding to a test operation object represented by the test object text in a display page of the business system; executing each operation instruction information at the operation object position in the display page of the business system, obtaining operation results of each operation instruction information, and displaying the operation results in the automatic test interface. The method can improve the automatic test efficiency of the business system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an automatic testing method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] With the development of computer technology, an automatic testing technology for a business system appears, a user can pre-write an automatic testing script, and the automatic testing script is set in a testing system to run the automatic testing script to realize an automatic testing process of the business system.

[0003] In the prior art, the automatic testing process of the business system usually needs to call the automatic testing script according to the business process steps of the business system to realize the automatic testing, however, the process needs a tester to have a deep understanding of the business process steps of the business system, so as to accurately perform the automatic testing process, therefore, the existing automatic testing method for the business system has a small number of testers who can meet the testing requirements, and the automatic testing efficiency for the business system is low. SUMMARY

[0004] Therefore, it is necessary to provide an automatic testing method and device, computer equipment, computer readable storage medium and computer program product capable of improving the automatic testing efficiency.

[0005] In a first aspect, the present application provides an automatic testing method. The method comprises:

[0006] displaying an automatic testing interface, and obtaining test text information input in the automatic testing interface for a business system to be tested, obtaining at least one test operation text and a test object text corresponding to the test operation text according to the test text information;

[0007] obtaining a plurality of operation instruction information pre-bound with the test operation text, and obtaining an operation object position corresponding to a test operation object in a display page of the business system represented by the test object text;

[0008] executing each operation instruction information at the operation object position in the display page of the business system, obtaining an operation result of each operation instruction information, and displaying the operation result in the automatic testing interface.

[0009] In one of the embodiments, the obtaining the at least one test operation text and the test object text corresponding to the test operation text from the test text information comprises: obtaining a plurality of target text segments corresponding to the test text information, and obtaining a segment type corresponding to each of the target text segments and an association relationship of each of the target text segments; obtaining a first text segment of a verb type from the target text segments, and taking the first text segment as the test operation text if the first text segment belongs to a preconfigured operation text segment library; the operation text segment library pre-stores text segments corresponding to a plurality of test operations; obtaining a second text segment associated with the test operation text according to the association relationship, and taking the second text segment as the test object text corresponding to the test operation text; the second text segment is a text segment of a noun type from the target text segments.

[0010] In one of the embodiments, the obtaining the plurality of target text segments corresponding to the test text information comprises: obtaining a plurality of first initial text segments corresponding to the test text information by using a forward maximum matching algorithm, and obtaining a plurality of second initial text segments corresponding to the test text information by using a reverse maximum matching algorithm; and obtaining the target text segment from the first initial text segments or the second initial text segments.

[0011] In one of the embodiments, the obtaining the target text segment from the plurality of first initial text segments or the plurality of second initial text segments comprises: obtaining a first segment matching iteration number of the first initial text segments and a second segment matching iteration number of the second initial text segments; taking an initial text segment with a smaller segment matching iteration number as the target text segment if the first segment matching iteration number is different from the second segment matching iteration number; obtaining a first segment number of the first initial text segments and a second segment number of the second initial text segments if the first segment matching iteration number is the same as the second segment matching iteration number; taking an initial text segment with a smaller segment number as the target text segment if the first segment number is different from the second segment number; and taking the second initial text segment as the target text segment if the first segment number is the same as the second segment number.

[0012] In one of the embodiments, before the obtaining of the operation object position in the display page of the business system corresponding to the test object text, the method further comprises: obtaining text content corresponding to a text image contained in the display page of the business system and a text position of the text image in the display page; obtaining an operation object area contained in the display page of the business system and a region position of the operation object area in the display page; constructing a corresponding relationship between the text content and the region position according to the text position and the region position; and the obtaining of the operation object position in the display page of the business system corresponding to the test object text comprises: obtaining a target region position corresponding to the text content of the test object text according to the corresponding relationship between the text content and the region position, and taking the target region position as the operation object position.

[0013] In one of the embodiments, the constructing of the corresponding relationship between the text content and the region position according to the text position and the region position comprises: obtaining current text content, a current text center position corresponding to the current text content, and a region center position of each of the operation object areas in the display page; determining a position difference value between the current text center position and each of the region center positions, and taking an operation object area with a minimum position difference value as a current operation object area corresponding to the current text content; and constructing a corresponding relationship between the current text content and a region position of the current operation object area.

[0014] In one of the embodiments, after the taking of the operation object area with the minimum position difference value as the current operation object area corresponding to the current text content, the method further comprises: in a case where a position difference value between the current text content and the current operation object area is less than a preset difference threshold value, combining the current text content with a preset text content to form combined text content; and constructing a corresponding relationship between the combined text content and a region position of the current operation object area.

[0015] In one of the embodiments, the obtaining of the text content corresponding to the text image contained in the display page of the business system and the text position of the text image in the display page comprises: performing contour recognition on a page image of the display page, obtaining a text image contained in the page image according to a contour recognition result, and obtaining a text position of each of the text images in the display page; inputting each of the text images into a pre-trained text recognition model, extracting text features of the text images and a text sequence of each of the text images by the text recognition model; and obtaining text content corresponding to the text image based on the text features and the text sequence.

[0016] In one of the embodiments, the test operation text comprises an input operation text, and the operation instruction information bound to the test operation text comprises click instruction information and input instruction information; and the executing each operation instruction information at the operation object position in the display page of the business system comprises: executing a click instruction at the operation object position in the display page, and obtaining to-be-input data according to a data table bound to the operation object position; and inputting the to-be-input data to the operation object position.

[0017] In one of the embodiments, the automatic test interface further displays object description information of each test operation object contained in the display page of the business system; and before the obtaining to-be-input data according to the data table bound to the operation object position, the method further comprises: in response to a data table binding operation triggered in the automatic test interface for target object description information, obtaining a target data table corresponding to the data table binding operation; and binding an operation object position of a test operation object described by the target object description information to the target data table.

[0018] In a second aspect, the present application further provides an automatic test device. The device comprises:

[0019] a test text input module, configured to display an automatic test interface, and obtain test text information input in the automatic test interface for a business system to be tested, obtain at least one test operation text according to the test text information, and obtain test object text corresponding to the test operation text;

[0020] an operation information obtaining module, configured to obtain a plurality of operation instruction information bound to the test operation text in advance, and obtain an operation object position in a display page of the business system corresponding to a test operation object represented by the test object text;

[0021] an operation result display module, configured to execute each operation instruction information at the operation object position in the display page of the business system, obtain an operation result of each operation instruction information, and display the operation result in the automatic test interface.

[0022] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0023] display an automatic test interface, and obtain test text information input in the automatic test interface for a business system to be tested, obtain at least one test operation text and test object text corresponding to the test operation text according to the test text information;

[0024] obtain a plurality of operation instruction information pre-bound with the test operation text, and obtain an operation object position in a display page of the business system corresponding to a test operation object represented by the test object text;

[0025] execute each operation instruction information at the operation object position in the display page of the business system, obtain an operation result of each operation instruction information, and display the operation result in the automatic test interface.

[0026] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0027] display an automatic test interface, and obtain test text information input in the automatic test interface for a business system to be tested, obtain at least one test operation text and test object text corresponding to the test operation text according to the test text information;

[0028] obtain a plurality of operation instruction information pre-bound with the test operation text, and obtain an operation object position in a display page of the business system corresponding to a test operation object represented by the test object text;

[0029] execute each operation instruction information at the operation object position in the display page of the business system, obtain an operation result of each operation instruction information, and display the operation result in the automatic test interface.

[0030] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0031] display an automatic test interface, and obtain test text information input in the automatic test interface for a business system to be tested, obtain at least one test operation text and test object text corresponding to the test operation text according to the test text information;

[0032] obtain a plurality of operation instruction information pre-bound with the test operation text, and obtain an operation object position in a display page of the business system corresponding to a test operation object represented by the test object text;

[0033] The operation instructions are executed at the operation object location on the display page of the business system, the operation results of each operation instruction are obtained, and the operation results are displayed on the automated testing interface.

[0034] The aforementioned automated testing method, apparatus, computer equipment, storage medium, and computer program product, by displaying an automated testing interface and acquiring test text information input for the business system under test in the automated testing interface, acquires at least one test operation text and the test object text corresponding to the test operation text based on the test text information; acquires multiple operation instruction information pre-bound to the test operation text, and acquires the position of the operation object represented by the test object text in the display page of the business system; executes each operation instruction information at the operation object position in the display page of the business system, acquires the operation result of each operation instruction information, and displays the operation result in the automated testing interface. This application demonstrates an automated testing interface. Users only need to input test text information into the interface to determine multiple bound operation instructions based on the test operation text. The interface then executes these instructions on the corresponding operation object location within the business system's display page, as represented by the test object text. The operation results are then displayed on the automated testing interface. This method allows users to automate business system testing simply by inputting test text. Furthermore, since multiple operation instructions are pre-bound to the test operation text, users do not need in-depth knowledge of all business process steps; a general understanding of the test steps is sufficient for automated testing. This reduces the testing requirements for testers, increases the number of testers, and ultimately improves the efficiency of automated testing of the business system. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an automated testing method in one embodiment;

[0036] Figure 2 This is a flowchart illustrating the process of obtaining test operation text and test object text in one embodiment;

[0037] Figure 3 This is a schematic diagram of the process of obtaining target text word segmentation in one embodiment;

[0038] Figure 4 This is a flowchart illustrating the process of constructing the correspondence between text content and region location in one embodiment;

[0039] Figure 5 This is a flowchart illustrating the process of constructing the correspondence between text content and region location in one embodiment;

[0040] Figure 6 This is a flowchart illustrating the process of obtaining the text content and text position of a text image in one embodiment;

[0041] Figure 7 This is a flowchart of an automated testing process based on image recognition and natural language processing in one embodiment;

[0042] Figure 8 This is a schematic diagram of the image recognition process in one embodiment;

[0043] Figure 9 This is a flowchart illustrating the natural language processing process in one embodiment;

[0044] Figure 10 This is a schematic diagram of the syntactic analysis results in one embodiment;

[0045] Figure 11 This is a flowchart illustrating the process of generating operation instructions in one embodiment;

[0046] Figure 12 This is a schematic diagram of the system processing in one embodiment;

[0047] Figure 13 This is a schematic diagram of the test interface of an automated testing tool in one embodiment;

[0048] Figure 14 This is a structural block diagram of an automated testing device in one embodiment;

[0049] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one embodiment, such as Figure 1 As shown, an automated testing method is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0052] Step S101: Display the automated testing interface and obtain the test text information input for the business system to be tested in the automated testing interface. Obtain at least one test operation text and the test object text corresponding to the test operation text based on the test text information.

[0053] The automated testing interface is a platform interface displayed on the terminal for automating testing of the business system. The business system to be tested is the business system that needs to be automated tested. The test text information is the test language text information entered by the user for automating testing of the business system.

[0054] For example, a business system can be a linked system, and the test text information is the language text used to automate the business processes of the linked system. This test text can be obtained by user input, and can be: "Enter card number, account number, click the query button to query, enter the linked account, and click the linked button to link the relationship."

[0055] The test operation text is the text information used to represent the test operation, while the test object text is the text of the object corresponding to the test operation. For example, for the test text information "Enter card number, account number", the test operation text could be "Enter", and the corresponding test object text could be "Card number, account number". Similarly, for the text information "Click the query button to query", the test operation text could be "Click", and the corresponding test object text could be "Query button".

[0056] Specifically, when a user performs automated testing, an automated testing interface for automating business system testing can be displayed on the terminal. Within this interface, the user can select the business system to be automated and the text information used for testing that system, serving as the test text information. The terminal can then further extract at least one test operation text representing a test operation, and the corresponding test object text for each test operation text, based on the user's input text information.

[0057] Step S102: Obtain multiple operation instruction information pre-bound to the test operation text, and obtain the position of the operation object corresponding to the test operation object represented by the test object text in the display page of the business system.

[0058] The operation instruction information consists of pre-bound operation instructions to the test operation text. For a business system, executing a test operation may require multiple operation instructions. For example, for an input operation, the user needs to click the input box first and then input data. In this case, if the user is not familiar with the specific business process of the system, the automated test may fail. Therefore, in this embodiment, the terminal pre-bound corresponding operation instruction information to the test operation text. That is, for an input operation, both click instructions and input instructions can be bound simultaneously.

[0059] The location of the operation object refers to the position of the test operation object represented by the test object text in the display page of the business system. For example, the test object text may refer to "account". Then the position of the corresponding test operation object in the display page of the business system may refer to the position of the account input box used to enter the account in the business system. This position can be obtained in advance and the correspondence between it and the test object text can be pre-built.

[0060] Specifically, once the terminal receives the test operation text, it can obtain multiple operation instruction information bound to the test operation text. At the same time, it can also obtain the pre-constructed operation object location corresponding to the test object text based on the test object text.

[0061] Step S103: Execute each operation instruction information at the operation object location on the business system's display page, obtain the operation results of each operation instruction information, and display the operation results on the automated testing interface.

[0062] After obtaining the operation instruction information corresponding to the test operation text and the operation object location corresponding to the test object text in step S102, each operation instruction information can be executed in the operation object location on the display page of the business system to obtain the corresponding operation results, which are then displayed in the automated test interface to achieve automated testing.

[0063] For example, for the test text information "Enter Account", after the terminal determines that the operation instruction corresponding to the input operation is a click instruction + input instruction, it can trigger the above operation instruction at the location of the account input box in the business system. That is, first click the location of the account input box in the business system, and then perform the account input operation at that location. After obtaining the account input result, it can be displayed, thereby realizing automated testing.

[0064] In the aforementioned automated testing method, the terminal displays an automated testing interface and obtains test text information input for the business system under test in the automated testing interface. Based on the test text information, it obtains at least one test operation text and the test object text corresponding to the test operation text. It obtains multiple operation instruction information pre-bound to the test operation text and obtains the position of the operation object represented by the test object text in the display page of the business system. It executes each operation instruction information at the operation object position in the display page of the business system, obtains the operation results of each operation instruction information, and displays the operation results in the automated testing interface. This application displays an automated testing interface via a terminal. Users only need to input test text information into the automated testing interface. Based on the test operation text in the test text, multiple bound operation instructions can be determined, and operation instructions can be executed on the corresponding operation object positions in the business system's display page, as represented by the test object text. The operation results are then displayed in the automated testing interface. In this way, users only need to input test text to achieve automated testing of the business system. Furthermore, since multiple operation instructions are pre-bound to the test operation text, users do not need to have an in-depth understanding of all business process steps of the business system. They only need to have a rough understanding of the test steps to achieve automated testing of the business system. This reduces the testing requirements for testers, increases the number of testers, and thus improves the efficiency of automated testing of the business system.

[0065] In one embodiment, such as Figure 2 As shown, step S101 may further include:

[0066] Step S201: Obtain multiple target text segments corresponding to the test text information, and obtain the segmentation type corresponding to each target text segment, as well as the correlation between each target text segment.

[0067] Target text segmentation refers to the segmented text contained in the test text information, while segmentation type refers to the segmentation type corresponding to each target text segment, such as noun type, verb type, conjunction type, and punctuation type, etc. The correlation of target text segmentation refers to the correlation between each target text segment.

[0068] Specifically, the terminal can first obtain multiple text segments contained in the test text information through a text segmentation algorithm, which are then used as target text segments. Afterwards, it can use a syntactic analysis algorithm, such as Stanford Parser, to obtain the segmentation type corresponding to each target text segment and the relationship between each target text segment.

[0069] Step S203: Obtain the first text segmentation in the target text segmentation whose segmentation type represents the verb type, and if the first text segmentation belongs to the pre-set operation text segmentation library, use the first text segmentation as the test operation text; the operation text segmentation library pre-stores text segments corresponding to multiple test operations.

[0070] The first text segmentation refers to the segmentation of target text where the segmentation type is represented as verb. The operational text segmentation library is pre-built and includes a segmentation library for text segments corresponding to various test operations. Since automated testing generally only requires performing operations such as input and click, a segmentation library composed of text segments corresponding to the above operations can be pre-built. For example, this segmentation library can contain entries such as: "[click, double-click, right-click, scroll, click the scroll bar, click both left and right buttons simultaneously, etc.]".

[0071] Specifically, after obtaining the target text segmentation, the terminal can also determine the text segmentation type representing the verb type from the target text segmentation and use it as the first text segmentation. It can also determine whether each first text segmentation belongs to the pre-set operation text segmentation library. If it does, the first text segmentation is used as the test operation text; otherwise, the first text segmentation is discarded.

[0072] Step S203: Based on the association relationship, obtain the second text segment associated with the test operation text, and use the second text segment as the test object text corresponding to the test operation text; the second text segment is the text segment representing the noun type in the target text segment.

[0073] The second text segmentation refers to the target text segmentation with the segmentation type of noun. After the terminal obtains the test operation text in step S202, it can also find the second text segmentation associated with each test operation text based on the association relationship between each target text segmentation obtained in step S201, and use the above second text segmentation as the test object text corresponding to the test operation text.

[0074] In this embodiment, after obtaining the test text information, the terminal can also perform word segmentation on the test text information to obtain target text words, determine the word segmentation type of the target text words, and the relationship between the target text words. Thus, the target text words with the word segmentation type of verb and belonging to the pre-set operation text word segmentation library are used as test operation text, while the target text words with the word segmentation type of noun and having a relationship with the test operation text are used as test object text, thereby improving the recognition efficiency of test operation text and test object text.

[0075] Further, step S201 may further include: obtaining multiple first initial text segments corresponding to the test text information through a forward maximum matching algorithm, and obtaining multiple second initial text segments corresponding to the test text information through a reverse maximum matching algorithm; and obtaining the target text segment from the first initial text segments or the second initial text segments.

[0076] The forward maximum matching algorithm works by starting from the beginning of the test text and matching the longest and most meaningful word, continuing until the end to obtain the text segmentation. The first initial text segmentation refers to the initial text segmentation obtained through the forward maximum matching algorithm. The reverse maximum matching algorithm, on the other hand, matches in the opposite way, starting from the end of the test text and matching backwards until the beginning. The second initial text segmentation is obtained through the reverse maximum matching algorithm.

[0077] In this embodiment, the terminal can perform word segmentation on the test text information using the forward maximum matching algorithm and the reverse maximum matching algorithm respectively, to obtain the first initial text segmentation and the second initial text segmentation respectively. Then, the terminal can select one of the first initial text segmentation and the second initial text segmentation as the target text segmentation.

[0078] In this embodiment, the terminal can obtain the first initial text segmentation and the second initial text segmentation corresponding to the test text information through the forward maximum matching algorithm and the reverse maximum matching algorithm, respectively. Then, it can select one of the two initial text segmentations as the target text segmentation. In this way, the two segmentation matching algorithms can be integrated to obtain the target text segmentation, thereby improving the accuracy of target text segmentation acquisition.

[0079] Furthermore, such as Figure 3 As shown, obtaining the target text segment from the first initial text segmentation or the second initial text segmentation may further include:

[0080] Step S301: Obtain the first word segmentation matching iteration number of the first initial text segmentation and the second word segmentation matching iteration number of the second initial text segmentation.

[0081] The first word segmentation matching iteration count refers to the number of iterations required to obtain the first initial text segment using the forward maximum matching algorithm, while the second initial text segment refers to the number of iterations required to obtain the second initial text segment using the reverse maximum matching algorithm. In this embodiment, both the forward maximum matching algorithm and the reverse maximum matching algorithm find the longest and most meaningful word text through iteration. Therefore, the terminal can count the number of iterations required to obtain the first initial text segment using the forward maximum matching algorithm and the number of iterations required to obtain the second initial text segment using the reverse maximum matching algorithm, respectively, as the first word segmentation matching iteration count and the second word segmentation matching iteration count.

[0082] Step S302: If the number of iterations for the first word segmentation matching is different from the number of iterations for the second word segmentation matching, the initial text segment with the smaller number of word segmentation matching iterations is used as the target text segment.

[0083] Step S303: If the number of iterations for the first word segmentation matching is the same as the number of iterations for the second word segmentation matching, obtain the first word count of the first initial text segmentation and the second word count of the second initial text segmentation.

[0084] If the number of iterations for the first and second word segmentation matching are different, the terminal can select the initial text segment with the smaller number of iterations from the first and second initial text segments as the target text segment. If the number of iterations for the first and second word segmentation matching are the same, the terminal can further obtain the number of segments contained in the first initial text segment (i.e., the first segment count) and the number of segments contained in the second initial text segment (i.e., the second segment count).

[0085] Step S304: If the number of first segmented words and the number of second segmented words are different, the initial text segment with the smaller number of segmented words is used as the target text segment.

[0086] Step S305: If the number of first segmented words is the same as the number of second segmented words, the second initial text segmented words are used as the target text segmented words.

[0087] After obtaining the first and second word segmentation counts, the terminal can further compare them. If the first and second word segmentation counts are different (e.g., the first word segmentation count is less than the second word segmentation count), the terminal will use the initial text segment with the smaller number of segments, i.e., the first initial text segmentation, as the target text segmentation. If the first and second word segmentation counts are the same, the terminal will default to using the second initial text segmentation as the target text segmentation.

[0088] In this embodiment, after the terminal obtains the first initial text segmentation and the second initial text segmentation, it can determine the target text segmentation based on the number of iterations and the number of segments obtained from the initial text segmentation, thereby further improving the accuracy of target text segmentation acquisition.

[0089] In one embodiment, such as Figure 4 As shown, before step S102, the following may also be included:

[0090] Step S401: Obtain the text content corresponding to the text image contained in the display page of the business system, and the text position of the text image in the display page.

[0091] A text image refers to an image of text contained in the display page of a business system. This image can be obtained by performing image recognition on the display page. The text content refers to the text content contained in the text image, and the text position refers to the location of the text image on the display page of the business system. This position can be represented by coordinates.

[0092] Specifically, the terminal can perform image recognition on the display page of the business system in advance to find the text images contained in the display page. Then, it can perform text recognition on the text images to obtain the text content contained in the text images and determine the position coordinates of the text images on the display page of the business system.

[0093] Step S402: Obtain the operation object area contained in the display page of the business system, and the location of the operation object area in the display page.

[0094] The location of an operation target area refers to its position on the display page. Similar to text location, this location can also be represented by coordinates. For business systems, the operation target area can generally consist of selection boxes, buttons, input boxes, output boxes, etc., and these areas are usually rectangular. Therefore, the terminal can use a rectangular area recognition algorithm to identify the rectangular areas contained in the business system's display page and use these rectangular areas as operation target areas. It can also identify the position of each rectangular area on the business system's display page, thus determining the location of the operation target area within the display page.

[0095] Step S403: Based on the text position and the region position, construct the correspondence between the text content and the region position.

[0096] Finally, after obtaining the text position of the text image in step S401 and the region position of the operation object area in step S402, the text content of the corresponding text image and the correspondence between the region positions can be constructed based on the relationship between the text position and the region position.

[0097] Step S102 may further include: obtaining the target region location corresponding to the text content of the test object text based on the correspondence between text content and region location, and using the target region location as the operation object location.

[0098] After establishing the correspondence between text content and region location, once the terminal obtains the text content of the test object text, it can determine the region location corresponding to the text content based on the above correspondence, which is then used as the target region location. At the same time, the target region location is used as the location of the operation object represented by the test object text in the display page of the business system.

[0099] For example, a pre-built correspondence between account number and location A, card number and location B, and query button and location C can be established. After the terminal obtains the test object text, if the test object text is "account", then the corresponding operation object location is location A, and if the test object text is "card number", then the corresponding operation object location is location B.

[0100] In this embodiment, the terminal can pre-identify the text content of the text image in the business system and the regional location of the text image, and combine it with the regional location of the identified operation object area to construct the correspondence between the text content and the regional location. In this way, the pre-construction of the correspondence between the text content and the regional location can be realized, thereby improving the efficiency of obtaining the location of the operation object.

[0101] Furthermore, such as Figure 5 As shown, step S403 may further include:

[0102] Step S501: Obtain the current text content, the current text center position corresponding to the current text content, and the center position of each operation object area on the display page.

[0103] The current text content can be any text content contained in the business system's display page as identified by the terminal. The current text center position refers to the coordinates of the center position of the current text content, while the region center position refers to the coordinates of the region center position of each operation object region on the display page. In this embodiment, after obtaining each text content, the terminal can select one as the current text content. Furthermore, it can obtain the center position of the current text content based on its corresponding text position, and determine the corresponding region center position based on the region position of each operation object region on the business system's display page.

[0104] Step S502: Determine the position difference between the current text center position and the center positions of each region, and take the operation object region with the smallest position difference as the current operation object region corresponding to the current text content.

[0105] The position difference refers to the coordinate difference between the current text center position and the center position of each area. Generally speaking, the text content and its corresponding operation object area are relatively close. For example, the text content "account" on the display page is usually close to the input box used to enter the account. Therefore, the terminal can filter the operation object area based on the position difference, that is, set the operation object area with the smallest position difference from the current text center position as the operation object area corresponding to the current text content, that is, the current operation object area.

[0106] Step S503: Construct the correspondence between the current text content and the region position of the current operation object area.

[0107] Finally, after the terminal determines the current operation object area corresponding to the current text content, it can construct the correspondence between the current text content and the area position of the current operation object area. By repeating the above process, the correspondence between each text content and the area position of the operation object area can be obtained.

[0108] In this embodiment, the terminal can identify the operation object area corresponding to each text content based on the text center position of each text content and the difference between the center positions of each operation object area on the display page, thereby constructing a correspondence between the regional positions of each text content and the operation object area, thereby further improving the efficiency of the correspondence construction.

[0109] In addition, after step S502, the method may further include: if the position difference between the current text content and the current operation object area is less than a preset difference threshold, combining the current text content with preset text content to form combined text content; and constructing the correspondence between the combined text content and the area position of the current operation object area.

[0110] If the difference between the center position of the current text and the center position of the area being manipulated is small—for example, less than a pre-set threshold—it indicates that the current text content may be located within the area being manipulated. This typically occurs with buttons on a display page. For instance, the center coordinates of the text "Search" on a search button are close to the center coordinates of the button itself. Therefore, if the difference between the center position of the current text and the center position of the area being manipulated is small, the current text content can be combined with a pre-defined text content to create a combined text content. This combined text content can then establish a correspondence between its location and the area being manipulated. For example, if the pre-defined text content is "button," the terminal can combine the current text content "Search" with the pre-defined text content "button" to create the combined text content "Search Button," and establish a correspondence between the "Search Button" and the area being manipulated.

[0111] In this embodiment, if the position difference between the current text center position and the regional center position of the current operation object area is small, the terminal can also combine the current text content and the preset text content to obtain combined text content, thereby constructing a correspondence between the combined text content and the regional position of the current operation object area. Through the above process, the correspondence between the pre-constructed text content and the regional position can be made more accurate.

[0112] In one embodiment, such as Figure 6 As shown, step S401 may further include:

[0113] Step S601: Perform contour recognition on the page image of the display page, obtain the text images contained in the page image based on the contour recognition result, and obtain the text position of each text image in the display page.

[0114] In this embodiment, the text image can be obtained by performing contour recognition on the page image. This process can first convert the page image of the displayed page into a grayscale image, and then perform binarization, dilation and erosion on the grayscale image. After dilation, the text image contained in the page image and the position coordinates of the text image in the page image can be obtained by using a contour recognition algorithm, such as the findcontour contour recognition algorithm provided by OpenCV.

[0115] Step S602: Input each text image into a pre-trained text recognition model, and extract the text features of the text images and the text order of each text image through the text recognition model;

[0116] Step S603: Based on text features and text order, obtain the text content corresponding to the text image.

[0117] The text recognition model is a pre-trained neural network model used to identify the text content contained in a text image. Text features refer to the image features extracted from the text image, and text order refers to the image sequence of the text image. In this embodiment, the neural network model used to recognize the text content, i.e., the text recognition model, can be a CRNN model. The input of this model can be a sequence of feature vectors from the text image. Through this model, the text features of the text image can be extracted first to obtain the corresponding feature map. Then, Bi-LSTM can be used to extract the sequential association information of the text. This information can be used with the feature map to form a text feature vector sequence, thereby obtaining the text content corresponding to the text image.

[0118] In this embodiment, the terminal can perform contour recognition on the page image of the displayed page to obtain the included text image and the position of the text image on the displayed page. Then, it can use a text recognition model to obtain the text content of the text image. Through the above process, the recognition accuracy of text content and text position can be improved.

[0119] In one embodiment, the test operation text includes input operation text, and the multiple operation instruction information bound to the test operation text includes: click instruction information and input instruction information; step S103 may further include: executing a click instruction at the operation object location on the display page, and obtaining the data to be input according to the data table bound to the operation object location; and inputting the data to be input to the operation object location.

[0120] In this embodiment, the test operation text can be text representing an input operation, and the text can be pre-bound with click instruction information and input instruction information. Since when performing an input operation, it is usually necessary to click on the input area before input can be performed, the pre-bound operation instruction information for the input operation text can include click instruction information and input instruction information.

[0121] Specifically, if the test operation text obtained by the terminal is input operation text, then the multiple operation instruction information bound to it can include click instruction information and input instruction information. Then, the click instruction can be executed first at the operation object position on the display page. After that, the data to be input can be obtained from the data table bound to the object position, i.e. the data to be input, and then the input operation can be executed to input it into the operation object position.

[0122] For example, a test text might be "Enter Account," with the corresponding test operation text being "Enter," which is an input operation text. The test object text is "Account," and the corresponding operation object location could be the account input box. Therefore, to execute the input operation, a click operation could be triggered first at the account input box location. Then, one account information could be retrieved from the data table bound to the account input box—for example, an account data table storing multiple account information—as the data to be entered. This account information could then be entered into the account input box to achieve account input.

[0123] In this embodiment, if the test operation text is input operation text, the terminal can first execute a click command at the operation object location on the display page, and after obtaining the data to be input from the data table bound to the operation object location, it can then perform an input operation on the data to be input and input it to the operation object location, thereby realizing data input and further improving the efficiency of data input.

[0124] In addition, the automated testing interface can also display object description information of each test operation object contained in the business system's display page. Before obtaining the input data based on the data table bound to the operation object's location, it can also include: responding to the data table binding operation triggered by the target object description information in the automated testing interface, obtaining the target data table corresponding to the data table binding operation; and binding the operation object location of the test operation object described by the target object description information to the target data table.

[0125] Among them, the object description information is a text description information used to describe each test operation object. For example, it can be the object name information of the test operation object, such as "account", "card number", etc. This information can be pre-bound to the business system to be tested. After the user selects the business system to be tested, the corresponding object description information can be displayed in the automated test interface. Alternatively, it can be obtained by recognizing the display page of the business system. Through image recognition, the object description information of the test operation objects contained in the display page can be identified.

[0126] Data table binding is used to bind test operation objects to their corresponding data tables. Since inputting data at the location of a test operation object requires querying the corresponding data table, users need to pre-build the mapping between these locations. This mapping can be achieved through the automated testing interface. Specifically, users can trigger the data table binding operation by selecting the target data table from the displayed object description information (the description of the object to be bound). The terminal can then build the binding relationship between the location of the test operation object described in the description and the target data table.

[0127] For example, the automated testing interface can contain object description information such as "account" and "card number" to represent the account and card number respectively. If the user needs to bind the data table for the account input, the user can trigger the data table binding operation on the "account" text displayed in the automated testing interface. The data table bound to this binding operation can be the account table. Therefore, the terminal can build the account table and the operation object position corresponding to the account, that is, the binding operation of the account input box.

[0128] In this embodiment, the terminal can also display the object description information of the test operation object in the automated testing interface, so that the user can directly bind the operation object location of the test operation object to the target data table in the automated testing interface, thereby reducing the operation steps of data table binding and improving the binding efficiency of data table.

[0129] In one embodiment, an automated testing tool based on image recognition and natural language processing is also provided. This tool utilizes cross-sectional image morphological manipulation algorithms and CRNN text recognition algorithms to identify text, and contour detection algorithms to identify input and output regions in images. It then uses natural language processing to generate computer code instructions and allows users to pre-define SQL data query conditions. These steps enable an automated testing tool, significantly lowering the barrier to entry for end-to-end testing, saving considerable time on data preparation and understanding functional business relationships, and improving the efficiency of testing tasks. Figure 7 As shown, the specific steps may include:

[0130] Step 1: Image Recognition

[0131] The process of image recognition can be as follows: Figure 8 As shown, image recognition includes a text location detection step, a text recognition step, a graphic detection step, and a text-graphic mapping step. The text recognition step includes three processes: feature extraction, obtaining preceding and following text information, and transcription recognition.

[0132] Step 1-1: Since the gate timer interface of the business system is relatively simple, basically consisting of text, input boxes, output boxes, and checkboxes, it does not contain much interfering information. Therefore, it falls under the category of text recognition in a simple scenario and can be processed using image morphology operations in computer vision. The specific process is as follows:

[0133] (1) Convert the interface image to a grayscale image to obtain the grayscale value of each pixel.

[0134] (2) Binarize the grayscale image to obtain a binary image, that is, the image only contains pixels with pixel values ​​of 0 and 255.

[0135] (3) Expansion connects adjacent areas, making the outline more prominent.

[0136] (4) Erosion removes details and highlights the outline information.

[0137] (5) It expands again, making the outline more obvious.

[0138] (6) Use the findcontour contour recognition algorithm provided by OpenCV to obtain the coordinate values ​​of the text region and store the coordinates.

[0139] (7) Based on the image coordinates, crop and save the image and name it using the coordinates. This step aims to obtain the relationship between the cropped image and its coordinates in the original image.

[0140] Steps 1-2: The core steps of text recognition can be summarized as follows:

[0141] Step 1-2-1: As shown in Table 1, feature extraction uses a CNN network structure containing 7 convolutional layers and 4 pooling layers. A standard CNN model has multiple convolutional layers, pooling layers, and fully connected layers. The purpose of convolution and pooling is to extract features from the original image, while the purpose of fully connected layers is to extract the correlations between these features for segmentation or classification. However, in CRNN, the purpose of CNN is feature extraction, not segmentation or classification, so fully connected layers are unnecessary. The image is then used for feature extraction, reducing its size and increasing its channels. The input height is 32, and the width is a fixed value, chosen here as 160. Because the last two pooling layers changed from 2×2 to 1×2, the image height is halved four times, and the width is halved twice. The input and output dimensions (channel, height, width) are (1, 32, 160) and (512, 1, 40), respectively, resulting in 512 feature maps with a height of 1 and a width of 40 after convolution. Since the input image has a fixed height of 32, the cropped image needs to be stretched beforehand.

[0142] Table 1. CRNN Model Network Parameters

[0143]

[0144]

[0145] Where K, s, and p represent the kernel size, stride, and padding size, respectively.

[0146] Step 1-2-2: Obtaining the contextual information of the text using Bi-LSTM. The input to the RNN cannot directly use the feature maps obtained from the CNN; it needs to be adjusted. A sequence of feature vectors needs to be extracted from the feature maps generated by the CNN. Each feature vector is obtained column-by-column on the feature map, meaning each column contains 512-dimensional features. The i-th feature vector is a sequence formed by connecting the pixels in the i-th column of the feature map. These feature vector sequences can then be used as the input to the RNN, with each feature vector serving as the input to the RNN at the first time step.

[0147] Because RNNs suffer from the vanishing gradient problem and cannot capture more contextual information, CRNN uses LSTM. LSTM's special design allows it to capture long-range dependencies. LSTM is unidirectional, using only past information; however, text recognition requires information in both forward and backward directions, so a bidirectional LSTM is used to capture both preceding and following text information. The input feature vector can be understood as a small region in the original image. The goal of RNN is to predict which character this rectangular region represents, i.e., based on the input feature vector, to predict the softmax probability distribution of all characters. This is a vector with a length equal to the number of character categories, used as input to CTC, since there is an input feature vector x at each time step. T Output a probability distribution Y for all characters. T Therefore, the output is a posterior probability matrix consisting of 40 vectors with a length equal to the number of character categories. This posterior probability matrix is ​​then input into the transcription layer.

[0148] Steps 1-2-3: This layer is the transcription layer, which converts the predictions generated by the RNN into label sequences. After obtaining the label sequences, decoding them yields the recognition results. The training dataset uses the Caffe-ocr Chinese synthetic dataset, containing 3.6 million images. The data utilizes a Chinese corpus (news + classical Chinese texts), randomly generated through variations in font, size, grayscale, blur, perspective, and stretching. The dictionary contains 5990 characters, including Chinese characters, punctuation, English letters, and numbers (based on corpus character frequency statistics, merging full-width and half-width characters). Each sample has a fixed 10 characters, randomly extracted from sentences in the corpus. The dataset is divided into training and validation sets in a 9:1 ratio, with approximately 60,000 images in the test set.

[0149] After recognizing the text and coordinates on the counter interface, a relationship is established. The data format is {"text":[coordinates]}, for example, {"account":[23,67]}, where the coordinates are the horizontal and vertical offsets from the top left corner of the interface image.

[0150] Steps 1-3: Since the image contains selection boxes, buttons, input boxes, output boxes, etc., it's convenient that these areas are represented by rectangles in the business system's display interface. Therefore, we only need to detect the rectangles in the interface image to obtain these rectangles. Detecting rectangles uses a similar operation to the text detection described above:

[0151] (1) First, perform median filtering to remove noise.

[0152] (2) Extract different color channels to detect rectangles in sequence.

[0153] (3) Binarize each channel.

[0154] (4) Use the findCountour algorithm to find the outline to obtain the coordinates of the rectangle, which are the coordinates of the four corners of the rectangle.

[0155] Steps 1-4: After obtaining the rectangle's coordinates, it's necessary to associate the text description with the rectangular area. The rules for establishing this association are as follows:

[0156] (1) First calculate the center coordinates of each rectangle, then calculate the center coordinates of each text description. Take the difference between the two coordinates. When the absolute value of the difference is less than the threshold (the threshold can be modified, and the threshold is set to 3 in this article), it can be understood that the text description is in the rectangle. This situation only occurs for buttons. Therefore, you can add "button" to the text description and then establish a connection with the coordinates of the rectangle. For example, ["Query button", [45, 78]] means that the query button is located at a position 45 degrees horizontally and 78 degrees vertically on the interface.

[0157] (2) Since all input and output boxes on this interface are located to the left of the description field of this area, the difference between the center coordinates of the rectangle and the center coordinates of the text can be taken. The difference can be positive or negative. The absolute value of the difference is used to determine whether the area is a button area. This part needs to be greater than the above threshold. The area with the smallest difference is taken as the area corresponding to the description field. For example, ["Account", [12, 22]] means that the center coordinates of the input area corresponding to the account are 12 horizontally offset and 22 vertically offset.

[0158] Step 2: Natural Language Processing

[0159] like Figure 9 As shown, the natural language processing module includes word segmentation, syntactic analysis, and tree structure generation steps.

[0160] Step 2-1: For raw natural language, word segmentation is required first. The purpose of word segmentation is to separate each word in the natural language description for subsequent processing, making it easier to understand the meaning of the language description. Word segmentation uses the open-source Chinese dictionary THUOCL. Common segmentation algorithms include forward longest matching, backward longest matching, and bidirectional longest matching. The principle of forward longest matching is: starting from the beginning of the sentence, matching the longest meaningful word, and continuing until the end of the sentence. Backward longest matching is the opposite, starting from the end of the sentence and matching backward. Bidirectional longest matching is a compromise between forward and backward longest matching. It is a complex rule set that combines the two matching algorithms, and the process is as follows: forward matching and backward longest matching are performed simultaneously. If the number of occurrences is different, the one with fewer occurrences is returned; if the number of occurrences is the same, the one with fewer words is returned; when the number of words is also the same, the result of backward longest matching is returned first. In this embodiment, the bidirectional longest matching algorithm is used for word segmentation of the sentence.

[0161] Step 2-2: After obtaining the word segmentation results, syntactic analysis is needed to understand the meaning of the sentence. The purpose is to identify the subject, predicate, object, and the relationships between the segmented words. The open-source StanfordParser can perform syntactic analysis. For the segmented results, the syntactic analysis algorithm is called. The input is "Enter card number, account number, click the query button to query, enter the linked account, click the link button to link the relationship". The syntactic analysis results label and associate the words in the sentence. VV, NN, CC, and PU represent verbs, common nouns, conjunctions, and punctuation marks, respectively. This syntactic analysis result can be shown as follows: Figure 10 As shown.

[0162] Step 2-3: After obtaining the syntactic analysis results, a tree-like storage structure is built based on the part of speech and relationships of each word. As in the example used in Step 2-2, the generated tree structure is as follows:

[0163] [{VV:"Input",NN:["Card Number","Account Number"]},

[0164] {VV:"Click",NN:["Query"]},

[0165] {VV:"Input",NN:["Linked Account"]},

[0166] {VV:"Click",NN:["Connect"]} ]

[0168] Since the system only needs to perform input and click operations when automating operations, this embodiment pre-establishes an operation instruction glossary. Words not in this glossary will not be added to the tree structure. Operation instructions are mainly divided into mouse operations and keyboard operations. Some examples of the established instruction glossary are as follows: ["click", "double-click", "right-click", "scroll", "click the scroll bar", "click both left and right buttons simultaneously"].

[0169] Step 3: Generate operation instructions

[0170] like Figure 11 As shown, the instruction generation module includes a tree structure to natural language instruction conversion step, a natural language instruction preprocessing step, and a natural language code generation instruction step.

[0171] Step 3-1: Before converting to computer code instructions, the tree structure needs to be converted into a natural language instruction sequence. The conversion is performed by traversal, and the traversal result is [["Input", "Account"], ["Input", "Card Number"], ["Click", "Query"], ["Input", "Linked Account"], ["Click", "Linked"]]. An array is used to store the instruction sequence processed for the first time. Since a certain area needs to be clicked before inputting or selecting, this paper pre-sets click instructions for non-click operations that require clicking before generating instructions. For example, ["Input", "Account"] will be further processed into ["Click", "Account Area"] and ["Input", "Account"].

[0172] Step 3-2: After the first instruction conversion of the natural language, further conversion is required. For the instruction corresponding to the click, since we have already established a connection between the recognized text and the input box, button, selection box, etc. during the text recognition process, the natural language instruction can be converted into computer code instruction in this step. For example, ["click", "query button"] is converted into ["click", [23, 45]], where click is the click command, and the following array is the interface coordinates corresponding to the "query button". This array can be passed as a parameter to the pre-written click script to execute the corresponding operation.

[0173] Step 3-3: For the input instructions, since the input data requires corresponding selection, the automated testing tool developed in this paper provides an SQL command writing area. This paper has already written SQL statements for some business-required data and established mapping relationships. When converted into computer code instructions, it only needs to execute the corresponding SQL statement according to the mapping relationship to find the corresponding data. If a business is being executed for the first time and no SQL is pre-defined, testers need to manually write the query SQL conditions. They can choose to save the query, which will then be saved to the previously pre-defined SQL mapping for easy use and subsequent repeated calls. The table records the database name, address, test task, and the corresponding SQL statement for the data under the test task. The data, table names, and field names used here are all anonymized; the actual data in use is not displayed.

[0174] Step 4: System Processing

[0175] The tool described in this embodiment is mainly implemented by executing corresponding mouse and keyboard events based on generated instructions. To make it more convenient for users, a simple visual interface has been developed, providing a test entry point. For example... Figure 12 As shown, the system processing module includes steps for file selection, image recognition, task selection, inputting test points, writing SQL, and displaying execution results.

[0176] Step 4-1: This tool uses a monthly version, recognizing new interface content once a month. You can also manually add an .exe file for recognition during each test; otherwise, the system's default recognition interface for the current month will be used. The recognized text results are displayed in the interface recognition results.

[0177] Step 4-2: Image recognition is invoked, and the results of the text recognition are displayed in the interface.

[0178] Step 4-3: Select the task to be tested. The SQL you enter later will also be saved in the SQL query condition database corresponding to that task.

[0179] Step 4-4: Input the test points and invoke natural language processing.

[0180] Steps 4-5: The recognition results will be displayed on the interface. The returned text is the recognized text, separated by newline characters. If this type of task does not have pre-defined SQL query conditions, you need to add them manually. In the output field, add "#" after the text where you want to add the SQL query conditions, and then enter the SQL statement, such as... Figure 13 As shown, if not manually added, the default preset SQL will be used or an error will be reported. After submission, execution will yield the test results.

[0181] Steps 4-6: Test results are fed back in the execution results.

[0182] This embodiment lowers the barrier to entry for full-process testing, saves a significant amount of time on data preparation and understanding of functional business relationships, and improves the testing efficiency of testing tasks.

[0183] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0184] Based on the same inventive concept, this application also provides an automated testing apparatus for implementing the automated testing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the automated testing apparatus provided below can be found in the limitations of the automated testing method described above, and will not be repeated here.

[0185] In one embodiment, such as Figure 14 As shown, an automated testing device is provided, including: a test text input module 1401, an operation information acquisition module 1402, and an operation result display module 1403, wherein:

[0186] The test text input module 1401 is used to display the automated test interface and obtain the test text information input for the business system to be tested in the automated test interface. Based on the test text information, it obtains at least one test operation text and the test object text corresponding to the test operation text.

[0187] The operation information acquisition module 1402 is used to acquire multiple operation instruction information pre-bound to the test operation text, and to acquire the position of the operation object represented by the test object text in the display page of the business system.

[0188] The operation result display module 1403 is used to execute various operation instructions at the operation object location in the display page of the business system, obtain the operation results of each operation instruction, and display the operation results in the automated test interface.

[0189] In one embodiment, the test text input module 1401 is further configured to retrieve multiple target text segments corresponding to the test text information, and obtain the segmentation type corresponding to each target text segment, as well as the association relationship of each target text segment; retrieve the first text segment whose segmentation type represents a verb type among the target text segments, and if the first text segment belongs to a pre-set operation text segmentation library, use the first text segment as the test operation text; the operation text segmentation library pre-stores multiple text segments corresponding to test operations; according to the association relationship, retrieve the second text segment associated with the test operation text, and use the second text segment as the test object text corresponding to the test operation text; the second text segment is the text segment whose segmentation type represents a noun type among the target text segments.

[0190] In one embodiment, the test text input module 1401 is further configured to obtain multiple first initial text segments corresponding to the test text information through a forward maximum matching algorithm, and to obtain multiple second initial text segments corresponding to the test text information through a reverse maximum matching algorithm; and to obtain the target text segment from the first initial text segments or the second initial text segments.

[0191] In one embodiment, the test text input module 1401 is further configured to obtain the first word segmentation matching iteration number of the first initial text segmentation and the second word segmentation matching iteration number of the second initial text segmentation; if the first word segmentation matching iteration number and the second word segmentation matching iteration number are different, the initial text segmentation with the smaller word segmentation matching iteration number is used as the target text segmentation; if the first word segmentation matching iteration number and the second word segmentation matching iteration number are the same, the first word segmentation count of the first initial text segmentation and the second word segmentation count of the second initial text segmentation are obtained; if the first word segmentation count and the second word segmentation count are different, the initial text segmentation with the smaller word segmentation count is used as the target text segmentation; if the first word segmentation count and the second word segmentation count are the same, the second initial text segmentation is used as the target text segmentation.

[0192] In one embodiment, the automated testing device further includes: a display page recognition module, configured to acquire the text content corresponding to the text image contained in the display page of the business system, and the text position of the text image in the display page; acquire the operation object area contained in the display page of the business system, and the area position of the operation object area in the display page; construct a correspondence between text content and area position based on the text position and area position; and an operation information acquisition module 1402, further configured to acquire the target area position corresponding to the text content of the test object text based on the correspondence between text content and area position, and use the target area position as the operation object position.

[0193] In one embodiment, the display page recognition module is further used to obtain the current text content, the current text center position corresponding to the current text content, and the region center position of each operation object region in the display page; determine the position difference between the current text center position and the center positions of each region, and take the operation object region with the smallest position difference as the current operation object region corresponding to the current text content; and construct the correspondence between the current text content and the region position of the current operation object region.

[0194] In one embodiment, the display page recognition module is further configured to combine the current text content with preset text content to form combined text content when the position difference between the current text content and the current operation object area is less than a preset difference threshold; and to construct a correspondence between the combined text content and the regional position of the current operation object area.

[0195] In one embodiment, the display page recognition module is further configured to perform contour recognition on the page image of the display page, obtain the text images contained in the page image based on the contour recognition result, and obtain the text position of each text image in the display page; input each text image into a pre-trained text recognition model, extract the text features of the text image and the text order of each text image through the text recognition model; and obtain the text content corresponding to the text image based on the text features and the text order.

[0196] In one embodiment, the test operation text includes input operation text, and the multiple operation instruction information bound to the test operation text includes: click instruction information and input instruction information; the operation result display module 1403 is further used to execute the click instruction at the operation object position in the display page, and obtain the data to be input according to the data table bound to the operation object position; and input the data to be input to the operation object position.

[0197] In one embodiment, the automated testing interface also displays object description information of each test operation object contained in the display page of the business system; the operation result display module 1403 is also used to respond to the data table binding operation triggered by the target object description information in the automated testing interface, obtain the target data table corresponding to the data table binding operation, and bind the operation object position of the test operation object described by the target object description information to the target data table.

[0198] Each module in the aforementioned automated testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0199] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an automated testing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0200] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0201] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0202] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0203] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0205] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0206] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An automated testing method, characterized in that, The method includes: Display the automated testing interface and obtain the test text information input for the business system to be tested in the automated testing interface. Obtain at least one test operation text and the test object text corresponding to the test operation text based on the test text information. Obtain multiple operation instruction information pre-bound to the test operation text, and obtain the position of the operation object corresponding to the test operation object represented by the test object text in the display page of the business system; The operation instructions are executed at the operation object location on the display page of the business system, the operation results of each operation instruction are obtained, and the operation results are displayed on the automated testing interface. The step of obtaining at least one test operation text and the test object text corresponding to the test operation text based on the test text information includes: Obtain multiple target text segments corresponding to the test text information, and obtain the segmentation type corresponding to each target text segment, as well as the association relationship between each target text segment; The first text segment representing the verb type in the target text segmentation is obtained, and if the first text segment belongs to a pre-defined operation text segmentation library, the first text segment is used as the test operation text; the operation text segmentation library pre-stores text segments corresponding to multiple test operations. Based on the association relationship, a second text segment associated with the test operation text is obtained, and the second text segment is used as the test object text corresponding to the test operation text; the second text segment is the text segment of the target text segment whose segmentation type represents the noun type.

2. The method according to claim 1, characterized in that, The step of obtaining multiple target text segmentations corresponding to the test text information includes: The forward maximum matching algorithm is used to obtain multiple first initial text segments corresponding to the test text information, and the reverse maximum matching algorithm is used to obtain multiple second initial text segments corresponding to the test text information. The target text segment is obtained from either the first initial text segment or the second initial text segment.

3. The method according to claim 2, characterized in that, The step of obtaining the target text segment from the plurality of first initial text segments or the plurality of second initial text segments includes: Obtain the first word segmentation matching iteration number of the first initial text segmentation, and the second word segmentation matching iteration number of the second initial text segmentation; If the number of the first word segmentation matching iterations is different from the number of the second word segmentation matching iterations, the initial text segment with the smaller number of word segmentation matching iterations will be used as the target text segment. If the number of iterations for the first word segmentation matching is the same as the number of iterations for the second word segmentation matching, obtain the first word segmentation count of the first initial text segmentation and the second word segmentation count of the second initial text segmentation. If the number of the first segmented words is different from the number of the second segmented words, the initial text segment with the smaller number of segmented words shall be used as the target text segment. If the number of the first segmented words is the same as the number of the second segmented words, the second initial text segmented words are used as the target text segmented words.

4. The method according to claim 1, characterized in that, Before obtaining the position of the test operation object represented by the test object text in the corresponding operation object position on the display page of the business system, the process further includes: Obtain the text content corresponding to the text image contained in the display page of the business system, and the text position of the text image in the display page; Obtain the operation object area contained in the display page of the business system, and the location of the operation object area in the display page; Based on the text location and the region location, construct the correspondence between the text content and the region location; The step of obtaining the location of the operation object in the display page of the business system corresponding to the test operation object represented by the test object text includes: Based on the correspondence between the text content and the region location, the target region location corresponding to the text content of the test object is obtained, and the target region location is used as the operation object location.

5. The method according to claim 4, characterized in that, The step of constructing the correspondence between the text content and the region position based on the text position and the region position includes: Obtain the current text content, the current text center position corresponding to the current text content, and the center position of each operation object area in the display page; Determine the position difference between the current text center position and the center positions of each region, and take the operation object region with the smallest position difference as the current operation object region corresponding to the current text content; Construct the correspondence between the current text content and the region location of the current operation object area.

6. The method according to claim 5, characterized in that, After determining the region of operation with the smallest position difference as the current operation region corresponding to the current text content, the method further includes: If the positional difference between the current text content and the current operation object area is less than a preset difference threshold, the current text content is combined with preset text content to form combined text content. Construct the correspondence between the combined text content and the regional position of the currently operated object area.

7. The method according to claim 4, characterized in that, The step of obtaining the text content corresponding to the text image contained in the display page of the business system, and the text position of the text image in the display page, includes: Contour recognition is performed on the page image of the display page, and the text images contained in the page image are obtained based on the contour recognition results. The text positions of each text image in the display page are also obtained. Each of the text images is input into a pre-trained text recognition model, and the text features of the text images and the text order of each text image are extracted by the text recognition model. Based on the text features and the text order, the text content corresponding to the text image is obtained.

8. The method according to claim 1, characterized in that, The test operation text includes input operation text, and the multiple operation instruction information bound to the test operation text includes: click instruction information and input instruction information; The execution of each operation instruction information at the operation object location on the display page of the business system includes: A click command is executed at the location of the operation object on the display page, and the data to be input is obtained from the data table bound to the location of the operation object; The data to be input is entered into the location of the operation object.

9. The method according to claim 8, characterized in that, The automated testing interface also displays object description information for each test operation object contained in the display page of the business system; Before retrieving the input data from the data table bound to the location of the operation object, the method further includes: In response to a data table binding operation triggered by the target object description information in the automated testing interface, the target data table corresponding to the data table binding operation is obtained. The location of the test operation object described in the target object description information is bound to the target data table.

10. An automated testing device, characterized in that, The device includes: The test text input module is used to display the automated testing interface and obtain the test text information input in the automated testing interface for the business system to be tested. Based on the test text information, it obtains at least one test operation text and the test object text corresponding to the test operation text. The operation information acquisition module is used to acquire multiple operation instruction information pre-bound to the test operation text, and to acquire the position of the operation object corresponding to the test operation object represented by the test object text in the display page of the business system. The operation result display module is used to execute each operation instruction information at the operation object location in the display page of the business system, obtain the operation result of each operation instruction information, and display the operation result in the automated test interface. The test text input module is further used for: The process involves obtaining multiple target text segments corresponding to the test text information, acquiring the segmentation type of each target text segment, and the association relationship between each target text segment; acquiring a first text segment whose segmentation type represents a verb, and using the first text segment as the test operation text if it belongs to a pre-defined operation text segmentation library; the operation text segmentation library pre-stores multiple text segments corresponding to test operations; acquiring a second text segment associated with the test operation text according to the association relationship, and using the second text segment as the test object text corresponding to the test operation text; the second text segment is a text segment whose segmentation type represents a noun among the target text segments.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images