Application system testing method and device based on multi-modal retrieval, equipment and medium

Abstract

This disclosure provides embodiments of a method, apparatus, device, and medium for testing application systems based on multimodal retrieval. One specific implementation of the method includes: acquiring a detailed design document; performing image conversion processing on the detailed design pages to obtain detailed design page diagrams; performing feature extraction processing on each obtained detailed design page diagram to obtain detailed design feature information; determining a target historical page diagram set; determining each target historical test point set; generating each test point set; generating each test case set corresponding to each test point set; and executing system testing tasks on the application system. This implementation can reduce the waste of computational resources when testing application systems.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to testing methods, apparatus, devices, and media for application systems based on multimodal retrieval. Background Technology

[0002] Application system testing is a method for testing the functionality and performance of an application. Currently, the common approach to application system testing is as follows: First, extract the test points from the detailed design document. Then, automatically populate the detailed design document and each test point into a pre-defined test case template to obtain test cases. Finally, write scripts based on the generated test cases and use these scripts to test the application.

[0003] However, when using the above methods to test application systems, the following technical problems often arise: Extracting test points directly from detailed design documents can easily lead to a high false positive rate for each test point, resulting in numerous redundant steps in the test cases generated based on these test points. Consequently, this leads to a waste of computational resources when testing the application system based on these redundant steps.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide application system testing methods, apparatuses, electronic devices, and computer-readable media based on multimodal retrieval to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide an application system testing method based on multimodal retrieval. The method includes: in response to receiving test information sent by an application system, obtaining a detailed design document, wherein the detailed design document includes various detailed design pages; for each detailed design page included in the detailed design document, performing image conversion processing on the detailed design page to obtain a detailed design page diagram; performing feature extraction processing on the obtained detailed design page diagrams to obtain various detailed design feature information; for each detailed design feature information, determining a target historical page diagram set based on a vector database and the detailed design feature information; determining a target historical test point set based on the determined target historical page diagram sets; generating various test point sets based on the target historical page diagram sets, the target historical test point sets, and the detailed design page diagrams; generating various test case sets corresponding to the various test point sets based on the various test point sets, the detailed design page diagrams, the target historical test point sets, and the target historical page diagrams; and performing system testing tasks on the application system based on the various test case sets.

[0008] Secondly, some embodiments of this disclosure provide an application system testing apparatus based on multimodal retrieval. The apparatus includes: an acquisition unit configured to acquire a detailed design document in response to receiving test information sent by an application system, wherein the detailed design document includes various detailed design pages; an image conversion unit configured to perform image conversion processing on each detailed design page included in the detailed design document to obtain a detailed design page image; a feature extraction unit configured to perform feature extraction processing on the obtained detailed design page images to obtain detailed design feature information; and a first determination unit configured to determine each detailed design feature information. The system comprises: a first generation unit, configured to determine target historical page graph sets based on a vector database and the aforementioned detailed design feature information; a second determination unit, configured to determine target historical test point sets based on the determined target historical page graph sets; a first generation unit, configured to generate test point sets based on the aforementioned target historical page graph sets, target historical test point sets, and detailed design page graphs; a second generation unit, configured to generate test case sets corresponding to the aforementioned test point sets based on the aforementioned test point sets, detailed design page graphs, target historical test point sets, and target historical page graph sets; and an execution unit, configured to perform system testing tasks on the aforementioned application system based on the aforementioned test case sets.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above embodiments of this disclosure have the following beneficial effects: the application system testing method based on multimodal retrieval in some embodiments of this disclosure can reduce the waste of computing resources. Specifically, the reason for the waste of computing resources is that the test points directly extracted from the detailed design document are prone to a high false detection rate, resulting in many redundant steps in the test cases generated based on the test points, and thus wasting computing resources when testing the application system based on the redundant steps in the test cases. Based on this, the application system testing method based on multimodal retrieval in some embodiments of this disclosure firstly obtains a detailed design document in response to receiving test information sent by the application system, wherein the detailed design document includes various detailed design pages. Thus, the detailed design document to be processed can be obtained. Secondly, for each detailed design page in the detailed design document, image conversion processing is performed on the detailed design page to obtain a detailed design page diagram. Thus, each page in the detailed design document can be converted into an image. Then, feature extraction processing is performed on the obtained detailed design page diagrams to obtain detailed design feature information. Thus, various features of each detailed design page diagram can be obtained. Then, for each of the detailed design feature information mentioned above, a target historical page atlas is determined based on the vector database and the detailed design feature information. This yields the target historical page atlas corresponding to the detailed design feature information. Next, based on the determined target historical page atlases, a target historical test point set is determined. This yields the target historical test point set. Then, based on the target historical page atlases, the target historical test point sets, and the detailed design page atlases, a test point set is generated. This yields the test point set. Finally, based on the test point sets, the detailed design page atlases, the target historical test point sets, and the target historical page atlases, a test case set corresponding to each test point set is generated. This yields the test case set. Finally, based on the test case sets, system testing tasks are performed on the application system. This allows for the testing of the application system. Because it can filter out the historical page sets and historical test point sets of each target based on the feature vectors of the detailed design document, and then generate new test points based on the differences between the detailed design document and the historical page sets of each target, and then generate test cases based on the generated test points, instead of directly generating test points and test cases from the detailed design document, it can reduce the false positive rate of each generated test point, reduce redundant steps in the test cases, and thus reduce the waste of computing resources when testing the application system based on the test cases. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the application system testing method based on multimodal retrieval according to the present disclosure; Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the testing apparatus for an application system based on multimodal retrieval according to this disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1A flowchart 100 is shown, illustrating some embodiments of an application system testing method based on multimodal retrieval according to this disclosure. This application system testing method based on multimodal retrieval includes the following steps: Step 101: In response to receiving test information from the application system, obtain the detailed design document.

[0021] In some embodiments, the execution entity (e.g., a computing device) of the application system testing method based on multimodal retrieval can obtain a detailed design document in response to receiving test information sent by the application system. The test information can be information used to characterize the system to be tested. For example, the test information could be "Please start system testing." The application system can be the application system to be tested. The application system has a corresponding system name. The system name can be the name of the application system. The detailed design document can be a document used to describe the specific operation steps of the application during the application system development process. For example, the content of the detailed design document could be "Step 1, click the 'Start' button; Step 2, enter the username; Step 3, click the 'Login' button." The detailed design document can include various detailed design pages. Each detailed design page can be a page in the detailed design document. Each detailed design page can contain a UI component area. The UI component area can be an image of a UI component. The execution entity can be a server.

[0022] In practice, the aforementioned executing entity can send the system name corresponding to the application system and preset data acquisition information to the target server. This data acquisition information can be used to obtain detailed design documents. For example, the data acquisition information could be "Please send detailed design documents." The target server can be a server used to send detailed design documents. Then, the entity can receive the detailed design documents sent by the target server.

[0023] Optionally, prior to step 101, the aforementioned execution entity may also perform the following steps: The first step is to acquire various historical test data. Each piece of historical test data can be data used to test the application or system before the current time. Each piece of historical test data can include historical detailed design documents, historical test point groups, and historical test cases. The historical detailed design documents can be detailed design documents acquired before the current time. These documents can include various historical detailed design pages. Each historical detailed design page can be a single page within the historical detailed design document. Each historical detailed design page can contain UI component areas. Each historical detailed design page corresponds to a historical test point. Each historical test point in the historical test point group can be a test point corresponding to the historical detailed design document. These test points can be information used to characterize the functions that need to be tested when testing the application. The historical test cases can be test cases corresponding to the historical detailed design documents.

[0024] In practice, the aforementioned implementing entity can obtain the various historical test data from a pre-set historical test database. This historical test database can be a database used to store historical test data.

[0025] The second step is to generate various historical detailed feature data groups based on the historical test data and the historical detailed design documents mentioned above.

[0026] Each historical detailed feature data point in the aforementioned historical detailed feature data groups can be data used to characterize the features corresponding to historical detailed design pages in the historical detailed design document. Each historical detailed feature data point in the aforementioned historical detailed feature data groups can include historical detailed feature information and historical image patch feature information groups. The aforementioned historical detailed feature information can be the feature vector corresponding to the historical detailed design page in the historical detailed design document. The aforementioned historical image patch feature information can be the feature vector corresponding to a local region of the historical detailed design page in the historical detailed design document. Each historical detailed feature data point in the aforementioned historical detailed feature data groups corresponds to a historical test point.

[0027] The third step is to construct a vector database based on the aforementioned historical detailed feature data groups, the historical test point groups included in the aforementioned historical test data, and the historical test cases. This vector database can be a relational database used to store the aforementioned historical detailed feature data groups, the historical test point groups included in the aforementioned historical test data, and the historical test cases.

[0028] In practice, firstly, a relational database can be created as the initial database using database creation commands. These commands can be any commands used to create a database, such as "CREATE DATABASE". Then, four tables can be created in the initial database using table creation commands to update the initial database and to designate the updated initial database as the database to be stored. These table creation commands can be any commands used to create tables within the database, such as "CREATE TABLE".

[0029] Then, for each historical detailed feature data in the aforementioned historical detailed feature data groups, firstly, the executing entity can determine the historical test point corresponding to the aforementioned historical detailed feature data as the historical test point to be matched. Secondly, it can determine the historical test data corresponding to the aforementioned historical test point to be matched as the historical test data to be matched. Then, it can determine the historical test cases included in the aforementioned historical test data to be matched as the historical test cases to be matched. Then, the aforementioned historical detailed feature data can be stored in the first table of the database to be stored using a data insertion instruction, and the ID of the aforementioned historical detailed feature data returned by the database to be stored can be used as the feature ID, in order to update the database to be stored. The aforementioned data insertion instruction can be any instruction capable of storing data in the database. For example, the aforementioned data insertion instruction can be "INSERT INTO". Then, the aforementioned historical test point to be matched can be stored in the second table of the updated database to be stored using the aforementioned data insertion instruction, and the ID of the aforementioned historical test point to be matched can be used as the historical test point ID, in order to update the database to be stored. Then, the aforementioned historical test cases to be matched can be stored in the third table of the updated database using the data insertion command described above, obtaining the ID of the aforementioned historical test cases as the historical test case ID, for updating the database. Next, the aforementioned feature ID, the aforementioned historical test point ID, and the aforementioned historical test case ID can be combined into a data group, and the aforementioned data group can be stored in the fourth table of the updated database using the data insertion command, for updating the database. Finally, in response to determining that each historical detailed feature data in each of the aforementioned historical detailed feature data groups has been processed, the updated database can be defined as a vector database.

[0030] In some optional implementations of certain embodiments, the aforementioned execution entity may generate various historical detailed feature data groups based on the various historical detailed design documents included in the aforementioned historical test data through the following steps: For each of the above historical detailed design documents, perform the following steps: The first step is to perform the following steps for each historical detailed design page included in the aforementioned historical detailed design documents: The first sub-step involves generating historical image block feature information groups and historical detailed feature information based on the aforementioned historical detailed design page.

[0031] The second sub-step involves combining the aforementioned historical detailed feature information and the aforementioned historical image block feature information into historical detailed feature data.

[0032] The second step is to combine the obtained historical detailed feature data into a historical detailed feature data group.

[0033] In some optional implementations of certain embodiments, the aforementioned execution entity may generate historical image block feature information groups and historical detailed feature information based on the aforementioned historical detailed design page through the following steps: The first step is to perform image conversion processing on the aforementioned historical detailed design pages to obtain historical detailed design page images. These images can be the corresponding pictures of the historical detailed design pages. In practice, firstly, the implementing entity can use a data conversion tool to convert the historical detailed design pages into PDF format, obtaining PDF versions of the historical detailed design pages as historical PDF pages. Secondly, the aforementioned data conversion tool can be used to convert the historical PDF pages into images, serving as historical detailed design page images. This data conversion tool can be anything capable of converting documents to PDF format and then to images. For example, Smallpdf could be a suitable data conversion tool.

[0034] The second step involves image segmentation of the aforementioned historical detailed design page diagrams to obtain individual historical page image blocks. Each historical page image block can be an image block from the aforementioned historical detailed design page diagram. All the aforementioned historical page image blocks are of the same size.

[0035] In practice, the aforementioned execution entity can use a block function to divide the historical detailed design page diagram into image blocks of equal size, serving as individual historical page image blocks. This block function can be any function capable of dividing an image into blocks. For example, the block function could be the `blockproc` function.

[0036] The third step is to perform feature encoding processing on each of the aforementioned historical page image blocks to obtain the feature information of the historical image blocks.

[0037] The aforementioned historical image patch feature information can be the feature vector corresponding to the historical page image patch. In practice, for each historical page image patch, the execution entity can input the historical page image patch into a pre-trained graph feature extraction model to obtain the historical image patch feature information. The graph feature extraction model can be a neural network model that takes historical page image patches as input and outputs the historical image patch feature information. For example, the graph feature extraction model can be a pre-trained graph convolutional neural network. The pre-training can be a process of fine-tuning the graph convolutional neural network using a set of historical page image patches and a cross-entropy loss function.

[0038] The fourth step involves performing global pooling on the obtained feature information of each historical image patch to obtain detailed historical feature information. This detailed historical feature information can be the feature vector obtained by average pooling the feature information of each historical image patch. In practice, the executing entity can use average pooling technology to perform average pooling on the feature information of each historical image patch to obtain detailed historical feature information.

[0039] The fifth step is to combine the feature information of each historical image block into a historical image block feature information group.

[0040] Step 102: For each detailed design page in the detailed design document, perform image conversion processing on the detailed design page to obtain a detailed design page diagram.

[0041] In some embodiments, the executing entity may perform image conversion processing on each detailed design page included in the detailed design document to obtain a detailed design page diagram. The detailed design page diagram can be an image of the detailed design page. In practice, firstly, the executing entity can use a data conversion tool to convert the detailed design document into PDF format, obtaining a PDF format detailed design document as the converted document. Secondly, for each page included in the converted document, the data conversion tool can convert the page into an image to obtain a detailed design page diagram. The data conversion tool can be a tool capable of converting a document to PDF format and then to an image. For example, the data conversion tool could be Smallpdf.

[0042] Step 103: Perform feature extraction processing on the obtained detailed design page diagrams to obtain the feature information of each detailed design.

[0043] In some embodiments, the execution entity can perform feature extraction processing on the obtained detailed design page diagrams to obtain detailed design feature information. Each detailed design feature information can be a feature vector corresponding to a detailed design page diagram. In practice, for each detailed design page diagram, the execution entity can perform feature extraction processing on the detailed design page diagram using a feature extraction model to obtain detailed design feature information. The feature extraction model can be a neural network model that takes the detailed design page diagram as input and outputs the detailed design feature information corresponding to the detailed design page diagram. For example, the feature extraction model can be a pre-trained graph convolutional neural network. The pre-training can be a process of fine-tuning the graph convolutional neural network using a detailed design page diagram set and a cross-entropy loss function.

[0044] Step 104: For each detailed design feature in each detailed design feature information, determine the target historical page atlas based on the vector database and the detailed design feature information.

[0045] In some embodiments, the execution entity may determine the target historical page atlas for each of the detailed design feature information based on the vector database and the detailed design feature information.

[0046] In addressing the technical problems mentioned above by adopting technical solutions, the following technical issues arise regarding the application scenario: non-stop updates to a large-scale intelligent claims system. Directly extracting test points from detailed design documents can lead to low accuracy of the generated test points, resulting in low reliability of the test cases. This can cause non-core functionalities to consume significant computing power while core functionalities suffer from insufficient testing, necessitating further testing and wasting computing resources. Considering the specific requirements of this application scenario—that during non-stop updates to a large-scale intelligent claims system, the server needs to allocate additional computing resources to test the updated system while ensuring a good user experience—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity described above can determine the target historical page atlas based on the vector database and the detailed design feature information described above through the following steps: The first step is to perform the following steps for each historical detailed feature data in the aforementioned vector database: The first sub-step involves identifying the historical detailed feature information included in the aforementioned historical detailed feature data as the historical feature information to be processed.

[0047] The second sub-step involves comparing the similarity between the aforementioned historical feature information to be processed and the aforementioned detailed design feature information to obtain the first feature similarity.

[0048] The first feature similarity can be the cosine similarity between the historical feature information to be processed and the detailed design feature information.

[0049] The second step is to determine the first target similarity based on the obtained first feature similarities. Each of the aforementioned first target similarities can be a filtered first feature similarity.

[0050] In practice, firstly, the executing entity can arrange the aforementioned first feature similarities into a first feature similarity sequence in descending order. Then, the first preset number of first feature similarities in the first feature similarity sequence can be determined as each first target similarity. The preset number can be a pre-defined value. Here, the specific setting of the preset number is not limited.

[0051] The third step involves determining the historical feature data for each target based on the aforementioned first target similarity scores and historical detailed feature data. Each historical feature data point can be the historical detailed feature data corresponding to the first target similarity score. In practice, the historical detailed feature data corresponding to the aforementioned first target similarities can be used as the historical feature data for each target.

[0052] The fourth step is to determine the historical image block feature information groups included in the historical feature data of each target as the historical feature information groups of each target.

[0053] Fifth, for each of the above target historical feature information groups, perform the following steps: The first sub-step is to determine the first target similarity corresponding to the above-mentioned target historical feature information group as the first target feature similarity.

[0054] The second sub-step involves comparing the similarity between each historical feature information in the aforementioned target historical feature information group and the aforementioned detailed design feature information to obtain a second feature similarity. This second feature similarity can be the cosine similarity between the historical feature information and the detailed design feature information. In practice, for each historical feature information in the aforementioned target historical feature information group, the cosine similarity between the historical feature information and the detailed design feature information can be determined as the second feature similarity.

[0055] The third sub-step involves generating a second target feature similarity corresponding to the aforementioned target historical feature information group, based on the obtained second feature similarities. This second target feature similarity can be the average of the aforementioned second feature similarities. In practice, the executing entity can determine the average of the aforementioned second feature similarities as the second target feature similarity.

[0056] The fourth sub-step involves weighting the first target feature similarity and the second target feature similarity based on preset weighting parameters to obtain a weighted feature similarity corresponding to the target historical feature information group. The weighting parameters can be preset values ​​greater than 0 and less than 1. The weighted feature similarity can be the value obtained by weighting the first target feature similarity and the second target feature similarity.

[0057] In practice, firstly, the difference between the preset value and the aforementioned weighting parameters can be determined as the weighting coefficient. Then, the product of the aforementioned weighting parameters and the similarity of the first target feature can be determined as the first weighted value. The weighting coefficient and the similarity of the second target feature can be used as the second weighted value. Finally, the sum of the first and second weighted values ​​can be determined as the weighted feature similarity.

[0058] Step 6: Based on the generated weighted feature similarities, determine the target historical page map. This target historical page map can be a detailed historical design page map determined based on the weighted feature similarities.

[0059] In practice, the weighted feature similarity with the highest value among the aforementioned weighted feature similarities can be determined as the target weighted feature similarity. Secondly, the historical detailed feature data corresponding to the target weighted feature similarity can be determined as the target historical detailed feature data. Then, the historical detailed design page diagram corresponding to the target historical detailed feature data can be determined as the target historical page diagram.

[0060] The above technical solution and its related content, combined with step 108, serve as an inventive point of this disclosure, solving the problem of "waste of computing resources." Factors leading to wasted computing resources often include: directly extracting test points from detailed design documents can easily result in low accuracy of the generated test points, leading to low reliability of the test cases generated based on these test points. This can cause non-core functionalities to consume a large amount of computing power, while core functionalities may lack sufficient computing power. Insufficient testing of core functionalities can lead to the need for further testing, resulting in wasted computing resources. Solving these factors can reduce wasted computing resources. To achieve this effect, this disclosure firstly performs the following steps for each historical detailed feature data in the vector database: Secondly, the historical detailed feature information included in the historical detailed feature data is determined as historical feature information to be processed. Then, a similarity comparison is performed between the historical feature information to be processed and the detailed design feature information to obtain a first feature similarity. Thus, the cosine similarity between the historical feature information to be processed and the detailed design feature information can be obtained. Then, based on the obtained first feature similarities, each first target similarity is determined. Therefore, the first feature similarity with the largest value can be obtained. Then, based on the first target similarity and the historical detailed feature data, the historical feature data of each target is determined. Thus, the historical feature data of each target that needs to be processed can be selected based on the first target similarity. Then, the historical image block feature information groups included in the historical feature data of each target are determined as the historical feature information groups of each target. Thus, the historical feature information groups of each target are obtained. Then, for each historical feature information group in the historical feature information groups, the following steps are performed: The first target similarity corresponding to the historical feature information group is determined as the first target feature similarity. Then, for each historical feature information in the historical feature information group, the historical feature information and the detailed design feature information are compared for similarity to obtain the second feature similarity. Thus, the cosine similarity between the historical feature information and the detailed design feature information is obtained. Then, based on the obtained second feature similarities, the second target feature similarity corresponding to the historical feature information group is generated. Thus, the second target feature similarity is obtained. Then, based on preset weighting parameters, the similarity of the first target feature and the similarity of the second target feature are weighted to obtain the weighted feature similarity corresponding to the target historical feature information group. From this, the weighted cosine similarity can be obtained. Finally, based on the generated weighted feature similarities, the target historical page image is determined. Thus, the target historical page image can be selected.Because it can filter out target historical page diagrams that are highly similar to the detailed design page diagrams corresponding to the detailed design feature information in terms of structure and details through two similarity comparisons, and then generate test points and test cases based on the historical test points and test cases corresponding to the target historical page diagrams, it can improve the accuracy and reliability of the generated test points and test cases. It can reduce the probability of performing a large number of tests on non-core functional points while undertesting core functional points, and reduce the probability of needing to spend computing resources to retest due to insufficient testing of core functional points, thus reducing the waste of computing resources.

[0061] Step 105: Based on the determined historical page atlases of each target, determine the historical test point set of each target.

[0062] In some embodiments, the aforementioned execution entity may determine each target historical test point set based on the determined target historical page atlas.

[0063] In this context, each target historical test point in the aforementioned target historical test point sets can be a historical test point corresponding to a target historical page image. In practice, for each target historical page image set in the aforementioned target historical page image sets, the historical test points corresponding to those images can be determined as target historical test points. Then, these determined target historical test points can be defined as the target historical test point sets corresponding to the aforementioned target historical page image sets. Thus, each target historical test point set can be determined.

[0064] Step 106: Generate each test point set based on each target's historical page atlas, each target's historical test point set, and each detailed design page atlas.

[0065] In some embodiments, the execution entity may generate various test point sets based on the various target historical page atlases, the various target historical test point sets, and the various detailed design page diagrams.

[0066] In some optional implementations of certain embodiments, the execution entity may generate various test point sets based on the various target historical page atlases, the various target historical test point sets, and the various detailed design page diagrams through the following steps: The first step is to perform the following steps for each target historical page image set in the above target historical page image sets: The first sub-step is to identify the detailed design page diagrams corresponding to the target historical page diagram set in each of the above detailed design page diagrams as the page diagrams to be processed.

[0067] The second sub-step involves performing the following steps for each target historical page image in the aforementioned target historical page image set: Sub-step one: Determine the target historical test points corresponding to the target historical page diagram in the above target historical test point set as the historical test points to be processed.

[0068] Sub-step two involves generating test points based on preset input prompts, the aforementioned page image to be processed, the aforementioned target historical page image, and the aforementioned historical test points to be processed. The input prompts can be information used to instruct the multimodal large model to output test points based on the input content. For example, the input prompts could be "Reference to the input content, output test points." The multimodal large model can be a model capable of processing data from two or more modalities. For example, the multimodal large model could be GPT-4O. In practice, the executing entity can input the aforementioned input prompts, the aforementioned page image to be processed, the aforementioned target historical page image, and the aforementioned historical test points to be processed into the multimodal large model, and use the data returned by the multimodal large model as test points.

[0069] The second step is to define the generated test points as a test point set.

[0070] In addressing the technical problems mentioned above, and considering the application scenario of testing an insurance system before the Spring Festival, the following technical issues often arise: directly extracting test points from detailed design documents can lead to low reliability of these test points, resulting in numerous invalid test cases generated from them. This, in turn, can lead to wasted computational resources when testing the application system based on these invalid test cases. Given the specific requirements of this application scenario—a sharp increase in policy renewals before the Spring Festival—while utilizing computational resources for comprehensive stress testing, performance testing, and regression testing of the insurance system, it is also necessary to ensure the system's normal operation. Computational resources are relatively scarce, thus necessitating minimizing waste and conserving computational resources. Therefore, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may generate various test point sets based on the various target historical page atlases, the various target historical test point sets, and the various detailed design page diagrams through the following steps: For each of the detailed design page diagrams mentioned above, perform the following steps: The first step is to identify the target historical page sets corresponding to the detailed design page sets mentioned above as the historical page sets to be processed.

[0071] The second step is to perform the following steps for each historical page image in the above-mentioned historical page image set: The first sub-step is to determine the target historical test point corresponding to the above-mentioned historical page image to be processed as the historical test point to be processed.

[0072] The second sub-step involves generating a page difference matrix based on the detailed design page diagram and the historical page diagram to be processed. This page difference matrix can be a matrix used to characterize the differences between the detailed design page diagram and the historical page diagram to be processed.

[0073] In practice, firstly, the aforementioned executing entity can input the detailed design page image into a grayscale function to obtain a grayscale image corresponding to the detailed design page image as the page grayscale image. The grayscale function can be any function capable of generating a grayscale image. For example, the grayscale function could be the rgb2gray function. Secondly, the aforementioned historical page image to be processed can be input into the aforementioned grayscale function to obtain a grayscale image corresponding to the historical page image as the historical page grayscale image. Then, the image matrix of the page grayscale image can be determined as the page matrix. The image matrix of the historical page grayscale image can be determined as the historical page matrix. Then, the page matrix and the historical page matrix can be input into a difference function to obtain the matrix output by the difference function as the page difference matrix. The difference function can be any function capable of generating the absolute difference between each element in the two matrices. For example, the difference function could be the cv2.absdiff() function.

[0074] The third sub-step involves generating bounding box data based on the aforementioned page difference matrix. This bounding box data can be the data corresponding to the vertices of the target's bounding rectangle. The target's bounding rectangle can be the smallest bounding rectangle corresponding to the contour in the aforementioned binarization matrix. The bounding box data can include the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the rectangle width, and the rectangle height. The x-coordinate of the top-left corner can be the x-coordinate of the top-left vertex of the target's bounding rectangle in the aforementioned binarization matrix. The y-coordinate of the top-left corner can be the y-coordinate of the top-left vertex of the target's bounding rectangle in the aforementioned binarization matrix. The rectangle width can be the width of the target's bounding rectangle. The rectangle height can be the height of the target's bounding rectangle.

[0075] In practice, firstly, the page difference matrix can be binarized using a binarization function to obtain the binarized page difference matrix as a binary matrix. The binarization function can be any function capable of binarizing a matrix. For example, the binarization function could be `cv2.threshold()`.

[0076] Then, the binarized matrix can be input into the boundary detection function to obtain the contour point data of the binarized matrix. Each contour point data can be a pixel coordinate representing the position of the contour in the binarized matrix. For example, when the binarized matrix is ​​a matrix composed of 0s and 255s, each contour point data can be the pixel coordinates of the elements with a value of 255 that are adjacent to 0. The boundary detection function can be a function capable of extracting contours from the binary matrix. For example, the boundary detection function could be the cvFindContours function.

[0077] Then, the contour point data can be input into the bounding rectangle generation function to obtain the bounding box data. This bounding rectangle generation function can be any function capable of generating the bounding rectangle of the contour. For example, the bounding rectangle generation function could be cv2.boundingRect().

[0078] The fourth sub-step involves generating a page image region and a historical page image region based on the aforementioned bounding box data, the aforementioned detailed design page diagram, and the aforementioned historical page diagram to be processed. The page image region can be an image region from the aforementioned detailed design page diagram. The historical page image region can be an image region from the aforementioned historical page diagram to be processed.

[0079] In practice, firstly, the executing entity can determine the sum of the top-left corner x-coordinate and the rectangle width included in the bounding box data as the x-coordinate segmentation data. It can also determine the sum of the top-left corner y-coordinate and the rectangle height included in the bounding box data as the y-coordinate segmentation data. Then, the top-left corner x-coordinate and top-left corner y-coordinate, the x-coordinate segmentation data, and the y-coordinate segmentation data can be combined to form the segmentation data. For example, when the top-left corner x-coordinate is 3, the top-left corner y-coordinate is 4, the x-coordinate segmentation data is 5, and the y-coordinate segmentation data is 7, the combined segmentation data can be "[4:7,3:5]".

[0080] Then, the detailed design page image and the historical page image to be processed can be segmented according to the range represented by the above segmentation data using a slicing operation to obtain the page image region and the historical page image region. The slicing operation can be any operation capable of image segmentation. For example, the slicing operation can be array slicing in NumPy. As an example: after reading the detailed design page image and naming it "image", the image region in the detailed design page image can be segmented according to the above segmentation data using the "image[4:7,3:5]" command, resulting in the page image region, with a width from the 3rd element point to the 5th pixel and a height from the 4th element point to the 7th pixel.

[0081] The fifth sub-step involves performing element recognition processing on the aforementioned page image area to obtain component element data. Each component element data item can be data describing the position and category of the corresponding component rectangle within the page image area. Each component rectangle can be the smallest bounding rectangle of the UI component area included in the page image area. Each component element data item can include component boundary data and component category. The component boundary data can be data describing the position of the component rectangle. The component boundary data can include vertex x-coordinate, vertex y-coordinate, component width, and component height. As an example, the component boundary data can be "[2,3,8,9]", where "2" is the vertex x-coordinate, "3" is the vertex y-coordinate, "8" is the component width, and "9" is the component height. The vertex x-coordinate can be the x-coordinate of the top-left corner vertex of the component rectangle within the page image area. The vertex y-coordinate can be the y-coordinate of the top-left corner vertex of the component rectangle within the page image area. The component width can be the width of the component rectangle. The component height can be the height of the component rectangle. The component categories mentioned above can be information used to describe the categories of UI components represented by the UI component area. For example, the component categories mentioned above can be buttons, input boxes, or icons.

[0082] In practice, the aforementioned page image regions can be input into an element recognition model to obtain data for each component element. This element recognition model can be a neural network model that takes page image regions as input and outputs the data for each component element. For example, the element recognition model could be a pre-trained Mask R-CNN model (Mask Region-based Convolutional Neural Network). The pre-training process involves fine-tuning the Mask R-CNN model using an annotated UI component graph and a cross-entropy loss function. Each UI component graph in the UI component graph set can be image data containing UI components.

[0083] The sixth sub-step involves performing element recognition processing on the aforementioned historical page image region to obtain element data for each historical component. Each historical component element data can be data describing the position and category of the rectangle corresponding to the aforementioned historical page image region. Each historical component element data can include historical component boundary data and a historical component category. The historical component boundary data can be the component boundary data corresponding to the historical page image region. The historical component category can be the component category corresponding to the historical page image region. In practice, the aforementioned historical page image region can be input into the aforementioned element recognition model to obtain the element data for each historical component.

[0084] The seventh sub-step generates new element groups, deleted element groups, and modified element groups based on the element data of each component and the element data of each historical component.

[0085] Specifically, each newly added element in the newly added element group can be a UI component area that is added to the page image area compared to the historical page image area. Each deleted element in the deleted element group can be a UI component area that is missing from the page image area compared to the historical page image area. Each modified element in the modified element group can be a UI component area that exists in both the page image area and the historical page image area, but whose position has changed.

[0086] In practice, for each component element data in the above component element data, firstly, the component boundary data and component category included in the above component element data can be determined as the target component boundary data and target component category, respectively.

[0087] Secondly, the historical component element data whose categories are the same as the target component category can be identified as the historical component element dataset. In response to the determination that the historical component element dataset is empty, the component element data can be added to the new element list using an add function to update the new element list. The add function can be any function capable of adding data to a list. For example, the add() function can be used. The new element list can be a list used to store the new elements.

[0088] Then, in response to determining that the aforementioned historical component element dataset is not empty, for each historical component boundary data in the aforementioned historical component element dataset, a data comparison function can be used to compare the historical component boundary data with the target component boundary data, and the return value of the data comparison function is used as the data comparison result. The data comparison function can be any function that compares whether two data are the same. For example, the data comparison function can be the `array_equal()` function in NumPy. The data comparison result can be a label used to characterize whether the historical component boundary data and the target component boundary data are the same. For example, the data comparison result can be "True" or "False". "True" indicates that the historical component boundary data and the target component boundary data are the same. "False" indicates that the historical component boundary data and the target component boundary data are different. Then, in response that all obtained data comparison results are "False", the aforementioned component element data can be added to the modified element list using the aforementioned add function to update the modified element list. The modified element list can be a list used to store modified elements.

[0089] Next, for each historical component element in the aforementioned historical component element data, the historical component category included in the aforementioned historical component element data can be determined as the target historical component category. Then, the component element data whose component categories are the same as the target historical component category can be determined as the component set to be processed. In response to determining that the component set to be processed is empty, the aforementioned historical component element data can be added to the deleted element list using the aforementioned add function to update the deleted element list. The deleted element list can be a list used to store deleted elements.

[0090] Finally, each newly added element in the updated list of newly added elements can be identified as a group of newly added elements. Each modified element in the updated list of modified elements can be identified as a group of modified elements. Each deleted element in the updated list of deleted elements can be identified as a group of deleted elements.

[0091] The eighth sub-step generates test points based on the preset test point generation information, the aforementioned newly added element group, the aforementioned deleted element group, the aforementioned modified element group, the aforementioned pending historical page diagram, the aforementioned pending historical test points, and the aforementioned detailed design page diagram. The aforementioned test point generation information can be used to prompt the aforementioned multimodal large model to output test point information based on the input content. For example, the aforementioned test point generation information could be: "Based on the input content, especially referring to the content of the newly added element group, the deleted element group, and the modified element group, generate new test points based on the pending historical test points."

[0092] In practice, the aforementioned executing entity can input the aforementioned test point generation information, the aforementioned newly added element group, the aforementioned deleted element group, the aforementioned modified element group, the aforementioned pending historical page diagram, the aforementioned pending historical test points, and the aforementioned detailed design page diagram into the aforementioned multimodal large model, and obtain the data returned by the aforementioned multimodal large model as test points.

[0093] The second step is to define the generated test points as a test point set.

[0094] The above technical solution and its related content, combined with step 108, serve as an inventive point of this disclosure, solving the problem of "waste of computing resources." Factors leading to wasted computing resources often include: directly extracting test points from detailed design documents can easily result in low reliability of the obtained test points, leading to a large number of invalid test cases in the test cases generated based on these test points, and consequently, wasted computing resources when performing system testing on the application system based on these invalid test cases. Solving these factors can reduce wasted computing resources. To achieve this effect, this disclosure firstly performs the following steps for each detailed design page diagram: secondly, the target historical page diagram sets corresponding to the detailed design page diagrams in the target historical page diagram sets are determined as the historical page diagram sets to be processed. Then, for each historical page diagram to be processed in the historical page diagram set, the following steps are performed: then, the target historical test points corresponding to the historical page diagrams to be processed are determined as the historical test points to be processed. Then, a page difference matrix is ​​generated based on the detailed design page diagrams and the historical page diagrams to be processed. Thus, a difference matrix between the detailed design page diagrams and the historical page diagrams to be processed can be generated. Next, based on the aforementioned page difference matrix, bounding box data is generated. This yields the outline data of the page difference matrix. Then, based on the bounding box data, the detailed design page diagram, and the historical page diagram to be processed, page image regions and historical page image regions are generated. This identifies the image regions in the detailed design page diagram and the historical page diagram to be processed that exhibit differences. Element recognition processing is then performed on the page image regions to obtain element data for each component. This generates element data for each component. Next, element recognition processing is performed on the historical page image regions to obtain element data for each historical component. This generates element data for each historical component. Based on the element data for each component and the element data for each historical component, new element groups, deleted element groups, and modified element groups are generated. This allows the identification of newly added, deleted, and modified components by comparing the element data for each component with the element data for each historical component. Finally, test points are generated based on preset test point generation information, the new element groups, deleted element groups, modified element groups, the historical page diagram to be processed, the historical test points to be processed, and the detailed design page diagram. Therefore, based on the historical test points to be processed, new test points can be generated according to the newly added element group, the deleted element group, and the modified element group. Finally, the generated test points are defined as the test point set.Because we can first find the image differences between the detailed design page diagram and the target historical page diagram, and then generate new test points based on the historical test points corresponding to the target historical page diagram according to the obtained image differences, instead of directly extracting new test points from the detailed design document, we can improve the reliability of the generated test points, thereby reducing the number of invalid test cases generated based on the test points, reducing the probability of functional testing relying on invalid test cases, and thus reducing the waste of computing resources.

[0095] Step 107: Based on each test point set, each detailed design page diagram, each target historical test point set, and each target historical page diagram set, generate each test case set corresponding to each test point set.

[0096] In some embodiments, the execution entity may generate test case sets corresponding to the test point sets based on the test point sets, detailed design page diagrams, target historical test point sets, and target historical page diagram sets.

[0097] In some optional implementations of certain embodiments, the execution entity can generate test case sets corresponding to the various test point sets based on the various test point sets, the various detailed design page diagrams, the various target historical test point sets, and the various target historical page diagram sets through the following steps: For each test point set in the above test point sets, perform the following steps: The first step is to identify the detailed design page diagrams corresponding to the test point set in each of the above detailed design page diagrams as the page diagrams to be input.

[0098] The second step is to perform the following steps for each test point in the above test point set: The first sub-step is to determine the target historical test points corresponding to the above test points in the set of target historical test points as the historical test points to be input.

[0099] The second sub-step involves determining the target historical page images corresponding to the test points in each of the aforementioned target historical page image sets as the historical page images to be input.

[0100] The third sub-step involves generating test cases corresponding to the aforementioned test points based on the preset test case prompts, the aforementioned page image to be input, the aforementioned historical test points to be input, and the aforementioned historical page image to be input. The aforementioned test case prompts can be information used to prompt the aforementioned multimodal large model to output test cases based on the input content. For example, the aforementioned test case prompts could be "Output test cases based on the input content."

[0101] The test cases described above can include preconditions, operation steps, and expected results. Preconditions can be information representing the requirements that must be met before performing an operation. For example, a precondition could be "The user has been verified with their real name." Each operation step can be information representing the necessary steps. For example, an operation step could be "Enter the mini-program." Each expected result can be information representing the result obtained after performing the operation. For example, an expected result could be "The new H5 page displays the individual insurance policy correctly." Each operation step corresponds to an expected result. Each expected result can be the result that should appear after performing the corresponding operation step. Each expected result corresponds to an expected result image. The expected result image can be an image corresponding to the expected result. As an example, when the expected result is "The new H5 page displays the individual insurance policy correctly," the corresponding expected result image could be a screenshot of the interface containing a pop-up window indicating "The individual insurance policy is correct."

[0102] In practice, the aforementioned executing entity can input the aforementioned test case prompts, the aforementioned page diagram to be input, the aforementioned historical test points to be input, and the aforementioned historical page diagram to be input into the aforementioned multimodal large model, and obtain the data returned by the multimodal large model as test cases.

[0103] The third step is to define the generated test cases as a test case set.

[0104] Step 108: Execute system testing tasks on the application system based on each test case set.

[0105] In some embodiments, the aforementioned execution entity may perform system testing tasks on the aforementioned application system based on the aforementioned test case sets.

[0106] In addressing the technical problems mentioned above by adopting technical solutions, the following technical issues often arise in the application scenario: during large-scale insurance e-commerce promotions. When running automated scripts to test the application system, these scripts are prone to element positioning failures due to non-functional adjustments such as icon changes and color variations, resulting in numerous invalid test errors. This necessitates modifications to the automated scripts to continue testing the application system, leading to low testing efficiency. Considering the following requirements for this application scenario: during large-scale insurance e-commerce promotions, insurance systems often need to launch new insurance services or limited-time activities. Given the short duration of these promotions, the server must complete comprehensive testing of the insurance system within a specified timeframe to ensure timely launch of new features. Therefore, high testing efficiency is required. We have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned execution entity may perform system testing tasks on the aforementioned application system based on the aforementioned test case sets through the following steps: For each test case in each of the above test case sets, perform the following steps: The first step is to perform data parsing and processing on each of the above operation steps to obtain an operation information sequence. Each operation information in this sequence can be an instruction used to execute the operation step. For example, when the operation step is "swipe up", the corresponding operation information could be "action: "swipe", direction: "up".

[0107] In practice, the aforementioned executing entity can send each of the above-mentioned operational steps to the target terminal. Then, it can receive the sequence of operational information returned by the target terminal. The target terminal can be a terminal belonging to a technician.

[0108] The second step is to perform the following steps for each operation information in the above sequence: The first sub-step involves conducting system testing on the application system based on the aforementioned operational information. In practice, the executing entity can send the aforementioned operational information to the application system to control the application system to execute the operational steps corresponding to the aforementioned operational information, thereby conducting system testing on the application system.

[0109] The second sub-step involves controlling a pre-set test and monitoring system to monitor the application system and obtain a system interface diagram. This test and monitoring system can be a tool capable of taking screenshots of the application system's response interface. For example, the test and monitoring system could be Snipaste. The system interface diagram can be a screenshot of the interface displayed by the application system after performing the operation steps corresponding to the aforementioned operation information.

[0110] The third sub-step is to determine the expected result corresponding to the above-mentioned operational information from among the above expected results as the target expected result.

[0111] The fourth sub-step is to determine the expected result diagram corresponding to the above-mentioned target expected result as the target expected result diagram.

[0112] The fifth sub-step involves performing feature extraction processing on the aforementioned system interface diagram and the aforementioned target expected result diagram to obtain the interface diagram feature information and the expected diagram feature information.

[0113] The aforementioned interface image feature information can be the feature vector corresponding to the aforementioned system interface image. The aforementioned expected image feature information can be the feature vector corresponding to the aforementioned target expected result image. In practice, the aforementioned execution entity can use a feature extraction tool to perform feature extraction processing on the aforementioned system interface image, obtaining the feature vector of the system interface image as the interface image feature information. The aforementioned feature extraction tool can be a tool capable of extracting feature vectors from an image. For example, the aforementioned feature extraction tool can be Img2Vec. Then, the aforementioned feature extraction tool can be used to perform feature extraction processing on the aforementioned target expected result image, obtaining the feature vector of the aforementioned target expected result image as the expected image feature information.

[0114] The sixth sub-step involves performing character recognition processing on the aforementioned system interface diagram and the aforementioned target expected result diagram to obtain interface diagram text information and expected result diagram text information. The interface diagram text information can be the text content contained in the aforementioned system interface diagram. The expected result diagram text information can be the text content contained in the aforementioned target expected result diagram. In practice, the executing entity can use character recognition technology to perform character recognition processing on the aforementioned system interface diagram to obtain the text content in the system interface diagram as the interface diagram text information. The character recognition technology can be any technology capable of extracting text from an image. For example, the character recognition technology can be optical character recognition (OCR).

[0115] The seventh sub-step involves semantically encoding the aforementioned interface image text information and the aforementioned expected image text information to obtain interface image semantic feature information and expected image semantic feature information. The interface image semantic feature information can be the feature vector corresponding to the aforementioned interface image text information. Similarly, the expected image semantic feature information can be the feature vector corresponding to the aforementioned expected image text information.

[0116] In practice, the aforementioned executing entity can use a text feature extraction tool to perform feature extraction processing on the text information of the interface image, thereby semantically encoding the text information of the interface image and obtaining semantic feature information of the interface image. The text feature extraction tool can be any tool capable of extracting features from text. For example, the text feature extraction tool could be TfidfVectorizer. Then, the aforementioned text feature extraction tool can be used to perform feature extraction processing on the text information of the expected image, thereby semantically encoding the text information of the expected image and obtaining semantic feature information of the expected image.

[0117] The eighth sub-step involves generating interface graph fusion feature information and expected graph fusion feature information based on the aforementioned interface graph feature information, interface graph semantic feature information, expected graph feature information, and expected graph semantic feature information.

[0118] The aforementioned interface graph fusion feature information can be a feature vector obtained by fusing the aforementioned interface graph feature information and the aforementioned interface graph semantic feature information. Similarly, the aforementioned expected graph fusion feature information can be a feature vector obtained by fusing the aforementioned expected graph feature information and the aforementioned expected graph semantic feature information.

[0119] In practice, the aforementioned execution entity can input the interface graph feature information and the interface graph semantic feature information into a feature concatenation function to fuse them and obtain fused interface graph feature information. The feature concatenation function can be a function capable of concatenating multiple feature vectors. For example, the feature concatenation function could be the `concatenate()` function from the NumPy library. Then, the expected graph feature information and the expected graph semantic feature information can be input into the feature concatenation function to fuse them and obtain fused expected graph feature information.

[0120] The ninth sub-step involves comparing the similarity between the interface image fusion feature information and the expected image fusion feature information to obtain the image comparison result.

[0121] The image comparison result can be the cosine similarity between the interface image fusion feature information and the expected image fusion feature information. In practice, the executing entity can determine the image comparison result as the cosine similarity between the interface image fusion feature information and the expected image fusion feature information.

[0122] The tenth sub-step involves sending the operation information and the system interface diagram to the target terminal in response to the determination that the image comparison result does not meet the preset image comparison conditions. The image comparison conditions can be that the value corresponding to the image comparison result is greater than a preset similarity threshold. The similarity threshold can be a pre-set value. Here, the specific setting of the similarity threshold is not limited.

[0123] The above-described technical solution and related content, as an inventive point of this disclosure, solve the problem of "low testing efficiency." Factors leading to low testing efficiency often include: when running automated scripts to test application systems, the scripts are prone to element location failures due to non-functional adjustments such as icon changes and color changes, resulting in numerous invalid test errors. This necessitates modifications to the automated scripts to continue testing the application system, leading to low testing efficiency. Solving these factors can improve testing efficiency. To achieve this, this disclosure firstly performs the following steps for each test case in the aforementioned test case sets: Secondly, it performs data parsing processing on each of the aforementioned operation steps to obtain an operation information sequence. This allows the aforementioned operation steps to be parsed into instructions. Then, for each operation information in the aforementioned operation information sequence, it performs the following steps: Firstly, based on the aforementioned operation information, it performs system testing on the aforementioned application system. This allows for system testing of the application system. Secondly, it controls a preset test monitoring system to monitor the aforementioned application system and obtain a system interface diagram. This allows for obtaining the interface diagram of the tested application system. Then, the expected results corresponding to the above-mentioned operation information are determined as the target expected results. Then, the expected result image corresponding to the target expected result is determined as the target expected result image. Next, feature extraction processing is performed on the system interface image and the target expected result image to obtain interface image feature information and expected image feature information. Thus, interface image feature information and expected image feature information are obtained. Then, character recognition processing is performed on the system interface image and the target expected result image to obtain interface image text information and expected image text information. Thus, interface image text information and expected image text information are obtained. Then, semantic encoding processing is performed on the interface image text information and the expected image text information to obtain interface image semantic feature information and expected image semantic feature information. Thus, interface image semantic feature information and expected image semantic feature information are obtained. Then, based on the interface image feature information, the interface image semantic feature information, the expected image feature information, and the expected image semantic feature information, interface image fusion feature information and expected image fusion feature information are generated. Then, similarity comparison is performed on the interface image fusion feature information and the expected image fusion feature information to obtain image comparison results. Therefore, the similarity between the interface image fusion feature information and the expected image fusion feature information can be compared to determine whether the operation steps corresponding to the above operation information have passed the test. Finally, in response to the determination that the above image comparison result does not meet the preset image comparison conditions, the above operation information and the above system interface image are sent to the target terminal. Thus, when the test fails, the operation information and the system interface image can be sent to the target terminal.Because functional testing involves converting test steps into operation commands and then using these commands to test the application system, the testing process is not affected by visual attributes such as color and font size. Instead of having to rewrite the script to continue testing when the system's color attributes change, this improves testing efficiency.

[0124] The above embodiments of this disclosure have the following beneficial effects: the application system testing method based on multimodal retrieval in some embodiments of this disclosure can reduce the waste of computing resources. Specifically, the reason for the waste of computing resources is that the test points directly extracted from the detailed design document are prone to a high false detection rate, resulting in many redundant steps in the test cases generated based on the test points, and thus wasting computing resources when testing the application system based on the redundant steps in the test cases. Based on this, the application system testing method based on multimodal retrieval in some embodiments of this disclosure firstly obtains a detailed design document in response to receiving test information sent by the application system, wherein the detailed design document includes various detailed design pages. Thus, the detailed design document to be processed can be obtained. Secondly, for each detailed design page in the detailed design document, image conversion processing is performed on the detailed design page to obtain a detailed design page diagram. Thus, each page in the detailed design document can be converted into an image. Then, feature extraction processing is performed on the obtained detailed design page diagrams to obtain detailed design feature information. Thus, various features of each detailed design page diagram can be obtained. Then, for each of the detailed design feature information mentioned above, a target historical page atlas is determined based on the vector database and the detailed design feature information. This yields the target historical page atlas corresponding to the detailed design feature information. Next, based on the determined target historical page atlases, a target historical test point set is determined. This yields the target historical test point set. Then, based on the target historical page atlases, the target historical test point sets, and the detailed design page atlases, a test point set is generated. This yields the test point set. Finally, based on the test point sets, the detailed design page atlases, the target historical test point sets, and the target historical page atlases, a test case set corresponding to each test point set is generated. This yields the test case set. Finally, based on the test case sets, system testing tasks are performed on the application system. This allows for the testing of the application system. Because it can filter out the historical page sets and historical test point sets of each target based on the feature vectors of the detailed design document, and then generate new test points based on the differences between the detailed design document and the historical page sets of each target, and then generate test cases based on the generated test points, instead of directly generating test points and test cases from the detailed design document, it can reduce the false positive rate of each generated test point, reduce redundant steps in the test cases, and thus reduce the waste of computing resources when testing the application system based on the test cases.

[0125] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an application system testing device based on multimodal retrieval. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0126] like Figure 2 As shown, an application system testing device 200 based on multimodal retrieval in some embodiments includes: an acquisition unit 201, an image conversion unit 202, a feature extraction unit 203, a first determination unit 204, a second determination unit 205, a first generation unit 206, a second generation unit 207, and an execution unit 208. The acquisition unit 201 is configured to acquire a detailed design document in response to receiving test information sent by the application system, wherein the detailed design document includes various detailed design pages; the image conversion unit 202 is configured to perform image conversion processing on each detailed design page included in the detailed design document to obtain a detailed design page diagram; the feature extraction unit 203 is configured to perform feature extraction processing on the obtained detailed design page diagrams to obtain various detailed design feature information; the first determination unit 204 is configured to, for each detailed design feature information, determine the detailed design feature based on a vector database and the detailed design features... The system comprises: a first generation unit 206, configured to determine target historical page diagrams; a second generation unit 207, configured to determine target historical test point sets based on the determined target historical page diagrams; a first generation unit 208, configured to generate test point sets based on the target historical page diagrams, target historical test point sets, and detailed design page diagrams; and an execution unit 208, configured to perform system testing tasks on the application system based on the test point sets.

[0127] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0128] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0129] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0130] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0131] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0132] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0133] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: In response to receiving test information sent by the application system, it acquires a detailed design document, wherein the detailed design document includes various detailed design pages; for each detailed design page included in the detailed design document, it performs image conversion processing on the detailed design page to obtain a detailed design page diagram; it performs feature extraction processing on the obtained detailed design page diagrams to obtain various detailed design feature information; for each detailed design feature information, it determines a target historical page diagram set based on a vector database and the detailed design feature information; it determines a target historical test point set based on the determined target historical page diagram sets; it generates various test point sets based on the target historical page diagram sets, the target historical test point sets, and the detailed design page diagrams; it generates various test case sets corresponding to the various test point sets based on the various test point sets, the detailed design page diagrams, the target historical test point sets, and the target historical page diagrams; and it performs system testing tasks on the application system based on the various test case sets.

[0135] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, an image conversion unit, a feature extraction unit, a first determination unit, a second determination unit, a first generation unit, a second generation unit, and an execution unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as a "unit for acquiring detailed design documents."

[0138] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0139] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A testing method for application systems based on multimodal retrieval, comprising: In response to receiving test information sent by the application system, a detailed design document is obtained, wherein the detailed design document includes various detailed design pages; For each detailed design page included in the detailed design document, image conversion processing is performed on the detailed design page to obtain a detailed design page diagram; Feature extraction processing is performed on each of the detailed design page diagrams to obtain the feature information of each detailed design. For each detailed design feature in the detailed design feature information, the target historical page atlas is determined based on the vector database and the detailed design feature information; Based on the determined historical page atlases of each target, determine the historical test point set of each target; Based on the historical page image sets of each target, the historical test point sets of each target, and the detailed design page images of each target, generate each test point set; Based on the various test point sets, the various detailed design page diagrams, the various target historical test point sets, and the various target historical page diagram sets, generate various test case sets corresponding to the various test point sets; Based on the various test case sets, system testing tasks are performed on the application system.

2. The method according to claim 1, wherein, Before obtaining the detailed design document in response to receiving test information sent by the application system, the method further includes: Obtain various historical test data, wherein each historical test data includes historical detailed design documents, historical test point groups, and historical test cases; Based on the historical detailed design documents included in the historical test data, a set of historical detailed feature data is generated, wherein each set of historical detailed feature data includes historical detailed feature information and a set of historical image block feature information. A vector database is constructed based on the various historical detailed feature data groups, the various historical test point groups included in the various historical test data, and the various historical test cases.

3. The method according to claim 2, wherein, The historical detailed design document includes various historical detailed design pages; and the generation of various historical detailed feature data groups based on the various historical detailed design documents included in the various historical test data includes: For each of the aforementioned historical detailed design documents, perform the following steps: For each historical detailed design page included in the historical detailed design document, perform the following steps: Based on the historical detailed design page, generate historical image block feature information groups and historical detailed feature information; The historical detailed feature information and the historical image block feature information are combined into historical detailed feature data; The obtained historical detailed feature data are combined into a historical detailed feature data group.

4. The method according to claim 3, wherein, The generation of historical image patch feature information groups and historical detailed feature information based on the historical detailed design page includes: The historical detailed design page is processed by image conversion to obtain a historical detailed design page image; The historical detailed design page diagram is processed into image blocks to obtain image blocks of each historical page. For each historical page image block in the various historical page image blocks, feature encoding processing is performed on the historical page image block to obtain historical image block feature information; Global pooling is performed on the feature information of each historical image patch to obtain detailed historical feature information; The feature information of each historical image block is combined into a historical image block feature information group.

5. The method according to claim 1, wherein, The generation of each test point set based on the respective target historical page image sets, the respective target historical test point sets, and the respective detailed design page images includes: For each target historical page atlas, perform the following steps: The detailed design page diagrams corresponding to the target historical page diagram set in each of the detailed design page diagrams are determined as the page diagrams to be processed; For each target historical page image in the target historical page image set, perform the following steps: The target historical test points corresponding to the target historical page diagram in each target historical test point set are determined as historical test points to be processed. Test points are generated based on preset input prompts, the page image to be processed, the target historical page image, and the historical test points to be processed. The generated test points are defined as a test point set.

6. The method according to claim 1, wherein, The step of generating test case sets corresponding to each test point set based on each test point set, each detailed design page diagram, each target historical test point set, and each target historical page diagram set includes: For each test point set in the aforementioned test point sets, perform the following steps: The detailed design page diagrams corresponding to the test point set in each of the detailed design page diagrams are determined as the page diagrams to be input; For each test point in the test point set, perform the following steps: The target historical test points corresponding to the test points in the set of each target historical test point are determined as the historical test points to be input. The target historical page images corresponding to the test points in each target historical page image set are determined as the historical page images to be input. Based on the preset test case prompts, the page image to be input, the historical test points to be input, and the historical page image to be input, test cases corresponding to the test points are generated. The generated test cases are defined as a test case set.

7. A testing device for an application system based on multimodal retrieval, comprising: The acquisition unit is configured to acquire a detailed design document in response to receiving test information sent by the application system, wherein the detailed design document includes various detailed design pages; The image conversion unit is configured to perform image conversion processing on each detailed design page included in the detailed design document to obtain a detailed design page diagram. The feature extraction unit is configured to perform feature extraction processing on each of the obtained detailed design page diagrams to obtain feature information of each detailed design. The first determining unit is configured to determine the target historical page atlas based on the vector database and the detailed design feature information for each of the detailed design feature information. The second determining unit is configured to determine the set of historical test points for each target based on the determined set of historical pages for each target. The first generation unit is configured to generate each test point set based on the each target historical page image set, the each target historical test point set, and the each detailed design page image. The second generation unit is configured to generate a set of test cases corresponding to each set of test points based on each set of test points, each set of detailed design page diagrams, each set of target historical test points, and each set of target historical page diagrams. The execution unit is configured to perform system test tasks on the application system based on the various test case sets.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images