Test result generation method and device, equipment and medium

Abstract

The invention discloses a test result generation method and device, equipment and a medium, which are applied to the technical field of computers, in the method, a first test text describing an expected display result and first test image data are obtained for a test process of executing first test operation on a first page; and calling the first language model to analyze the first test text and the first test image data to obtain a test result for the first test operation. The test result represents whether the first test image data conforms to an expected display result described by the first test text or not. Automatic assertion is achieved through the first language model, a tester does not need to set mark points for test content and write program codes used for determining the test result, and the editing cost of the tester is reduced. The expandability of the first language model is high, the first language model can adapt to page updating of the first page, and the maintenance cost of a tester is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for generating test results. Background Art

[0002] User Interface (UI) automated testing is a testing method for UI. UI automated testing uses automated tools to execute test cases to verify whether the tested user interface runs as expected. UI automated testing involves testing the functions of the business logic running on the page. The tester needs to mark the functional points in the page according to the business logic in advance and write program code to generate test results. Automated testing is implemented based on the marked functional points and program code to generate test results.

[0003] However, the cost for testers to mark function points and write program codes is high, and they need to maintain function points and program codes based on changes in page functions. The maintenance cost is also high, which makes it difficult to meet the needs of UI automation testing. Summary of the invention

[0004] In view of this, the present application provides a method, apparatus, device and medium for generating test results, which can reduce the cost of generating test results.

[0005] In a first aspect, the present application provides a method for generating a test result, the method comprising:

[0006] Acquire a first test text, where the first test text is used to describe an expected display result related to performing a first test operation on a first page;

[0007] Acquire first test image data of the first page, wherein the first test image data includes a page display image related to performing the first test operation on the first page;

[0008] The first test text and the first test image data are sent to a first language model to obtain a test result for the first test operation fed back by the first language model, wherein the test result indicates whether the first test image data conforms to an expected display result described by the first test text.

[0009] In a possible implementation, the method further includes:

[0010] Sending the first test text to a second language model to obtain test content fed back by the second language model;

[0011] Determine an image type corresponding to the test content, where the image type is used to indicate a type of the acquired page display image;

[0012] The obtaining of the first test image data of the first page includes:

[0013] According to the image type, first test image data of the first page is acquired.

[0014] In a possible implementation manner, the image type is a comparison type, and obtaining first test image data of the first page according to the image type includes:

[0015] Acquire a page display image before executing the first test operation on the first page and a page display image after executing the first test operation;

[0016] or,

[0017] The image type is a result type, and obtaining first test image data of the first page according to the image type includes:

[0018] A page display image is obtained after the first test operation is performed on the first page.

[0019] In a possible implementation, the method further includes:

[0020] If the image similarity between the page display image before the first test operation and the page display image after the first test operation is performed is greater than an image similarity threshold, the page display image before the first test operation is deleted from the first test image data.

[0021] In a possible implementation, the method further includes:

[0022] Performing region division on the page display image included in the first test image data to obtain a region marking image, wherein the region marking image includes a divided region and a region identifier;

[0023] Sending the region annotated image and the first test text to a fourth language model to obtain a region identifier of a target segmented region fed back by the fourth language model, where the target segmented region is a segmented region related to the first test operation;

[0024] The page display image included in the first test image data is updated to the area display image of the target divided area.

[0025] In a possible implementation, dividing the page display image included in the first test image data into regions to obtain a region-labeled image includes:

[0026] Analyze the document object model tree of the first page to obtain a page element area of ​​the first page;

[0027] Filtering the repeated page element regions and merging the overlapping page element regions to obtain a region division image;

[0028] Based on the region division image, a region annotation image is generated.

[0029] In a possible implementation, dividing the page display image included in the first test image data into regions to obtain a region-labeled image includes:

[0030] Processing the first test image data using the target detection model to obtain a region division result, wherein the region division result is used to indicate a divided region;

[0031] Based on the region division result, a region annotation image is generated.

[0032] In a possible implementation manner, sending the region annotated image and the first test text to the fourth language model includes:

[0033] The region annotated image, the first test text, and training data are sent to a fourth language model, where the training data includes test image sample data and region identifiers of corresponding correct target segmentation data, and the training data is used to assist the fourth language model in generating region identifiers of the target segmentation regions.

[0034] In a possible implementation, the method further includes:

[0035] If it is determined that the first test text does not meet the analysis condition, the first test text is sent to a third language model to obtain a modified first test text fed back by the third language model.

[0036] In a possible implementation, the method further includes:

[0037] Obtaining an evaluation result corresponding to the test result, wherein the evaluation result is used to measure the correctness of the test result;

[0038] The first case data and the first case result are stored in a database, wherein the first case data includes the first test text and the first test image data, and the first case result includes the test result and the evaluation result.

[0039] In a possible implementation, the method further includes:

[0040] Acquire a second test text and second test image data, wherein the second test text is used to describe an expected display result related to performing a second test operation on a second page, and the second test image data includes a page display image related to performing the second test operation on the second page;

[0041] If the data similarity between the second test text and the second test image data and the first case data is greater than the data similarity threshold, and the accuracy of the test result is greater than the accuracy threshold, the test result is used as the test result of the second test operation.

[0042] In a possible implementation, the method further includes:

[0043] Acquire a third test text and third test image data, wherein the third test text is used to describe an expected display result associated with performing a third test operation on a third page, and the third test image data includes a page display image associated with performing the third test operation on the third page;

[0044] Determine that a scene similarity between a test scene to which the third test operation belongs and a test scene to which the first test operation belongs is greater than a scene similarity threshold;

[0045] The third test text, the third test image data, the first case data, and the first case result are sent to the first language model to obtain a test result for the third test operation fed back by the first language model.

[0046] In a possible implementation, sending the first test text and the first test image data to the first language model to obtain a test result for the first test operation fed back by the first language model includes:

[0047] Sending the first test image data and a first prompt word to the first language model, where the first prompt word is used to instruct the first language model to describe image content;

[0048] Obtaining the image content text fed back by the first language model;

[0049] Sending the first test image data, the image content text, the first test text, and a second prompt word to the first language model, where the second prompt word is used to instruct the first language model to generate a test result;

[0050] A test result of obtaining feedback of the first language model.

[0051] In a second aspect, the present application provides a device for generating a test result, the device comprising:

[0052] A first acquisition unit, configured to acquire a first test text, wherein the first test text is used to describe an expected display result related to performing a first test operation on a first page;

[0053] A second acquiring unit, configured to acquire first test image data of the first page, wherein the first test image data includes a page display image related to performing the first test operation on the first page;

[0054] A first generating unit is used to send the first test text and the first test image data to a first language model, and obtain a test result for the first test operation fed back by the first language model, wherein the test result indicates whether the first test image data conforms to an expected display result described by the first test text.

[0055] In a possible implementation manner, the device further includes:

[0056] A first processing unit is configured to send the first test text to a second language model to obtain test content fed back by the second language model;

[0057] A type determination unit, used to determine the image type corresponding to the test content, wherein the image type is used to indicate the type of the acquired page display image;

[0058] The second acquisition unit is specifically configured to acquire first test image data of the first page according to the image type.

[0059] In a possible implementation, the image type is a comparison type, and the second acquisition unit is specifically configured to acquire a page display image before the first test operation is performed on the first page and a page display image after the first test operation is performed on the first page;

[0060] or,

[0061] The image type is a result type, and the second acquisition unit is specifically configured to acquire a page display image after the first test operation is performed on the first page.

[0062] In a possible implementation manner, the device further includes:

[0063] The deleting unit is configured to delete the page display image before the first test operation from the first test image data if the image similarity between the page display image before the first test operation and the page display image after the first test operation is greater than an image similarity threshold.

[0064] In a possible implementation manner, the device further includes:

[0065] a dividing unit, configured to divide the page display image included in the first test image data into regions to obtain a region marking image, wherein the region marking image includes divided regions and region identifiers;

[0066] a second processing unit, configured to send the region annotation image and the first test text to a fourth language model, and obtain a region identifier of a target segmentation region fed back by the fourth language model, wherein the target segmentation region is a segmentation region related to the first test operation;

[0067] An updating unit is used to update the page display image included in the first test image data to the area display image of the target divided area.

[0068] In a possible implementation, the division unit is specifically used to analyze the document object model tree of the first page to obtain the page element area of ​​the first page; filter the repeated page element areas and merge the overlapping page element areas to obtain a region division image; and generate a region annotation image based on the region division image.

[0069] In a possible implementation, the segmentation unit is used to process the first test image data using a target detection model to obtain a region segmentation result, where the region segmentation result is used to indicate a segmented region; and generate a region annotation image based on the region segmentation result.

[0070] In a possible implementation manner, the second processing unit, configured to send the region annotation image and the first test text to a fourth language model, includes:

[0071] The second processing unit is used to send the region annotation image, the first test text and training data to a fourth language model, where the training data includes test image sample data and region identifiers of corresponding correct target segmentation data, and the training data is used to assist the fourth language model in generating region identifiers of the target segmentation regions.

[0072] In a possible implementation manner, the device further includes:

[0073] The third processing unit is configured to send the first test text to a third language model to obtain a modified first test text fed back by the third language model if it is determined that the first test text does not meet the analysis condition.

[0074] In a possible implementation manner, the device further includes:

[0075] A third acquisition unit, used to acquire an evaluation result corresponding to the test result, wherein the evaluation result is used to measure the correctness of the test result;

[0076] The storage unit is used to store first case data and first case results in a database, wherein the first case data includes the first test text and the first test image data, and the first case results include the test result and the evaluation result.

[0077] In a possible implementation manner, the device further includes:

[0078] a fourth acquisition unit, configured to acquire a second test text and second test image data, wherein the second test text is used to describe an expected display result associated with performing a second test operation on a second page, and the second test image data includes a page display image associated with performing the second test operation on the second page;

[0079] The second generating unit is used to use the test result as the test result of the second test operation if the data similarity between the second test text and the second test image data and the first case data is greater than the data similarity threshold, and the correctness of the test result is greater than the correctness threshold.

[0080] In a possible implementation manner, the device further includes:

[0081] a fifth acquisition unit, configured to acquire a third test text and third test image data, wherein the third test text is used to describe an expected display result associated with performing a third test operation on a third page, and the third test image data includes a page display image associated with performing the third test operation on the third page;

[0082] a scene determination unit, configured to determine whether a scene similarity between a test scene to which the third test operation belongs and a test scene to which the first test operation belongs is greater than a scene similarity threshold;

[0083] The third generating unit is used to send the third test text, the third test image data, the first case data and the first case result to the first language model to obtain the test result of the third test operation fed back by the first language model.

[0084] In a possible implementation, the first generation unit is specifically used to send the first test image data and a first prompt word to the first language model, where the first prompt word is used to instruct the first language model to describe the image content; obtain the image content text fed back by the first language model; send the first test image data, the image content text, the first test text, and a second prompt word to the first language model, where the second prompt word is used to instruct the first language model to generate a test result; and obtain the test result fed back by the first language model.

[0085] In a third aspect, the present application provides an electronic device, including:

[0086] one or more processors;

[0087] a storage device having one or more programs stored thereon,

[0088] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect and any implementation manner of the first aspect.

[0089] In a fourth aspect, the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in the first aspect and any implementation method of the first aspect is implemented.

[0090] In a fifth aspect, the present application provides a computer program product. When the computer program product is executed on a device, the device executes the method described in the first aspect and any implementation method of the first aspect.

[0091] It can be seen that this application has the following beneficial effects:

[0092] The present application provides a method, apparatus, device and medium for generating test results. In the method, a test process of performing a first test operation on a first page is performed to obtain a first test text describing an expected display result and first test image data. The first test image data includes a page display image related to the test process. A first language model is called to analyze the first test text and the first test image data to obtain a test result for the first test operation. The test result characterizes whether the first test image data meets the expected display result described by the first test text. The first language model is used to implement automated assertions, and there is no need for the tester to set marking points for the test content and write program code for determining the test result, thereby reducing the editing cost of the tester. In addition, the first language model has strong scalability and can adapt to page updates of the first page, thereby reducing the maintenance cost of the tester. In this way, the test results are generated using the first language model, which reduces the cost of generating test results and meets the needs of automated testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A schematic diagram of an application scenario of UI automated testing provided in an embodiment of the present application;

[0094] Figure 2 A flowchart of a method for generating test results provided in an embodiment of the present application;

[0095] Figure 3 A schematic diagram of a region annotation image provided in an embodiment of the present application;

[0096] Figure 4 A schematic diagram of a region division result provided in an embodiment of the present application;

[0097] Figure 5 A flowchart of another method for generating test results provided in an embodiment of the present application;

[0098] Figure 6 A schematic diagram of the structure of a test result generating device provided in an embodiment of the present application;

[0099] Figure 7 A schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0100] In order to facilitate the understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

[0101] In UI automated testing, assertion is a technique for verifying whether certain conditions are met during the test. The purpose of assertion is to ensure that the UI display meets expectations, thereby ensuring the normal interaction between the application and the user and achieving the normal operation of the application. In the functional testing scenario for the business logic of the page, the tester is required to manually mark the function points in the page and write program code for the implementation assertion of the function. Based on the changes in the function points during the function operation of the page, the program code is run to determine whether the function of the page meets expectations. At present, the cost of functional testing for pages is high. On the one hand, the tester is required to mark the function points in the page and write specific program code to implement the test judgment. The reusability of the program code is not high. The cost of marking and writing program code for the tester is high. On the other hand, in the development and maintenance stage of the software, the function of the page is updated frequently, and the tester needs to update the function points and program code based on the function update, which has a high maintenance cost. The method of marking function points and writing program code to implement assertions is difficult to meet the low-cost requirements of UI automated testing.

[0102] Based on this, an embodiment of the present application provides a method for generating test results, and obtains a first test text and a first test image data for a test process after executing a first test operation on a first page. The first test text is used to describe the expected display result of executing the first test operation. The first test image data includes a page display image related to the test process. The first test text is a reference text for the test result. The first test text and the first test image data are analyzed using a first language model to determine whether the first test image data meets the expected display result described by the first test text, and a test result for the first test operation is generated. By generating test results using the first language model, automated assertions can be realized without the need to manually set function points and write program code, greatly reducing the cost of generating test results.

[0103] In order to facilitate understanding of the technical solution provided in the embodiment of the present application, the application scenario of the method for generating test results provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0104] The page testing method provided in the embodiment of the present application can be applied to the UI testing system. The UI testing system can be used as an independent system, or integrated into other systems as a subsystem or functional module, which is not limited in the embodiment of the present application.

[0105] See also Figure 1 As shown, the figure is a schematic diagram of an application scenario of a UI automated test provided by an embodiment of the present application. The tester writes the test case text in advance. The test case text is used to describe the test operation for the page, and to describe the expected display result related to the test operation. The test case text can be stored in the test case text library. The tester selects the test case text required for this test from the test case text library through the UI test system. The tester triggers the UI test system to perform the test task based on the test case text. The UI test system performs the test operation for the page described in the test case text based on the test case text, and obtains the test image data of the page. The test image data includes a page display image related to the test operation performed on the page. The UI test system filters the test text describing the expected display result related to the test operation from the test case text. The UI test system sends the test text and the test image data of the page to the language model. The language model processes the test text and the test image data to generate the test result. The UI test system obtains the test result fed back by the language model and realizes the automatic assertion. In this way, the assertion can be realized by using the language model to obtain the test result of the test operation for the page. Using language models to implement assertions eliminates the need for testers to mark function points and write specific program codes, reducing testing costs. In addition, language models are highly scalable, reducing maintenance costs caused by functional changes to pages. Using language models to implement assertions for page functional testing meets the need for lower costs for UI automated testing.

[0106] The above application scenario is only used as a possible implementation method. For example, in another possible implementation method, the UI test system's test of the page also includes the judgment of page display abnormalities. After obtaining the test image data of the page, first determine whether the page has a display abnormality based on the test image data. If it is determined that there is a display abnormality, the UI test system directly feeds back the test result of the display abnormality. If it is determined that there is no display abnormality, the UI test system then calls the language model to process the test text and the test image data of the page to obtain the test result.

[0107] The following is an introduction to the method for generating the test results provided in the embodiment of the present application. Figure 2 As shown, the figure is a flow chart of a method for generating test results provided in an embodiment of the present application, the method comprising:

[0108] S201: Obtain a first test text.

[0109] When performing a test on a first page, a first test operation may be performed on the first page. The first test text is used to describe an expected display result related to the first test operation. Based on the first test text, an expected display result of performing the first test operation on the first page may be determined, thereby determining a test result of the first test operation.

[0110] The first test text may be a test case text, or may be extracted from the test case text. The test case text includes text describing the execution of a first test operation on the first page and text describing the expected display result. As an example, the test case text for the first page is: open the session page, and the group avatar is normally displayed at the top of the session page. Among them, "open the session page" is the text describing the execution of an opening operation on the session page. "The group avatar is normally displayed at the top of the session page" is the first test text, which describes the expected display result of executing the opening session page. The test case text can be used to automatically execute the first test operation on the first page, and generate a test result based on the expected display result.

[0111] In some possible implementations, after obtaining the first test text, it is determined whether the first test text meets the analysis conditions. The analysis conditions are pre-set conditions regarding the language style and description content of the first test text. The analysis conditions are used to standardize the first test text so that the first language model can fully understand the first test text. The analysis conditions can be set in advance according to the test needs and the needs of the language model to understand the text. If it is determined that the first test text does not meet the analysis conditions, a third language model can be called to process the first test text. The third language model is, for example, a large language model (LLM). A large language model refers to a model trained using a large amount of text data based on natural language technology, which can generate natural language text or understand the meaning of a language text. The third language model and the first language model can be the same model or different models.

[0112] The third language model can modify the first test text. For example, the third language model is used to polish and adjust the language of the first test text. The modified first test text fed back by the third language model is more standardized, and the description of the expected display result is clearer and easier to understand. In some possible implementations, the first test text and the prompt words indicating the text modification are sent to the third language model. The prompt words are used to indicate the way in which the third language model modifies the first test text. The prompt words indicating the text modification, for example, include a description of the analysis conditions, so that the third language model generates a first test text that meets the analysis conditions by analyzing and understanding the first test text.

[0113] S202: Acquire first test image data of a first page.

[0114] Obtain a page display image related to performing a first test operation on a first page to obtain first test image data. The first test image data may include one or more images, such as a screenshot of the first page. Alternatively, the first test image data is a video, such as a screen recording video of the first test operation. The first test image data may be test image data generated by a UI automated test performing the first test operation, or may be test image data generated by a tester manually performing the first test operation.

[0115] The image type included in the first test image data is determined based on the test content. The test content reflects the target to be tested in this test. For example, for some test contents, the test result can be determined by only executing the page display image after the first test operation. For example, for determining whether one or more page elements exist, determining whether the state of the page elements is correct, determining whether the page elements or page display is normal, etc., for such test contents, the test result can be determined by using the page display image after executing the first test operation.

[0116] For other test contents, it is necessary to compare the page display images before and after the first test operation to determine the test results. For example, to determine whether the page elements are changed normally after the test operation, it is necessary to use the page display image before and after the first test operation to determine the test results.

[0117] As an example, the test content consists of three parts, namely, the target object, adverbial and assertion description. The target object is the object tested in the page. For example, the target object is a page element. Adverbial is used to modify the target object. Adverbial is, for example, a location adverbial, such as the position of a page element in the page. The assertion description is used to describe the changes in the target object before and after the first test operation. The assertion description is, for example, from situation A to situation B. For example, for the test scenario in which the target object appears after the first test operation, situation A is, for example, non-existence, and situation B is existence. For another example, for the test scenario in which the target object disappears after the first test operation, situation A is, for example, existence, and situation B is non-existence. For another example, for the test scenario in which the target object changes its display mode in the first test operation, situation A is, for example, display mode 1, and situation B is display mode 2.

[0118] An embodiment of the present application provides a possible implementation method for determining the image type. The test content can be determined based on a first test text. After obtaining the first test text, the first test text is sent to the second language model to obtain the test content fed back by the second language model. The second language model is, for example, LLM. The second language model and the first language model can be the same model or different models. Using the second language model, the first test text can be analyzed to obtain the test content. The test content is more concise and can directly reflect the goal of the test, that is, the expected display result of the test. As an example, the first test text and the prompt words indicating the generation of the test content are sent to the second language model to obtain the test content generated by the second language model. The prompt words indicating the generation of the test content, for example, include text describing the composition of the test content.

[0119] Using the determined test content, determine the image type corresponding to the test content. For example, the image type is a comparison type or a result type. The comparison type includes the page display image after the first test operation and the page display image before the first test operation. The result type includes the page display image after the first test operation.

[0120] The correspondence between the test content and the image type can be preset. Take the above test content consisting of the target object, adverbial and assertion as an example. The assertion description is the test content from nothing to something, which corresponds to the result type. The assertion description is from something to nothing, or other test content except from nothing to something, which corresponds to the comparison type. The target object is a pronoun or a template, which is not a specific object, and the test content corresponds to the comparison type.

[0121] According to the determined image type, first test image data of the first page is obtained. In one possible implementation, based on the first test operation, a page display image after the first test operation and a page display image before the first test operation are obtained. According to the determined image type, the first test image data is determined from the page display image after the first test operation and the page display image before the first test operation. In another possible implementation, during the execution of the first test operation, the first test image data is obtained according to the determined image type.

[0122] As an example, if the image type is a comparison type, the page display image before the first test operation is performed on the first page and the page display image after the first test operation is performed are obtained. As another example, if the image content is a result type, the page display image after the first test operation is performed on the first page is obtained.

[0123] When the image type is a result type, the first language model only needs to process the page display image after executing the first test operation, and does not need to process the page display image before executing the first test operation, thereby reducing the cost of the first language model to process the image and improving the efficiency of the first language model in generating test results.

[0124] Furthermore, when the image type is a comparison type, the page display images before and after the first test operation are compared. The image similarity between the page display image before the first test operation and the page display image after the first test operation is determined. The embodiment of the present application does not limit the method of determining the image similarity between the two images, for example, using the histograms of the two page display images, calculating the similarity of the histograms, or using an artificial intelligence model to determine the image similarity.

[0125] If the image similarity is greater than the image similarity threshold, it means that the page display image does not change much before and after the first test operation is performed. The image similarity threshold can be set according to the assertion needs. As an example, the image similarity threshold is 99%. In the case that the page display image does not change much before and after the first test operation is performed, it is not necessary to analyze the page display image before the first test operation is performed, and only the page display image after the first test operation is processed using the first language model. The page display image before the first test operation is deleted from the first test image data. In this way, the number of page display images processed by the first language model can be reduced, the cost of processing images using the first language model can be reduced, and the efficiency of generating test results can be improved.

[0126] It should be noted that the embodiment of the present application does not limit the order of executing S201 to obtain the first test text and executing S202 to obtain the first test image data. S201 may be executed first, and then S202. Alternatively, S202 may be executed first, and then S201. Alternatively, S201 and S202 may be executed simultaneously.

[0127] S203: Sending a first test text and a first test image data to a first language model, and obtaining a test result for a first test operation fed back by the first language model.

[0128] The first language model is used to analyze the first test text and the first test image data to generate a test result. The test result indicates whether the first test image data conforms to the expected display result described by the first test text.

[0129] The first language model is, for example, a multimodal LLM, which can process a variety of data such as text, image, and video.

[0130] The embodiment of the present application does not limit the manner of generating the test result using the first language model.

[0131] In a possible implementation, the first test image data, the first test text, and the third prompt word are sent to the first language model. The third prompt word is used to instruct the first language model to analyze the first test image data and the first test text to generate a test result. The third prompt word can be set according to test needs. The third prompt word can be a pre-set text template, or it can be a text edited and input by the tester based on test needs.

[0132] In another possible implementation, the first test image data and the first prompt word are first sent to the first language model. The first prompt word is used to instruct the first language model to describe the image content. The content included in the first prompt word can be seen in the introduction below. In this way, the first language model can fully understand the image content of the page display image included in the first test image data, which is conducive to the subsequent generation of more accurate test results by the first language model. The first language model generates an image content text based on the first test image data and the first prompt word. As another example, the first test image data, the first test text and the first prompt word can be sent to the first language model so that the first language model combines the information included in the first test text to generate a more effective image content text.

[0133] The acquired image content text, the first test image data, the first test text and the second prompt word are sent to the first language model. The second prompt word is used to instruct the first language model to generate a test result. The content included in the second prompt word can be seen in the following introduction. In this way, the first language model can determine whether the page display image meets the expected display result of the first test text based on the image content text. The test result of the feedback of the first language model is obtained to obtain the test result of the first test operation. Among them, the image content text can be used as auxiliary content to help the first language model fully understand the content of the page display image and obtain a more accurate test result.

[0134] It should be noted that, when the image type is a comparison type, that is, when the first test image data includes the page display image before the first test operation is performed and the page display image after the first test operation is performed, it is necessary to first process the page display images before and after the first test operation respectively to obtain the image content texts for the two page display images. In other words, the page display image before the first test operation and the first prompt word are sent to the first language model to obtain the first image content text fed back by the first language model. The first image content text is for the page display image before the first test operation is performed. The page display image after the first test operation is performed and the first prompt word are sent to the first language model to obtain the second image content text fed back by the first language model. The second image content text is for the page display image after the first test operation is performed. Then the first image content text, the second image content text, the first test image data, the first test text and the second prompt word are sent to the first language model to obtain the test result of the first language model.

[0135] In addition, during the process of generating the first image content text and the second image content text, the first test text may be sent to the first language model so that the first language model generates the first image content text and the second image content text in combination with the first test text.

[0136] As an example, the contents included in the first prompt word and the second prompt word are introduced below.

[0137] The first prompt word includes, for example, one or more of the following five contents:

[0138] The first: An overall summary page shows what the image represents.

[0139] The second method is to describe the page elements, the order of the page elements, the status of the page elements, etc. in the page display image in a specific order, and infer the meaning of the page elements.

[0140] The specific order is, for example, a preset order of describing the image from top to bottom, from left to right, etc. The specific order can be set based on the description needs of the image. The page elements are, for example, the text, numbers, buttons, icons, and pictures included in the page, which constitute the page.

[0141] Furthermore, if the first language model also uses the first test text to analyze the image content text, the meaning of the page element can also be inferred in combination with the first test text.

[0142] The third method: when the structure of the page display image is relatively complex, the page display image is divided into multiple areas, and each area is described separately.

[0143] The fourth type: describing the objective attributes of the page elements in the page display image, and inferring the meaning of the page elements based on the objective attributes of the page elements.

[0144] Objective attributes include, for example, text content, numerical value, button shape, icon size, etc. Furthermore, if the first language model also uses the first test text to analyze the image content text, the meaning of the page element can also be inferred in combination with the first test text.

[0145] Fifth: The inference page displays the product name represented by the page element in the image.

[0146] The product name refers to the name of the product dimension of the page element. For example, the group chat page includes an icon of the group avatar, and the product name is the group avatar. The group chat page includes text representing the group name, and the product name is the group name.

[0147] Furthermore, if the first language model also uses the first test text to analyze the image content text, the product name represented by the page element can also be inferred in combination with the first test text.

[0148] The second prompt word includes, for example, one or more of the following four contents:

[0149] The first method is to judge whether the page display image meets the expected display result based on the page display image as the main factor and the image content text as the auxiliary factor.

[0150] The second type: If the test text involves multiple target objects, all target objects meet the expected display results, which is considered to be fully in line with expectations. Three test results are distinguished: fully in line with expectations, partially in line with expectations, and completely not in line with expectations.

[0151] The third type: If the display status of the target object changes, it is necessary to combine the page display images before and after the test operation to first find the status of the target object before and after the test operation, and then determine whether it meets the expected display results.

[0152] Fourth type: When the target object is a pronoun or a template, the specific object represented by the target object is determined based on the description of the test text and the page display images before and after the test operation is performed, and then it is analyzed whether the expected display result is met.

[0153] The content of the first prompt word and the second prompt word is only used as an example. The specific content of the first prompt word and the second prompt word can be set according to the test needs. The first prompt word and the second prompt word can be generated based on a preset prompt word template, or can be edited and input by the tester.

[0154] Based on the above S201-S203, it can be known that the use of language models can realize low-cost automatic assertions. In addition, the use of language models can also determine test results for the test process of more complex pages, expanding the application scope of automatic assertions.

[0155] In a possible implementation, the page display image included in the first test image data includes more content. For example, the page display image is a page image of a computer. The size of the page image of the computer is large, the structure of the page image is complex, and the number of page elements included may be large. Before using the first language model to process the first test image data, the first test image data is first divided into regions for processing, which specifically includes the following three steps:

[0156] A1: Divide the page display image included in the first test image data into regions to obtain a region marking image.

[0157] See also Figure 3 As shown in FIG. 1 , the figure is a schematic diagram of a region annotation image provided by an embodiment of the present application. The region annotation image includes a divided region and a region identifier. The divided region is, for example, a divided region included in a rectangular frame. The region identifier is, for example, a digital identifier in the divided region, such as Figure 3 The number in the upper left corner of each partition area.

[0158] As some examples, the embodiments of the present application provide two possible implementation methods for dividing the page display image included in the first test image data into regions to obtain a region-annotated image.

[0159] The first method is to use the Document Object Model tree (DOM tree) to generate a region annotation image.

[0160] The DOM tree reflects the page structure and content. Using the DOM tree of the first page, obtain the information of the page elements of the first page and obtain the coordinates of the page elements. Based on the coordinates of the page elements, determine the page element area. Page element areas may be repeated or overlapping. Repeated page element areas refer to different page elements having the same page element area. Overlapping page element areas refer to areas of different page elements having overlapping parts. Filter out repeated page element areas, that is, retain one page element area among multiple identical page element areas. Merge overlapping page element areas, that is, merge overlapping page element areas into a larger page element area that includes overlapping page element areas.

[0161] After filtering and merging, a region division image is obtained. Based on the region division image, the divided regions are marked to obtain a region annotation image. As an example, OpenCV (a cross-platform computer vision library) is used to process the region division image to generate a region annotation image.

[0162] The second method: using the target detection model to generate region annotation images.

[0163] The target detection model is, for example, a YOLO (You Only Look Once) model. The target detection model is pre-trained using sample data. The sample data includes a page image and a logo image corresponding to the page image. The logo image includes a marked area.

[0164] The first test image data is processed using the target detection model to obtain a region division result generated by the target detection model. The region division result is used to indicate page elements belonging to the same region. The region division result is, for example, the first test image data including a detection box. The detection box includes one or more page elements. The detection box marks the divided region. As an example, see Figure 4 As shown, this figure is a schematic diagram of a region division result provided in an embodiment of the present application.

[0165] On the basis of the region division image, the divided regions are marked to obtain a region annotation image.

[0166] The embodiments of the present application do not limit the use of the above-mentioned generated region annotation image. In a possible implementation, the DOM tree can be used to generate the region annotation image. If the region division image cannot be generated normally, or the region included in the generated region division image is relatively complex, the target detection model is used to generate the region annotation image.

[0167] A2: Send the region annotated image and the first test text to a fourth language model to obtain a region identifier of a target divided region fed back by the fourth language model.

[0168] The fourth language model is used to identify a relatively critical region in the region annotation image, that is, a target segmentation region. The target segmentation region is a segmentation region related to the first test operation.

[0169] The fourth language model is, for example, a multimodal large language model. The fourth language model and the first language model may be the same model or different models.

[0170] The region annotation image and the first test text are sent to the fourth language model. The fourth language model can analyze the sent region annotation image and the first test text, determine the target divided region in the region annotation image, and feed back the region identifier of the target divided region.

[0171] In another possible implementation, training data is obtained. The training data includes data related to cases where the target segmentation area is incorrectly identified. For example, the training data includes test image sample data and corresponding region identifications of correct target segmentation data. The training data can be determined based on historical feedback results of the fourth language model for the target segmentation area and user evaluation results for the feedback results. The training data, as a counterexample, can assist the fourth language model in better identifying the target segmentation area, improve the accuracy of the target segmentation area fed back by the fourth language model, and thereby improve the accuracy of the test results generated by the first language model.

[0172] A3: updating the page display image included in the first test image data to the area display image of the target divided area.

[0173] After determining the target divided area, the page display image is updated to the regional display image of the target divided area. The regional display image is a partial image of the page display area.

[0174] This makes it easier for the first language model to process the first test image data, reduces the impact of image content irrelevant to the first test operation on the test result, reduces the cost of the first language model generating the test result, and improves the accuracy of the test result.

[0175] After generating the test result using the first language model, the tester may also evaluate the test result to generate an evaluation result. The evaluation result is used to measure the accuracy of the test result. In some possible implementations, if the test result is incorrect, that is, the accuracy is low, the evaluation result also includes the correct test result.

[0176] The test results and evaluation results mainly involve four situations:

[0177] The first case: The test result is: the first test image data meets the expected display result described by the first test text. The evaluation result is: the first test image data actually meets the expected display result described by the first test text. The first case is TP (True Positive), that is, the positive sample is correctly identified.

[0178] The second case is FP (False Positive), which means that the positive sample is incorrectly identified, which is a false positive case.

[0179] The third case is TN (True Negative), which means that the negative sample is correctly identified and is an effective recall case.

[0180] The fourth case: The test result is: the first test image data does not meet the expected display result described in the first test text. The evaluation result is: the first test image data does not actually meet the expected display result described in the first test text. The fourth case is FN (False Negative), that is, the positive sample is misidentified, which is a missed recall case.

[0181] Obtain the evaluation result corresponding to the test result. And store the first test text, the first test image data test result and the evaluation result in a database. The first test text and the first test image data constitute the first case data. The test result and the evaluation result constitute the first case result.

[0182] The database may be a vector database. Retrieval-augmented Generation (RAG) may be implemented based on the vector database. In a possible implementation, the first case information and the first case result are vectorized, and the obtained vectors are stored in the vector database for subsequent retrieval and calling.

[0183] Among them, the first test text in the first case information and the first case result are both texts. The first test text and the first case result are vectorized. As an example, the first test text and the first case result are processed by bge-m3 (a vectorization model) respectively to obtain the vectorized first test text and the vectorized first case result, that is, the text vector. For the first test image data in the first case information, a balancing process is first performed and then a vectorization process is performed. The balancing process can be processed by a multi-scale MSR image enhancement algorithm to achieve color enhancement of the page display image and improve the image quality of the page display image. The first test image data can be vectorized by a histogram to obtain an image vector. The text vector and the image vector are merged and written into a vector database, and a hierarchical navigable small world graph (hnsw) index of the merged vector is established to facilitate vector query.

[0184] For the first and third correct recognition cases mentioned above, the test results of the database can be directly reused for similar test text and test image data.

[0185] Taking the first case data and the first case result as an example, the process of reusing the test results included in the first case result is described below.

[0186] Obtain a second test text and a second test image data. The second test text is used to describe the expected display result related to performing a second test operation on the second page. The second test image data includes a page display image related to performing a second test operation on the second page. It should be noted that the second page may be the same as the first page or different from the first page. The second test operation may be the same as the first test operation or different from the first test operation. This embodiment of the present application is not limited to this.

[0187] Determine the data similarity between the second test text and the second test image data and the first case data. The embodiment of the present application does not limit the method of determining the data similarity. In one possible implementation, the second test text and the second test image data are vectorized, and the obtained text vector and image vector are used to query the hnsw index in the database. If the corresponding vector is found in the database, it is determined that the data similarity is greater than the data similarity threshold.

[0188] In the case where the evaluation result included in the first case result indicates that the accuracy of the test result is greater than the accuracy threshold, the test result included in the first case result can be used as the test result of the second test operation. In this way, the test result in the TN or TP case can be directly used without calling the first language model to process the second test text and the second test image data, thereby reducing the cost of calling the first language model and improving the speed of feeding back the test result.

[0189] For similar test scenarios, the case information and case results stored in the database can be used to assist the first language model in understanding.

[0190] Taking the first case data and the first case result as an example, the process of assisting the first language model in processing data is described below.

[0191] Obtain a third test text and a third test image data. The third test text is used to describe the expected display result related to performing a third test operation on a third page. The third test image data includes a page display image related to performing a third test operation on the third page. It should be noted that the third page may be the same as or different from the first page. The third test operation may be the same as or different from the first test operation. This embodiment of the present application is not limited to this.

[0192] The third test operation has a test scene to which it belongs. The test scene can be determined based on the operation type of the third test operation, or based on the test content included in the third test text. The test scene to which the third test operation belongs is compared with the test scene to which the first test operation belongs to determine the scene similarity. The scene similarity can be determined based on the relevance of the pre-set test scenes. If the scene similarity is greater than the scene similarity threshold, it means that the third test operation and the first test operation are relatively similar. The first case data and the evaluation result are used as data to assist the first language model in understanding. The third test text, the third test image data, the first case data and the first case result are sent to the first language model to obtain the test result for the third test operation fed back by the first language model.

[0193] The first case data and the first case result may belong to TN or TP, or may belong to FP or FP. Using the case data and the case result in the FP or FP case may help the first language model analyze the error situation, help the first language model better understand and analyze new test text and test image data, and improve the accuracy of the first language model.

[0194] As an example, the following takes a large language model in which the first language model, the second language model, the third language model and the fourth language model are all multimodal as an example. Figure 5As shown, the method for generating the test results provided in the embodiment of the present application is described.

[0195] The test text and initial test image data are acquired. The initial test image data includes page display images before and after the first test operation is performed.

[0196] The test text and the initial test image data are used to calculate the similarity with the TN and TP in the RAG vector database. If the data similarity is greater than the data similarity threshold, the test result of the similar TN or TP is used as the test result of this test operation. If there is no similar TN or TP, the subsequent processing is performed.

[0197] Send the test text to the multimodal large language model and get the test content as feedback. Determine the corresponding image type based on the test content. According to the image type, determine whether it is necessary to compare the page display images before and after the test operation, and then determine the test image data.

[0198] The page display image included in the test image data is divided into regions to obtain a region annotation image. The region annotation image can be generated based on a document object model tree or based on an object detection model.

[0199] Send the region annotation image and the test text to the multimodal large language model to obtain the region identification of the target divided region. Update the page display image included in the test image data to the region display image of the target divided region. In addition, training data can also be sent to the multimodal large language model to assist the model in identifying the target divided region and improve the accuracy of identifying the target divided region.

[0200] Send test image data and prompt words indicating the description of image content to a multimodal large language model to obtain image content text.

[0201] Send the test image data, image content text, test text, and prompt words for indicating the generation of test results to the multimodal large language model to obtain the test results. In addition, the FP or FN data similar to the current test scenario can also be sent to the multimodal large language model to assist the model in generating more accurate test results.

[0202] In addition, you can also use FP or FN data to train the model and improve the accuracy of the model.

[0203] Based on the method for generating a test result provided in the above method embodiment, the present application embodiment further provides a device for generating a test result. The device for generating a test result will be described below in conjunction with the accompanying drawings.

[0204] See also Figure 6As shown in FIG. 1 , this figure is a schematic diagram of the structure of a test result generating device provided in an embodiment of the present application. Figure 6 As shown, the test result generating device includes:

[0205] A first acquisition unit 601 is used to acquire a first test text, where the first test text is used to describe an expected display result related to performing a first test operation on a first page;

[0206] A second acquiring unit 602 is configured to acquire first test image data of the first page, where the first test image data includes a page display image related to performing the first test operation on the first page;

[0207] The first generating unit 603 is used to send the first test text and the first test image data to the first language model, and obtain a test result for the first test operation fed back by the first language model, wherein the test result indicates whether the first test image data meets the expected display result described by the first test text.

[0208] In a possible implementation manner, the device further includes:

[0209] A first processing unit is configured to send the first test text to a second language model to obtain test content fed back by the second language model;

[0210] A type determination unit, used to determine the image type corresponding to the test content, wherein the image type is used to indicate the type of the acquired page display image;

[0211] The second acquisition unit 602 is specifically configured to acquire first test image data of the first page according to the image type.

[0212] In a possible implementation, the image type is a comparison type, and the second acquisition unit 602 is specifically configured to acquire a page display image before the first test operation is performed on the first page and a page display image after the first test operation is performed on the first page;

[0213] or,

[0214] The image type is a result type, and the second acquisition unit 602 is specifically configured to acquire a page display image after the first test operation is performed on the first page.

[0215] In a possible implementation manner, the device further includes:

[0216] The deleting unit is configured to delete the page display image before the first test operation from the first test image data if the image similarity between the page display image before the first test operation and the page display image after the first test operation is greater than an image similarity threshold.

[0217] In a possible implementation manner, the device further includes:

[0218] a dividing unit, configured to divide the page display image included in the first test image data into regions to obtain a region marking image, wherein the region marking image includes divided regions and region identifiers;

[0219] a second processing unit, configured to send the region annotation image and the first test text to a fourth language model, and obtain a region identifier of a target segmentation region fed back by the fourth language model, wherein the target segmentation region is a segmentation region related to the first test operation;

[0220] An updating unit is used to update the page display image included in the first test image data to the area display image of the target divided area.

[0221] In a possible implementation, the division unit is specifically used to analyze the document object model tree of the first page to obtain the page element area of ​​the first page; filter the repeated page element areas and merge the overlapping page element areas to obtain a region division image; and generate a region annotation image based on the region division image.

[0222] In a possible implementation, the segmentation unit is used to process the first test image data using a target detection model to obtain a region segmentation result, where the region segmentation result is used to indicate a segmented region; and generate a region annotation image based on the region segmentation result.

[0223] In a possible implementation manner, the second processing unit, configured to send the region annotation image and the first test text to a fourth language model, includes:

[0224] The second processing unit is used to send the region annotation image, the first test text and training data to a fourth language model, where the training data includes test image sample data and region identifiers of corresponding correct target segmentation data, and the training data is used to assist the fourth language model in generating region identifiers of the target segmentation regions.

[0225] In a possible implementation manner, the device further includes:

[0226] The third processing unit is configured to send the first test text to a third language model to obtain a modified first test text fed back by the third language model if it is determined that the first test text does not meet the analysis condition.

[0227] In a possible implementation manner, the device further includes:

[0228] A third acquisition unit, used to acquire an evaluation result corresponding to the test result, wherein the evaluation result is used to measure the correctness of the test result;

[0229] The storage unit is used to store first case data and first case results in a database, wherein the first case data includes the first test text and the first test image data, and the first case results include the test result and the evaluation result.

[0230] In a possible implementation manner, the device further includes:

[0231] a fourth acquisition unit, configured to acquire a second test text and second test image data, wherein the second test text is used to describe an expected display result associated with performing a second test operation on a second page, and the second test image data includes a page display image associated with performing the second test operation on the second page;

[0232] The second generating unit is used to use the test result as the test result of the second test operation if the data similarity between the second test text and the second test image data and the first case data is greater than the data similarity threshold, and the correctness of the test result is greater than the correctness threshold.

[0233] In a possible implementation manner, the device further includes:

[0234] a fifth acquisition unit, configured to acquire a third test text and third test image data, wherein the third test text is used to describe an expected display result associated with performing a third test operation on a third page, and the third test image data includes a page display image associated with performing the third test operation on the third page;

[0235] a scene determination unit, configured to determine whether a scene similarity between a test scene to which the third test operation belongs and a test scene to which the first test operation belongs is greater than a scene similarity threshold;

[0236] The third generating unit is used to send the third test text, the third test image data, the first case data and the first case result to the first language model to obtain the test result of the third test operation fed back by the first language model.

[0237] In a possible implementation, the first generation unit 603 is specifically configured to send the first test image data and a first prompt word to the first language model, where the first prompt word is used to instruct the first language model to describe the image content; obtain the image content text fed back by the first language model; send the first test image data, the image content text, the first test text, and a second prompt word to the first language model, where the second prompt word is used to instruct the first language model to generate a test result; and obtain the test result fed back by the first language model.

[0238] Based on a method for generating a test result provided in the above method embodiment, the present application also provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a test result as described in any of the above embodiments.

[0239] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0240] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0241] Typically, the following devices may be connected to the I / O interface 705: input devices including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0242] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0243] The electronic device provided in the embodiment of the present application and the method for generating test results provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0244] Based on a method for generating test results provided in the above method embodiments, an embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the program is executed by a processor, the method for generating test results applied to an operating system of a device as described in any of the above embodiments is implemented.

[0245] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0246] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0247] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0248] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method for generating the test result.

[0249] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0250] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0251] The units involved in the embodiments described in this application may be implemented by software or hardware. The name of a unit / module does not limit the unit itself in some cases. For example, a voice data acquisition module may also be described as a "data acquisition module".

[0252] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0253] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0254] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0255] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0256] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0257] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a test result, characterized in that: The method comprises: Acquire a first test text, where the first test text is used to describe an expected display result related to performing a first test operation on a first page; Acquire first test image data of the first page, wherein the first test image data includes a page display image related to performing the first test operation on the first page; The first test text and the first test image data are sent to a first language model to obtain a test result for the first test operation fed back by the first language model, wherein the test result indicates whether the first test image data conforms to an expected display result described by the first test text.

2. The method according to claim 1, characterized in that The method further comprises: Sending the first test text to a second language model to obtain test content fed back by the second language model; Determine an image type corresponding to the test content, where the image type is used to indicate a type of the acquired page display image; The obtaining of the first test image data of the first page includes: According to the image type, first test image data of the first page is acquired.

3. The method according to claim 2, characterized in that The image type is a comparison type, and obtaining first test image data of the first page according to the image type includes: Acquire a page display image before executing the first test operation on the first page and a page display image after executing the first test operation; or, The image type is a result type, and obtaining first test image data of the first page according to the image type includes: A page display image is obtained after the first test operation is performed on the first page.

4. The method according to claim 2, characterized in that: The method further comprises: If the image similarity between the page display image before the first test operation and the page display image after the first test operation is performed is greater than an image similarity threshold, the page display image before the first test operation is deleted from the first test image data.

5. The method according to claim 1, characterized in that The method further comprises: Performing region division on the page display image included in the first test image data to obtain a region marking image, wherein the region marking image includes a divided region and a region identifier; Sending the region annotated image and the first test text to a fourth language model to obtain a region identifier of a target segmented region fed back by the fourth language model, where the target segmented region is a segmented region related to the first test operation; The page display image included in the first test image data is updated to the area display image of the target divided area.

6. The method according to claim 5, characterized in that The step of dividing the page display image included in the first test image data into regions to obtain a region-marked image includes: Analyze the document object model tree of the first page to obtain a page element area of ​​the first page; Filtering the repeated page element regions and merging the overlapping page element regions to obtain a region division image; Based on the region division image, a region annotation image is generated.

7. The method according to claim 5, characterized in that The step of dividing the page display image included in the first test image data into regions to obtain a region-marked image includes: Processing the first test image data using the target detection model to obtain a region division result, wherein the region division result is used to indicate a divided region; Based on the region division result, a region annotation image is generated.

8. The method according to claim 5, characterized in that The sending the region annotated image and the first test text to the fourth language model includes: The region annotated image, the first test text, and training data are sent to a fourth language model, where the training data includes test image sample data and region identifiers of corresponding correct target segmentation data, and the training data is used to assist the fourth language model in generating region identifiers of the target segmentation regions.

9. The method according to claim 1, characterized in that: The method further comprises: If it is determined that the first test text does not meet the analysis condition, the first test text is sent to a third language model to obtain a modified first test text fed back by the third language model.

10. The method according to claim 1, characterized in that The method further comprises: Obtaining an evaluation result corresponding to the test result, wherein the evaluation result is used to measure the correctness of the test result; The first case data and the first case result are stored in a database, wherein the first case data includes the first test text and the first test image data, and the first case result includes the test result and the evaluation result.

11. The method according to claim 10, characterized in that The method further comprises: Acquire a second test text and second test image data, wherein the second test text is used to describe an expected display result related to performing a second test operation on a second page, and the second test image data includes a page display image related to performing the second test operation on the second page; If the data similarity between the second test text and the second test image data and the first case data is greater than the data similarity threshold, and the accuracy of the test result is greater than the accuracy threshold, the test result is used as the test result of the second test operation.

12. The method according to claim 10, characterized in that The method further comprises: Acquire a third test text and third test image data, wherein the third test text is used to describe an expected display result associated with performing a third test operation on a third page, and the third test image data includes a page display image associated with performing the third test operation on the third page; Determine that a scene similarity between a test scene to which the third test operation belongs and a test scene to which the first test operation belongs is greater than a scene similarity threshold; The third test text, the third test image data, the first case data, and the first case result are sent to the first language model to obtain a test result for the third test operation fed back by the first language model.

13. The method according to any one of claims 1 to 12, characterized in that: The sending the first test text and the first test image data to the first language model to obtain a test result for the first test operation fed back by the first language model includes: Sending the first test image data and a first prompt word to the first language model, where the first prompt word is used to instruct the first language model to describe image content; Obtaining the image content text fed back by the first language model; Sending the first test image data, the image content text, the first test text, and a second prompt word to the first language model, where the second prompt word is used to instruct the first language model to generate a test result; A test result of obtaining feedback of the first language model.

14. A device for generating test results, characterized in that: The device comprises: A first acquisition unit, configured to acquire a first test text, wherein the first test text is used to describe an expected display result related to performing a first test operation on a first page; A second acquiring unit, configured to acquire first test image data of the first page, wherein the first test image data includes a page display image related to performing the first test operation on the first page; A first generating unit is used to send the first test text and the first test image data to a first language model, and obtain a test result for the first test operation fed back by the first language model, wherein the test result indicates whether the first test image data conforms to an expected display result described by the first test text.

15. An electronic device, characterized in that: include: one or more processors; A storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.

16. A computer readable medium, characterized in that A computer program is stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images