Page test method and device based on large model inspection, medium and equipment

Abstract

According to the page testing method and device based on large model inspection, the medium and the equipment provided by the invention, the focus is accurately anchored by capturing the operation event and positioning the corresponding target control. Afterwards, the local graph is intercepted according to the control coordinates and the expansion proportion, on one hand, dynamic noise such as advertisements and recommendation streams in the page is effectively eliminated, and the model is enabled to focus on the to-be-inspected area. On the other hand, a certain proportion is expanded outwards during interception, and necessary context information is reserved. And then the resolution of the local image is further detected and adaptive amplification is carried out, so that the definition of fine elements in the local image is ensured, and the problem of detail blurring caused by image compression is solved. And finally, inputting the processed high-definition target local graph into the large model for inspection and judgment, so that the large model can concentrate on accurate analysis of the state of the target control on the premise of keeping complete context and key visual details, thereby improving the accuracy and anti-interference capability of UI test inspection.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to a page testing method, apparatus, medium and equipment based on large model inspection. Background Technology

[0002] In the current field of automated UI testing, using multimodal large models for interface inspection is becoming an emerging technical approach. This solution typically relies on automated scripts to perform single-step operations, capture a full-screen image of the device, and directly input it into a large model, which then determines whether the interface has any anomalies or whether the state of specific elements is normal.

[0003] However, full-screen screenshots contain numerous dynamic visual elements irrelevant to the testing objective, such as advertising banners and recommendation feeds. This noise can interfere with the model's attention, leading to misjudgments of unrelated regions. Furthermore, image compression to adapt to model input limitations can blur crucial details like price tags and inventory labels, affecting the recognition of subtle errors. In summary, the current testing accuracy based on large-model inspection is relatively low. Summary of the Invention

[0004] The purpose of this application is to address at least one of the aforementioned technical shortcomings, particularly the issue that existing full-screen screenshots contain numerous dynamic visual elements unrelated to the testing target, leading to misjudgments of unrelated areas. Furthermore, image compression to adapt to model input limitations blurs crucial details such as price tags and inventory labels, affecting the identification of subtle errors. This, in turn, results in the relatively low accuracy of current large-model-based inspection tests.

[0005] Firstly, this application provides a page testing method based on large-model inspection, the method comprising:

[0006] When an operation event is captured in the target application, the target control that the test task corresponding to the operation event is concerned with is determined;

[0007] Obtain the current application page, determine the expansion ratio based on the target control, and capture a partial image of the application page according to the coordinate position of the target control and the expansion ratio;

[0008] The resolution of the local image is detected, and when the resolution is lower than a preset recognition threshold, the local image is magnified to obtain a target local image;

[0009] The target local graph is input into a preset large model to obtain the UI test inspection results for the operation event.

[0010] In one embodiment, determining the target control of interest for the test task corresponding to the operation event includes:

[0011] Obtain the test document and the tree structure of the page where the operation event was triggered;

[0012] The operation event is parsed to determine the test task corresponding to the operation event in the test document, and to obtain the expected change elements of the test task;

[0013] The control element corresponding to the expected change element is located in the tree structure, and the found control element is determined as the target control.

[0014] In one embodiment, determining the expansion ratio based on the target control includes:

[0015] Determine the control type and control size of the target control;

[0016] Query the base expansion ratio corresponding to the control type and the control size;

[0017] The density of interfering elements on the application page is analyzed, and the basic expansion ratio is dynamically adjusted based on the analysis results to obtain the expansion ratio.

[0018] In one embodiment, the step of capturing a partial image in the application page based on the coordinate position of the target control and the scaling factor includes:

[0019] The target area is determined based on the coordinate position of the target control within the application page, and then expanded outward from the periphery of that area according to the expansion ratio to form the target area.

[0020] The target area is cropped out from the application page to obtain a partial image.

[0021] In one embodiment, the step of magnifying the local image to obtain the target local image includes:

[0022] Obtain the preset super-resolution model;

[0023] The local map is input into the super-resolution model so that the super-resolution model reconstructs a high-resolution image on the local map and outputs the high-resolution image as the target local map.

[0024] In one embodiment, the step of inputting the target local image into a preset large model to obtain UI test inspection results for the operation event includes:

[0025] Determine the semantic information and control type of the target control;

[0026] Based on the business type of the application page and the semantic information and control type of the target control, determine the model prompt words;

[0027] The model prompt and the target local image are input into a preset large model, so that the large model can perform verification analysis on the target local image based on the model prompt and output UI test inspection results for the operation event.

[0028] In one embodiment, after obtaining the UI test inspection results for the operation event, the method further includes:

[0029] Capture a full-screen image of the application page and evaluate the overall layout of the application page based on the full-screen image;

[0030] Evaluate the business logic of the target control based on the target partial diagram;

[0031] Based on the overall layout and business logic, the credibility of the UI test inspection results is determined. If the credibility does not reach the preset threshold, the page test method based on large model inspection is re-executed.

[0032] Secondly, this application provides a page testing device based on large-model inspection, the device comprising:

[0033] The control determination module is used to determine the target control that the test task corresponding to the operation event is interested in when an operation event is captured in the target application.

[0034] The partial capture module is used to obtain the current application page, determine the expansion ratio according to the target control, and capture a partial image in the application page according to the coordinate position of the target control and the expansion ratio.

[0035] An image processing module is used to detect the resolution of the local image, and when the resolution is lower than a preset recognition threshold, to magnify the local image to obtain a target local image;

[0036] The test inspection module is used to input the target local image into a preset large model to obtain the UI test inspection results for the operation event.

[0037] Thirdly, this application provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the page testing method based on large model inspection as described in any of the above embodiments.

[0038] Fourthly, this application provides a computer device, including: one or more processors, and a memory;

[0039] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, they perform the steps of the page testing method based on large model inspection as described in any of the above embodiments.

[0040] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0041] This application provides a page testing method, apparatus, medium, and equipment based on a large model inspection. By capturing operation events and locating the target controls of interest in the testing task, it achieves precise anchoring of the expected focus points for testing. Then, based on the control coordinates and expansion ratio, a partial image is captured from the current application page. This effectively eliminates dynamic noise such as advertisements and recommendation streams, allowing the model to focus its attention on the area to be inspected. Furthermore, the image is expanded outwards by a certain ratio during capture to retain necessary contextual information. The resolution of the partial image is then further detected and adaptively enlarged to ensure the clarity of subtle elements and overcome the blurring of details caused by image compression. Finally, the processed high-resolution target partial image is input into a large model for inspection and judgment. This allows the large model to focus on the precise analysis of the target control's state while preserving complete context and key visual details, thereby improving the accuracy and anti-interference capability of UI testing inspection. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a page testing method based on large model inspection provided in this application embodiment;

[0044] Figure 2 A schematic diagram of the structure of a page testing device based on large model inspection provided in an embodiment of this application;

[0045] Figure 3 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In one embodiment, this application provides a page testing method based on large model inspection. The following embodiments illustrate the application of this method to a server. It is understood that the page testing method based on large model inspection can be performed on a single server or a server cluster consisting of multiple servers, and this application does not impose any specific limitations on this.

[0048] like Figure 1 As shown, this application provides a page testing method based on large model inspection, the method comprising:

[0049] S101: When an operation event is captured in the target application, determine the target control that the test task corresponding to the operation event is concerned with.

[0050] In this context, an action event refers to a user interaction triggered by the test script, including but not limited to clicks, swipes, long presses, and input. A target control is a page element directly associated with the current action event and whose state needs to be verified or monitored, such as a button, input box, or menu item. A test task is a predefined, specific test objective that specifies the page function or state to be verified, as well as the expected performance.

[0051] In this step, testers can use test scripts to automate page testing. Each time an action event is triggered on the page, the server can capture the event through a predefined event listener mechanism. Then, based on the captured action events, the current test task is determined, and further, the target controls whose state needs to be monitored within that test task are identified.

[0052] S102: Obtain the current application page, determine the expansion ratio based on the target control, and capture a partial image from the application page based on the coordinate position of the target control and the expansion ratio.

[0053] The expansion ratio refers to the proportion by which the target control is extended outwards from its original size. A partial view refers to an image captured from the application page, including the target control and a certain area around it.

[0054] In this step, after identifying the target control, the current application page of the target application can be obtained, and the expansion ratio can be dynamically determined based on information such as the control type of the target control. For example, the expansion ratio can be dynamically adjusted based on the size of the target control, the complexity of the page layout, and the specific needs of the test target; this application does not impose specific limitations on this. For instance, if the target control is a small button, the expansion ratio may be set relatively large to ensure that the captured image contains sufficient contextual information; while if the target control is a large form area, the expansion ratio can be appropriately reduced. Then, the boundary of the area to be captured is calculated based on the coordinate position of the target control and the expansion ratio, and this area is then captured from the application page to obtain a partial image.

[0055] S103: Detect the resolution of the local image, and when the resolution is lower than the preset recognition threshold, magnify the local image to obtain the target local image.

[0056] The preset recognition threshold is an empirical value and can be set or adjusted according to actual conditions. This application does not impose specific restrictions on it. The target local image refers to the local image after resolution optimization.

[0057] In this step, after capturing a partial image containing the target control, the server first checks the resolution of that partial image. Resolution detection can be done by analyzing the image's pixel density or size, ensuring the image's sharpness meets a preset recognition threshold so that the larger model can make judgments based on more accurate and richer pixel information. If the resolution of the partial image is detected to be lower than the preset recognition threshold, a magnification process will be automatically initiated. Specifically, magnification can be implemented using various algorithms, such as bilinear interpolation, more advanced image magnification algorithms, or deep learning models, to ensure that the image maintains high sharpness even after partial capture.

[0058] For example, suppose the captured partial image contains a price tag, but its low resolution causes the price number to be blurry. When this situation is detected, the partial image will be automatically enlarged to make the details of the price tag clearer for testing and inspection.

[0059] S104: Input the target local image into the preset large model to obtain the UI test inspection results for the operation event.

[0060] Among them, the large model refers to an artificial intelligence model that has been trained and possesses multimodal understanding capabilities. UI test inspection results are used to reflect the difference between the feedback of operational events and the expected performance.

[0061] In this step, when the target local image is determined, it is input into the preset large model so that the large model can analyze the target local image, determine whether it is the same as the expected performance, and then output the UI test inspection results about the operation event.

[0062] Specifically, the large model of this application can use local map samples as training data and the corresponding real inspection results as label data. The pre-trained model is iteratively trained using the training data. During training, a loss function is constructed based on the output of the pre-trained model and the label data, and the pre-trained model is updated based on the loss function until a preset iteration condition is met. The completed pre-trained model is then determined as the large model. Furthermore, the preset iteration condition can be set to reach a preset number of iterations or to the loss accuracy being less than a preset threshold. This application does not impose specific limitations on this.

[0063] In the above embodiments, by capturing operation events and locating the target controls of interest in the test task, precise anchoring of the expected focus points of the test is achieved. Then, a partial image is captured from the current application page based on the control coordinates and expansion ratio. This effectively eliminates dynamic noise such as advertisements and recommendation streams on the page, allowing the model to focus its attention on the area to be inspected. Furthermore, the image is expanded outwards by a certain ratio during capture to retain necessary contextual information. The resolution of the partial image is then further detected and adaptively enlarged to ensure the clarity of subtle elements and overcome the blurring of details caused by image compression. Finally, the processed high-resolution target partial image is input into the large model for inspection and judgment. This allows the large model to focus on the precise analysis of the target control's state while preserving complete context and key visual details, thereby improving the accuracy and anti-interference capability of UI test inspection.

[0064] In one embodiment, determining the target control of interest for the test task corresponding to the operation event includes:

[0065] S1: Obtain the tree structure of the test document and the page where the operation event was triggered.

[0066] S2: Parse the operation event to identify the test task corresponding to the operation event in the test document, and obtain the expected change elements of the test task.

[0067] S3: Locate the control element in the tree structure that corresponds to the element expected to change, and identify the found control element as the target control.

[0068] In this context, "test documentation" refers to a document that details the test tasks, test cases, expected results, and operational steps. "Tree structure" refers to the hierarchical relationship of page elements, typically represented as a DOM (Document Object Model) tree. "Expectedly changing elements" are page elements defined in the test task that are expected to change after the execution of operational events.

[0069] In this embodiment, the test document and the tree structure of the page where the operation event is triggered can be obtained first. The test document provides detailed information about the test task, including operation steps and expected results, while the tree structure describes the hierarchical relationship of page elements, facilitating the location of the target control. During execution, the server parses the test document, determines the test task corresponding to the operation event, and extracts the expected change element from it. For example, the test document may define a test task that requires verifying that clicking the "Submit" button will cause a pop-up window to appear on the page asking whether to confirm submission. The expected change element is the area where this pop-up window is located. Next, the control element corresponding to the expected change element can be found in the page's tree structure. This can be achieved by traversing the DOM tree and matching the control's attributes (such as ID, class name, or tag name). Once a matching control element is found, it is identified as the target control.

[0070] It is understandable that parsing operation events and extracting expected change elements allows the system to focus on page elements directly related to the operation events, avoiding misjudgments and redundant operations on irrelevant elements, thereby improving the accuracy and reliability of subsequent large-scale model inspections.

[0071] In one embodiment, determining the expansion ratio based on the target control includes:

[0072] S1: Determine the control type and size of the target control.

[0073] S2: Queries the base extension ratio corresponding to the control type and control size.

[0074] S3: Analyze the density of interfering elements on the application page, and dynamically adjust the basic expansion ratio based on the analysis results to obtain the expansion ratio.

[0075] Among them, control type refers to the function and appearance of the target control in the UI interface, such as buttons, text boxes, drop-down menus, and images. Interference element density refers to the number and distribution of dynamic or static elements unrelated to the test target within a unit area around the target control.

[0076] In this embodiment, the control type and size of the target control can first be determined by parsing the tree structure of the page. Then, the label name, attribute values, and size information of the target control on the page can be obtained. For example, if the target control is a button, the system will recognize it as... <button>The label is identified, and its width and height are obtained to determine its control type and size. Next, a pre-defined mapping is retrieved, and then the base expansion ratio corresponding to the control type and size is queried from that mapping. For example, the base expansion ratio might be 1.2 for a small button, while it might be 1.5 for a large form.

[0077] Next, the density of interfering elements on the application page is analyzed by detecting the number or proportion of irrelevant elements in the area surrounding the target control. Then, the base expansion ratio is dynamically adjusted based on this density to obtain the final expansion ratio of the application. It can be understood that in areas with high interference density, the expansion ratio may be moderately reduced to decrease noise introduction; while in areas with low interference density and a simple layout, the expansion ratio can be maintained or even moderately increased to retain more benign context that aids in judgment.

[0078] Specifically, the mapping relationship described above is used to record the correspondence between control types and sizes in the target application and the basic expansion ratio. Furthermore, when analyzing the density of interfering elements on the application page, visual feature analysis of the surrounding area of ​​the target control can be performed in conjunction with the test task to determine the number or proportion of irrelevant elements, thus obtaining the interfering element density. Further, after obtaining the interfering element density, it can be matched with predefined density grading conditions to determine the basic expansion ratio.

[0079] Understandably, this embodiment determines the basic expansion ratio based on the control type and size, and dynamically adjusts it in conjunction with the density of page distracting elements. This achieves intelligent and context-adaptive cropping range. It ensures that the cropping strategy matches the functional characteristics and visual requirements of the control itself, providing differentiated context preservation benchmarks for different types of controls. More importantly, by dynamically sensing the degree of interference in the local page environment, it flexibly achieves the optimal balance between "preserving necessary context" and "eliminating irrelevant noise," thereby generating a highly focused local image with the highest quality contextual information.

[0080] In one embodiment, a partial image is captured from the application page based on the coordinate position and scaling of the target control, including:

[0081] S1: Determine the area of ​​the target control in the application page based on its coordinate position, and expand outward from the periphery of the area according to the expansion ratio to form the target area.

[0082] S2: Crops out the target area in the application page to obtain a partial image.

[0083] In this embodiment, the coordinates of the target control are obtained through the tree structure of the application page. These coordinates are typically expressed in pixels, with the top-left corner of the page as the origin. Simultaneously, the initial area of ​​the target control within the application page is determined by combining its size information (width and height). For example, if the target control's coordinates are (x, y), its width is w, and its height is h, then its initial area can be represented as a rectangular region starting from (x, y), with a width of w and a height of h.

[0084] Subsequently, the target area can be expanded outward from the perimeter of the rectangular area containing the target control, based on the determined expansion ratio. For example, assuming the coordinates of the rectangular area containing the target control are (x1=100, y1=1800, x2=280, y2=1850), with a width of 180 and a height of 50, if the expansion ratio is 10%, then each side expands by 18 pixels horizontally and 5 pixels vertically. The coordinates of the expanded target area can then be represented as (x1'=100-18=72, y1'=1800-5=1795, x2'=280+18=298, y2'=1850+5=1855), or (x1'=72, y1'=1795, x2'=298, y2'=1855).

[0085] Specifically, determining the initial area by the coordinates of the target control ensures a clear and precise focus for the test, avoiding misjudgments of irrelevant areas. Then, expanding outwards from the perimeter according to the expansion ratio to form the target area provides sufficient contextual information for the test while avoiding excessive interference from irrelevant content. This expansion mechanism can be flexibly adjusted according to the characteristics of the target control and the actual page environment, ensuring that the partial image contains the necessary contextual information without affecting the accuracy of the test due to excessive distracting elements.

[0086] In one embodiment, the local image is magnified to obtain the target local image, including:

[0087] S1: Obtain the preset super-resolution model.

[0088] S2: Input the local map into the super-resolution model so that the super-resolution model can reconstruct a high-resolution image on the local map and output the high-resolution image as the target local map.

[0089] Among them, the super-resolution model can learn the complex mapping relationship between low-resolution and high-resolution images, intelligently predict and supplement detailed information, thereby reconstructing the input low-resolution image into a visually clearer and more detailed high-resolution image.

[0090] In this embodiment, to ensure the clarity and detail of the local image, a pre-trained super-resolution model is obtained. This super-resolution model is a trained deep learning model capable of reconstructing a low-resolution image into a high-resolution image. Specifically, the local image is input into the super-resolution model, which processes the local image to reconstruct a high-resolution image, i.e., the target local image.

[0091] Specifically, during training, the super-resolution model of this application can use a large number of high- and low-resolution image sets as training data to learn the mapping relationship between low-resolution and high-resolution images. During training, a loss function is constructed to minimize the difference between the reconstructed image and the real high-resolution image, and backpropagation is performed based on the loss function to continuously adjust the parameters of the super-resolution model until a preset training condition is met, resulting in the final super-resolution model. The preset training condition can be set to reaching a preset number of training iterations, or it can be set to the absolute value of the difference between the loss values ​​of two adjacent iterations being less than a preset loss threshold; this application does not impose specific restrictions on this.

[0092] Understandably, this embodiment utilizes prior knowledge learned by the model from massive amounts of data to perform intelligent reasoning and realistic reconstruction of lost textures, edges, and structural details. This improves the visual clarity and recognizability of tiny text, fine icons, and complex interface elements in the target local image, providing higher-density and higher-quality visual input for subsequent large-scale model analysis, thereby improving the accuracy and reliability of UI testing.

[0093] In one embodiment, in addition to using a super-resolution model to process the local map, an image interpolation algorithm can be used to process the local map to obtain a higher resolution target local map, taking into account factors such as performance.

[0094] In one embodiment, the target local graph is input into a preset large model to obtain UI test inspection results for the operation event, including:

[0095] S1: Determine the semantic information and control type of the target control.

[0096] S2: Determine model prompt words based on the business type of the application page, the semantic information of the target control, and the control type.

[0097] S3: Input the model prompts and target local graph into the preset large model, so that the large model can perform verification analysis on the target local graph based on the model prompts and output the UI test inspection results about the operation events.

[0098] Here, "business type" refers to the business domain or functional module to which the application page belongs, such as a shopping cart module or a homepage module. "Model prompts" refer to additional contextual information or instructions provided to the large model to guide it in performing specific validation analyses on the target local graph. For example, a model prompt could be set to "Based on this enlarged screenshot of the product price area, determine whether the price numbers are obscured by lines and whether the currency symbols are displayed completely."

[0099] In this embodiment, the semantic information and control type of the target control are determined. The semantic information refers to the function and content description of the control. Next, based on the business type of the application page and the semantic information and control type of the target control, matching prompts are searched in a pre-set prompt word library, and the found prompts are identified as model prompts. Then, the model prompts and the target local... Figure 1 The data is then input into the large model, which uses the model prompts as task guidance, focuses on the target local graph for verification analysis, and finally outputs the UI test inspection results for the operation events.

[0100] Specifically, determining model hints based on business type and control information provides concrete guidance to the large model, helping it better understand test objectives and expected results. The model hints and target local images are then input into the large model for verification analysis, enabling the model to accurately analyze and judge the image based on the hints, outputting accurate test inspection results. This approach effectively improves the accuracy and reliability of testing, reducing misjudgments caused by insufficient contextual information.

[0101] In one embodiment, after obtaining the UI test inspection results for the operation event, the page testing method for large model inspection further includes:

[0102] S1: Capture a full-screen image of the application page and evaluate the overall layout of the application page based on the full-screen image.

[0103] S2: Evaluate the business logic of the target control based on the target local diagram.

[0104] S3: Based on the overall layout and business logic, determine the credibility of the UI test inspection results. If the credibility does not reach the preset threshold, re-execute the page test method based on the large model inspection.

[0105] The overall layout refers to the global layout status of the application page, including the arrangement, alignment, spacing, and other visual effects of page elements. The business logic status refers to the assessment of whether the execution and presentation of the specific functions or business processes carried by the target controls are correct.

[0106] In this embodiment, on the one hand, a full-screen image of the application page is captured, and the overall layout of the application page is evaluated based on the full-screen image. The full-screen image provides overall visual information about the page, including the arrangement, alignment, and spacing of elements. At this time, image analysis algorithms can be used to detect whether the page layout conforms to design specifications, such as checking whether elements are aligned, whether there is overlap, blank screens, or misalignment. On the other hand, the business logic of the target control is evaluated based on the target partial image. The target partial image focuses on the target control and its surrounding area. At this time, the control state, displayed content, and interaction logic in the partial image can be analyzed to determine whether it meets expectations. For example, price display, text display, and icon integrity.

[0107] Subsequently, based on the overall layout and business logic, these two aspects can be scored, and the credibility of the UI test inspection results can be calculated using a weighted summation method. If the credibility does not reach a preset threshold, the operation events of this test are re-tested. Specifically, when scoring the overall layout and business logic, pre-set deduction and bonus items can be used. When a deduction or bonus item is matched, the corresponding score is added or subtracted to obtain the score for the overall layout and business logic.

[0108] By introducing a full-screen view to assess the overall layout and combining it with business logic evaluation of the target local view for a comprehensive credibility judgment, retrying is triggered when credibility is insufficient. Based on macro-layout checks, invalid or misleading fine-grained analyses caused by global issues such as page loading failures and frame corruption are effectively filtered out. Simultaneously, local logic checks can identify result fluctuations caused by transient interface states, image capture defects, or uncertainties in single-inference iterations of the model. Through this process, low-quality or uncertain inspection conclusions are automatically discarded, and more reliable verification is proactively sought, improving the reliability of the test.

[0109] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0110] The following describes the page testing device based on large model inspection provided in the embodiments of this application. The page testing device based on large model inspection described below can be referred to in correspondence with the page testing method based on large model inspection described above.

[0111] like Figure 2 As shown, this application provides a page testing device 200 based on large model inspection, the device comprising:

[0112] The control determination module 201 is used to determine the target control of the test task corresponding to the operation event when an operation event is captured in the target application.

[0113] The partial capture module 202 is used to obtain the current application page, determine the expansion ratio according to the target control, and capture a partial image in the application page based on the coordinate position of the target control and the expansion ratio.

[0114] Image processing module 203 is used to detect the resolution of the local image and, when the resolution is lower than a preset recognition threshold, to magnify the local image to obtain the target local image;

[0115] The test inspection module 204 is used to input the target local image into the preset large model to obtain the UI test inspection results for the operation event.

[0116] In the above embodiments, by capturing operation events and locating the target controls of interest in the test task, precise anchoring of the expected focus points of the test is achieved. Then, a partial image is captured from the current application page based on the control coordinates and expansion ratio. This effectively eliminates dynamic noise such as advertisements and recommendation streams on the page, allowing the model to focus its attention on the area to be inspected. Furthermore, the image is expanded outwards by a certain ratio during capture to retain necessary contextual information. The resolution of the partial image is then further detected and adaptively enlarged to ensure the clarity of subtle elements and overcome the blurring of details caused by image compression. Finally, the processed high-resolution target partial image is input into the large model for inspection and judgment. This allows the large model to focus on the precise analysis of the target control's state while preserving complete context and key visual details, thereby improving the accuracy and anti-interference capability of UI test inspection.

[0117] In one embodiment, the control determination module includes:

[0118] The structure acquisition submodule is used to obtain the tree structure of the test document and the page where the operation event is triggered;

[0119] The event parsing submodule is used to parse operation events to identify the test tasks corresponding to the operation events in the test document and obtain the expected change elements of the test tasks.

[0120] The control determination submodule is used to find the control element that corresponds to the expected change element in the tree structure, and to determine the found control element as the target control.

[0121] In one embodiment, the partial capture module includes:

[0122] The information determination submodule is used to determine the control type and control size of the target control;

[0123] The ratio query submodule is used to query the basic extended ratio corresponding to the control type and control size;

[0124] The scaling adjustment submodule is used to analyze the density of interfering elements on the application page and dynamically adjust the basic scaling ratio based on the analysis results to obtain the scaling ratio.

[0125] In one embodiment, the partial capture module includes:

[0126] The region expansion submodule is used to determine the region of the target control in the application page based on its coordinate position, and expand outward from the periphery of the region according to the expansion ratio to form the target region;

[0127] The region cropping submodule is used to crop out the target region in the application page to obtain a partial image.

[0128] In one embodiment, the image processing module includes:

[0129] The model acquisition submodule is used to acquire preset super-resolution models;

[0130] The image processing submodule is used to input the local map into the super-resolution model so that the super-resolution model can reconstruct a high-resolution image on the local map and output the high-resolution image as the target local map.

[0131] In one embodiment, the test inspection module includes:

[0132] The information determination submodule is used to determine the semantic information and control type of the target control;

[0133] The prompt word determination submodule is used to determine model prompt words based on the business type of the application page, as well as the semantic information and control type of the target control;

[0134] The test inspection submodule is used to input model prompts and target local images into a preset large model, so that the large model can perform verification analysis on the target local image based on the model prompts and output UI test inspection results about operation events.

[0135] In one embodiment, the page testing apparatus based on large model inspection further includes:

[0136] The first evaluation module is used to capture a full-screen image of the application page and evaluate the overall layout of the application page based on the full-screen image.

[0137] The second evaluation module is used to evaluate the business logic of the target control based on the target partial diagram.

[0138] The feasibility assessment module is used to determine the credibility of the UI test inspection results based on the overall layout and business logic. If the credibility does not reach the preset threshold, the page test method based on the large model inspection will be re-executed.

[0139] The division of modules in the above-described page testing device based on large-model inspection is merely illustrative. In other embodiments, the page testing device based on large-model inspection can be divided into different modules as needed to complete all or part of its functions. Each module in the above-described page testing device based on large-model inspection can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0140] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the page testing method based on large model inspection as described in any of the above embodiments.

[0141] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the page testing method based on large model inspection as described in any of the above embodiments.

[0142] Indicatively, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 3 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the page testing method based on large model inspection of any of the above embodiments.

[0143] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0144] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, the singular forms "a," "an," and "the" may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "comprising / including" or "having" specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term "and / or" as used in this specification includes any and all combinations of the associated listed items.

[0146] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0147] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / button>

Claims

1. A page testing method based on large model inspection, characterized in that, The method includes: When an operation event is captured in the target application, the target control that the test task corresponding to the operation event is concerned with is determined; Obtain the current application page, determine the expansion ratio based on the target control, and capture a partial image of the application page according to the coordinate position of the target control and the expansion ratio; The resolution of the local image is detected, and when the resolution is lower than a preset recognition threshold, the local image is magnified to obtain a target local image; The target local graph is input into a preset large model to obtain the UI test inspection results for the operation event.

2. The page testing method based on large model inspection according to claim 1, characterized in that, The step of determining the target control of the test task corresponding to the operation event includes: Obtain the test document and the tree structure of the page where the operation event was triggered; The operation event is parsed to determine the test task corresponding to the operation event in the test document, and to obtain the expected change elements of the test task; The control element corresponding to the expected change element is located in the tree structure, and the found control element is determined as the target control.

3. The page testing method based on large model inspection according to claim 1, characterized in that, Determining the expansion ratio based on the target control includes: Determine the control type and control size of the target control; Query the base expansion ratio corresponding to the control type and the control size; The density of interfering elements on the application page is analyzed, and the basic expansion ratio is dynamically adjusted based on the analysis results to obtain the expansion ratio.

4. The page testing method based on large model inspection according to claim 1, characterized in that, The step of capturing a partial image from the application page based on the coordinate position of the target control and the scaling factor includes: The target area is determined based on the coordinate position of the target control within the application page, and then expanded outward from the periphery of that area according to the expansion ratio to form the target area. The target area is cropped out from the application page to obtain a partial image.

5. The page testing method based on large model inspection according to claim 1, characterized in that, The process of magnifying the local image to obtain the target local image includes: Obtain the preset super-resolution model; The local map is input into the super-resolution model so that the super-resolution model reconstructs a high-resolution image on the local map and outputs the high-resolution image as the target local map.

6. The page testing method based on large model inspection according to claim 1, characterized in that, The step of inputting the target local image into a preset large model to obtain UI test inspection results for the operation event includes: Determine the semantic information and control type of the target control; Based on the business type of the application page and the semantic information and control type of the target control, determine the model prompt words; The model prompt and the target local image are input into a preset large model, so that the large model can perform verification analysis on the target local image based on the model prompt and output UI test inspection results for the operation event.

7. The page testing method based on large model inspection according to any one of claims 1 to 6, characterized in that, After obtaining the UI test inspection results for the operation event, the method further includes: Capture a full-screen image of the application page and evaluate the overall layout of the application page based on the full-screen image; Evaluate the business logic of the target control based on the target partial diagram; Based on the overall layout and business logic, the credibility of the UI test inspection results is determined. If the credibility does not reach the preset threshold, the page test method based on large model inspection is re-executed.

8. A page testing device based on large-scale model inspection, characterized in that, The device includes: The control determination module is used to determine the target control that the test task corresponding to the operation event is interested in when an operation event is captured in the target application. The partial capture module is used to obtain the current application page, determine the expansion ratio according to the target control, and capture a partial image in the application page according to the coordinate position of the target control and the expansion ratio. An image processing module is used to detect the resolution of the local image, and when the resolution is lower than a preset recognition threshold, to magnify the local image to obtain a target local image; The test inspection module is used to input the target local image into a preset large model to obtain the UI test inspection results for the operation event.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the page testing method based on large model inspection as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the page testing method based on large model inspection as described in any one of claims 1 to 7.

Images